FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(deepseek-v41): support text and vision serving (#603)
Xiaoze Fan127c85665b49
diff --git a/docs/models.md b/docs/models.mdindex 2cd3baa3..b999d29e 100644--- a/docs/models.md+++ b/docs/models.md@@ -5,6 +5,7 @@ for them; other checkpoints of the same architectures work too.| Model | HF checkpoints ||---|---|+| DeepSeek-V4.1-Flash | [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) || DeepSeek-V4 | [deepseek-ai/DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) || GLM-5.3-Flash | [RedHatAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4) || GLM-5.2 | [nvidia/GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) |@@ -27,6 +28,7 @@ These families accept image input by default; pass `--text-model-only` to skip t| Family | Image tokens | `--image-min-tokens` / `--image-max-tokens` | `--mm-processor-kwargs` example || --- | --- | --- | --- |+| DeepSeek-V4.1-Flash (ViT tower, streamed under `--mm-encoder-weights host`) | one feature per 42x42 pixels, plus a learned row break per row and two delimiters; checkpoint cap 1024 total span tokens | maximum caps the entire span; minimum maps to a pixel-area floor before the cap; checkpoint minimum area is 544x544 | `{"max_image_tokens": 512, "min_pixels": 295936}` || Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next, Qwen3-VL | one token per 32x32 pixels of the resized image, dynamic resolution | pixel areas in `size.shortest_edge` / `longest_edge`; checkpoint defaults 64 to 16384 tokens | `{"size": {"longest_edge": 1048576}}` || Gemma-4 26B-A4B, 31B (`gemma4`: ViT tower, streamed under `--mm-encoder-weights host`) | one of the soft-token budgets 70 / 140 / 280 / 560 / 1120, every image scaled to its budget as far as the aspect ratio allows | the maximum picks the largest budget within it, below 70 is refused at start-up; the minimum has no effect | `{"max_soft_tokens": 1120}` || Gemma-4 12B (`gemma4_unified`: linear patch embedder, resident under either placement) | same budgets, one 48x48 super-patch per soft token | same as the tower releases | same |@@ -86,4 +88,34 @@ unset on PCIe GPUs: wider grids add no bandwidth there.with `ft checkpoint`, or add the encoder in place with [scripts/ftw_hotfix.py](ftw-hotfix.md).- DeepSeek-V4 checkpoints must keep the `inference/config.json` subdir — theauthoritative model args are read from there.+- DeepSeek-V4.1-Flash serves text and images from the HF checkpoint as shipped (fp8 block-32+ dense weights, fp4 experts on the offload cache, 890 B/token global KV in packed+ fp4/fp8 pools). Its two 98 GiB Engram tables stream from the checkpoint shards on+ demand (keep them on a fast NVMe; the 6 GiB of table scales stay in host RAM), using+ bounded pinned staging buffers; an FTW conversion copies the shards that hold them next+ to the checkpoint.+ SWA bounded replay (DeepSeek_V41_Tech_Report.pdf, shipped in the checkpoint, §3.2.2): a replayed+ segment recomputes only the SWA KV of its last `n_win` tokens and truncates each query's window to+ the segment. *Encoder* replay rebuilds the encoder SWA KV behind a prefix hit from the global KV+ alone (approximate; the report's fallback when the SWA KV of a hit has been evicted). *Decoder*+ replay runs the decoder layers on each prompt's last `n_win` tokens only; their SWA KV is never+ prefix-cached and post-training simulated it. FreeToken keeps encoder SWA KV in the radix cache (no+ encoder replay); `--swa-decoder-replay bounded` (default) is the report's decoder replay, with a+ prefix hit stopping at least `n_win` tokens before the prompt end so those tokens are prefilled;+ `exact` runs the decoder on every prompt token (reference numerics).+ Bounded output differs from exact by construction and does not depend on the prefill chunk size.+ Under expert offload the prefill time is bounded by streaming each layer's experts, so decoder+ replay saves decoder-layer compute, not prefill time.+ Run `ft bench bw --model dsv4.1-flash` to measure local PCIe and CPU bandwidth before+ choosing a MoE strategy; `--moe-strategy hybrid` splits expert misses using those+ measurements. With `--moe-cache-auto`, `--kv-reserve-tokens` sets the token capacity reserved+ before the expert cache is allocated, subject to the attention pool's structural minimum; size it+ for the intended context and concurrency. `--max-extend-length` controls the prefill chunk size+ and its workspace. Image input uses the shared multimodal processor and encoder cache, including+ chunked prefill and prefix replay; `--text-model-only` skips the vision weights. DSpark+ speculative decoding is not served.+ `reasoning_effort` takes `low`, `high` or `max`; the 1-100 integer budget goes through the+ template kwargs with thinking on, `"chat_template_kwargs": {"enable_thinking": true,+ "reasoning_effort": 37}`, and the checkpoint's encoder validates it. Runtime window-cache+ controls use 128-token pages.- Qwen3.8-Flash-Next keeps a 47.7 GiB PLE n-gram table pinned in host RAM.diff --git a/python/freetoken/attention/__init__.py b/python/freetoken/attention/__init__.pyindex 8a410ffe..eda3b61f 100644--- a/python/freetoken/attention/__init__.py+++ b/python/freetoken/attention/__init__.py@@ -102,6 +102,16 @@ def create_dsv4_sparse_backend(config: ModelConfig):return DSV4SparseAttnBackend(config)+@SUPPORTED_ATTENTION_BACKENDS.register(+ "dsv41_sparse",+ BackendInfo(supported_types=frozenset({AttnType.DSV41})),+)+def create_dsv41_sparse_backend(config: ModelConfig):+ from .dsv41_sparse import DSV41SparseAttnBackend++ return DSV41SparseAttnBackend(config)++@SUPPORTED_ATTENTION_BACKENDS.register("dsa",BackendInfo(supported_types=frozenset({AttnType.MLA, AttnType.DSA})),diff --git a/python/freetoken/attention/base.py b/python/freetoken/attention/base.pyindex 9115b809..846e9944 100644--- a/python/freetoken/attention/base.py+++ b/python/freetoken/attention/base.py@@ -20,6 +20,9 @@ class AttnType(str, Enum):MLA = "mla" # plain latent-KV MLA -> MLAKVCacheDSA = "dsa" # latent-KV MLA + DSA sparse indexer -> DSAKVCacheDSV4 = "dsv4" # DSV4 window+compressed sparse -> DSV4PagedKVCache+ # DSV41 (DeepSeek-V4.1): window + cross-layer-shared compressed KV in packed fp8/fp4+ # rows, Full/Reindex/Reuse layer modes -> DSV41PagedKVCache+ DSV41 = "dsv41"LINEAR = "linear" # GDN/mamba state layers -> LinearStatePool# GQA block-sparse (MiniMax-M3): paged GQA K/V + a per-sparse-layer index-key# slab; the indexer picks top-k 128-token blocks per query -> BSAKVCachediff --git a/python/freetoken/attention/dsv41_sparse.py b/python/freetoken/attention/dsv41_sparse.pynew file mode 100644index 00000000..baaa2cf0--- /dev/null+++ b/python/freetoken/attention/dsv41_sparse.py@@ -0,0 +1,438 @@+"""DeepSeek-V4.1 sparse-attention backend.++Same contract and division of labour as ``dsv4_sparse``: the backend owns the per-forward KV+ADDRESSING over the shared page table (window ring slots, compressed rows, compress-state carry,+the decode snapshot and the CUDA-graph staging buffers) plus the selection helpers that are pure+index arithmetic (causal top-k, the hierarchical candidate pool); the model computes projections+and hands the picks back to be resolved into pool rows.++What DSV41 adds on top of DSV4's vocabulary:++* **cross-layer sharing** -- the compressed tiers belong to the kv-source layer; a consumer layer+ addresses ``source_of(layer)``'s pools. The Top-K rows and the candidate pool the Full / Reindex+ layers produce ride on the per-forward ``SharedSelection`` for the Reuse layers to read.+* **packed rows** -- writes quantize into the pool (``pool.store_*``), reads dequantize in-kernel.+* **prefill passes with a window floor** -- a ``PrefillSegment`` carries ``window_floor``: the+ absolute position a query's sliding window may not reach below (Decoder SWA Bounded Replay+ truncates the decoder's window at ``L - W``, the start of the prompt's last window). The metadata+ carries the encoder pass (every new token) and the decoder pass (the new tokens inside the+ prompt's last window, or every new token in exact mode).+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import TYPE_CHECKING, List++import torch+from freetoken.core import Batch, get_global_ctx++from .base import AttentionSpec, BaseAttnBackend, BaseAttnMetadata++if TYPE_CHECKING:+ from freetoken.models import ModelConfig+++@dataclass(frozen=True)+class PrefillSegment:+ """One request's slice of a flat prefill token stream.++ ``offset``/``n`` tile the stream; ``start_pos`` is the absolute position of the first token;+ ``window_floor`` bounds every query's sliding window from below (0 = the exact window; bounded+ replay sets it to the start of the prompt's last window).+ """++ offset: int+ n: int+ table_idx: int+ start_pos: int+ window_floor: int = 0++ @property+ def end(self) -> int:+ return self.start_pos + self.n+++@dataclass+class SharedSelection:+ """What the index-producing layers hand down the stack within ONE forward.++ ``topk_rows`` are GLOBAL main-pool rows (``-1`` masked) for the current kv source, laid out+ exactly as the attention kernel consumes them; ``cmp_counts`` is the decode-only per-row live+ count that bounds the kernel's loop; ``candidates`` is the Hierarchical Sparse Indexer's+ compressed-position pool (``-1`` = empty slot). Layers run in order and every producer writes+ before its consumers read, so one slot each suffices.+ """++ source: int | None = None+ topk_rows: torch.Tensor | None = None+ cmp_counts: torch.Tensor | None = None+ candidates: torch.Tensor | None = None+++@dataclass+class DSV41AttnMetadata(BaseAttnMetadata):+ last_indices: torch.Tensor+ # Prefill: the encoder pass tiles the batch's new tokens; the decoder pass tiles the tokens the+ # decoder layers process (``decoder_rows`` gathers them out of the encoder stream; None = all of+ # it). ``decoder_pad``: some request has no decoder row this chunk, and its ``last_indices`` entry+ # points at one placeholder row the model appends after the decoder stream (its logits are unused).+ segments: List[PrefillSegment] | None = None+ decoder_segments: List[PrefillSegment] | None = None+ decoder_rows: torch.Tensor | None = None+ decoder_pad: bool = False+ # Decode: the whole-history full-loc snapshot (captured buffer under a replay, lazy eager copy).+ full_snap: torch.Tensor | None = None+ table_rows: torch.Tensor | None = None+ window_ar: torch.Tensor | None = None+ selection: SharedSelection = field(default_factory=SharedSelection)++ def get_last_indices(self, bs: int) -> torch.Tensor:+ return self.last_indices[:bs]++ @property+ def stage_width(self) -> int:+ assert self.full_snap is not None, "stage_width is capture/replay-only"+ return self.full_snap.shape[1]++ def full_snapshot(self) -> torch.Tensor:+ if self.full_snap is None:+ assert self.table_rows is not None, "snapshot is decode-only"+ pool = get_global_ctx().kv_cache+ self.full_snap = pool.full_loc_map.index_select(0, self.table_rows).to(torch.int64)+ return self.full_snap++ def window_ctx(self, pos: torch.Tensor, rows: torch.Tensor):+ """Layer-invariant decode ring context ``(window_slots, prev_window_slots,+ window_slots_topk [B, 1, win])``; computed fresh on every call (never cached -- a capture+ must record these gathers), off the snapshot so a concurrent allocation cannot redirect+ an in-flight replay."""+ snap = self.full_snapshot()+ translate = get_global_ctx().kv_cache.translate_full_to_window+ bs = pos.shape[0]+ j = self.window_ar+ assert j is not None, "window_ctx is decode-only"+ win = j.shape[0]+ window_slots = translate(snap[rows, pos])+ prev_window_slots = translate(snap[rows, (pos - 1).clamp_min(0)])+ p = pos[:, None] - ((pos[:, None] - j[None, :]) % win)+ ws = translate(snap[rows[:, None], p.clamp(min=0)])+ window_slots_topk = torch.where((p >= 0) & (j[None, :] <= pos[:, None]), ws, -1).view(bs, 1, win)+ return window_slots, prev_window_slots, window_slots_topk++ def private_window_ctx(self, pos: torch.Tensor, rows: torch.Tensor):+ """The private-ring counterpart of ``window_ctx`` for request-private window layers:+ ``(window_slots [B], window_slots_topk [B, 1, win])`` in the layer's ring, where a request's+ row holds positions by ``pos % win`` (candidate ``j`` is the position congruent to ``j``)."""+ assert self.table_rows is not None and self.window_ar is not None, "private_window_ctx is decode-only"+ pool = get_global_ctx().kv_cache+ bs = pos.shape[0]+ j = self.window_ar+ win = j.shape[0]+ table = self.table_rows[rows]+ window_slots = pool.ring_slots(table, pos)+ p = pos[:, None] - ((pos[:, None] - j[None, :]) % win) # the position at ring index j (< 0: none yet)+ window_slots_topk = pool.ring_slots(table[:, None], p).view(bs, 1, win)+ return window_slots, window_slots_topk+++@dataclass+class DSV41CaptureData:+ full_snap: torch.Tensor+ last_indices: torch.Tensor+ table_rows: torch.Tensor # each batch row's page-table row (the private window rings key on it)++ @classmethod+ def create(cls, max_bs: int, width: int, device: torch.device) -> "DSV41CaptureData":+ return cls(+ full_snap=torch.full((max_bs, width), -1, dtype=torch.int64, device=device),+ last_indices=torch.arange(max_bs, dtype=torch.int32, device=device),+ table_rows=torch.zeros(max_bs, dtype=torch.int64, device=device),+ )+++class DSV41SparseAttnBackend(BaseAttnBackend):+ def __init__(self, config: ModelConfig):+ self.config = config+ self.device = get_global_ctx().kv_cache.device+ args = config.dsv41_args+ self.window_size = args.window_size+ self.swa_decoder_replay: str = args.swa_decoder_replay+ self.capture: DSV41CaptureData | None = None+ self.capture_bs: List[int] = []+ self.max_graph_bs = 0+ self._window_ar = torch.arange(self.window_size, device=self.device)++ @property+ def pool(self):+ return get_global_ctx().kv_cache++ # ----- generic contract -------------------------------------------------------------+ def forward(self, q, k, v, layer_id, batch, attn_spec: AttentionSpec | None = None):+ raise NotImplementedError("DSV41 attention is driven per-tier from the model module; use DSV41SparseAttnBackend.attend().")++ def prepare_metadata(self, batch: Batch) -> None:+ if not batch.is_decode:+ batch.attn_metadata = self._prefill_metadata(batch)+ return+ last = torch.tensor([r.extend_len for r in batch.padded_reqs], dtype=torch.int32, device=self.device).cumsum_(0) - 1+ batch.attn_metadata = DSV41AttnMetadata(last_indices=last, table_rows=self._table_rows(batch), window_ar=self._window_ar)++ def _prefill_metadata(self, batch: Batch) -> DSV41AttnMetadata:+ """The encoder pass runs every request's new tokens ``[cached_len, device_len)``. Exact mode runs+ the decoder on the same stream. Decoder SWA Bounded Replay runs it on the new tokens inside the+ prompt's last window ``[L - W, L)`` (``L = prompt_len``) with every query's window floored at+ ``L - W``: the chunks of a chunked prefill each take their part of that window (earlier rows+ come from the request's private ring), and a chunk that ends before it runs no decoder row."""+ segments: List[PrefillSegment] = []+ off = 0+ for r in batch.reqs:+ segments.append(PrefillSegment(off, r.extend_len, r.table_idx, r.cached_len))+ off += r.extend_len+ if self.swa_decoder_replay == "exact":+ last = torch.tensor([s.n for s in segments], dtype=torch.int32, device=self.device).cumsum_(0) - 1+ return DSV41AttnMetadata(last_indices=last, segments=segments, decoder_segments=segments)+ win = self.window_size+ dec: List[PrefillSegment] = []+ rows: list[torch.Tensor] = []+ last: list[int] = []+ doff = 0+ for r, s in zip(batch.reqs, segments):+ floor = max(0, r.prompt_len - win)+ lo = max(s.start_pos, floor)+ n = max(0, s.end - lo)+ if n:+ # the decoder's window KV goes to the request's private rings, never to radix-shared pages+ dec.append(PrefillSegment(doff, n, s.table_idx, lo, window_floor=floor))+ rows.append(torch.arange(s.offset + lo - s.start_pos, s.offset + s.n))+ doff += n+ last.append(doff - 1 if n else -1)+ pad = -1 in last+ last_indices = torch.tensor([doff if i < 0 else i for i in last], dtype=torch.int32, device=self.device)+ decoder_rows = torch.cat(rows).to(self.device, non_blocking=True) if rows else torch.empty(0, dtype=torch.int64, device=self.device)+ return DSV41AttnMetadata(last_indices=last_indices, segments=segments, decoder_segments=dec, decoder_rows=decoder_rows, decoder_pad=pad)++ def init_capture_graph(self, max_seq_len: int, bs_list: List[int]) -> None:+ assert self.capture is None, "Capture already initialized."+ self.max_graph_bs = max(bs_list)+ self.capture = DSV41CaptureData.create(self.max_graph_bs, max_seq_len, self.device)+ self.capture_bs = sorted(bs_list)++ def prepare_for_capture(self, batch: Batch) -> None:+ bs = batch.size+ rows = torch.full((bs,), batch.padded_reqs[0].table_idx, dtype=torch.int64, device=self.device)+ self._point_to_capture(batch, bs, rows)++ def prepare_for_replay(self, batch: Batch) -> None:+ self._point_to_capture(batch, batch.padded_size, self._table_rows(batch))++ def _table_rows(self, batch: Batch) -> torch.Tensor:+ assert batch.active_table_idx is not None, "decode batch is missing its page-table rows"+ return batch.active_table_idx.to(torch.int64)++ def _point_to_capture(self, batch: Batch, bs: int, rows_ti: torch.Tensor) -> None:+ assert self.capture is not None and bs <= self.max_graph_bs+ cap = self.capture+ src = self.pool.full_loc_map.index_select(0, rows_ti)+ w = min(src.shape[1], cap.full_snap.shape[1])+ cap.full_snap[:bs, :w].copy_(src[:bs, :w])+ cap.table_rows[:bs].copy_(rows_ti[:bs])+ batch.attn_metadata = DSV41AttnMetadata(+ last_indices=cap.last_indices[:bs], full_snap=cap.full_snap[:bs], table_rows=cap.table_rows[:bs], window_ar=self._window_ar,+ )++ # ----- metadata / selection access -------------------------------------------------+ @property+ def metadata(self) -> DSV41AttnMetadata:+ md = get_global_ctx().batch.attn_metadata+ assert isinstance(md, DSV41AttnMetadata)+ return md++ @property+ def selection(self) -> SharedSelection:+ return self.metadata.selection++ def snapshot(self) -> torch.Tensor:+ return self.metadata.full_snapshot()++ # ----- window tier -------------------------------------------------------------------+ def window_slots_of(self, ti: int, lo: int, hi: int) -> torch.Tensor:+ """SHARED window slots of positions ``[lo, hi)`` off the request's LIVE full locs (slid-out -> -1)."""+ return self.pool.translate_full_to_window(self.pool.full_loc_map[ti, lo:hi])++ def ring_slots_of(self, ti: int, lo: int, hi: int) -> torch.Tensor:+ """PRIVATE-ring slots of positions ``[lo, hi)`` of page-table row ``ti`` (``hi - lo <= win``)."""+ assert hi - lo <= self.window_size, f"a private ring holds one window, not {hi - lo} positions"+ return self.pool.ring_slots(torch.tensor(ti, device=self.device), torch.arange(lo, hi, device=self.device))++ def layer_window_slots_of(self, layer_id: int, ti: int, lo: int, hi: int) -> torch.Tensor:+ """Where ``layer_id`` keeps the window KV of positions ``[lo, hi)``: its private ring or the shared pool."""+ if self.pool.is_private_window(layer_id):+ return self.ring_slots_of(ti, lo, hi)+ return self.window_slots_of(ti, lo, hi)++ def store_window(self, kv: torch.Tensor, layer_id: int, window_slots: torch.Tensor) -> None:+ self.pool.store_window(kv, layer_id, window_slots)++ def window_topk_prefill(self, seg: PrefillSegment, layer_id: int | None = None) -> torch.Tensor:+ """Per-query window candidates for a prefill segment as GLOBAL window slots ``[1, n, win]``+ (``-1`` where empty) in the tier ``layer_id`` reads (shared pool by default). Query at+ absolute ``p`` sees ``[max(floor, first_retained, p - win + 1), p]``."""+ win, device = self.window_size, self.device+ lo = max(0, seg.start_pos - win + 1, seg.window_floor)+ private = layer_id is not None and self.pool.is_private_window(layer_id)+ ws_pool = (self.ring_slots_of if private else self.window_slots_of)(seg.table_idx, lo, seg.end) # [end - lo]+ abs_p = seg.start_pos + torch.arange(seg.n, device=device).unsqueeze(1)+ cand = (abs_p - win + 1).clamp(min=lo) + torch.arange(win, device=device)+ cols = torch.where(cand > abs_p, -1, cand - lo)+ g = ws_pool[cols.clamp_min(0)]+ return torch.where(cols < 0, -1, g).unsqueeze(0)++ # ----- compressed tiers (per kv source) ----------------------------------------------+ def source_of(self, layer_id: int) -> int:+ return self.pool.source_of(layer_id)++ def compressed_rows_of(self, ti: int, group_starts: torch.Tensor, ratio: int) -> torch.Tensor:+ """Main / index rows of the compressed groups whose ABSOLUTE first positions are+ ``group_starts``, off the request's LIVE full locs (a page is ratio-divisible, so every+ position of a group shares one row)."""+ return self.pool.cmp_rows(self.pool.full_loc_map[ti, group_starts], ratio)++ def locs_prefill(self, ti: int, end: int) -> torch.Tensor:+ """``[1, end]`` int32: the request's live full locs for positions ``[0, end)`` -- what the indexer+ derives compressed rows from (``loc // ratio``) for the positions its queries may see."""+ return self.pool.full_loc_map[ti : ti + 1, :end]++ def store_main(self, latent: torch.Tensor, source: int, rows: torch.Tensor) -> None:+ self.pool.store_main(latent, source, rows)++ def store_index(self, k: torch.Tensor, source: int, rows: torch.Tensor) -> None:+ self.pool.store_index(k, source, rows)++ def decode_store_rows(self, rows: torch.Tensor, pos: torch.Tensor, ratio: int, source: int, completed: torch.Tensor) -> torch.Tensor:+ """Per-row decode destination: the completed group's arithmetic row, or the row's OWN scratch+ row when this step did not complete a group (graph-safe, collision-free masked store)."""+ row_of_group = self.pool.cmp_rows(self.snapshot()[rows, pos], ratio)+ scratch = rows + self.pool.scratch_base[source]+ return torch.where(completed, row_of_group, scratch)++ # ----- compress-state ring (ratio > 1 sources) ---------------------------------------+ def ring_page_base(self, window_slots: torch.Tensor, ring_size: int) -> torch.Tensor:+ return torch.div(window_slots, self.window_size, rounding_mode="floor") * ring_size++ def carry_state_loc(self, window_slot: int, ring_size: int) -> torch.Tensor:+ base = (window_slot // self.window_size) * ring_size+ return torch.arange(base, base + ring_size, device=self.device, dtype=torch.int64)++ def read_carry(self, source: int, window_slot: int) -> torch.Tensor:+ """The ``[ring_size, 2 * head_dim]`` carry block at ``window_slot``'s page (kv | score)."""+ ring = self.pool.state_ring[source]+ return ring.get(self.carry_state_loc(window_slot, ring.ring_size))++ def write_carry(self, source: int, window_slot: int, kv_score: torch.Tensor) -> None:+ ring = self.pool.state_ring[source]+ ring.set(self.carry_state_loc(window_slot, ring.ring_size), kv_score)++ def read_carry_blocks(self, source: int, window_slots: torch.Tensor) -> torch.Tensor:+ ring = self.pool.state_ring[source]+ return ring.get_blocks(self.ring_page_base(window_slots, ring.ring_size))++ def write_carry_blocks(self, source: int, window_slots: torch.Tensor, blocks: torch.Tensor) -> None:+ ring = self.pool.state_ring[source]+ ring.set_blocks(self.ring_page_base(window_slots, ring.ring_size), blocks)++ def write_boundary_carries(self, source: int, *, lo: int, hi: int, window_slots: torch.Tensor) -> None:+ """Persist the compressor carry at every window-page boundary ``B`` in ``(lo, hi]`` so a+ page-aligned radix match can resume by value. A page holds whole groups (``P % ratio == 0``),+ so the carry at a boundary is the empty group -- the reset block is written so a resume never+ reads a stale one. ``window_slots`` is indexed by ``pos - lo``. One batched ring write, no host syncs."""+ ring = self.pool.state_ring.get(source)+ if ring is None:+ return+ P = self.window_size+ first = (lo // P + 1) * P+ if first > hi:+ return+ bounds = torch.arange(first, hi + 1, P, device=self.device)+ page_slots = window_slots[bounds - 1 - lo].to(torch.int64) # the last slot of each page+ locs = (torch.div(page_slots, P, rounding_mode="floor") * ring.ring_size)[:, None] + torch.arange(ring.ring_size, device=self.device)+ empty = torch.cat([+ torch.zeros(ring.ring_size, ring.item_size, dtype=torch.float32, device=self.device),+ torch.full((ring.ring_size, ring.item_size), float("-inf"), dtype=torch.float32, device=self.device),+ ], dim=-1)+ ring.set(locs.flatten(), empty.repeat(bounds.numel(), 1))++ # ----- indexer ----------------------------------------------------------------------+ def indexer_logits(+ self, q: torch.Tensor, weights: torch.Tensor, source: int, locs: torch.Tensor, ratio: int, live: torch.Tensor,+ T: int | None = None, candidates: torch.Tensor | None = None,+ ) -> torch.Tensor:+ """``[B, S, T]`` full-range scores (live columns written) or ``[B, S, NC]`` candidate scores; the+ row of compressed position ``t`` is ``locs[b, t * ratio] // ratio``."""+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed++ return indexer_logits_packed(q, weights, self.pool.idx_pool[source], self.pool.idx_fmt, locs, ratio, live, T=T, candidates=candidates)++ @staticmethod+ def select_topk(scores: torch.Tensor, live: torch.Tensor, topk: int) -> torch.Tensor:+ """Exact causal top-k over ``[B, S, T]`` full-range scores (columns at or past ``live`` unread):+ compressed positions ``[B, S, topk]`` int32 in ascending order, ``-1`` in the tail for picks+ that do not exist (fewer live or finite positions than ``topk``). Ties: lowest position wins."""+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk++ b, s, t = scores.shape+ return dsv41_topk(scores.reshape(b * s, t), live.reshape(b * s), topk).view(b, s, topk)++ @staticmethod+ def select_topk_in_candidates(scores: torch.Tensor, candidates: torch.Tensor, topk: int) -> torch.Tensor:+ """Top-k over ``[B, S, NC]`` candidate-aligned scores -> the picked compressed positions+ ``[B, S, topk]`` int32. The candidate list is a sorted valid prefix (``-1`` tail) and empty /+ unreachable slots score ``-inf``, so the ascending winning slots map to ascending positions."""+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk++ b, s, nc = scores.shape+ live = torch.full((b * s,), nc, dtype=torch.int32, device=scores.device)+ slots = dsv41_topk(scores.reshape(b * s, nc), live, topk).view(b, s, topk)+ pos = candidates.gather(-1, slots.clamp_min(0).to(torch.int64)).to(torch.int32)+ return torch.where(slots < 0, -1, pos)++ @staticmethod+ def select_candidate_blocks(scores: torch.Tensor, live: torch.Tensor, topk_blocks: int, block_size: int) -> torch.Tensor:+ """Level one of the Hierarchical Sparse Indexer: the ``topk_blocks`` best live ``block_size``+ blocks by their best position, the block holding the newest live position always kept,+ expanded to the compact candidate list ``[B, S, topk_blocks * block_size]`` int32 of+ compressed positions (ascending, ``-1`` tail). Mirrors the reference ``select_candidate_blocks``+ (which returns the equivalent boolean mask)."""+ from freetoken.kernel.triton.dsv41.topk import dsv41_candidate_blocks++ b, s, t = scores.shape+ return dsv41_candidate_blocks(scores.reshape(b * s, t), live.reshape(b * s), topk_blocks, block_size).view(b, s, -1)++ @staticmethod+ def positions_to_rows(positions: torch.Tensor, locs: torch.Tensor, ratio: int) -> torch.Tensor:+ """Compressed positions ``[B, S, K]`` (``-1`` masked) -> GLOBAL main rows ``locs[b, p * ratio] // ratio``."""+ b = positions.shape[0]+ idx = (positions.to(torch.int64) * ratio).clamp_min(0).flatten(1)+ full = locs.to(torch.int64).gather(1, idx.clamp(max=locs.shape[1] - 1)).view_as(positions)+ rows = torch.div(full, ratio, rounding_mode="floor")+ return torch.where((positions < 0) | (full < 0), -1, rows)++ # ----- attention -----------------------------------------------------------------------+ def attend(+ self, q: torch.Tensor, layer_id: int, topk_idxs: torch.Tensor, n_window: int, attn_sink: torch.Tensor,+ softmax_scale: float, cmp_counts: torch.Tensor | None = None,+ ) -> torch.Tensor:+ """Paged sparse attention over ``[window | compressed]`` global rows; the compressed half reads+ the layer's kv source (window-only layers pass ``n_window == topk``)."""+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed++ pool, args = self.pool, self.config.dsv41_args+ src = args.roles[layer_id].kv_source+ cmp = pool.main_pool[src] if src is not None else pool.main_pool[args.backbone_kv_sources[0]]+ return sparse_attn_packed(+ q, pool.window_pool[layer_id], pool.win_fmt, cmp, pool.main_fmt, attn_sink,+ topk_idxs, n_window, softmax_scale, cmp_counts=cmp_counts,+ )+++__all__ = ["DSV41SparseAttnBackend", "DSV41AttnMetadata", "PrefillSegment", "SharedSelection"]diff --git a/python/freetoken/core.py b/python/freetoken/core.pyindex fd691d18..45a763c0 100644--- a/python/freetoken/core.py+++ b/python/freetoken/core.py@@ -43,6 +43,8 @@ class Req:mm_items: list | None = Nonemrope_positions_full: torch.Tensor | None = None # [3, prompt_len] int32, CPUmrope_delta: int = 0+ # The whole prompt's length; 0 = this request's input_ids (a ChunkedReq's end at its chunk).+ prompt_len: int = 0# --- hybrid-radix (GDN linear-state) per-request slots; None for non-hybrid models or# until allocated from LinearStatePool. Set by the scheduler (P2). ---@@ -69,6 +71,7 @@ class Req:assert self.input_ids.is_cpuself.device_len = len(self.input_ids)self.max_device_len = len(self.input_ids) + self.output_len+ self.prompt_len = self.prompt_len or self.device_lenassert 0 <= self.cached_len < self.device_len <= self.max_device_lenself._alloc_ids_buf()diff --git a/python/freetoken/engine/cache_budget.py b/python/freetoken/engine/cache_budget.pyindex d522be49..b5f07ee8 100644--- a/python/freetoken/engine/cache_budget.py+++ b/python/freetoken/engine/cache_budget.py@@ -24,7 +24,7 @@ def expert_bytes_per_slot(sources: dict[str, "list[torch.Tensor]"]) -> int:# marlin/b12x gate_up/down alpha scales are fixed [L*E] residency (do not scale# with cache_size), so they are intentionally excluded from the per-slot growth term.# tensor[0].numel() is the per-row element count (one expert slot); see the matching- # slot-byte idiom in kvcache/linear_state_pool.py and kvcache/dsv4_paged_pool.py.+ # slot-byte idiom in kvcache/linear_state_pool.py and kvcache/dsv4/v4_pool.py.return sum(t[0][0].numel() * t[0].element_size() for t in sources.values())diff --git a/python/freetoken/engine/config.py b/python/freetoken/engine/config.pyindex 3e38cb21..b6fa8b89 100644--- a/python/freetoken/engine/config.py+++ b/python/freetoken/engine/config.py@@ -34,6 +34,9 @@ class EngineConfig:quant_backend: str | None = None# PLE table backend: "disk" (default) reads rows from the checkpoint files per fill, "pinned" preloads the table into page-locked host RAM.ple_backend: str = "disk"+ # DeepSeek-V4.1 Decoder SWA Bounded Replay: "bounded" (default) runs the decoder layers on each+ # prompt's last window; "exact" runs them on every token.+ swa_decoder_replay: str = "bounded"# Expert-bank host load (--expert-load): auto|serial|parallel. "auto" reads scattered# experts in parallel but falls back to serial when free RAM can't cover the banks + the# parallel reader's extra (non-reclaimable) whole-shard buffer; "serial" forces thediff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.pyindex e5ccf710..bf3960ab 100644--- a/python/freetoken/engine/engine.py+++ b/python/freetoken/engine/engine.py@@ -129,6 +129,8 @@ def _resolve_auto_attention_backend(required: frozenset[AttnType]) -> str:candidates: list[tuple[str, bool]] = []if AttnType.DSV4 in required:candidates.append(("dsv4_sparse", True))+ if AttnType.DSV41 in required:+ candidates.append(("dsv41_sparse", True))if required & {AttnType.MLA, AttnType.DSA}:candidates.append(("dsa", True))if AttnType.BSA in required:@@ -812,6 +814,7 @@ class Engine:# Decode batches never exceed max_running_req, but CUDA-graph padding can# round a batch up to the largest captured size; cover both.max_tokens = max(config.max_running_req, config.cuda_graph_max_bs or 0, 1)+ method = sample.quant_methodexecutor = CpuMoeExecutor(cache,top_k=sample.top_k,@@ -823,7 +826,8 @@ class Engine:swiglu_alpha=float(sample.alpha),swiglu_limit=sample.limit,# FIXME: the None branch serves GGUF q4_0 banks, which have no quant method yet; drop it once GGUF joins the quant path- fmt=sample.quant_method.cpu_format if sample.quant_method is not None else None,+ fmt=method.cpu_format if method is not None else None,+ act_block=method.cfg.act_block if method is not None else None,)cache.set_cpu_executor(executor)self.cpu_moe_executor = executor@@ -1200,6 +1204,39 @@ def _resolve_cache_type(has_linear_attention: bool, requested: str) -> str:return requested+def _adjust_dsv41_config(config: EngineConfig, override) -> None:+ """DeepSeek-V4.1 config reconciliation, the DSV4 policy with the replay knob: sync the+ runtime into ``dsv41_args``, page_size = the window page P, radix -> swa_radix, decode graph+ batches <= max_running_req. Unlike DSV4 the prefill chunk keeps ``max_extend_tokens`` (whole+ window pages): at a 1M ceiling the window pool alone would admit ~40K-token chunks whose+ activations (64 x 512 latent queries per token) do not fit next to the expert cache."""+ model_config = config.model_config+ args = model_config.dsv41_args+ args.max_seq_len = config.max_seq_len+ args.max_batch_size = config.max_running_req + 1 # +1 dummy+ args.swa_decoder_replay = config.swa_decoder_replay+ P = args.window_size+ override("page_size", P)+ logger.info_rank0(+ f"DSV41 KV pages are {P}-token window pages; page_size set to {P}; SWA decoder replay: {args.swa_decoder_replay}"+ + ("" if args.swa_decoder_replay == "exact" else f" (prefix hits stop {P} tokens before the prompt end)")+ )+ if getattr(config, "cache_type", "radix") != "naive":+ override("cache_type", "swa_radix")+ current = getattr(config, "max_extend_tokens", None) # a SchedulerConfig field+ if current is not None and (extend := max(P, current // P * P)) != current:+ override("max_extend_tokens", extend)+ mr = config.max_running_req+ if config.cuda_graph_max_bs is not None and config.cuda_graph_max_bs > mr:+ logger.warning_rank0(f"cuda_graph_max_bs {config.cuda_graph_max_bs} exceeds max_running_req {mr}; clamping (larger decode batches never occur).")+ override("cuda_graph_max_bs", mr)+ if config.cuda_graph_bs is not None:+ kept = [bs for bs in config.cuda_graph_bs if bs <= mr]+ if kept != list(config.cuda_graph_bs):+ logger.warning_rank0(f"dropping cuda_graph_bs entries above max_running_req {mr}: {[bs for bs in config.cuda_graph_bs if bs > mr]}")+ override("cuda_graph_bs", kept)++def _adjust_dsv4_config(config: EngineConfig, override) -> None:"""DSV4 engine-config reconciliation at config-resolution time (before the pool exists).Syncs the resolved runtime config into the opaque ``dsv4_args`` payload, sets@@ -1477,6 +1514,7 @@ def _adjust_config(config: EngineConfig):model_config = config.model_configsingle_stream_only = getattr(model_config, "single_stream_only", False)is_dsv4 = getattr(model_config, "dsv4_args", None) is not None+ is_dsv41 = getattr(model_config, "dsv41_args", None) is not Nonehas_swa_attention = getattr(model_config, "has_swa_attention", False)has_linear_attention = getattr(model_config, "has_linear_attention", False)is_moe = getattr(model_config, "is_moe", False)@@ -1520,6 +1558,8 @@ def _adjust_config(config: EngineConfig):if is_dsv4:_adjust_dsv4_config(config, override)+ if is_dsv41:+ _adjust_dsv41_config(config, override)if has_swa_attention:# Both SWA cache paths use the global-paged swa pool (page_size==1 only for now).@@ -1806,7 +1846,7 @@ def _adjust_config(config: EngineConfig):# DSV4 is exempt: it sizes its own table from the resolved max_seq_len (_adjust_dsv4_config).rotary = getattr(model_config, "rotary_config", None)seq_override = getattr(config, "max_seq_len_override", None)- if seq_override is not None and rotary is not None and not is_dsv4:+ if seq_override is not None and rotary is not None and not (is_dsv4 or is_dsv41):if seq_override > rotary.table_positions:raise ValueError(f"--max-seq-len-override {seq_override} exceeds the model's "diff --git a/python/freetoken/kernel/aot_models.py b/python/freetoken/kernel/aot_models.pyindex a8ef85d4..ecd1f850 100644--- a/python/freetoken/kernel/aot_models.py+++ b/python/freetoken/kernel/aot_models.py@@ -332,6 +332,17 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (expert_formats=("ds_fp4",),embed_indexing=False, # plain nn.Embedding),+ AotModel(+ # DSV41 pools are packed byte rows written by the dsv41 pack kernel, not store_cache;+ # the routed experts share DSV4's ds_fp4 bank layout (hidden 5120, intermediate 2304).+ name="deepseek-ai/DeepSeek-V4.1-Flash",+ architecture="DeepseekV41ForCausalLM",+ hidden_size=5120,+ kv_groups=(),+ top_k=6,+ moe_intermediate_size=2304,+ expert_formats=("ds_fp4",),+ ),# ---- dense checkpoints (store/index only, no expert banks) ----AotModel(name="Qwen/Qwen3.6-27B",diff --git a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cppindex fb6a33f4..86765013 100644--- a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp+++ b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp@@ -1063,11 +1063,12 @@ inline void deinterleave_bf16_f32(const bf16_t* src, float* even, float* odd, in}}-// DeepSeek-V4 activation FP8 round-trip (bf16 in/out): per 128-block,-// s = 2^ceil(log2(max(|x|,1e-4)/448)); y = round_e4m3(clamp(x/s,+-448)) * s.-void fp8_roundtrip_bf16(const bf16_t* src, bf16_t* dst, int K) {- for (int b0 = 0; b0 < K; b0 += 128) {- const int b1 = std::min(K, b0 + 128);+// DeepSeek-V4 / V4.1 activation FP8 round-trip (bf16 in/out): per `block` (the checkpoint's+// fp8 block: 128 on V4, 32 on V4.1), s = 2^ceil(log2(max(|x|,1e-4)/448));+// y = round_e4m3(clamp(x/s,+-448)) * s.+void fp8_roundtrip_bf16(const bf16_t* src, bf16_t* dst, int K, int block) {+ for (int b0 = 0; b0 < K; b0 += block) {+ const int b1 = std::min(K, b0 + block);float amax = 1e-4f;for (int i = b0; i < b1; ++i) amax = std::max(amax, std::fabs(bf16_to_f32(src[i])));const float s = std::ldexp(1.0f, ceil_log2_pos(amax * (1.0f / 448.0f)));@@ -1275,6 +1276,8 @@ struct CpuMoeExecutor {// it to a captured GPU elementwise kernel removes it while keeping the official// W4A8 numerics bit-exact. Set via set_input_prequant (see cpu_executor.py).bool input_prequant = false;+ // ds_fp4 activation round-trip block (the checkpoint's fp8 block); set via set_act_block.+ int act_block = 128;// Q4_0 packed-row byte strides (H/32*18 for gate_up over K=H, I/32*18 for down over K=I).int q4_gu_row_bytes = 0, q4_dn_row_bytes = 0;float e2m1_lut[16];@@ -1817,7 +1820,7 @@ struct CpuMoeExecutor {gas_scratch.data() + (size_t)r * (I / 32));return;}- if (fmt == WF_DSFP4) fp8_roundtrip_bf16(g, g, I);+ if (fmt == WF_DSFP4) fp8_roundtrip_bf16(g, g, I, act_block);float* ge = ge_scratch.data() + (size_t)r * (I / 2);float* go = go_scratch.data() + (size_t)r * (I / 2);deinterleave_bf16_f32(g, ge, go, I);@@ -1957,7 +1960,7 @@ struct CpuMoeExecutor {const bf16_t* src = t->x + (size_t)tok * H;if (ds) { // DSV4 FP8-round-trips the input before the gate_up GEMVbf16_t* xq = xq_scratch.data() + (size_t)tok * H;- fp8_roundtrip_bf16(src, xq, H);+ fp8_roundtrip_bf16(src, xq, H, act_block);src = xq;}float* xe = xe_scratch.data() + (size_t)tok * (H / 2);@@ -2185,6 +2188,9 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {.def("set_input_prequant",[](CpuMoeExecutor& e, bool v) { e.input_prequant = v; },py::arg("value"))+ .def("set_act_block",+ [](CpuMoeExecutor& e, int v) { e.act_block = v; },+ py::arg("value")).def("isa_name", &CpuMoeExecutor::isa_name).def("set_spin_wait",[](CpuMoeExecutor& e, bool v) { e.spin_wait.store(v, std::memory_order_relaxed); },diff --git a/python/freetoken/kernel/triton/dsv4/fp8_linear.py b/python/freetoken/kernel/triton/dsv4/fp8_linear.pyindex 47e3c9f5..fb17a351 100644--- a/python/freetoken/kernel/triton/dsv4/fp8_linear.py+++ b/python/freetoken/kernel/triton/dsv4/fp8_linear.py@@ -1,23 +1,25 @@-"""Block-scaled FP8 (e4m3) linear for DeepSeek-V4, matching the reference numerics.+"""Block-scaled FP8 (e4m3) linear for DeepSeek-V4 / V4.1, matching the reference numerics.The reference (``inference/model.py`` ``linear`` + ``inference/kernel.py``-``act_quant``/``fp8_gemm``) quantizes the *activation* to FP8 with a per-128 block+``act_quant``/``fp8_gemm``) quantizes the *activation* to FP8 with a per-``block``power-of-two (ue8m0) scale, then runs an FP8xFP8 block-scaled GEMM against the FP8-weight (which carries its own 128x128 ue8m0 block scale). Both operands' scales are-applied per 128-K block to a separate FP32 accumulator. This module reproduces that:+weight (which carries its own ``block x block`` ue8m0 scale). Both operands' scales are+applied per ``block``-K slab to a separate FP32 accumulator. This module reproduces that:- ``y = fp8_gemm(act_quant(x, 128, ue8m0), weight_fp8, weight_scale_e8m0)``+ ``y = fp8_gemm(act_quant(x, block, ue8m0), weight_fp8, weight_scale_e8m0)``++``block`` is the checkpoint's ``weight_block_size``: 128 for DeepSeek-V4, 32 for V4.1.``act_quant`` (reference): per block ``s = 2**ceil(log2(max(|x|,1e-4)/448))`` (exactvia IEEE bit ops -> matches ``fast_round_scale``), ``x_fp8 = round_e4m3(clamp(x/s,+-448))``, scale stored e8m0. The GEMM accumulates ``sum_k (A_fp8 @ B_fp8) * s_a * s_b``-per 128-K block in FP32.+per ``block``-K slab in FP32.Also provides ``act_quant_fp8_inplace`` -- the fused FP8 quant+dequant round-trip thereference applies in-place to the window / compressor KV (``act_quant(..., 64, ...,inplace=True)``), returning BF16.-Assumes ``K % 128 == 0`` and ``N % 128 == 0`` (true for every DeepSeek-V4 projection).+Assumes ``K % block == 0`` and ``N % block == 0`` (true for every DeepSeek-V4 / V4.1 projection)."""from __future__ import annotations@@ -218,8 +220,8 @@ def fp4_act_quant_inplace(x: torch.Tensor, block: int = 32) -> torch.Tensor:def _fp8_act_gemm_kernel(a_ptr, # [M, K] float8_e4m3fn (quantized activation)w_ptr, # [N, K] float8_e4m3fn- sa_ptr, # [M, K//128] uint8 (e8m0 act codes)- sb_ptr, # [N//128, K//128] uint8 (e8m0 weight codes)+ sa_ptr, # [M, K//BLOCK_K] uint8 (e8m0 act codes)+ sb_ptr, # [N//BLOCK_K, K//BLOCK_K] uint8 (e8m0 weight codes; square BLOCK_K blocks)c_ptr, # [M, N] compute dtypeM, N, K,stride_am, stride_ak, stride_wn, stride_wk,@@ -228,6 +230,11 @@ def _fp8_act_gemm_kernel(BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr,compute_type: tl.constexpr,):+ """``BLOCK_K`` is the weight's square block edge, so each K step maps to exactly one scale+ column. When the N tile IS one scale row (``BLOCK_N == BLOCK_K``, the DeepSeek-V4 block-128+ case) the weight scale is a scalar per step and the N masks vanish -- the original kernel;+ otherwise (block 32) the tile spans ``BLOCK_N // BLOCK_K`` scale rows and N is masked."""+ SCALAR_SCALE: tl.constexpr = BLOCK_N == BLOCK_Kpid_m = tl.program_id(0)pid_n = tl.program_id(1)offs_m = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)@@ -238,112 +245,217 @@ def _fp8_act_gemm_kernel(w_ptrs = w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :] * stride_wkacc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)num_k = tl.cdiv(K, BLOCK_K)- for k in range(num_k):- a = tl.load(a_ptrs, mask=m_mask[:, None], other=0.0)- w = tl.load(w_ptrs)- if e4m3_native_cx():- p = tl.dot(a, tl.trans(w), out_dtype=tl.float32)- else:- # bf16 dot on the same e4m3 grid: operands exact in bf16, fp32 acc- p = tl.dot(a, tl.trans(e4m3_u8_to_f32(w).to(tl.bfloat16)), out_dtype=tl.float32)- sa_code = tl.load(sa_ptr + offs_m * stride_sam + k * stride_sak, mask=m_mask, other=0)- sca = tl.exp2(sa_code.to(tl.float32) - 127.0) # [BLOCK_M]- sb_code = tl.load(sb_ptr + pid_n * stride_sbn + k * stride_sbk)- scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # scalar (one 128-N block)- acc += p * sca[:, None] * scb- a_ptrs += BLOCK_K * stride_ak- w_ptrs += BLOCK_K * stride_wk- c_ptrs = c_ptr + offs_m[:, None] * stride_cm + offs_n[None, :] * stride_cn- tl.store(c_ptrs, acc.to(compute_type), mask=m_mask[:, None])+ if SCALAR_SCALE:+ for k in range(num_k):+ a = tl.load(a_ptrs, mask=m_mask[:, None], other=0.0)+ w = tl.load(w_ptrs)+ if e4m3_native_cx():+ p = tl.dot(a, tl.trans(w), out_dtype=tl.float32)+ else:+ # bf16 dot on the same e4m3 grid: operands exact in bf16, fp32 acc+ p = tl.dot(a, tl.trans(e4m3_u8_to_f32(w).to(tl.bfloat16)), out_dtype=tl.float32)+ sa_code = tl.load(sa_ptr + offs_m * stride_sam + k * stride_sak, mask=m_mask, other=0)+ sca = tl.exp2(sa_code.to(tl.float32) - 127.0) # [BLOCK_M]+ sb_code = tl.load(sb_ptr + pid_n * stride_sbn + k * stride_sbk)+ scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # scalar (one N block)+ acc += p * sca[:, None] * scb+ a_ptrs += BLOCK_K * stride_ak+ w_ptrs += BLOCK_K * stride_wk+ c_ptrs = c_ptr + offs_m[:, None] * stride_cm + offs_n[None, :] * stride_cn+ tl.store(c_ptrs, acc.to(compute_type), mask=m_mask[:, None])+ else:+ n_mask = offs_n < N # N is a multiple of BLOCK_K, not necessarily of BLOCK_N+ sn = offs_n // BLOCK_K+ for k in range(num_k):+ a = tl.load(a_ptrs, mask=m_mask[:, None], other=0.0)+ w = tl.load(w_ptrs, mask=n_mask[:, None], other=0.0)+ if e4m3_native_cx():+ p = tl.dot(a, tl.trans(w), out_dtype=tl.float32)+ else:+ p = tl.dot(a, tl.trans(e4m3_u8_to_f32(w).to(tl.bfloat16)), out_dtype=tl.float32)+ sa_code = tl.load(sa_ptr + offs_m * stride_sam + k * stride_sak, mask=m_mask, other=0)+ sca = tl.exp2(sa_code.to(tl.float32) - 127.0) # [BLOCK_M]+ sb_code = tl.load(sb_ptr + sn * stride_sbn + k * stride_sbk, mask=n_mask, other=0)+ scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # [BLOCK_N]+ acc += p * sca[:, None] * scb[None, :]+ a_ptrs += BLOCK_K * stride_ak+ w_ptrs += BLOCK_K * stride_wk+ c_ptrs = c_ptr + offs_m[:, None] * stride_cm + offs_n[None, :] * stride_cn+ tl.store(c_ptrs, acc.to(compute_type), mask=m_mask[:, None] & n_mask[None, :])@triton.jit-def _fp8_act_gemv_splitk_kernel(- a_ptr, # [K] float8_e4m3fn- sa_ptr, # [K//128] uint8 (e8m0 act codes)- w_ptr, # [N, K] float8_e4m3fn- sb_ptr, # [N//128, K//128] uint8 (e8m0 weight codes)- part_ptr, # [SPLIT_K, N] fp32- N, K,- stride_ak, stride_wn, stride_wk, stride_sbn, stride_sbk, stride_pk, stride_pn,- BLOCK_N: tl.constexpr, SPLIT_K: tl.constexpr,+def _fp8_gemv_kernel(+ x_ptr, # [M, G, K] activation in the compute dtype (bf16); QUANT_ACT: quantized in-kernel per SLAB+ w_ptr, # [G * N, K] float8_e4m3fn (or its uint8 view on pre-Ada parts)+ sb_ptr, # [G * N // SLAB, K // SLAB] uint8 (e8m0 weight codes)+ out_ptr, # [M, G * N] OUT dtype (SPLIT_K == 1)+ part_ptr, # [SPLIT_K, M, G * N] fp32 partials (SPLIT_K > 1)+ M, N, K,+ stride_xm, stride_xg, stride_wn, stride_wk, stride_sbn, stride_sbk, stride_om, stride_pk, stride_pm,+ BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr, SLAB: tl.constexpr, SPLIT_K: tl.constexpr,+ QUANT_ACT: tl.constexpr, OUT: tl.constexpr,):+ """Small-M (decode) block-scaled fp8 GEMV/GEMM: one pass over the weight at near HBM bandwidth.++ Each program owns ``BLOCK_N`` rows and ``K // SPLIT_K`` contiguous K, walked ``BLOCK_K`` (several+ ``SLAB``-wide scale slabs) at a time: a wide tile keeps enough bytes in flight per SM to saturate+ the memory system, which a slab-at-a-time walk (512 B per load at the V4.1 slab of 32) cannot.+ Every weight tile is applied to all ``M`` (<= ``BLOCK_M``) activation rows, so a decode batch+ costs one weight read like a single token does (the tensor-core GEMM path is 3-6x slower at+ M = 2..8: too few tiles, tiny K steps). ``QUANT_ACT`` quantizes the activation here rather than+ by a separate ``act_quant`` launch: every program recomputes the ue8m0 scale and the e4m3 rounding+ of the slabs it touches (the reference formula, bit-identical to ``act_quant_fp8``) from the+ L2-resident bf16 rows, so the path is one kernel (plus a small reduce when ``SPLIT_K > 1``).+ Without it the activation stays bf16 (W8A16: the reference's bf16 einsum over a dequantized+ weight, e.g. ``wo_a``). The third grid axis is an independent group (block-diagonal weights:+ group ``g`` reads ``x[:, g]`` and rows ``[g * N, (g + 1) * N)``)."""+ tl.static_assert(BLOCK_K % SLAB == 0, "the K tile holds whole scale slabs")+ NSLAB: tl.constexpr = BLOCK_K // SLABpid_n = tl.program_id(0)pid_k = tl.program_id(1)+ g = tl.program_id(2).to(tl.int64)+ x_ptr += g * stride_xgoffs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)n_mask = offs_n < N- sn = offs_n // 128+ sn = (g * N + offs_n) // SLAB+ offs_n = g * N + offs_n+ rows = tl.arange(0, BLOCK_M)k_per = K // SPLIT_Kk_start = pid_k * k_per- acc = tl.zeros((BLOCK_N,), dtype=tl.float32)- for k0 in range(0, k_per, 128):- offs_k = k_start + k0 + tl.arange(0, 128)- a = tl.load(a_ptr + offs_k * stride_ak).to(tl.float32)+ acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)+ for k0 in range(0, k_per, BLOCK_K):+ offs_k = k_start + k0 + tl.arange(0, BLOCK_K)+ # --- the weight tile, dequantized per slab (read once for every activation row) ---w_ptrs = w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :] * stride_wkif e4m3_native_cx():w = tl.load(w_ptrs, mask=n_mask[:, None], other=0.0).to(tl.float32)else:w = e4m3_u8_to_f32(tl.load(w_ptrs, mask=n_mask[:, None], other=0))- kb = (k_start + k0) // 128- sb_code = tl.load(sb_ptr + sn * stride_sbn + kb * stride_sbk, mask=n_mask, other=0)- scb = tl.exp2(sb_code.to(tl.float32) - 127.0)- sa_code = tl.load(sa_ptr + kb)- sca = tl.exp2(sa_code.to(tl.float32) - 127.0)- acc += tl.sum(w * a[None, :], axis=1) * scb * sca- tl.store(part_ptr + pid_k * stride_pk + offs_n * stride_pn, acc, mask=n_mask)+ kb = (k_start + k0) // SLAB + tl.arange(0, NSLAB)+ sb_code = tl.load(sb_ptr + sn[:, None] * stride_sbn + kb[None, :] * stride_sbk, mask=n_mask[:, None], other=0)+ scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # [BLOCK_N, NSLAB]+ for m in tl.static_range(BLOCK_M):+ if m < M:+ xs = tl.reshape(tl.load(x_ptr + m * stride_xm + offs_k).to(tl.float32), (NSLAB, SLAB))+ if QUANT_ACT:+ # --- reference act_quant on this row's slabs: s = 2**ceil(log2(max(|x|, 1e-4) / 448)) ---+ amax = tl.maximum(tl.max(tl.abs(xs), axis=1), 1e-4)+ sa = tl.exp2(_log2_ceil(amax * (1.0 / 448.0)).to(tl.float32)) # [NSLAB]+ q = tl.clamp(xs / sa[:, None], -448.0, 448.0)+ if e4m3_native_cx():+ q = q.to(tl.float8e4nv).to(tl.float32)+ else:+ q = round_e4m3(q).to(tl.float32)+ else:+ sa = tl.full((NSLAB,), 1.0, tl.float32)+ q = xs+ a = tl.reshape(q, (BLOCK_K,))+ slab = tl.sum(tl.reshape(w * a[None, :], (BLOCK_N, NSLAB, SLAB)), axis=2) # [BLOCK_N, NSLAB]+ contrib = tl.sum(slab * scb * sa[None, :], axis=1) # [BLOCK_N]+ acc = tl.where(rows[:, None] == m, acc + contrib[None, :], acc)+ row_mask = rows < M+ if SPLIT_K == 1:+ tl.store(out_ptr + rows[:, None] * stride_om + offs_n[None, :], acc.to(OUT), mask=row_mask[:, None] & n_mask[None, :])+ else:+ tl.store(part_ptr + pid_k * stride_pk + rows[:, None] * stride_pm + offs_n[None, :], acc, mask=row_mask[:, None] & n_mask[None, :])@triton.jit-def _splitk_reduce_kernel(part_ptr, out_ptr, N, SPLIT_K: tl.constexpr,- stride_pk, stride_pn, BLOCK: tl.constexpr, OUT: tl.constexpr):+def _splitk_reduce_kernel(part_ptr, out_ptr, M, N, SPLIT_K: tl.constexpr,+ stride_pk, stride_pm, stride_om, BLOCK: tl.constexpr, OUT: tl.constexpr):+ m = tl.program_id(1)offs = tl.program_id(0) * BLOCK + tl.arange(0, BLOCK)mask = offs < Nacc = tl.zeros((BLOCK,), dtype=tl.float32)for k in tl.static_range(SPLIT_K):- acc += tl.load(part_ptr + k * stride_pk + offs * stride_pn, mask=mask, other=0.0)- tl.store(out_ptr + offs, acc.to(OUT), mask=mask)---# Swept-best decode GEMV config per (N, K) -> (BLOCK_N, SPLIT_K, num_warps).-_DECODE_FP8_CFG = {- (1024, 4096): (16, 16, 1), # wq_a- (32768, 1024): (16, 1, 2), # wq_b- (512, 4096): (16, 16, 1), # attn wkv- (4096, 8192): (16, 8, 1), # wo_b- (2048, 4096): (16, 8, 1), # shared w1 / w3- (4096, 2048): (16, 8, 1), # shared w2- (8192, 1024): (16, 2, 4), # indexer wq_b-}---def _decode_cfg(N: int, K: int) -> tuple[int, int, int]:- cfg = _DECODE_FP8_CFG.get((N, K))- if cfg is not None:- bn, sk, nw = cfg- return bn, max(1, min(sk, K // 128)), nw- bn = 16- n_tiles = triton.cdiv(N, bn)- sk = max(1, 1536 // n_tiles)- sk = 1 << (sk.bit_length() - 1)- return bn, max(1, min(sk, K // 128)), 1---def _fp8_act_gemv(a_fp8: torch.Tensor, sa: torch.Tensor, weight: torch.Tensor,- sb: torch.Tensor, out_dtype: torch.dtype) -> torch.Tensor:- N, K = weight.shape- BLOCK_N, split_k, num_warps = _decode_cfg(N, K)- n_tiles = triton.cdiv(N, BLOCK_N)- part = torch.empty((split_k, N), dtype=torch.float32, device=a_fp8.device)- _fp8_act_gemv_splitk_kernel[(n_tiles, split_k)](- a_fp8, sa, weight, sb, part, N, K,- a_fp8.stride(0), weight.stride(0), weight.stride(1),- sb.stride(0), sb.stride(1), part.stride(0), part.stride(1),- BLOCK_N=BLOCK_N, SPLIT_K=split_k, num_warps=num_warps,+ acc += tl.load(part_ptr + k * stride_pk + m * stride_pm + offs, mask=mask, other=0.0)+ tl.store(out_ptr + m * stride_om + offs, acc.to(OUT), mask=mask)+++_GEMV_BLOCK_N = 32+_GEMV_NARROW_BLOCK_N = 8+def _gemv_cfg(N: int, K: int, slab: int, sm_count: int) -> tuple[int, int, int, int]:+ """Choose whole scale tiles and enough independent programs to cover the device."""+ block_k = next((bk for bk in (512, 256, 128) if bk >= slab and K % bk == 0), slab)+ if triton.cdiv(N, _GEMV_BLOCK_N) >= sm_count:+ return _GEMV_BLOCK_N, block_k, 1, 4+ n_tiles = triton.cdiv(N, _GEMV_NARROW_BLOCK_N)+ split = 1+ while n_tiles * split < sm_count and (K // block_k) % (split * 2) == 0:+ split *= 2+ return _GEMV_NARROW_BLOCK_N, block_k, split, 4+++GEMV_MAX_M = 4 # bound the unrolled row loop's register footprint+++def _fp8_gemv(+ x: torch.Tensor, weight: torch.Tensor, sb: torch.Tensor, out_dtype: torch.dtype, block: int, *, groups: int = 1, quant_act: bool = True,+) -> torch.Tensor:+ """``x [M, groups, K]`` bf16 x block-diagonal ``weight [groups * N, K]`` fp8 -> ``[M, groups * N]``+ in ``out_dtype`` (``M <= GEMV_MAX_M``). ``quant_act`` applies the reference fp8 activation quant+ (W8A8); off, the activation stays bf16 (W8A16). ``groups == 1`` is the plain GEMV."""+ GN, K = weight.shape+ N = GN // groups+ x = x.reshape(-1, groups, K)+ M = x.shape[0]+ assert 1 <= M <= GEMV_MAX_M, M+ block_m = triton.next_power_of_2(M)+ block_n, block_k, split_k, num_warps = _gemv_cfg(GN, K, block, torch.cuda.get_device_properties(x.device).multi_processor_count)+ out = torch.empty((M, GN), dtype=out_dtype, device=x.device)+ part = out if split_k == 1 else torch.empty((split_k, M, GN), dtype=torch.float32, device=x.device)+ _fp8_gemv_kernel[(triton.cdiv(N, block_n), split_k, groups)](+ x, weight, sb, out, part, M, N, K,+ x.stride(0), x.stride(1), weight.stride(0), weight.stride(1), sb.stride(0), sb.stride(1), out.stride(0),+ part.stride(0) if split_k > 1 else 0, part.stride(1) if split_k > 1 else 0,+ BLOCK_M=block_m, BLOCK_N=block_n, BLOCK_K=block_k, SLAB=block, SPLIT_K=split_k, QUANT_ACT=quant_act, OUT=_TL_DTYPE[out_dtype],+ num_warps=num_warps,)- out = torch.empty(N, dtype=out_dtype, device=a_fp8.device)- _splitk_reduce_kernel[(triton.cdiv(N, 256),)](- part, out, N, split_k, part.stride(0), part.stride(1),- BLOCK=256, OUT=_TL_DTYPE[out_dtype], num_warps=2,+ if split_k > 1:+ _splitk_reduce_kernel[(triton.cdiv(GN, 256), M)](+ part, out, M, GN, split_k, part.stride(0), part.stride(1), out.stride(0),+ BLOCK=256, OUT=_TL_DTYPE[out_dtype], num_warps=2,+ )+ return out+++def grouped_w8a16_gemv(x: torch.Tensor, weight: torch.Tensor, scale: torch.Tensor, *, block: int) -> torch.Tensor:+ """Decode form of the reference's grouped bf16 einsum ``tgd,grd->tgr`` over a block-diagonal fp8+ weight: ``x [t, G, d]`` bf16 (``t <= GEMV_MAX_M``), ``weight [G * r, d]`` fp8 with ``[G * r //+ block, d // block]`` e8m0 ``scale`` -> ``[t, G * r]`` bf16. The weight is dequantized in-kernel+ (exact in bf16: e4m3 x 2^k), the activation is not quantized, so this is the einsum's math with a+ different fp32 summation order."""+ assert x.ndim == 3 and weight.dtype == FP8+ G = x.shape[1]+ sb = (scale.view(torch.uint8) if scale.dtype == torch.float8_e8m0fnu else scale).contiguous()+ return _fp8_gemv(x.contiguous(), e4m3_kernel_view(weight), sb, x.dtype, block, groups=G, quant_act=False)+++def _dequant_block_fp8_kernel(w_ptr, sb_ptr, out_ptr, N, K, stride_wn, stride_sbn, stride_on, SLAB: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr):+ pid_n = tl.program_id(0)+ pid_k = tl.program_id(1)+ offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)+ offs_k = pid_k * BLOCK_K + tl.arange(0, BLOCK_K)+ mask = (offs_n < N)[:, None] & (offs_k < K)[None, :]+ if e4m3_native_cx():+ w = tl.load(w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :], mask=mask, other=0.0).to(tl.float32)+ else:+ w = e4m3_u8_to_f32(tl.load(w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :], mask=mask, other=0))+ code = tl.load(sb_ptr + (offs_n // SLAB)[:, None] * stride_sbn + (offs_k // SLAB)[None, :], mask=mask, other=0)+ tl.store(out_ptr + offs_n[:, None] * stride_on + offs_k[None, :], (w * tl.exp2(code.to(tl.float32) - 127.0)).to(tl.bfloat16), mask=mask)+++def dequant_block_fp8(weight: torch.Tensor, scale: torch.Tensor, *, block: int) -> torch.Tensor:+ """``[N, K]`` block-scaled e4m3 -> bf16 (exact), one pass; the prefill-side companion of+ :func:`grouped_w8a16_gemv` for weights the reference consumes through a bf16 einsum."""+ N, K = weight.shape+ sb = (scale.view(torch.uint8) if scale.dtype == torch.float8_e8m0fnu else scale).contiguous()+ out = torch.empty((N, K), dtype=torch.bfloat16, device=weight.device)+ _dequant_block_fp8_kernel[(triton.cdiv(N, 64), triton.cdiv(K, 256))](+ e4m3_kernel_view(weight), sb, out, N, K, weight.stride(0), sb.stride(0), out.stride(0),+ SLAB=block, BLOCK_N=64, BLOCK_K=256, num_warps=4,)return out@@ -353,36 +465,41 @@ def block_fp8_linear(weight: torch.Tensor,scale: torch.Tensor,bias: torch.Tensor | None = None,+ *,+ block: int = 128,) -> torch.Tensor:"""``y = act_quant(x) @ weight^T`` (reference FP8 path).``x``: ``[..., K]`` bf16; ``weight``: ``[N, K]`` float8_e4m3fn; ``scale``:- ``[N//128, K//128]`` float8_e8m0fnu (weight block scale). Activation is quantized- to FP8 with a per-128 ue8m0 scale; the GEMM applies both scales per 128-K block.+ ``[N//block, K//block]`` float8_e8m0fnu (weight block scale). Activation is quantized+ to FP8 with a per-``block`` ue8m0 scale; the GEMM applies both scales per ``block``-K slab."""assert weight.dtype == FP8*lead, K = x.shapeN = weight.shape[0]assert weight.shape[1] == K- assert K % 128 == 0 and N % 128 == 0, (N, K)+ assert K % block == 0 and N % block == 0, (N, K, block)+ assert tuple(scale.shape) == (N // block, K // block), (tuple(scale.shape), N, K, block)compute_dtype = x.dtype if x.dtype in _TL_DTYPE else torch.bfloat16sb = scale.view(torch.uint8) if scale.dtype == torch.float8_e8m0fnu else scalesb = sb.contiguous()w = e4m3_kernel_view(weight)- a_fp8, sa = act_quant_fp8(x, 128) # [M,K] fp8, [M,K//128] e8m0 codes- M = a_fp8.shape[0]-- if M == 1:- out = _fp8_act_gemv(a_fp8[0], sa[0], w, sb, compute_dtype).reshape(*lead, N)+ x2d = x.reshape(-1, K)+ if x2d.shape[0] == 1:+ # decode: one kernel quantizes the activation rows and streams the weight once+ out = _fp8_gemv(x2d.contiguous(), w, sb, compute_dtype, block).reshape(*lead, N)if bias is not None:out = out + bias.to(out.dtype)return out+ a_fp8, sa = act_quant_fp8(x, block) # [M,K] fp8, [M,K//block] e8m0 codes+ M = a_fp8.shape[0]+out = torch.empty((M, N), dtype=compute_dtype, device=x.device)BLOCK_M = 32BLOCK_N = 128- BLOCK_K = 128+ BLOCK_K = blockgrid = (triton.cdiv(M, BLOCK_M), triton.cdiv(N, BLOCK_N))_fp8_act_gemm_kernel[grid](a_fp8, w, sa, sb, out,@@ -399,4 +516,4 @@ def block_fp8_linear(return out-__all__ = ["block_fp8_linear", "act_quant_fp8", "act_quant_fp8_inplace", "fp4_act_quant_inplace"]+__all__ = ["block_fp8_linear", "act_quant_fp8", "act_quant_fp8_inplace", "fp4_act_quant_inplace", "grouped_w8a16_gemv", "dequant_block_fp8", "GEMV_MAX_M"]diff --git a/python/freetoken/kernel/triton/dsv4/fused_moe.py b/python/freetoken/kernel/triton/dsv4/fused_moe.pyindex 6eea606c..0adfbdd3 100644--- a/python/freetoken/kernel/triton/dsv4/fused_moe.py+++ b/python/freetoken/kernel/triton/dsv4/fused_moe.py@@ -24,6 +24,9 @@ import torchimport tritonimport triton.language as tl+from freetoken.kernel.triton.dsv4.fp8_linear import _log2_ceil+from freetoken.kernel.triton.e4m3_compat import e4m3_native_cx, round_e4m3+_E2M1_VALUES = [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0,-0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0,@@ -71,6 +74,12 @@ def _decode_dsfp4_moe_kernel(n_mask = offs_n < Nslot = tl.load(topk_ids_ptr + token_id * stride_tid_m + route_k * stride_tid_k).to(tl.int64)+ c_ptrs = c_ptr + token_id * stride_cm + route_k * stride_ck + offs_n * stride_cn+ if slot < 0:+ # an inactive route (hybrid decode computes it on the CPU): zero contribution, and no read of+ # any slot -- a slot's storage may be uninitialized, and 0 * NaN is not 0+ tl.store(c_ptrs, tl.zeros((BLOCK_SIZE_N,), dtype=compute_type), mask=(route_id < total_routes) & n_mask)+ returna_row = route_id if A_ROW_IS_ROUTE else token_ida_base = a_ptr + a_row * stride_am@@ -113,7 +122,6 @@ def _decode_dsfp4_moe_kernel(weight = tl.load(topk_weights_ptr + token_id * stride_tw_m + route_k * stride_tw_k)accumulator = accumulator * weight- c_ptrs = c_ptr + token_id * stride_cm + route_k * stride_ck + offs_n * stride_cntl.store(c_ptrs, accumulator.to(compute_type), mask=(route_id < total_routes) & n_mask)@@ -124,11 +132,15 @@ def _swiglu_kernel(R, I, limit,stride_gr, stride_gi, stride_or, stride_oi,BLOCK: tl.constexpr, HAS_LIMIT: tl.constexpr, compute_type: tl.constexpr,+ ACT_BLOCK: tl.constexpr, # > 0: fp8 (e4m3, ue8m0 per ACT_BLOCK) round-trip of the stored value, fused):"""Fused SwiGLU: ``out = silu(min(gate, limit)) * clamp(up, -limit, limit)``.Collapses the gate/up split + two clamps + silu + mul + the fp32 round-trip- (6 elementwise launches over [R, I]) between the two FP4 GEMVs into one pass."""+ (6 elementwise launches over [R, I]) between the two FP4 GEMVs into one pass. With+ ``ACT_BLOCK`` the down projection's activation quant (``act_quant_fp8_inplace`` on the bf16+ value: ``s = 2**ceil(log2(max(|x|, 1e-4) / 448))``, e4m3 rounding, dequant) is applied to the+ same tile before the store -- one launch and one memory pass fewer, bit-identical."""row = tl.program_id(0)cb = tl.program_id(1)offs = cb * BLOCK + tl.arange(0, BLOCK)@@ -139,6 +151,17 @@ def _swiglu_kernel(g = tl.minimum(g, limit)u = tl.minimum(tl.maximum(u, -limit), limit)act = (g * tl.sigmoid(g)) * u+ if ACT_BLOCK > 0:+ tl.static_assert(BLOCK % ACT_BLOCK == 0, "the tile holds whole quant blocks")+ xs = tl.reshape(act.to(compute_type).to(tl.float32), (BLOCK // ACT_BLOCK, ACT_BLOCK)) # the value the store would hold+ amax = tl.maximum(tl.max(tl.abs(xs), axis=1), 1e-4)+ sc = tl.exp2(_log2_ceil(amax * (1.0 / 448.0)).to(tl.float32))+ q = tl.clamp(xs / sc[:, None], -448.0, 448.0)+ if e4m3_native_cx():+ q = q.to(tl.float8e4nv).to(tl.float32)+ else:+ q = round_e4m3(q)+ act = tl.reshape(q * sc[:, None], (BLOCK,))tl.store(out_ptr + row * stride_or + offs * stride_oi, act.to(compute_type), mask=mask)@@ -274,20 +297,22 @@ def _compute_type(dtype: torch.dtype):return {torch.bfloat16: tl.bfloat16, torch.float16: tl.float16, torch.float32: tl.float32}[dtype]-def fused_swiglu(gate_up: torch.Tensor, limit: float) -> torch.Tensor:- """``[..., 2I] -> [..., I]`` SwiGLU in a single kernel (see ``_swiglu_kernel``)."""+def fused_swiglu(gate_up: torch.Tensor, limit: float, *, act_block: int = 0) -> torch.Tensor:+ """``[..., 2I] -> [..., I]`` SwiGLU in a single kernel (see ``_swiglu_kernel``). ``act_block > 0``+ also applies the down projection's fp8 activation round-trip (block ``act_block``) in the same pass."""*lead, two_I = gate_up.shapeI = two_I // 2gu = gate_up.reshape(-1, two_I)R = gu.shape[0]out = torch.empty((R, I), dtype=gate_up.dtype, device=gate_up.device)BLOCK = 1024+ assert act_block == 0 or (I % act_block == 0 and BLOCK % act_block == 0), (I, act_block)grid = (R, triton.cdiv(I, BLOCK))_swiglu_kernel[grid](gu, out, R, I, float(limit),gu.stride(0), gu.stride(1), out.stride(0), out.stride(1),BLOCK=BLOCK, HAS_LIMIT=limit > 0,- compute_type=_compute_type(gate_up.dtype), num_warps=4,+ compute_type=_compute_type(gate_up.dtype), ACT_BLOCK=act_block, num_warps=4,)return out.reshape(*lead, I)diff --git a/python/freetoken/kernel/triton/dsv4/router.py b/python/freetoken/kernel/triton/dsv4/router.pynew file mode 100644index 00000000..17e3506a--- /dev/null+++ b/python/freetoken/kernel/triton/dsv4/router.py@@ -0,0 +1,83 @@+"""Fused MoE gate selection (DeepSeek-V4 / V4.1 ``Gate``): score activation, selection bias, top-k,+weight normalization and route scale in one program per token.++ s = act(scores) act: softmax | sigmoid | sqrt(softplus)+ ids = topk(s + bias) selection is biased, the weights are not+ w = s[ids] [/ (sum + 1e-20)] * route_scale++One launch replaces the eight or so elementwise / top-k / gather launches of the torch expression --+at decode each of those moves a few hundred floats and costs its launch floor. Ties in ``s + bias``+go to the lowest expert id (torch.topk leaves them unspecified).+"""++from __future__ import annotations++import torch+import triton+import triton.language as tl+from triton.language.extra import libdevice++_SCORE = {"softmax": 0, "sigmoid": 1, "sqrtsoftplus": 2}+++def _router_select_kernel(+ scores_ptr, bias_ptr, alternate_bias_ptr, alternate_mask_ptr, w_ptr, id_ptr, E, route_scale,+ stride_s, stride_w, stride_id,+ SCORE: tl.constexpr, TOPK: tl.constexpr, BLOCK_K: tl.constexpr, NORMALIZE: tl.constexpr, BLOCK_E: tl.constexpr, ALTERNATE: tl.constexpr,+):+ t = tl.program_id(0).to(tl.int64)+ offs = tl.arange(0, BLOCK_E)+ mask = offs < E+ s = tl.load(scores_ptr + t * stride_s + offs, mask=mask, other=float("-inf")).to(tl.float32)+ if SCORE == 0:+ s = tl.exp(s - tl.max(s))+ s = s / tl.sum(tl.where(mask, s, 0.0))+ elif SCORE == 1:+ s = tl.sigmoid(s)+ else:+ s = tl.sqrt(tl.where(s > 20.0, s, libdevice.log1p(tl.exp(s)))) # torch softplus (threshold 20)+ if ALTERNATE:+ bias_ptr = tl.where(tl.load(alternate_mask_ptr + t), alternate_bias_ptr, bias_ptr)+ bias = tl.load(bias_ptr + offs, mask=mask, other=0.0).to(tl.float32)+ key = tl.where(mask, s + bias, float("-inf"))+ slots = tl.arange(0, BLOCK_K)+ picked_w = tl.zeros((BLOCK_K,), dtype=tl.float32)+ picked_id = tl.zeros((BLOCK_K,), dtype=tl.int32)+ for k in tl.static_range(TOPK):+ best = tl.max(key)+ idx = tl.min(tl.where(key == best, offs, BLOCK_E)) # lowest id among ties+ w = tl.sum(tl.where(offs == idx, s, 0.0))+ picked_w = tl.where(slots == k, w, picked_w)+ picked_id = tl.where(slots == k, idx.to(tl.int32), picked_id)+ key = tl.where(offs == idx, float("-inf"), key)+ if NORMALIZE:+ picked_w = picked_w / (tl.sum(picked_w) + 1e-20)+ picked_w = picked_w * route_scale+ tl.store(w_ptr + t * stride_w + slots, picked_w, mask=slots < TOPK)+ tl.store(id_ptr + t * stride_id + slots, picked_id, mask=slots < TOPK)+++def router_select(+ scores: torch.Tensor, bias: torch.Tensor, topk: int, *, score_func: str, normalize: bool, route_scale: float,+ alternate_bias: torch.Tensor | None = None, alternate_mask: torch.Tensor | None = None,+) -> tuple[torch.Tensor, torch.Tensor]:+ """``scores [T, E]`` fp32 (pre-activation gate logits), ``bias [E]`` -> ``(weights [T, topk] fp32,+ ids [T, topk] int32)`` in selection order (best first)."""+ T, E = scores.shape+ assert scores.dtype == torch.float32 and scores.stride(1) == 1, (scores.dtype, scores.stride())+ weights = torch.empty((T, topk), dtype=torch.float32, device=scores.device)+ ids = torch.empty((T, topk), dtype=torch.int32, device=scores.device)+ if T == 0:+ return weights, ids+ _router_select_kernel[(T,)](+ scores, bias, alternate_bias if alternate_bias is not None else bias,+ alternate_mask if alternate_mask is not None else scores, weights, ids, E, float(route_scale),+ scores.stride(0), weights.stride(0), ids.stride(0),+ SCORE=_SCORE[score_func], TOPK=topk, BLOCK_K=triton.next_power_of_2(topk), NORMALIZE=normalize and topk > 1,+ BLOCK_E=triton.next_power_of_2(E), ALTERNATE=alternate_mask is not None, num_warps=4,+ )+ return weights, ids+++__all__ = ["router_select"]diff --git a/python/freetoken/kernel/triton/dsv41/__init__.py b/python/freetoken/kernel/triton/dsv41/__init__.pynew file mode 100644index 00000000..00ec72da--- /dev/null+++ b/python/freetoken/kernel/triton/dsv41/__init__.py@@ -0,0 +1,9 @@+"""Triton kernels for DeepSeek-V4.1 Compressed Sparse Attention 2 over packed KV pools.++``row_format`` device helpers: load a tile of packed rows (bf16 / fp8+e8m0 / fp4+e4m3 / fp4+e8m0)+ and dequantize it to fp32, encode / decode e2m1 nibbles.+``pack`` host wrappers: quantize rows into a packed pool (``pack_rows``) and read them back+ (``unpack_rows``), matching the reference ``act_quant`` / ``fp4_act_quant`` numerics.+``sparse_attn`` paged sparse attention gathering window (fp8) and compressed (fp4) rows.+``indexer`` Lightning-indexer logits over fp4 index keys, full-range or candidate-restricted.+"""diff --git a/python/freetoken/kernel/triton/dsv41/indexer.py b/python/freetoken/kernel/triton/dsv41/indexer.pynew file mode 100644index 00000000..37213158--- /dev/null+++ b/python/freetoken/kernel/triton/dsv41/indexer.py@@ -0,0 +1,176 @@+"""Lightning-Indexer logits over PACKED index keys (DeepSeek-V4.1, Full and Reindex modes).++ logits[b, s, j] = sum_h relu(q[b, s, h, :] . k(b, pos_j)) * weights[b, s, h]++``q`` carries the ``index_n_heads`` fp4-baked index queries, ``k`` is the single shared index key of+a compressed position, gathered from the kv-source's packed pool and dequantized in-kernel. The+row of compressed position ``t`` is derived in-kernel from the request's full-token locs (the page+table row at prefill, the decode snapshot): ``locs[b, t * ratio] // ratio`` (negative = no row) --+no ``[B, T]`` row table is materialized per layer. ``weights`` folds ``softmax_scale *+n_heads**-0.5`` in.++Two position sets, one kernel:++* **full range** (Full mode): position ``j`` is the compressed position ``j``; the output is+ ``[B, S, T]`` over the staged width ``T``, but only the live tiles (columns below ``live[b, s]``+ rounded up to ``BLOCK_T``, the rest of the last tile ``-inf``) are WRITTEN. A fixed grid of worker+ programs strides over the live tiles, so both the work and the launch geometry follow the live+ history rather than the staged width; consumers (``dsv41/topk.py``) read below ``live`` only, so+ the dead area needs no fill.+* **candidate pool** (Reindex mode, Hierarchical Sparse Indexer): position ``j`` is+ ``candidates[b, s, j]`` (a compressed position, ``-1`` = empty slot); the output is ``[B, S, NC]``+ aligned with the candidate list and fully written (``-inf`` for empty / unreachable slots), so+ a top-k over it indexes straight back into the list.++Causality lives in the kernel: ``live[b, s]`` is the number of compressed positions query ``s`` may+see, and every position at or past it (or with no row) scores ``-inf``. The dot product, weighted+scores and head reduction round to bf16 at the same boundaries as the reference expression.+"""++from __future__ import annotations++import torch+import triton+import triton.language as tl++from freetoken.kvcache.dsv4.v41_row_format import RowFormat++from .row_format import load_rows++BLOCK_T = 64+++def _score_tile(+ q, w, pool_ptr, locs_row, live, pos, store_mask, out_ptrs,+ n_locs, RATIO: tl.constexpr, D: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr,+):+ """Score one tile of compressed positions ``pos`` against the query (``q [BLOCK_H, D]`` bf16,+ ``w [BLOCK_H]`` fp32) and store ``[BLOCK_T]`` logits (``-inf`` where unreachable)."""+ pos_ok = store_mask & (pos >= 0) & (pos < live)+ full_idx = pos * RATIO+ full = tl.load(locs_row + tl.maximum(full_idx, 0), mask=pos_ok & (full_idx < n_locs), other=-1)+ rows = tl.where(full >= 0, full // RATIO, -1).to(tl.int32)+ valid = pos_ok & (rows >= 0)+ offs_d = tl.arange(0, D)+ k = load_rows(pool_ptr, rows, valid, offs_d, D, ROW_BYTES, FMT).to(tl.bfloat16) # [BLOCK_T, D]+ score = tl.dot(q, tl.trans(k)).to(tl.bfloat16).to(tl.float32)+ score = (tl.maximum(score, 0.0) * w[:, None]).to(tl.bfloat16).to(tl.float32)+ logits = tl.sum(score, axis=0).to(tl.bfloat16).to(tl.float32)+ tl.store(out_ptrs, tl.where(valid, logits, float("-inf")), mask=store_mask)+++def _indexer_logits_packed_kernel(+ q_ptr, w_ptr, pool_ptr, locs_ptr, live_ptr, cand_ptr, out_ptr,+ T, NC, n_locs,+ stride_qb, stride_qs, stride_qh, stride_qd,+ stride_wb, stride_ws, stride_wh,+ stride_lb, stride_lt,+ stride_vb, stride_vs,+ stride_cb, stride_cs, stride_cn,+ stride_ob, stride_os, stride_on,+ RATIO: tl.constexpr,+ H: tl.constexpr,+ D: tl.constexpr,+ ROW_BYTES: tl.constexpr,+ FMT: tl.constexpr,+ BLOCK_H: tl.constexpr,+ BLOCK_T: tl.constexpr,+ HAS_CAND: tl.constexpr,+ N_WORKERS: tl.constexpr,+):+ pid_s = tl.program_id(0)+ pid_b = tl.program_id(1)+ pid_t = tl.program_id(2)++ live = tl.load(live_ptr + pid_b * stride_vb + pid_s * stride_vs)+ locs_row = locs_ptr + pid_b * stride_lb+ out_row = out_ptr + pid_b * stride_ob + pid_s * stride_os+ offs_d = tl.arange(0, D)+ offs_h = tl.arange(0, BLOCK_H)+ h_mask = offs_h < H+ q = tl.load(+ q_ptr + pid_b * stride_qb + pid_s * stride_qs + offs_h[:, None] * stride_qh + offs_d[None, :] * stride_qd,+ mask=h_mask[:, None], other=0.0,+ ).to(tl.bfloat16)+ w = tl.load(w_ptr + pid_b * stride_wb + pid_s * stride_ws + offs_h * stride_wh, mask=h_mask, other=0.0).to(tl.bfloat16).to(tl.float32)+ offs = tl.arange(0, BLOCK_T)++ if HAS_CAND:+ # one program per candidate tile; the list is fixed-width and fully written+ offs_t = pid_t * BLOCK_T + offs+ store_mask = offs_t < NC+ pos = tl.load(cand_ptr + pid_b * stride_cb + pid_s * stride_cs + offs_t * stride_cn, mask=store_mask, other=-1)+ _score_tile(q, w, pool_ptr, locs_row, live, pos, store_mask, out_row + offs_t * stride_on, n_locs, RATIO, D, ROW_BYTES, FMT)+ else:+ # worker pid_t scores live tiles pid_t, pid_t + N_WORKERS, ...; nothing past live is written+ for tile in range(pid_t, tl.cdiv(tl.minimum(live, T), BLOCK_T), N_WORKERS):+ offs_t = tile * BLOCK_T + offs+ store_mask = offs_t < T+ _score_tile(q, w, pool_ptr, locs_row, live, offs_t, store_mask, out_row + offs_t * stride_on, n_locs, RATIO, D, ROW_BYTES, FMT)+++def indexer_logits_packed(+ q: torch.Tensor, # [B, S, H, D] bf16 (fp4 round-tripped index queries)+ weights: torch.Tensor, # [B, S, H] (softmax_scale * n_heads**-0.5 folded in)+ k_pool: torch.Tensor, # [R, fmt.row_bytes(D)] uint8, the kv-source's packed index keys+ fmt: RowFormat,+ locs: torch.Tensor, # [B, W] int32/int64 full-token locs of each request (page-table row / decode snapshot)+ ratio: int, # compressed position t -> full loc locs[b, t * ratio]; its row is loc // ratio+ live: torch.Tensor, # [B, S] int32: compressed positions query (b, s) may see+ T: int | None = None, # staged width of the full-range output (default: W // ratio)+ candidates: torch.Tensor | None = None, # [B, S, NC] int32 compressed positions (-1 = empty)+ out: torch.Tensor | None = None,+) -> torch.Tensor:+ """Head-reduced indexer logits: ``[B, S, T]`` over the full staged range (columns below ``live``+ written, the rest untouched), or ``[B, S, NC]`` aligned with ``candidates`` (fully written).+ Unreachable / empty positions are ``-inf``."""+ B, S, H, D = q.shape+ assert locs.ndim == 2 and locs.shape[0] == B and live.shape == (B, S), (locs.shape, live.shape, (B, S))+ assert weights.shape == (B, S, H), (weights.shape, (B, S, H))+ assert D == triton.next_power_of_2(D), f"index_head_dim must be pow2, got {D}"+ assert k_pool.dtype == torch.uint8 and k_pool.shape[1] == fmt.row_bytes(D) and k_pool.is_contiguous()+ assert locs.dtype in (torch.int32, torch.int64) and locs.stride(1) == 1, (locs.dtype, locs.stride())+ n_locs = locs.shape[1]+ if T is None:+ T = n_locs // ratio+ live = live.to(torch.int32)+ has_cand = candidates is not None+ if has_cand:+ assert candidates.shape[:2] == (B, S), (candidates.shape, (B, S))+ cand = candidates.to(torch.int32)+ NC = cand.shape[2]+ cs = cand.stride()+ else:+ cand, NC, cs = live, 0, (0, 0, 0)+ n_out = NC if has_cand else T+ if out is None:+ out = torch.empty((B, S, n_out), dtype=torch.float32, device=q.device)+ assert out.shape == (B, S, n_out) and out.dtype == torch.float32+ if n_out == 0 or S == 0:+ return out++ n_tiles = triton.cdiv(n_out, BLOCK_T)+ # A device-sized worker grid strides over live history without launching a+ # program for every reserved tile. Independent queries share the SM budget.+ sm_count = torch.cuda.get_device_properties(q.device).multi_processor_count+ workers = n_tiles if has_cand else min(n_tiles, max(1, triton.cdiv(sm_count, B * S)))+ grid = (S, B, workers)+ _indexer_logits_packed_kernel[grid](+ q, weights, k_pool, locs, live, cand, out,+ T, NC, n_locs,+ q.stride(0), q.stride(1), q.stride(2), q.stride(3),+ weights.stride(0), weights.stride(1), weights.stride(2),+ locs.stride(0), locs.stride(1),+ live.stride(0), live.stride(1),+ cs[0], cs[1], cs[2],+ out.stride(0), out.stride(1), out.stride(2),+ RATIO=ratio, H=H, D=D, ROW_BYTES=fmt.row_bytes(D), FMT=fmt.code,+ BLOCK_H=triton.next_power_of_2(H), BLOCK_T=BLOCK_T, HAS_CAND=has_cand, N_WORKERS=workers,+ num_warps=4, num_stages=2,+ )+ return out+++__all__ = ["indexer_logits_packed", "BLOCK_T"]diff --git a/python/freetoken/kernel/triton/dsv41/pack.py b/python/freetoken/kernel/triton/dsv41/pack.pynew file mode 100644index 00000000..47a9164e--- /dev/null+++ b/python/freetoken/kernel/triton/dsv41/pack.py@@ -0,0 +1,183 @@+"""Quantize rows into a packed pool and read them back (DeepSeek-V4.1 KV storage formats).++``pack_rows`` reproduces the reference quantizers (``inference/kernel.py``) exactly:++* fp8_e8m0_b32 ``act_quant(x, 32, "ue8m0", e8m0)``: per 32-group ``s = 2**ceil(log2(max(amax, 1e-4) / 448))``,+ ``q = round_e4m3(clamp(x / s, +-448))``+* fp4_e4m3_b16 ``fp4_act_quant(x, 16, scale_dtype=e4m3)``: ``s = e4m3(max(amax, 6 * 2^-9) / 6)`` (round to+ nearest e4m3), ``q = round_e2m1(clamp(x / s, +-6))``+* fp4_e8m0_b32 ``fp4_act_quant(x, 32)``: ``s = 2**ceil(log2(max(amax, 6 * 2^-126) / 6))``, same e2m1 rounding+* bf16 a plain byte copy++Rows land at ``pool[row_ids[m]]`` (or ``pool[m]`` without ids), so the same launch serves the KV write+path (scatter into slots) and plain buffer conversion. ``unpack_rows`` is the inverse gather + dequant+(bf16-rounded like the reference cache); the attention / indexer kernels use the same device helper+(``row_format.load_rows``) in-line instead of materializing rows.+"""++from __future__ import annotations++import torch+import triton+import triton.language as tl++from freetoken.kernel.triton.dsv4.fp8_linear import _log2_ceil, _round_fp4+from freetoken.kernel.triton.e4m3_compat import e4m3_f32_to_u8, e4m3_native_cx, round_e4m3+from freetoken.kvcache.dsv4.v41_row_format import BF16, RowFormat++from .row_format import fp4_f32_to_nibble, load_rows+++def _e4m3_codes(q):+ """fp32 (|q| <= 448) -> e4m3 bit pattern, RNE."""+ if e4m3_native_cx():+ return q.to(tl.float8e4nv).to(tl.uint8, bitcast=True)+ else:+ return e4m3_f32_to_u8(round_e4m3(q))+++def _e4m3_round(q):+ """fp32 -> nearest e4m3 value, as fp32."""+ if e4m3_native_cx():+ return q.to(tl.float8e4nv).to(tl.float32)+ else:+ return round_e4m3(q)+++def _pack_rows_kernel(+ x_ptr, out_ptr, idx_ptr, stride_xm,+ D: tl.constexpr, GROUP: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr, HAS_IDX: tl.constexpr,+):+ """One program per input row ``m``; writes packed bytes to pool row ``idx[m]`` (or ``m``)."""+ m = tl.program_id(0).to(tl.int64)+ if HAS_IDX:+ row = tl.load(idx_ptr + m).to(tl.int64)+ else:+ row = m+ dst = out_ptr + row * ROW_BYTES+ src = x_ptr + m * stride_xm++ if FMT == 0:+ offs_d = tl.arange(0, D)+ bits = tl.load(src + offs_d).to(tl.bfloat16).to(tl.uint16, bitcast=True)+ tl.store(dst + 2 * offs_d, (bits & 0xFF).to(tl.uint8))+ tl.store(dst + 2 * offs_d + 1, (bits >> 8).to(tl.uint8))+ elif FMT == 1:+ NG: tl.constexpr = D // GROUP+ offs_d = tl.arange(0, D)+ offs_g = tl.arange(0, NG)+ x = tl.reshape(tl.load(src + offs_d).to(tl.float32), [NG, GROUP])+ amax = tl.maximum(tl.max(tl.abs(x), axis=1), 1e-4)+ e = _log2_ceil(amax * (1.0 / 448.0))+ s = tl.exp2(e.to(tl.float32))+ q = tl.clamp(tl.math.div_rn(x, s[:, None]), -448.0, 448.0)+ tl.store(dst + offs_d, tl.reshape(_e4m3_codes(q), [D]))+ tl.store(dst + D + offs_g, (e + 127).to(tl.uint8))+ else:+ # e2m1 pairs: even channels from one strided load, odd from another, so a byte is (lo | hi << 4)+ NG: tl.constexpr = D // GROUP+ HALF: tl.constexpr = GROUP // 2+ offs_h = tl.arange(0, D // 2)+ offs_g = tl.arange(0, NG)+ xe = tl.reshape(tl.load(src + 2 * offs_h).to(tl.float32), [NG, HALF])+ xo = tl.reshape(tl.load(src + 2 * offs_h + 1).to(tl.float32), [NG, HALF])+ amax = tl.maximum(tl.max(tl.abs(xe), axis=1), tl.max(tl.abs(xo), axis=1))+ if FMT == 2:+ amax = tl.maximum(amax, 6.0 * (2.0 ** -9))+ s = _e4m3_round(amax * (1.0 / 6.0))+ tl.store(dst + D // 2 + offs_g, _e4m3_codes(s))+ else:+ amax = tl.maximum(amax, 6.0 * (2.0 ** -126))+ e = _log2_ceil(amax * (1.0 / 6.0))+ s = tl.exp2(e.to(tl.float32))+ tl.store(dst + D // 2 + offs_g, (e + 127).to(tl.uint8))+ # IEEE division: the reference's CUDA ``x / s`` is correctly rounded, and with an e4m3+ # (non power-of-two) scale triton's default approximate div would land a hair off the+ # e2m1 midpoints and flip their round-to-even+ qe = fp4_f32_to_nibble(_round_fp4(tl.clamp(tl.math.div_rn(xe, s[:, None]), -6.0, 6.0)))+ qo = fp4_f32_to_nibble(_round_fp4(tl.clamp(tl.math.div_rn(xo, s[:, None]), -6.0, 6.0)))+ tl.store(dst + offs_h, tl.reshape(qe | (qo << 4), [D // 2]))+++def _unpack_rows_kernel(+ pool_ptr, idx_ptr, out_ptr, M,+ D: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr, BLOCK_T: tl.constexpr, HAS_IDX: tl.constexpr,+):+ pid = tl.program_id(0)+ offs_t = pid * BLOCK_T + tl.arange(0, BLOCK_T)+ valid = offs_t < M+ if HAS_IDX:+ rows = tl.load(idx_ptr + offs_t, mask=valid, other=0)+ else:+ rows = offs_t+ offs_d = tl.arange(0, D)+ vals = load_rows(pool_ptr, rows, valid, offs_d, D, ROW_BYTES, FMT)+ tl.store(+ out_ptr + offs_t[:, None].to(tl.int64) * D + offs_d[None, :],+ vals.to(out_ptr.dtype.element_ty), mask=valid[:, None],+ )+++def pack_rows(+ x: torch.Tensor, fmt: RowFormat, pool: torch.Tensor | None = None, row_ids: torch.Tensor | None = None,+) -> torch.Tensor:+ """Quantize ``x [M, D]`` (bf16 / fp16 / fp32) into packed rows.++ With ``pool [rows, row_bytes] uint8`` and ``row_ids [M]``, row ``m`` is written to ``pool[row_ids[m]]``+ (the KV write path). Without a pool a fresh ``[M, row_bytes]`` uint8 tensor is returned.+ """+ assert x.dim() == 2, x.shape+ m, d = x.shape+ fmt.validate_dim(d)+ row_bytes = fmt.row_bytes(d)+ assert d & (d - 1) == 0 and d >= 16, f"pack_rows wants a power-of-two D >= 16, got {d}"+ if pool is None:+ assert row_ids is None+ pool = torch.empty((m, row_bytes), dtype=torch.uint8, device=x.device)+ else:+ assert pool.dtype == torch.uint8 and pool.shape[1] == row_bytes and pool.is_contiguous(), (pool.shape, row_bytes)+ assert row_ids is not None and row_ids.shape == (m,), (None if row_ids is None else row_ids.shape, m)+ if m == 0:+ return pool+ if x.stride(1) != 1:+ x = x.contiguous()+ _pack_rows_kernel[(m,)](+ x, pool, row_ids if row_ids is not None else pool, x.stride(0),+ D=d, GROUP=fmt.group or d, ROW_BYTES=row_bytes, FMT=fmt.code, HAS_IDX=row_ids is not None,+ num_warps=4,+ )+ return pool+++def unpack_rows(+ pool: torch.Tensor, fmt: RowFormat, dim: int, row_ids: torch.Tensor | None = None,+ out_dtype: torch.dtype = torch.bfloat16,+) -> torch.Tensor:+ """Gather ``pool[row_ids]`` (or every row) and dequantize to ``[M, dim]`` in ``out_dtype``."""+ fmt.validate_dim(dim)+ assert pool.dtype == torch.uint8 and pool.shape[1] == fmt.row_bytes(dim) and pool.is_contiguous(), pool.shape+ m = pool.shape[0] if row_ids is None else row_ids.numel()+ out = torch.empty((m, dim), dtype=out_dtype, device=pool.device)+ if m == 0:+ return out+ block_t = 16+ _unpack_rows_kernel[(triton.cdiv(m, block_t),)](+ pool, row_ids if row_ids is not None else pool, out, m,+ D=dim, ROW_BYTES=fmt.row_bytes(dim), FMT=fmt.code, BLOCK_T=block_t, HAS_IDX=row_ids is not None,+ num_warps=4,+ )+ return out+++def quant_roundtrip(x: torch.Tensor, fmt: RowFormat) -> torch.Tensor:+ """``unpack(pack(x))`` -- the value the cache will hand back; the reference's ``inplace=True`` semantics."""+ if fmt is BF16:+ return x.to(torch.bfloat16)+ return unpack_rows(pack_rows(x, fmt), fmt, x.shape[1], out_dtype=torch.bfloat16)+++__all__ = ["pack_rows", "unpack_rows", "quant_roundtrip"]diff --git a/python/freetoken/kernel/triton/dsv41/row_format.py b/python/freetoken/kernel/triton/dsv41/row_format.pynew file mode 100644index 00000000..032d77e9--- /dev/null+++ b/python/freetoken/kernel/triton/dsv41/row_format.py@@ -0,0 +1,92 @@+"""Device-side row-format helpers shared by the DSV41 kernels.++The Python-side descriptors live in ``freetoken.kvcache.dsv4.v41_row_format``; the ``FMT`` constexpr a+kernel receives is ``RowFormat.code``:++ 0 bf16 2 B / channel, no scale+ 1 fp8_e8m0_b32 e4m3 byte / channel, ue8m0 scale byte per 32 channels+ 2 fp4_e4m3_b16 e2m1 nibble / channel (even channel low nibble), e4m3 scale byte per 16+ 3 fp4_e8m0_b32 e2m1 nibble / channel, ue8m0 scale byte per 32++A row is ``[values | scales]``; ``ROW_BYTES`` is the pool's row stride. Every dequantized value+is rounded through bf16 before use -- the reference bakes its quant round-trip into a bf16 cache,+so this is what its attention / indexer kernels actually read.+"""++from __future__ import annotations++import triton+import triton.language as tl++from freetoken.kernel.triton.e4m3_compat import e4m3_u8_to_f32++FMT_BF16 = 0+FMT_FP8_E8M0_B32 = 1+FMT_FP4_E4M3_B16 = 2+FMT_FP4_E8M0_B32 = 3+++def fp4_nibble_to_f32(nib):+ """e2m1 code (uint8, 0..15) -> fp32: bit 3 sign, bits 2-1 exponent, bit 0 mantissa;+ codes 0/1 are the subnormals 0 and 0.5, the rest ``(1 + m/2) * 2^(e-1)``."""+ c = (nib & 7).to(tl.int32)+ neg = (nib & 8) != 0+ sub = c.to(tl.float32) * 0.5+ normal = (1.0 + 0.5 * (c & 1).to(tl.float32)) * tl.exp2(((c >> 1) - 1).to(tl.float32))+ mag = tl.where(c < 2, sub, normal)+ return tl.where(neg, -mag, mag)+++def fp4_f32_to_nibble(q):+ """fp32 value already on the e2m1 grid ({0, .5, 1, 1.5, 2, 3, 4, 6}, signed) -> e2m1 code.+ Thresholds sit between grid points, so a value a hair off the grid still maps to its point."""+ a = tl.abs(q)+ c = tl.where(+ a < 1.75,+ tl.where(a < 0.75, tl.where(a < 0.25, 0, 1), tl.where(a < 1.25, 2, 3)),+ tl.where(a < 3.5, tl.where(a < 2.5, 4, 5), tl.where(a < 5.0, 6, 7)),+ )+ return (c + tl.where(q < 0, 8, 0)).to(tl.uint8)+++def load_rows(pool_ptr, rows, valid, offs_d, D: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr):+ """Gather + dequantize ``[T, D]`` fp32 from a packed pool.++ ``pool_ptr`` uint8 pool base; ``rows`` [T] row ids (any int width; masked-out rows may be -1);+ ``valid`` [T] bool; ``offs_d`` [D] = ``tl.arange(0, D)``. Invalid rows read as 0.+ """+ row_i64 = tl.maximum(rows, 0).to(tl.int64)+ base = pool_ptr + row_i64[:, None] * ROW_BYTES+ m = valid[:, None] & (offs_d >= 0)[None, :]+ if FMT == 0:+ lo = tl.load(base + 2 * offs_d[None, :], mask=m, other=0).to(tl.uint16)+ hi = tl.load(base + 2 * offs_d[None, :] + 1, mask=m, other=0).to(tl.uint16)+ val = (lo | (hi << 8)).to(tl.bfloat16, bitcast=True).to(tl.float32)+ elif FMT == 1:+ v = tl.load(base + offs_d[None, :], mask=m, other=0)+ s = tl.load(base + D + (offs_d // 32)[None, :], mask=m, other=127)+ val = e4m3_u8_to_f32(v) * tl.exp2(s.to(tl.float32) - 127.0)+ else:+ b = tl.load(base + (offs_d // 2)[None, :], mask=m, other=0)+ nib = tl.where((offs_d % 2 == 0)[None, :], b & 0xF, b >> 4)+ q = fp4_nibble_to_f32(nib)+ if FMT == 2:+ s = e4m3_u8_to_f32(tl.load(base + D // 2 + (offs_d // 16)[None, :], mask=m, other=0))+ else:+ s = tl.exp2(tl.load(base + D // 2 + (offs_d // 32)[None, :], mask=m, other=127).to(tl.float32) - 127.0)+ val = q * s+ return val.to(tl.bfloat16).to(tl.float32)+++__all__ = [+ "FMT_BF16",+ "FMT_FP8_E8M0_B32",+ "FMT_FP4_E4M3_B16",+ "FMT_FP4_E8M0_B32",+ "fp4_nibble_to_f32",+ "fp4_f32_to_nibble",+ "load_rows",+]diff --git a/python/freetoken/kernel/triton/dsv41/sparse_attn.py b/python/freetoken/kernel/triton/dsv41/sparse_attn.pynew file mode 100644index 00000000..9d6b0816--- /dev/null+++ b/python/freetoken/kernel/triton/dsv41/sparse_attn.py@@ -0,0 +1,323 @@+"""Sparse gathered-KV flash attention over PACKED pools (DeepSeek-V4.1).++Same contract as ``kernel/triton/dsv4/sparse_attn.py`` -- each query attends to a per-query list of+GLOBAL slots laid out ``[window part | compressed part]`` plus a per-head attention sink, K == V is one+shared latent head -- but the two pools are byte pools in (possibly different) ``RowFormat``s: the+window ring holds fp8 rows, the compressed KV holds fp4 rows, and every gathered row is dequantized+in-kernel (``row_format.load_rows``) to the bf16 value the reference cache would have held.++Because the two pools dequantize differently, a tile never mixes them: the window part is+``n_window`` columns with ``n_window % BLOCK_T == 0`` (the caller pads it with ``-1``), so the kernel+walks the window tiles and then the compressed tiles in two loops, each with a single load site.+Both dots take bf16 operands with fp32 accumulation, like the reference ``sparse_attn`` kernel.++Two implementations, picked from the launch shape (no knob):+ * prefill (``m`` > 1): one program per (query, request, head block)+ * decode (``m`` == 1): flash-decoding split over the candidate axis, merged through log-sum-exp+ (the sink joins in the merge). Not bit-identical to the single-program path.++``cmp_counts`` (device ``[b, m]`` int32) bounds the compressed columns a CUDA-graph replay visits;+without it the full ``topk_idxs`` width is walked.+"""++from __future__ import annotations++import torch+import triton+import triton.language as tl++from freetoken.kvcache.dsv4.v41_row_format import RowFormat++from .row_format import load_rows++BLOCK_H = 16+BLOCK_T = 32+MAX_SPLITS = 32+# Decode splits at tile boundaries, bounded by the device's SM count below.+MIN_TILES_PER_SPLIT = 1+++def _attend_tile(q, kv, valid, scale, m_i, l_i, acc):+ """One online-softmax step over a ``[BLOCK_T, D]`` fp32 KV tile (bf16-exact values)."""+ kv16 = kv.to(tl.bfloat16)+ scores = tl.dot(q, tl.trans(kv16)) * scale # [BLOCK_H, BLOCK_T] fp32+ scores = tl.where(valid[None, :], scores, -float("inf"))+ m_new = tl.maximum(m_i, tl.max(scores, axis=1))+ # an all-masked tile keeps m_new at -inf; short-circuit so -inf - -inf never forms+ alpha = tl.where(m_new == -float("inf"), 1.0, tl.exp(m_i - m_new))+ p = tl.where(valid[None, :], tl.exp(scores - m_new[:, None]), 0.0)+ l_i = l_i * alpha + tl.sum(p, axis=1)+ acc = acc * alpha[:, None] + tl.dot(p.to(tl.bfloat16), kv16)+ return m_new, l_i, acc+++def _sparse_attn_packed_kernel(+ q_ptr, win_ptr, cmp_ptr, o_ptr, sink_ptr, idx_ptr, cnt_ptr,+ scale,+ H, TOPK, N_WINDOW,+ stride_qb, stride_qm, stride_qh, stride_qd,+ stride_ob, stride_om, stride_oh, stride_od,+ stride_ib, stride_im, stride_it,+ stride_nb, stride_nm,+ D: tl.constexpr,+ WIN_FMT: tl.constexpr, WIN_ROW_BYTES: tl.constexpr,+ CMP_FMT: tl.constexpr, CMP_ROW_BYTES: tl.constexpr,+ BLOCK_H: tl.constexpr,+ BLOCK_T: tl.constexpr,+ HAS_COUNTS: tl.constexpr,+):+ pid_m = tl.program_id(0)+ pid_b = tl.program_id(1)+ pid_h = tl.program_id(2)++ offs_h = pid_h * BLOCK_H + tl.arange(0, BLOCK_H)+ h_mask = offs_h < H+ offs_d = tl.arange(0, D)++ q_ptrs = q_ptr + pid_b * stride_qb + pid_m * stride_qm + offs_h[:, None] * stride_qh + offs_d[None, :] * stride_qd+ q = tl.load(q_ptrs, mask=h_mask[:, None], other=0.0).to(tl.bfloat16) # [BLOCK_H, D]++ m_i = tl.full((BLOCK_H,), -float("inf"), dtype=tl.float32)+ l_i = tl.zeros((BLOCK_H,), dtype=tl.float32)+ acc = tl.zeros((BLOCK_H, D), dtype=tl.float32)++ n_active = TOPK+ if HAS_COUNTS:+ n_active = N_WINDOW + tl.load(cnt_ptr + pid_b * stride_nb + pid_m * stride_nm)++ idx_base = idx_ptr + pid_b * stride_ib + pid_m * stride_im+ for start in range(0, N_WINDOW, BLOCK_T):+ offs_t = start + tl.arange(0, BLOCK_T)+ idxs = tl.load(idx_base + offs_t * stride_it)+ valid = idxs >= 0+ kv = load_rows(win_ptr, idxs, valid, offs_d, D, WIN_ROW_BYTES, WIN_FMT)+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)+ for start in range(N_WINDOW, n_active, BLOCK_T):+ offs_t = start + tl.arange(0, BLOCK_T)+ t_mask = offs_t < n_active+ idxs = tl.load(idx_base + offs_t * stride_it, mask=t_mask, other=-1)+ valid = idxs >= 0+ kv = load_rows(cmp_ptr, idxs, valid, offs_d, D, CMP_ROW_BYTES, CMP_FMT)+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)++ sink = tl.load(sink_ptr + offs_h, mask=h_mask, other=0.0).to(tl.float32)+ l_i = l_i + tl.exp(sink - m_i)+ o = acc / l_i[:, None]++ o_ptrs = o_ptr + pid_b * stride_ob + pid_m * stride_om + offs_h[:, None] * stride_oh + offs_d[None, :] * stride_od+ tl.store(o_ptrs, o.to(o_ptr.dtype.element_ty), mask=h_mask[:, None])+++def _sparse_attn_packed_splitk_kernel(+ q_ptr, win_ptr, cmp_ptr, mid_o_ptr, mid_lse_ptr, idx_ptr, cnt_ptr,+ scale,+ H, TOPK, N_WINDOW,+ stride_qb, stride_qm, stride_qh, stride_qd,+ stride_mb, stride_mm, stride_mh, stride_ms, stride_md,+ stride_lb, stride_lm, stride_lh, stride_ls,+ stride_ib, stride_im, stride_it,+ stride_nb, stride_nm,+ D: tl.constexpr,+ WIN_FMT: tl.constexpr, WIN_ROW_BYTES: tl.constexpr,+ CMP_FMT: tl.constexpr, CMP_ROW_BYTES: tl.constexpr,+ BLOCK_H: tl.constexpr,+ BLOCK_T: tl.constexpr,+ HAS_COUNTS: tl.constexpr,+ NUM_SPLITS: tl.constexpr,+):+ """Stage 1: each program reduces one BLOCK_T-aligned slice of the candidate list and writes+ its normalized partial output + log-sum-exp. A slice may straddle the window / compressed+ boundary; it is walked as (window tiles, compressed tiles) since N_WINDOW is tile-aligned."""+ pid_ms = tl.program_id(0)+ pid_m = pid_ms // NUM_SPLITS+ split_id = pid_ms % NUM_SPLITS+ pid_b = tl.program_id(1)+ pid_h = tl.program_id(2)++ offs_h = pid_h * BLOCK_H + tl.arange(0, BLOCK_H)+ h_mask = offs_h < H+ offs_d = tl.arange(0, D)++ n_active = TOPK+ if HAS_COUNTS:+ n_active = N_WINDOW + tl.load(cnt_ptr + pid_b * stride_nb + pid_m * stride_nm)++ per_split = tl.cdiv(tl.cdiv(n_active, NUM_SPLITS), BLOCK_T) * BLOCK_T+ split_start = per_split * split_id+ split_end = tl.minimum(split_start + per_split, n_active)++ m_i = tl.full((BLOCK_H,), -float("inf"), dtype=tl.float32)+ l_i = tl.zeros((BLOCK_H,), dtype=tl.float32)+ acc = tl.zeros((BLOCK_H, D), dtype=tl.float32)++ if split_end > split_start:+ q_ptrs = (+ q_ptr + pid_b * stride_qb + pid_m * stride_qm+ + offs_h[:, None] * stride_qh + offs_d[None, :] * stride_qd+ )+ q = tl.load(q_ptrs, mask=h_mask[:, None], other=0.0).to(tl.bfloat16)+ idx_base = idx_ptr + pid_b * stride_ib + pid_m * stride_im++ win_end = tl.minimum(split_end, N_WINDOW)+ for start in range(split_start, win_end, BLOCK_T):+ offs_t = start + tl.arange(0, BLOCK_T)+ idxs = tl.load(idx_base + offs_t * stride_it)+ valid = idxs >= 0+ kv = load_rows(win_ptr, idxs, valid, offs_d, D, WIN_ROW_BYTES, WIN_FMT)+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)+ cmp_start = tl.maximum(split_start, N_WINDOW)+ for start in range(cmp_start, split_end, BLOCK_T):+ offs_t = start + tl.arange(0, BLOCK_T)+ t_mask = offs_t < split_end+ idxs = tl.load(idx_base + offs_t * stride_it, mask=t_mask, other=-1)+ valid = idxs >= 0+ kv = load_rows(cmp_ptr, idxs, valid, offs_d, D, CMP_ROW_BYTES, CMP_FMT)+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)++ out = tl.where(l_i[:, None] == 0.0, 0.0, acc / l_i[:, None])+ lse = tl.where(l_i == 0.0, -float("inf"), m_i + tl.log(l_i))++ mid_base = (+ mid_o_ptr + pid_b * stride_mb + pid_m * stride_mm+ + offs_h[:, None] * stride_mh + split_id * stride_ms + offs_d[None, :] * stride_md+ )+ tl.store(mid_base, out, mask=h_mask[:, None])+ lse_base = (+ mid_lse_ptr + pid_b * stride_lb + pid_m * stride_lm+ + offs_h * stride_lh + split_id * stride_ls+ )+ tl.store(lse_base, lse, mask=h_mask)+++def _sparse_attn_splitk_merge_kernel(+ mid_o_ptr, mid_lse_ptr, o_ptr, sink_ptr,+ stride_mb, stride_mm, stride_mh, stride_ms, stride_md,+ stride_lb, stride_lm, stride_lh, stride_ls,+ stride_ob, stride_om, stride_oh, stride_od,+ D: tl.constexpr,+ NUM_SPLITS: tl.constexpr,+):+ """Stage 2: log-sum-exp merge over the splits; the attention sink joins here once, as a null+ key with logit ``attn_sink[h]`` and zero value. One program per (query, request, head)."""+ pid_m = tl.program_id(0)+ pid_b = tl.program_id(1)+ pid_h = tl.program_id(2)+ offs_d = tl.arange(0, D)++ m_i = tl.load(sink_ptr + pid_h).to(tl.float32)+ l_i = 1.0+ acc = tl.zeros((D,), dtype=tl.float32)++ mid_base = mid_o_ptr + pid_b * stride_mb + pid_m * stride_mm + pid_h * stride_mh + offs_d * stride_md+ lse_base = mid_lse_ptr + pid_b * stride_lb + pid_m * stride_lm + pid_h * stride_lh++ for split_id in tl.range(0, NUM_SPLITS, num_stages=2):+ partial = tl.load(mid_base + split_id * stride_ms)+ lse = tl.load(lse_base + split_id * stride_ls)+ m_new = tl.maximum(m_i, lse)+ alpha = tl.exp(m_i - m_new)+ beta = tl.where(lse == -float("inf"), 0.0, tl.exp(lse - m_new))+ acc = acc * alpha + partial * beta+ l_i = l_i * alpha + beta+ m_i = m_new++ o = acc / l_i+ o_ptrs = o_ptr + pid_b * stride_ob + pid_m * stride_om + pid_h * stride_oh + offs_d * stride_od+ tl.store(o_ptrs, o.to(o_ptr.dtype.element_ty))+++def split_count(b: int, m: int, h: int, topk: int, device) -> int:+ """How many ways to split the candidate axis; 0 means run the single-program kernel. Decode+ splits until every SM has a program or every split is down to one tile."""+ if m != 1:+ return 0+ sm_count = torch.cuda.get_device_properties(device).multi_processor_count+ n_splits = min(+ MAX_SPLITS,+ triton.cdiv(topk, MIN_TILES_PER_SPLIT * BLOCK_T),+ max(1, sm_count // (b * triton.cdiv(h, BLOCK_H))),+ )+ return n_splits if n_splits > 1 else 0+++def sparse_attn_packed(+ q: torch.Tensor, # [b, m, h, d] bf16+ window_pool: torch.Tensor, # [n_win_slots, win_fmt.row_bytes(d)] uint8+ win_fmt: RowFormat,+ cmp_pool: torch.Tensor, # [n_cmp, cmp_fmt.row_bytes(d)] uint8+ cmp_fmt: RowFormat,+ attn_sink: torch.Tensor, # [h]+ topk_idxs: torch.Tensor, # [b, m, topk] int32, GLOBAL rows, layout [window | compressed]+ n_window: int, # window columns (a multiple of BLOCK_T; pad with -1)+ softmax_scale: float,+ cmp_counts: torch.Tensor | None = None, # [b, m] int32 live compressed columns per query+) -> torch.Tensor:+ """Paged sparse MLA attention over packed KV pools; see the module docstring for the contract."""+ b, m, h, d = q.shape+ topk = topk_idxs.shape[-1]+ assert n_window % BLOCK_T == 0 and 0 <= n_window <= topk, (n_window, topk)+ assert window_pool.dtype == torch.uint8 and window_pool.shape[1] == win_fmt.row_bytes(d), (window_pool.shape, win_fmt)+ assert cmp_pool.dtype == torch.uint8 and cmp_pool.shape[1] == cmp_fmt.row_bytes(d), (cmp_pool.shape, cmp_fmt)+ assert window_pool.is_contiguous() and cmp_pool.is_contiguous()+ q = q.contiguous()+ idx = topk_idxs.contiguous().to(torch.int32)+ sink = attn_sink.contiguous().to(torch.float32)+ o = torch.empty_like(q)++ has_counts = cmp_counts is not None+ if has_counts:+ cnt = cmp_counts.contiguous().to(torch.int32).view(b, m)+ stride_nb, stride_nm = cnt.stride()+ else:+ cnt, stride_nb, stride_nm = idx, 0, 0++ fmt_args = dict(+ WIN_FMT=win_fmt.code, WIN_ROW_BYTES=win_fmt.row_bytes(d),+ CMP_FMT=cmp_fmt.code, CMP_ROW_BYTES=cmp_fmt.row_bytes(d),+ )+ n_splits = split_count(b, m, h, topk, q.device)+ if n_splits:+ head_blocks = triton.cdiv(h, BLOCK_H)+ mid_o = torch.empty((b, m, h, n_splits, d), dtype=torch.float32, device=q.device)+ mid_lse = torch.empty((b, m, h, n_splits), dtype=torch.float32, device=q.device)+ _sparse_attn_packed_splitk_kernel[(m * n_splits, b, head_blocks)](+ q, window_pool, cmp_pool, mid_o, mid_lse, idx, cnt,+ float(softmax_scale),+ h, topk, int(n_window),+ q.stride(0), q.stride(1), q.stride(2), q.stride(3),+ mid_o.stride(0), mid_o.stride(1), mid_o.stride(2), mid_o.stride(3), mid_o.stride(4),+ mid_lse.stride(0), mid_lse.stride(1), mid_lse.stride(2), mid_lse.stride(3),+ idx.stride(0), idx.stride(1), idx.stride(2),+ stride_nb, stride_nm,+ D=d, BLOCK_H=BLOCK_H, BLOCK_T=BLOCK_T, HAS_COUNTS=has_counts, NUM_SPLITS=n_splits,+ **fmt_args, num_warps=8, num_stages=2,+ )+ _sparse_attn_splitk_merge_kernel[(m, b, h)](+ mid_o, mid_lse, o, sink,+ mid_o.stride(0), mid_o.stride(1), mid_o.stride(2), mid_o.stride(3), mid_o.stride(4),+ mid_lse.stride(0), mid_lse.stride(1), mid_lse.stride(2), mid_lse.stride(3),+ o.stride(0), o.stride(1), o.stride(2), o.stride(3),+ D=d, NUM_SPLITS=n_splits, num_warps=4,+ )+ return o++ grid = (m, b, triton.cdiv(h, BLOCK_H))+ _sparse_attn_packed_kernel[grid](+ q, window_pool, cmp_pool, o, sink, idx, cnt,+ float(softmax_scale),+ h, topk, int(n_window),+ q.stride(0), q.stride(1), q.stride(2), q.stride(3),+ o.stride(0), o.stride(1), o.stride(2), o.stride(3),+ idx.stride(0), idx.stride(1), idx.stride(2),+ stride_nb, stride_nm,+ D=d, BLOCK_H=BLOCK_H, BLOCK_T=BLOCK_T, HAS_COUNTS=has_counts,+ **fmt_args, num_warps=8, num_stages=2,+ )+ return o+++__all__ = ["sparse_attn_packed", "split_count", "BLOCK_T"]diff --git a/python/freetoken/kernel/triton/dsv41/topk.py b/python/freetoken/kernel/triton/dsv41/topk.pynew file mode 100644index 00000000..4d638e4d--- /dev/null+++ b/python/freetoken/kernel/triton/dsv41/topk.py@@ -0,0 +1,265 @@+"""Exact top-k and block-candidate selection whose work follows the live history (DeepSeek-V4.1).++The indexer scores a request's compressed positions into a buffer as wide as the staged history+(``max_seq_len / ratio``, fixed while a CUDA graph is captured), but only the first ``live[row]``+columns are meaningful -- and at 1M staged positions a 4K-token request must not pay for the+other 99.6%. Every kernel here takes the live count per row on the device, keeps its output+addresses and shapes fixed, and restricts what it READS and reduces to the live prefix; nothing+past ``live`` is ever read, so the score buffer's dead area needs no initialization.++Selection is a fixed pipeline of small kernels with explicit workspace contracts:++* ``dsv41_topk``: exact top-k of ``[R, T]`` fp32 scores below ``live``. Level 0 gives every+ ``CHUNK``-wide slice of a row its own program, which radix-selects the slice's top-k into a+ ``(key, column)`` candidate workspace (a global winner beats at most k-1 columns, so it is+ inside its own slice's top-k: the union is exact). Further levels apply the same program to the+ candidates until one register-resident tile remains; the final program ranks it and emits the+ winners in ascending column order, ``-1`` padded. Slices past ``live`` exit at once and a+ slice with no more entries than ``k`` is copied through, so a short history costs a handful of+ near-empty launches. The radix primitives are the QSA block top-k's (``kernel/triton/qsa/topk``).+* ``dsv41_block_scores``: per-``block_size`` block maxima over the live prefix (the Hierarchical+ Sparse Indexer's level one), the block holding the newest live position pinned to ``+inf`` so it+ is always kept; a fixed worker grid strides over the live blocks.+* ``dsv41_candidate_blocks``: the two above plus a fixed-width expansion into the compact candidate+ list ``[R, topk_blocks * block_size]`` of compressed positions, ascending with ``-1`` in the tail.++Tie rule (documented, deterministic): scores order descending, equal scores by ascending column,+so the lowest positions win among exact ties. ``torch.topk`` leaves ties unspecified.+"""++from __future__ import annotations++import torch+import triton+import triton.language as tl++from freetoken.kernel.triton.qsa.topk import _BINS, _PASSES, _RADIX, _monotone_key, _resident_prefix, _tile_ranks++CHUNK = 4096 # register-resident slice per level program (fp32 keys + int32 columns)+FINAL_MAX = 4096 # the final program's resident candidate tile+BLOCK_SCORE_TILE = 512 # blocks per block-max program tile+++def _load_entries(src_ptr, col_ptr, base, offsets, limit, SRC_LOGITS: tl.constexpr):+ """One slice of a level's input as ``(monotone key, column)``; key 0 = dead (past ``limit``,+ ``-inf``, or a padding slot of the previous level)."""+ live = offsets < limit+ if SRC_LOGITS:+ value = tl.load(src_ptr + base + offsets, mask=live, other=-float("inf"))+ key = tl.where(live & (value > -float("inf")), _monotone_key(value), 0)+ cols = (base + offsets).to(tl.int32)+ else:+ key = tl.load(src_ptr + base + offsets, mask=live, other=0).to(tl.uint32, bitcast=True)+ cols = tl.load(col_ptr + base + offsets, mask=live, other=-1)+ return key, cols+++def _dsv41_topk_level_kernel(+ src_ptr, src_col_ptr, live_in_ptr, key_out_ptr, col_out_ptr, live_out_ptr,+ stride_src_row, stride_out_row, num_columns,+ TOP_K: tl.constexpr, PAD_K: tl.constexpr, CHUNK: tl.constexpr, N_SPLITS: tl.constexpr,+ SRC_LOGITS: tl.constexpr, BINS: tl.constexpr, RADIX: tl.constexpr, PASSES: tl.constexpr,+):+ """One program per (row, slice): the slice's top-k into candidate slot ``split * TOP_K``. Slot+ ``split`` is read downstream only when ``split < cdiv(live, CHUNK)``; program 0 publishes that+ live candidate count for the next level."""+ tl.static_assert(CHUNK <= 0xFFFF, "packed cumsum keeps 16 bits per half")+ tl.static_assert(TOP_K <= CHUNK, "a slice must hold its own top-k")+ row = tl.program_id(0)+ split = tl.program_id(1)+ visible = tl.maximum(tl.minimum(tl.load(live_in_ptr + row), num_columns), 0)+ if split == 0:+ tl.store(live_out_ptr + row, tl.minimum(tl.cdiv(visible, CHUNK), N_SPLITS) * TOP_K)+ base = split * CHUNK+ limit = tl.minimum(tl.maximum(visible - base, 0), CHUNK)+ if limit <= 0:+ return # a dead slice: nothing downstream reads its slot+ slot = row.to(tl.int64) * stride_out_row + split * TOP_K+ offsets = tl.arange(0, CHUNK)+ key, cols = _load_entries(src_ptr + row.to(tl.int64) * stride_src_row, src_col_ptr + row.to(tl.int64) * stride_src_row, base, offsets, limit, SRC_LOGITS)+ if limit <= TOP_K:+ # every live entry is a winner: copy in order (dead keys stay 0 for the next level)+ inside = offsets < TOP_K+ tl.store(key_out_ptr + slot + offsets, key.to(tl.int32, bitcast=True), mask=inside)+ tl.store(col_out_ptr + slot + offsets, cols, mask=inside)+ return+ prefix, ties = _resident_prefix(key, TOP_K, BINS, RADIX, PASSES)+ rank, take, above, equal = _tile_ranks(key, prefix, ties, 0, 0)+ write = take & (rank < TOP_K)+ tl.store(key_out_ptr + slot + rank, key.to(tl.int32, bitcast=True), mask=write)+ tl.store(col_out_ptr + slot + rank, cols, mask=write)+ emitted = above + tl.minimum(equal, ties)+ pad = tl.arange(0, PAD_K)+ tl.store(key_out_ptr + slot + pad, 0, mask=(pad >= emitted) & (pad < TOP_K))+++def _final_tile(src_row, col_row, out_row, limit, TOP_K: tl.constexpr, BLOCK: tl.constexpr, SRC_LOGITS: tl.constexpr, BINS: tl.constexpr, RADIX: tl.constexpr, PASSES: tl.constexpr):+ tl.static_assert(BLOCK <= 0xFFFF, "packed cumsum keeps 16 bits per half")+ offsets = tl.arange(0, BLOCK)+ key, cols = _load_entries(src_row, col_row, 0, offsets, limit, SRC_LOGITS)+ prefix, ties = _resident_prefix(key, tl.minimum(limit, TOP_K), BINS, RADIX, PASSES)+ rank, take, above, equal = _tile_ranks(key, prefix, ties, 0, 0)+ tl.store(out_row + rank, cols, mask=take & (rank < TOP_K))+ return above + tl.minimum(equal, ties)+++def _dsv41_topk_final_kernel(+ src_ptr, src_col_ptr, live_ptr, out_ptr,+ stride_src_row, stride_out_row, num_columns,+ TOP_K: tl.constexpr, PAD_K: tl.constexpr,+ BLOCK_SMALL: tl.constexpr, BLOCK_MID: tl.constexpr, BLOCK_FULL: tl.constexpr,+ SRC_LOGITS: tl.constexpr, BINS: tl.constexpr, RADIX: tl.constexpr, PASSES: tl.constexpr,+):+ """One program per row over the last level's live candidates (or the scores themselves when the+ row fits one tile): winners at their ascending-column ranks, ``-1`` from ``emitted`` on."""+ row = tl.program_id(0)+ limit = tl.maximum(tl.minimum(tl.load(live_ptr + row), num_columns), 0)+ src_row = src_ptr + row.to(tl.int64) * stride_src_row+ col_row = src_col_ptr + row.to(tl.int64) * stride_src_row+ out_row = out_ptr + row.to(tl.int64) * stride_out_row+ emitted = 0+ if limit > 0:+ # register residency costs the whole tile, so a short row takes a narrower one+ if limit <= BLOCK_SMALL:+ emitted = _final_tile(src_row, col_row, out_row, limit, TOP_K, BLOCK_SMALL, SRC_LOGITS, BINS, RADIX, PASSES)+ elif limit <= BLOCK_MID:+ emitted = _final_tile(src_row, col_row, out_row, limit, TOP_K, BLOCK_MID, SRC_LOGITS, BINS, RADIX, PASSES)+ else:+ emitted = _final_tile(src_row, col_row, out_row, limit, TOP_K, BLOCK_FULL, SRC_LOGITS, BINS, RADIX, PASSES)+ pad = tl.arange(0, PAD_K)+ tl.store(out_row + pad, -1, mask=(pad >= emitted) & (pad < TOP_K))+++MAX_K = CHUNK // 2 # a slice must shed at least half its entries per level, or the plan cannot converge+++def topk_plan(columns: int, k: int) -> list[tuple[int, int]]:+ """``[(input width, n_splits), ...]`` of the reduction levels before the final tile; empty when+ the row already fits the final tile. Derived from the buffer width alone (graph-static). Each+ level maps ``width`` to ``cdiv(width, CHUNK) * k`` candidates, which shrinks the row by at least+ ``CHUNK / k >= 2`` per level for ``k <= MAX_K``; larger ``k`` would stall (``k == CHUNK`` never+ shrinks) and is rejected here, before any launch."""+ if not 0 < k <= MAX_K:+ raise ValueError(f"dsv41_topk supports 1 <= k <= {MAX_K}, got {k}")+ levels: list[tuple[int, int]] = []+ width = columns+ while width > FINAL_MAX:+ n_splits = -(-width // CHUNK)+ levels.append((width, n_splits))+ nxt = n_splits * k+ assert nxt < width, (width, k) # guaranteed by k <= CHUNK // 2 for width > FINAL_MAX == CHUNK+ width = nxt+ return levels+++def dsv41_topk(scores: torch.Tensor, live: torch.Tensor, k: int, out: torch.Tensor | None = None) -> torch.Tensor:+ """Exact top-``k`` columns of every ``scores`` row among its first ``live[row]`` columns.++ ``scores [R, T]`` fp32 row-contiguous (columns at or past ``live`` are never read; ``-inf``+ columns are dead), ``live [R]`` int32. Returns ``out [R, k]`` int32: the winners in ascending+ column order packed at the front, ``-1`` from the ``emitted`` count on (fewer live or finite+ columns than ``k`` leave a ``-1`` tail). Ties: lower column wins."""+ assert scores.ndim == 2 and scores.dtype == torch.float32 and scores.stride(1) == 1, (scores.shape, scores.dtype)+ R, T = scores.shape+ assert live.shape == (R,), (live.shape, R)+ live = live.to(torch.int32).contiguous()+ if out is None:+ out = torch.empty((R, k), dtype=torch.int32, device=scores.device)+ assert out.shape == (R, k) and out.dtype == torch.int32 and out.stride(1) == 1+ if R == 0 or k == 0:+ return out+ if T == 0:+ out.fill_(-1)+ return out+ pad_k = triton.next_power_of_2(k)+ src, src_col, src_live, width, src_logits = scores, scores, live, T, True+ for level_width, n_splits in topk_plan(T, k):+ keys = torch.empty((R, n_splits * k), dtype=torch.int32, device=scores.device)+ cols = torch.empty((R, n_splits * k), dtype=torch.int32, device=scores.device)+ live_out = torch.empty((R,), dtype=torch.int32, device=scores.device)+ _dsv41_topk_level_kernel[(R, n_splits)](+ src, src_col, src_live, keys, cols, live_out,+ src.stride(0), keys.stride(0), level_width,+ TOP_K=k, PAD_K=pad_k, CHUNK=CHUNK, N_SPLITS=n_splits, SRC_LOGITS=src_logits,+ BINS=_BINS, RADIX=_RADIX, PASSES=_PASSES, num_warps=8, num_stages=1,+ )+ src, src_col, src_live, width, src_logits = keys, cols, live_out, n_splits * k, False+ block = triton.next_power_of_2(max(width, 1))+ _dsv41_topk_final_kernel[(R,)](+ src, src_col, src_live, out,+ src.stride(0), out.stride(0), width,+ TOP_K=k, PAD_K=pad_k,+ BLOCK_SMALL=min(1024, block), BLOCK_MID=min(4096, block), BLOCK_FULL=block,+ SRC_LOGITS=src_logits, BINS=_BINS, RADIX=_RADIX, PASSES=_PASSES, num_warps=8, num_stages=1,+ )+ return out+++def _dsv41_block_scores_kernel(+ scores_ptr, live_ptr, blk_ptr,+ stride_s_row, stride_b_row, num_blocks,+ BLOCK_SIZE: tl.constexpr, TILE: tl.constexpr, N_WORKERS: tl.constexpr,+):+ """Worker ``w`` of a row scores block tiles ``w, w + N_WORKERS, ...`` below the live block count:+ the block's best live position, ``+inf`` for the block holding the newest live position."""+ row = tl.program_id(0)+ worker = tl.program_id(1)+ live = tl.maximum(tl.load(live_ptr + row), 0)+ nb_live = tl.minimum(tl.cdiv(live, BLOCK_SIZE), num_blocks)+ s_row = scores_ptr + row.to(tl.int64) * stride_s_row+ b_row = blk_ptr + row.to(tl.int64) * stride_b_row+ offs_b = tl.arange(0, TILE)+ offs_p = tl.arange(0, BLOCK_SIZE)+ for tile in range(worker, tl.cdiv(nb_live, TILE), N_WORKERS):+ blocks = tile * TILE + offs_b+ b_mask = blocks < nb_live+ pos = blocks[:, None] * BLOCK_SIZE + offs_p[None, :]+ v = tl.load(s_row + pos, mask=b_mask[:, None] & (pos < live), other=-float("inf"))+ best = tl.max(v, axis=1)+ best = tl.where(blocks == nb_live - 1, float("inf"), best)+ tl.store(b_row + blocks, best, mask=b_mask)+++def dsv41_block_scores(scores: torch.Tensor, live: torch.Tensor, block_size: int, out: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor]:+ """``(block scores [R, cdiv(T, block_size)] fp32, live blocks [R] int32)``: each live block's best+ live position (``+inf`` for the newest block); blocks at or past the live count are not written."""+ R, T = scores.shape+ nb = -(-T // block_size)+ live = live.to(torch.int32).contiguous()+ if out is None:+ out = torch.empty((R, nb), dtype=torch.float32, device=scores.device)+ live_blocks = torch.div(live + block_size - 1, block_size, rounding_mode="floor").clamp_(min=0).to(torch.int32)+ if R == 0 or nb == 0:+ return out, live_blocks+ assert block_size == triton.next_power_of_2(block_size), block_size+ sm_count = torch.cuda.get_device_properties(scores.device).multi_processor_count+ workers = min(triton.cdiv(nb, BLOCK_SCORE_TILE), max(1, triton.cdiv(sm_count, R)))+ _dsv41_block_scores_kernel[(R, workers)](+ scores, live, out, scores.stride(0), out.stride(0), nb,+ BLOCK_SIZE=block_size, TILE=BLOCK_SCORE_TILE, N_WORKERS=workers, num_warps=4,+ )+ return out, live_blocks+++def dsv41_candidate_blocks(scores: torch.Tensor, live: torch.Tensor, topk_blocks: int, block_size: int) -> torch.Tensor:+ """Level one of the Hierarchical Sparse Indexer over ``[R, T]`` scores: the ``topk_blocks`` best+ live blocks (the newest always among them) expanded to the compact candidate list ``[R,+ topk_blocks * block_size]`` of compressed positions -- ascending, ``-1`` for positions at or past+ ``live`` and for unfilled blocks, all in the tail (a sorted valid prefix)."""+ R, T = scores.shape+ blk, live_blocks = dsv41_block_scores(scores, live, block_size)+ keep = dsv41_topk(blk, live_blocks, min(topk_blocks, blk.shape[1])) # [R, KB] ascending, -1 tail+ pos = keep.to(torch.int64).unsqueeze(-1) * block_size + torch.arange(block_size, device=scores.device)+ dead = (keep < 0).unsqueeze(-1) | (pos >= live.to(torch.int64).view(R, 1, 1))+ out = torch.where(dead, -1, pos).flatten(1).to(torch.int32)+ if out.shape[1] < topk_blocks * block_size:+ out = torch.nn.functional.pad(out, (0, topk_blocks * block_size - out.shape[1]), value=-1)+ return out+++__all__ = ["dsv41_topk", "dsv41_block_scores", "dsv41_candidate_blocks", "topk_plan", "CHUNK", "FINAL_MAX", "MAX_K"]diff --git a/python/freetoken/kernel/triton/e4m3_compat.py b/python/freetoken/kernel/triton/e4m3_compat.pyindex 6ea0c753..14a7acd8 100644--- a/python/freetoken/kernel/triton/e4m3_compat.py+++ b/python/freetoken/kernel/triton/e4m3_compat.py@@ -109,6 +109,20 @@ def e4m3_u8_to_f16_x128(v):return h.to(tl.float16, bitcast=True) * 32768.0+@jit+def e4m3_f32_to_u8(x):+ """Encode an fp32 value that already sits on the e4m3 grid (see :func:`round_e4m3`, |x| <= 448)+ into its e4m3 bit pattern (uint8). Normals: fp32 exponent rebased 127 -> 7 with the top three+ mantissa bits; subnormals (|x| < 2^-6, grid 2^-9): mantissa = |x| / 2^-9, exponent field 0."""+ b = x.to(tl.uint32, bitcast=True)+ sign = ((b >> 31) & 1).to(tl.uint8) << 7+ exp = ((b >> 23) & 0xFF).to(tl.int32) - 127 + 7+ man = ((b >> 20) & 0x7).to(tl.uint8)+ normal = (exp.to(tl.uint8) << 3) | man+ sub = (tl.abs(x) * 512.0).to(tl.uint8) # 0..7 on the 2^-9 grid+ return sign | tl.where(tl.abs(x) >= 0.015625, normal, sub)++@jitdef round_e4m3(x):"""Round fp32 onto the e4m3 value grid (RNE), fp32 -> fp32, in a SINGLEdiff --git a/python/freetoken/kernel/triton/mhc.py b/python/freetoken/kernel/triton/mhc.pyindex 0782efe3..876643c1 100644--- a/python/freetoken/kernel/triton/mhc.py+++ b/python/freetoken/kernel/triton/mhc.py@@ -27,7 +27,7 @@ def _mhc_stage1_kernel(H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,SPLIT: tl.constexpr, # hidden elems per split (multiple of BLOCK_H)BLOCK_H: tl.constexpr,- NS: tl.constexpr,+ NS: tl.constexpr, # splits per token (the grid's second axis; need not be a power of two)HAS_POST: tl.constexpr,):"""Split-K stage of the fused mHC: each program owns one hidden slice of one@@ -79,24 +79,24 @@ def _mhc_stage1_kernel(@triton.jit-def _mhc_stage2_kernel(- sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,+def _mhc_gates(+ t, sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,post_out_ptr, comb_out_ptr, pre_out_ptr,rms_eps, hc_eps, post_mult,SINKHORN: tl.constexpr,H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,- NS: tl.constexpr,+ NS: tl.constexpr, BLK_NS: tl.constexpr,):- """Reduce the split partials and run the tiny gate math (sigmoid gates,- row-softmax + in-register 4x4 Sinkhorn); emits pre gates for stage 3."""- t = tl.program_id(0).to(tl.int64)+ """Reduce token ``t``'s split partials and run the tiny gate math (sigmoid gates,+ row-softmax + in-register 4x4 Sinkhorn); writes post / comb / pre for the next sublayer."""offs_mix = tl.arange(0, BLK_MIX)mix_mask = offs_mix < MIX- offs_s = tl.arange(0, NS)+ offs_s = tl.arange(0, BLK_NS)+ s_mask = offs_s < NS- sqsum = tl.sum(tl.load(sq_part_ptr + t * NS + offs_s))+ sqsum = tl.sum(tl.load(sq_part_ptr + t * NS + offs_s, mask=s_mask, other=0.0))acc = tl.sum(- tl.load(mix_part_ptr + (t * NS + offs_s)[:, None] * BLK_MIX + offs_mix[None, :]),+ tl.load(mix_part_ptr + (t * NS + offs_s)[:, None] * BLK_MIX + offs_mix[None, :], mask=s_mask[:, None], other=0.0),axis=0,)inv_rms = tl.math.rsqrt(sqsum / (N * H) + rms_eps)@@ -142,6 +142,23 @@ def _mhc_stage2_kernel(tl.store(comb_out_ptr + t * N * N + offs_n2[:, None] * N + offs_n2[None, :], comb)++def _mhc_stage2_kernel(+ sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,+ post_out_ptr, comb_out_ptr, pre_out_ptr,+ rms_eps, hc_eps, post_mult,+ SINKHORN: tl.constexpr,+ H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,+ NS: tl.constexpr, BLK_NS: tl.constexpr,+):+ """Gate math alone (the two-pass variant, whose stage 3 consumes these pre gates)."""+ _mhc_gates(+ tl.program_id(0).to(tl.int64), sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr, post_out_ptr, comb_out_ptr, pre_out_ptr,+ rms_eps, hc_eps, post_mult, SINKHORN, H, N, MIX, BLK_MIX, NS, BLK_NS,+ )++@triton.jitdef _mhc_stage3_kernel(res_out_ptr, pre_ptr, li_out_ptr,@@ -161,7 +178,46 @@ def _mhc_stage3_kernel(tl.store(li_out_ptr + t * H + offs_h, li.to(li_out_ptr.dtype.element_ty), mask=h_mask)-def mhc_fused_post_pre_triton(+def _mhc_stage23_kernel(+ res_out_ptr, pre_ptr, li_out_ptr,+ sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,+ post_out_ptr, comb_out_ptr, pre_out_ptr,+ rms_eps, hc_eps, post_mult,+ SINKHORN: tl.constexpr,+ H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,+ NS: tl.constexpr, BLK_NS: tl.constexpr, BLOCK_H: tl.constexpr,+):+ """Single-pass mHC: the layer input mixes with the PREVIOUS sublayer's pre gates, so the+ gate math for the next sublayer (stage 2) does not gate stage 3 -- one launch does both:+ every program mixes one hidden chunk, program 0 of each token also reduces the split+ partials and writes the next sublayer's gates."""+ t = tl.program_id(0).to(tl.int64)+ hb = tl.program_id(1)+ offs_h = hb * BLOCK_H + tl.arange(0, BLOCK_H)+ h_mask = offs_h < H+ offs_n = tl.arange(0, N)+ pre = tl.load(pre_ptr + t * N + offs_n)+ li = tl.zeros([BLOCK_H], dtype=tl.float32)+ for n in tl.static_range(N):+ r = tl.load(res_out_ptr + (t * N + n) * H + offs_h, mask=h_mask, other=0.0).to(tl.float32)+ li += tl.sum(tl.where(offs_n == n, pre, 0.0)) * r+ tl.store(li_out_ptr + t * H + offs_h, li.to(li_out_ptr.dtype.element_ty), mask=h_mask)+ if hb == 0:+ _mhc_gates(+ t, sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr, post_out_ptr, comb_out_ptr, pre_out_ptr,+ rms_eps, hc_eps, post_mult, SINKHORN, H, N, MIX, BLK_MIX, NS, BLK_NS,+ )+++def _stage1_hidden_tile(h: int) -> int:+ """The stage-1 hidden split. It fixes the fp32 reduction order over the hidden width, so it+ depends on the width alone and results reproduce across processes, machines and chunk sizes.+ 128 gives 32 / 40 programs per token at widths 4096 / 5120."""+ return min(128, triton.next_power_of_2(h))+++def _mhc_fused_launch(x: torch.Tensor,residual: torch.Tensor,post_mix: torch.Tensor | None,@@ -173,11 +229,12 @@ def mhc_fused_post_pre_triton(hc_eps: float,post_mult: float,sinkhorn_repeat: int,-) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:- """Fused hc_post (skipped when ``post_mix is None``) + hc_pre. Three-stage- split-K: the GEMV/sq-sum reduction fans out over NS hidden slices. Returns- (residual_new [T,N,H] bf16, post [T,N,1] fp32, comb [T,N,N] fp32,- layer_input [T,H] bf16)."""+ pre_mix_in: torch.Tensor | None,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:+ """The three stages behind both variants. ``pre_mix_in`` selects single-pass: the+ layer input is mixed with it instead of with the pre gates stage 2 just computed+ (those are still returned, for the next sublayer). Returns (residual_new, post,+ comb, pre_new [T,N] fp32, layer_input)."""t, n, h = residual.shapemix = 2 * n + n * nassert fn.shape == (mix, n * h) and fn.dtype == torch.float32@@ -190,12 +247,7 @@ def mhc_fused_post_pre_triton(comb_out = torch.empty(t, n, n, dtype=torch.float32, device=dev)li_out = torch.empty(t, h, dtype=residual.dtype, device=dev)- block_h = min(512, triton.next_power_of_2(h))- # NS feeds a tl.arange in stage 2 -> keep it a power of two.- ns = 1- while ns * 2 <= min(16, h // block_h):- ns *= 2- split = triton.cdiv(triton.cdiv(h, ns), block_h) * block_h+ block_h = split = _stage1_hidden_tile(h)ns = triton.cdiv(h, split)blk_mix = triton.next_power_of_2(mix)sq_part = torch.empty(t, ns, dtype=torch.float32, device=dev)@@ -213,20 +265,83 @@ def mhc_fused_post_pre_triton(HAS_POST=has_post,num_warps=4, num_stages=2,)- _mhc_stage2_kernel[(t,)](- sq_part, mix_part, hc_scale, hc_base,- post_out, comb_out, pre_out,- rms_eps, hc_eps, post_mult,- SINKHORN=sinkhorn_repeat,- H=h, N=n, MIX=mix, BLK_MIX=blk_mix, NS=ns,- num_warps=1,+ block_h3 = min(1024, triton.next_power_of_2(h))+ if pre_mix_in is None:+ # two-pass: stage 3 mixes with the gates stage 2 just produced+ _mhc_stage2_kernel[(t,)](+ sq_part, mix_part, hc_scale, hc_base,+ post_out, comb_out, pre_out,+ rms_eps, hc_eps, post_mult,+ SINKHORN=sinkhorn_repeat,+ H=h, N=n, MIX=mix, BLK_MIX=blk_mix, NS=ns, BLK_NS=triton.next_power_of_2(ns),+ num_warps=1,+ )+ _mhc_stage3_kernel[(t, triton.cdiv(h, block_h3))](+ res_out, pre_out, li_out,+ H=h, N=n, BLOCK_H=block_h3,+ num_warps=4,+ )+ else:+ # single-pass: the mix uses the previous sublayer's gates, so stages 2 and 3 share a launch+ assert pre_mix_in.shape == (t, n), (pre_mix_in.shape, (t, n))+ _mhc_stage23_kernel[(t, triton.cdiv(h, block_h3))](+ res_out, pre_mix_in.to(torch.float32).contiguous(), li_out,+ sq_part, mix_part, hc_scale, hc_base,+ post_out, comb_out, pre_out,+ rms_eps, hc_eps, post_mult,+ SINKHORN=sinkhorn_repeat,+ H=h, N=n, MIX=mix, BLK_MIX=blk_mix, NS=ns, BLK_NS=triton.next_power_of_2(ns), BLOCK_H=block_h3,+ num_warps=4,+ )+ return res_out, post_out.view(t, n, 1), comb_out, pre_out, li_out+++def mhc_fused_post_pre_triton(+ x: torch.Tensor,+ residual: torch.Tensor,+ post_mix: torch.Tensor | None,+ comb_mix: torch.Tensor | None,+ fn: torch.Tensor,+ hc_scale: torch.Tensor,+ hc_base: torch.Tensor,+ rms_eps: float,+ hc_eps: float,+ post_mult: float,+ sinkhorn_repeat: int,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:+ """Fused hc_post (skipped when ``post_mix is None``) + hc_pre. Three-stage+ split-K: the GEMV/sq-sum reduction fans out over NS hidden slices. Returns+ (residual_new [T,N,H] bf16, post [T,N,1] fp32, comb [T,N,N] fp32,+ layer_input [T,H] bf16)."""+ res_out, post_out, comb_out, _pre_out, li_out = _mhc_fused_launch(+ x, residual, post_mix, comb_mix, fn, hc_scale, hc_base,+ rms_eps, hc_eps, post_mult, sinkhorn_repeat, pre_mix_in=None,)- _mhc_stage3_kernel[(t, triton.cdiv(h, 1024))](- res_out, pre_out, li_out,- H=h, N=n, BLOCK_H=min(1024, triton.next_power_of_2(h)),- num_warps=4,+ return res_out, post_out, comb_out, li_out+++def mhc_fused_post_pre_single_pass_triton(+ x: torch.Tensor,+ residual: torch.Tensor,+ post_mix: torch.Tensor | None,+ comb_mix: torch.Tensor | None,+ pre_mix_in: torch.Tensor,+ fn: torch.Tensor,+ hc_scale: torch.Tensor,+ hc_base: torch.Tensor,+ rms_eps: float,+ hc_eps: float,+ post_mult: float,+ sinkhorn_repeat: int,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:+ """Single-pass mHC (DeepSeek-V4.1): same fused stages, but the layer input is mixed+ with the PREVIOUS sublayer's ``pre_mix_in`` and this sublayer's pre gates come back+ as ``pre_new`` for the next one. Returns (residual_new, post [T,N,1], comb [T,N,N],+ pre_new [T,N] fp32, layer_input)."""+ return _mhc_fused_launch(+ x, residual, post_mix, comb_mix, fn, hc_scale, hc_base,+ rms_eps, hc_eps, post_mult, sinkhorn_repeat, pre_mix_in=pre_mix_in,)- return res_out, post_out.view(t, n, 1), comb_out, li_out-__all__ = ["mhc_fused_post_pre_triton"]+__all__ = ["mhc_fused_post_pre_triton", "mhc_fused_post_pre_single_pass_triton"]diff --git a/python/freetoken/kvcache/__init__.py b/python/freetoken/kvcache/__init__.pyindex 1792da2b..e0cf606d 100644--- a/python/freetoken/kvcache/__init__.py+++ b/python/freetoken/kvcache/__init__.py@@ -34,7 +34,7 @@ def resolve_pool_class(model_config: ModelConfig) -> type[BaseKVCachePool]:specs_fn = getattr(model_config, "kv_cache_group_specs", None)if specs_fn is None:if getattr(model_config, "dsv4_args", None) is not None:- from .dsv4_paged_pool import DSV4PagedKVCache+ from .dsv4.v4_pool import DSV4PagedKVCachereturn DSV4PagedKVCachefrom .mha_pool import MHAKVCache@@ -43,9 +43,13 @@ def resolve_pool_class(model_config: ModelConfig) -> type[BaseKVCachePool]:specs = list(specs_fn())types = {spec.attn_type for spec in specs}if AttnType.DSV4 in types:- from .dsv4_paged_pool import DSV4PagedKVCache+ from .dsv4.v4_pool import DSV4PagedKVCachereturn DSV4PagedKVCache+ if AttnType.DSV41 in types:+ from .dsv4.v41_pool import DSV41PagedKVCache++ return DSV41PagedKVCacheif AttnType.SWA in types:from .hybrid_swa_pool import HybridSWAKVCache@@ -80,9 +84,9 @@ def create_kv_pool(config, num_pages: int, device: torch.device, dtype: torch.dt"""Build the engine's KV pool for ``num_pages`` USABLE pages (the dummy page and everysecondary tier -- window pool, index slab, state rings -- are derived here or insidethe pool). Single factory entry for all pool families, DSV4 included."""- from .dsv4_cost_model import _dsv4_pool_sizes+ from .dsv4.v4_cost_model import _dsv4_pool_sizesfrom .hybrid_swa_pool import _naive_swa_num_tokens, _swa_paged_num_tokens- from .dsv4_paged_pool import DSV4PagedKVCache+ from .dsv4.v4_pool import DSV4PagedKVCachemodel_config = config.model_configif resolve_pool_class(model_config) is DSV4PagedKVCache:@@ -99,6 +103,21 @@ def create_kv_pool(config, num_pages: int, device: torch.device, dtype: torch.dt)pool._init_paged_state(config.max_running_req, config.cache_type != "naive")return pool+ if getattr(model_config, "dsv41_args", None) is not None:+ from .dsv4.v41_cost_model import _dsv41_pool_sizes+ from .dsv4.v41_pool import DSV41PagedKVCache++ # Same route as DSV4: the generic CacheManager over the shared page table; the pool is the+ # swa_pool plug-in (window tier + per-source packed main / index pools + state rings).+ pool = DSV41PagedKVCache(+ sizes=_dsv41_pool_sizes(config, num_pages + 1), # +1 for dummy page+ args=model_config.dsv41_args,+ device=device,+ dtype=dtype,+ n_scratch=config.max_running_req + 1,+ )+ pool._init_paged_state(config.max_running_req, config.cache_type != "naive")+ return poolnum_swa_tokens = None# Both the naive and radix SWA paths share the global-paged swa pool; radix sizes it bydiff --git a/python/freetoken/kvcache/base.py b/python/freetoken/kvcache/base.pyindex 95669e8c..569da1cb 100644--- a/python/freetoken/kvcache/base.py+++ b/python/freetoken/kvcache/base.py@@ -35,6 +35,14 @@ def spec_kv_bytes_per_token(spec, config) -> int:return per_token + spec.index_head_dim * spec.num_index_layers * 2 // spec.index_ratio+@dataclass(frozen=True)+class WindowPoolSpec:+ """Window resize units and minimum usable capacity, excluding reserved pages."""++ page_size: int+ min_pages: int++class BaseKVCachePool(ABC):"""Base class for key-value caches.@@ -45,6 +53,15 @@ class BaseKVCachePool(ABC):# model re-bound after a rebuild; the engine asks before it resizes.needs_rebind_on_rebuild: ClassVar[bool] = False+ @classmethod+ def window_spec(cls, config) -> WindowPoolSpec | None:+ return None++ @property+ def window_pages(self) -> int:+ """Current usable window pages in the units declared by window_spec."""+ return 0+# ---- sizing/cost classmethods: run BEFORE the pool exists (startup budget solve,# --moe-cache-auto). The engine measures memory and passes bytes in; each pool family# implements kv_cost for ITS OWN buffers only (the engine sums families, e.g. addsdiff --git a/python/freetoken/kvcache/cache_status.py b/python/freetoken/kvcache/cache_status.pyindex 6aa13e7f..5b531ce2 100644--- a/python/freetoken/kvcache/cache_status.py+++ b/python/freetoken/kvcache/cache_status.py@@ -3,13 +3,14 @@ from __future__ import annotationsfrom typing import Any, Dict+def window_pool_spec(config):+ from . import resolve_pool_class++ return resolve_pool_class(config.model_config).window_spec(config)++def _supports_swa_ratio(config) -> bool:- """Whether ``swa_full_tokens_ratio`` sizes a separate window pool for this model -- DSV4- (always) or a radix-SWA model (Gemma). Gates the ratio in telemetry and rebuild."""- mc = config.model_config- if mc.dsv4_args is not None:- return True- return mc.has_swa_attention and config.cache_type == "swa_radix"+ return window_pool_spec(config) is not Nonedef compute_cache_unit_bytes(engine: "Engine") -> Dict[str, int]:@@ -120,18 +121,8 @@ def compute_cache_floors(engine: "Engine") -> Dict[str, int]:return int(_linear_pool_min_slots(config) - 1)def _swa() -> int:- # Report usable tokens, matching num_swa_pages in rebuild requests.- # DSV4's physical floor includes a dummy page; the SWA floor already excludes slot 0.- from .dsv4_cost_model import _dsv4_window_floor_pages- from .hybrid_swa_pool import _swa_pool_floor-- mc = config.model_config- if mc.dsv4_args is not None:- P = mc.dsv4_args.window_size- return int((_dsv4_window_floor_pages(config, P) - 1) * P)- if not (mc.has_swa_attention and config.cache_type == "swa_radix"):- return 0- return int(_swa_pool_floor(config))+ spec = window_pool_spec(config)+ return spec.page_size * spec.min_pages if spec is not None else 0for key, fn in (("kv_tokens", _kv), ("moe_experts", _moe), ("mamba_slots", _mamba),("swa_tokens", _swa)):@@ -157,19 +148,10 @@ def compute_cache_pools(engine: "Engine") -> Dict[str, int]:pools["num_pages"] = int(engine.num_pages or 0)if config is not None:pools["page_size"] = int(config.page_size or 0)- # Window pool: its own page unit (swa_page_size -- DSV4 windows are P-token pages,- # radix-SWA is token-granular page_size 1) and the concrete current size in that unit- # (num_swa_pages, usable count). Same source as the scheduler's _current_cache_geometry.- # Both 0 for models without a window pool. Lets a client denominate the swa control.- mc = config.model_config- if mc.dsv4_args is not None:- pools["swa_page_size"] = int(mc.dsv4_args.window_size or 0)- sizes = getattr(engine.kv_cache, "sizes", None) # usable = physical minus dummy- if sizes is not None:- pools["num_swa_pages"] = max(0, int(sizes.n_win_pages) - 1)- elif mc.has_swa_attention and config.cache_type == "swa_radix":- pools["swa_page_size"] = 1 # usable = pool tokens minus the slot-0 sentinel- pools["num_swa_pages"] = max(0, int(getattr(engine.kv_cache, "swa_num_tokens", 0) or 0) - 1)+ spec = window_pool_spec(config)+ if spec is not None:+ pools["swa_page_size"] = spec.page_size+ pools["num_swa_pages"] = max(0, int(engine.kv_cache.window_pages))moe = engine.moe_offload_cacheif moe is not None:pools["moe_cache_size"] = int(moe.cache_size or 0)diff --git a/python/freetoken/kvcache/dsa_pool.py b/python/freetoken/kvcache/dsa_pool.pyindex 13225cec..d7c36fd7 100644--- a/python/freetoken/kvcache/dsa_pool.py+++ b/python/freetoken/kvcache/dsa_pool.py@@ -2,7 +2,7 @@``MLAKVCache`` is the MHA pool's sibling for latent-KV models: ONE slab holding theper-token latent ``ckv (kv_lora_rank) | kpe (qk_rope_head_dim)`` -- there is no-separate V (``v_cache`` aliases ``k_cache``, same convention as dsv4_paged_pool's+separate V (``v_cache`` aliases ``k_cache``, same convention as dsv4/v4_pool'ssingle-latent tiers). ``DSAKVCache`` extends it with the DeepSeek-Sparse-Attentionindex-key slab: one ``index_head_dim``-wide bf16 row per token per full-indexerlayer, addressed by the SAME physical rows as the latent slab (GLM-5.2, page 1;@@ -75,7 +75,7 @@ class MLAKVCache(BaseKVCachePool):)def v_cache(self, layer_id: int) -> torch.Tensor:- # MLA: K == V (single latent); same buffer, dsv4_paged_pool precedent.+ # MLA: K == V (single latent); same buffer, dsv4/v4_pool precedent.return self.k_cache(layer_id)def latent_rows(self, layer_id: int) -> torch.Tensor:diff --git a/python/freetoken/kvcache/dsv4/__init__.py b/python/freetoken/kvcache/dsv4/__init__.pynew file mode 100644index 00000000..7531470f--- /dev/null+++ b/python/freetoken/kvcache/dsv4/__init__.py@@ -0,0 +1 @@+"""Paged KV pools for the DeepSeek-V4 family."""diff --git a/python/freetoken/kvcache/dsv4/v41_cost_model.py b/python/freetoken/kvcache/dsv4/v41_cost_model.pynew file mode 100644index 00000000..4db1fb18--- /dev/null+++ b/python/freetoken/kvcache/dsv4/v41_cost_model.py@@ -0,0 +1,216 @@+"""Cost model + per-tier sizing for the DSV41 paged KV pool (DeepSeek-V4.1).++Same shape as ``v4_cost_model``: one bytes-per-P-token number for the budget division, and+independent per-tier sizing from the anchor ``full_token = num_pages * P``. The tiers:++* window pool, every layer -- ``swa_ratio`` of the full history, packed fp8 rows+* main KV pool, per kv source -- ``full_token // ratio`` packed fp4 rows+* index-key pool, per kv source -- ``full_token // ratio`` packed fp4 rows+* compress-state ring, ratio>1 sources -- ``ring_size`` fp32 ``kv|score`` slots per WINDOW page++Sizing reads ``ModelConfig.dsv41_args`` by attribute, like ``v4_cost_model`` reads ``dsv4_args``:+the kvcache package never imports the model package. All byte widths come from the tier row formats+below, so there is exactly one place that knows what a row costs.+"""++from __future__ import annotations++from dataclasses import dataclass, field++from .v41_row_format import FP4_E4M3_B16, FP4_E8M0_B32, FP8_E8M0_B32+from .window_tier import reserved_window_pages++_INT64_BYTES = 8+_FP32_BYTES = 4++WIN_FMT = FP8_E8M0_B32+MAIN_FMT = FP4_E4M3_B16+IDX_FMT = FP4_E8M0_B32+++def _win_row_bytes(args) -> int:+ return WIN_FMT.row_bytes(args.head_dim)+++def _main_row_bytes(args) -> int:+ return MAIN_FMT.row_bytes(args.head_dim)+++def _idx_row_bytes(args) -> int:+ return IDX_FMT.row_bytes(args.index_head_dim)+++def _state_bytes(args) -> int:+ """One compress-state ring slot: fp32 ``kv | score`` of the latent width."""+ return 2 * args.head_dim * _FP32_BYTES+++def private_window_layer_ids(args) -> tuple[int, ...]:+ """Layers whose window KV is REQUEST-PRIVATE: kept in a per-page-table-row ring of ``window``+ slots (slot = row * window + pos % window) instead of the shared, page-bound window pool, so+ it never lands in radix-shared pages. The decoder layers under Decoder SWA Bounded Replay:+ their KV is recomputed per request over the prompt's last window (positions before it are+ never computed), so a shared page could hold rows another request never wrote. Exact mode+ computes every position and keeps the decoder on the shared pool."""+ if args.swa_decoder_replay == "exact":+ return ()+ return tuple(range(args.decoder_start_layer, args.n_layers))+++@dataclass+class DSV41PoolSizes:+ """Per-tier row counts derived from the budget anchor ``full_token``; the global tiers are+ keyed by kv-source layer id."""++ P: int+ swa_ratio: float+ full_token: int+ n_win_slots: int+ n_win_pages: int+ main_rows: dict[int, int] = field(default_factory=dict)+ idx_rows: dict[int, int] = field(default_factory=dict)+ state_slots: dict[int, int] = field(default_factory=dict) # ratio>1 sources only+++def dsv41_pool_sizes(+ num_pages: int, args, swa_ratio: float, P: int, n_win_pages: int | None = None+) -> DSV41PoolSizes:+ full_token = num_pages * P+ if n_win_pages is None:+ n_win_pages = (round(swa_ratio * full_token) + P - 1) // P+ n_win_pages = min(n_win_pages, num_pages)+ sizes = DSV41PoolSizes(P=P, swa_ratio=swa_ratio, full_token=full_token, n_win_slots=n_win_pages * P, n_win_pages=n_win_pages)+ for src in args.backbone_kv_sources:+ ratio = args.compress_ratios[src]+ sizes.main_rows[src] = full_token // ratio+ sizes.idx_rows[src] = full_token // ratio+ if ratio > 1:+ sizes.state_slots[src] = n_win_pages * ratio # ring size == ratio+ return sizes+++def dsv41_pool_bytes(sizes: DSV41PoolSizes, args, n_scratch: int = 1) -> int:+ """Exact bytes a ``DSV41PagedKVCache`` built from ``sizes`` allocates (mirror of ``total_bytes``)."""+ n_private = len(private_window_layer_ids(args))+ total = (args.n_layers - n_private) * sizes.n_win_slots * _win_row_bytes(args)+ total += n_private * n_scratch * sizes.P * _win_row_bytes(args) # per-request rings+ total += (sizes.full_token + 1) * _INT64_BYTES # full_to_window (+ sentinel row)+ for src in args.backbone_kv_sources:+ total += (sizes.main_rows[src] + n_scratch) * _main_row_bytes(args)+ total += (sizes.idx_rows[src] + n_scratch) * _idx_row_bytes(args)+ if src in sizes.state_slots:+ total += (sizes.state_slots[src] + 1) * _state_bytes(args)+ return int(total)+++def dsv41_cache_per_page(args, swa_ratio: float, P: int) -> int:+ """Marginal bytes per P-token page across all tiers (window scaled by ``swa_ratio``)."""+ n_shared = args.n_layers - len(private_window_layer_ids(args))+ total = n_shared * round(swa_ratio * P) * _win_row_bytes(args)+ for src in args.backbone_kv_sources:+ ratio = args.compress_ratios[src]+ total += (P // ratio) * (_main_row_bytes(args) + _idx_row_bytes(args))+ if ratio > 1:+ total += round(swa_ratio * ratio) * _state_bytes(args)+ return int(total)+++def dsv41_kv_unit_bytes(args, P: int) -> int:+ """FULL-tier bytes per full-history token (main + index pools + the mapping); the slider's+ ``kv_bytes_per_token``. 890 B/token for DeepSeek-V4.1-Flash before the mapping."""+ per_page = P * _INT64_BYTES+ for src in args.backbone_kv_sources:+ per_page += (P // args.compress_ratios[src]) * (_main_row_bytes(args) + _idx_row_bytes(args))+ return -(-per_page // P)+++def dsv41_window_unit_bytes(args, P: int) -> int:+ """WINDOW-tier bytes per window token (sliding KV on every shared-window layer + the state rings)."""+ per_page = (args.n_layers - len(private_window_layer_ids(args))) * P * _win_row_bytes(args)+ for src in args.backbone_kv_sources:+ ratio = args.compress_ratios[src]+ if ratio > 1:+ per_page += ratio * _state_bytes(args)+ return -(-per_page // P)+++def dsv41_solve_num_pages(+ available_bytes: int, args, swa_ratio: float, floor_win_pages: int, P: int, n_scratch: int = 1+) -> DSV41PoolSizes:+ """Largest budget-respecting pool; the window floor is honored in pages and the total is+ byte-checked. Raises ``ValueError`` when even the minimal pool does not fit."""++ def _sizes(num: int) -> DSV41PoolSizes:+ win = max(floor_win_pages, (round(swa_ratio * num * P) + P - 1) // P)+ return dsv41_pool_sizes(num, args, swa_ratio, P, n_win_pages=win)++ lo = max(floor_win_pages, 2)+ if dsv41_pool_bytes(_sizes(lo), args, n_scratch) > available_bytes:+ raise ValueError(+ f"DSV41 KV budget {available_bytes} bytes cannot fit the minimal pool ({lo} pages incl. the "+ f"window working-set floor {floor_win_pages}); raise memory_ratio or lower max_running_req/max_seq_len"+ )+ hi = max(lo, available_bytes // max(1, dsv41_cache_per_page(args, 0.0, P)))+ while dsv41_pool_bytes(_sizes(hi), args, n_scratch) <= available_bytes:+ hi *= 2+ while lo < hi - 1:+ mid = (lo + hi) // 2+ if dsv41_pool_bytes(_sizes(mid), args, n_scratch) <= available_bytes:+ lo = mid+ else:+ hi = mid+ return _sizes(lo)+++def dsv41_auto_cost_model(args, swa_ratio: float, floor_win_pages: int, P: int, n_scratch: int = 1):+ """Affine ``(cache_per_page, fixed_cache_size, min_reserve_tokens)`` for the MoE-first auto planner."""+ per_page = dsv41_cache_per_page(args, swa_ratio, P) + P * _INT64_BYTES+ n0 = max(floor_win_pages, 2)+ win0 = max(floor_win_pages, (round(swa_ratio * n0 * P) + P - 1) // P)+ base = dsv41_pool_bytes(dsv41_pool_sizes(n0, args, swa_ratio, P, n_win_pages=win0), args, n_scratch)+ # The engine combines this structural floor with the configured KV reserve.+ return per_page, max(0, base - n0 * per_page), n0 * P+++# ---- config-facing sizing (EngineConfig in, sizes out) ----+++def _dsv41_swa_ratio(config) -> float:+ return float(config.swa_full_tokens_ratio)+++def _dsv41_window_floor_pages(config, args) -> int:+ """Minimum window pages: one prefill chunk's reach (capped at 8 pages) + the running requests'+ working set (see ``reserved_window_pages``)."""+ P = args.window_size+ prefill_reach_pages = (config.max_seq_len + P - 1) // P+ radix = config.cache_type != "naive"+ return min(prefill_reach_pages, 8) + reserved_window_pages(config.max_running_req, radix)+++def _dsv41_pool_sizes(config, num_pages: int, num_swa_pages: int | None = None) -> DSV41PoolSizes:+ """Sizes for ``num_pages`` PHYSICAL pages (dummy included). Window precedence: an explicit+ ``num_swa_pages`` > ``config.swa_num_pages_override`` > ``swa_ratio`` x full, floored ONCE in pages."""+ args = config.model_config.dsv41_args+ P = args.window_size+ swa_ratio = _dsv41_swa_ratio(config)+ floor_pages = _dsv41_window_floor_pages(config, args)+ target = num_swa_pages if num_swa_pages is not None else config.swa_num_pages_override+ if target is not None:+ win = min(num_pages, max(floor_pages, int(target) + 1))+ else:+ win = max(floor_pages, (round(swa_ratio * num_pages * P) + P - 1) // P)+ return dsv41_pool_sizes(num_pages, args, swa_ratio, P, n_win_pages=win)+++__all__ = [+ "DSV41PoolSizes",+ "dsv41_auto_cost_model",+ "dsv41_cache_per_page",+ "dsv41_kv_unit_bytes",+ "dsv41_pool_bytes",+ "dsv41_pool_sizes",+ "dsv41_solve_num_pages",+ "dsv41_window_unit_bytes",+ "private_window_layer_ids",+]diff --git a/python/freetoken/kvcache/dsv4/v41_pool.py b/python/freetoken/kvcache/dsv4/v41_pool.pynew file mode 100644index 00000000..36d3986a--- /dev/null+++ b/python/freetoken/kvcache/dsv4/v41_pool.py@@ -0,0 +1,292 @@+"""DSV41 paged KV pool (DeepSeek-V4.1): packed byte tiers over the shared page table.++Tiers, sized from a budget (``v41_cost_model``) not from ``num_requests``:++* ``window_pool[L]`` -- every layer; the P-sliding KV ring in ``win_fmt`` (fp8) rows, page-granular.+* ``main_pool[src]`` -- per kv-source layer; compressed KV latents in ``main_fmt`` (fp4) rows.+* ``idx_pool[src]`` -- per kv-source layer; indexer keys in ``idx_fmt`` (fp4) rows.+* ``state_ring[src]`` -- ratio>1 sources; the per-window-page compress-state ring (fp32 ``kv|score``).++Consumer layers (Reindex / Reuse modes) alias their source's pools: ``main_pool_of(layer)``.++Addressing is the shared ``WindowTierPagedPool``'s (``kvcache/dsv4/window_tier.py``, also under the DSV4+pool): the page table maps ``(table_idx, pos)`` to a full-token loc; ``full_to_window`` maps that to a+window slot (page-bound by the free list); the main / index rows are pure arithmetic ``full_loc //+ratio``; the ring slot derives from the window slot. Scratch rows past every main / index pool's+arithmetic range give a graph-safe destination for the decode rows whose compressed group did not+complete this step.++Rows are written through ``pack_rows`` (quantize + scatter) and read inside the attention /+indexer kernels; ``read_*`` helpers dequantize for tests and torch reference paths.+"""++from __future__ import annotations++import torch++from freetoken.utils import init_logger++from .v41_cost_model import (+ IDX_FMT,+ MAIN_FMT,+ WIN_FMT,+ DSV41PoolSizes,+ _idx_row_bytes,+ _main_row_bytes,+ _win_row_bytes,+ dsv41_kv_unit_bytes,+ dsv41_window_unit_bytes,+ private_window_layer_ids,+)+from .window_tier import CompressStateRing, WindowTierPagedPool++logger = init_logger(__name__)+++class DSV41PagedKVCache(WindowTierPagedPool):+ win_fmt = WIN_FMT+ main_fmt = MAIN_FMT+ idx_fmt = IDX_FMT++ def __init__(+ self,+ sizes: DSV41PoolSizes,+ args,+ device: torch.device,+ dtype: torch.dtype = torch.bfloat16,+ n_scratch: int = 1,+ ) -> None:+ for fmt, dim in ((self.win_fmt, args.head_dim), (self.main_fmt, args.head_dim), (self.idx_fmt, args.index_head_dim)):+ fmt.validate_dim(dim)+ for src in args.backbone_kv_sources:+ if args.window_size % args.compress_ratios[src]:+ raise ValueError(f"window {args.window_size} must be a multiple of compress ratio {args.compress_ratios[src]}")+ self.args = args+ self.private_window_layer_ids = private_window_layer_ids(args)+ self.sizes = sizes+ self._device = device+ self._dtype = dtype # the model's compute dtype; the tiers themselves are packed bytes+ self.P = args.window_size+ self.head_dim = args.head_dim+ self.index_head_dim = args.index_head_dim+ self.n_scratch = int(n_scratch)+ self._paged_params: tuple[int, bool] | None = None+ self.full_loc_map: torch.Tensor | None = None+ self._alloc_buffers()++ # ----- buffers -----+ def _alloc_buffers(self) -> None:+ sizes, args, dev = self.sizes, self.args, self._device+ self.full_to_window = torch.full((sizes.full_token + 1,), -1, dtype=torch.int64, device=dev)+ # shared layers: the page-bound window pool; private layers: one ring of P slots per+ # page-table row (n_scratch rows, the dummy included), addressed row * P + pos % P+ self.window_pool: list[torch.Tensor] = [+ torch.zeros(self.n_scratch * self.P if self.is_private_window(l) else sizes.n_win_slots, _win_row_bytes(args), dtype=torch.uint8, device=dev)+ for l in range(args.n_layers)+ ]+ self.main_pool: dict[int, torch.Tensor] = {}+ self.idx_pool: dict[int, torch.Tensor] = {}+ self.state_ring: dict[int, CompressStateRing] = {}+ self.scratch_base: dict[int, int] = {}+ for src in args.backbone_kv_sources:+ rows = sizes.main_rows[src]+ self.scratch_base[src] = rows+ self.main_pool[src] = torch.zeros(rows + self.n_scratch, _main_row_bytes(args), dtype=torch.uint8, device=dev)+ self.idx_pool[src] = torch.zeros(sizes.idx_rows[src] + self.n_scratch, _idx_row_bytes(args), dtype=torch.uint8, device=dev)+ if src in sizes.state_slots:+ self.state_ring[src] = CompressStateRing(+ n_slots=sizes.state_slots[src], ring_size=args.compress_ratios[src], overlap=False,+ head_dim=args.head_dim, device=dev,+ )++ def total_bytes(self) -> int:+ n = self.full_to_window.numel() * self.full_to_window.element_size()+ n += sum(t.numel() for t in self.window_pool)+ n += sum(t.numel() for t in self.main_pool.values())+ n += sum(t.numel() for t in self.idx_pool.values())+ n += sum(r.buffer.numel() * r.buffer.element_size() for r in self.state_ring.values())+ return int(n)++ @property+ def prefix_replay_tokens(self) -> int:+ """Prompt-tail tokens a prefix hit must leave to the prefill: bounded replay runs the decoder on+ the prompt's last window from that window's encoder outputs, which no cache keeps."""+ return self.P if self.private_window_layer_ids else 0++ # ----- private window rings -----+ def is_private_window(self, layer_id: int) -> bool:+ return layer_id in self.private_window_layer_ids++ def ring_slots(self, table_rows: torch.Tensor, positions: torch.Tensor) -> torch.Tensor:+ """Ring slots of ``positions`` (int64, broadcastable with ``table_rows``) for private window layers;+ negative positions map to ``-1``."""+ slots = table_rows.to(torch.int64) * self.P + positions % self.P+ return torch.where(positions < 0, -1, slots)++ # ----- source aliasing -----+ def source_of(self, layer_id: int) -> int:+ src = self.args.roles[layer_id].kv_source+ assert src is not None, f"layer {layer_id} is window-only"+ return src++ def main_pool_of(self, layer_id: int) -> torch.Tensor:+ return self.main_pool[self.source_of(layer_id)]++ def idx_pool_of(self, layer_id: int) -> torch.Tensor:+ return self.idx_pool[self.source_of(layer_id)]++ def ratio_of(self, layer_id: int) -> int:+ return self.args.roles[layer_id].ratio++ # ----- engine-facing sizing / rebuild surface -----+ @classmethod+ def kv_cost(cls, config) -> tuple[int, int, int, int]:+ from .v41_cost_model import _dsv41_swa_ratio, _dsv41_window_floor_pages, dsv41_auto_cost_model++ args = config.model_config.dsv41_args+ P = args.window_size+ per_page, fixed, min_reserve = dsv41_auto_cost_model(+ args, _dsv41_swa_ratio(config), _dsv41_window_floor_pages(config, args), P, n_scratch=config.max_running_req + 1+ )+ return per_page, fixed, config.page_size, min_reserve++ @classmethod+ def solve_num_pages(cls, config, available_memory: int) -> int:+ from freetoken.utils import mem_GB++ from .v41_cost_model import (+ _dsv41_pool_sizes,+ _dsv41_swa_ratio,+ _dsv41_window_floor_pages,+ dsv41_pool_bytes,+ dsv41_solve_num_pages,+ )++ args = config.model_config.dsv41_args+ P = args.window_size+ num_pages = config.num_page_override+ if num_pages is None:+ sizes = dsv41_solve_num_pages(+ available_memory, args, _dsv41_swa_ratio(config), _dsv41_window_floor_pages(config, args), P,+ n_scratch=config.max_running_req + 1,+ )+ num_pages = sizes.full_token // P - 1 # one physical page is the dummy+ else:+ floor = _dsv41_window_floor_pages(config, args)+ if num_pages < floor:+ raise ValueError(+ f"--num-pages {num_pages} ({num_pages * P} tokens) is below the DSV41 window working-set "+ f"floor {floor} pages ({floor * P} tokens); raise --num-pages or lower max_running_req/max_seq_len"+ )+ sizes = _dsv41_pool_sizes(config, num_pages + 1)+ assert num_pages > 1, "Not enough memory for KV cache, try reducing --num-pages"+ real = dsv41_pool_bytes(sizes, args, config.max_running_req + 1)+ logger.info(+ f"Allocating {num_pages * P} tokens for DSV41 KV cache ({sizes.n_win_pages} window pages), total = {mem_GB(real)}"+ )+ return num_pages++ @classmethod+ def window_spec(cls, config):+ from ..base import WindowPoolSpec+ from .v41_cost_model import _dsv41_window_floor_pages++ args = config.model_config.dsv41_args+ return WindowPoolSpec(args.window_size, _dsv41_window_floor_pages(config, args) - 1)++ @classmethod+ def min_kv_tokens(cls, config) -> int:+ from .v41_cost_model import _dsv41_window_floor_pages++ args = config.model_config.dsv41_args+ return _dsv41_window_floor_pages(config, args) * args.window_size++ def validate_rebuild(+ self, config, *, num_pages: int | None, target_moe: int, per_expert_bytes: int,+ baseline_free: int, weights_bytes: int, current_num_pages: int,+ extra_fixed_bytes: int = 0, extra_note: str = "",+ num_swa_pages: int | None = None, **targets,+ ) -> None:+ from freetoken.engine.cache_budget import net_cache_budget_bytes+ from freetoken.utils import mem_GB++ from ..base import CacheRebuildRejected+ from .v41_cost_model import _dsv41_pool_sizes, _dsv41_window_floor_pages, dsv41_pool_bytes++ if num_pages is not None:+ floor = _dsv41_window_floor_pages(config, self.args)+ if num_pages < floor:+ raise CacheRebuildRejected(+ f"num_pages {num_pages} is below the DSV41 window working-set floor {floor} "+ f"(max_running_req={config.max_running_req}); admission would deadlock"+ )+ if num_pages is not None or num_swa_pages is not None:+ target_pages = num_pages if num_pages is not None else current_num_pages+ kv_sizes = _dsv41_pool_sizes(config, target_pages + 1, num_swa_pages=num_swa_pages)+ else:+ kv_sizes = self.sizes+ budget = net_cache_budget_bytes(config.memory_ratio, baseline_free, weights_bytes, 0)+ need = target_moe * per_expert_bytes + dsv41_pool_bytes(kv_sizes, self.args, config.max_running_req + 1)+ if need > budget:+ kv_part = f"kv={num_pages} P-pages" if num_pages is not None else "kv=current pool"+ raise CacheRebuildRejected(+ f"requested cache (moe={target_moe} slots, {kv_part}) needs {mem_GB(need)} > budget {mem_GB(budget)}; "+ "old cache kept, still serving"+ )++ def rebuild_from_config(self, config, num_pages: int, *, num_swa_pages: int | None = None) -> None:+ from .v41_cost_model import _dsv41_pool_sizes++ self.rebuild(_dsv41_pool_sizes(config, num_pages + 1, num_swa_pages=num_swa_pages))++ def _drop_buffers(self) -> None:+ self.window_pool = self.main_pool = self.idx_pool = self.state_ring = None # type: ignore[assignment]++ def unit_bytes(self) -> tuple[int, int]:+ return dsv41_kv_unit_bytes(self.args, self.P), dsv41_window_unit_bytes(self.args, self.P)++ # ----- writes (quantize + scatter) -----+ def store_window(self, kv: torch.Tensor, layer_id: int, window_slot: torch.Tensor) -> None:+ from freetoken.kernel.triton.dsv41.pack import pack_rows++ pack_rows(kv, self.win_fmt, pool=self.window_pool[layer_id], row_ids=window_slot)++ def store_main(self, latent: torch.Tensor, source: int, rows: torch.Tensor) -> None:+ from freetoken.kernel.triton.dsv41.pack import pack_rows++ pack_rows(latent, self.main_fmt, pool=self.main_pool[source], row_ids=rows)++ def store_index(self, k: torch.Tensor, source: int, rows: torch.Tensor) -> None:+ from freetoken.kernel.triton.dsv41.pack import pack_rows++ pack_rows(k, self.idx_fmt, pool=self.idx_pool[source], row_ids=rows)++ # ----- reads (gather + dequantize; tests and torch reference paths) -----+ def read_window(self, layer_id: int, window_slot: torch.Tensor) -> torch.Tensor:+ from freetoken.kernel.triton.dsv41.pack import unpack_rows++ return unpack_rows(self.window_pool[layer_id], self.win_fmt, self.head_dim, window_slot)++ def read_main(self, source: int, rows: torch.Tensor) -> torch.Tensor:+ from freetoken.kernel.triton.dsv41.pack import unpack_rows++ return unpack_rows(self.main_pool[source], self.main_fmt, self.head_dim, rows)++ def read_index(self, source: int, rows: torch.Tensor) -> torch.Tensor:+ from freetoken.kernel.triton.dsv41.pack import unpack_rows++ return unpack_rows(self.idx_pool[source], self.idx_fmt, self.index_head_dim, rows)++ # ----- compress-state ring accessors -----+ def get_state(self, source: int, state_loc: torch.Tensor) -> torch.Tensor:+ return self.state_ring[source].get(state_loc)++ def set_state(self, source: int, state_loc: torch.Tensor, kv_score: torch.Tensor) -> None:+ self.state_ring[source].set(state_loc, kv_score)++ @property+ def num_layers(self) -> int:+ return self.args.n_layers+++__all__ = ["DSV41PagedKVCache"]diff --git a/python/freetoken/kvcache/dsv4/v41_row_format.py b/python/freetoken/kvcache/dsv4/v41_row_format.pynew file mode 100644index 00000000..f273be4f--- /dev/null+++ b/python/freetoken/kvcache/dsv4/v41_row_format.py@@ -0,0 +1,69 @@+"""RowFormat: how one DeepSeek-V4.1 KV row (a head_dim-wide vector) is stored in a packed byte pool.++DeepSeek-V4.1 keeps three cache tiers in three storage formats (tech report sec. 2.4.4 and+``inference/model.py``):++* SWA KV -- ``fp8_e8m0_b32``: e4m3 values, one ue8m0 (power-of-two) scale per 32 channels+* main KV -- ``fp4_e4m3_b16``: e2m1 values, one e4m3 scale per 16 channels (NVFP4 without the global scale)+* indexer K -- ``fp4_e8m0_b32``: e2m1 values, one ue8m0 scale per 32 channels (OCP MXFP4)++A row is laid out as ``[values | scales]``: ``value_bytes(dim)`` bytes of (packed) values followed by+``scale_bytes(dim)`` scale bytes, so one gather fetches both. e2m1 pairs pack the even channel into the low+nibble and the odd channel into the high nibble (``torch.float4_e2m1fn_x2`` / ``convert.py`` order).++The Triton side (``kernel/triton/dsv41/row_format.py``) selects the dequant path from ``RowFormat.code``+(a constexpr); the pack / unpack wrappers in ``kernel/triton/dsv41/pack.py`` reproduce the reference+``act_quant`` / ``fp4_act_quant`` numerics exactly, including the bf16 rounding of the dequantized value+the reference bakes into its cache.+"""++from __future__ import annotations++from dataclasses import dataclass+++@dataclass(frozen=True)+class RowFormat:+ name: str+ elem_bits: int # 16 (bf16), 8 (e4m3) or 4 (e2m1)+ group: int # channels per scale; 0 = unscaled+ scale: str | None # None | "e8m0" | "e4m3"+ code: int # Triton constexpr selector++ def value_bytes(self, dim: int) -> int:+ assert (dim * self.elem_bits) % 8 == 0, (dim, self.elem_bits)+ return dim * self.elem_bits // 8++ def scale_bytes(self, dim: int) -> int:+ if not self.group:+ return 0+ assert dim % self.group == 0, (dim, self.group)+ return dim // self.group++ def row_bytes(self, dim: int) -> int:+ return self.value_bytes(dim) + self.scale_bytes(dim)++ def num_groups(self, dim: int) -> int:+ return dim // self.group if self.group else 0++ def validate_dim(self, dim: int) -> None:+ if dim <= 0 or (dim * self.elem_bits) % 8 or (self.group and dim % self.group):+ raise ValueError(f"row format {self.name} cannot hold a {dim}-wide row")+++BF16 = RowFormat("bf16", elem_bits=16, group=0, scale=None, code=0)+FP8_E8M0_B32 = RowFormat("fp8_e8m0_b32", elem_bits=8, group=32, scale="e8m0", code=1)+FP4_E4M3_B16 = RowFormat("fp4_e4m3_b16", elem_bits=4, group=16, scale="e4m3", code=2)+FP4_E8M0_B32 = RowFormat("fp4_e8m0_b32", elem_bits=4, group=32, scale="e8m0", code=3)++ROW_FORMATS: dict[str, RowFormat] = {f.name: f for f in (BF16, FP8_E8M0_B32, FP4_E4M3_B16, FP4_E8M0_B32)}+++def row_format(name: str) -> RowFormat:+ try:+ return ROW_FORMATS[name]+ except KeyError:+ raise ValueError(f"unknown row format {name!r}; one of {sorted(ROW_FORMATS)}") from None+++__all__ = ["RowFormat", "BF16", "FP8_E8M0_B32", "FP4_E4M3_B16", "FP4_E8M0_B32", "ROW_FORMATS", "row_format"]diff --git a/python/freetoken/kvcache/dsv4_cost_model.py b/python/freetoken/kvcache/dsv4/v4_cost_model.pysimilarity index 96%rename from python/freetoken/kvcache/dsv4_cost_model.pyrename to python/freetoken/kvcache/dsv4/v4_cost_model.pyindex 2c084aff..c6f5b7ce 100644--- a/python/freetoken/kvcache/dsv4_cost_model.py+++ b/python/freetoken/kvcache/dsv4/v4_cost_model.py@@ -28,14 +28,12 @@ _FP32_BYTES = 4def dsv4_reserved_window_pages(max_running_req: int, radix: bool) -> int:- """Window pages the sliding pool must always keep for the concurrent working set: each- running request's decode transients (2 per req + dummy) plus, in radix mode, PER concurrent- request one locked live-tail page AND a retained (soft-pinned) prompt-end window -- the- window is 2 pages here because the retention gap page-aligns to a whole extra page at- P==window==128, so a distinct follow-up per running request can re-lock 2 pages each.- Shared by the engine's window-floor and the manager's prefill_chunk_budget so both- reserve the same set."""- return 2 * (max_running_req + 1) + (3 * max_running_req if radix else 0) + 1+ """Window pages the sliding pool must always keep for the concurrent working set; the+ formula is shared with every sliding-window paged pool (``window_tier.reserved_window_pages``)+ and by the engine's window-floor and the manager's prefill_chunk_budget."""+ from .window_tier import reserved_window_pages++ return reserved_window_pages(max_running_req, radix)def ring_size_for_ratio(ratio: int) -> int:diff --git a/python/freetoken/kvcache/dsv4_paged_pool.py b/python/freetoken/kvcache/dsv4/v4_pool.pysimilarity index 52%rename from python/freetoken/kvcache/dsv4_paged_pool.pyrename to python/freetoken/kvcache/dsv4/v4_pool.pyindex 3ce23eea..d5eae6d0 100644--- a/python/freetoken/kvcache/dsv4_paged_pool.py+++ b/python/freetoken/kvcache/dsv4/v4_pool.py@@ -23,135 +23,22 @@ import torchfrom freetoken.utils import init_logger-from .base import BaseKVCachePool-from .dsv4_cost_model import (+from .v4_cost_model import (DSV4PoolSizes,dsv4_kv_unit_bytes,dsv4_window_unit_bytes,ring_size_for_ratio,)+# The window-tier building blocks and the pool base live in window_tier.py (shared with the DSV41+# pool); the first two are re-exported here for the DSV4 callers and tests that import them from+# this module.+from .window_tier import CompressStateRing, FreeListAllocator, WindowTierPagedPool # noqa: F401logger = init_logger(__name__)-# LIFO free-list allocator for the window tier: one allocated unit spans ``page_unit = P``-# slots, so a unit base is always a multiple of ``P`` (the spec's ``G*P`` page-base-# invariant). The cmp/idx tiers have no allocator -- their rows are ``full_loc // ratio``-# pure arithmetic.-class FreeListAllocator:- def __init__(- self,- capacity: int,- device: torch.device,- page_unit: int = 1,- ) -> None:- assert capacity % page_unit == 0, (- f"capacity {capacity} must be a multiple of page_unit {page_unit}"- )- self._capacity = int(capacity)- self._page_unit = int(page_unit)- self._device = device- # Unit base slots: [0, page_unit, 2*page_unit, ...]. LIFO -> pop the tail.- self._free = self._fresh_free()-- def _fresh_free(self) -> torch.Tensor:- n_units = self._capacity // self._page_unit- return torch.arange(n_units, dtype=torch.int64, device=self._device) * self._page_unit-- def alloc(self, n_units: int) -> torch.Tensor:- """Return ``n_units`` unit base slots (each a multiple of ``page_unit``)."""- if n_units < 0:- raise ValueError(f"n_units must be non-negative, got {n_units}")- if n_units > self._free.numel():- raise RuntimeError(- f"FreeListAllocator out of slots: requested {n_units} units, "- f"have {self._free.numel()} (capacity {self._capacity}, unit {self._page_unit})"- )- if n_units == 0:- return self._free[:0].clone()- taken = self._free[-n_units:].clone()- self._free = self._free[:-n_units]- return taken-- def free(self, units: torch.Tensor) -> None:- """Return previously-allocated unit base slots to the pool (LIFO recycle)."""- if units.numel() == 0:- return- units = units.to(device=self._device, dtype=torch.int64).reshape(-1)- self._free = torch.cat([self._free, units])-- def available(self) -> int:- """Free capacity in slots (units * page_unit)."""- return int(self._free.numel()) * self._page_unit-- @property- def capacity(self) -> int:- return self._capacity-- def reset(self) -> None:- self._free = self._fresh_free()----class CompressStateRing:- """Per-layer fp32 compress-state ring: ``[n_slots + 1, 2*(1+overlap)*head_dim]``.-- Last row (index ``-1``) is a permanent scratch slot, re-cleared on every- write. Last dim is split ``kv | score``; ``set_state`` writes both halves.- """-- def __init__(- self,- n_slots: int,- ring_size: int,- overlap: bool,- head_dim: int,- device: torch.device,- dtype: torch.dtype = torch.float32,- ) -> None:- self.ring_size = ring_size- self.head_dim = head_dim- self._item_size = (1 + int(overlap)) * head_dim # width of kv (== width of score)- last_dim = 2 * self._item_size- # +1 trailing scratch row at index -1.- self.buffer = torch.zeros((n_slots + 1, last_dim), dtype=dtype, device=device)- self.n_slots = n_slots- self._clear_scratch()-- def _clear_scratch(self) -> None:- self.buffer[-1, : self._item_size].zero_()- self.buffer[-1, self._item_size :].fill_(float("-inf"))-- @property- def item_size(self) -> int:- return self._item_size-- def get(self, state_loc: torch.Tensor) -> torch.Tensor:- """Gather ``kv_score`` rows at ``state_loc`` (``-1`` -> scratch row)."""- return self.buffer[state_loc]-- def set(self, state_loc: torch.Tensor, kv_score: torch.Tensor) -> None:- """Scatter ``kv_score`` rows to ``state_loc`` then re-clear the scratch row."""- self.buffer[state_loc] = kv_score- self._clear_scratch()-- def get_blocks(self, page_base: torch.Tensor) -> torch.Tensor:- """Batched per-row carry-block read. ``page_base`` is ``[B]`` (the ring block base- row per row, == ``(window_slot // P) * ring_size``). Returns ``[B, ring_size, 2*item]``- — each row's whole rolling carry block. Distinct pages -> disjoint blocks (isolation)."""- rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)- return self.buffer[rows]-- def set_blocks(self, page_base: torch.Tensor, blocks: torch.Tensor) -> None:- """Batched per-row carry-block write: ``buffer[base+arange(ring_size)] = blocks[row]``.- ``page_base`` ``[B]``, ``blocks`` ``[B, ring_size, 2*item]``. Re-clears scratch."""- rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)- self.buffer[rows] = blocks- self._clear_scratch()---class DSV4PagedKVCache(BaseKVCachePool):+class DSV4PagedKVCache(WindowTierPagedPool):def __init__(self,sizes: DSV4PoolSizes,@@ -273,50 +160,11 @@ class DSV4PagedKVCache(BaseKVCachePool):))- # ----- full-loc translation (gather-only -1 safety; see field comment) ------ def translate_full_to_window(self, full_locs: torch.Tensor) -> torch.Tensor:- # int64 gather indices: the shared page_table stores full locs as int32.- return self.full_to_window[full_locs.to(dtype=torch.int64)]-- @staticmethod- def cmp_rows(full_locs: torch.Tensor, ratio: int) -> torch.Tensor:- return torch.div(full_locs.to(dtype=torch.int64), ratio, rounding_mode="floor")-- def bind_window_pages(self, full_page_base: int, window_page_base: int) -> None:- assert full_page_base % self.P == 0 and window_page_base % self.P == 0- self.full_to_window[full_page_base : full_page_base + self.P] = torch.arange(- window_page_base, window_page_base + self.P, dtype=torch.int64, device=self._device- )-- def unbind_window_pages(self, full_locs: torch.Tensor) -> None:- self.full_to_window[full_locs[full_locs >= 0]] = -1-- # ----- generic swa_pool duck-type (the CacheManager plug-in surface) ------ # ShadowRadix layering: the shared page_table is the virtual full-token coordinate; this pool- # projects it into the physical tiers. The window tier is the managed "second currency" --- # token-face signatures (what the generic CacheManager speaks), PAGE-ATOMIC internals (window- # pages are 1:1 page-bound to full pages; the per-page state ring requires it). alloc_swa- # receives whole ascending pages (allocate_paged's _page_to_token expansion) and free paths- # are page-complete by construction (padded finish tails, align_down frontiers, page-aligned- # tree nodes) -- asserted here, not assumed.- swa_paged = True-- @property- def sliding_window_size(self) -> int:- return self.P-- @property- def prefill_chunk_budget(self) -> int:- return self._chunk_budget-# ----- engine-facing rebuild surface ------ # The tier buffers are bound into per-forward model scratch, invalid after a realloc.- needs_rebind_on_rebuild = True-@classmethoddef kv_cost(cls, config) -> tuple[int, int, int, int]:- from .dsv4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages- from .dsv4_cost_model import dsv4_auto_cost_model+ from .v4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages+ from .v4_cost_model import dsv4_auto_cost_modeldsv4_args = config.model_config.dsv4_argsP = dsv4_args.window_size@@ -335,8 +183,8 @@ class DSV4PagedKVCache(BaseKVCachePool):# graceful config error, not a late OOM.from freetoken.utils import mem_GB- from .dsv4_cost_model import _dsv4_pool_sizes, _dsv4_swa_ratio, _dsv4_window_floor_pages- from .dsv4_cost_model import dsv4_pool_bytes, dsv4_solve_num_pages+ from .v4_cost_model import _dsv4_pool_sizes, _dsv4_swa_ratio, _dsv4_window_floor_pages+ from .v4_cost_model import dsv4_pool_bytes, dsv4_solve_num_pagesdsv4_args = config.model_config.dsv4_argsP = dsv4_args.window_size@@ -369,11 +217,19 @@ class DSV4PagedKVCache(BaseKVCachePool):)return num_pages+ @classmethod+ def window_spec(cls, config):+ from ..base import WindowPoolSpec+ from .v4_cost_model import _dsv4_window_floor_pages++ P = config.model_config.dsv4_args.window_size+ return WindowPoolSpec(P, _dsv4_window_floor_pages(config, P) - 1)+@classmethoddef min_kv_tokens(cls, config) -> int:# The full anchor must cover the window working-set floor (full >= window always), so# that floor -- the value validate_rebuild enforces -- is the pool's floor in tokens.- from .dsv4_cost_model import _dsv4_window_floor_pages+ from .v4_cost_model import _dsv4_window_floor_pagesP = config.model_config.dsv4_args.window_sizereturn _dsv4_window_floor_pages(config, P) * P@@ -387,9 +243,9 @@ class DSV4PagedKVCache(BaseKVCachePool):from freetoken.engine.cache_budget import net_cache_budget_bytesfrom freetoken.utils import mem_GB- from .base import CacheRebuildRejected- from .dsv4_cost_model import _dsv4_pool_sizes, _dsv4_window_floor_pages- from .dsv4_cost_model import dsv4_pool_bytes+ from ..base import CacheRebuildRejected+ from .v4_cost_model import _dsv4_pool_sizes, _dsv4_window_floor_pages+ from .v4_cost_model import dsv4_pool_bytesdsv4_args = config.model_config.dsv4_argsif num_pages is not None:@@ -426,120 +282,19 @@ class DSV4PagedKVCache(BaseKVCachePool):def rebuild_from_config(self, config, num_pages: int, *, num_swa_pages: int | None = None) -> None:- from .dsv4_cost_model import _dsv4_pool_sizes+ from .v4_cost_model import _dsv4_pool_sizes# +1 for the dummy pageself.rebuild(_dsv4_pool_sizes(config, num_pages + 1, num_swa_pages=num_swa_pages))- def attach_page_table(self, page_table: torch.Tensor) -> None:- # The model reads full locs through full_loc_map; under the shared route that IS the- # page table. The attention backend re-allocates its decode snapshot on re-capture.- self.full_loc_map = page_table-def unit_bytes(self) -> tuple[int, int]:# No measurable flat buffer (owned paged pool): the full (cmp/idx + mapping) and window# (sliding KV + state rings) per-token costs come from the per-tier cost model.return dsv4_kv_unit_bytes(self.args, self.P), dsv4_window_unit_bytes(self.args, self.P)- def rebuild(self, sizes) -> None:- """In-place resize to ``sizes`` (identity-preserving; free-before-alloc). The manager's- tree/page bookkeeping reset is the scheduler's generic cache_manager.rebuild; the engine- re-attaches the page table via attach_page_table afterwards."""- import gc-- assert self._paged_params is not None, "rebuild before _init_paged_state"- self.sizes = sizes+ def _drop_buffers(self) -> None:self.window_pool = self.cmp_pool = self.idx_pool = Noneself.state_ring = self.indexer_state_ring = None- self.full_to_window = None- gc.collect()- if torch.cuda.is_available():- torch.cuda.empty_cache()- self._alloc_buffers()- self._init_paged_state(*self._paged_params)-- def _init_paged_state(self, max_running_req: int, radix: bool) -> None:- """Build the pool-owned window free-list + the tail dummy binding. The LAST full page and- LAST window page are the reserved dummy region: page_table's dummy row points at- ``full_token - P`` (== the generic ``fill_(num_tokens)`` convention with num_tokens = the- allocatable token count), permanently bound so graph-padded rows scatter to a real slot."""- from .dsv4_cost_model import dsv4_reserved_window_pages- P = self.P- self._paged_params = (int(max_running_req), bool(radix))- self.full_to_window.fill_(-1)- self._win_alloc = FreeListAllocator(self.sizes.n_win_slots - P, self._device, page_unit=P)- self.bind_window_pages(self.sizes.full_token - P, self.sizes.n_win_slots - P)- # Chunk cap: a batched prefill holds the whole chunk's window live at once (sliding frees- # only between chunks; peak ~2x the chunk), so reserve the concurrent working set and- # halve the rest -- the same formula the bespoke manager used.- n_win_pages = (self.sizes.n_win_slots // P) - 1- reserved = dsv4_reserved_window_pages(max_running_req, radix)- self._chunk_budget = max(P, (n_win_pages - reserved) // 2 * P)-- @property- def swa_num_tokens(self) -> int:- # Allocatable window slots + 1: the generic capacity convention reserves slot 0 as a- # sentinel (cap == swa_num_tokens - 1); DSV4's reserved unit is the tail dummy page,- # already excluded from the free-list, so +1 re-encodes the same cap.- return (self.sizes.n_win_slots - self.P) + 1-- def swa_available_size(self) -> int:- return int(self._win_alloc.available())-- def alloc_swa(self, full_indices: torch.Tensor) -> None:- """Bind one window page per incoming FULL page. ``full_indices`` must be whole ascending- pages (the ``_page_to_token`` expansion); the in-page offsets are preserved- (``window_slot = wbase + pos % P``), which the state ring's page-block layout requires."""- n = int(full_indices.numel())- if n == 0:- return- P = self.P- assert n % P == 0, f"alloc_swa needs whole pages, got {n} slots"- fi = full_indices.to(device=self._device, dtype=torch.int64).view(-1, P)- fbases = fi[:, 0]- assert torch.equal(fi, fbases[:, None] + torch.arange(P, device=self._device)), (- "alloc_swa pages must be contiguous ascending"- )- wbases = self._win_alloc.alloc(fbases.numel()) # raises when exhausted (caller gated)- offsets = torch.arange(P, dtype=torch.int64, device=self._device)- self.full_to_window[(fbases[:, None] + offsets).flatten()] = (- wbases[:, None] + offsets- ).flatten()-- def free_swa(self, full_indices: torch.Tensor) -> None:- """Return the window pages backing these FULL locs and unbind the mapping. Page-atomic:- the incoming locs must cover each touched page completely (guaranteed by the padded- finish tails / aligned frontiers / page-aligned tree values). Idempotent over already- unbound (slid/tombstoned) pages."""- if full_indices.numel() == 0:- return- P = self.P- fi = full_indices.to(device=self._device, dtype=torch.int64)- fi = fi[fi >= 0]- if fi.numel() == 0:- return- fbases, counts = torch.unique(- torch.div(fi, P, rounding_mode="floor") * P, return_counts=True- )- assert bool((counts == P).all()), (- f"free_swa got partial pages (counts {counts[counts != P].tolist()[:4]})"- )- ws = self.full_to_window[fbases]- live = ws[ws >= 0]- offsets = torch.arange(P, dtype=torch.int64, device=self._device)- self.full_to_window[(fbases[:, None] + offsets).flatten()] = -1- if live.numel():- self._win_alloc.free(torch.div(live, P, rounding_mode="floor") * P)-- def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor) -> torch.Tensor:- return self.full_to_window[kv_indices.to(dtype=torch.int64)]-- # ----- state_loc derivation (vectorized, LongTensor in/out) ------ @staticmethod- def state_loc(window_slot: torch.Tensor, ring_size: int, P: int) -> torch.Tensor:- pages = torch.div(window_slot, P, rounding_mode="floor")- loc = pages * ring_size + (window_slot % ring_size)- return torch.where(window_slot < 0, torch.full_like(loc, -1), loc)def ring_size(self, layer_id: int) -> int:return ring_size_for_ratio(self.compress_ratios[layer_id])@@ -586,26 +341,6 @@ class DSV4PagedKVCache(BaseKVCachePool):)return int(n)- # ----- BaseKVCachePool interface ------ def k_cache(self, index: int) -> torch.Tensor:- return self.window_pool[index]-- def v_cache(self, index: int) -> torch.Tensor: # MLA: K == V (single latent)- return self.window_pool[index]-- def store_kv(self, k, v, out_loc, layer_id) -> None:- # Thin window-write shim for ABC compat; DSV4 writes via the specialized- # setters above. K == V (single latent), so the window slot is out_loc.- self.store_window(k, layer_id, out_loc)-- @property- def device(self) -> torch.device:- return self._device-- @property- def dtype(self) -> torch.dtype:- return self._dtype-@propertydef num_layers(self) -> int:return self._n_layersdiff --git a/python/freetoken/kvcache/dsv4/window_tier.py b/python/freetoken/kvcache/dsv4/window_tier.pynew file mode 100644index 00000000..0a25c058--- /dev/null+++ b/python/freetoken/kvcache/dsv4/window_tier.py@@ -0,0 +1,331 @@+"""Shared pieces of the sliding-window paged pools (DSV4, DSV41).++* ``FreeListAllocator`` -- LIFO page allocator for the window tier (unit = one P-slot page).+* ``CompressStateRing`` -- per-window-page fp32 ``kv | score`` carry ring with a scratch row.+* ``reserved_window_pages`` -- the window working set every such pool keeps for its running requests.+* ``window_state_loc`` -- the derived (never stored) ring slot of a window slot.+* ``WindowTierPagedPool`` -- the pool base: the full-loc -> window-slot mapping over the shared page+ table, the page-atomic window allocator behind the CacheManager's ``swa_pool`` duck-type, the+ in-place rebuild, and the ABC glue every such pool shares. Subclasses allocate their own tiers+ (``_alloc_buffers`` / ``_drop_buffers``), size themselves (the ``kv_cost`` family) and write rows.++``DSV4PagedKVCache`` (the origin of this code) and ``DSV41PagedKVCache`` both build on these, so a+fix to allocation, freeing or the mapping lands in one place.+"""++from __future__ import annotations++import gc+from abc import abstractmethod++import torch++from ..base import BaseKVCachePool+++class FreeListAllocator:+ """LIFO free list over ``capacity // page_unit`` units; a unit's base slot is a multiple of+ ``page_unit`` (the window tier's ``G*P`` page-base invariant)."""++ def __init__(self, capacity: int, device: torch.device, page_unit: int = 1) -> None:+ assert capacity % page_unit == 0, f"capacity {capacity} must be a multiple of page_unit {page_unit}"+ self._capacity = int(capacity)+ self._page_unit = int(page_unit)+ self._device = device+ self._free = self._fresh_free()++ def _fresh_free(self) -> torch.Tensor:+ n_units = self._capacity // self._page_unit+ return torch.arange(n_units, dtype=torch.int64, device=self._device) * self._page_unit++ def alloc(self, n_units: int) -> torch.Tensor:+ """Return ``n_units`` unit base slots (each a multiple of ``page_unit``)."""+ if n_units < 0:+ raise ValueError(f"n_units must be non-negative, got {n_units}")+ if n_units > self._free.numel():+ raise RuntimeError(+ f"FreeListAllocator out of slots: requested {n_units} units, "+ f"have {self._free.numel()} (capacity {self._capacity}, unit {self._page_unit})"+ )+ if n_units == 0:+ return self._free[:0].clone()+ taken = self._free[-n_units:].clone()+ self._free = self._free[:-n_units]+ return taken++ def free(self, units: torch.Tensor) -> None:+ """Return previously-allocated unit base slots to the pool (LIFO recycle)."""+ if units.numel() == 0:+ return+ units = units.to(device=self._device, dtype=torch.int64).reshape(-1)+ self._free = torch.cat([self._free, units])++ def available(self) -> int:+ """Free capacity in slots (units * page_unit)."""+ return int(self._free.numel()) * self._page_unit++ @property+ def capacity(self) -> int:+ return self._capacity++ def reset(self) -> None:+ self._free = self._fresh_free()+++class CompressStateRing:+ """Per-layer fp32 compress-state ring: ``[n_slots + 1, 2*(1+overlap)*head_dim]``.++ Last row (index ``-1``) is a permanent scratch slot, re-cleared on every write. Last dim is+ split ``kv | score``; ``set`` writes both halves.+ """++ def __init__(+ self,+ n_slots: int,+ ring_size: int,+ overlap: bool,+ head_dim: int,+ device: torch.device,+ dtype: torch.dtype = torch.float32,+ ) -> None:+ self.ring_size = ring_size+ self.head_dim = head_dim+ self._item_size = (1 + int(overlap)) * head_dim # width of kv (== width of score)+ last_dim = 2 * self._item_size+ self.buffer = torch.zeros((n_slots + 1, last_dim), dtype=dtype, device=device)+ self.n_slots = n_slots+ self._clear_scratch()++ def _clear_scratch(self) -> None:+ self.buffer[-1, : self._item_size].zero_()+ self.buffer[-1, self._item_size :].fill_(float("-inf"))++ @property+ def item_size(self) -> int:+ return self._item_size++ def get(self, state_loc: torch.Tensor) -> torch.Tensor:+ """Gather ``kv_score`` rows at ``state_loc`` (``-1`` -> scratch row)."""+ return self.buffer[state_loc]++ def set(self, state_loc: torch.Tensor, kv_score: torch.Tensor) -> None:+ """Scatter ``kv_score`` rows to ``state_loc`` then re-clear the scratch row."""+ self.buffer[state_loc] = kv_score+ self._clear_scratch()++ def get_blocks(self, page_base: torch.Tensor) -> torch.Tensor:+ """Batched per-row carry-block read: ``page_base`` ``[B]`` (``(window_slot // P) * ring_size``)+ -> ``[B, ring_size, 2*item]``. Distinct pages -> disjoint blocks."""+ rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)+ return self.buffer[rows]++ def set_blocks(self, page_base: torch.Tensor, blocks: torch.Tensor) -> None:+ """Batched per-row carry-block write of ``[B, ring_size, 2*item]``; re-clears scratch."""+ rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)+ self.buffer[rows] = blocks+ self._clear_scratch()+++def reserved_window_pages(max_running_req: int, radix: bool) -> int:+ """Window pages the sliding pool must always keep for the concurrent working set: each+ running request's decode transients (2 per req + dummy) plus, in radix mode, PER concurrent+ request one locked live-tail page AND a retained (soft-pinned) prompt-end window -- the+ window is 2 pages here because the retention gap page-aligns to a whole extra page at+ P == window."""+ return 2 * (max_running_req + 1) + (3 * max_running_req if radix else 0) + 1+++def window_state_loc(window_slot: torch.Tensor, ring_size: int, P: int) -> torch.Tensor:+ """Ring slot of a window slot: ``(ws // P) * ring_size + ws % ring_size``; ``-1`` stays ``-1``+ (the scratch row). ``ring_size | P`` keeps distinct pages on disjoint blocks."""+ pages = torch.div(window_slot, P, rounding_mode="floor")+ loc = pages * ring_size + (window_slot % ring_size)+ return torch.where(window_slot < 0, torch.full_like(loc, -1), loc)+++class WindowTierPagedPool(BaseKVCachePool):+ """A paged KV pool whose window tier is page-bound to the shared full-token page table.++ Subclasses set ``P`` (the window page), ``_device``, ``_dtype`` and ``sizes`` (with+ ``full_token`` and ``n_win_slots``) before calling ``_alloc_buffers``, which must create+ ``self.full_to_window`` (``[full_token + 1]`` int64, ``-1`` = unmapped, the trailing row a+ permanent ``-1`` sentinel so a gather at ``-1`` is safe) and ``self.window_pool`` (one tensor+ per layer, indexed by window slot). Scatter paths must never see a negative slot.++ ShadowRadix layering: the shared page table is the virtual full-token coordinate; the pool+ projects it into physical tiers. The window tier is the managed "second currency" -- token-face+ signatures (what the generic CacheManager speaks), PAGE-ATOMIC internals (window pages are 1:1+ page-bound to full pages; the per-page state ring requires it). ``alloc_swa`` receives whole+ ascending pages and free paths are page-complete by construction -- asserted here, not assumed.+ """++ swa_paged = True+ # the tier buffers are bound into per-forward model scratch, invalid after a realloc+ needs_rebind_on_rebuild = True++ P: int+ _device: torch.device+ _dtype: torch.dtype+ sizes: object+ full_to_window: torch.Tensor+ window_pool: list++ # ----- tiers (subclass) -----+ @abstractmethod+ def _alloc_buffers(self) -> None: ...++ @abstractmethod+ def _drop_buffers(self) -> None:+ """Release every tier buffer before a rebuild reallocates them."""++ # ----- full-loc translation (gather-only -1 safety; see the class docstring) -----+ def translate_full_to_window(self, full_locs: torch.Tensor) -> torch.Tensor:+ # int64 gather indices: the shared page_table stores full locs as int32+ return self.full_to_window[full_locs.to(dtype=torch.int64)]++ @staticmethod+ def cmp_rows(full_locs: torch.Tensor, ratio: int) -> torch.Tensor:+ """The compressed row of a full loc: ``full_loc // ratio`` (negative stays negative)."""+ return torch.div(full_locs.to(dtype=torch.int64), ratio, rounding_mode="floor")++ def bind_window_pages(self, full_page_base: int, window_page_base: int) -> None:+ assert full_page_base % self.P == 0 and window_page_base % self.P == 0+ self.full_to_window[full_page_base : full_page_base + self.P] = torch.arange(+ window_page_base, window_page_base + self.P, dtype=torch.int64, device=self._device+ )++ def unbind_window_pages(self, full_locs: torch.Tensor) -> None:+ self.full_to_window[full_locs[full_locs >= 0]] = -1++ @staticmethod+ def state_loc(window_slot: torch.Tensor, ring_size: int, P: int) -> torch.Tensor:+ return window_state_loc(window_slot, ring_size, P)++ # ----- generic swa_pool duck-type (the CacheManager plug-in surface) -----+ @property+ def sliding_window_size(self) -> int:+ return self.P++ @property+ def prefill_chunk_budget(self) -> int:+ return self._chunk_budget++ def _init_paged_state(self, max_running_req: int, radix: bool) -> None:+ """Build the pool-owned window free list + the tail dummy binding. The LAST full page and+ LAST window page are the reserved dummy region: page_table's dummy row points at+ ``full_token - P`` (the generic ``fill_(num_tokens)`` convention), permanently bound so+ graph-padded rows scatter to a real slot."""+ P = self.P+ self._paged_params = (int(max_running_req), bool(radix))+ self.full_to_window.fill_(-1)+ self._win_alloc = FreeListAllocator(self.sizes.n_win_slots - P, self._device, page_unit=P)+ self.bind_window_pages(self.sizes.full_token - P, self.sizes.n_win_slots - P)+ # Chunk cap: a batched prefill holds the whole chunk's window live at once (sliding frees+ # only between chunks; peak ~2x the chunk), so reserve the concurrent working set and+ # halve the rest.+ n_win_pages = (self.sizes.n_win_slots // P) - 1+ reserved = reserved_window_pages(max_running_req, radix)+ self._chunk_budget = max(P, (n_win_pages - reserved) // 2 * P)++ @property+ def window_pages(self) -> int:+ return self.sizes.n_win_pages - 1++ @property+ def swa_num_tokens(self) -> int:+ # Allocatable window slots + 1: the generic capacity convention reserves slot 0 as a+ # sentinel (cap == swa_num_tokens - 1); here the reserved unit is the tail dummy page,+ # already excluded from the free list, so +1 re-encodes the same cap.+ return (self.sizes.n_win_slots - self.P) + 1++ def swa_available_size(self) -> int:+ return int(self._win_alloc.available())++ def alloc_swa(self, full_indices: torch.Tensor) -> None:+ """Bind one window page per incoming FULL page. ``full_indices`` must be whole ascending+ pages (the ``_page_to_token`` expansion); the in-page offsets are preserved+ (``window_slot = wbase + pos % P``), which the state ring's page-block layout requires."""+ n = int(full_indices.numel())+ if n == 0:+ return+ P = self.P+ assert n % P == 0, f"alloc_swa needs whole pages, got {n} slots"+ fi = full_indices.to(device=self._device, dtype=torch.int64).view(-1, P)+ fbases = fi[:, 0]+ assert torch.equal(fi, fbases[:, None] + torch.arange(P, device=self._device)), "alloc_swa pages must be contiguous ascending"+ wbases = self._win_alloc.alloc(fbases.numel()) # raises when exhausted (caller gated)+ offsets = torch.arange(P, dtype=torch.int64, device=self._device)+ self.full_to_window[(fbases[:, None] + offsets).flatten()] = (wbases[:, None] + offsets).flatten()++ def free_swa(self, full_indices: torch.Tensor) -> None:+ """Return the window pages backing these FULL locs and unbind the mapping. Page-atomic: the+ incoming locs must cover each touched page completely (guaranteed by the padded finish+ tails / aligned frontiers / page-aligned tree values). Idempotent over already unbound+ (slid / tombstoned) pages."""+ if full_indices.numel() == 0:+ return+ P = self.P+ fi = full_indices.to(device=self._device, dtype=torch.int64)+ fi = fi[fi >= 0]+ if fi.numel() == 0:+ return+ fbases, counts = torch.unique(torch.div(fi, P, rounding_mode="floor") * P, return_counts=True)+ assert bool((counts == P).all()), f"free_swa got partial pages (counts {counts[counts != P].tolist()[:4]})"+ ws = self.full_to_window[fbases]+ live = ws[ws >= 0]+ offsets = torch.arange(P, dtype=torch.int64, device=self._device)+ self.full_to_window[(fbases[:, None] + offsets).flatten()] = -1+ if live.numel():+ self._win_alloc.free(torch.div(live, P, rounding_mode="floor") * P)++ def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor) -> torch.Tensor:+ return self.full_to_window[kv_indices.to(dtype=torch.int64)]++ # ----- rebuild / page table -----+ def rebuild(self, sizes) -> None:+ """In-place resize to ``sizes`` (identity-preserving; free-before-alloc). The manager's+ tree / page bookkeeping reset is the scheduler's generic cache_manager.rebuild; the engine+ re-attaches the page table via attach_page_table afterwards."""+ assert self._paged_params is not None, "rebuild before _init_paged_state"+ self.sizes = sizes+ self._drop_buffers()+ self.full_to_window = None # type: ignore[assignment]+ gc.collect()+ if torch.cuda.is_available():+ torch.cuda.empty_cache()+ self._alloc_buffers()+ self._init_paged_state(*self._paged_params)++ def attach_page_table(self, page_table: torch.Tensor) -> None:+ # the model reads full locs through full_loc_map; under the shared route that IS the page table+ self.full_loc_map = page_table++ # ----- BaseKVCachePool glue: the window tier is the K == V latent cache -----+ def k_cache(self, index: int) -> torch.Tensor:+ return self.window_pool[index]++ def v_cache(self, index: int) -> torch.Tensor:+ return self.window_pool[index]++ def store_kv(self, k, v, out_loc, layer_id) -> None:+ self.store_window(k, layer_id, out_loc)++ @abstractmethod+ def store_window(self, kv: torch.Tensor, layer_id: int, window_slot: torch.Tensor) -> None: ...++ @property+ def device(self) -> torch.device:+ return self._device++ @property+ def dtype(self) -> torch.dtype:+ return self._dtype+++__all__ = [+ "FreeListAllocator",+ "CompressStateRing",+ "WindowTierPagedPool",+ "reserved_window_pages",+ "window_state_loc",+]diff --git a/python/freetoken/kvcache/hybrid_swa_pool.py b/python/freetoken/kvcache/hybrid_swa_pool.pyindex 41c3880e..022eb499 100644--- a/python/freetoken/kvcache/hybrid_swa_pool.py+++ b/python/freetoken/kvcache/hybrid_swa_pool.py@@ -8,7 +8,7 @@ from freetoken.distributed import get_tp_infofrom freetoken.models.config import KVCacheGroupSpecfrom freetoken.utils import align_ceil, div_even-from .base import BaseKVCachePool+from .base import BaseKVCachePool, WindowPoolSpec@dataclass(frozen=True)@@ -295,6 +295,16 @@ class HybridSWAKVCache(BaseKVCachePool):# by CacheManager.rebuild is then consistent by construction (idle-only).self._init_swa_paged_state()+ @classmethod+ def window_spec(cls, config) -> WindowPoolSpec | None:+ if config.cache_type != "swa_radix":+ return None+ return WindowPoolSpec(1, _swa_pool_floor(config))++ @property+ def window_pages(self) -> int:+ return self.swa_num_tokens - 1+@classmethoddef kv_cost(cls, config, *, num_swa_pages: int | None = None) -> tuple[int, int, int, int]:"""Full groups ride cache_per_page; the window pool is either a pinned absolutediff --git a/python/freetoken/layers/mhc.py b/python/freetoken/layers/mhc.pyindex 1cd6459d..dcf49151 100644--- a/python/freetoken/layers/mhc.py+++ b/python/freetoken/layers/mhc.py@@ -19,6 +19,16 @@ PR #53906) bit-for-bit in fp32; the per-layer weights are ``hc_{attn,ffn}_fn`` /correctness baseline; the fused triton kernel (``kernel/triton/mhc.py``,dispatched by ``mhc_fused_post_pre`` below) replaces it on CUDA and isvalidated against this file.++Two variants share the gate math:++* **multi-pass** (GLM-5.3, DeepSeek-V4): ``pre`` is computed from the residual the+ sublayer is about to read and applied immediately -- ``mhc_pre`` / ``mhc_fused_post_pre``.+* **single-pass** (DeepSeek-V4.1 "Single-Pass mHC", tech report sec. 2.4.1): every+ sublayer consumes the ``pre`` gates the *previous* sublayer produced, so the input+ mix no longer waits on this sublayer's own reduction -- ``mhc_pre_single_pass`` /+ ``mhc_fused_post_pre_single_pass``. The returned ``pre_new`` is what the next+ sublayer applies; the model seeds the chain with a one-hot pre on stream 0."""from __future__ import annotations@@ -36,8 +46,8 @@ def hc_contract(x: torch.Tensor) -> torch.Tensor:return x.mean(dim=1)-def mhc_pre(- residual: torch.Tensor, # [T, n, hidden] bf16+def mhc_gates(+ residual: torch.Tensor, # [T, n, hidden]fn: torch.Tensor, # [2n + n^2, n*hidden] fp32hc_scale: torch.Tensor, # [3] fp32hc_base: torch.Tensor, # [2n + n^2] fp32@@ -46,8 +56,8 @@ def mhc_pre(post_mult: float,sinkhorn_repeat: int,) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:- """Returns (post_mix [T, n, 1] fp32, comb_mix [T, n, n] fp32,- layer_input [T, hidden] bf16)."""+ """The coefficient predictor H(X): returns (pre_mix [T, n] fp32, post_mix [T, n, 1]+ fp32, comb_mix [T, n, n] fp32) from one fp32 GEMM over the flattened streams."""n, hidden = residual.shape[-2], residual.shape[-1]t = residual.shape[0]@@ -68,11 +78,46 @@ def mhc_pre(for _ in range(sinkhorn_repeat - 1):comb = comb / (comb.sum(dim=-1, keepdim=True) + hc_eps)comb = comb / (comb.sum(dim=-2, keepdim=True) + hc_eps)+ return pre_mix, post_mix.view(t, n, 1), comb- layer_input = (- (pre_mix.unsqueeze(-1) * residual.to(torch.float32)).sum(dim=1).to(residual.dtype)- )- return post_mix.view(t, n, 1), comb, layer_input++def mhc_mix_input(residual: torch.Tensor, pre_mix: torch.Tensor) -> torch.Tensor:+ """The sublayer input ``sum_i pre_i * res_i``: [T, n, hidden] x [T, n] -> [T, hidden]."""+ return (pre_mix.unsqueeze(-1) * residual.to(torch.float32)).sum(dim=1).to(residual.dtype)+++def mhc_pre(+ residual: torch.Tensor, # [T, n, hidden] bf16+ fn: torch.Tensor, # [2n + n^2, n*hidden] fp32+ hc_scale: torch.Tensor, # [3] fp32+ hc_base: torch.Tensor, # [2n + n^2] fp32+ rms_eps: float,+ hc_eps: float,+ post_mult: float,+ sinkhorn_repeat: int,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """Multi-pass: gates from ``residual``, pre applied to it right away. Returns+ (post_mix [T, n, 1] fp32, comb_mix [T, n, n] fp32, layer_input [T, hidden] bf16)."""+ pre_mix, post_mix, comb = mhc_gates(residual, fn, hc_scale, hc_base, rms_eps, hc_eps, post_mult, sinkhorn_repeat)+ return post_mix, comb, mhc_mix_input(residual, pre_mix)+++def mhc_pre_single_pass(+ residual: torch.Tensor, # [T, n, hidden]+ pre_mix_in: torch.Tensor, # [T, n] fp32 -- the previous sublayer's pre gates+ fn: torch.Tensor,+ hc_scale: torch.Tensor,+ hc_base: torch.Tensor,+ rms_eps: float,+ hc_eps: float,+ post_mult: float,+ sinkhorn_repeat: int,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:+ """Single-pass: gates from ``residual`` for THIS sublayer's post/comb and the NEXT+ sublayer's pre; the input is mixed with ``pre_mix_in``. Returns (pre_new [T, n] fp32,+ post_mix [T, n, 1] fp32, comb_mix [T, n, n] fp32, layer_input [T, hidden])."""+ pre_new, post_mix, comb = mhc_gates(residual, fn, hc_scale, hc_base, rms_eps, hc_eps, post_mult, sinkhorn_repeat)+ return pre_new, post_mix, comb, mhc_mix_input(residual, pre_mix_in.to(torch.float32))def mhc_post(@@ -140,11 +185,69 @@ def mhc_fused_post_pre()+def mhc_fused_post_pre_single_pass_torch(+ x: torch.Tensor,+ residual: torch.Tensor,+ post_mix: torch.Tensor | None,+ comb_mix: torch.Tensor | None,+ pre_mix_in: torch.Tensor,+ fn: torch.Tensor,+ hc_scale: torch.Tensor,+ hc_base: torch.Tensor,+ rms_eps: float,+ hc_eps: float,+ post_mult: float,+ sinkhorn_repeat: int,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:+ """Decomposed single-pass reference: the previous sublayer's hc_post (skipped when+ ``post_mix is None`` -- the model's first sublayer), then this sublayer's gates with+ the input mixed by ``pre_mix_in``. Returns (residual_new, post_new [T, n, 1],+ comb_new [T, n, n], pre_new [T, n], layer_input)."""+ residual_new = residual if post_mix is None else mhc_post(x, residual, post_mix, comb_mix)+ pre_new, post_new, comb_new, layer_input = mhc_pre_single_pass(+ residual_new, pre_mix_in, fn, hc_scale, hc_base, rms_eps, hc_eps, post_mult, sinkhorn_repeat+ )+ return residual_new, post_new, comb_new, pre_new, layer_input+++def mhc_fused_post_pre_single_pass(+ x: torch.Tensor,+ residual: torch.Tensor,+ post_mix: torch.Tensor | None,+ comb_mix: torch.Tensor | None,+ pre_mix_in: torch.Tensor,+ fn: torch.Tensor,+ hc_scale: torch.Tensor,+ hc_base: torch.Tensor,+ rms_eps: float,+ hc_eps: float,+ post_mult: float,+ sinkhorn_repeat: int,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:+ """Single-pass sublayer boundary (DeepSeek-V4.1): fused triton on CUDA, torch elsewhere."""+ if residual.is_cuda:+ from freetoken.kernel.triton.mhc import mhc_fused_post_pre_single_pass_triton++ return mhc_fused_post_pre_single_pass_triton(+ x, residual, post_mix, comb_mix, pre_mix_in, fn, hc_scale, hc_base,+ rms_eps, hc_eps, post_mult, sinkhorn_repeat,+ )+ return mhc_fused_post_pre_single_pass_torch(+ x, residual, post_mix, comb_mix, pre_mix_in, fn, hc_scale, hc_base,+ rms_eps, hc_eps, post_mult, sinkhorn_repeat,+ )++__all__ = ["hc_expand","hc_contract",+ "mhc_gates",+ "mhc_mix_input","mhc_pre",+ "mhc_pre_single_pass","mhc_post","mhc_fused_post_pre","mhc_fused_post_pre_torch",+ "mhc_fused_post_pre_single_pass",+ "mhc_fused_post_pre_single_pass_torch",]diff --git a/python/freetoken/layers/moe.py b/python/freetoken/layers/moe.pyindex 996beaa7..4918a032 100644--- a/python/freetoken/layers/moe.py+++ b/python/freetoken/layers/moe.py@@ -308,18 +308,21 @@ class OffloadMoELayer(MoELayer):The CPU pool is kicked off (``decode_submit``) before the GPU PCIe fetch + GEMM sothe CPU overflow GEMV runs concurrently with the GPU work. Capture-safe: therouting split is device-side elementwise and the CPU submit/sync are host nodes.- Each route is computed exactly once -- the GPU weights are zeroed for CPU-assigned- routes and the CPU ids are -1 for GPU-assigned routes (the C++ kernel skips id<0).+ Each route is computed exactly once -- CPU-assigned routes reach the GPU as slot -1 with+ weight 0 (a kernel with ``supports_inactive_slots`` skips them without a read; any other reads+ slot 0, whose storage is a payload or the bank's zero fill) and the CPU ids are -1 for+ GPU-assigned routes (the C++ kernel skips id<0)."""executor = cache.cpu_executorassert executor is not None, "CPU MoE executor was not initialized"- raw = topk_ids.clone() # raw expert ids for the CPU partial- cache.ensure_experts_hybrid(self.layer_id, topk_ids) # -> slot (hit/fetched) or -1+ raw_ids = topk_ids.clone()+ cache.ensure_experts_hybrid(self.layer_id, topk_ids)+ on_gpu = topk_ids >= 0+ cpu_ids = torch.where(on_gpu, -1, raw_ids).contiguous()+ gpu_w = torch.where(on_gpu, topk_weights, 0.0).contiguous()+ gpu_slots = topk_idsif cache.collect_stats:cache.record_decode_stats_hybrid(self.layer_id)- on_gpu = topk_ids >= 0-- cpu_ids = torch.where(on_gpu, raw.new_full((), -1), raw).contiguous()pending = executor.decode_submit(self.layer_id, hidden_states, topk_weights, cpu_ids)# Measurement knob: FREETOKEN_HYBRID_OVERLAP=0 syncs the CPU pool *before* the@@ -329,8 +332,10 @@ class OffloadMoELayer(MoELayer):)cache.copy_missing()- gpu_slots = topk_ids.clamp_min(0) # -1 -> slot 0 (zero-weighted below)- gpu_w = torch.where(on_gpu, topk_weights, topk_weights.new_zeros(())).contiguous()+ if not (self.quant_method is not None and self.quant_method.kernel.supports_inactive_slots):+ # the kernel reads every route's slot: give the inactive ones slot 0, which holds a+ # complete payload or the bank's zero fill (never uninitialized storage), times weight 0+ gpu_slots = gpu_slots.clamp_min(0)gpu_routed = self._expert_gemm(cache,hidden_states,diff --git a/python/freetoken/layers/quantization/configs/base.py b/python/freetoken/layers/quantization/configs/base.pyindex e84dcbcb..a6413610 100644--- a/python/freetoken/layers/quantization/configs/base.py+++ b/python/freetoken/layers/quantization/configs/base.py@@ -89,9 +89,13 @@ class QuantConfig(ABC):layer_kind = layer.quant_layer_kindkind = scheme.kind if scheme else QuantKind.NONEcls = method_class(kind, layer_kind)- cfg = LAYER_CONFIGS[layer_kind].from_layer(layer, scheme)+ cfg = self.layer_config(layer, layer_kind, scheme)return cls(cfg, get_quant_backend().select(layer_kind, kind))+ def layer_config(self, layer: Any, layer_kind: LayerKind, scheme: QuantScheme | None) -> LinearConfig | MoEConfig:+ """The layer's Linear / MoE config; a dialect overrides it to add what only its checkpoint knows."""+ return LAYER_CONFIGS[layer_kind].from_layer(layer, scheme)+@staticmethoddef from_hf(hf_config: Any,diff --git a/python/freetoken/layers/quantization/configs/fp8.py b/python/freetoken/layers/quantization/configs/fp8.pyindex c23cbfde..3e74f0c4 100644--- a/python/freetoken/layers/quantization/configs/fp8.py+++ b/python/freetoken/layers/quantization/configs/fp8.py@@ -1,20 +1,25 @@from __future__ import annotations+import dataclassesfrom typing import Any, ClassVar+from ..linear import LinearConfig+from ..moe import MoEConfigfrom ..names import is_routed_expert, name_set, substr_set-from ..registry import register_dialect+from ..registry import LayerKind, register_dialectfrom ..scheme import QuantKind, QuantScheme-from ..scheme import FP8_BLOCK, fp8_block_scheme, fp8_tensor_scheme, mxfp4_scheme+from ..scheme import FP8_BLOCK_SIZES, fp8_block_scheme, fp8_tensor_scheme, mxfp4_schemefrom .base import QuantConfig, Stored, cfg_get@register_dialectclass Fp8BlockConfig(QuantConfig):- """HF ``quant_method: fp8`` (DeepSeek-V3 style 128x128 block scales) plus the DeepSeek-V4 e8m0 / fp4-expert variant."""+ """HF ``quant_method: fp8`` (DeepSeek-V3 style 128x128 block scales) plus the DeepSeek-V4 e8m0 / fp4-expert+ variant; DeepSeek-V4.1 keeps that dialect with 32x32 blocks (``weight_block_size: [32, 32]``)."""dialect = "fp8"+ # the 128-block schemes every dialect table names; ``block_scheme`` carries the checkpoint's actual block edgeSCHEMES: ClassVar[dict[str, QuantScheme]] = {"BLOCK": fp8_block_scheme("float"),"BLOCK_E8M0": fp8_block_scheme("e8m0"),@@ -32,15 +37,23 @@ class Fp8BlockConfig(QuantConfig):def __init__(self, q: dict[str, Any], hf_config: Any = None, *, name_map=None, unquantized=()):super().__init__(name_map, unquantized)block = tuple(int(x) for x in (q.get("weight_block_size") or ()))- if q.get("weight_per_tensor") or block != (FP8_BLOCK, FP8_BLOCK):- raise NotImplementedError(f"fp8 checkpoint with weight_block_size={block} per_tensor={q.get('weight_per_tensor')} is not supported; only 128x128 blocks are")+ square = len(block) == 2 and block[0] == block[1] and block[0] in FP8_BLOCK_SIZES+ if q.get("weight_per_tensor") or not square:+ raise NotImplementedError(+ f"fp8 checkpoint with weight_block_size={block} per_tensor={q.get('weight_per_tensor')} is not supported; "+ f"only square blocks of {FP8_BLOCK_SIZES} are"+ )+ self.block = block[0]# transformers skips lm_head when the checkpoint gives no listnot_convert = tuple(q.get("modules_to_not_convert") or ("lm_head",))self.not_convert = name_set(not_convert)self.not_convert_substr = substr_set(not_convert)self.convert_tables = name_set(tuple(q.get("modules_to_convert") or ()))self.e8m0 = str(q.get("scale_fmt") or "").lower() == "ue8m0"- self.expert_fp4 = str(cfg_get(hf_config, "expert_dtype") or "").lower() == "fp4"+ # ``expert_dtype`` sits at the config top level (V4) or inside quantization_config (V4.1)+ expert_dtype = q.get("expert_dtype") or cfg_get(hf_config, "expert_dtype")+ self.expert_fp4 = str(expert_dtype or "").lower() == "fp4"+ self.block_scheme = fp8_block_scheme("e8m0" if self.e8m0 else "float", self.block)def storage(self, scheme: QuantScheme) -> dict[str, Stored]:names = super().storage(scheme)@@ -55,4 +68,11 @@ class Fp8BlockConfig(QuantConfig):return Noneif self.expert_fp4 and is_routed_expert(name):return self.SCHEMES["EXPERT_MXFP4"]- return self.SCHEMES["BLOCK_E8M0" if self.e8m0 else "BLOCK"]+ return self.block_scheme++ def layer_config(self, layer: Any, layer_kind: LayerKind, scheme: QuantScheme | None) -> LinearConfig | MoEConfig:+ cfg = super().layer_config(layer, layer_kind, scheme)+ if layer_kind is LayerKind.MOE and scheme is not None and scheme.kind is QuantKind.MXFP4:+ # the fp4 experts quantize their activations at the same block as the dense linears+ cfg = dataclasses.replace(cfg, act_block=self.block)+ return cfgdiff --git a/python/freetoken/layers/quantization/linear/fp8_block.py b/python/freetoken/layers/quantization/linear/fp8_block.pyindex 3b78e977..3eafdfdf 100644--- a/python/freetoken/layers/quantization/linear/fp8_block.py+++ b/python/freetoken/layers/quantization/linear/fp8_block.py@@ -1,4 +1,4 @@-"""fp8 e4m3 weight with 128x128 block scales (DeepSeek-V3 style)."""+"""fp8 e4m3 weight with square block scales: 128x128 (DeepSeek-V3 style) or 32x32 (DeepSeek-V4.1)."""from __future__ import annotations@@ -7,7 +7,7 @@ from typing import Anyimport torchfrom ..registry import LayerKind, register_method-from ..scheme import FP8_BLOCK as BLOCK, QuantKind+from ..scheme import FP8_BLOCK, QuantKind, fp8_block_sizefrom .base import LinearConfig, LinearKernel, LinearMethodFP8 = torch.float8_e4m3fn@@ -18,6 +18,11 @@ def _e8m0(cfg: LinearConfig) -> bool:return cfg.scheme is not None and cfg.scheme.weight.scale == "e8m0"+def layer_block_size(layer: Any) -> int:+ """The block edge a finalized layer was declared with: ``K / (scale columns)``, exact by construction."""+ return layer.weight.shape[1] // layer.weight_scale_inv.shape[1]++class Dsv4Fp8BlockLinearKernel(LinearKernel):"""DeepSeek-V4's reference path: activations quantized to fp8 with power-of-two block scales, e8m0 weight scales read as codes."""@@ -29,7 +34,7 @@ class Dsv4Fp8BlockLinearKernel(LinearKernel):def apply(self, layer: Any, x: torch.Tensor) -> torch.Tensor:from freetoken.kernel.triton.dsv4.fp8_linear import block_fp8_linear- return block_fp8_linear(x, layer.weight, layer.weight_scale_inv, layer.bias)+ return block_fp8_linear(x, layer.weight, layer.weight_scale_inv, layer.bias, block=layer_block_size(layer))class TritonFp8BlockLinearKernel(LinearKernel):@@ -38,7 +43,11 @@ class TritonFp8BlockLinearKernel(LinearKernel):name = "triton"def unusable_reason(self, cfg: LinearConfig) -> str | None:- return "reads float block scales; e8m0 codes go to the dsv4 kernel" if _e8m0(cfg) else None+ if _e8m0(cfg):+ return "reads float block scales; e8m0 codes go to the dsv4 kernel"+ if fp8_block_size(cfg.scheme) != FP8_BLOCK:+ return f"float-scale kernel serves {FP8_BLOCK}x{FP8_BLOCK} blocks only"+ return Nonedef apply(self, layer: Any, x: torch.Tensor) -> torch.Tensor:from freetoken.kernel.triton.fp8_block_linear import block_fp8_linear@@ -52,9 +61,10 @@ class Fp8BlockLinearMethod(LinearMethod):def create_weights(self, layer: Any) -> None:g = self.cfg- if g.in_features % BLOCK or any(o % BLOCK for o in g.output_sizes):- raise ValueError(f"block-fp8 needs in/out sizes divisible by {BLOCK}, got K={g.in_features} N={g.output_sizes}")+ block = fp8_block_size(g.scheme)+ if g.in_features % block or any(o % block for o in g.output_sizes):+ raise ValueError(f"block-fp8 needs in/out sizes divisible by {block}, got K={g.in_features} N={g.output_sizes}")layer.weight = torch.empty(g.out_features, g.in_features, dtype=FP8)# e8m0 codes stay codes for the dsv4 kernel; float scales are bf16 as the readers push them todayscale_dtype = E8M0 if _e8m0(g) else torch.bfloat16- layer.weight_scale_inv = torch.empty(g.out_features // BLOCK, g.in_features // BLOCK, dtype=scale_dtype)+ layer.weight_scale_inv = torch.empty(g.out_features // block, g.in_features // block, dtype=scale_dtype)diff --git a/python/freetoken/layers/quantization/moe/base.py b/python/freetoken/layers/quantization/moe/base.pyindex e4a428d3..de1ce004 100644--- a/python/freetoken/layers/quantization/moe/base.py+++ b/python/freetoken/layers/quantization/moe/base.py@@ -28,6 +28,8 @@ class MoEConfig:beta: float = 0.0limit: float | None = Noneinterleaved: bool = False+ # dynamic fp8 activation quant block of a W4A8 expert kernel, set by the quant dialect; None = no activation quant+ act_block: int | None = Nonehas_bias: bool = Falseapply_router_weight_on_input: bool = Falsestrategy: str = "resident"@@ -138,6 +140,10 @@ class MoEKernel(ABC):name: ClassVar[str]cpu_format: ClassVar[str | None] = Nonemax_slots: ClassVar[int | None] = None+ # Whether ``apply`` treats a slot of -1 as an inactive route (zero output, no read of any slot).+ # Hybrid decode hands the GPU a route split with such holes; kernels without this get the holes+ # pointed at slot 0 with weight 0 (slot storage is zero-filled at allocation, so that is exact).+ supports_inactive_slots: ClassVar[bool] = Falsedef unusable_reason(self, cfg: MoEConfig) -> str | None:return Nonediff --git a/python/freetoken/layers/quantization/moe/fp8_block.py b/python/freetoken/layers/quantization/moe/fp8_block.pyindex 20c3dde9..7552978d 100644--- a/python/freetoken/layers/quantization/moe/fp8_block.py+++ b/python/freetoken/layers/quantization/moe/fp8_block.py@@ -5,7 +5,7 @@ from __future__ import annotationsimport torchfrom ..registry import LayerKind, register_method-from ..scheme import FP8_BLOCK as BLOCK, QuantKind+from ..scheme import FP8_BLOCK as BLOCK, QuantKind, fp8_block_sizefrom .base import BankSpec, ExpertView, fused_piece, gated_epilogue_reason, limit_or_inf, MoEConfig, MoEKernel, MoEMethodFP8 = torch.float8_e4m3fn@@ -24,6 +24,8 @@ class TritonFp8BlockMoEKernel(MoEKernel):reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=False)if reason:return reason+ if cfg.scheme is not None and fp8_block_size(cfg.scheme) != BLOCK:+ return f"fp8 block MoE kernel serves {BLOCK}x{BLOCK} expert blocks only"reason = gated_epilogue_reason(cfg)if reason:return f"fp8 block MoE kernel: {reason}"diff --git a/python/freetoken/layers/quantization/moe/mxfp4.py b/python/freetoken/layers/quantization/moe/mxfp4.pyindex 13cd9746..1af6d9f4 100644--- a/python/freetoken/layers/quantization/moe/mxfp4.py+++ b/python/freetoken/layers/quantization/moe/mxfp4.py@@ -12,16 +12,20 @@ E8M0 = torch.float8_e8m0fnuclass TritonMxfp4MoEKernel(MoEKernel):- """Standard OCP MXFP4 experts (DeepSeek-V4 ds_fp4): e2m1 pairs + e8m0 scales, no bias."""+ """Standard OCP MXFP4 experts (DeepSeek-V4 / V4.1 ds_fp4): e2m1 pairs + e8m0 scales, no bias;+ W4A8 with the fp8 activation round-trips at the checkpoint's block."""name = "triton"cpu_format = "ds_fp4"+ supports_inactive_slots = Truedef unusable_reason(self, cfg: MoEConfig) -> str | None:if cfg.interleaved:return "standard MXFP4 kernel reads the concatenated gate|up row order"if (cfg.alpha, cfg.beta) != (1.0, 0.0):return "standard MXFP4 kernel has no alpha / beta in its swiglu"+ if cfg.act_block is None:+ return "W4A8 ds_fp4 experts need the fp8 activation quant block"return self._common_reject(cfg, tp_ok=False, cpu_ok=True, plain_silu_only=False)def layout(self, cfg: MoEConfig) -> dict[str, BankSpec]:@@ -44,13 +48,14 @@ class TritonMxfp4MoEKernel(MoEKernel):t = view.tensorsbanks = (t["gate_up"], t["gate_up_scale"], t["down"], t["down_scale"])limit = limit_or_inf(layer)+ act_block = layer.quant_method.cfg.act_blockif is_prefill and view.n is not None:from freetoken.moe.fused_ds_fp4 import routed_experts_fp4_prefill- return routed_experts_fp4_prefill(x, topk_ids, topk_weights, *banks, limit, view.n)+ return routed_experts_fp4_prefill(x, topk_ids, topk_weights, *banks, limit, view.n, act_block=act_block)from freetoken.moe.fused_ds_fp4 import routed_experts_fp4- return routed_experts_fp4(x, topk_ids, topk_weights, *banks, limit)+ return routed_experts_fp4(x, topk_ids, topk_weights, *banks, limit, act_block=act_block)class TritonGptossMxfp4MoEKernel(MoEKernel):diff --git a/python/freetoken/layers/quantization/scheme.py b/python/freetoken/layers/quantization/scheme.pyindex 668eb6fb..89f968af 100644--- a/python/freetoken/layers/quantization/scheme.py+++ b/python/freetoken/layers/quantization/scheme.py@@ -25,7 +25,8 @@ class QuantKind(Enum):return self.value-FP8_BLOCK = 128+FP8_BLOCK = 128 # DeepSeek-V3 style square block; V4.1 exports 32x32+FP8_BLOCK_SIZES = (32, 128)NVFP4_GROUP = 16MX_GROUP = 32@@ -80,8 +81,17 @@ def fp8_tensor_scheme(scale: str, *, per_row: bool = False, input_scale: bool =return QuantScheme(QuantKind.FP8_TENSOR, WeightDesc("e4m3", (1, -1) if per_row else (-1, -1), scale), roles)-def fp8_block_scheme(scale: str) -> QuantScheme:- return QuantScheme(QuantKind.FP8_BLOCK, WeightDesc("e4m3", (FP8_BLOCK, FP8_BLOCK), scale), {"weight", "weight_scale_inv"})+def fp8_block_scheme(scale: str, block: int = FP8_BLOCK) -> QuantScheme:+ """Square ``block x block`` e4m3 weight blocks, one scale each; ``block`` is the checkpoint's ``weight_block_size``."""+ if block not in FP8_BLOCK_SIZES:+ raise NotImplementedError(f"fp8 block size {block} is not supported; one of {FP8_BLOCK_SIZES}")+ return QuantScheme(QuantKind.FP8_BLOCK, WeightDesc("e4m3", (block, block), scale), {"weight", "weight_scale_inv"})+++def fp8_block_size(scheme: QuantScheme) -> int:+ """The square block edge of an FP8_BLOCK scheme (128 for DeepSeek-V3 style exports, 32 for DeepSeek-V4.1)."""+ assert scheme.kind is QuantKind.FP8_BLOCK, scheme+ return scheme.weight.group[0]def mxfp8_scheme() -> QuantScheme:diff --git a/python/freetoken/mm/processors/deepseek_v41.py b/python/freetoken/mm/processors/deepseek_v41.pynew file mode 100644index 00000000..e40c9615--- /dev/null+++ b/python/freetoken/mm/processors/deepseek_v41.py@@ -0,0 +1,88 @@+"""DeepSeek-V4.1 image resize/patch layout and learned image-span delimiters."""++from __future__ import annotations++import math+import struct++import numpy as np+import torch+from PIL import ImageOps++from freetoken.message import MMItem+from freetoken.mm import mm_pad_value+from freetoken.mm.processor import MMProcessor, PromptReplacement, content_hash+++def image_grid(width, height, patch, ratio, min_pixels, max_tokens, max_wh_ratio=None):+ if max_wh_ratio is not None:+ width = min(width, height * max_wh_ratio)+ if width * height < min_pixels:+ scale = (min_pixels / (width * height)) ** 0.5+ width, height = int(width * scale), int(height * scale)+ w, h = math.ceil(width / patch) * patch, math.ceil(height / patch) * patch+ if math.ceil(h / (patch * ratio)) * (math.ceil(w / (patch * ratio)) + 1) + 2 > max_tokens:+ aspect, cell = height / width, patch * ratio+ wf = math.sqrt((max_tokens - 2) / aspect + 0.25) - 0.5+ hf = wf * aspect+ if wf < 1:+ h, w = (max_tokens - 2) // 2 * cell, cell+ elif hf < 1:+ h, w = cell, (max_tokens - 3) * cell+ else:+ scale = min(math.floor(wf) * cell / width, math.floor(hf) * cell / height)+ h, w = math.floor(height * scale / patch) * patch, math.floor(width * scale / patch) * patch+ return h, w+++class DeepseekV41MMProcessor(MMProcessor):+ def __init__(self, hf_config, model_path, mm):+ super().__init__(model_path, mm)+ self.vc = hf_config.vision_config+ self.placeholder = [hf_config.image_token_id]+ self.min_pixels = self.vc.min_pixels+ self.max_tokens = mm.image_max_tokens if mm.image_max_tokens is not None else self.vc.max_image_tokens+ if mm.image_min_tokens is not None:+ # Delimiters and row breaks count against the total span budget.+ self.min_pixels = max(1, mm.image_min_tokens - 2) * (self.vc.patch_size * self.vc.downsample_ratio)**2+ allowed = {"min_pixels", "max_image_tokens", "max_wh_ratio"}+ unknown = set(mm.processor_kwargs) - allowed+ if unknown:+ raise ValueError(f"unsupported DeepSeek image processor options: {sorted(unknown)}")+ self.min_pixels = mm.processor_kwargs.get("min_pixels", self.min_pixels)+ self.max_tokens = mm.processor_kwargs.get("max_image_tokens", self.max_tokens)+ self.max_wh_ratio = mm.processor_kwargs.get("max_wh_ratio", self.vc.max_wh_ratio)+ if self.max_tokens < 4 or self.min_pixels < 0:+ raise ValueError("image token budget must be >= 4 and min_pixels must be nonnegative")+ if self.max_wh_ratio is not None and self.max_wh_ratio <= 0:+ raise ValueError("max_wh_ratio must be positive")++ def process(self, images):+ p, r = self.vc.patch_size, self.vc.downsample_ratio+ items = []+ for image in images:+ h, w = image_grid(image.width, image.height, p, r, self.min_pixels, self.max_tokens, self.max_wh_ratio)+ if self.max_wh_ratio is not None and image.width >= self.max_wh_ratio * image.height:+ image = image.resize((w, h))+ else:+ image = ImageOps.pad(image, (w, h), color=(127, 127, 127))+ x = torch.from_numpy(np.asarray(image, dtype=np.float32)).permute(2, 0, 1) / 255+ x = ((x - 0.5) / 0.5).to(torch.bfloat16)+ nh, nw = h // p, w // p+ patches = x.reshape(3, nh, p, nw, p).permute(1, 3, 0, 2, 4).reshape(nh * nw, 3, p, p).contiguous()+ item_hash = content_hash(patches, struct.pack("<3i", 1, nh, nw))+ items.append(MMItem(modality="image", hash=item_hash, pad_value=mm_pad_value(item_hash), offsets=[], feature=patches,+ model_specific_data={"grid_thw": [1, nh, nw]}))+ return items++ def prompt_replacement(self, item):+ _, h, w = item.grid_thw+ r = self.vc.downsample_ratio+ n = ((h + r - 1) // r) * ((w + r - 1) // r + 1) + 2+ # Every span row, including delimiters/newlines, has a learned visual embedding.+ return PromptReplacement([self.placeholder[0]] * n)++ def dummy_items(self, dtype, device):+ p = self.vc.patch_size+ return [MMItem(modality="image", hash=0, pad_value=0, offsets=[[0, 4]],+ feature=torch.zeros(1, 3, p, p, dtype=dtype, device=device), model_specific_data={"grid_thw": [1, 1, 1]})]diff --git a/python/freetoken/models/config.py b/python/freetoken/models/config.pyindex c5943dce..55b90bb5 100644--- a/python/freetoken/models/config.py+++ b/python/freetoken/models/config.py@@ -205,7 +205,8 @@ class DSV4AttentionGroupConfig(BaseAttentionGroupConfig):"""DSV4 sparse attention (window + compressed tiers + Lightning Indexer). Standaloneclass on purpose: subclassing SWAAttentionGroupConfig would flip has_swa_attentionand reroute DSV4 through the SWA gates. Geometry beyond the latent width lives in- dsv4_args; the pool prices itself from there, not from this spec."""+ dsv4_args (``dsv41_args`` for the ``v41`` variant); the pool prices itself from there,+ not from this spec."""kind: ClassVar[Literal["dsv4"]] = "dsv4"cache_kind: ClassVar[Literal["dsv4_paged"]] = "dsv4_paged"@@ -213,6 +214,8 @@ class DSV4AttentionGroupConfig(BaseAttentionGroupConfig):num_kv_heads: inthead_dim: intsliding_window: int # the P-token window page+ # "v41": DeepSeek-V4.1's cross-layer-shared packed tiers (AttnType.DSV41, its own pool and backend)+ variant: Literal["v4", "v41"] = "v4"AttentionGroupConfig: TypeAlias = (@@ -327,6 +330,9 @@ class ModelConfig:# CSA/HCA compressors, Lightning Indexer, manifold-constrained Hyper-Connections,# hash routing). Opaque to model-agnostic engine code; None for non-DSV4 models.dsv4_args: Any | None = None+ # DeepSeek-V4.1 payload (DeepseekV41Args): DSV41 layer roles, Engram, replay policy. Opaque+ # to model-agnostic engine code; None for every other model.+ dsv41_args: Any | None = None# GLM-5.2 (glm_moe_dsa) MLA/DSA payload (GlmMoeDsaArgs): the MLA low-rank dims and the# DSA indexer geometry the model module needs. Opaque to model-agnostic engine code;# None for every other model.@@ -448,7 +454,7 @@ class ModelConfig:if isinstance(group, SWAAttentionGroupConfig):return AttnType.SWAif isinstance(group, DSV4AttentionGroupConfig):- return AttnType.DSV4+ return AttnType.DSV41 if group.variant == "v41" else AttnType.DSV4return _full_group_attn_type(group)def kv_cache_group_specs(self) -> Tuple[KVCacheGroupSpec, ...]:@@ -502,7 +508,7 @@ class ModelConfig:num_kv_heads=group.num_kv_heads,head_dim=group.head_dim,sliding_window=group.sliding_window,- attn_type=AttnType.DSV4,+ attn_type=AttnType.DSV41 if group.variant == "v41" else AttnType.DSV4,))return tuple(specs)diff --git a/python/freetoken/models/deepseek_v4/__init__.py b/python/freetoken/models/deepseek_v4/__init__.pyindex 37f5076b..e19438d1 100644--- a/python/freetoken/models/deepseek_v4/__init__.py+++ b/python/freetoken/models/deepseek_v4/__init__.py@@ -10,7 +10,7 @@ of experts is resident on the GPU (the framework's core acceleration).DeepSeek-V4-Flash is a first-class registered model on the shared paged-KV engine:its window / compressed-attention / compressed-index KV live in DSV4-owned poolsaddressed by page tables, and sparse attention is a physical-slot gather (see-:mod:`freetoken.attention.dsv4_sparse` and :mod:`freetoken.kvcache.dsv4_paged_pool`).+:mod:`freetoken.attention.dsv4_sparse` and :mod:`freetoken.kvcache.dsv4.v4_pool`)."""from .args import DeepseekV4Args, load_argsdiff --git a/python/freetoken/models/deepseek_v4/model.py b/python/freetoken/models/deepseek_v4/model.pyindex a2d4387a..9faefb83 100644--- a/python/freetoken/models/deepseek_v4/model.py+++ b/python/freetoken/models/deepseek_v4/model.py@@ -4,7 +4,7 @@ A faithful single-stream port of the reference (MLA attention with a sliding win+ stateful KV compressors / Lightning Indexer, manifold-constrained Hyper-Connections,sqrtsoftplus / hash MoE), wired onto FreeToken's shared paged engine:- - KV lives in DSV4-owned paged pools (:class:`~freetoken.kvcache.dsv4_paged_pool.DSV4PagedKVCache`):+ - KV lives in DSV4-owned paged pools (:class:`~freetoken.kvcache.dsv4.v4_pool.DSV4PagedKVCache`):each layer's window-ring + compressed KV is a region of a shared global pool, addressed byper-layer slot maps. Sparse attention is a PAGED physical-slot gather:``sparse_attn_paged`` reads KV directly from the two global pools (window / compressed) atdiff --git a/python/freetoken/models/deepseek_v41/__init__.py b/python/freetoken/models/deepseek_v41/__init__.pynew file mode 100644index 00000000..bcdef078--- /dev/null+++ b/python/freetoken/models/deepseek_v41/__init__.py@@ -0,0 +1,14 @@+"""DeepSeek-V4.1-Flash: Causal Encoder-Decoder MoE with Compressed Sparse Attention 2.++40 layers (20 encoder + 20 decoder), every layer a sliding window (128) plus -- from layer 2 -- global+sparse attention over cross-layer-shared compressed KV in packed fp4 (Full / Reindex / Reuse+modes, a hierarchical candidate pool in the decoder), single-pass mHC residual streams, Engram n-gram+memory at layers 1 and 14 streamed from disk, one shared + 384 routed MXFP4 experts per layer served+from the offload cache. Images use the DeepSeek ViT; DSpark speculative decoding is not served.+"""++from .config import parse_config+from .model import DeepseekV41ForCausalLM+from .weight import ftw_side_files, iter_expert_pieces, iter_weights, iter_vision_weights++__all__ = ["DeepseekV41ForCausalLM", "parse_config", "iter_weights", "iter_vision_weights", "iter_expert_pieces", "ftw_side_files"]diff --git a/python/freetoken/models/deepseek_v41/args.py b/python/freetoken/models/deepseek_v41/args.pynew file mode 100644index 00000000..867dcb21--- /dev/null+++ b/python/freetoken/models/deepseek_v41/args.py@@ -0,0 +1,225 @@+"""DeepSeek-V4.1-Flash hyperparameters and the static per-layer attention roles.++``DeepseekV41Args`` mirrors the reference ``ModelArgs`` (``inference/model.py``) field for field,+read from the HF ``config.json``'s ``text_config`` (the ``inference/config.json`` dialect is also+accepted). Everything the engine reconciles at runtime (``max_seq_len``, the replay policy) is+attached by ``parse_config`` / the engine, not read from the checkpoint.++``LayerRole`` is the attention mode table (tech report sec. 2.3.1): every compressing layer is Full+(owns the compressed KV + indexer K), Reindex (own indexer over a shared K) or Reuse (shared+Top-K); the candidate pool of the Hierarchical Sparse Indexer (sec. 2.3.2) is produced by the+first decoder Full layer and consumed by the decoder Reindex layers after it.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from enum import Enum+from typing import Any, Literal++# text_config key -> ModelArgs key, where the two dialects differ+_HF_TO_ARGS = {+ "hidden_size": "dim",+ "moe_intermediate_size": "moe_inter_dim",+ "num_hidden_layers": "n_layers",+ "num_nextn_predict_layers": "n_mtp_layers",+ "num_attention_heads": "n_heads",+ "num_experts_per_tok": "n_activated_experts",+ "scoring_func": "score_func",+ "routed_scaling_factor": "route_scale",+ "qk_rope_head_dim": "rope_head_dim",+ "rms_norm_eps": "norm_eps",+ "sliding_window": "window_size",+ "kv_source_layer_ids": "kv_source_layers",+ "index_source_layer_ids": "index_source_layers",+ "candidate_source_layer_id": "candidate_source_layer",+ "engram_pad_token_id": "engram_pad_id",+ "dspark_num_experts_per_tok": "dspark_n_activated_experts",+}+++class Mode(str, Enum):+ WINDOW = "window" # sliding window only (compress_ratio == 0)+ FULL = "full" # computes main KV + indexer K, runs its indexer+ REINDEX = "reindex" # reuses main KV + indexer K, runs its own indexer+ REUSE = "reuse" # reuses main KV and the latest Top-K+++@dataclass(frozen=True)+class LayerRole:+ layer_id: int+ ratio: int+ mode: Mode+ kv_source: int | None # the layer whose main KV / indexer K this layer reads (itself when Full)+ is_candidate_source: bool # builds the hierarchical candidate pool+ uses_candidates: bool # indexes only inside the candidate pool++ @property+ def compresses(self) -> bool:+ return self.ratio > 0++ @property+ def has_indexer(self) -> bool:+ return self.mode in (Mode.FULL, Mode.REINDEX)+++@dataclass+class DeepseekV41Args:+ vocab_size: int = 129280+ dim: int = 5120+ moe_inter_dim: int = 2304+ n_layers: int = 40+ n_mtp_layers: int = 3+ n_heads: int = 64+ # moe+ n_routed_experts: int = 384+ n_shared_experts: int = 1+ n_activated_experts: int = 6+ score_func: Literal["softmax", "sigmoid", "sqrtsoftplus"] = "sqrtsoftplus"+ norm_topk_prob: bool = True+ route_scale: float = 1.5+ swiglu_limit: float = 10.0+ # attention: latent q, one shared latent kv head, grouped low-rank output projection+ q_lora_rank: int = 1280+ head_dim: int = 512+ rope_head_dim: int = 64+ norm_eps: float = 1e-20+ o_groups: int = 8+ o_lora_rank: int = 1024+ # sparse attention+ window_size: int = 128+ compress_ratios: tuple[int, ...] = () # one per layer, MTP layers included+ kv_source_layers: tuple[int, ...] = ()+ index_source_layers: tuple[int, ...] = ()+ compress_rope_theta: float = 160000.0+ original_seq_len: int = 65536+ rope_theta: float = 10000.0+ rope_factor: float = 16.0+ beta_fast: int = 32+ beta_slow: int = 1+ index_n_heads: int = 32+ index_head_dim: int = 128+ index_topk: int = 512+ candidate_source_layer: int = -1+ candidate_topk_blocks: int = 0+ candidate_block_size: int = 0+ # hyper-connections (single-pass mHC)+ hc_mult: int = 4+ hc_sinkhorn_iters: int = 20+ hc_eps: float = 1e-6+ # engram+ engram_layer_ids: tuple[int, ...] = ()+ engram_num_embeddings: tuple[int, ...] = ()+ engram_max_ngram_size: int = 4+ engram_vocab_size: int = 0+ engram_n_heads: int = 0+ engram_head_dim: int = 0+ engram_pad_id: int = 2+ engram_compressed_vocab_size: int = 0+ # dspark (weights are not served in v1; kept so the config round-trips)+ dspark_block_size: int = 0+ dspark_target_layer_ids: tuple[int, ...] = ()+ # ---- runtime (engine-reconciled, never read from the checkpoint) ----+ max_seq_len: int = 4096+ max_batch_size: int = 1+ # "bounded": the decoder runs on the prompt's last window_size tokens with the sliding window+ # truncated there (Decoder SWA Bounded Replay, tech report sec. 3.2.2); "exact": on every token.+ swa_decoder_replay: Literal["bounded", "exact"] = "bounded"+ roles: tuple[LayerRole, ...] = field(default_factory=tuple, repr=False)++ def __post_init__(self) -> None:+ for name in ("compress_ratios", "kv_source_layers", "index_source_layers", "engram_layer_ids",+ "engram_num_embeddings", "dspark_target_layer_ids"):+ setattr(self, name, tuple(int(v) for v in getattr(self, name)))+ if len(self.compress_ratios) < self.n_layers:+ raise ValueError(f"compress_ratios has {len(self.compress_ratios)} entries for {self.n_layers} layers")+ if self.n_shared_experts != 1:+ raise ValueError("DeepSeek-V4.1 has exactly one shared expert")+ if self.hc_mult < 1:+ raise ValueError("hc_mult must be positive")+ self.roles = tuple(self._role(layer) for layer in range(self.n_layers))+ if len(self.engram_layer_ids) != len(self.engram_num_embeddings):+ raise ValueError("engram_layer_ids and engram_num_embeddings differ in length")++ # ---- derived ----+ def _role(self, layer: int) -> LayerRole:+ ratio = self.compress_ratios[layer]+ if ratio == 0:+ return LayerRole(layer, 0, Mode.WINDOW, None, False, False)+ sources = [s for s in self.kv_source_layers if s <= layer]+ if not sources:+ raise ValueError(f"layer {layer} compresses but no kv source precedes it")+ source = max(sources)+ if self.compress_ratios[source] != ratio:+ raise ValueError(f"layer {layer} (ratio {ratio}) reads kv source {source} (ratio {self.compress_ratios[source]})")+ if layer == source:+ mode = Mode.FULL+ elif layer in self.index_source_layers:+ mode = Mode.REINDEX+ else:+ mode = Mode.REUSE+ cand = self.candidate_source_layer+ is_cand_src = cand >= 0 and layer == cand+ uses = cand >= 0 and cand < layer and mode is Mode.REINDEX+ if uses and self.kv_source_layers and max(s for s in self.kv_source_layers if s <= cand) != source:+ raise ValueError(f"layer {layer} would index the candidate pool of layer {cand} over a different kv source")+ return LayerRole(layer, ratio, mode, source, is_cand_src, uses)++ @property+ def nope_head_dim(self) -> int:+ return self.head_dim - self.rope_head_dim++ @property+ def backbone_kv_sources(self) -> tuple[int, ...]:+ return tuple(s for s in self.kv_source_layers if s < self.n_layers)++ @property+ def decoder_start_layer(self) -> int:+ """The first decoder layer: the last kv source (CED projects the decoder's global KV from+ its input, the final encoder hidden state)."""+ return max(self.backbone_kv_sources) if self.backbone_kv_sources else self.n_layers++ @property+ def engram_hash_cols(self) -> int:+ return (self.engram_max_ngram_size - 1) * self.engram_n_heads++ @property+ def index_softmax_scale(self) -> float:+ return self.index_head_dim**-0.5++ def freqs_params(self, layer: int) -> tuple[int, int, float, float, int, int]:+ """``(rotary_dim, original_seq_len, theta, factor, beta_fast, beta_slow)``: compressing layers+ rotate at the compressed theta under YaRN, window-only layers at the base theta without it."""+ if self.compress_ratios[layer]:+ return self.rope_head_dim, self.original_seq_len, self.compress_rope_theta, self.rope_factor, self.beta_fast, self.beta_slow+ return self.rope_head_dim, 0, self.rope_theta, self.rope_factor, self.beta_fast, self.beta_slow++ @classmethod+ def from_hf(cls, hf_config: Any) -> "DeepseekV41Args":+ """From the HF config (``text_config`` of the multimodal wrapper, or the text config itself,+ or the ``inference/config.json`` dialect)."""+ text = getattr(hf_config, "text_config", None) or hf_config+ if isinstance(text, dict):+ raw = text+ elif hasattr(text, "to_dict"): # PretrainedConfig / RawConfigShim+ raw = text.to_dict()+ else:+ raw = vars(text)+ names = {f.name for f in cls.__dataclass_fields__.values()} - {"roles", "max_seq_len", "max_batch_size", "swa_decoder_replay"}+ kwargs: dict[str, Any] = {}+ for key, value in raw.items():+ key = _HF_TO_ARGS.get(key, key)+ if key in names:+ kwargs[key] = value+ # HF nests the YaRN parameters; the inference dialect keeps them flat+ scaling = raw.get("rope_scaling") or raw.get("rope_parameters")+ if isinstance(scaling, dict):+ if "factor" in scaling:+ kwargs["rope_factor"] = scaling["factor"]+ for src, dst in (("beta_fast", "beta_fast"), ("beta_slow", "beta_slow"), ("original_max_position_embeddings", "original_seq_len")):+ if src in scaling:+ kwargs[dst] = scaling[src]+ return cls(**kwargs)+++__all__ = ["DeepseekV41Args", "LayerRole", "Mode"]diff --git a/python/freetoken/models/deepseek_v41/attention.py b/python/freetoken/models/deepseek_v41/attention.pynew file mode 100644index 00000000..1643ddb6--- /dev/null+++ b/python/freetoken/models/deepseek_v41/attention.py@@ -0,0 +1,305 @@+"""DSV41 attention layer (reference ``Attention``): latent MLA over a sliding window plus, on+compressing layers, ``index_topk`` compressed positions -- one paged sparse-attention call.++Per layer mode (``LayerRole``):++* Full -- ``publish_*`` compresses its input into the kv source's main KV (fp4) and index keys+ (fp4), then its indexer selects fresh Top-K rows and publishes them on the forward's+ ``SharedSelection``; the decoder's first Full layer also builds the candidate pool.+* Reindex -- selects fresh Top-K rows with its own indexer over the shared keys (inside the+ candidate pool in the decoder) and publishes them.+* Reuse -- attends over the latest published Top-K rows.+* Window -- the sliding window only.++All modes compute their own query and sliding-window KV (fp8 into the layer's window ring). The+attention output is rotated back (K == V share one rotated latent) and projected through the+grouped low-rank ``wo_a`` (block-diagonal over ``o_groups``; fp8 payload kept, W8A16 at decode) and ``wo_b``.++Prefill runs ragged over a flat token stream tiled by ``PrefillSegment``s; the compressor / indexer+state is per request, so those run per segment. Decode is batched (one token per row) and+CUDA-graph safe: every gather reads the backend's snapshot, never the live page table.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import List++import torch+from freetoken.attention.dsv41_sparse import PrefillSegment+from freetoken.core import get_global_ctx+from freetoken.kernel.triton.dsv4.fp8_linear import GEMV_MAX_M, dequant_block_fp8, grouped_w8a16_gemv+from freetoken.layers import BaseOP, LinearReplicated, RMSNorm+from freetoken.layers.quantization import QuantKind+from freetoken.layers.quantization.scheme import fp8_block_size++from .args import DeepseekV41Args, LayerRole, Mode+from .compressor import Compressor+from .indexer import Indexer+from .rope import apply_rotary_emb, apply_rotary_emb_decode, get_freqs_cis+++class DSV41Attention(BaseOP):+ def __init__(self, args: DeepseekV41Args, role: LayerRole, *, quant_config=None, prefix: str = ""):+ self.args = args+ self.role = role+ self.layer_id = role.layer_id+ self.n_heads = args.n_heads+ self.head_dim = args.head_dim+ self.rope_dim = args.rope_head_dim+ self.n_groups = args.o_groups+ self.o_lora_rank = args.o_lora_rank+ self.window = args.window_size+ self.softmax_scale = args.head_dim**-0.5++ self.wq_a = LinearReplicated(args.dim, args.q_lora_rank, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wq_a")+ self.q_norm = RMSNorm(args.q_lora_rank, args.norm_eps)+ self.wq_b = LinearReplicated(args.q_lora_rank, args.n_heads * args.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wq_b")+ self.wkv = LinearReplicated(args.dim, args.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wkv")+ self.kv_norm = RMSNorm(args.head_dim, args.norm_eps)+ # Block-diagonal over groups. The reference dequantizes it to bf16 and runs a grouped einsum+ # (the activation stays bf16); a quantized checkpoint keeps the e4m3 payload + e8m0 scale+ # here and dequantizes in-kernel -- the same math, half the bytes per decode step, and no+ # bf16 copy in VRAM. An unquantized checkpoint ships it bf16.+ wo_a_scheme = quant_config.scheme_for(f"{prefix}.wo_a") if quant_config is not None else None+ self._wo_a_block = fp8_block_size(wo_a_scheme) if wo_a_scheme is not None and wo_a_scheme.kind is QuantKind.FP8_BLOCK else None+ rows, cols = args.o_groups * args.o_lora_rank, args.n_heads * args.head_dim // args.o_groups+ if self._wo_a_block is not None:+ self.wo_a = torch.empty(rows, cols, dtype=torch.float8_e4m3fn)+ self.wo_a_scale = torch.empty(rows // self._wo_a_block, cols // self._wo_a_block, dtype=torch.float8_e8m0fnu)+ else:+ self.wo_a = torch.empty(rows, cols, dtype=torch.bfloat16)+ self.wo_b = LinearReplicated(args.o_groups * args.o_lora_rank, args.dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wo_b")+ self.attn_sink = torch.empty(args.n_heads, dtype=torch.float32)++ self.compressor = Compressor(args, self.layer_id, role.ratio, quant_config=quant_config, prefix=f"{prefix}.compressor") if role.mode is Mode.FULL else None+ self.indexer = Indexer(args, role, quant_config=quant_config, prefix=f"{prefix}.indexer") if role.has_indexer else None+ self._freqs_params = args.freqs_params(self.layer_id)+ self._freqs: torch.Tensor | None = None++ # addressing lives on the live backend, pool buffers on the live pool (both read per access)+ @property+ def attn(self):+ return get_global_ctx().attn_backend++ @property+ def freqs_key(self) -> tuple:+ """The RoPE table this layer rotates with (window layers: base theta; compressing layers: the+ compressed theta under YaRN). Layers with the same key share one table."""+ return self._freqs_params++ def bind(self, device: torch.device, tables: dict | None = None) -> None:+ """Attach the RoPE table; ``tables`` (key -> table) lets the model share one table per key+ instead of every layer holding its own ``[max_seq_len, rd // 2]`` copy."""+ if tables is not None and self.freqs_key in tables:+ self._freqs = tables[self.freqs_key]+ return+ rd, original_seq_len, theta, factor, beta_fast, beta_slow = self._freqs_params+ self._freqs = get_freqs_cis(rd, self.args.max_seq_len, original_seq_len, theta, factor, beta_fast, beta_slow, device)+ if tables is not None:+ tables[self.freqs_key] = self._freqs++ @property+ def freqs(self) -> torch.Tensor:+ assert self._freqs is not None, "attention layer is not bound (call bind first)"+ return self._freqs++ # ----- shared pieces -------------------------------------------------------------------+ def _q(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:+ qr = self.q_norm.forward(self.wq_a.forward(x))+ q = self.wq_b.forward(qr).unflatten(-1, (self.n_heads, self.head_dim))+ return qr, q++ def _kv(self, x: torch.Tensor) -> torch.Tensor:+ return self.kv_norm.forward(self.wkv.forward(x))++ def _wo(self, o: torch.Tensor) -> torch.Tensor:+ """``wo_b(wo_a(o))``: up to ``GEMV_MAX_M`` rows the grouped W8A16 GEMV over the fp8 payload, else the+ reference's grouped einsum over the bf16 (or dequantized) ``wo_a``."""+ t = o.shape[0]+ o = o.reshape(t, self.n_groups, -1)+ if self._wo_a_block is not None and t <= GEMV_MAX_M:+ return self.wo_b.forward(grouped_w8a16_gemv(o, self.wo_a, self.wo_a_scale, block=self._wo_a_block))+ wo_a = self.wo_a if self._wo_a_block is None else dequant_block_fp8(self.wo_a, self.wo_a_scale, block=self._wo_a_block)+ return self.wo_b.forward(torch.einsum("tgd,grd->tgr", o, wo_a.view(self.n_groups, self.o_lora_rank, -1)).flatten(1))++ def load_state_dict(self, state_dict, *, prefix: str = "", _internal: bool = False) -> None:+ key = f"{prefix}.wo_a" if prefix else "wo_a"+ scale_key = f"{key}_scale"+ if self._wo_a_block is None and scale_key in state_dict:+ # a bf16 layer (no fp8 scheme) fed a quantized checkpoint dequantizes wo_a on the way in+ from .weight import dequant_fp8_block++ state_dict[key] = dequant_fp8_block(state_dict[key], state_dict.pop(scale_key))+ elif self._wo_a_block is not None and scale_key not in state_dict:+ # an FTW conversion made before wo_a kept its fp8 payload carries a dequantized bf16+ # tensor and no scale: say so instead of failing on the missing key+ raise RuntimeError(+ f"{key}: the checkpoint has no {scale_key} for the fp8 wo_a layout; an FTW checkpoint "+ "converted by an earlier build stores wo_a dequantized -- reconvert it with `ft checkpoint`"+ )+ super().load_state_dict(state_dict, prefix=prefix, _internal=_internal)++ # ----- prefill -----------------------------------------------------------------------------+ def publish_prefill(self, x: torch.Tensor, segments: List[PrefillSegment]) -> None:+ """Full mode: compress this layer's input for every segment into the source's main KV and+ index keys. Runs on EVERY new token (under bounded replay the decoder source publishes for+ the whole prompt even though its attention runs on the prompt's last window only)."""+ assert self.compressor is not None and self.indexer is not None+ attn, ratio, src = self.attn, self.role.ratio, self.layer_id+ for seg in segments:+ slots = attn.window_slots_of(seg.table_idx, seg.start_pos, seg.end)+ tail = None+ if self.compressor.needs_tail_carry(seg.start_pos):+ tail = int(attn.window_slots_of(seg.table_idx, seg.start_pos - 1, seg.start_pos).item())+ latent, starts = self.compressor.forward_prefill(x[seg.offset : seg.offset + seg.n], seg.start_pos, slots, tail)+ if not latent.shape[0]:+ continue+ rows = attn.compressed_rows_of(seg.table_idx, starts, ratio)+ freqs = self.freqs.index_select(0, starts)+ attn.store_index(self.indexer.index_keys(latent, freqs), src, rows)+ apply_rotary_emb(latent[..., -self.rope_dim :], freqs)+ attn.store_main(latent, src, rows)++ def forward_prefill(self, x: torch.Tensor, segments: List[PrefillSegment], positions: torch.Tensor) -> torch.Tensor:+ """Ragged prefill over ``x [T, dim]`` tiled by ``segments`` (``positions [T]`` absolute)."""+ attn, role = self.attn, self.role+ freqs = self.freqs.index_select(0, positions)+ qr, q = self._q(x)+ apply_rotary_emb(q[..., -self.rope_dim :], freqs)+ kv = self._kv(x)+ apply_rotary_emb(kv[..., -self.rope_dim :], freqs)++ sel = attn.selection+ win_parts, cmp_parts, pools = [], [], []+ for seg in segments:+ lo, hi = seg.offset, seg.offset + seg.n+ attn.store_window(kv[lo:hi], self.layer_id, attn.layer_window_slots_of(self.layer_id, seg.table_idx, seg.start_pos, seg.end))+ win_parts.append(attn.window_topk_prefill(seg, self.layer_id))+ if not role.compresses:+ continue+ if role.has_indexer:+ locs = attn.locs_prefill(seg.table_idx, seg.end // role.ratio * role.ratio)+ cand = sel.candidates[:, lo:hi] if role.uses_candidates else None+ rows, pool = self.indexer.select_prefill(+ x[lo:hi], qr[lo:hi], freqs[lo:hi], start_pos=seg.start_pos, ratio=role.ratio, locs=locs, candidates=cand,+ )+ cmp_parts.append(rows)+ if pool is not None:+ pools.append(pool)+ else:+ assert sel.source == role.kv_source, "Reuse layer without a published selection for its source"+ cmp_parts.append(sel.topk_rows[:, lo:hi])+ if role.has_indexer:+ sel.source, sel.topk_rows = role.kv_source, torch.cat(cmp_parts, dim=1)+ if pools:+ sel.candidates = torch.cat(pools, dim=1)+ win = torch.cat(win_parts, dim=1)+ topk = torch.cat([win, torch.cat(cmp_parts, dim=1)], dim=-1) if role.compresses else win++ o = attn.attend(q.unsqueeze(0), self.layer_id, topk.to(torch.int32), self.window, self.attn_sink, self.softmax_scale)[0]+ apply_rotary_emb(o[..., -self.rope_dim :], freqs, inverse=True)+ return self._wo(o)++ # ----- decode ------------------------------------------------------------------------------+ def forward_decode(self, x: torch.Tensor, pos: torch.Tensor, rows: torch.Tensor, dctx: "DecodeStepContext", cmp_stage_cap: int) -> torch.Tensor:+ """Batched single-token attention. ``pos [B]`` device positions, ``rows [B]`` local rows into+ the decode snapshot, ``dctx`` the step's layer-invariant context (ring slots, window+ candidates, RoPE rows), ``cmp_stage_cap`` the static compressed staging bound (max position+ any row reaches; the capture width under a graph)."""+ attn, role = self.attn, self.role+ B = x.shape[0]+ freqs_t = dctx.freqs[self.freqs_key]+ qr, q = self._q(x)+ apply_rotary_emb_decode(q[..., -self.rope_dim :].unsqueeze(1), freqs_t)+ kv = self._kv(x)+ apply_rotary_emb_decode(kv[..., -self.rope_dim :].unsqueeze(1), freqs_t)+ # the shared ring context feeds the compressor's carry ring on every layer; a request-private+ # window layer stores and reads its own KV through its ring context instead+ window_slots, prev_window_slots = dctx.window_slots, dctx.prev_window_slots+ private = dctx.private is not None and attn.pool.is_private_window(self.layer_id)+ tier = dctx.private if private else dctx.shared+ attn.store_window(kv, self.layer_id, tier.slots if private else window_slots)++ cmp_counts = None+ if role.compresses:+ ratio, src = role.ratio, role.kv_source+ n_stage = (cmp_stage_cap + 1) // ratio+ sel = attn.selection+ if role.mode is Mode.FULL:+ latent, completed = self.compressor.forward_decode(x, pos, prev_window_slots, window_slots)+ dst = attn.decode_store_rows(rows, pos, ratio, src, completed)+ freqs_g = self.freqs.index_select(0, (pos + 1 - ratio).clamp_min(0))+ k = self.indexer.k_norm.forward(self.indexer.wk.forward(latent))+ apply_rotary_emb_decode(k[..., -self.rope_dim :].unsqueeze(1), freqs_g)+ attn.store_index(k, src, dst)+ apply_rotary_emb_decode(latent[..., -self.rope_dim :].unsqueeze(1), freqs_g)+ attn.store_main(latent, src, dst)+ if role.has_indexer:+ cand = sel.candidates if role.uses_candidates else None+ cmp_rows, counts, pool = self.indexer.select_decode(x, qr, freqs_t, pos=pos, ratio=ratio, locs=attn.snapshot(), T=n_stage, candidates=cand)+ sel.source, sel.topk_rows, sel.cmp_counts = src, cmp_rows, counts+ if pool is not None:+ sel.candidates = pool+ # the step's [window | compressed] candidate buffer of this tier takes the new rows once;+ # the Reuse layers that follow read them in place (no per-layer cat / cast)+ tier.topk[..., self.window :].copy_(cmp_rows)+ else:+ assert sel.source == src, "Reuse layer without a published selection for its source"+ counts = sel.cmp_counts+ topk = tier.topk+ cmp_counts = counts+ else:+ topk = tier.window_topk++ o = attn.attend(q.view(B, 1, self.n_heads, self.head_dim), self.layer_id, topk, self.window, self.attn_sink, self.softmax_scale, cmp_counts=cmp_counts)+ apply_rotary_emb_decode(o[..., -self.rope_dim :], freqs_t, inverse=True)+ return self._wo(o.view(B, self.n_heads, self.head_dim))+++@dataclass+class TierCandidates:+ """One window tier's per-step decode candidates: the request's store slot and the int32+ ``[B, 1, win]`` window candidates, plus the ``[B, 1, win + topk]`` ``[window | compressed]``+ buffer whose compressed half the index layers of that tier fill as they publish."""++ slots: torch.Tensor+ window_topk: torch.Tensor+ topk: torch.Tensor+++@dataclass+class DecodeStepContext:+ """Everything a decode step resolves once for all layers (a capture records it once): the shared+ pool's ring slots (the compressor's carry ring keys on them), the shared / private tier+ candidates, and each RoPE table gathered at the step's positions."""++ window_slots: torch.Tensor+ prev_window_slots: torch.Tensor+ shared: TierCandidates+ private: TierCandidates | None+ freqs: dict++ @classmethod+ def build(cls, layers, md, pos: torch.Tensor, rows: torch.Tensor, window: int, topk: int) -> "DecodeStepContext":+ B = pos.shape[0]++ def tier(slots, window_topk):+ win32 = window_topk.to(torch.int32)+ buf = torch.empty((B, 1, window + topk), dtype=torch.int32, device=pos.device)+ buf[..., :window].copy_(window_topk)+ return TierCandidates(slots, win32, buf)++ window_slots, prev_window_slots, window_topk = md.window_ctx(pos, rows)+ private = None+ if get_global_ctx().kv_cache.private_window_layer_ids:+ private = tier(*md.private_window_ctx(pos, rows))+ tables: dict = {}+ for layer in layers:+ key = layer.freqs_key+ if key not in tables:+ tables[key] = layer.freqs.index_select(0, pos)+ return cls(window_slots, prev_window_slots, tier(window_slots, window_topk), private, tables)+++__all__ = ["DSV41Attention", "DecodeStepContext", "TierCandidates"]diff --git a/python/freetoken/models/deepseek_v41/compressor.py b/python/freetoken/models/deepseek_v41/compressor.pynew file mode 100644index 00000000..5d5d07fe--- /dev/null+++ b/python/freetoken/models/deepseek_v41/compressor.py@@ -0,0 +1,131 @@+"""DSV41 KV compressor: ``compress_ratio`` consecutive tokens -> one latent (reference ``Compressor``).++* ratio 1 (the CED decoder source): a plain bf16 projection + RMSNorm, one latent per token.+* ratio > 1 (the encoder sources): ``wkv`` / ``wgate`` promoted to fp32, a softmax over the group's+ gate scores pools the group's ``wkv`` rows; the trailing partial group is carried across chunks+ and decode steps in the pool's compress-state ring (per window page, so a page-aligned radix hit+ resumes it by value -- see ``DSV41SparseAttnBackend`` for the addressing).++Returns the latent BEFORE RoPE: the indexer key is projected from the unrotated latent, the+attention layer rotates and quantizes it afterwards.+"""++from __future__ import annotations++import torch+import torch.nn.functional as F+from freetoken.core import get_global_ctx+from freetoken.layers import BaseOP, LinearReplicated, RMSNorm++from .args import DeepseekV41Args+++class Compressor(BaseOP):+ def __init__(self, args: DeepseekV41Args, layer_id: int, ratio: int, *, quant_config=None, prefix: str = ""):+ assert ratio >= 1+ self.layer_id = layer_id+ self.ratio = ratio+ self.dim = args.dim+ self.head_dim = args.head_dim+ self.window = args.window_size+ if ratio > 1:+ # fp32 like the reference (the softmax pooling runs in fp32); the checkpoint ships bf16+ self.wkv = torch.empty(args.head_dim, args.dim, dtype=torch.float32)+ self.wgate = torch.empty(args.head_dim, args.dim, dtype=torch.float32)+ else:+ self.wkv = LinearReplicated(args.dim, args.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wkv")+ self.norm = RMSNorm(args.head_dim, args.norm_eps)++ @property+ def attn(self):+ return get_global_ctx().attn_backend++ def _project(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:+ """fp32 ``wkv`` / ``wgate`` projections."""+ xf = x.float()+ return F.linear(xf, self.wkv), F.linear(xf, self.wgate)++ # ----- prefill -----------------------------------------------------------------------+ def needs_tail_carry(self, start_pos: int) -> bool:+ """Whether a prefill from ``start_pos`` resumes a partial group (never at a page-aligned start)."""+ return self.ratio > 1 and start_pos % self.ratio != 0++ def forward_prefill(+ self, x: torch.Tensor, start_pos: int, window_slots: torch.Tensor, tail_window_slot: int | None = None,+ ) -> tuple[torch.Tensor, torch.Tensor]:+ """Compress the tokens ``[start_pos, start_pos + n)`` of one request.++ ``window_slots`` are the tokens' window slots (the ring block of each window page carries the+ partial group); ``tail_window_slot`` is the previous token's slot, needed only when+ ``needs_tail_carry(start_pos)``. Returns ``(latent [G, head_dim] bf16 pre-RoPE, group_starts [G]+ int64)`` for every group that completed in this call.+ """+ n = x.shape[0]+ device = x.device+ if self.ratio == 1:+ latent = self.norm.forward(self.wkv.forward(x))+ return latent, torch.arange(start_pos, start_pos + n, device=device)++ ratio, d = self.ratio, self.head_dim+ kv, score = self._project(x)+ # the partial group in progress at start_pos: its rows live in the previous token's ring block+ offset = start_pos % ratio+ if offset:+ assert tail_window_slot is not None, "resuming mid-group needs the previous token's window slot"+ carry = self.attn.read_carry(self.layer_id, tail_window_slot) # [ratio, 2d]+ kv = torch.cat([carry[:offset, :d], kv], dim=0)+ score = torch.cat([carry[:offset, d:], score], dim=0)+ total = kv.shape[0]+ groups = total // ratio+ remainder = total % ratio+ cut = groups * ratio+ if groups:+ kv_g = kv[:cut].view(groups, ratio, d)+ score_g = score[:cut].view(groups, ratio, d)+ pooled = (kv_g * score_g.softmax(dim=1)).sum(dim=1)+ latent = self.norm.forward(pooled.to(x.dtype))+ else:+ latent = x.new_empty(0, d)+ group_starts = (start_pos - offset) + ratio * torch.arange(groups, device=device)+ # ring: the reset block at every page boundary crossed (a page holds whole groups), and the+ # in-progress tail (the remainder rows) on the last page so decode / the next chunk resume+ end = start_pos + n+ self.attn.write_boundary_carries(self.layer_id, lo=start_pos, hi=end, window_slots=window_slots)+ if end % self.window != 0:+ block = self._carry_block(kv[cut:], score[cut:], remainder)+ self.attn.write_carry(self.layer_id, int(window_slots[-1].item()), block)+ return latent, group_starts++ def _carry_block(self, kv_tail: torch.Tensor, score_tail: torch.Tensor, filled: int) -> torch.Tensor:+ ratio, d = self.ratio, self.head_dim+ ks = kv_tail.new_zeros(ratio, d)+ ss = kv_tail.new_full((ratio, d), float("-inf"))+ if filled:+ ks[:filled] = kv_tail+ ss[:filled] = score_tail+ return torch.cat([ks, ss], dim=-1)++ # ----- decode ------------------------------------------------------------------------+ def forward_decode(self, x: torch.Tensor, pos: torch.Tensor, prev_window_slots: torch.Tensor, window_slots: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:+ """One token per row. Returns ``(latent [B, head_dim] bf16 pre-RoPE, completed [B] bool)``;+ a row whose group did not complete carries a meaningless latent the caller discards."""+ B = x.shape[0]+ if self.ratio == 1:+ latent = self.norm.forward(self.wkv.forward(x))+ return latent, torch.ones(B, dtype=torch.bool, device=x.device)++ from freetoken.kernel.triton.dsv4.compress import gated_pool++ ratio, d = self.ratio, self.head_dim+ kv, score = self._project(x)+ idx = (pos % ratio).view(B, 1, 1).expand(B, 1, d)+ block = self.attn.read_carry_blocks(self.layer_id, prev_window_slots) # [B, ratio, 2d]+ ks = block[..., :d].clone().scatter_(1, idx, kv.unsqueeze(1))+ ss = block[..., d:].clone().scatter_(1, idx, score.unsqueeze(1))+ self.attn.write_carry_blocks(self.layer_id, window_slots, torch.cat([ks, ss], dim=-1))+ pooled = gated_pool(ks, ss, x.dtype).view(B, d)+ completed = (pos + 1) % ratio == 0+ return self.norm.forward(pooled), completed+++__all__ = ["Compressor"]diff --git a/python/freetoken/models/deepseek_v41/config.py b/python/freetoken/models/deepseek_v41/config.pynew file mode 100644index 00000000..9e738939--- /dev/null+++ b/python/freetoken/models/deepseek_v41/config.py@@ -0,0 +1,95 @@+"""Engine-facing config for DeepSeek-V4.1-Flash.++``parse_config`` maps the standard transformer fields into :class:`ModelConfig`, carries the full+:class:`DeepseekV41Args` in ``ModelConfig.dsv41_args`` for the model module and the KV pool (which+sizes its tiers from it), and declares the ``v41`` variant of the DSV4 attention group.+The engine reconciles the runtime knobs (``max_seq_len``, ``swa_decoder_replay``) onto ``dsv41_args``+at config resolution.+"""++from __future__ import annotations++from dataclasses import dataclass, fields+from typing import Any++from freetoken.models.config import DSV4AttentionGroupConfig, ModelConfig, RotaryConfig++from .args import DeepseekV41Args+++@dataclass(frozen=True)+class VisionConfig:+ num_hidden_layers: int = 32+ hidden_size: int = 1024+ num_attention_heads: int = 16+ intermediate_size: int = 2816+ patch_size: int = 14+ rope_theta: float = 10000.0+ downsample_ratio: int = 3+ max_image_tokens: int = 1024+ min_pixels: int = 544 * 544+ max_wh_ratio: float | None = None+++def parse_config(hf_config: Any) -> ModelConfig:+ args = DeepseekV41Args.from_hf(hf_config)+ text = getattr(hf_config, "text_config", None) or hf_config+ max_position = int(getattr(text, "max_position_embeddings", 0) or getattr(hf_config, "max_position_embeddings", 0) or 0)+ if max_position <= 0:+ max_position = int(args.original_seq_len * args.rope_factor)+ vc = getattr(hf_config, "vision_config", None)+ vision = VisionConfig(**{f.name: getattr(vc, f.name, f.default) for f in fields(VisionConfig)}) if vc is not None else None++ return ModelConfig(+ num_layers=args.n_layers,+ num_qo_heads=args.n_heads,+ num_kv_heads=1, # one shared latent KV head (K == V)+ head_dim=args.head_dim,+ hidden_size=args.dim,+ vocab_size=args.vocab_size,+ intermediate_size=args.moe_inter_dim,+ hidden_act="silu",+ rms_norm_eps=args.norm_eps,+ tie_word_embeddings=False,+ rotary_config=RotaryConfig(+ head_dim=args.head_dim,+ rotary_dim=args.rope_head_dim,+ max_position=max_position,+ base=args.rope_theta,+ scaling={+ "rope_type": "yarn",+ "factor": args.rope_factor,+ "beta_fast": args.beta_fast,+ "beta_slow": args.beta_slow,+ "original_max_position_embeddings": args.original_seq_len,+ },+ ),+ num_experts=args.n_routed_experts,+ num_experts_per_tok=args.n_activated_experts,+ moe_intermediate_size=args.moe_inter_dim,+ norm_topk_prob=args.norm_topk_prob,+ n_shared_experts=args.n_shared_experts,+ routed_scaling_factor=args.route_scale,+ swiglu_limit=args.swiglu_limit,+ model_type="deepseek_v41",+ architectures=["DeepseekV41ForCausalLM"],+ moe_enabled=True,+ expert_quant="ds_fp4",+ attn_sm_scale=args.head_dim**-0.5,+ dsv41_args=args,+ vision_config=vision,+ image_token_id=getattr(hf_config, "image_token_id", None),+ attention_groups=(+ DSV4AttentionGroupConfig(+ name="dsv41",+ layer_ids=tuple(range(args.n_layers)),+ num_kv_heads=1,+ head_dim=args.head_dim,+ sliding_window=args.window_size,+ variant="v41",+ ),+ ),+ )+++__all__ = ["parse_config"]diff --git a/python/freetoken/models/deepseek_v41/engram.py b/python/freetoken/models/deepseek_v41/engram.pynew file mode 100644index 00000000..1c997a7c--- /dev/null+++ b/python/freetoken/models/deepseek_v41/engram.py@@ -0,0 +1,262 @@+"""Engram conditional memory (reference ``inference/engram.py`` + ``Engram``): n-gram hash lookups+added into the residual streams before layers 1 and 14.++Hashing (host side, ``EngramHash``): token ids are first mapped through a compressed vocabulary+(tokens that normalize alike collapse together), then each position is hashed with the+``max_ngram_size - 1`` tokens before it -- a rolling XOR of ``id * multiplier`` over the lookback,+one bucket range (a distinct prime modulus) per (n-gram order, head). Lookback stops at the sequence+start (pad id) and never crosses an image span. Addresses depend+only on the token ids, so the rows can be fetched before the forward starts.++Lookup (``EngramTable`` protocol): the 384M-row fp8 tables live on disk; a backend gathers the+``n_hash_cols`` rows of every token into a device tensor ``[T, n_hash_cols * head_dim]`` in bf16.++Gate (``EngramLayer``): ``wkv`` (fp8 linear) turns the gathered rows into one key per residual+stream plus a shared value; the gate is a normalized dot product of stream and key, signed-sqrt+then sigmoid; ``stream += gate * value``.+"""++from __future__ import annotations++from typing import Protocol++import numpy as np+import torch+from freetoken.layers import BaseOP, LinearReplicated++from .args import DeepseekV41Args+++# ----- hashing ---------------------------------------------------------------------------------+++def _is_prime(n: int) -> bool:+ """Deterministic Miller-Rabin for n < 3.3e24 (the bases below are a proven set), matching+ ``sympy.isprime`` on every 64-bit input."""+ if n < 2:+ return False+ small = (2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41)+ for p in small:+ if n % p == 0:+ return n == p+ d, s = n - 1, 0+ while d % 2 == 0:+ d //= 2+ s += 1+ for a in small:+ x = pow(a, d, n)+ if x in (1, n - 1):+ continue+ for _ in range(s - 1):+ x = pow(x, 2, n)+ if x == n - 1:+ break+ else:+ return False+ return True+++def find_next_prime(start: int, seen: set[int]) -> int:+ """The smallest prime above ``start`` not handed out yet (reference ``find_next_prime``)."""+ candidate = start + 1+ while not _is_prime(candidate) or candidate in seen:+ candidate += 1+ return candidate+++def bucket_primes(args: DeepseekV41Args) -> list[list[list[int]]]:+ """``[layer][ngram order - 2][head]`` bucket moduli: distinct primes drawn in order from+ ``engram_vocab_size - 1`` upward, never reused across layers / orders / heads."""+ primes: list[list[list[int]]] = []+ seen: set[int] = set()+ for _ in args.engram_layer_ids:+ per_ngram = []+ for _ in range(args.engram_max_ngram_size - 1):+ sizes, current = [], args.engram_vocab_size - 1+ for _ in range(args.engram_n_heads):+ current = find_next_prime(current, seen)+ seen.add(current)+ sizes.append(current)+ per_ngram.append(sizes)+ primes.append(per_ngram)+ return primes+++def hash_multipliers(layer_ids: tuple[int, ...], max_ngram_size: int, compressed_vocab_size: int) -> torch.Tensor:+ """``[n_layers, max_ngram_size]`` odd int64 multipliers from a per-layer RNG (reference+ ``compute_hash_multipliers``), bounded so ``id * multiplier`` cannot overflow int64."""+ max_long = np.iinfo(np.int64).max+ bound = max(1, (max_long // compressed_vocab_size) // 2)+ rows = []+ for layer_id in layer_ids:+ gen = np.random.default_rng(10007 * layer_id)+ values = gen.integers(low=0, high=bound, size=(max_ngram_size,), dtype=np.int64)+ rows.append(torch.tensor(values * 2 + 1))+ return torch.stack(rows)+++def build_compressed_token_map(tokenizer) -> tuple[list[int], int]:+ """Token id -> compressed id: tokens that normalize alike (NFKC, accents stripped, lowercased,+ whitespace collapsed) share one id. Reference ``build_compressed_token_map``; the size feeds the+ multipliers, so it is asserted against ``engram_compressed_vocab_size``."""+ from tokenizers import Regex, normalizers++ sentinel = "\ue000"+ normalizer = normalizers.Sequence([+ normalizers.NFKC(),+ normalizers.NFD(),+ normalizers.StripAccents(),+ normalizers.Lowercase(),+ normalizers.Replace(Regex(r"[ \t\r\n]+"), " "),+ normalizers.Replace(Regex(r"^ $"), sentinel),+ normalizers.Strip(),+ normalizers.Replace(sentinel, " "),+ ])+ backend = tokenizer.backend_tokenizer+ key_to_new: dict[str, int] = {}+ lookup = [0] * len(tokenizer)+ for token_id in range(len(tokenizer)):+ text = backend.decode([token_id], skip_special_tokens=False)+ if "\ufffd" in text:+ key = backend.id_to_token(token_id)+ else:+ normalized = normalizer.normalize_str(text)+ key = normalized if normalized else text+ new_id = key_to_new.get(key)+ if new_id is None:+ new_id = len(key_to_new)+ key_to_new[key] = new_id+ lookup[token_id] = new_id+ return lookup, len(key_to_new)+++class EngramHash:+ """Maps token runs to table rows: ``row_ids(tokens) -> [T, n_layers, n_hash_cols]`` int64.++ ``token_map`` is the compressed vocabulary (``build_compressed_token_map``); ``pad_id`` is the+ compressed id of ``engram_pad_id``. Runs on the host (numpy / CPU torch) since the addresses+ are needed before the forward launches.+ """++ def __init__(self, args: DeepseekV41Args, token_map: list[int] | torch.Tensor, compressed_vocab_size: int):+ if compressed_vocab_size != args.engram_compressed_vocab_size:+ raise ValueError(+ f"compressed vocab has {compressed_vocab_size} ids, config says {args.engram_compressed_vocab_size}; "+ "the hash multipliers derive from it, so the tokenizer does not match the checkpoint"+ )+ self.args = args+ self.max_ngram = args.engram_max_ngram_size+ self.token_map = torch.as_tensor(token_map, dtype=torch.int64)+ self.pad_id = int(self.token_map[args.engram_pad_id])+ primes = bucket_primes(args)+ flat = [[p for per_ngram in layer for p in per_ngram] for layer in primes]+ self.primes = torch.tensor(primes, dtype=torch.int64) # [L, ngram-1, heads]+ self.offsets = torch.tensor(np.array([np.cumsum([0, *sizes[:-1]]) for sizes in flat]), dtype=torch.int64) # [L, cols]+ self.multipliers = hash_multipliers(args.engram_layer_ids, self.max_ngram, compressed_vocab_size) # [L, ngram]+ self.rows_per_layer = [int(o[-1] + s[-1]) for o, s in zip(self.offsets.tolist(), [[p for pn in layer for p in pn] for layer in primes])]++ @property+ def n_cols(self) -> int:+ return (self.max_ngram - 1) * self.args.engram_n_heads++ def row_ids(self, tokens: torch.Tensor, context: torch.Tensor | None = None) -> torch.Tensor:+ """Rows for ``tokens [T]`` (raw ids, one contiguous run). ``context`` holds the+ ``max_ngram_size - 1`` raw ids immediately before the run (fewer -> the run starts a+ sequence and the missing lookback is the pad id). Returns ``[T, n_layers, n_hash_cols]``."""+ tokens = tokens.to(torch.int64).cpu()+ ctx_len = self.max_ngram - 1+ if context is None:+ context = tokens.new_empty(0)+ context = context.to(torch.int64).cpu()[-ctx_len:] if ctx_len else tokens.new_empty(0)+ raw = torch.cat([context, tokens])+ # Content-pad ids identify image spans; n-grams stop at them, including across chunks.+ dead = (raw < 0) | (raw >= self.token_map.numel())+ compressed = self.token_map[raw.clamp(0, self.token_map.numel() - 1)]+ n_ctx = context.numel()+ T = tokens.numel()+ positions = torch.arange(n_ctx, n_ctx + T)+ cols = []+ blocked = torch.zeros(T, dtype=torch.bool)+ for shift in range(self.max_ngram):+ src = positions - shift+ blocked |= (src < 0) | dead[src.clamp_min(0)]+ cols.append(torch.where(blocked, torch.full_like(src, self.pad_id), compressed[src.clamp_min(0)]))+ stack = torch.stack(cols, dim=-1) # [T, max_ngram]; column s is the token s places back+ products = stack.unsqueeze(1) * self.multipliers # [T, L, max_ngram]+ rolling, hashes = products[..., 0], []+ for i in range(1, self.max_ngram):+ rolling = torch.bitwise_xor(rolling, products[..., i])+ hashes.append(rolling.unsqueeze(-1) % self.primes[:, i - 1]) # [T, L, heads]+ return torch.cat(hashes, dim=-1) + self.offsets # [T, L, cols]+++# ----- lookup ------------------------------------------------------------------------------------+++class EngramTable(Protocol):+ """Row store of one Engram layer: ``lookup`` returns the dequantized rows of the current+ forward's tokens as ``[T, n_hash_cols * head_dim]`` bf16 (staged by the host before launch)."""++ def lookup(self, num_tokens: int) -> torch.Tensor: ...+++class ZeroEngramTable:+ """Stand-in when no table is attached (dummy weights / unit tests): every lookup reads zeros."""++ def __init__(self, width: int, device: torch.device) -> None:+ self.width = width+ self.device = device++ def lookup(self, num_tokens: int) -> torch.Tensor:+ return torch.zeros(num_tokens, self.width, dtype=torch.bfloat16, device=self.device)+++# ----- the layer -----------------------------------------------------------------------------------+++class EngramLayer(BaseOP):+ """``streams [T, hc, dim] += gate * value`` with key / value from the gathered n-gram rows."""++ def __init__(self, args: DeepseekV41Args, layer_id: int, *, quant_config=None, prefix: str = ""):+ self.layer_id = layer_id+ self.dim = args.dim+ self.hc_mult = args.hc_mult+ self.eps = args.norm_eps+ self.clamp_value = 1e-6+ self.width = args.engram_hash_cols * args.engram_head_dim+ self.wkv = LinearReplicated(self.width, args.dim * (args.hc_mult + 1), has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wkv")+ self.q_weight = torch.empty(args.hc_mult, args.dim, dtype=torch.float32)+ self.k_weight = torch.empty(args.hc_mult, args.dim, dtype=torch.float32)+ self._table: EngramTable | None = None++ def attach_table(self, table: EngramTable) -> None:+ self._table = table++ def forward(self, streams: torch.Tensor, rows: torch.Tensor | None = None, *, image_mask: torch.Tensor | None = None) -> torch.Tensor:+ """``streams [T, hc, dim]``; ``rows`` overrides the table lookup (tests / reference parity)."""+ T = streams.shape[0]+ if rows is None:+ assert self._table is not None, "Engram table was never attached"+ rows = self._table.lookup(T)+ kv = self.wkv.forward(rows.to(streams.dtype))+ key, value = kv.split([self.hc_mult * self.dim, self.dim], dim=-1)+ key = key.float().view(T, self.hc_mult, self.dim)+ weight = self.q_weight * self.k_weight # only ever used as a product+ h = streams.float()+ # normalized per (token, stream) over dim, not jointly over the streams+ rstd = torch.rsqrt(h.square().mean(-1) + self.eps) * torch.rsqrt(key.square().mean(-1) + self.eps)+ dot = (h * weight * key).sum(-1) * rstd * self.dim**-0.5+ gate = torch.sigmoid(torch.copysign(dot.abs().clamp_min(self.clamp_value).sqrt(), dot))+ out = (h + gate.unsqueeze(-1) * value.float().unsqueeze(1)).to(streams.dtype)+ return out if image_mask is None else torch.where(image_mask[:, None, None], streams, out)+++__all__ = [+ "EngramHash",+ "EngramLayer",+ "EngramTable",+ "ZeroEngramTable",+ "build_compressed_token_map",+ "bucket_primes",+ "hash_multipliers",+]diff --git a/python/freetoken/models/deepseek_v41/engram_table.py b/python/freetoken/models/deepseek_v41/engram_table.pynew file mode 100644index 00000000..55450bbd--- /dev/null+++ b/python/freetoken/models/deepseek_v41/engram_table.py@@ -0,0 +1,296 @@+"""Engram tables streamed directly from checkpoint shards.++Each Engram layer owns a ``[384M, 256]`` fp8 table (98 GB) plus a ``[384M, 8]`` ue8m0 scale+tensor (3 GB). The tables stay where the checkpoint put them: ``EngramDiskTable`` maps the shard+tensor as one ``RowStore`` extent and reads the 24 rows a token needs per fill (io_uring, O_DIRECT,+duplicates deduplicated), while the scales are host-resident and gathered next to the values, so+the staged row is exactly the pool row format ``fp8_e8m0_b32`` and the device dequantizes it with+the same kernel the KV pools use.++Protocol (the Qwen3.8 PLE disk table's, ``models/qwen4_exp/ple_disk.py``): the engine wraps every+dispatch in ``EngramHost.forward_host_ctx``. Eager: hash + stage + read before the launch. CUDA-graph+decode: the sampled token lives on the device under overlap scheduling, so the host reads it back,+fills the graph's pinned staging and releases a per-table flag the captured ``lookup`` WAITs on (or,+without stream memops, fills before the replay). The fill runs on a worker thread that is started+BEFORE the graph is launched: with a graph this large (40 layers) the driver's launch call itself+blocks once the GPU sits on the WAIT with the rest of the graph queued behind it, so a fill scheduled+after the launch returns would deadlock. The worker works from an immutable dispatch snapshot plus+the readback event (which the GPU completes on its own), and the dispatch context waits for it after+the launch so its failure surfaces before the step's output is consumed.+"""++from __future__ import annotations++import json+import os+import struct+from concurrent.futures import Future, ThreadPoolExecutor+from contextlib import contextmanager+from dataclasses import dataclass+from typing import Sequence++import torch+from freetoken.core import Batch+from freetoken.kernel.pinned import alloc_pinned_tensor+from freetoken.kvcache.dsv4.v41_row_format import FP8_E8M0_B32+from freetoken.utils import init_logger++from .engram import EngramHash++logger = init_logger(__name__)+++@dataclass(frozen=True)+class _TensorExtent:+ """Where a safetensors tensor's bytes sit in its shard: a contiguous ``[offset, offset + nbytes)``."""++ path: str+ offset: int+ nbytes: int+ shape: tuple[int, ...]+ dtype: str # the safetensors dtype string (e.g. "F8_E4M3")+++def _tensor_extent(folder: str, name: str) -> _TensorExtent:+ """Locate ``name`` in the checkpoint directory without reading it (the table is streamed in place)."""+ from freetoken.models.loader import safetensors_weight_map++ path = os.path.join(folder, safetensors_weight_map(folder)[name])+ with open(path, "rb") as fh:+ n = struct.unpack("<Q", fh.read(8))[0]+ header = json.loads(fh.read(n))+ meta = header[name]+ start, end = meta["data_offsets"]+ return _TensorExtent(path, 8 + n + start, end - start, tuple(meta["shape"]), meta["dtype"])+++@dataclass(frozen=True)+class EngramRowSource:+ """The fp8 table tensor in its shard (one extent) and its host-resident scales."""++ path: str+ base: int+ num_rows: int+ head_dim: int+ scales: torch.Tensor # [num_rows, head_dim // 32] uint8 (e8m0 codes), CPU+++def engram_row_source(folder: str, layer_id: int) -> EngramRowSource:+ """Locate a layer's table in whatever safetensors files the checkpoint directory holds: the HF+ shards (through the index) or the shards an FTW conversion copied next to it (through their headers)."""+ from safetensors import safe_open++ weight = _tensor_extent(folder, f"layers.{layer_id}.engram.embed.weight")+ if weight.dtype != "F8_E4M3":+ raise ValueError(f"engram table of layer {layer_id} is {weight.dtype}, expected F8_E4M3")+ rows, head_dim = weight.shape+ scale = _tensor_extent(folder, f"layers.{layer_id}.engram.embed.scale")+ with safe_open(scale.path, framework="pt", device="cpu") as f:+ scales = f.get_tensor(f"layers.{layer_id}.engram.embed.scale").view(torch.uint8).contiguous()+ if tuple(scales.shape) != (rows, head_dim // 32):+ raise ValueError(f"engram scales of layer {layer_id} are {tuple(scales.shape)}, expected {(rows, head_dim // 32)}")+ return EngramRowSource(weight.path, weight.offset, rows, head_dim, scales)+++class EngramDiskTable:+ """One layer's table: stages ``[T, n_cols]`` rows into pinned memory on the host, dequantizes on+ the device. Implements the model's ``EngramTable`` protocol (``lookup``)."""++ def __init__(+ self, source: EngramRowSource, n_cols: int, device: torch.device, *, max_graph_rows: int, max_extend_tokens: int,+ use_io_uring: bool = True,+ ) -> None:+ from freetoken.kernel.row_store import RowStore++ self.head_dim = source.head_dim+ self.n_cols = n_cols+ self.num_rows = source.num_rows+ self.row_bytes = FP8_E8M0_B32.row_bytes(source.head_dim) # values | scales+ self.scales = source.scales+ self.device = device+ self.store = RowStore(+ paths=[source.path], extent_file=[0], extent_base=[source.base], rows_per_extent=source.num_rows,+ row_bytes=source.head_dim, row_stride=source.head_dim, use_io_uring=use_io_uring,+ )+ # the graph staging is allocated up front (pinned alloc inside capture is illegal) and+ # outlives any one capture so a re-capture keeps the same pointers; the eager staging+ # grows to the largest prefill chunk seen (never inside a capture)+ self.max_graph_rows = max_graph_rows+ self._graph_pinned = alloc_pinned_tensor(max_graph_rows * n_cols * self.row_bytes, dtype=torch.uint8)+ self._graph_pinned.zero_()+ self._graph_dev = torch.empty(max_graph_rows * n_cols * self.row_bytes, dtype=torch.uint8, device=device)+ # completes once the last graph launch that read ``_graph_pinned`` has run (recorded by the+ # host coordinator after the launch: a record inside the capture would not be observable)+ self.graph_consumed = torch.cuda.Event()+ # the overlap scheduler stages batch k+1 while batch k's lookup copy may still be queued: the+ # eager staging alternates two pinned buffers, and each is rewritten only once the H2D copy+ # that last read it has run (``_eager_read[i]``, recorded right after that copy)+ self._eager_pinned: list[torch.Tensor] = []+ self._eager_read = [torch.cuda.Event(), torch.cuda.Event()]+ self._eager_slot = 0+ self._eager_dev = None+ self._grow_eager(max_extend_tokens)+ self.flag = alloc_pinned_tensor(1, dtype=torch.int64)+ self.flag.zero_()+ self.wait_sync = False # set by the host coordinator after its probe++ def _grow_eager(self, num_tokens: int) -> None:+ nbytes = num_tokens * self.n_cols * self.row_bytes+ if self._eager_pinned and self._eager_pinned[0].numel() >= nbytes:+ return+ for event in self._eager_read:+ event.synchronize()+ self._eager_pinned = [alloc_pinned_tensor(nbytes, dtype=torch.uint8) for _ in self._eager_read]+ for buf in self._eager_pinned:+ buf.zero_() # a warmup prefill stages nothing and reads whatever sits here+ self._eager_dev = torch.empty(nbytes, dtype=torch.uint8, device=self.device)++ # ----- host -----+ def stage(self, row_ids: torch.Tensor, *, graph: bool) -> None:+ """Queue ``row_ids [T, n_cols]`` (int64, host): values through the store, scales from the resident tensor."""+ n = row_ids.numel()+ if graph:+ self.graph_consumed.synchronize()+ pinned = self._graph_pinned+ assert n <= self.max_graph_rows * self.n_cols, f"graph staging holds {self.max_graph_rows} rows, batch has {n // self.n_cols}"+ else:+ self._grow_eager(n // self.n_cols)+ self._eager_slot ^= 1+ self._eager_read[self._eager_slot].synchronize()+ pinned = self._eager_pinned[self._eager_slot]+ ids = row_ids.reshape(-1).to(torch.int64).contiguous()+ rows = pinned[: n * self.row_bytes].view(n, self.row_bytes)+ self.store.stage_rows(ids.data_ptr(), n, rows.data_ptr(), self.row_bytes)+ rows[:, self.head_dim :].copy_(self.scales.index_select(0, ids))++ def flush(self, *, signal: bool) -> None:+ self.store.flush(self.flag.data_ptr() if signal else 0)++ # ----- device (EngramTable protocol) -----+ def lookup(self, num_tokens: int) -> torch.Tensor:+ from freetoken.kernel.row_store import wait_reset+ from freetoken.kernel.triton.dsv41.pack import unpack_rows++ stream = torch.cuda.current_stream(self.device)+ capturing = torch.cuda.is_current_stream_capturing()+ if capturing and self.wait_sync:+ wait_reset(stream, self.flag)+ pinned, dev = (self._graph_pinned, self._graph_dev) if capturing else (self._eager_pinned[self._eager_slot], self._eager_dev)+ n = num_tokens * self.n_cols+ nbytes = n * self.row_bytes+ dev[:nbytes].copy_(pinned[:nbytes], non_blocking=True)+ if not capturing:+ self._eager_read[self._eager_slot].record(stream)+ values = unpack_rows(dev[:nbytes].view(n, self.row_bytes), FP8_E8M0_B32, self.head_dim)+ return values.view(num_tokens, self.n_cols * self.head_dim)+++def _context(ids: torch.Tensor, position: int, width: int) -> torch.Tensor:+ """The up-to-``width`` raw ids before ``position`` (fewer at the sequence start)."""+ return ids[max(0, position - width) : position]+++class EngramHost:+ """The per-model coordinator: hashes each batch's token runs and fills every layer's table.++ The token range a prefill hashes is the attention metadata's encoder segments -- the one+ execution plan the scheduler, the attention layers and the model already agreed on. A graph+ decode's fill runs on a worker thread from an immutable dispatch snapshot (position + the+ ``max_ngram - 1`` preceding ids per row) plus the readback of the sampled token; the request+ objects themselves advance on the main thread right after dispatch and are never read late.+ """++ def __init__(self, hash: EngramHash, tables: Sequence[EngramDiskTable], device: torch.device, *, sync_mode: str = "auto") -> None:+ from freetoken.kernel.row_store import probe_wait_sync++ self.hash = hash+ self.tables = list(tables)+ self.device = device+ self.wait_sync = probe_wait_sync(sync_mode, device)+ for t in self.tables:+ t.wait_sync = self.wait_sync+ self._token_readback = alloc_pinned_tensor(min(t.max_graph_rows for t in self.tables), dtype=torch.int32)+ self._readback_event = torch.cuda.Event()+ self._worker = ThreadPoolExecutor(max_workers=1, thread_name_prefix="engram-fill")+ self._pending: Future | None = None+ logger.info_rank0(f"Engram disk backend: {self.tables[0].store.io_backend()}, {'wait-sync' if self.wait_sync else 'launch-gating'}")++ def close(self) -> None:+ self._worker.shutdown(wait=True)++ def __del__(self) -> None:+ worker = getattr(self, "_worker", None)+ if worker is not None:+ worker.shutdown(wait=False)++ def fill(self, runs: Sequence[tuple[torch.Tensor, torch.Tensor]], *, graph: bool) -> None:+ """``runs``: per request ``(context ids, token ids)`` in batch order; stages every layer's rows."""+ ids = torch.cat([self.hash.row_ids(tokens, ctx) for ctx, tokens in runs]) if runs else torch.empty(0, len(self.tables), self.hash.n_cols, dtype=torch.int64)+ for l, table in enumerate(self.tables):+ table.stage(ids[:, l], graph=graph)+ table.flush(signal=graph and self.wait_sync)++ @torch.inference_mode()+ def _deferred_fill(self, bs: int, histories: list[tuple[int, ...]]) -> None:+ """Worker side of a graph decode: wait for the sampled tokens, hash them against the dispatch+ snapshot, stage, and release the tables' flags. Reads nothing that the main thread mutates.+ Inference mode is thread-local; staging buffers were created under it by the server."""+ try:+ self._readback_event.synchronize()+ tokens = self._token_readback[:bs].to(torch.int64)+ runs = [(torch.tensor(hist, dtype=torch.int64), tokens[i : i + 1]) for i, hist in enumerate(histories)]+ self.fill(runs, graph=True)+ except BaseException:+ from freetoken.kernel.row_store import signal++ for t in self.tables: # unblock the stream before surfacing; the step's output is discarded+ signal(t.flag)+ raise++ def _await_pending(self) -> None:+ """Surface the deferred fill's outcome (it depends only on GPU work already enqueued, so this+ cannot wait on the host)."""+ if self._pending is not None:+ pending, self._pending = self._pending, None+ pending.result()++ def host_fill_batch(self, batch: Batch, use_graph: bool) -> None:+ """Stage this batch's rows: eagerly, or -- a graph decode under flag-sync -- on the worker+ thread once the sampled tokens have been read back."""+ self._await_pending()+ width = self.hash.max_ngram - 1+ if batch.is_decode:+ reqs = list(batch.padded_reqs)+ # the dispatch snapshot: each row's position context, taken before anything advances+ histories = [tuple(_context(r.input_ids, r.device_len - 1, width).tolist()) for r in reqs]+ if use_graph and self.wait_sync:+ bs = batch.padded_size+ self._token_readback[:bs].copy_(batch.input_ids, non_blocking=True)+ self._readback_event.record(torch.cuda.current_stream(self.device))+ self._pending = self._worker.submit(self._deferred_fill, bs, histories)+ return+ tokens = batch.input_ids.to("cpu").to(torch.int64)+ runs = [(torch.tensor(hist, dtype=torch.int64), tokens[i : i + 1]) for i, hist in enumerate(histories)]+ self.fill(runs, graph=use_graph)+ return+ # prefill: the encoder segments are the token range every consumer of this forward executes+ segments = batch.attn_metadata.segments+ assert segments is not None and len(segments) == len(batch.reqs)+ runs = [(_context(r.input_ids, seg.start_pos, width), r.input_ids[seg.start_pos : seg.end]) for r, seg in zip(batch.reqs, segments)]+ self.fill(runs, graph=False)++ @contextmanager+ def forward_host_ctx(self, batch: Batch, use_graph: bool):+ self.host_fill_batch(batch, use_graph)+ yield+ # the launch has been enqueued: wait for the deferred fill here so a failure surfaces before+ # this step's output is consumed (the fill only waits on the GPU readback, never on the host)+ self._await_pending()+ if use_graph:+ # only after this step's fill: the next fill waits on this launch, not the fill on it+ stream = torch.cuda.current_stream(self.device)+ for t in self.tables:+ t.graph_consumed.record(stream)+++__all__ = ["EngramDiskTable", "EngramHost", "EngramRowSource", "engram_row_source"]diff --git a/python/freetoken/models/deepseek_v41/indexer.py b/python/freetoken/models/deepseek_v41/indexer.pynew file mode 100644index 00000000..4ab3da2b--- /dev/null+++ b/python/freetoken/models/deepseek_v41/indexer.py@@ -0,0 +1,127 @@+"""DSV41 Lightning Indexer (reference ``Indexer``): scores the compressed positions and keeps the+``index_topk`` best per query.++A Full-mode layer owns the index keys: ``k = k_norm(wk(latent))`` from its compressor's unrotated+latent, RoPE'd at the group position, fp4-quantized (ue8m0 per 32) into the kv source's index pool.+Every Full / Reindex layer scores its own fp4-baked queries ``wq_b(qr)`` against those keys,+``sum_h relu(q_h . k) * w_h`` with ``w = weights_proj(x) * softmax_scale * n_heads**-0.5``.++The Hierarchical Sparse Indexer (sec. 2.3.2): the candidate-source layer (the decoder's first Full+layer) additionally keeps the ``candidate_topk_blocks`` best ``candidate_block_size``-position blocks+as a candidate pool; the decoder's Reindex layers score only that pool. Logits are produced in query+blocks bounded by ``LOGITS_BUDGET_BYTES`` so a long-context prefill never materializes ``[T, 1M]``.+"""++from __future__ import annotations++import torch+from freetoken.core import get_global_ctx+from freetoken.kernel.triton.dsv4.fp8_linear import fp4_act_quant_inplace+from freetoken.layers import BaseOP, LinearReplicated, RMSNorm++from .args import DeepseekV41Args, LayerRole, Mode+from .rope import apply_rotary_emb, apply_rotary_emb_decode++LOGITS_BUDGET_BYTES = 512 << 20+++class Indexer(BaseOP):+ def __init__(self, args: DeepseekV41Args, role: LayerRole, *, quant_config=None, prefix: str = ""):+ self.role = role+ self.n_heads = args.index_n_heads+ self.head_dim = args.index_head_dim+ self.rope_dim = args.rope_head_dim+ self.topk = args.index_topk+ self.weights_scale = args.index_softmax_scale * args.index_n_heads**-0.5+ self.candidate_topk_blocks = args.candidate_topk_blocks+ self.candidate_block_size = args.candidate_block_size+ self.wq_b = LinearReplicated(args.q_lora_rank, self.n_heads * self.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wq_b")+ self.weights_proj = LinearReplicated(args.dim, self.n_heads, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.weights_proj")+ if role.mode is Mode.FULL:+ self.wk = LinearReplicated(args.head_dim, self.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wk")+ self.k_norm = RMSNorm(self.head_dim, args.norm_eps)++ @property+ def attn(self):+ return get_global_ctx().attn_backend++ # ----- keys (Full mode) ----------------------------------------------------------------+ def index_keys(self, latent: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:+ """``[G, index_head_dim]`` keys from the unrotated latents, rotated at their group positions. The+ fp4 quantization is the pool's packing."""+ k = self.k_norm.forward(self.wk.forward(latent))+ apply_rotary_emb(k[..., -self.rope_dim :], freqs)+ return k++ # ----- queries ---------------------------------------------------------------------------+ def _queries(self, x: torch.Tensor, qr: torch.Tensor, rotate) -> tuple[torch.Tensor, torch.Tensor]:+ q = self.wq_b.forward(qr).unflatten(-1, (self.n_heads, self.head_dim))+ rotate(q[..., -self.rope_dim :])+ fp4_act_quant_inplace(q, 32)+ weights = self.weights_proj.forward(x) * self.weights_scale+ return q, weights++ # ----- selection -------------------------------------------------------------------------+ def select_prefill(+ self, x: torch.Tensor, qr: torch.Tensor, freqs: torch.Tensor, *, start_pos: int, ratio: int, locs: torch.Tensor,+ candidates: torch.Tensor | None,+ ) -> tuple[torch.Tensor, torch.Tensor | None]:+ """Per-query picks for one request's ``n`` new tokens at ``[start_pos, start_pos + n)``.++ ``locs [1, W]`` are the request's live full locs for the positions its queries may see (the+ source's row of compressed position ``t`` is ``locs[t * ratio] // ratio``); ``candidates [1, n,+ NC]`` restricts a Reindex layer to the pool. Returns ``(rows [1, n, topk] int32 global main rows,+ -1 tail; candidates [1, n, NC] | None -- the pool this layer built as candidate source)``.+ """+ n = x.shape[0]+ T = locs.shape[1] // ratio+ if T == 0: # a prompt shorter than one compression block: no source row exists yet+ rows = torch.full((1, n, self.topk), -1, dtype=torch.int32, device=x.device)+ pool = None+ if self.role.is_candidate_source:+ pool = torch.full((1, n, self.candidate_topk_blocks * self.candidate_block_size), -1, dtype=torch.int32, device=x.device)+ return rows, pool+ q, w = self._queries(x, qr, lambda t: apply_rotary_emb(t, freqs))+ live = ((start_pos + torch.arange(1, n + 1, device=x.device)) // ratio).to(torch.int32).view(1, n)+ width = candidates.shape[-1] if candidates is not None else T+ qb = max(1, LOGITS_BUDGET_BYTES // max(1, width * 4))+ picks, pools = [], []+ for s0 in range(0, n, qb):+ s1 = min(n, s0 + qb)+ cand = candidates[:, s0:s1] if candidates is not None else None+ logits = self.attn.indexer_logits(q[None, s0:s1], w[None, s0:s1], self.role.kv_source, locs, ratio, live[:, s0:s1], T=T, candidates=cand)+ if cand is not None:+ pos = self.attn.select_topk_in_candidates(logits, cand, self.topk)+ else:+ pos = self.attn.select_topk(logits, live[:, s0:s1], self.topk)+ if self.role.is_candidate_source:+ pools.append(self.attn.select_candidate_blocks(logits, live[:, s0:s1], self.candidate_topk_blocks, self.candidate_block_size))+ picks.append(self.attn.positions_to_rows(pos, locs, ratio))+ rows = torch.cat(picks, dim=1)+ pool = torch.cat(pools, dim=1) if pools else None+ return rows.to(torch.int32), pool++ def select_decode(+ self, x: torch.Tensor, qr: torch.Tensor, freqs: torch.Tensor, *, pos: torch.Tensor, ratio: int, locs: torch.Tensor,+ T: int, candidates: torch.Tensor | None,+ ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]:+ """Batched one-token selection. ``locs [B, W]`` is the decode snapshot (graph-static width),+ ``T`` the staged compressed width, ``pos [B]``. Returns ``(rows [B, 1, topk] int32, counts [B, 1]+ int32 valid picks, candidates [B, 1, NC] | None)``."""+ B = x.shape[0]+ q, w = self._queries(x, qr, lambda t: apply_rotary_emb_decode(t.unsqueeze(1), freqs))+ live = ((pos + 1) // ratio).to(torch.int32).view(B, 1)+ logits = self.attn.indexer_logits(q.view(B, 1, self.n_heads, self.head_dim), w.view(B, 1, -1), self.role.kv_source, locs, ratio, live, T=T, candidates=candidates)+ pool = None+ if candidates is not None:+ picks = self.attn.select_topk_in_candidates(logits, candidates, self.topk)+ else:+ picks = self.attn.select_topk(logits, live, self.topk)+ if self.role.is_candidate_source:+ pool = self.attn.select_candidate_blocks(logits, live, self.candidate_topk_blocks, self.candidate_block_size)+ rows = self.attn.positions_to_rows(picks, locs, ratio)+ counts = (picks >= 0).sum(dim=-1, dtype=torch.int32) # valid picks form a prefix+ return rows.to(torch.int32), counts, pool+++__all__ = ["Indexer", "LOGITS_BUDGET_BYTES"]diff --git a/python/freetoken/models/deepseek_v41/model.py b/python/freetoken/models/deepseek_v41/model.pynew file mode 100644index 00000000..8112022f--- /dev/null+++ b/python/freetoken/models/deepseek_v41/model.py@@ -0,0 +1,289 @@+"""DeepSeek-V4.1-Flash model (engine-native port of ``inference/model.py``).++ embed -> hc_mult residual streams -> [Engram at 1, 14] -> 40 blocks (single-pass mHC around+ DSV41 attention and MoE) -> collapse with the last block's pre gates -> norm -> head++Single-pass mHC (``layers/mhc.py``): each sublayer boundary applies the PREVIOUS sublayer's post /+comb to the streams, predicts this sublayer's (pre, post, comb) from the updated streams, and mixes+the sublayer input with the pre gates the previous sublayer predicted. The chain state+``(y, residual, post, comb, pre)`` threads through the layer loop; layer 0's attention starts it+with a one-hot pre on stream 0 and no pending post.++Causal Encoder-Decoder: the decoder's kv source (layer 20) compresses ITS INPUT -- the final encoder+hidden state -- into the global KV every decoder layer reads. Under Decoder SWA Bounded Replay the+decoder layers run on each request's last ``window_size`` prompt tokens only (their sliding window+truncated at the start of that window, see the backend; a prefix hit stops before it, so those tokens+are always prefilled); layer 20 still publishes global KV for every prompt token.+``--swa-decoder-replay exact`` runs the decoder on every token (the reference numerics).++KV addressing is the attention backend's; pool buffers are read off the live pool per access, so a+runtime cache rebuild needs no unbind. Decode is batched and CUDA-graph safe (the DSV4 precedent).+"""++from __future__ import annotations++from typing import TYPE_CHECKING, List, NamedTuple++import torch+from freetoken.attention.dsv41_sparse import DSV41AttnMetadata, PrefillSegment+from freetoken.core import get_global_ctx+from freetoken.layers import BaseOP, OPList, ParallelLMHead, RMSNorm, VocabParallelEmbedding+from freetoken.layers.mhc import mhc_fused_post_pre_single_pass, mhc_mix_input, mhc_post+from freetoken.models.blocks import BaseLLMModel, embed_input_ids++from .args import DeepseekV41Args+from .attention import DSV41Attention, DecodeStepContext+from .engram import EngramLayer+from .moe import MoE++if TYPE_CHECKING:+ from freetoken.models.config import ModelConfig++POST_MULT = 2.0 # post gates are 2 * sigmoid (reference hc_split_sinkhorn)+++class Streams(NamedTuple):+ """The single-pass mHC chain state between sublayers: the last sublayer output ``y`` with its+ not-yet-applied ``post`` / ``comb``, the residual streams ``[T, hc, dim]`` they apply to, and+ the ``pre`` gates ``[T, hc]`` the next sublayer mixes its input with."""++ y: torch.Tensor | None+ residual: torch.Tensor+ post: torch.Tensor | None+ comb: torch.Tensor | None+ pre: torch.Tensor++ def materialize(self) -> torch.Tensor:+ """The streams with the pending post applied."""+ if self.post is None:+ return self.residual+ return mhc_post(self.y, self.residual, self.post, self.comb)+++class Block(BaseOP):+ def __init__(self, config: ModelConfig, layer_id: int, *, prefix: str = ""):+ args: DeepseekV41Args = config.dsv41_args+ self.layer_id = layer_id+ self.role = args.roles[layer_id]+ self.norm_eps = args.norm_eps+ self.hc_eps = args.hc_eps+ self.sinkhorn = args.hc_sinkhorn_iters+ hc, dim = args.hc_mult, args.dim+ mix = (2 + hc) * hc+ self.attn = DSV41Attention(args, self.role, quant_config=config.quant, prefix=f"{prefix}.attn")+ self.ffn = MoE(config, layer_id, prefix=f"{prefix}.ffn")+ self.attn_norm = RMSNorm(dim, args.norm_eps)+ self.ffn_norm = RMSNorm(dim, args.norm_eps)+ self.engram = EngramLayer(args, layer_id, quant_config=config.quant, prefix=f"{prefix}.engram") if layer_id in args.engram_layer_ids else None+ self.hc_attn_fn = torch.empty(mix, hc * dim, dtype=torch.float32)+ self.hc_ffn_fn = torch.empty(mix, hc * dim, dtype=torch.float32)+ self.hc_attn_base = torch.empty(mix, dtype=torch.float32)+ self.hc_ffn_base = torch.empty(mix, dtype=torch.float32)+ self.hc_attn_scale = torch.empty(3, dtype=torch.float32)+ self.hc_ffn_scale = torch.empty(3, dtype=torch.float32)++ def _boundary(self, s: Streams, fn, scale, base):+ residual, post, comb, pre_next, x = mhc_fused_post_pre_single_pass(+ s.y if s.y is not None else s.residual[:, 0], s.residual, s.post, s.comb, s.pre, fn, scale, base,+ self.norm_eps, self.hc_eps, POST_MULT, self.sinkhorn,+ )+ return residual, post, comb, pre_next, x++ def attn_input(self, s: Streams, *, image_mask: torch.Tensor | None = None):+ """Apply the pending post, predict the attention gates, mix and norm the attention input.+ Returns ``(residual, post, comb, pre_next, x)``."""+ if self.engram is not None:+ s = Streams(None, self.engram.forward(s.materialize(), image_mask=image_mask), None, None, s.pre)+ residual, post, comb, pre_next, x = self._boundary(s, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)+ return residual, post, comb, pre_next, self.attn_norm.forward(x)++ def ffn_step(self, y: torch.Tensor, residual, post, comb, pre, *, image_mask: torch.Tensor | None = None) -> Streams:+ """The FFN sublayer after the attention output ``y``: boundary, norm, MoE; returns the chain state."""+ residual, post, comb, pre_next, x = self._boundary(Streams(y, residual, post, comb, pre), self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)+ return Streams(self.ffn.forward(self.ffn_norm.forward(x), image_mask=image_mask), residual, post, comb, pre_next)++ def forward_prefill(self, s: Streams, segments: List[PrefillSegment], positions: torch.Tensor, image_mask: torch.Tensor | None = None) -> Streams:+ residual, post, comb, pre_next, x = self.attn_input(s, image_mask=image_mask)+ if self.attn.compressor is not None:+ self.attn.publish_prefill(x, segments)+ y = self.attn.forward_prefill(x, segments, positions)+ return self.ffn_step(y, residual, post, comb, pre_next, image_mask=image_mask)++ def forward_prefill_narrowed(+ self, s: Streams, segments: List[PrefillSegment], rows: torch.Tensor, query_segments: List[PrefillSegment],+ query_positions: torch.Tensor, image_mask: torch.Tensor | None = None, query_image_mask: torch.Tensor | None = None,+ ) -> Streams | None:+ """``forward_prefill`` with a row cut after the global KV is published: the attention input and+ ``publish_prefill`` cover every token of ``segments``, the attention and the FFN only ``rows``+ (tiled by ``query_segments``) -- the CED decoder's kv source under bounded replay. None when no+ query segment is left (every chunk in the batch ends before its prompt's last window)."""+ residual, post, comb, pre_next, x = self.attn_input(s, image_mask=image_mask)+ if self.attn.compressor is not None:+ self.attn.publish_prefill(x, segments)+ if not query_segments:+ return None+ y = self.attn.forward_prefill(x[rows], query_segments, query_positions)+ return self.ffn_step(y, residual[rows], post[rows], comb[rows], pre_next[rows], image_mask=query_image_mask)++ def forward_decode(self, s: Streams, pos: torch.Tensor, rows: torch.Tensor, dctx, cmp_stage_cap: int) -> Streams:+ residual, post, comb, pre_next, x = self.attn_input(s)+ y = self.attn.forward_decode(x, pos, rows, dctx, cmp_stage_cap)+ return self.ffn_step(y, residual, post, comb, pre_next)+++class Transformer(BaseOP):+ def __init__(self, config: ModelConfig, *, prefix: str = "model"):+ args: DeepseekV41Args = config.dsv41_args+ self.args = args+ self.embed = VocabParallelEmbedding(num_embeddings=args.vocab_size, embedding_dim=args.dim)+ self.layers = OPList([Block(config, i, prefix=f"{prefix}.layers.{i}") for i in range(args.n_layers)])+ self.norm = RMSNorm(args.dim, args.norm_eps)++ def bind(self, device: torch.device) -> None:+ tables: dict = {} # one RoPE table per distinct parameter set, shared by the layers using it+ for block in self.layers.op_list:+ block.attn.bind(device, tables)++ def _entry(self, input_ids: torch.Tensor) -> Streams:+ h = embed_input_ids(self.embed, input_ids, get_global_ctx().batch)+ residual = h.unsqueeze(1).expand(-1, self.args.hc_mult, -1).contiguous()+ pre = torch.zeros(h.shape[0], self.args.hc_mult, dtype=torch.float32, device=h.device)+ pre[:, 0] = 1.0+ return Streams(None, residual, None, None, pre)++ def _exit(self, s: Streams) -> torch.Tensor:+ return self.norm.forward(mhc_mix_input(s.materialize(), s.pre))++ def prefill(self, input_ids: torch.Tensor, positions: torch.Tensor, md: DSV41AttnMetadata) -> torch.Tensor:+ """Ragged prefill over the flat new-token stream; returns the hidden states the head reads (the+ decoder stream under bounded replay, where ``md.last_indices`` also point, plus the placeholder+ row ``md.decoder_pad`` asks for)."""+ s = self._entry(input_ids)+ dtype = s.residual.dtype+ image_mask = input_ids >= self.args.vocab_size if get_global_ctx().batch.mm_embeds is not None else None+ segments, dec_segments, rows = md.segments, md.decoder_segments, md.decoder_rows+ decoder_start = self.args.decoder_start_layer+ if rows is not None:+ dec_positions = positions[rows]+ dec_image_mask = None if image_mask is None else image_mask[rows]+ for block in self.layers.op_list:+ if rows is None or block.layer_id < decoder_start:+ s = block.forward_prefill(s, segments, positions, image_mask)+ elif block.layer_id == decoder_start:+ s = block.forward_prefill_narrowed(s, segments, rows, dec_segments, dec_positions, image_mask, dec_image_mask)+ if s is None:+ break+ else:+ s = block.forward_prefill(s, dec_segments, dec_positions, dec_image_mask)+ pad = [torch.zeros(1, self.args.dim, dtype=dtype, device=input_ids.device)] if md.decoder_pad else []+ return torch.cat([self._exit(s)] + pad) if s is not None else pad[0]++ def decode(self, input_ids: torch.Tensor, pos: torch.Tensor, md: DSV41AttnMetadata, cmp_stage_cap: int) -> torch.Tensor:+ B = input_ids.shape[0]+ rows = torch.arange(B, device=input_ids.device)+ s = self._entry(input_ids)+ # the layer-invariant step context, resolved once (a capture records it once): ring slots and+ # window candidates of the shared pool -- and of the private rings when bounded replay keeps+ # the decoder's window KV per request -- plus each RoPE table at the step's positions+ dctx = DecodeStepContext.build([b.attn for b in self.layers.op_list], md, pos, rows, self.args.window_size, self.args.index_topk)+ for block in self.layers.op_list:+ s = block.forward_decode(s, pos, rows, dctx, cmp_stage_cap)+ return self._exit(s)+++class DeepseekV41ForCausalLM(BaseLLMModel):+ def __init__(self, config: ModelConfig):+ self._config = config+ self._args: DeepseekV41Args = config.dsv41_args+ self.model = Transformer(config)+ self.head = ParallelLMHead(num_embeddings=self._args.vocab_size, embedding_dim=self._args.dim, quant_config=config.quant, prefix="head")+ self._bound = False+ if config.is_multimodal:+ from .vision import DeepseekV41Vision++ self.visual = DeepseekV41Vision(config.vision_config, config.hidden_size,+ num_layers=self._args.n_layers, num_experts=self._args.n_routed_experts)++ def encode(self, item):+ return self.visual.forward(item)++ def place_encoder_weights(self, mode: str) -> None:+ self.visual.place_weights(mode)++ def _ensure_bound(self) -> None:+ if not self._bound:+ self.model.bind(get_global_ctx().kv_cache.device)+ if self._config.is_multimodal:+ for i, block in enumerate(self.model.layers.op_list):+ block.ffn.gate._bias_vl = self.visual.routing_bias[i]+ self._bound = True++ def mark_for_rebind(self) -> None:+ """A runtime pool rebuild changes nothing the model caches (rope tables are pool-independent),+ but the engine asks; re-bind is cheap and keeps the contract with the DSV4 sibling."""+ self._bound = False++ def engram_layers(self) -> List[EngramLayer]:+ return [b.engram for b in self.model.layers.op_list if b.engram is not None]++ def load_host_tables(self, engine_config) -> int:+ """Attach the Engram tables (streamed from the checkpoint shards; zeros for dummy weights)+ and take over ``forward_host_ctx`` so every dispatch stages its rows. Returns the pinned+ host bytes the engine reserves from its pin budget."""+ import os++ layers = self.engram_layers()+ if not layers:+ return 0+ device = torch.device("cuda", torch.cuda.current_device())+ if getattr(engine_config, "use_dummy_weight", False):+ from .engram import ZeroEngramTable++ for layer in layers:+ layer.attach_table(ZeroEngramTable(layer.width, device))+ return 0+ from freetoken.utils import download_hf_weight+ from transformers import AutoTokenizer++ from .engram import EngramHash, build_compressed_token_map+ from .engram_table import EngramDiskTable, EngramHost, engram_row_source++ folder = download_hf_weight(engine_config.model_path)+ token_map, compressed_vocab = build_compressed_token_map(AutoTokenizer.from_pretrained(folder, trust_remote_code=True))+ hash = EngramHash(self._args, token_map, compressed_vocab)+ graph_rows = max(engine_config.max_running_req, engine_config.cuda_graph_max_bs or 0, 1)+ tables = [+ EngramDiskTable(+ engram_row_source(folder, layer.layer_id), hash.n_cols, device,+ max_graph_rows=graph_rows,+ max_extend_tokens=engine_config.max_extend_tokens,+ use_io_uring=os.getenv("FREETOKEN_ENGRAM_IO_URING", "1") != "0",+ )+ for layer in layers+ ]+ for layer, table in zip(layers, tables):+ layer.attach_table(table)+ self._engram_host = EngramHost(hash, tables, device, sync_mode=os.getenv("FREETOKEN_ENGRAM_SYNC", "auto"))+ self.forward_host_ctx = self._engram_host.forward_host_ctx+ return sum(t._graph_pinned.numel() + sum(b.numel() for b in t._eager_pinned) for t in tables)++ def forward(self) -> torch.Tensor:+ self._ensure_bound()+ batch = get_global_ctx().batch+ md = batch.attn_metadata+ assert isinstance(md, DSV41AttnMetadata)+ if batch.is_prefill:+ hidden = self.model.prefill(batch.input_ids, batch.positions.long(), md)+ else:+ input_ids = batch.input_ids+ B = batch.padded_size+ pos = batch.positions.long().view(-1)[:B]+ if torch.cuda.is_current_stream_capturing():+ cmp_stage_cap = md.stage_width - 1+ else:+ cmp_stage_cap = int(pos.max().item())+ hidden = self.model.decode(input_ids.view(B), pos, md, cmp_stage_cap)+ return self.head.forward(hidden)+++__all__ = ["DeepseekV41ForCausalLM", "Transformer", "Block", "Streams"]diff --git a/python/freetoken/models/deepseek_v41/moe.py b/python/freetoken/models/deepseek_v41/moe.pynew file mode 100644index 00000000..91c62ecb--- /dev/null+++ b/python/freetoken/models/deepseek_v41/moe.py@@ -0,0 +1,90 @@+"""DeepSeek-V4.1 MoE block: sqrtsoftplus router with a selection-only correction bias, one shared+clamped-SwiGLU expert, and the routed MXFP4 experts on the shared offload cache.++Router (reference ``Gate``): ``scores = sqrt(softplus(x @ W^T))`` in fp32; the bias picks the experts,+the unbiased scores weight them (renormalized, ``+1e-20`` as in training), times ``route_scale``.+Image-span tokens select experts using the checkpoint's separate ``bias_vl``.+"""++from __future__ import annotations++from typing import TYPE_CHECKING, Tuple++import torch+from freetoken.kernel.triton.dsv4.bf16_linear import bf16_linear_fp32+from freetoken.kernel.triton.dsv4.router import router_select+from freetoken.kernel.triton.dsv4.swiglu import fused_swiglu+from freetoken.layers import BaseOP, LinearReplicated, make_moe_layer++from .args import DeepseekV41Args++if TYPE_CHECKING:+ from freetoken.models.config import ModelConfig++TopK = Tuple[torch.Tensor, torch.Tensor]+++class Router(BaseOP):+ def __init__(self, args: DeepseekV41Args):+ self.topk = args.n_activated_experts+ self.score_func = args.score_func+ self.norm_topk_prob = args.norm_topk_prob+ self.route_scale = args.route_scale+ self.weight = torch.empty(args.n_routed_experts, args.dim, dtype=torch.bfloat16)+ self.bias = torch.empty(args.n_routed_experts, dtype=torch.float32)+ self._bias_vl = None # bound to the visual stack's per-layer routing bias after loading++ def forward(self, x: torch.Tensor, *, image_mask: torch.Tensor | None = None) -> TopK:+ if image_mask is not None:+ assert self._bias_vl is not None, "image routing bias was not loaded"+ scores = bf16_linear_fp32(x, self.weight)+ return router_select(+ scores.view(-1, scores.shape[-1]), self.bias, self.topk,+ score_func=self.score_func, normalize=self.norm_topk_prob, route_scale=self.route_scale,+ alternate_bias=self._bias_vl, alternate_mask=image_mask,+ )+++class SharedExpert(BaseOP):+ """The dense clamped-SwiGLU expert every token passes through (fp8 block-32 linears)."""++ def __init__(self, args: DeepseekV41Args, *, quant_config=None, prefix: str = ""):+ self.w1 = LinearReplicated(args.dim, args.moe_inter_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.w1")+ self.w2 = LinearReplicated(args.moe_inter_dim, args.dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.w2")+ self.w3 = LinearReplicated(args.dim, args.moe_inter_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.w3")+ self.limit = args.swiglu_limit++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ h = fused_swiglu(self.w1.forward(x), self.w3.forward(x), self.limit, x.dtype)+ return self.w2.forward(h)+++class MoE(BaseOP):+ def __init__(self, config: ModelConfig, layer_id: int, *, prefix: str = ""):+ args: DeepseekV41Args = config.dsv41_args+ self.dim = args.dim+ self.gate = Router(args)+ self.shared_experts = SharedExpert(args, quant_config=config.quant, prefix=f"{prefix}.shared_experts")+ # every layer is MoE: the offload cache indexes experts by layer id directly+ self.experts = make_moe_layer(+ config,+ layer_id=layer_id,+ activation="swiglu_clamp",+ renormalize=args.norm_topk_prob,+ limit=args.swiglu_limit,+ quant_config=config.quant,+ prefix=f"{prefix}.experts",+ )++ def forward(self, x: torch.Tensor, *, image_mask: torch.Tensor | None = None) -> torch.Tensor:+ shape = x.shape+ x = x.view(-1, self.dim)+ weights, indices = self.gate.forward(x, image_mask=image_mask)+ # the shared expert runs before the routed experts (same stream, so it does not overlap the+ # hybrid decode's CPU work, whose submit follows it -- reordering that is follow-up work)+ shared = self.shared_experts.forward(x)+ routed = self.experts.routed_forward(x, weights, indices)+ return (routed + shared).view(shape)+++__all__ = ["MoE", "Router", "SharedExpert"]diff --git a/python/freetoken/models/deepseek_v41/rope.py b/python/freetoken/models/deepseek_v41/rope.pynew file mode 100644index 00000000..6d7df9f8--- /dev/null+++ b/python/freetoken/models/deepseek_v41/rope.py@@ -0,0 +1,58 @@+"""Rotary embeddings for DeepSeek-V4.1 (verbatim reference math, ``inference/model.py``).++Interleaved complex rotation on the last ``rope_head_dim`` channels, with YaRN when+``original_seq_len > 0``. Two regimes share two cached tables: window-only layers at the base+theta without YaRN, compressing layers at ``compress_rope_theta`` under YaRN. A compressed latent+stands for the first token of its group, so group ``j`` rotates at position ``j * ratio``; the+attention output is rotated back (``inverse=True``) because K == V share one rotated latent.+"""++from __future__ import annotations++import math+from functools import lru_cache++import torch+++def precompute_freqs_cis(dim: int, seqlen: int, original_seq_len: int, base: float, factor: float, beta_fast: int, beta_slow: int) -> torch.Tensor:+ freqs = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))+ if original_seq_len > 0:++ def corrected_dim(rotations):+ return dim * math.log(original_seq_len / (rotations * 2 * math.pi)) / (2 * math.log(base))++ low = max(math.floor(corrected_dim(beta_fast)), 0)+ high = min(math.ceil(corrected_dim(beta_slow)), dim - 1)+ ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / max(high - low, 1e-3)).clamp(0, 1)+ smooth = 1 - ramp+ freqs = freqs / factor * (1 - smooth) + freqs * smooth+ freqs = torch.outer(torch.arange(seqlen), freqs)+ return torch.polar(torch.ones_like(freqs), freqs)+++@lru_cache(2) # the model's two rope regimes; a 1M table is ~256 MB, so never one per layer+def get_freqs_cis(dim: int, seqlen: int, original_seq_len: int, base: float, factor: float, beta_fast: int, beta_slow: int, device: torch.device) -> torch.Tensor:+ return precompute_freqs_cis(dim, seqlen, original_seq_len, base, factor, beta_fast, beta_slow).to(device)+++def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor, inverse: bool = False) -> torch.Tensor:+ """In-place rotation of ``x[..., :]`` (``[T, rd]`` or ``[T, H, rd]``) by per-token ``freqs_cis [T, rd//2]``."""+ xc = torch.view_as_complex(x.float().unflatten(-1, (-1, 2)))+ if inverse:+ freqs_cis = freqs_cis.conj()+ if xc.ndim == 3:+ freqs_cis = freqs_cis.unsqueeze(1)+ x.copy_(torch.view_as_real(xc * freqs_cis).flatten(-2))+ return x+++def apply_rotary_emb_decode(x: torch.Tensor, freqs_cis: torch.Tensor, inverse: bool = False) -> torch.Tensor:+ """Batched decode rotation with PER-ROW freqs: ``x [B, 1, ..., rd]``, ``freqs_cis [B, rd//2]``+ (the fused dsv4 triton kernel; bit-identical to ``apply_rotary_emb``)."""+ from freetoken.kernel.triton.dsv4.rope import rope_decode_inplace++ return rope_decode_inplace(x, freqs_cis, inverse)+++__all__ = ["precompute_freqs_cis", "get_freqs_cis", "apply_rotary_emb", "apply_rotary_emb_decode"]diff --git a/python/freetoken/models/deepseek_v41/vision.py b/python/freetoken/models/deepseek_v41/vision.pynew file mode 100644index 00000000..f5fea969--- /dev/null+++ b/python/freetoken/models/deepseek_v41/vision.py@@ -0,0 +1,139 @@+"""DeepSeek ViT and 3x3 pixel-unshuffle aligner, with the shared encoder weight streamer."""++from __future__ import annotations++import torch+import torch.nn.functional as F+from freetoken.layers import BaseOP, LinearReplicated, OPList+from freetoken.models.weight_stream import BlockWeightStreamer+++class VisionNorm(BaseOP):+ def __init__(self, dim: int):+ self.weight = torch.empty(dim, dtype=torch.float32)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ y = x.float()+ return (y * torch.rsqrt(y.square().mean(-1, keepdim=True) + 1e-6) * self.weight).to(x.dtype)+++class VisionAttention(BaseOP):+ def __init__(self, vc):+ self.n_heads = vc.num_attention_heads+ self.head_dim = vc.hidden_size // self.n_heads+ self.wqkv = LinearReplicated(vc.hidden_size, 3 * vc.hidden_size, has_bias=True)+ self.wo = LinearReplicated(vc.hidden_size, vc.hidden_size, has_bias=True)++ def forward(self, x, cos, sin):+ n = x.shape[0]+ q, k, v = (t.reshape(n, self.n_heads, self.head_dim) for t in self.wqkv.forward(x).chunk(3, -1))+ def rotary(t):+ a, b = t.float().chunk(2, -1)+ return torch.cat((a * cos - b * sin, b * cos + a * sin), -1).to(t.dtype)+ q, k = rotary(q), rotary(k)+ q, k, v = (t.transpose(0, 1).unsqueeze(0) for t in (q, k, v))+ y = F.scaled_dot_product_attention(q, k, v)+ return self.wo.forward(y[0].transpose(0, 1).reshape(n, -1))+++class VisionMLP(BaseOP):+ def __init__(self, vc):+ self.w1 = LinearReplicated(vc.hidden_size, 2 * vc.intermediate_size, has_bias=False)+ self.w2 = LinearReplicated(vc.intermediate_size, vc.hidden_size, has_bias=False)++ def forward(self, x):+ gate, up = self.w1.forward(x).chunk(2, -1)+ return self.w2.forward(F.silu(gate) * up)+++class VisionBlock(BaseOP):+ def __init__(self, vc):+ self.norm1 = VisionNorm(vc.hidden_size)+ self.attn = VisionAttention(vc)+ self.norm2 = VisionNorm(vc.hidden_size)+ self.mlp = VisionMLP(vc)++ def forward(self, x, cos, sin):+ x = x + self.attn.forward(self.norm1.forward(x), cos, sin)+ return x + self.mlp.forward(self.norm2.forward(x))+++class PatchEmbed(BaseOP):+ def __init__(self, vc):+ self.proj = LinearReplicated(3 * vc.patch_size**2, vc.hidden_size, has_bias=True)++ def forward(self, x):+ return self.proj.forward(x.flatten(1))+++class VisionTransformer(BaseOP):+ def __init__(self, vc):+ self.patch_embed = PatchEmbed(vc)+ self.blocks = OPList([VisionBlock(vc) for _ in range(vc.num_hidden_layers)])+ self.norm = VisionNorm(vc.hidden_size)+ self._vc = vc+ self._streamer = None++ def place_weights(self, mode: str) -> None:+ if mode == "host" and self._streamer is None:+ self._streamer = BlockWeightStreamer(self.blocks.op_list, self.patch_embed.proj.weight.device)+ elif mode == "gpu" and self._streamer is not None:+ self._streamer.unstream()+ self._streamer = None+ elif mode not in ("gpu", "host"):+ raise ValueError(f"unknown vision weight placement {mode!r}")++ def forward(self, patches, h: int, w: int):+ x = self.patch_embed.forward(patches)+ dim = self._vc.hidden_size // self._vc.num_attention_heads // 2+ inv = 1.0 / (self._vc.rope_theta ** (torch.arange(0, dim, 2, dtype=torch.float32, device=x.device) / dim))+ rows = torch.arange(h, device=x.device)[:, None].expand(h, w)+ cols = torch.arange(w, device=x.device)[None, :].expand(h, w)+ angles = (torch.stack((rows, cols), -1).reshape(-1, 2, 1).float() * inv).flatten(1)+ cos, sin = angles.cos().unsqueeze(1), angles.sin().unsqueeze(1)+ blocks = enumerate(self.blocks.op_list) if self._streamer is None else self._streamer.blocks(self.blocks.op_list)+ for _, block in blocks:+ x = block.forward(x, cos, sin)+ return self.norm.forward(x)+++class VisionAligner(BaseOP):+ def __init__(self, vc, dim: int):+ self.ratio = vc.downsample_ratio+ self.w1 = LinearReplicated(vc.hidden_size * self.ratio**2, dim, has_bias=True)+ self.w2 = LinearReplicated(dim, dim, has_bias=True)++ def forward(self, x, h: int, w: int):+ r = self.ratio+ x = x.view(h, w, -1).permute(2, 0, 1)+ x = F.pad(x, (0, -w % r, 0, -h % r))+ x = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).T+ return self.w2.forward(F.gelu(self.w1.forward(x)))+++class DeepseekV41Vision(BaseOP):+ def __init__(self, vc, dim: int, *, num_layers: int, num_experts: int):+ self.vision = VisionTransformer(vc)+ self.aligner = VisionAligner(vc, dim)+ self.image_start = torch.empty(dim)+ self.image_end = torch.empty(dim)+ self.image_newline = torch.empty(dim)+ # Keep all image-only parameters under visual, including FTW/text-only filtering.+ self.routing_bias = torch.empty(num_layers, num_experts, dtype=torch.float32)++ def place_weights(self, mode: str) -> None:+ self.vision.place_weights(mode)++ @torch.inference_mode()+ def forward(self, item):+ _, h, w = item.grid_thw+ patches = item.feature.to(device=self.image_start.device, dtype=self.image_start.dtype)+ x = self.aligner.forward(self.vision.forward(patches, h, w), h, w)+ r = self.aligner.ratio+ nh, nw = (h + r - 1) // r, (w + r - 1) // r+ out = x.new_empty(nh * (nw + 1) + 2, x.shape[-1])+ out[0], out[-1] = self.image_start, self.image_end+ rows = out[1:-1].view(nh, nw + 1, -1)+ rows[:, :nw] = x.view(nh, nw, -1)+ rows[:, nw] = self.image_newline+ return outdiff --git a/python/freetoken/models/deepseek_v41/weight.py b/python/freetoken/models/deepseek_v41/weight.pynew file mode 100644index 00000000..1ccd23ea--- /dev/null+++ b/python/freetoken/models/deepseek_v41/weight.py@@ -0,0 +1,242 @@+"""Weight loading for DeepSeek-V4.1-Flash.++* :func:`iter_weights` streams the resident (non-expert) tensors keyed by the model's attribute+ paths: checkpoint ``layers.N.*`` -> ``model.layers.N.*``, ``embed`` / ``norm`` -> ``model.*``,+ ``head`` -> ``head``. fp8 linears keep their e4m3 payload and e8m0 ``scale`` (declared by the quant+ method as ``weight_scale_inv``); ``wo_a`` keeps its e4m3 payload and ``wo_a_scale`` too (the layer+ dequantizes in-kernel for the reference's bf16 grouped einsum). The engine casts everything else to+ the declared parameter dtype (fp32 compressor weights, hc gates, engram gate weights).+* :func:`iter_expert_pieces` streams the routed MXFP4 experts (e2m1 pairs + e8m0 per-32 scales) per+ expert for the offload banks.++Dropped: the DSpark ``mtp.*`` stage; vision weights and ``gate.bias_vl`` when serving text-only.+The Engram tables (``layers.{1,14}.engram.embed.*``) are+not state-dict tensors: the disk row store maps them in place (``engram_table.py``); an FTW conversion+carries them as side files (:func:`ftw_side_files`).+"""++from __future__ import annotations++from contextlib import ExitStack+import json+import os+import re++import safetensors+import torch+from freetoken.distributed import get_tp_info+from freetoken.layers.quantization import QuantKind+from freetoken.models.loader import drop_page_cache+from tqdm import tqdm++from .args import DeepseekV41Args, Mode+++class _ShardReader:+ def __init__(self, folder: str, weight_map: dict, device):+ self._folder = folder+ self._weight_map = weight_map+ self._device = str(device)+ self._handles: dict[str, object] = {}+ self._stack = ExitStack()++ def has(self, name: str) -> bool:+ return name in self._weight_map++ def get(self, name: str) -> torch.Tensor:+ shard = self._weight_map[name]+ handle = self._handles.get(shard)+ if handle is None:+ handle = self._stack.enter_context(safetensors.safe_open(os.path.join(self._folder, shard), framework="pt", device=self._device))+ self._handles[shard] = handle+ return handle.get_tensor(name)++ def close(self) -> None:+ try:+ self._stack.close()+ finally:+ for shard in self._handles:+ drop_page_cache(os.path.join(self._folder, shard))+ self._handles.clear()+++def weight_map(model_path: str) -> dict:+ with open(os.path.join(model_path, "model.safetensors.index.json")) as f:+ return json.load(f)["weight_map"]+++def load_args(model_path: str) -> DeepseekV41Args:+ from freetoken.utils.hf import cached_load_hf_config++ return DeepseekV41Args.from_hf(cached_load_hf_config(model_path))+++def dequant_fp8_block(weight: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:+ """Square block-scaled e4m3 -> bf16; ``scale`` holds e8m0 codes (``2^(code-127)``), the block+ edge follows from the shapes."""+ n, k = weight.shape+ block = n // scale.shape[0]+ assert (n // block, k // block) == tuple(scale.shape), (weight.shape, scale.shape)+ s = torch.exp2(scale.view(torch.uint8).to(torch.float32) - 127.0)+ s = s.repeat_interleave(block, dim=0).repeat_interleave(block, dim=1)+ return (weight.to(torch.float32) * s).to(torch.bfloat16)+++def iter_weights(model_path: str, device, *, include_moe_experts: bool = True, include_non_moe: bool = True, include_vision: bool = True):+ args = load_args(model_path)+ reader = _ShardReader(model_path, weight_map(model_path), device)+ get = reader.get+ def linear(src: str, dst: str):+ yield f"{dst}.weight", get(f"{src}.weight")+ if reader.has(f"{src}.scale"):+ yield f"{dst}.weight_scale_inv", get(f"{src}.scale")++ try:+ if include_moe_experts:+ # Resident experts support unquantized checkpoints; FP4 uses the offload cache.+ if reader.has("layers.0.ffn.experts.0.w1.scale"):+ raise ValueError(+ "DeepSeek-V4.1's fp4 routed experts are served from the offload cache; run with "+ "--moe-strategy offload (include_moe_experts must be False)."+ )+ for L in range(args.n_layers):+ p = f"layers.{L}.ffn.experts"+ gate_up = torch.stack([torch.cat([get(f"{p}.{e}.w1.weight"), get(f"{p}.{e}.w3.weight")]) for e in range(args.n_routed_experts)])+ down = torch.stack([get(f"{p}.{e}.w2.weight") for e in range(args.n_routed_experts)])+ yield f"model.layers.{L}.ffn.experts.gate_up_proj", gate_up+ yield f"model.layers.{L}.ffn.experts.down_proj", down+ if not include_non_moe:+ return+ if include_vision and reader.has("vision.patch_embed.proj.weight"):+ yield from iter_vision_weights(model_path, device)+ yield "model.embed.weight", get("embed.weight")+ yield "model.norm.weight", get("norm.weight")+ yield "head.weight", get("head.weight")+ for L in range(args.n_layers):+ role = args.roles[L]+ a, m = f"layers.{L}.attn", f"model.layers.{L}.attn"+ yield from linear(f"{a}.wq_a", f"{m}.wq_a")+ yield f"{m}.q_norm.weight", get(f"{a}.q_norm.weight")+ yield from linear(f"{a}.wq_b", f"{m}.wq_b")+ yield from linear(f"{a}.wkv", f"{m}.wkv")+ yield f"{m}.kv_norm.weight", get(f"{a}.kv_norm.weight")+ yield f"{m}.wo_a", get(f"{a}.wo_a.weight")+ if reader.has(f"{a}.wo_a.scale"): # the fp8 payload + scale; the layer dequantizes in-kernel+ yield f"{m}.wo_a_scale", get(f"{a}.wo_a.scale")+ yield from linear(f"{a}.wo_b", f"{m}.wo_b")+ yield f"{m}.attn_sink", get(f"{a}.attn_sink")+ if role.mode is Mode.FULL:+ c = f"{a}.compressor"+ yield f"{m}.compressor.norm.weight", get(f"{c}.norm.weight")+ if role.ratio > 1:+ yield f"{m}.compressor.wkv", get(f"{c}.wkv.weight")+ yield f"{m}.compressor.wgate", get(f"{c}.wgate.weight")+ else:+ yield from linear(f"{c}.wkv", f"{m}.compressor.wkv")+ yield from linear(f"{a}.indexer.wk", f"{m}.indexer.wk")+ yield f"{m}.indexer.k_norm.weight", get(f"{a}.indexer.k_norm.weight")+ if role.has_indexer:+ yield from linear(f"{a}.indexer.wq_b", f"{m}.indexer.wq_b")+ yield from linear(f"{a}.indexer.weights_proj", f"{m}.indexer.weights_proj")++ p = f"layers.{L}"+ yield f"model.{p}.attn_norm.weight", get(f"{p}.attn_norm.weight")+ yield f"model.{p}.ffn_norm.weight", get(f"{p}.ffn_norm.weight")+ yield f"model.{p}.ffn.gate.weight", get(f"{p}.ffn.gate.weight")+ yield f"model.{p}.ffn.gate.bias", get(f"{p}.ffn.gate.bias")+ for proj in ("w1", "w2", "w3"):+ yield from linear(f"{p}.ffn.shared_experts.{proj}", f"model.{p}.ffn.shared_experts.{proj}")+ for nm in ("hc_attn_fn", "hc_ffn_fn", "hc_attn_base", "hc_ffn_base", "hc_attn_scale", "hc_ffn_scale"):+ yield f"model.{p}.{nm}", get(f"{p}.{nm}")+ if L in args.engram_layer_ids:+ e = f"{p}.engram"+ yield from linear(f"{e}.wkv", f"model.{e}.wkv")+ yield f"model.{e}.q_weight", get(f"{e}.q_weight")+ yield f"model.{e}.k_weight", get(f"{e}.k_weight")+ finally:+ reader.close()+++def iter_vision_weights(model_path: str, device):+ names = weight_map(model_path)+ reader = _ShardReader(model_path, names, device)+ try:+ for name in names:+ if name.startswith(("vision.", "aligner.")) or name in ("image_start", "image_end", "image_newline"):+ yield f"visual.{name}", reader.get(name)+ args = load_args(model_path)+ yield "visual.routing_bias", torch.stack([reader.get(f"layers.{i}.ffn.gate.bias_vl") for i in range(args.n_layers)])+ finally:+ reader.close()+++# ----- FTW side files: the Engram tables ----------------------------------------------------------+def engram_table_names(layer_id: int) -> tuple[str, str]:+ return f"layers.{layer_id}.engram.embed.weight", f"layers.{layer_id}.engram.embed.scale"+++def ftw_side_files(model_path: str, out_dir: str) -> list[str]:+ """Copy the checkpoint shards holding the Engram tables (fp8 rows + ue8m0 scales) next to an FTW+ checkpoint. The tables are never FTW entries: ``load_host_tables`` maps them in place from whatever+ safetensors file holds them (``engram_table.engram_row_source`` reads the headers), and the FTW+ loader never reads ``.safetensors`` as weights, so the shard's other tensors ride along unused."""+ import shutil++ from freetoken.models.loader import safetensors_weight_map+ from freetoken.utils import download_hf_weight++ folder = download_hf_weight(model_path)+ args = load_args(folder)+ shard_of = safetensors_weight_map(folder)+ shards = sorted({shard_of[n] for layer_id in args.engram_layer_ids if layer_id < args.n_layers for n in engram_table_names(layer_id)})+ for shard in shards:+ shutil.copyfile(os.path.join(folder, shard), os.path.join(out_dir, shard))+ return shards+++# ----- routed MXFP4 expert pieces ---------------------------------------------------------------+_EXPERT_RE = re.compile(r"^layers\.(?P<layer>\d+)\.ffn\.experts\.(?P<expert>\d+)\.(?P<proj>w1|w2|w3)\.(?P<kind>weight|scale)$")+_PROJ_ROLE = {"w1": "gate", "w3": "up", "w2": "down"}+_KIND_SUFFIX = {"weight": "", "scale": "_scale"}+++def iter_expert_pieces(model_path: str, config, kind: QuantKind, *, parallel: bool | None = False, workers: int = 8, chunk: int = 8 << 20):+ """One piece per expert: ``{gate, up, down}`` e2m1 pairs and their e8m0 ``_scale`` companions.+ The DSpark stage's experts (``mtp.*``) never match."""+ if kind is not QuantKind.MXFP4:+ return None+ if get_tp_info().size > 1:+ raise NotImplementedError("DeepSeek-V4.1 expert banks support TP=1 only")+ from freetoken.models.weight import iter_expert_tensors_parallel+ from freetoken.moe.expert_pieces import per_expert_pieces++ args = load_args(model_path)+ L, E = args.n_layers, args.n_routed_experts++ def locate(raw_name: str):+ m = _EXPERT_RE.match(raw_name)+ if m is None or int(m["layer"]) >= L:+ return None+ return int(m["layer"]), int(m["expert"]), _PROJ_ROLE[m["proj"]] + _KIND_SUFFIX[m["kind"]]++ if parallel:+ tensors = iter_expert_tensors_parallel(model_path, lambda n: locate(n) is not None, workers=workers, chunk=chunk)+ return per_expert_pieces(tensors, locate, tensors_per_expert=6)++ def _serial():+ reader = _ShardReader(model_path, weight_map(model_path), torch.device("cpu"))+ try:+ for li in tqdm(range(L), desc="Loading DeepSeek-V4.1 experts (serial)", disable=not get_tp_info().is_primary()):+ for e in range(E):+ base = f"layers.{li}.ffn.experts.{e}"+ for proj in ("w1", "w3", "w2"):+ for kind_ in ("weight", "scale"):+ name = f"{base}.{proj}.{kind_}"+ yield name, reader.get(name)+ finally:+ reader.close()++ return per_expert_pieces(_serial(), locate, tensors_per_expert=6)+++__all__ = ["iter_weights", "iter_vision_weights", "iter_expert_pieces", "ftw_side_files", "load_args", "weight_map", "dequant_fp8_block"]diff --git a/python/freetoken/models/register.py b/python/freetoken/models/register.pyindex 2a4ced10..fe150ddc 100644--- a/python/freetoken/models/register.py+++ b/python/freetoken/models/register.py@@ -156,6 +156,20 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {# the head, the KV compressors and the indexer's scorer ship bf16; the fp8 config has no modules_to_not_convertunquantized_modules=("head", "*.compressor.wkv", "*.compressor.wgate", "*.indexer.weights_proj"),),+ # DeepSeek-V4.1-Flash (model_type deepseek_v41): multimodal wrapper config (text tower in+ # text_config). The checkpoint has no ``model.`` root; the compressors, the indexer's key projection /+ # scorer, the head and the vision tower + aligner ship bf16 (the fp8 config has no modules_to_not_convert).+ "DeepseekV41ForCausalLM": ModelSpec(+ "freetoken.models.deepseek_v41",+ "DeepseekV41ForCausalLM",+ mm_processor="freetoken.mm.processors.deepseek_v41:DeepseekV41MMProcessor",+ encoders=(EncoderSpec("vision", "vision_config", ("image",)),),+ checkpoint_roots=(("model.layers", "layers"), ("model.embed", "embed"), ("model.norm", "norm")),+ packed_modules_mapping=_EXPERTS_W123_PACKED,+ unquantized_modules=(+ "head", "*.compressor.wkv", "*.compressor.wgate", "*.indexer.wk", "*.indexer.weights_proj", "vision.*", "aligner.*",+ ),+ ),"Qwen3_5MoeForConditionalGeneration": ModelSpec("freetoken.models.qwen3_5_moe","Qwen3_5MoeForConditionalGeneration",diff --git a/python/freetoken/moe/benchbw.py b/python/freetoken/moe/benchbw.pyindex 83b3d196..e9d6ffe5 100644--- a/python/freetoken/moe/benchbw.py+++ b/python/freetoken/moe/benchbw.py@@ -108,6 +108,7 @@ class Workload:activation: str = "silu"swiglu_alpha: float = 1.702swiglu_limit: float | None = None+ act_block: int | None = None # ds_fp4 W4A8 activation quant block# Preset workloads. Dims from the model configs / benchmarks/bench_offload_cache_copy.py.@@ -120,7 +121,8 @@ WORKLOADS: dict[str, Workload] = {activation="gpt_oss_swiglu", swiglu_limit=7.0),"gpt-oss-20b": Workload("gpt-oss-20b", 2880, 2880, 32, 4, ("mxfp4_triton",),activation="gpt_oss_swiglu", swiglu_limit=7.0),- "dsv4": Workload("dsv4", 4096, 2048, 256, 6, ("ds_fp4",), swiglu_limit=7.0),+ "dsv4": Workload("dsv4", 4096, 2048, 256, 6, ("ds_fp4",), swiglu_limit=7.0, act_block=128),+ "dsv4.1-flash": Workload("dsv4.1-flash", 5120, 2304, 384, 6, ("ds_fp4",), swiglu_limit=10.0, act_block=32),"glm4.7-nvfp4": Workload("glm4.7-nvfp4", 5120, 1536, 160, 8, ("nvfp4",)),"glm5.3-flash-nvfp4": Workload("glm5.3-flash-nvfp4", 4096, 2048, 288, 8, ("nvfp4",),@@ -142,7 +144,7 @@ DTYPE_WORKLOADS: dict[str, Workload] = {"fp8_block": Workload("dtype:fp8", 2048, 512, 128, 8, ("fp8_block",)),"mxfp4_triton": Workload("dtype:mxfp4", 2880, 2880, 128, 4, ("mxfp4_triton",),activation="gpt_oss_swiglu", swiglu_limit=7.0),- "ds_fp4": Workload("dtype:ds_fp4", 4096, 2048, 128, 6, ("ds_fp4",), swiglu_limit=7.0),+ "ds_fp4": Workload("dtype:ds_fp4", 4096, 2048, 128, 6, ("ds_fp4",), swiglu_limit=7.0, act_block=128),}@@ -460,7 +462,7 @@ def _build_cpu_moe_executor(fmt: str, wl: Workload, banks: dict, num_threads: incache, top_k=wl.top_k, activation=wl.activation,apply_router_weight_on_input=False, num_threads=num_threads, max_tokens=1,device=torch.device("cuda"), swiglu_alpha=wl.swiglu_alpha,- swiglu_limit=wl.swiglu_limit,+ swiglu_limit=wl.swiglu_limit, act_block=wl.act_block,)diff --git a/python/freetoken/moe/cpu_executor.py b/python/freetoken/moe/cpu_executor.pyindex a4680674..c4809075 100644--- a/python/freetoken/moe/cpu_executor.py+++ b/python/freetoken/moe/cpu_executor.py@@ -171,6 +171,7 @@ class CpuMoeExecutor:swiglu_alpha: float = 1.702,swiglu_limit: float | None = None,fmt: str | None = None,+ act_block: int | None = None,) -> None:from freetoken.kernel import _cpu_moefrom freetoken.moe.legacy_format import canonical_role@@ -323,6 +324,11 @@ class CpuMoeExecutor:# the C++ side to skip its own. Measured on DeepSeek-V4-Flash bs=1 decode:# 12.85 -> 15.65 tok/s, output bit-identical (tests/moe/test_dsfp4_prequant.py).self._gpu_prequant = fmt == "ds_fp4" and device.type == "cuda"+ # the W4A8 activation round-trip block follows the checkpoint's fp8 block (128 on V4, 32 on V4.1)+ self._act_block = act_block+ if fmt == "ds_fp4":+ assert act_block is not None, "ds_fp4 experts need act_block, the checkpoint's fp8 activation quant block"+ self._ext.set_act_block(act_block)if self._gpu_prequant:self._ext.set_input_prequant(True)logger.info_rank0(@@ -482,7 +488,7 @@ class CpuMoeExecutor:"""DeepSeek-V4 ``ds_fp4`` schema: row-major e2m1 (2/byte) + e8m0 per-32 blockscales, no global, no bias. Layout matches nvfp4 (K contiguous per output row),so the C++ GEMV reads it in place. The kernel additionally FP8-round-trips the- activations (block 128) to match DSV4's W4A8 reference, hence the %128 dims."""+ activations (the checkpoint's fp8 block) to match the DeepSeek W4A8 reference."""gup, gus = banks["gate_up"], banks["gate_up_scale"]dnp, dns = banks["down"], banks["down_scale"]assert gup[0].dtype == torch.uint8 and dnp[0].dtype == torch.uint8, (gup[0].dtype, dnp[0].dtype)@@ -490,7 +496,7 @@ class CpuMoeExecutor:I = int(gup[0].shape[1] // 2)H = int(gup[0].shape[2] * 2)assert gup[0].shape[1] == 2 * I- assert H % 128 == 0 and I % 128 == 0, (H, I) # FP8 activation round-trip block=128+ assert H % 32 == 0 and I % 32 == 0, (H, I) # e8m0 scales per 32 along Kassert tuple(dnp[0].shape[1:]) == (H, I // 2), (dnp[0].shape, H, I)assert tuple(gus[0].shape[1:]) == (2 * I, H // 32), (gus[0].shape, I, H)assert tuple(dns[0].shape[1:]) == (H, I // 32), (dns[0].shape, H, I)@@ -580,7 +586,7 @@ class CpuMoeExecutor:# pre-quantized activations and skips its serial scalar pass.from freetoken.kernel.triton.dsv4.fp8_linear import act_quant_fp8_roundtrip- hidden_states = act_quant_fp8_roundtrip(hidden_states, block=128)+ hidden_states = act_quant_fp8_roundtrip(hidden_states, block=self._act_block)# D2H: ship this step's activations + routing to pinned host memory.io["x"].copy_(hidden_states, non_blocking=True)diff --git a/python/freetoken/moe/fused_ds_fp4.py b/python/freetoken/moe/fused_ds_fp4.pyindex 65fffb5c..b4dc5cd0 100644--- a/python/freetoken/moe/fused_ds_fp4.py+++ b/python/freetoken/moe/fused_ds_fp4.py@@ -12,10 +12,7 @@ from __future__ import annotationsimport torchimport triton-from freetoken.kernel.triton.dsv4.fp8_linear import (- act_quant_fp8_inplace,- act_quant_fp8_roundtrip,-)+from freetoken.kernel.triton.dsv4.fp8_linear import act_quant_fp8_roundtripfrom freetoken.kernel.triton.dsv4.fused_moe import (_decode_dsfp4_moe_kernel,_e2m1_lut,@@ -50,7 +47,8 @@ def _grouped_decode("""Grouped per-route GEMM ``c[t,r,:] = a[row] @ dequant(W[slot])^T``.Returns ``[T, top_k, N]``. Used for both gate_up (a_row=token) and down- (a_row=route). FP4 weights are dequantized inline from the slot cache.+ (a_row=route). FP4 weights are dequantized inline from the slot cache. A slot+ of ``-1`` is an inactive route: its output is zero and its storage is never read."""T, top_k = slots.shapeN = packed_cache.shape[1]@@ -62,14 +60,13 @@ def _grouped_decode(topk_weights = out.new_empty((1, 1), dtype=torch.float32)scale_u8 = scale_cache.view(torch.uint8)- # Decode is a per-route GEMV, bound by the inline FP4 dequant (LUT gather + block- # scale), not weight HBM bandwidth -- so it tops out ~29% of HBM peak. BN=16/- # BKB=128/1 warp maximizes memory-level parallelism (many single-warp CTAs, no- # cross-warp reduction) -> ~950 GB/s on H100 (was ~830 at BN=8/BKB=256/2). K_BYTES- # must be a multiple of BLOCK_SIZE_KB (gate_up 2048, down 1024 -- both /128).+ # One fixed launch geometry, never tuned per machine: the row / warp split changes the per-row+ # K reduction layout (and so the output bits) on some Triton versions. 16 rows on 2 warps is at+ # or near the fastest split for the V4 and V4.1 decode shapes. Each K tile contains whole+ # packed FP4 scale groups.BLOCK_SIZE_N = 16BLOCK_SIZE_KB = 128- _NW = 1+ _NW = 2assert (K // 2) % BLOCK_SIZE_KB == 0, (K, BLOCK_SIZE_KB)grid = (total_routes, triton.cdiv(N, BLOCK_SIZE_N))_decode_dsfp4_moe_kernel[grid](@@ -103,28 +100,28 @@ def routed_experts_fp4(down_packed: torch.Tensor, # [S, H, I//2] uint8down_scale: torch.Tensor, # [S, H, I//32] e8m0swiglu_limit: float,+ act_block: int = 128,) -> torch.Tensor:"""Full routed-expert output (summed over the top-k routes), excludes shared expert.- Precision matches the reference ``fp4_gemm(act_quant(x, 128), W_fp4)``: the gate_up- and down activations are FP8-round-tripped (block 128, ue8m0) before each GEMM. Since- an fp8 value x pow2 scale is exact in bf16, the round-tripped activation entering the- bf16 decode kernel is bit-identical to the reference's dequantized FP8 activation- (validated max diff = 0 vs the tilelang ``fp4_gemm`` reference)."""+ Precision matches the reference ``fp4_gemm(act_quant(x, act_block), W_fp4)``: the gate_up+ and down activations are FP8-round-tripped (block ``act_block`` -- the checkpoint's fp8 block,+ 128 on V4 and 32 on V4.1 -- ue8m0) before each GEMM. Since an fp8 value x pow2 scale is exact+ in bf16, the round-tripped activation entering the bf16 decode kernel is bit-identical to the+ reference's dequantized FP8 activation (validated max diff = 0 vs the tilelang ``fp4_gemm``+ reference). The routing weight scales each route's down output in the GEMM epilogue."""T, top_k = slots.shapeH = x.shape[1]two_I = gate_up_packed.shape[1]I = two_I // 2- x = act_quant_fp8_roundtrip(x, 128) # gate_up activation -> FP8 round-trip (no clone)+ x = act_quant_fp8_roundtrip(x, act_block) # gate_up activation -> FP8 round-trip (no clone)gate_up = _grouped_decode(x, gate_up_packed, gate_up_scale, slots, None,a_row_is_route=False, mul_routed_weight=False,) # [T, top_k, 2I]- act = fused_swiglu(gate_up, swiglu_limit) # [T, top_k, I]-- act = act.reshape(T * top_k, I)- act_quant_fp8_inplace(act, 128) # down activation -> FP8 round-trip+ # [T, top_k, I] with the down activation's FP8 round-trip in the same pass+ act = fused_swiglu(gate_up, swiglu_limit, act_block=act_block).reshape(T * top_k, I)down = _grouped_decode(act, down_packed, down_scale, slots, topk_weights,a_row_is_route=True, mul_routed_weight=True,@@ -191,6 +188,7 @@ def routed_experts_fp4_prefill(down_scale: torch.Tensor, # [S, H, I//32] e8m0swiglu_limit: float,num_rows: int,+ act_block: int = 128,) -> torch.Tensor:"""Grouped-GEMM counterpart of :func:`routed_experts_fp4` for dense prefillchunks: one moe_align sort shared by both GEMMs, each expert's weights@@ -201,7 +199,7 @@ def routed_experts_fp4_prefill(if T * top_k < _GROUPED_MIN_ROUTES:return routed_experts_fp4(x, slots, topk_weights,- gate_up_packed, gate_up_scale, down_packed, down_scale, swiglu_limit,+ gate_up_packed, gate_up_scale, down_packed, down_scale, swiglu_limit, act_block=act_block,)H = x.shape[1]two_I = gate_up_packed.shape[1]@@ -217,16 +215,14 @@ def routed_experts_fp4_prefill(sorted_ids, expert_ids, ntpp = moe_align_block_size(slots, cfg["BLOCK_SIZE_M"], num_rows)tw = topk_weights.reshape(-1).contiguous()- x = act_quant_fp8_roundtrip(x, 128) # gate_up activation -> FP8 round-trip (no clone)+ x = act_quant_fp8_roundtrip(x, act_block) # gate_up activation -> FP8 round-trip (no clone)gate_up = torch.empty((T, top_k, two_I), dtype=x.dtype, device=x.device)_grouped_prefill(x, gate_up_packed, gate_up_scale, gate_up, tw,sorted_ids, expert_ids, ntpp, routes, top_k, False, cfg,)- act = fused_swiglu(gate_up, swiglu_limit) # [T, top_k, I]-- act = act.reshape(routes, I)- act_quant_fp8_inplace(act, 128) # down activation -> FP8 round-trip+ # [T, top_k, I] with the down activation's FP8 round-trip in the same pass+ act = fused_swiglu(gate_up, swiglu_limit, act_block=act_block).reshape(routes, I)down = torch.empty((T, top_k, H), dtype=x.dtype, device=x.device)_grouped_prefill(act, down_packed, down_scale, down, tw,diff --git a/python/freetoken/scheduler/cache.py b/python/freetoken/scheduler/cache.pyindex 8f77be93..772297e0 100644--- a/python/freetoken/scheduler/cache.py+++ b/python/freetoken/scheduler/cache.py@@ -66,6 +66,11 @@ class CacheManager:supports_runtime_rebuild = Trueprefill_chunk_budget = None # generic shared page pool: no per-model prefill chunk cap+ @property+ def prefix_replay_tokens(self) -> int:+ """Prompt-tail tokens a prefix hit must leave to the prefill (a pool whose model recomputes them)."""+ return getattr(self.swa_pool, "prefix_replay_tokens", 0) if self.swa_pool is not None else 0+@propertydef prefill_chunk_align(self) -> int:"""Granularity a non-final prefill chunk should end on. A hybrid snapshot is donated only@@ -90,10 +95,10 @@ class CacheManager:return SWARadixCache(device, page_size, self.sliding_window_size)return create_prefix_cache(device=device, type=type, page_size=page_size)- def match_req(self, req: PendingReq) -> MatchResult:+ def match_req(self, req: PendingReq, max_len: int | None = None) -> MatchResult:input_len = req.input_lenassert input_len > 0, "Input length must be greater than 0."- ids = req.input_ids[: input_len - 1]+ ids = req.input_ids[: input_len - 1 if max_len is None else max(0, min(input_len - 1, max_len))]if self.is_swa:from freetoken.kvcache.swa_radix_cache import SWACacheHandlem = self.prefix_cache.match_prefix(ids)diff --git a/python/freetoken/scheduler/prefill.py b/python/freetoken/scheduler/prefill.pyindex bd8129aa..de4bddfb 100644--- a/python/freetoken/scheduler/prefill.py+++ b/python/freetoken/scheduler/prefill.py@@ -71,8 +71,9 @@ class PrefillAdder:if self.table_manager.available_size == 0:return None- # TODO: consider host cache match case- mr = self.cache_manager.match_req(req)+ # TODO: consider host cache match case (it needs the same replay cap)+ replay = self.cache_manager.prefix_replay_tokens+ mr = self.cache_manager.match_req(req, max_len=max(0, req.input_len - replay) if replay else None)handle = mr.cuda_handlecached_len = handle.cached_len# TODO: better estimate policy@@ -214,6 +215,7 @@ class PrefillAdder:uid=pending_req.uid,cache_handle=cache_handle,sampling_params=pending_req.sampling_params,+ prompt_len=pending_req.input_len,)req.mm_items = pending_req.mm_itemsreq.mrope_positions_full = pending_req.mrope_positions_fulldiff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.pyindex fa0d96a7..d057d3b9 100644--- a/python/freetoken/scheduler/scheduler.py+++ b/python/freetoken/scheduler/scheduler.py@@ -349,7 +349,7 @@ class Scheduler(SchedulerIOMixin):next_token = int(next_token.item())# EOS / stop-string -> "stop", output budget exhausted -> "length";# EOS and stop strings win over length.- # Overlap can advance device_len ahead of the token delivered to the host.+ # Overlap can advance device_len one token ahead of this host reply.hit_length = req.input_ids.numel() >= req.max_device_lenhit_eos = (not req.sampling_params.ignore_eos and next_token in self.eos_token_ids@@ -737,17 +737,10 @@ class Scheduler(SchedulerIOMixin):the CONCRETE current window (usable pages) so a rollback restores it byte-for-byte,whether it was pinned or ratio-derived."""eng = self.engine- config = self.config- mc = config.model_config- num_swa_pages = None- if getattr(mc, "dsv4_args", None) is not None:- sizes = getattr(eng.kv_cache, "sizes", None)- if sizes is not None: # usable window pages = physical n_win_pages minus the dummy page- num_swa_pages = max(0, sizes.n_win_pages - 1)- elif getattr(mc, "has_swa_attention", False) and (- getattr(config, "cache_type", None) == "swa_radix"- ): # usable window tokens = pool tokens minus the slot-0 sentinel- num_swa_pages = max(0, int(getattr(eng.kv_cache, "swa_num_tokens", 0) or 0) - 1)+ from freetoken.kvcache.cache_status import window_pool_spec++ spec = window_pool_spec(self.config)+ num_swa_pages = eng.kv_cache.window_pages if spec is not None else Nonereturn dict(num_pages=eng.num_pages,moe_cache_size=eng.moe_offload_cache.cache_size if eng.moe_offload_cache is not None else None,diff --git a/python/freetoken/server/api_server.py b/python/freetoken/server/api_server.pyindex 5592a2b0..34221cfc 100644--- a/python/freetoken/server/api_server.py+++ b/python/freetoken/server/api_server.py@@ -563,8 +563,12 @@ def _resolve_num_swa_pages(state: FrontendManager, req: CacheRebuildRequest) ->or pools.get("num_pages", 0) or 0)page_size = int(pools.get("page_size", 0) or getattr(config, "page_size", 1) or 1)- is_dsv4 = getattr(getattr(config, "model_config", None), "dsv4_args", None) is not None- swa_page_size = page_size if is_dsv4 else 1+ from freetoken.kvcache.cache_status import window_pool_spec++ spec = window_pool_spec(config)+ if spec is None:+ raise ValueError("swa_full_tokens_ratio requested but this model has no resizable window pool")+ swa_page_size = spec.page_sizewindow_tokens = int(round(req.swa_full_tokens_ratio * num_pages * page_size))return max(1, -(-window_tokens // swa_page_size)) # ceil-div to the pool's page unit@@ -604,12 +608,16 @@ async def cache_rebuild(req: CacheRebuildRequest):{"status": "failed", "error": "swa_full_tokens_ratio must be in (0, 1]"},status_code=422,)+ try:+ num_swa_pages = _resolve_num_swa_pages(state, req)+ except ValueError as exc:+ return JSONResponse({"status": "failed", "error": str(exc)}, status_code=422)result = await dispatch_rebuild(state,moe_cache_size=req.moe_cache_size,num_pages=req.num_pages,num_mamba_slots=req.num_mamba_slots,- num_swa_pages=_resolve_num_swa_pages(state, req),+ num_swa_pages=num_swa_pages,mode=req.mode,timeout=req.timeout,)@@ -739,11 +747,11 @@ def cache_geometry(state: Any) -> dict:# the internal currency), else the load-time meta value. 0.0 for models without a window pool.swa_full_tokens_ratio = float(getattr(state, "swa_full_tokens_ratio", 0.0) or 0.0)last_swa_pages = last.get("num_swa_pages")- if last_swa_pages:- is_dsv4 = getattr(getattr(config, "model_config", None), "dsv4_args", None) is not None- full = num_pages if is_dsv4 else num_pages * page_size+ swa_page_size = int(pools.get("swa_page_size", 0) or 0)+ if last_swa_pages and swa_page_size:+ full = num_pages * page_sizeif full > 0:- swa_full_tokens_ratio = min(1.0, last_swa_pages / full)+ swa_full_tokens_ratio = min(1.0, last_swa_pages * swa_page_size / full)try:model_config = config.model_confignum_experts = int(getattr(model_config, "num_experts", 0) or 0)@@ -785,9 +793,8 @@ def cache_geometry(state: Any) -> dict:"moe_cache_policy": getattr(config, "moe_cache_policy", None),"unit_bytes": unit_bytes,"swa_full_tokens_ratio": swa_full_tokens_ratio,- # The window pool's page unit for num_swa_pages: DSV4 = P (== page_size), radix-SWA = 1- # token, 0 for models without a window pool. Lets a client denominate the swa control.- "swa_page_size": int(pools.get("swa_page_size", 0) or 0),+ # The pool's window page unit; 0 for models without a resizable window pool.+ "swa_page_size": swa_page_size,# Concrete current window size in that unit (usable pages): the last rebuild's value if it# pinned/derived one, else the load-time pool size. 0 for models without a window pool."num_swa_pages": int(last.get("num_swa_pages") or pools.get("num_swa_pages", 0) or 0),diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.pyindex 2194b82e..b8d06018 100644--- a/python/freetoken/server/args.py+++ b/python/freetoken/server/args.py@@ -204,6 +204,8 @@ def parse_args(return "qwen3_coder"if "qwen" in marker:return "qwen25"+ if "deepseek_v41" in marker or "deepseekv41" in marker:+ return "deepseekv41"if "deepseek" in marker and ("v4" in marker or "deepseek_v4" in marker):return "deepseekv32"if "deepseek" in marker and ("v3.2" in marker or "v32" in marker):@@ -576,6 +578,7 @@ def parse_args("qwen3_coder","mistral","deepseekv32",+ "deepseekv41","gemma4","glm47","minimax",@@ -645,6 +648,17 @@ def parse_args(),)+ parser.add_argument(+ "--swa-decoder-replay",+ default=ServerArgs.swa_decoder_replay,+ choices=["bounded", "exact"],+ help=(+ "DeepSeek-V4.1 Decoder SWA Bounded Replay. 'bounded' (default) runs the 20 decoder layers "+ "on each prompt's last 128 tokens with their sliding window truncated there, as in the "+ "tech report; 'exact' runs them on every prompt token (the reference numerics)."+ ),+ )+parser.add_argument("--nvfp4-backend",action=_DeprecatedAlias,diff --git a/python/freetoken/server/function_call_parser.py b/python/freetoken/server/function_call_parser.pyindex 0804dcc3..296cd560 100644--- a/python/freetoken/server/function_call_parser.py+++ b/python/freetoken/server/function_call_parser.py@@ -78,6 +78,8 @@ TOOLS_TAG_LIST = ["<|DSML|function_calls>","<|DSML|tool_calls>","<|DSML|invoke",+ "<|DSML| calls>",+ "<|DSML| invoke","<atem:function_calls>",]@@ -1563,37 +1565,45 @@ class DeepSeekV32Detector(BaseFormatDetector):Reference: https://huggingface.co/deepseek-ai/DeepSeek-V3.2"""+ # the DSML tag names: (block, alternative block), invoke, parameter+ block_tags = ("function_calls", "tool_calls")+ invoke_tag = "invoke"+ param_tag = "parameter"+def __init__(self):super().__init__()self.dsml_token = "|DSML|"- self.bot_token = f"<{self.dsml_token}function_calls>"- self.eot_token = f"</{self.dsml_token}function_calls>"- self.alt_bot_token = f"<{self.dsml_token}tool_calls>"- self.alt_eot_token = f"</{self.dsml_token}tool_calls>"- self.invoke_start_prefix = f"<{self.dsml_token}invoke"- self.invoke_end_token = f"</{self.dsml_token}invoke>"- self.param_end_token = f"</{self.dsml_token}parameter>"+ block, alt_block = self.block_tags+ self.bot_token = f"<{self.dsml_token}{block}>"+ self.eot_token = f"</{self.dsml_token}{block}>"+ self.alt_bot_token = f"<{self.dsml_token}{alt_block}>"+ self.alt_eot_token = f"</{self.dsml_token}{alt_block}>"+ self.invoke_start_prefix = f"<{self.dsml_token}{self.invoke_tag}"+ self.invoke_end_token = f"</{self.dsml_token}{self.invoke_tag}>"+ self.param_start_prefix = f"<{self.dsml_token}{self.param_tag}"+ self.param_end_token = f"</{self.dsml_token}{self.param_tag}>"# Regex for complete invoke extraction- _de = re.escape(self.dsml_token)+ _inv = re.escape(self.dsml_token + self.invoke_tag)+ _par = re.escape(self.dsml_token + self.param_tag)self.invoke_regex = re.compile(- rf'<{_de}invoke\s+name="([^"]+)"\s*>(.*?)</{_de}invoke>',+ rf'<{_inv}\s+name="([^"]+)"\s*>(.*?)</{_inv}>',re.DOTALL,)# Regex for parameter extractionself.param_regex = re.compile(- rf'<{_de}parameter\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>(.*?)</{_de}parameter>',+ rf'<{_par}\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>(.*?)</{_par}>',re.DOTALL,)# Regex for partial invoke (name known, body still streaming)self.partial_invoke_regex = re.compile(- rf'<{_de}invoke\s+name="([^"]+)"\s*>(.*)',+ rf'<{_inv}\s+name="([^"]+)"\s*>(.*)',re.DOTALL,)# Streaming state machine tag regexes (anchored matches over the buffer).- self.invoke_open_regex = re.compile(rf'<{_de}invoke\s+name="([^"]+)"\s*>')+ self.invoke_open_regex = re.compile(rf'<{_inv}\s+name="([^"]+)"\s*>')self.param_open_regex = re.compile(- rf'<{_de}parameter\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>'+ rf'<{_par}\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>')self._last_arguments = ""@@ -1687,7 +1697,7 @@ class DeepSeekV32Detector(BaseFormatDetector):normal_parts: List[str] = []calls: List[ToolCallItem] = []- param_open_token = f"<{self.dsml_token}parameter"+ param_open_token = self.param_start_prefixdef _emit_args(fragment: str) -> None:if fragment:@@ -3512,6 +3522,16 @@ class MuseGlimmerDetector(InvokeParamStreamMixin, BaseFormatDetector):return StreamingParseResult(normal_text="".join(normal_parts).strip(), calls=calls)+class DeepSeekV41Detector(DeepSeekV32Detector):+ """DeepSeek-V4.1's DSML: the V3.2 grammar with every tag name led by a space and the block+ named ``calls`` (``<|DSML| calls>`` / ``<|DSML| invoke name=...>`` / ``<|DSML| parameter+ ...>``, the checkpoint's ``encoding/encoding.py``)."""++ block_tags = (" calls", " calls")+ invoke_tag = " invoke"+ param_tag = " parameter"++class FunctionCallParser:"""Parser for function/tool calls in model outputs.@@ -3523,6 +3543,7 @@ class FunctionCallParser:ToolCallParserEnum: Dict[str, Type[BaseFormatDetector]] = {"deepseekv32": DeepSeekV32Detector,+ "deepseekv41": DeepSeekV41Detector,"gemma4": Gemma4Detector,"gpt-oss": GptOssDetector,"gpt_oss": GptOssDetector,diff --git a/python/freetoken/server/stats.py b/python/freetoken/server/stats.pyindex d01828e2..3d48ee96 100644--- a/python/freetoken/server/stats.py+++ b/python/freetoken/server/stats.py@@ -120,12 +120,10 @@ def derive_model_card(config: Any) -> dict:def _swa_page_size(config: Any) -> int:- """The window pool's own page unit: P (window_size) for DSV4, 1 token for radix-SWA.- Mirrors compute_cache_pools' swa_page_size."""- dsv4 = getattr(getattr(config, "model_config", None), "dsv4_args", None)- if dsv4 is not None:- return int(getattr(dsv4, "window_size", 0) or 1)- return 1+ from freetoken.kvcache.cache_status import window_pool_spec++ spec = window_pool_spec(config)+ return spec.page_size if spec is not None else 1def build_stats(state: Any, p95_ms: int, ttft_mean_ms: int) -> dict:diff --git a/python/freetoken/tokenizer/effort.py b/python/freetoken/tokenizer/effort.pyindex c81cd7be..2e9e3cd2 100644--- a/python/freetoken/tokenizer/effort.py+++ b/python/freetoken/tokenizer/effort.py@@ -84,7 +84,8 @@ class EffortProfile:template interpolates anything". ``strength_dialect`` True means thetemplate reads the ``reasoning_strength`` spelling (muse-glimmer's family,whose documented ladder tops out at xhigh) rather than only- ``reasoning_effort``.+ ``reasoning_effort``. ``integer_range`` records a validated 1-100 budget+ accepted independently of the named effort vocabulary."""supported: frozenset[str]@@ -92,6 +93,7 @@ class EffortProfile:consumes_effort: boolvalidates: bool = Falsestrength_dialect: bool = False+ integer_range: tuple[int, int] | None = None@dataclass(frozen=True)@@ -118,7 +120,7 @@ class ThinkingProfile:_MAX_QUANTIZE_DISTANCE = 0.15-def quantize_effort(value: Any, profile: EffortProfile) -> str | None:+def quantize_effort(value: Any, profile: EffortProfile) -> str | int | None:"""Map a client's effort onto ``profile``; ``None`` means "send nothing".In-vocabulary values pass through untouched. Other named levels land on the@@ -128,6 +130,11 @@ def quantize_effort(value: Any, profile: EffortProfile) -> str | None:the template default. With "max" excluded the remaining scale values areunique, so quantization is deterministic across processes."""+ if profile.integer_range is not None and isinstance(value, (int, float)):+ low, high = profile.integer_range+ if type(value) is not int or not low <= value <= high:+ raise ValueError(f"reasoning_effort must be an integer in [{low}, {high}]")+ return valuesupported = effective_efforts(profile)if not supported:return None@@ -181,6 +188,7 @@ def probe_effort_profile(diverged: set[str] = set()matches_baseline: dict[str, bool] = {name: True for name in KNOWN_REASONING_EFFORTS}ran_rounds = 0+ integer_range = Nonefor base_kwargs, tools in _PROBE_ROUNDS:try:@@ -198,6 +206,10 @@ def probe_effort_profile(if _renderings_differ(rendering, baseline):diverged.add(name)matches_baseline[name] = False+ # Only a validated budget range is a numeric capability; arbitrary Jinja+ # interpolation must not advertise support for numeric reasoning effort.+ if _accepts_integer_budget(render, base_kwargs, tools):+ integer_range = (1, 100)if ran_rounds == 0:# Nothing learnable: sending no effort is the only safe rendering.@@ -214,7 +226,7 @@ def probe_effort_profile(strength_dialect = Falsesupported = frozenset(name for name in KNOWN_REASONING_EFFORTS if name not in rejected)- consumes = bool(rejected or diverged)+ consumes = bool(rejected or diverged or integer_range)default = Noneif consumes:defaults = [name for name in supported if matches_baseline[name]]@@ -226,9 +238,26 @@ def probe_effort_profile(consumes_effort=consumes,validates=bool(rejected),strength_dialect=strength_dialect,+ integer_range=integer_range,)+def _accepts_integer_budget(render, base_kwargs, tools) -> bool:+ for value in (0, 101):+ try:+ render({**base_kwargs, "reasoning_effort": value}, tools)+ except Exception: # noqa: BLE001 -- the encoder owns validation+ continue+ return False+ outputs = []+ for value in range(1, 101):+ try:+ outputs.append(render({**base_kwargs, "reasoning_effort": value}, tools))+ except Exception: # noqa: BLE001+ return False+ return _renderings_differ(outputs[0], outputs[-1])++def probe_thinking_profile(render: Callable[[dict[str, Any], list[dict[str, Any]] | None], Any],efforts: EffortProfile,diff --git a/python/freetoken/tokenizer/tokenize.py b/python/freetoken/tokenizer/tokenize.pyindex f4736652..77247257 100644--- a/python/freetoken/tokenizer/tokenize.py+++ b/python/freetoken/tokenizer/tokenize.py@@ -59,7 +59,7 @@ class TokenizeManager:self._effort_profile: EffortProfile | None = Noneself._thinking_profile: ThinkingProfile | None = Noneself._effort_lock = threading.Lock()- self._logged_effort_maps: set[tuple[Any, str | None]] = set()+ self._logged_effort_maps: set[tuple[Any, str | int | None]] = set()self._inline_system_lock = threading.Lock()self._inline_system_modes: dict[str, str] = {}@@ -117,7 +117,7 @@ class TokenizeManager:if mode == "auto":mode = self.inline_system_mode(msg.tools, kwargs)messages = normalize_inline_system(messages, mode)- return self._render(messages, msg.tools, kwargs)+ return self._render(messages, msg.tools, kwargs, images=msg.images)def inline_system_mode(self, tools: list[dict] | None, kwargs: dict[str, Any]) -> str:key = json.dumps([tools, kwargs], sort_keys=True)@@ -137,12 +137,13 @@ class TokenizeManager:messages: list[dict[str, Any]],tools: list[dict[str, Any]] | None,chat_template_kwargs: dict[str, Any],+ *, images: list[bytes] | None = None,) -> str:"""Raw render, no effort sanitation — the probe needs unsupported valuesto actually reach the template so rejection is observable."""if self._dsv4_encoder is not None:return _apply_dsv4_chat_encoder(- self._dsv4_encoder, messages, tools, chat_template_kwargs+ self._dsv4_encoder, messages, tools, chat_template_kwargs, images=images)# Broadcast the effort in every spelling the ecosystem's templates read# (muse-glimmer grades ``reasoning_strength``; Jinja ignores undeclared@@ -222,8 +223,12 @@ def _load_dsv4_encoder_if_needed(tokenizer: PreTrainedTokenizerBase) -> ModuleTymodel_path = getattr(tokenizer, "name_or_path", None) or getattr(tokenizer, "_name_or_path", "")if not model_path:return None- encoder_path = os.path.join(str(model_path), "encoding", "encoding_dsv4.py")- if not os.path.isfile(encoder_path):+ # DeepSeek-V4 ships encoding/encoding_dsv4.py, V4.1 encoding/encoding.py; same encode_messages contract+ encoder_path = next(+ (p for p in (os.path.join(str(model_path), "encoding", name) for name in ("encoding_dsv4.py", "encoding.py")) if os.path.isfile(p)),+ None,+ )+ if encoder_path is None:return Nonespec = importlib.util.spec_from_file_location("encoding_dsv4", encoder_path)if spec is None or spec.loader is None:@@ -240,9 +245,28 @@ def _apply_dsv4_chat_encoder(messages: list[dict],tools: list[dict] | None,chat_template_kwargs: dict,+ *, images: list[bytes] | None = None,) -> str:rendered_messages = [dict(message) for message in messages]+ image_data = iter(images) if images is not None else Nonefor message in rendered_messages:+ if isinstance(message.get("content"), list):+ content = []+ for part in message["content"]:+ if isinstance(part, dict) and part.get("type") == "image":+ # Prerender still has refs; the tokenizer worker receives fetched bytes.+ # The checkpoint encoder validates sources even when rendering only text.+ ref = part.get("freetoken_ref")+ if image_data is not None:+ try:+ part = {"type": "image", "data": next(image_data)}+ except StopIteration:+ raise ValueError("image blocks outnumber fetched images") from None+ elif ref is not None:+ key = "data" if ref["kind"] == "b64" else "url"+ part = {"type": "image", key: ref["data"]}+ content.append(part)+ message["content"] = contentif message.get("tool_calls"):message["tool_calls"] = _dsv4_tool_calls(message["tool_calls"])if tools:diff --git a/scripts/ftw_hotfix.py b/scripts/ftw_hotfix.pyindex f9cb62d1..9640c7b7 100644--- a/scripts/ftw_hotfix.py+++ b/scripts/ftw_hotfix.py@@ -263,7 +263,9 @@ def is_checkpoint_tower_name(name: str) -> bool:head = name.split(".")if head[0] == "model" and len(head) > 1:head = head[1:]- return "vision" in head[0] or "visual" in head[0] or head[0] in _TOWER_SEGMENTS+ return ("vision" in head[0] or "visual" in head[0] or head[0] in _TOWER_SEGMENTS+ or head[0] in ("aligner", "image_start", "image_end", "image_newline")+ or name.endswith(".ffn.gate.bias_vl"))def read_tower(source: TensorSource, ftw_dir: str, checkpoint_names: list[str]) -> dict[str, torch.Tensor]:diff --git a/tests/README.md b/tests/README.mdindex be22c1fb..76b2b1fb 100644--- a/tests/README.md+++ b/tests/README.md@@ -62,6 +62,7 @@ checkpoint is set:| `FREETOKEN_TEST_MEM_RATIO` | `e2e/test_aime.py` — offload-mode memory_ratio (default `0.9`) || `FREETOKEN_REBUILD_TEST_MODEL` | `e2e/test_cache_rebuild.py` — a SMALL local model dir; boots a real server (falls back to `FREETOKEN_TEST_MODEL`) || `FREETOKEN_GEMMA4_GGUF_GLOB` | `models/test_gemma4_gguf_rope.py` — glob matching a local gemma-4 GGUF file |+| `FREETOKEN_TEST_MODELS_ROOT` | `models/test_quant_config.py` — directory whose checkpoints the stored-tensor scheme scan checks (default `/mnt/nvme/models`) |`test_aime.py` takes its sampling protocol from the checkpoint's own`generation_config.json` (pass@N at the recommended temperature, or a single greedydiff --git a/tests/attention/test_dsv41_backend.py b/tests/attention/test_dsv41_backend.pynew file mode 100644index 00000000..387742eb--- /dev/null+++ b/tests/attention/test_dsv41_backend.py@@ -0,0 +1,208 @@+"""DSV41 attention backend: addressing and selection contracts, CPU-only (kernels live in tests/kernels).++* prefill metadata: encoder segments, the decoder pass under exact vs bounded replay;+* window candidates with a floor (bounded replay truncates the window at the prompt's last window);+* decode snapshot staging and the layer-invariant ring context;+* the selection helpers' reshape, candidate gather and position-to-row mapping.+"""++from __future__ import annotations++from types import SimpleNamespace++import pytest+import torch++from freetoken.core import Batch, Context, Req, SamplingParams, get_global_ctx, set_global_ctx+from freetoken.kvcache.dsv4.v41_cost_model import dsv41_pool_sizes+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache+from freetoken.models.deepseek_v41.args import DeepseekV41Args++P, MRR, DEVICE = 128, 4, torch.device("cpu")+RATIOS = (0, 2, 2, 1, 1)+SOURCES = (1, 3)+++def _ctx(pool):+ try:+ ctx = get_global_ctx()+ except AssertionError:+ ctx = Context(page_size=P)+ set_global_ctx(ctx)+ ctx.kv_cache = pool+ return ctx+++def _stack(swa_decoder_replay="exact", num_pages=32, max_seq_len=8192):+ from freetoken.attention.dsv41_sparse import DSV41SparseAttnBackend++ args = DeepseekV41Args(n_layers=len(RATIOS), head_dim=512, index_head_dim=128, window_size=P, compress_ratios=RATIOS,+ kv_source_layers=SOURCES, swa_decoder_replay=swa_decoder_replay)+ pool = DSV41PagedKVCache(dsv41_pool_sizes(num_pages + 1, args, 1.0, P), args, DEVICE, n_scratch=MRR + 1)+ pool._init_paged_state(MRR, True)+ pt = torch.zeros(MRR + 1, max_seq_len, dtype=torch.int32)+ pt[MRR].fill_(num_pages * P)+ pt[2, :300] = torch.arange(300, dtype=torch.int32)+ for page in range(3):+ pool.bind_window_pages(page * P, page * P)+ pool.full_loc_map = pt+ _ctx(pool)+ backend = DSV41SparseAttnBackend(SimpleNamespace(dsv41_args=args))+ return backend, pool, pt+++def _req(table_idx, cached_len, n_new, output_len=16, uid=0, prompt_len=0):+ return Req(input_ids=torch.zeros(cached_len + n_new, dtype=torch.int32), table_idx=table_idx, cached_len=cached_len,+ output_len=output_len, uid=uid, sampling_params=SamplingParams(), cache_handle=None, prompt_len=prompt_len)+++def _prefill_batch(reqs):+ batch = Batch(reqs=reqs, phase="prefill")+ batch.padded_reqs = reqs+ return batch+++def test_prefill_metadata_exact_mode_runs_the_decoder_on_every_token():+ backend, pool, _ = _stack("exact")+ assert pool.prefix_replay_tokens == 0+ batch = _prefill_batch([_req(0, 0, 300), _req(2, 256, 44, uid=1)])+ backend.prepare_metadata(batch)+ md = batch.attn_metadata+ assert [(s.offset, s.n, s.table_idx, s.start_pos, s.window_floor) for s in md.segments] == [(0, 300, 0, 0, 0), (300, 44, 2, 256, 0)]+ assert md.decoder_segments is md.segments and md.decoder_rows is None+ assert md.last_indices.tolist() == [299, 343]+++def test_prefill_metadata_bounded_mode_runs_the_decoder_on_the_last_window():+ """The encoder runs every new token; the decoder runs each request's part of its prompt's last+ window ``[L - P, L)``, floored at ``L - P``. A chunk ending before that window runs no decoder row+ and its ``last_indices`` entry points at the placeholder row the model appends."""+ backend, pool, _ = _stack("bounded")+ assert pool.prefix_replay_tokens == P and pool.sliding_window_size == P+ cold = _req(0, 0, 300) # a whole prompt: L - P = 172+ early = _req(1, 256, 128, uid=1, prompt_len=600) # chunk [256, 384) of a 600-token prompt: before 472+ straddle = _req(2, 384, 128, uid=2, prompt_len=600) # chunk [384, 512): its rows [472, 512) are in the window+ batch = _prefill_batch([cold, early, straddle])+ backend.prepare_metadata(batch)+ md = batch.attn_metadata+ assert [(s.offset, s.n, s.start_pos, s.window_floor) for s in md.segments] == [(0, 300, 0, 0), (300, 128, 256, 0), (428, 128, 384, 0)]+ assert [(d.offset, d.n, d.table_idx, d.start_pos, d.window_floor) for d in md.decoder_segments] == [(0, 128, 0, 172, 172), (128, 40, 2, 472, 472)]+ assert md.decoder_rows.tolist() == list(range(172, 300)) + list(range(428 + 88, 556))+ assert md.decoder_pad and md.last_indices.tolist() == [127, 168, 167] # 168: the placeholder after 168 decoder rows+++def test_prefill_metadata_bounded_mode_without_any_decoder_row():+ backend, _, _ = _stack("bounded")+ batch = _prefill_batch([_req(0, 0, 256, prompt_len=1024)])+ backend.prepare_metadata(batch)+ md = batch.attn_metadata+ assert md.decoder_segments == [] and md.decoder_rows.numel() == 0+ assert md.decoder_pad and md.last_indices.tolist() == [0]+++def test_prefill_metadata_short_prompt_runs_the_decoder_everywhere():+ """A prompt shorter than the window floors the decoder's window at 0: it sees what exact mode sees."""+ backend, _, _ = _stack("bounded")+ batch = _prefill_batch([_req(0, 0, 50)])+ backend.prepare_metadata(batch)+ md = batch.attn_metadata+ assert [(d.offset, d.n, d.start_pos, d.window_floor) for d in md.decoder_segments] == [(0, 50, 0, 0)]+ assert not md.decoder_pad and md.last_indices.tolist() == [49]+++def test_private_window_layers_address_per_request_rings():+ """Under bounded replay the decoder layers keep their window KV in per-request rings (slot =+ table row * P + pos % P), never in the radix-shared window pages; the encoder layers stay on+ the shared pool."""+ from freetoken.attention.dsv41_sparse import PrefillSegment++ backend, pool, _ = _stack("bounded")+ assert pool.private_window_layer_ids == (3, 4) and not any(pool.is_private_window(l) for l in (0, 1, 2))+ assert pool.window_pool[3].shape[0] == (MRR + 1) * P and pool.window_pool[0].shape[0] == pool.sizes.n_win_slots+ # prefill: a decoder segment's slots are its ring row; an encoder segment's are page-bound+ seg = PrefillSegment(0, 44, 2, 256, window_floor=256)+ assert backend.layer_window_slots_of(3, 2, 256, 300).tolist() == [2 * P + p % P for p in range(256, 300)]+ assert backend.layer_window_slots_of(0, 2, 256, 300).tolist() == list(range(256, 300))+ g = backend.window_topk_prefill(seg, layer_id=3)[0]+ assert g.shape == (44, P) and g[0, 0].item() == 2 * P + 0 and (g[0, 1:] == -1).all() # position 256 sees itself only (floor)+ assert set(g[43].tolist()) - {-1} == {2 * P + p % P for p in range(256, 300)}+ # decode: the private ring context keys on the batch row's page-table row+ batch = _decode_batch([2], [300])+ backend.prepare_metadata(batch)+ md = batch.attn_metadata+ slots, topk = md.private_window_ctx(torch.tensor([300]), torch.arange(1))+ assert slots.tolist() == [2 * P + 300 % P] and topk.shape == (1, 1, P)+ assert set(topk[0, 0].tolist()) == {2 * P + j for j in range(P)} # every ring index holds a position <= 300+++def test_window_candidates_honor_the_floor_and_the_retained_prefix():+ from freetoken.attention.dsv41_sparse import PrefillSegment++ backend, pool, _ = _stack()+ # a cold segment at positions [0, 10): query p sees [0, p]+ g = backend.window_topk_prefill(PrefillSegment(0, 10, 2, 0))+ assert g.shape == (1, 10, P)+ assert g[0, 3, :4].tolist() == [0, 1, 2, 3] and (g[0, 3, 4:] == -1).all()+ # an extend at [256, 300) sees back into the retained prefix, but not below the floor+ g = backend.window_topk_prefill(PrefillSegment(0, 44, 2, 256, window_floor=200))+ row = g[0, 0] # query at 256 -> [200, 256]+ assert row[0].item() == 200 and row[56].item() == 256 and (row[57:] == -1).all()+ # without a floor the query at 256 sees [129, 256]+ g = backend.window_topk_prefill(PrefillSegment(0, 44, 2, 256))+ assert g[0, 0, 0].item() == 129 and g[0, 0, 127].item() == 256+++def _decode_batch(rows, positions):+ reqs = [_req(int(t), 0, 1, uid=i) for i, t in enumerate(rows)]+ batch = Batch(reqs=reqs, phase="decode")+ batch.padded_reqs = reqs+ batch.active_table_idx = torch.tensor(rows, dtype=torch.int64)+ batch.positions = torch.tensor(positions, dtype=torch.int64)+ return batch+++def test_decode_snapshot_and_ring_context():+ backend, pool, pt = _stack()+ batch = _decode_batch([2, MRR], [259, 0])+ backend.prepare_metadata(batch)+ md = batch.attn_metadata+ assert md.full_snap is None+ snap = md.full_snapshot()+ assert torch.equal(snap[0, :300], torch.arange(300))+ pt[2, :10] = 7 # a later mutation must not reach the snapshot+ assert torch.equal(snap[0, :300], torch.arange(300))+ pos, rows = batch.positions, torch.arange(2)+ ws, prev, topk = md.window_ctx(pos, rows)+ dummy_ws = pool.sizes.n_win_slots - P # the dummy row's full loc binds to the dummy window page+ assert ws.tolist() == [259, dummy_ws] and prev.tolist() == [258, dummy_ws]+ assert topk.shape == (2, 1, P) and (topk[0] >= 0).all()+ assert set(topk[0, 0].tolist()) == set(range(259 - 127, 260))+ assert topk[1, 0, 0].item() == dummy_ws and (topk[1, 0, 1:] == -1).all()+ # the indexer derives compressed rows from the live locs: prefill off the page table row,+ # decode off the snapshot (position t -> full_loc(t * ratio) // ratio)+ assert torch.equal(backend.locs_prefill(2, 20), pt[2:3, :20])+ assert snap[0, :16].tolist() == list(range(16)) # taken before the mutation above+ # capture staging reuses one buffer+ backend.init_capture_graph(512, [1, 2])+ backend.prepare_for_replay(batch)+ assert batch.attn_metadata.full_snap.data_ptr() == backend.capture.full_snap.data_ptr()+ assert batch.attn_metadata.stage_width == 512++[email protected](not torch.cuda.is_available(), reason="the selection kernels need CUDA")+def test_topk_helpers():+ from freetoken.attention.dsv41_sparse import DSV41SparseAttnBackend as B++ scores = torch.tensor([[[0.1, 5.0, 3.0, 7.0, 9.0]]], device="cuda") # columns past live are never read+ live = torch.tensor([[3]], device="cuda", dtype=torch.int32)+ assert B.select_topk(scores, live, 4).tolist() == [[[0, 1, 2, -1]]]+ assert B.select_topk(scores, live, 2).tolist() == [[[1, 2]]]+ # a candidate list is a sorted valid prefix; empty slots score -inf+ cand = torch.tensor([[[2, 7, 9, -1]]], device="cuda", dtype=torch.int32)+ cscores = torch.tensor([[[1.0, 4.0, -torch.inf, -torch.inf]]], device="cuda")+ assert B.select_topk_in_candidates(cscores, cand, 3).tolist() == [[[2, 7, -1]]]+ # rows come from the live locs: compressed position p -> locs[p * ratio] // ratio+ locs = torch.arange(20, 40, device="cuda", dtype=torch.int32).view(1, 20)+ locs[0, 14:16] = -1 # position 7 (ratio 2) has slid out+ rows = B.positions_to_rows(torch.tensor([[[2, 7, 9, -1]]], device="cuda", dtype=torch.int32), locs, 2)+ assert rows.tolist() == [[[12, -1, 19, -1]]]diff --git a/tests/checkpoint/test_ftw_hotfix_vision.py b/tests/checkpoint/test_ftw_hotfix_vision.pyindex d4a3eac6..eb0c592f 100644--- a/tests/checkpoint/test_ftw_hotfix_vision.py+++ b/tests/checkpoint/test_ftw_hotfix_vision.py@@ -14,7 +14,7 @@ _SCRIPT = pathlib.Path(__file__).resolve().parents[2] / "scripts" / "ftw_hotfix.# env var -> local checkpoint, as tests/README.md lists them_CHECKPOINTS = ("FREETOKEN_QWEN3VL_MODEL", "FREETOKEN_GEMMA4_MODEL", "FREETOKEN_GEMMA4_UNIFIED_MODEL",- "FREETOKEN_GLM53_MODEL", "FREETOKEN_MUSE_MODEL", "FREETOKEN_MINIMAX_M3_MODEL")+ "FREETOKEN_GLM53_MODEL", "FREETOKEN_MUSE_MODEL", "FREETOKEN_MINIMAX_M3_MODEL", "FREETOKEN_DSV41_MODEL")@pytest.fixture(scope="module")@@ -34,6 +34,10 @@ def hotfix():("vision_tower.vision_model.embeddings.patch_embedding.weight", True),("multi_modal_projector.linear_1.weight", True),("patch_merge_mlp.linear_2.bias", True),+ ("vision.blocks.0.norm1.weight", True),+ ("aligner.w1.weight", True),+ ("image_newline", True),+ ("layers.3.ffn.gate.bias_vl", True),("model.embed_audio.embedding_projection.weight", False),("model.language_model.layers.0.self_attn.q_proj.weight", False),("model.layers.3.mlp.experts.0.gate_proj.weight", False),diff --git a/tests/dsv4/test_dsv4_attn_backend.py b/tests/dsv4/test_dsv4_attn_backend.pyindex 8ffc6fd4..7bc72210 100644--- a/tests/dsv4/test_dsv4_attn_backend.py+++ b/tests/dsv4/test_dsv4_attn_backend.py@@ -11,8 +11,8 @@ from __future__ import annotationsimport torchfrom freetoken.core import Batch, Context, Req, SamplingParams, get_global_ctx, set_global_ctx-from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache-from freetoken.kvcache.dsv4_cost_model import dsv4_pool_sizes+from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache+from freetoken.kvcache.dsv4.v4_cost_model import dsv4_pool_sizesfrom freetoken.models.deepseek_v4.args import DeepseekV4ArgsP, MRR, DEVICE = 128, 4, torch.device("cpu")diff --git a/tests/kernels/test_dsv41_indexer.py b/tests/kernels/test_dsv41_indexer.pynew file mode 100644index 00000000..19b45ab8--- /dev/null+++ b/tests/kernels/test_dsv41_indexer.py@@ -0,0 +1,139 @@+"""DSV41 indexer logits over packed fp4 index keys: full-range (Full mode) and candidate-restricted+(Reindex mode) against a torch reference on the dequantized keys, with causality from ``live``. The+row of a compressed position comes from the request's full-token locs (``locs[t * ratio] // ratio``);+in Full mode only the live columns are written."""++from __future__ import annotations++import pytest+import torch++from freetoken.kvcache.dsv4.v41_row_format import FP4_E8M0_B32++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++D, H = 128, 32+ROWS = 4200+++RATIO = 2+++def _setup(b, s, t, seed):+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows++ torch.manual_seed(seed)+ keys = torch.randn(ROWS, D, device="cuda", dtype=torch.bfloat16)+ pool = pack_rows(keys, FP4_E8M0_B32)+ deq = unpack_rows(pool, FP4_E8M0_B32, D)+ q = torch.randn(b, s, H, D, device="cuda", dtype=torch.bfloat16)+ w = (torch.randn(b, s, H, device="cuda") * 0.3).bfloat16()+ # each request owns a random permutation of pool rows for its compressed positions; a few holes.+ # The kernel derives rows from full-token locs: compressed position p -> loc = row * RATIO (+ any+ # in-group offset) at locs[p * RATIO]; a hole is a negative loc.+ k_rows = torch.stack([torch.randperm(ROWS, device="cuda")[:t] for _ in range(b)]).to(torch.int32)+ k_rows[:, 5::11] = -1+ locs = torch.full((b, t * RATIO), -1, device="cuda", dtype=torch.int32)+ for i in range(b):+ for p in range(t):+ r = int(k_rows[i, p])+ if r >= 0:+ locs[i, p * RATIO : (p + 1) * RATIO] = torch.arange(r * RATIO, (r + 1) * RATIO, device="cuda", dtype=torch.int32)+ return pool, deq, q, w, k_rows, locs+++def _reference(q, w, deq, k_rows, live, positions):+ """positions: [B, S, N] compressed positions (-1 empty) -> logits [B, S, N]."""+ b, s, n = positions.shape+ out = torch.full((b, s, n), float("-inf"), device="cuda")+ for i in range(b):+ for j in range(s):+ for c in range(n):+ p = int(positions[i, j, c])+ if p < 0 or p >= int(live[i, j]):+ continue+ r = int(k_rows[i, p])+ if r < 0:+ continue+ score = (q[i, j] @ deq[r]).relu() * w[i, j]+ out[i, j, c] = score.sum()+ return out+++def test_full_range_matches_reference_with_causal_live():+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed++ b, s, t = 2, 5, 300+ pool, deq, q, w, k_rows, locs = _setup(b, s, t, seed=1)+ live = torch.tensor([[10, 64, 65, 200, 300], [0, 1, 128, 129, 250]], device="cuda", dtype=torch.int32)+ out = torch.full((b, s, t), 12345.0, device="cuda") # a sentinel: the dead area must stay untouched+ got = indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, RATIO, live, out=out)+ positions = torch.arange(t, device="cuda").view(1, 1, t).expand(b, s, t)+ want = _reference(q, w, deq, k_rows, live, positions)+ assert got.shape == (b, s, t)+ written = positions < live.unsqueeze(-1)+ touched = positions < (torch.div(live + 63, 64, rounding_mode="floor") * 64).unsqueeze(-1) # whole live tiles+ assert (got[~touched] == 12345.0).all() and torch.isneginf(got[touched & ~written]).all()+ assert torch.equal(torch.isinf(got[written]), torch.isinf(want[written]))+ finite = written & ~torch.isinf(want)+ torch.testing.assert_close(got[finite], want[finite], atol=2e-2, rtol=2e-2)+++def test_quantized_queries_preserve_reference_score_rounding():+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows++ torch.manual_seed(23)+ t, s, topk = 4096, 32, 512+ pool = pack_rows(torch.randn(t, D, device="cuda", dtype=torch.bfloat16), FP4_E8M0_B32)+ keys = unpack_rows(pool, FP4_E8M0_B32, D)+ q = torch.randn(s * H, D, device="cuda", dtype=torch.bfloat16)+ q = unpack_rows(pack_rows(q, FP4_E8M0_B32), FP4_E8M0_B32, D).view(1, s, H, D)+ w = torch.randn(1, s, H, device="cuda", dtype=torch.bfloat16) * (D * H) ** -0.5+ locs = torch.arange(t, device="cuda", dtype=torch.int32).view(1, t)+ live = torch.full((1, s), t, device="cuda", dtype=torch.int32)+ got = indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, 1, live)+ reference = torch.einsum("bshd,btd->bsht", q, keys.unsqueeze(0))+ reference = (reference.relu() * w.unsqueeze(-1)).sum(dim=2).float()+ torch.testing.assert_close(got, reference, atol=0, rtol=0)+ # BF16 scores can tie at the cutoff; compare selected scores, not arbitrary tie order.+ torch.testing.assert_close(got.topk(topk).values, reference.topk(topk).values, atol=0, rtol=0)+++def test_full_range_worker_grid_covers_wide_histories(monkeypatch):+ """More live tiles than worker programs: each worker strides over its share."""+ from freetoken.kernel.triton.dsv41 import indexer as mod++ b, s, t = 1, 2, 4096 # 64 tiles+ pool, deq, q, w, k_rows, locs = _setup(b, s, t, seed=4)+ live = torch.tensor([[4096, 4000]], device="cuda", dtype=torch.int32)+ from types import SimpleNamespace++ properties = torch.cuda.get_device_properties(q.device)+ with monkeypatch.context() as patch:+ patch.setattr(torch.cuda, "get_device_properties", lambda _: SimpleNamespace(+ multi_processor_count=5, major=properties.major, minor=properties.minor,+ ))+ got = mod.indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, RATIO, live)+ positions = torch.arange(t, device="cuda").view(1, 1, t).expand(b, s, t)+ want = _reference(q, w, deq, k_rows, live, positions)+ written = positions < live.unsqueeze(-1)+ assert torch.equal(torch.isinf(got[written]), torch.isinf(want[written]))+ finite = written & ~torch.isinf(want)+ torch.testing.assert_close(got[finite], want[finite], atol=2e-2, rtol=2e-2)+++def test_candidate_pool_matches_reference():+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed++ b, s, t, nc = 2, 3, 400, 96+ pool, deq, q, w, k_rows, locs = _setup(b, s, t, seed=2)+ live = torch.tensor([[400, 350, 33], [1, 400, 200]], device="cuda", dtype=torch.int32)+ cand = torch.randint(0, t, (b, s, nc), device="cuda", dtype=torch.int32)+ cand[..., -10:] = -1 # empty tail of the pool+ got = indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, RATIO, live, candidates=cand)+ want = _reference(q, w, deq, k_rows, live, cand)+ assert got.shape == (b, s, nc)+ assert torch.equal(torch.isinf(got), torch.isinf(want))+ finite = ~torch.isinf(want)+ torch.testing.assert_close(got[finite], want[finite], atol=2e-2, rtol=2e-2)diff --git a/tests/kernels/test_dsv41_pack.py b/tests/kernels/test_dsv41_pack.pynew file mode 100644index 00000000..176889dc--- /dev/null+++ b/tests/kernels/test_dsv41_pack.py@@ -0,0 +1,112 @@+"""Packed KV row formats (``kvcache/dsv4/v41_row_format.py`` + ``kernel/triton/dsv41/pack.py``) against a torch+transcription of the reference quantizers in DeepSeek-V4.1's ``inference/kernel.py``: the round-tripped+value the pool hands back must equal what the reference bakes into its bf16 cache, bit for bit."""++from __future__ import annotations++import pytest+import torch++from freetoken.kvcache.dsv4.v41_row_format import BF16, FP4_E4M3_B16, FP4_E8M0_B32, FP8_E8M0_B32, RowFormat++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++FP4_GRID = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])+++def _round_fp4(x: torch.Tensor) -> torch.Tensor:+ """Round-to-nearest-even onto the signed e2m1 grid (the ``float4_e2m1fn`` cast)."""+ a = x.abs()+ grid = FP4_GRID.to(x.device)+ # nearest grid point; ties go to the even-mantissa neighbour (0, 1, 2, 4 are even)+ dist = (a[..., None] - grid).abs()+ lo = torch.searchsorted(grid, a.reshape(-1).contiguous(), right=True).reshape(a.shape).clamp(1, 7)+ below, above = grid[lo - 1], grid[lo]+ mid = (below + above) / 2+ even_above = torch.isin(above, torch.tensor([0.0, 1.0, 2.0, 4.0], device=x.device))+ pick_above = (a > mid) | ((a == mid) & even_above)+ r = torch.where(pick_above, above, below)+ r = torch.where(a >= 6.0, torch.full_like(r, 6.0), r)+ del dist+ return torch.copysign(r, x)+++def reference_roundtrip(x: torch.Tensor, fmt: RowFormat) -> torch.Tensor:+ """``act_quant`` / ``fp4_act_quant`` with ``inplace=True``: quant + dequant back to bf16."""+ m, d = x.shape+ if fmt is BF16:+ return x.to(torch.bfloat16)+ g = x.float().view(m, d // fmt.group, fmt.group)+ amax = g.abs().amax(dim=-1)+ if fmt is FP8_E8M0_B32:+ s = torch.exp2(torch.ceil(torch.log2(amax.clamp_min(1e-4) / 448.0)))+ q = (g / s[..., None]).clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float()+ elif fmt is FP4_E8M0_B32:+ s = torch.exp2(torch.ceil(torch.log2(amax.clamp_min(6.0 * 2.0**-126) / 6.0)))+ q = _round_fp4((g / s[..., None]).clamp(-6.0, 6.0))+ elif fmt is FP4_E4M3_B16:+ s = (amax.clamp_min(6.0 * 2.0**-9) / 6.0).to(torch.float8_e4m3fn).float()+ q = _round_fp4((g / s[..., None]).clamp(-6.0, 6.0))+ else:+ raise AssertionError(fmt)+ return (q * s[..., None]).view(m, d).to(torch.bfloat16)++[email protected]("fmt", [BF16, FP8_E8M0_B32, FP4_E4M3_B16, FP4_E8M0_B32], ids=lambda f: f.name)[email protected]("dim", [128, 512])+def test_roundtrip_matches_the_reference_quantizer(fmt: RowFormat, dim: int):+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows++ torch.manual_seed(0)+ m = 300+ # mixed magnitudes, exact zeros, a tiny group and a huge one+ x = torch.randn(m, dim, device="cuda") * torch.exp2(torch.randint(-8, 6, (m, 1), device="cuda").float())+ x[7] = 0+ x[11, : fmt.group or 32] *= 1e-6+ x[13, -32:] = 400.0+ x = x.to(torch.bfloat16)++ packed = pack_rows(x, fmt)+ assert packed.shape == (m, fmt.row_bytes(dim)) and packed.dtype == torch.uint8+ got = unpack_rows(packed, fmt, dim)+ want = reference_roundtrip(x, fmt)+ assert torch.equal(got, want), f"{fmt.name}: {(got.float() - want.float()).abs().max().item()}"+++def test_scatter_into_pool_rows_and_gather_back():+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows++ torch.manual_seed(1)+ fmt, dim, rows = FP4_E4M3_B16, 512, 64+ pool = torch.zeros((rows, fmt.row_bytes(dim)), dtype=torch.uint8, device="cuda")+ x = torch.randn(5, dim, device="cuda", dtype=torch.bfloat16)+ ids = torch.tensor([3, 60, 0, 17, 42], device="cuda", dtype=torch.int64)+ assert pack_rows(x, fmt, pool=pool, row_ids=ids) is pool+ got = unpack_rows(pool, fmt, dim, row_ids=ids.to(torch.int32))+ assert torch.equal(got, reference_roundtrip(x, fmt))+ untouched = torch.ones(rows, dtype=torch.bool, device="cuda")+ untouched[ids] = False+ assert int(pool[untouched].sum()) == 0+++def test_e4m3_encode_helper_matches_torch_bits():+ """``e4m3_f32_to_u8`` (the pre-sm_89 scale encoder) on every finite e4m3 code."""+ import triton+ import triton.language as tl++ from freetoken.kernel.triton.e4m3_compat import e4m3_f32_to_u8++ @triton.jit+ def _enc(x_ptr, o_ptr, N: tl.constexpr):+ offs = tl.arange(0, N)+ tl.store(o_ptr + offs, e4m3_f32_to_u8(tl.load(x_ptr + offs)))++ codes = torch.arange(256, dtype=torch.uint8, device="cuda")+ finite = (codes & 0x7F) != 0x7F # skip the NaN codes+ vals = codes.view(torch.float8_e4m3fn).float()+ vals = torch.where(finite, vals, torch.zeros_like(vals))+ out = torch.empty(256, dtype=torch.uint8, device="cuda")+ _enc[(1,)](vals, out, N=256)+ want = torch.where(finite, codes, torch.zeros_like(codes))+ # -0.0 encodes to 0x80 in torch and 0x80 here; +0.0 -> 0x00+ assert torch.equal(out, want)diff --git a/tests/kernels/test_dsv41_sparse_attn.py b/tests/kernels/test_dsv41_sparse_attn.pynew file mode 100644index 00000000..8a4f5f2a--- /dev/null+++ b/tests/kernels/test_dsv41_sparse_attn.py@@ -0,0 +1,107 @@+"""DSV41 sparse attention over packed pools against a torch reference on the DEQUANTIZED pools.++``m == 1`` takes the split-k decode path, ``m > 1`` the single-program prefill one. Tolerance follows+the dsv4 sparse-attention tests (5e-2 on fp32-cast bf16): the online-softmax accumulation order differs+from the reference and split-k merges through log-sum-exp on top of that."""++from __future__ import annotations++import pytest+import torch++from freetoken.kvcache.dsv4.v41_row_format import BF16, FP4_E4M3_B16, FP8_E8M0_B32++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++D, H = 128, 8+N_WINDOW = 128 # a multiple of BLOCK_T+N_WIN_SLOTS, N_CMP = 512, 1024+TOL = dict(atol=5e-2, rtol=5e-2)+++def _reference(q, win, cmp, sink, idx, n_window, scale, counts):+ """Per (request, query, head) softmax over the live columns; ``sink`` is a null key."""+ b, m, h, d = q.shape+ out = torch.zeros_like(q, dtype=torch.float32)+ for i in range(b):+ for j in range(m):+ width = idx.shape[-1] if counts is None else n_window + int(counts[i, j])+ kv = []+ for c in range(min(width, idx.shape[-1])):+ slot = int(idx[i, j, c])+ if slot >= 0:+ kv.append((win if c < n_window else cmp)[slot].float())+ if not kv:+ continue+ kv_t = torch.stack(kv)+ logits = q[i, j].float() @ kv_t.T * scale+ mx = torch.maximum(logits.max(dim=-1).values, sink.float())+ probs = torch.exp(logits - mx[:, None])+ denom = probs.sum(dim=-1) + torch.exp(sink.float() - mx)+ out[i, j] = (probs @ kv_t) / denom[:, None]+ return out+++def _pools(win_fmt, cmp_fmt, seed):+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows++ torch.manual_seed(seed)+ win_vals = torch.randn(N_WIN_SLOTS, D, device="cuda", dtype=torch.bfloat16)+ cmp_vals = torch.randn(N_CMP, D, device="cuda", dtype=torch.bfloat16)+ win_pool = pack_rows(win_vals, win_fmt)+ cmp_pool = pack_rows(cmp_vals, cmp_fmt)+ return win_pool, cmp_pool, unpack_rows(win_pool, win_fmt, D), unpack_rows(cmp_pool, cmp_fmt, D)+++def _topk(b, m, n_cmp_cols, seed, ring_fill=None):+ torch.manual_seed(seed)+ win = torch.randint(0, N_WIN_SLOTS, (b, m, N_WINDOW), device="cuda", dtype=torch.int32)+ if ring_fill is not None: # a ring still filling: trailing window columns empty+ win[..., ring_fill:] = -1+ cmp = torch.randint(0, N_CMP, (b, m, n_cmp_cols), device="cuda", dtype=torch.int32)+ cmp[..., ::7] = -1 # scattered unreachable picks+ return torch.cat([win, cmp], dim=-1)++[email protected]("m", [1, 3])[email protected]("fmts", [(FP8_E8M0_B32, FP4_E4M3_B16), (BF16, BF16)], ids=["fp8+fp4", "bf16"])+def test_matches_reference_on_dequantized_pools(m, fmts):+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed++ win_fmt, cmp_fmt = fmts+ b, n_cmp_cols = 2, 256+ win_pool, cmp_pool, win_deq, cmp_deq = _pools(win_fmt, cmp_fmt, seed=1)+ q = torch.randn(b, m, H, D, device="cuda", dtype=torch.bfloat16)+ sink = torch.randn(H, device="cuda") * 0.5+ idx = _topk(b, m, n_cmp_cols, seed=2, ring_fill=100)+ scale = D**-0.5++ got = sparse_attn_packed(q, win_pool, win_fmt, cmp_pool, cmp_fmt, sink, idx, N_WINDOW, scale)+ want = _reference(q, win_deq, cmp_deq, sink, idx, N_WINDOW, scale, None)+ torch.testing.assert_close(got.float(), want, **TOL)+++def test_cmp_counts_bound_the_visited_columns():+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed++ b, m, n_cmp_cols = 3, 1, 512+ win_pool, cmp_pool, win_deq, cmp_deq = _pools(FP8_E8M0_B32, FP4_E4M3_B16, seed=3)+ q = torch.randn(b, m, H, D, device="cuda", dtype=torch.bfloat16)+ sink = torch.zeros(H, device="cuda")+ idx = _topk(b, m, n_cmp_cols, seed=4)+ counts = torch.tensor([[0], [37], [512]], device="cuda", dtype=torch.int32)+ scale = D**-0.5++ got = sparse_attn_packed(q, win_pool, FP8_E8M0_B32, cmp_pool, FP4_E4M3_B16, sink, idx, N_WINDOW, scale, cmp_counts=counts)+ want = _reference(q, win_deq, cmp_deq, sink, idx, N_WINDOW, scale, counts)+ torch.testing.assert_close(got.float(), want, **TOL)+++def test_all_masked_query_yields_zeros():+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed++ win_pool, cmp_pool, _, _ = _pools(FP8_E8M0_B32, FP4_E4M3_B16, seed=5)+ q = torch.randn(1, 2, H, D, device="cuda", dtype=torch.bfloat16)+ idx = torch.full((1, 2, N_WINDOW + 64), -1, device="cuda", dtype=torch.int32)+ out = sparse_attn_packed(q, win_pool, FP8_E8M0_B32, cmp_pool, FP4_E4M3_B16, torch.zeros(H, device="cuda"), idx, N_WINDOW, 0.1)+ assert torch.equal(out, torch.zeros_like(out))diff --git a/tests/kernels/test_dsv41_topk.py b/tests/kernels/test_dsv41_topk.pynew file mode 100644index 00000000..e7fc66f8--- /dev/null+++ b/tests/kernels/test_dsv41_topk.py@@ -0,0 +1,159 @@+"""dsv41_topk / dsv41_candidate_blocks against an independent torch reference with the documented tie+rule, over the contract's edge cases: empty and short histories, unequal live counts in one batch,+-inf holes, widths crossing the slice and level boundaries, exact ties, newest-block inclusion,+the sorted-valid-prefix / -1-tail output layout, and CUDA-graph replay across changing live lengths+(stale workspace reads would surface as wrong picks)."""++from __future__ import annotations++import pytest+import torch++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")+++def ref_topk(scores: torch.Tensor, live: torch.Tensor, k: int) -> torch.Tensor:+ """(score desc, column asc) over the live, finite columns; ascending columns, -1 padded."""+ R, T = scores.shape+ out = torch.full((R, k), -1, dtype=torch.int32)+ for r in range(R):+ n = int(live[r])+ s = scores[r, :n].double().cpu()+ cols = torch.arange(n)+ alive = torch.isfinite(s) | (s == float("inf"))+ cand = [(float(-s[c]), int(c)) for c in cols[alive]]+ cand.sort()+ picked = sorted(c for _, c in cand[:k])+ out[r, : len(picked)] = torch.tensor(picked, dtype=torch.int32)+ return out.to(scores.device)+++def ref_candidates(scores, live, topk_blocks, block_size):+ R, T = scores.shape+ nb = -(-T // block_size)+ out = torch.full((R, topk_blocks * block_size), -1, dtype=torch.int32)+ for r in range(R):+ n = int(live[r])+ if n == 0:+ continue+ nbl = -(-n // block_size)+ s = torch.nn.functional.pad(scores[r, :n].float().cpu(), (0, nbl * block_size - n), value=float("-inf")).view(nbl, block_size).amax(-1)+ s[nbl - 1] = float("inf")+ cand = sorted((float(-s[b]), b) for b in range(nbl) if s[b] > float("-inf"))+ keep = sorted(b for _, b in cand[:topk_blocks])+ pos = [p for b in keep for p in range(b * block_size, (b + 1) * block_size) if p < n]+ out[r, : len(pos)] = torch.tensor(pos, dtype=torch.int32)+ return out.to(scores.device)+++def _scores(R, T, seed=0, ties=False):+ g = torch.Generator(device="cpu").manual_seed(seed)+ s = torch.randn(R, T, generator=g)+ if ties:+ s = torch.randint(0, 5, (R, T), generator=g).float() # heavy exact ties+ return s.cuda()++[email protected]("T,k", [(64, 8), (4096, 512), (4097, 512), (8192, 512), (100_000, 512), (262_144, 512), (1_048_576, 512), (131_072, 2048)])+def test_topk_matches_reference_across_widths(T, k):+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk++ R = 3+ scores = _scores(R, T, seed=T)+ live = torch.tensor([0, min(T, 5000), T], dtype=torch.int32, device="cuda")[:R]+ # holes inside the live range+ scores[2, ::7] = float("-inf")+ got = dsv41_topk(scores, live, k)+ want = ref_topk(scores, live, k)+ assert torch.equal(got, want), (got[:, :8], want[:, :8])+++def test_topk_short_and_unequal_rows_and_exact_ties():+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk++ T, k = 20_000, 512+ scores = _scores(5, T, seed=3, ties=True)+ live = torch.tensor([0, 1, 300, 4096, 20_000], dtype=torch.int32, device="cuda")+ got = dsv41_topk(scores, live, k)+ want = ref_topk(scores, live, k)+ assert torch.equal(got, want)+ # the layout contract: a sorted valid prefix, then -1+ for r in range(5):+ row = got[r].tolist()+ n = sum(1 for v in row if v >= 0)+ assert row[n:] == [-1] * (k - n) and row[:n] == sorted(row[:n]) and n == min(int(live[r]), k)+++def test_topk_all_dead_row_and_fewer_finite_than_k():+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk++ scores = torch.full((2, 10_000), float("-inf"), device="cuda")+ scores[1, [10, 4095, 4096, 9000]] = torch.tensor([1.0, 2.0, 3.0, 0.5], device="cuda")+ live = torch.tensor([10_000, 9001], dtype=torch.int32, device="cuda")+ got = dsv41_topk(scores, live, 8)+ assert (got[0] == -1).all()+ assert got[1].tolist() == [10, 4095, 4096, 9000, -1, -1, -1, -1]++[email protected]("T", [4000, 200_000])+def test_candidate_blocks_match_reference_and_pin_the_newest_block(T):+ from freetoken.kernel.triton.dsv41.topk import dsv41_candidate_blocks++ R, KB, BS = 4, 2048, 8+ scores = _scores(R, T, seed=T + 1)+ scores[0, :] = -1000.0 # the newest block is kept even when every score is poor+ live = torch.tensor([13, 8, min(T, 16_000), T], dtype=torch.int32, device="cuda")+ got = dsv41_candidate_blocks(scores, live, KB, BS)+ want = ref_candidates(scores, live, KB, BS)+ assert got.shape == (R, KB * BS)+ assert torch.equal(got, want)+ # short histories: every live position is a candidate (the implicit fast path)+ assert got[0].tolist()[:13] == list(range(13)) and (got[0, 13:] == -1).all()+ assert got[1].tolist()[:8] == list(range(8)) and (got[1, 8:] == -1).all()+++def test_topk_under_graph_replay_across_live_lengths():+ """Capture once at the staged width, replay with live lengths that move up and down: every replay+ must equal the reference (a stale candidate slot or live count from a previous replay would not)."""+ from freetoken.kernel.triton.dsv41.topk import dsv41_candidate_blocks, dsv41_topk++ T, k, R = 300_000, 512, 2+ scores = torch.empty((R, T), device="cuda")+ live = torch.empty((R,), dtype=torch.int32, device="cuda")+ out = torch.empty((R, k), dtype=torch.int32, device="cuda")+ cand = torch.empty((R, 2048 * 8), dtype=torch.int32, device="cuda")+ stream = torch.cuda.Stream()+ with torch.cuda.stream(stream):+ scores.copy_(_scores(R, T, seed=9))+ live.fill_(T)+ dsv41_topk(scores, live, k, out) # warm-up compiles+ cand.copy_(dsv41_candidate_blocks(scores, live, 2048, 8))+ torch.cuda.synchronize()+ graph = torch.cuda.CUDAGraph()+ with torch.cuda.graph(graph, stream=stream):+ dsv41_topk(scores, live, k, out)+ cand.copy_(dsv41_candidate_blocks(scores, live, 2048, 8))+ torch.cuda.synchronize()+ for step, (l0, l1) in enumerate([(T, 7), (4096, 4097), (0, 1), (250_000, 12_000), (3, T), (16_384, 16_385)]):+ scores.copy_(_scores(R, T, seed=100 + step))+ live.copy_(torch.tensor([l0, l1], dtype=torch.int32))+ graph.replay()+ torch.cuda.synchronize()+ assert torch.equal(out, ref_topk(scores, live, k)), (step, l0, l1)+ assert torch.equal(cand, ref_candidates(scores, live, 2048, 8)), (step, l0, l1)+++def test_topk_plan_terminates_or_rejects():+ """Every accepted k converges (each level at least halves the row); k past CHUNK // 2 -- which+ would stall the planner (k == CHUNK never shrinks, 3072 stalls at 6144) -- is rejected up front."""+ from freetoken.kernel.triton.dsv41.topk import CHUNK, FINAL_MAX, MAX_K, topk_plan++ for k in (1, 8, 512, 2048, MAX_K):+ for columns in (1, 4096, 4097, 1 << 20, (1 << 20) + 3):+ levels = topk_plan(columns, k)+ widths = [w for w, _ in levels] + [(levels[-1][1] * k) if levels else columns]+ assert all(a > b for a, b in zip(widths, widths[1:])) and widths[-1] <= FINAL_MAX+ assert len(levels) <= 12+ for k in (MAX_K + 1, 2049, 3072, CHUNK, 0, -1):+ with pytest.raises(ValueError):+ topk_plan(1 << 20, k)diff --git a/tests/kernels/test_dsv4_fp8_block_linear.py b/tests/kernels/test_dsv4_fp8_block_linear.pynew file mode 100644index 00000000..2194a65e--- /dev/null+++ b/tests/kernels/test_dsv4_fp8_block_linear.py@@ -0,0 +1,101 @@+"""``kernel/triton/dsv4/fp8_linear.block_fp8_linear`` against a torch transcription of the reference+``act_quant`` + ``fp8_gemm`` (ue8m0 activation scales, ``block x block`` e8m0 weight scales), for the+DeepSeek-V4 block (128) and the DeepSeek-V4.1 block (32), on the GEMV (M=1) and GEMM (M>1) paths."""++from __future__ import annotations++import pytest+import torch++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++FP8 = torch.float8_e4m3fn+E8M0 = torch.float8_e8m0fnu+++def _pow2_ceil_scale(amax: torch.Tensor, qmax: float) -> torch.Tensor:+ """``2 ** ceil(log2(amax / qmax))`` -- the reference ``fast_round_scale``."""+ return torch.exp2(torch.ceil(torch.log2(amax / qmax)))+++def quantize_weight(w: torch.Tensor, block: int) -> tuple[torch.Tensor, torch.Tensor]:+ """Per ``block x block`` ue8m0-scaled e4m3 weight, scale as e8m0 codes."""+ n, k = w.shape+ blocks = w.float().view(n // block, block, k // block, block).permute(0, 2, 1, 3)+ amax = blocks.abs().amax(dim=(-1, -2)).clamp_min(1e-4)+ scale = _pow2_ceil_scale(amax, 448.0) # [n//block, k//block]+ q = (blocks / scale[..., None, None]).clamp(-448.0, 448.0).to(FP8)+ w_fp8 = q.permute(0, 2, 1, 3).reshape(n, k)+ return w_fp8, scale.to(E8M0)+++def act_quant_reference(x: torch.Tensor, block: int) -> tuple[torch.Tensor, torch.Tensor]:+ m, k = x.shape+ groups = x.float().view(m, k // block, block)+ amax = groups.abs().amax(dim=-1).clamp_min(1e-4)+ scale = _pow2_ceil_scale(amax, 448.0)+ q = (groups / scale[..., None]).clamp(-448.0, 448.0).to(FP8)+ return q.view(m, k), scale+++def linear_reference(x: torch.Tensor, w_fp8: torch.Tensor, w_scale: torch.Tensor, block: int) -> torch.Tensor:+ m, k = x.shape+ n = w_fp8.shape[0]+ a_fp8, a_scale = act_quant_reference(x, block)+ a = a_fp8.float().view(m, k // block, block)+ w = w_fp8.float().view(n // block, block, k // block, block)+ ws = w_scale.float() # [n//block, k//block]+ out = torch.zeros(m, n, dtype=torch.float32, device=x.device)+ for kb in range(k // block):+ wk = w[:, :, kb, :].reshape(n, block) # [n, block]+ partial = a[:, kb, :] @ wk.t() # [m, n]+ out += partial * a_scale[:, kb, None] * ws[:, kb].repeat_interleave(block)[None, :]+ return out++[email protected]("block", [32, 128])[email protected]("m", [1, 2, 5, 8, 9, 64])+def test_block_fp8_linear_matches_reference(block: int, m: int):+ from freetoken.kernel.triton.dsv4.fp8_linear import block_fp8_linear++ torch.manual_seed(0)+ # K // block = 5 slabs exercises the GEMV split that must divide the slab count; N spans two GEMM N-tiles+ k, n = 5 * block, 256 if block == 128 else 224+ x = torch.randn(m, k, device="cuda", dtype=torch.bfloat16) * 3+ w_fp8, w_scale = quantize_weight(torch.randn(n, k, device="cuda", dtype=torch.bfloat16), block)++ got = block_fp8_linear(x, w_fp8, w_scale, block=block).float()+ want = linear_reference(x, w_fp8, w_scale, block)+ # bf16 output; the fp32 reduction order inside the MMA may differ from the slab loop+ torch.testing.assert_close(got, want, rtol=2e-2, atol=2e-2 * want.abs().max().item())+++def test_block_argument_must_match_the_scale_shape():+ from freetoken.kernel.triton.dsv4.fp8_linear import block_fp8_linear++ x = torch.randn(4, 256, device="cuda", dtype=torch.bfloat16)+ w_fp8, w_scale = quantize_weight(torch.randn(256, 256, device="cuda", dtype=torch.bfloat16), 32)+ with pytest.raises(AssertionError):+ block_fp8_linear(x, w_fp8, w_scale, block=128)++[email protected]("block", [32, 128])+def test_grouped_w8a16_gemv_matches_the_dequantized_einsum(block):+ """``wo_a`` at decode: the fp8 payload dequantized in-kernel against a bf16 activation must equal+ the reference's bf16 einsum over the dequantized weight (fp32 accumulation, order aside)."""+ from freetoken.kernel.triton.dsv4.fp8_linear import dequant_block_fp8, grouped_w8a16_gemv++ G, r, d = 4, 256, 1024+ torch.manual_seed(block)+ w = torch.randn(G * r, d, device="cuda") * 0.05+ w_fp8, scale = quantize_weight(w, block)+ x = torch.randn(G, d, device="cuda", dtype=torch.bfloat16)+ deq = dequant_block_fp8(w_fp8, scale, block=block)+ ref_deq = (w_fp8.float() * torch.exp2(scale.view(torch.uint8).float() - 127.0).repeat_interleave(block, 0).repeat_interleave(block, 1)).to(torch.bfloat16)+ assert torch.equal(deq, ref_deq)+ for t in (1, 3):+ xt = torch.randn(t, G, d, device="cuda", dtype=torch.bfloat16) if t > 1 else x.unsqueeze(0)+ want = torch.einsum("tgd,grd->tgr", xt, ref_deq.view(G, r, d)).flatten(1)+ got = grouped_w8a16_gemv(xt, w_fp8, scale, block=block)+ assert got.shape == (t, G * r) and got.dtype == torch.bfloat16+ torch.testing.assert_close(got.float(), want.float(), atol=2e-2, rtol=2e-2)diff --git a/tests/kernels/test_dsv4_router_select.py b/tests/kernels/test_dsv4_router_select.pynew file mode 100644index 00000000..9829ed03--- /dev/null+++ b/tests/kernels/test_dsv4_router_select.py@@ -0,0 +1,52 @@+"""Fused gate selection (``kernel/triton/dsv4/router``) against the torch expression it replaces, for+every score function, with and without top-k weight normalization, at decode and prefill widths."""++from __future__ import annotations++import pytest+import torch+import torch.nn.functional as F++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")+++def reference(scores, bias, topk, score_func, normalize, route_scale):+ if score_func == "softmax":+ s = scores.softmax(dim=-1)+ elif score_func == "sigmoid":+ s = scores.sigmoid()+ else:+ s = F.softplus(scores).sqrt()+ ids = (s + bias).topk(topk, dim=-1)[1]+ w = s.gather(1, ids)+ if normalize and topk > 1:+ w = w / (w.sum(dim=-1, keepdim=True) + 1e-20)+ return w * route_scale, ids++[email protected]("score_func", ["sqrtsoftplus", "sigmoid", "softmax"])[email protected]("normalize", [True, False])[email protected]("T", [1, 3, 300])+def test_router_select_matches_torch(score_func, normalize, T):+ from freetoken.kernel.triton.dsv4.router import router_select++ g = torch.Generator(device="cpu").manual_seed(T)+ E, K = 384, 6+ scores = (torch.randn(T, E, generator=g) * 3).cuda()+ bias = (torch.randn(E, generator=g) * 0.1).cuda()+ w, ids = router_select(scores, bias, K, score_func=score_func, normalize=normalize, route_scale=2.5)+ w_ref, ids_ref = reference(scores, bias, K, score_func, normalize, 2.5)+ assert torch.equal(ids.sort(dim=-1).values, ids_ref.to(torch.int32).sort(dim=-1).values)+ # same expert set; compare the weights per expert (selection order may differ only on exact ties)+ order = ids.argsort(dim=-1)+ order_ref = ids_ref.argsort(dim=-1)+ torch.testing.assert_close(w.gather(1, order), w_ref.gather(1, order_ref), atol=1e-6, rtol=1e-5)+++def test_router_select_breaks_ties_toward_the_lowest_expert():+ from freetoken.kernel.triton.dsv4.router import router_select++ scores = torch.zeros(1, 384, device="cuda") # every expert ties+ bias = torch.zeros(384, device="cuda")+ _, ids = router_select(scores, bias, 6, score_func="sigmoid", normalize=True, route_scale=1.0)+ assert ids.tolist() == [[0, 1, 2, 3, 4, 5]]diff --git a/tests/kvcache/test_cache_unit_bytes.py b/tests/kvcache/test_cache_unit_bytes.pyindex 496622f9..cce5df62 100644--- a/tests/kvcache/test_cache_unit_bytes.py+++ b/tests/kvcache/test_cache_unit_bytes.py@@ -16,7 +16,7 @@ from freetoken.kvcache.cache_status import (compute_cache_status_meta,compute_cache_unit_bytes,)-from freetoken.kvcache.dsv4_cost_model import _dsv4_window_floor_pages+from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_window_floor_pagesdef _init_tp() -> None:@@ -204,7 +204,7 @@ def test_floors_dsv4_reports_real_window_floor():dsv4_args=SimpleNamespace(window_size=P), has_swa_attention=False),)- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCacheeng = SimpleNamespace(config=cfg, moe_offload_cache=None, linear_state_pool=None)eng.kv_cache = object.__new__(DSV4PagedKVCache) # min_kv_tokens is a classmethoddiff --git a/tests/kvcache/test_dsv41_pool.py b/tests/kvcache/test_dsv41_pool.pynew file mode 100644index 00000000..f41df851--- /dev/null+++ b/tests/kvcache/test_dsv41_pool.py@@ -0,0 +1,205 @@+"""DSV41 paged KV pool + cost model (CPU, no model): args validation, per-source tiers, sizing /+byte accounting, the window free-list duck-type, full-loc translation; packed writes on CUDA."""++from __future__ import annotations++import pytest+import torch++from freetoken.kvcache.dsv4.v41_cost_model import (+ dsv41_cache_per_page,+ dsv41_kv_unit_bytes,+ dsv41_pool_bytes,+ dsv41_pool_sizes,+ dsv41_solve_num_pages,+ dsv41_window_unit_bytes,+)+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache+from freetoken.kvcache.dsv4.v41_row_format import FP4_E4M3_B16, FP4_E8M0_B32, FP8_E8M0_B32+from freetoken.models.deepseek_v41.args import DeepseekV41Args++DEVICE = torch.device("cpu")+P = 128+# a 10-layer miniature of V4.1's layout: 2 window-only, ratio-2 encoder (source 2, source 5), ratio-1 decoder (source 7)+RATIOS = (0, 0, 2, 2, 2, 2, 2, 1, 1, 1)+SOURCES = (2, 5, 7)+++def _args(**over) -> DeepseekV41Args:+ base = dict(n_layers=10, head_dim=512, index_head_dim=128, window_size=P, compress_ratios=RATIOS,+ kv_source_layers=SOURCES, swa_decoder_replay="exact")+ base.update(over)+ return DeepseekV41Args(**base)+++def _pool(num_pages=8, swa_ratio=0.5, n_scratch=1, **over):+ args = _args(**over)+ sizes = dsv41_pool_sizes(num_pages, args, swa_ratio, P)+ return DSV41PagedKVCache(sizes, args, DEVICE, n_scratch=n_scratch), sizes, args+++def test_window_control_uses_pool_units_and_restores_concrete_capacity():+ from types import SimpleNamespace++ from freetoken.attention import AttnType+ from freetoken.kvcache.cache_status import (+ _supports_swa_ratio, compute_cache_floors, compute_cache_pools,+ )+ from freetoken.kvcache.dsv4.v41_cost_model import _dsv41_pool_sizes+ from freetoken.scheduler.scheduler import Scheduler+ from freetoken.server.api_server import CacheRebuildRequest, _resolve_num_swa_pages, cache_geometry+ from freetoken.server.stats import _swa_page_size++ pool, _, args = _pool(num_pages=64)+ config = SimpleNamespace(+ page_size=P, max_running_req=1, max_seq_len=1024, cache_type="swa_radix",+ swa_full_tokens_ratio=0.5, swa_num_pages_override=None,+ model_config=SimpleNamespace(+ dsv4_args=None, dsv41_args=_args(), has_swa_attention=False,+ kv_cache_group_specs=lambda: [SimpleNamespace(attn_type=AttnType.DSV41)],+ ),+ )+ engine = SimpleNamespace(+ config=config, kv_cache=pool, num_pages=63,+ moe_offload_cache=None, linear_state_pool=None,+ )+ assert _supports_swa_ratio(config)+ assert compute_cache_pools(engine)["swa_page_size"] == P+ assert compute_cache_pools(engine)["num_swa_pages"] == 31+ floor = _dsv41_pool_sizes(config, 64, num_swa_pages=1).n_win_pages - 1+ assert compute_cache_floors(engine)["swa_tokens"] == floor * P+ prior = Scheduler._current_cache_geometry(SimpleNamespace(engine=engine, config=config))+ assert prior["num_swa_pages"] == 31+ state = SimpleNamespace(config=config, cache_pools=compute_cache_pools(engine))+ req = CacheRebuildRequest(num_pages=64, swa_full_tokens_ratio=0.5)+ assert _resolve_num_swa_pages(state, req) == 32+ assert _swa_page_size(config) == P+ state.stats = SimpleNamespace(kv_total_pages=63, mamba_total_slots=0)+ state.last_rebuild = {"num_pages": 64, "num_swa_pages": 32}+ assert cache_geometry(state)["swa_full_tokens_ratio"] == 0.5+ pool._init_paged_state(1, True)+ pool.rebuild_from_config(config, 63, num_swa_pages=24)+ assert pool.window_pages == 24+ pool.rebuild_from_config(config, prior["num_pages"], num_swa_pages=prior["num_swa_pages"])+ assert pool.window_pages == 31+++def test_pool_reads_sources_and_rings_from_args():+ pool, _, _ = _pool()+ assert [pool.source_of(l) for l in range(2, 10)] == [2, 2, 2, 5, 5, 7, 7, 7]+ assert set(pool.state_ring) == {2, 5} and pool.state_ring[2].ring_size == 2+ with pytest.raises(ValueError): # the window is not a multiple of a compress ratio+ _pool(window_size=P - 1)+ with pytest.raises(ValueError): # an indexer key the fp4 row format cannot hold+ _pool(index_head_dim=48)++[email protected]("replay", ["bounded", "exact"])+def test_v41_flash_row_bytes_and_private_window_layers(replay):+ """The tech report's headline: 3 ratio-2 encoder sources + 1 ratio-1 decoder source = 890 B/token."""+ ratios = (0, 0) + (2,) * 18 + (1,) * 20+ args = DeepseekV41Args(n_layers=40, head_dim=512, index_head_dim=128, window_size=128, compress_ratios=ratios,+ kv_source_layers=(2, 8, 14, 20), swa_decoder_replay=replay)+ pool = DSV41PagedKVCache(dsv41_pool_sizes(4, args, 1.0, 128), args, DEVICE)+ win, main, idx = pool.window_pool[0].shape[1], pool.main_pool[2].shape[1], pool.idx_pool[2].shape[1]+ assert (win, main, idx) == (528, 288, 68)+ assert pool.state_ring[2].buffer.shape[1] * 4 == 4096+ assert sum((main + idx) // pool.ratio_of(s) for s in args.backbone_kv_sources) == 890+ assert dsv41_kv_unit_bytes(args, 128) == 890 + 8 # + the int64 full->window map slot+ assert pool.private_window_layer_ids == (tuple(range(20, 40)) if replay == "bounded" else ())+++def test_pool_tiers_per_source_and_aliasing():+ pool, sizes, args = _pool()+ assert len(pool.window_pool) == 10 and pool.window_pool[0].shape == (sizes.n_win_slots, 528)+ assert set(pool.main_pool) == set(SOURCES) == set(pool.idx_pool)+ assert set(pool.state_ring) == {2, 5} # the ratio-1 decoder source carries no partial group+ assert pool.main_pool[2].shape == (sizes.full_token // 2 + 1, 288)+ assert pool.main_pool[7].shape == (sizes.full_token + 1, 288)+ assert pool.idx_pool[5].shape == (sizes.full_token // 2 + 1, 68)+ assert pool.main_pool_of(4) is pool.main_pool[2] and pool.idx_pool_of(9) is pool.idx_pool[7]+ assert pool.scratch_base[2] == sizes.full_token // 2+ with pytest.raises(AssertionError):+ pool.main_pool_of(0)+ ring = pool.state_ring[2]+ assert ring.ring_size == 2 and ring.item_size == 512 and ring.buffer.shape == (sizes.state_slots[2] + 1, 1024)+ assert torch.all(ring.buffer[-1, :512] == 0) and torch.all(torch.isneginf(ring.buffer[-1, 512:]))+++def test_pool_bytes_match_allocation_and_solver_respects_budget():+ pool, sizes, args = _pool(num_pages=16, swa_ratio=0.25)+ assert pool.total_bytes() == dsv41_pool_bytes(sizes, args, n_scratch=1)+ assert dsv41_cache_per_page(args, 0.25, P) > 0+ assert dsv41_window_unit_bytes(args, P) == -(-(10 * P * 528 + 2 * 2 * 4096) // P)+ budget = 64 << 20+ solved = dsv41_solve_num_pages(budget, args, 0.25, floor_win_pages=4, P=P, n_scratch=3)+ assert dsv41_pool_bytes(solved, args, 3) <= budget+ # one more page, sized the way the solver sizes (window = max(floor, ceil(ratio * pages))), overflows+ more = solved.full_token // P + 1+ bigger = dsv41_pool_sizes(more, args, 0.25, P, n_win_pages=max(4, (round(0.25 * more * P) + P - 1) // P))+ assert dsv41_pool_bytes(bigger, args, 3) > budget+ assert solved.n_win_pages >= 4+ with pytest.raises(ValueError):+ dsv41_solve_num_pages(1 << 10, args, 0.25, floor_win_pages=4, P=P)+++def test_translation_state_loc_and_cmp_rows():+ pool, sizes, args = _pool(num_pages=16)+ pool.bind_window_pages(full_page_base=0, window_page_base=2 * P)+ pool.bind_window_pages(full_page_base=3 * P, window_page_base=0)+ assert pool.translate_full_to_window(torch.tensor([0, 1, 127])).tolist() == [2 * P, 2 * P + 1, 2 * P + 127]+ assert pool.translate_full_to_window(torch.tensor([3 * P + 5])).item() == 5+ assert pool.translate_full_to_window(torch.tensor([P + 7, -1])).tolist() == [-1, -1]+ full = torch.tensor([0, 1, 2, 127, 128, 4 * P - 1])+ assert pool.cmp_rows(full, 2).tolist() == [0, 0, 1, 63, 64, 2 * P - 1]+ assert pool.cmp_rows(full, 1).tolist() == full.tolist()+ assert pool.cmp_rows(torch.tensor([-1]), 2).item() < 0+ top = pool.cmp_rows(torch.tensor([sizes.full_token - 1]), 2).item()+ assert top < pool.scratch_base[2]+ ws = pool.translate_full_to_window(torch.tensor([127, 3 * P]))+ assert DSV41PagedKVCache.state_loc(ws, 2, P).tolist() == [2 * 2 + 1, 0]+ assert DSV41PagedKVCache.state_loc(torch.tensor([-1]), 2, P).item() == -1+++def _expand(bases):+ return (torch.tensor(bases, dtype=torch.int64)[:, None] + torch.arange(P)).flatten()+++def test_swa_duck_type_alloc_free_and_dummy():+ pool, sizes, _ = _pool(num_pages=8)+ pool._init_paged_state(max_running_req=2, radix=True)+ cap = sizes.n_win_slots - P+ assert pool.swa_available_size() == cap and pool.swa_num_tokens - 1 == cap+ assert pool.sliding_window_size == P and pool.prefill_chunk_budget >= P+ pool.alloc_swa(_expand([0, 2 * P]))+ assert pool.swa_available_size() == cap - 2 * P+ ws = pool.translate_loc_from_full_to_swa(torch.arange(2 * P, 3 * P))+ assert (ws >= 0).all() and int(ws[0]) % P == 0 and torch.equal(ws - ws[0], torch.arange(P))+ pool.free_swa(_expand([0]))+ pool.free_swa(_expand([0])) # idempotent+ assert pool.swa_available_size() == cap - P+ with pytest.raises(AssertionError):+ pool.free_swa(torch.arange(2 * P, 2 * P + 5))+ dummy = pool.translate_loc_from_full_to_swa(torch.arange(sizes.full_token - P, sizes.full_token))+ assert int(dummy[0]) == sizes.n_win_slots - P+ assert pool.k_cache(3) is pool.window_pool[3] and pool.num_layers == 10 and pool.unit_bytes()[0] > 0++[email protected](not torch.cuda.is_available(), reason="packed writes run through triton")+def test_packed_writes_round_trip_on_cuda():+ from kernels.test_dsv41_pack import reference_roundtrip # tests/ is on sys.path under pytest++ args = _args()+ sizes = dsv41_pool_sizes(4, args, 0.5, P)+ pool = DSV41PagedKVCache(sizes, args, torch.device("cuda"), n_scratch=2)+ kv = torch.randn(3, 512, device="cuda", dtype=torch.bfloat16)+ slots = torch.tensor([0, 130, 5], device="cuda")+ pool.store_window(kv, 4, slots)+ assert torch.equal(pool.read_window(4, slots), reference_roundtrip(kv, FP8_E8M0_B32))+ pool.store_main(kv, 2, slots)+ assert torch.equal(pool.read_main(2, slots), reference_roundtrip(kv, FP4_E4M3_B16))+ k = torch.randn(3, 128, device="cuda", dtype=torch.bfloat16)+ pool.store_index(k, 7, slots)+ assert torch.equal(pool.read_index(7, slots), reference_roundtrip(k, FP4_E8M0_B32))+ pool.store_kv(kv, kv, slots, 0)+ assert torch.equal(pool.read_window(0, slots), reference_roundtrip(kv, FP8_E8M0_B32))diff --git a/tests/kvcache/test_dsv4_free_list.py b/tests/kvcache/test_dsv4_free_list.pyindex a3ae8d0c..7b0dfb55 100644--- a/tests/kvcache/test_dsv4_free_list.py+++ b/tests/kvcache/test_dsv4_free_list.py@@ -5,7 +5,7 @@ from __future__ import annotationsimport pytestimport torch-from freetoken.kvcache.dsv4_paged_pool import FreeListAllocator+from freetoken.kvcache.dsv4.v4_pool import FreeListAllocatorDEVICE = torch.device("cpu")P = 128diff --git a/tests/kvcache/test_dsv4_pool.py b/tests/kvcache/test_dsv4_pool.pyindex c345b67c..c8e21997 100644--- a/tests/kvcache/test_dsv4_pool.py+++ b/tests/kvcache/test_dsv4_pool.py@@ -10,8 +10,8 @@ from __future__ import annotationsimport pytestimport torch-from freetoken.kvcache.dsv4_cost_model import dsv4_pool_sizes-from freetoken.kvcache.dsv4_paged_pool import CompressStateRing, DSV4PagedKVCache+from freetoken.kvcache.dsv4.v4_cost_model import dsv4_pool_sizes+from freetoken.kvcache.dsv4.v4_pool import CompressStateRing, DSV4PagedKVCachefrom freetoken.models.deepseek_v4.args import DeepseekV4ArgsDEVICE = torch.device("cpu")diff --git a/tests/kvcache/test_kv_cache_rebuild.py b/tests/kvcache/test_kv_cache_rebuild.pyindex a5ec94a0..0e3b7050 100644--- a/tests/kvcache/test_kv_cache_rebuild.py+++ b/tests/kvcache/test_kv_cache_rebuild.py@@ -178,13 +178,13 @@ def test_dsv4_rebuild_from_config_builds_the_pool_sizes_and_attaches_the_table()the engine used to do -- and re-points full_loc_map at the shared page table."""from types import SimpleNamespace- from freetoken.kvcache.dsv4_cost_model import _dsv4_pool_sizes- from freetoken.kvcache.dsv4_cost_model import (+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_pool_sizes+ from freetoken.kvcache.dsv4.v4_cost_model import (dsv4_kv_unit_bytes,dsv4_pool_sizes,dsv4_window_unit_bytes,)- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCachefrom freetoken.models.deepseek_v4.args import DeepseekV4ArgsP, mrr = 128, 1@@ -222,7 +222,7 @@ def test_dsv4_refresh_seq_state_tracks_page_table_width():from freetoken.engine.engine import Enginefrom freetoken.utils import align_ceil- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCachefrom freetoken.scheduler.table import TableManagerP, mrr = 128, 4@@ -264,7 +264,7 @@ def test_every_kv_pool_answers_the_sizing_surface():the model resolved to. DSV4 must carry its OWN overrides (getattr would resolve to thebase class and silently misprice the window floor if an override vanished)."""from freetoken.kvcache.dsa_pool import DSAKVCache, MLAKVCache- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCachefrom freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCachefrom freetoken.kvcache.mha_pool import MHAKVCache@@ -279,7 +279,7 @@ def test_every_kv_pool_answers_the_rebuild_surface():"""The engine drives every pool through this one surface; a pool that skips animplementation must fail loudly at import/instantiation, not at rebuild time."""from freetoken.kvcache.dsa_pool import DSAKVCache, MLAKVCache- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCachefrom freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCachefrom freetoken.kvcache.mha_pool import MHAKVCachediff --git a/tests/kvcache/test_pool_sizing_surface.py b/tests/kvcache/test_pool_sizing_surface.pyindex 516090b0..33f27983 100644--- a/tests/kvcache/test_pool_sizing_surface.py+++ b/tests/kvcache/test_pool_sizing_surface.py@@ -30,7 +30,7 @@ def _model_config(specs, **attrs):def test_resolve_pool_class_follows_attn_type():from freetoken.kvcache.dsa_pool import DSAKVCache, MLAKVCache- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCachefrom freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCachefrom freetoken.kvcache.mha_pool import MHAKVCache@@ -91,11 +91,11 @@ def _dsv4_config(num_page_override=None):def test_dsv4_kv_cost_and_floor_parity():- from freetoken.kvcache.dsv4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache- pytest.importorskip("freetoken.kvcache.dsv4_cost_model")- from freetoken.kvcache.dsv4_cost_model import dsv4_auto_cost_model+ pytest.importorskip("freetoken.kvcache.dsv4.v4_cost_model")+ from freetoken.kvcache.dsv4.v4_cost_model import dsv4_auto_cost_modelconfig = _dsv4_config()P = 128@@ -147,8 +147,8 @@ def test_generic_validate_rebuild_budget_check():def test_dsv4_validate_rebuild_floor():from freetoken.kvcache.base import CacheRebuildRejected- from freetoken.kvcache.dsv4_cost_model import _dsv4_window_floor_pages- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_window_floor_pages+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCacheconfig = _dsv4_config()pool = object.__new__(DSV4PagedKVCache) # floor path reads no instance state@@ -165,7 +165,7 @@ def test_rebuild_from_config_explicit_num_swa_pages_wins():# The explicit parameter must take precedence over config.swa_num_pages_override --# the engine's override write would otherwise mask a dropped parameter forever.from freetoken.distributed import set_tp_info, try_get_tp_info- from freetoken.kvcache.dsv4_cost_model import _dsv4_pool_sizes+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_pool_sizesfrom freetoken.kvcache.hybrid_swa_pool import _swa_paged_num_tokens, _swa_pool_floorif try_get_tp_info() is None:@@ -183,7 +183,7 @@ def test_rebuild_from_config_explicit_num_swa_pages_wins():assert _swa_paged_num_tokens(swa_cfg, 11, num_swa_pages=explicit) == explicit + 1assert _swa_paged_num_tokens(swa_cfg, 11) == 999_999 + 1 # override path unchanged- from freetoken.kvcache.dsv4_cost_model import _dsv4_window_floor_pages+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_window_floor_pagesd_cfg = _dsv4_config()d_floor = _dsv4_window_floor_pages(d_cfg, d_cfg.page_size)diff --git a/tests/layers/test_mhc.py b/tests/layers/test_mhc.pyindex 4cf144e7..0359bed9 100644--- a/tests/layers/test_mhc.py+++ b/tests/layers/test_mhc.py@@ -152,6 +152,75 @@ def test_triton_fused_respects_input_dtype(dtype, tol):assert err < tol, f"{name} [{dtype}]: max abs err {err}"+def test_single_pass_applies_the_previous_pre_and_hands_on_its_own():+ """Single-pass mHC (DeepSeek-V4.1): the input mix uses ``pre_mix_in``, the gates+ computed here are the next sublayer's pre; post/comb are unchanged from multi-pass."""+ from freetoken.layers.mhc import (+ mhc_fused_post_pre_single_pass,+ mhc_gates,+ mhc_mix_input,+ mhc_pre_single_pass,+ )++ fn, scale, base = _weights(seed=21)+ res = _residual(seed=22)+ x = torch.randn(T, HIDDEN, dtype=torch.bfloat16)+ pre_in = torch.rand(T, N, dtype=torch.float32)+ post0, comb0, _ = mhc_pre(res, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN)++ r1, p1, c1, pre1, li1 = mhc_fused_post_pre_single_pass(+ x, res, post0, comb0, pre_in, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN+ )+ r_ref = mhc_post(x, res, post0, comb0)+ pre_ref, p_ref, c_ref = mhc_gates(r_ref, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN)+ assert torch.equal(r1, r_ref) and torch.equal(p1, p_ref) and torch.equal(c1, c_ref)+ assert torch.equal(pre1, pre_ref)+ assert torch.equal(li1, mhc_mix_input(r_ref, pre_in))+ # the multi-pass input would have used pre_ref, not pre_in+ assert not torch.equal(li1, mhc_mix_input(r_ref, pre_ref))+ # a one-hot pre picks one stream (the model's seed for the chain)+ one_hot = torch.zeros(T, N)+ one_hot[:, 0] = 1.0+ _, _, _, li0 = mhc_pre_single_pass(res, one_hot, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN)+ assert torch.equal(li0, res[:, 0])+ # no post on the first sublayer: the streams pass through untouched+ r0, *_ = mhc_fused_post_pre_single_pass(+ x, res, None, None, one_hot, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN+ )+ assert torch.equal(r0, res)++[email protected](not torch.cuda.is_available(), reason="needs CUDA")[email protected]("t,hidden,with_post", [(1, 64, True), (9, 64, True), (3, 5120, True), (3, 5120, False)])+def test_triton_single_pass_matches_torch(t, hidden, with_post):+ from freetoken.layers.mhc import mhc_fused_post_pre_single_pass_torch+ from freetoken.kernel.triton.mhc import mhc_fused_post_pre_single_pass_triton++ torch.manual_seed(23)+ mix = 2 * N + N * N+ fn = torch.randn(mix, N * hidden, dtype=torch.float32, device="cuda") * 0.05+ scale = torch.rand(3, dtype=torch.float32, device="cuda") + 0.5+ base = torch.randn(mix, dtype=torch.float32, device="cuda") * 0.3+ res = torch.randn(t, N, hidden, dtype=torch.bfloat16, device="cuda")+ x = torch.randn(t, hidden, dtype=torch.bfloat16, device="cuda")+ pre_in = torch.rand(t, N, dtype=torch.float32, device="cuda")+ post0 = torch.rand(t, N, 1, dtype=torch.float32, device="cuda") * POST_MULT if with_post else None+ comb0 = torch.softmax(torch.randn(t, N, N, device="cuda"), dim=-1) if with_post else None++ ref = mhc_fused_post_pre_single_pass_torch(+ x, res, post0, comb0, pre_in, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN+ )+ got = mhc_fused_post_pre_single_pass_triton(+ x, res, post0, comb0, pre_in, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN+ )+ names = ("residual", "post", "comb", "pre_new", "layer_input")+ tols = (2e-2, 2e-3, 2e-3, 2e-3, 2e-2)+ for name, r, g, tol in zip(names, ref, got, tols):+ assert g.shape == r.shape, (name, g.shape, r.shape)+ err = (g.float() - r.float()).abs().max().item()+ assert err < tol, f"{name}: max abs err {err}"++@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")def test_triton_pre_only_matches_torch():"""HAS_POST=False path (layer 0's standalone hc_pre through the fused kernel)."""@@ -176,3 +245,31 @@ def test_triton_pre_only_matches_torch():assert (got_post.float() - ref_post.float()).abs().max().item() < 2e-3assert (got_comb.float() - ref_comb.float()).abs().max().item() < 2e-3assert (got_li.float() - ref_li.float()).abs().max().item() < 2e-2++[email protected](not torch.cuda.is_available(), reason="needs CUDA")+def test_mhc_split_preserves_chunked_results_and_graph_replay():+ from freetoken.kernel.triton.mhc import mhc_fused_post_pre_single_pass_triton as fused++ torch.manual_seed(26)+ hidden, tokens = 5120, 7+ res = torch.randn(tokens, N, hidden, device="cuda", dtype=torch.bfloat16)+ x = torch.randn(tokens, hidden, device="cuda", dtype=torch.bfloat16)+ post = torch.rand(tokens, N, 1, device="cuda")+ comb = torch.softmax(torch.randn(tokens, N, N, device="cuda"), -1)+ pre = torch.rand(tokens, N, device="cuda")+ fn = torch.randn(2 * N + N * N, N * hidden, device="cuda") * 0.05+ scale, base = torch.ones(3, device="cuda"), torch.zeros(2 * N + N * N, device="cuda")+ def run(lo, hi):+ return fused(x[lo:hi], res[lo:hi], post[lo:hi], comb[lo:hi], pre[lo:hi], fn, scale, base,+ RMS_EPS, EPS, POST_MULT, SINKHORN)+ whole = run(0, tokens)+ chunks = [run(0, 2), run(2, tokens)]+ for i, expected in enumerate(whole):+ assert torch.equal(expected, torch.cat([chunk[i] for chunk in chunks]))+ graph = torch.cuda.CUDAGraph()+ with torch.cuda.graph(graph):+ captured = run(0, tokens)+ graph.replay()+ for got, expected in zip(captured, whole):+ assert torch.equal(got, expected)diff --git a/tests/models/deepseek_v41/__init__.py b/tests/models/deepseek_v41/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/tests/models/deepseek_v41/common.py b/tests/models/deepseek_v41/common.pynew file mode 100644index 00000000..d0f8efee--- /dev/null+++ b/tests/models/deepseek_v41/common.py@@ -0,0 +1,176 @@+"""Shared fixtures for the deepseek_v41 tests: a tiny V4.1-shaped config (2 window layers, a ratio-2+encoder group with one Full + one Reuse layer, a ratio-1 decoder group with a Full candidate source and+a Reindex layer) and synthetic bf16 and FP8/MXFP4 checkpoints in the real tensor naming. Holds no tests itself."""++from __future__ import annotations++import json+import os+from types import SimpleNamespace++import pytest+import torch++requires_cuda = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU")++VOCAB = 512+DIM = 256+N_LAYERS = 6+RATIOS = [0, 0, 2, 2, 1, 1, 0] # + one trailing MTP layer the model ignores+KV_SOURCES = [2, 4]+INDEX_SOURCES = [2, 4, 5]+ENGRAM_LAYERS = [1]+ENGRAM_VOCAB = 500 # bucket prime start; 3 orders x 2 heads ~= 3000 rows+ENGRAM_ROWS = 4096+++def tiny_text_config(**over) -> dict:+ text = dict(+ model_type="deepseek_v41_text",+ vocab_size=VOCAB, hidden_size=DIM, moe_intermediate_size=64, num_hidden_layers=N_LAYERS, num_attention_heads=4,+ num_key_value_heads=1, head_dim=64, qk_rope_head_dim=16, q_lora_rank=64, o_lora_rank=32, o_groups=2,+ hidden_act="silu", swiglu_limit=10.0, rms_norm_eps=1e-20, max_position_embeddings=16384,+ rope_theta=10000, rope_scaling=dict(rope_type="yarn", factor=16, beta_fast=32, beta_slow=1, original_max_position_embeddings=1024),+ n_routed_experts=8, n_shared_experts=1, num_experts_per_tok=2, scoring_func="sqrtsoftplus", topk_method="noaux_tc",+ norm_topk_prob=True, routed_scaling_factor=1.5, sliding_window=128,+ compress_ratios=list(RATIOS), compress_rope_theta=160000, kv_source_layer_ids=list(KV_SOURCES),+ index_source_layer_ids=list(INDEX_SOURCES), index_n_heads=2, index_head_dim=32, index_topk=8,+ candidate_source_layer_id=4, candidate_topk_blocks=2, candidate_block_size=4,+ hc_mult=4, hc_sinkhorn_iters=20, hc_eps=1e-6,+ engram_layer_ids=list(ENGRAM_LAYERS), engram_num_embeddings=[ENGRAM_ROWS], engram_max_ngram_size=4,+ engram_vocab_size=ENGRAM_VOCAB, engram_n_heads=2, engram_head_dim=32, engram_pad_token_id=2,+ engram_compressed_vocab_size=VOCAB, # the synthetic tokenizer maps every id to itself+ num_nextn_predict_layers=1, dspark_block_size=0, dspark_target_layer_ids=[],+ )+ text.update(over)+ return text+++def tiny_hf_config(**over) -> SimpleNamespace:+ return SimpleNamespace(architectures=["DeepseekV41ForCausalLM"], model_type="deepseek_v41", text_config=tiny_text_config(**over))+++def checkpoint_tensors(text: dict, seed: int = 0) -> dict[str, torch.Tensor]:+ """Random bf16 weights under the checkpoint's names (the reference ``convert.py`` dialect without+ quantization: no ``.scale`` tensors). hc / sink / router-bias / engram gates in fp32 as shipped."""+ g = torch.Generator().manual_seed(seed)++ def rnd(*shape, scale=0.02, dtype=torch.bfloat16):+ return (torch.randn(*shape, generator=g) * scale).to(dtype)++ dim, heads, hd, rd = text["hidden_size"], text["num_attention_heads"], text["head_dim"], text["qk_rope_head_dim"]+ qr, ol, og = text["q_lora_rank"], text["o_lora_rank"], text["o_groups"]+ inter, E, hc = text["moe_intermediate_size"], text["n_routed_experts"], text["hc_mult"]+ ih, ihd = text["index_n_heads"], text["index_head_dim"]+ mix = (2 + hc) * hc+ t: dict[str, torch.Tensor] = {+ "embed.weight": rnd(text["vocab_size"], dim, scale=1.0),+ "norm.weight": torch.ones(dim, dtype=torch.bfloat16),+ "head.weight": rnd(text["vocab_size"], dim),+ }+ for L in range(text["num_hidden_layers"]):+ a = f"layers.{L}.attn"+ t[f"{a}.wq_a.weight"] = rnd(qr, dim)+ t[f"{a}.q_norm.weight"] = torch.ones(qr, dtype=torch.bfloat16)+ t[f"{a}.wq_b.weight"] = rnd(heads * hd, qr)+ t[f"{a}.wkv.weight"] = rnd(hd, dim)+ t[f"{a}.kv_norm.weight"] = torch.ones(hd, dtype=torch.bfloat16)+ t[f"{a}.wo_a.weight"] = rnd(og * ol, heads * hd // og)+ t[f"{a}.wo_b.weight"] = rnd(dim, og * ol)+ t[f"{a}.attn_sink"] = rnd(heads, scale=0.5, dtype=torch.float32)+ ratio = text["compress_ratios"][L]+ if L in text["kv_source_layer_ids"]:+ t[f"{a}.compressor.wkv.weight"] = rnd(hd, dim)+ if ratio > 1:+ t[f"{a}.compressor.wgate.weight"] = rnd(hd, dim)+ t[f"{a}.compressor.norm.weight"] = torch.ones(hd, dtype=torch.bfloat16)+ t[f"{a}.indexer.wk.weight"] = rnd(ihd, hd)+ t[f"{a}.indexer.k_norm.weight"] = torch.ones(ihd, dtype=torch.bfloat16)+ if L in text["index_source_layer_ids"]:+ t[f"{a}.indexer.wq_b.weight"] = rnd(ih * ihd, qr)+ t[f"{a}.indexer.weights_proj.weight"] = rnd(ih, dim)+ p = f"layers.{L}"+ t[f"{p}.attn_norm.weight"] = torch.ones(dim, dtype=torch.bfloat16)+ t[f"{p}.ffn_norm.weight"] = torch.ones(dim, dtype=torch.bfloat16)+ t[f"{p}.ffn.gate.weight"] = rnd(E, dim)+ t[f"{p}.ffn.gate.bias"] = rnd(E, scale=0.1, dtype=torch.float32)+ t[f"{p}.ffn.gate.bias_vl"] = rnd(E, scale=0.1, dtype=torch.float32)+ for w, shape in (("w1", (inter, dim)), ("w2", (dim, inter)), ("w3", (inter, dim))):+ t[f"{p}.ffn.shared_experts.{w}.weight"] = rnd(*shape)+ for e in range(E):+ t[f"{p}.ffn.experts.{e}.{w}.weight"] = rnd(*shape)+ for nm in ("hc_attn_fn", "hc_ffn_fn"):+ t[f"{p}.{nm}"] = rnd(mix, hc * dim, scale=0.05, dtype=torch.float32)+ for nm in ("hc_attn_base", "hc_ffn_base"):+ t[f"{p}.{nm}"] = rnd(mix, scale=0.3, dtype=torch.float32)+ for nm in ("hc_attn_scale", "hc_ffn_scale"):+ t[f"{p}.{nm}"] = (torch.rand(3, generator=g) + 0.5).to(torch.float32)+ if L in text["engram_layer_ids"]:+ e = f"{p}.engram"+ cols = (text["engram_max_ngram_size"] - 1) * text["engram_n_heads"]+ t[f"{e}.wkv.weight"] = rnd(dim * (hc + 1), cols * text["engram_head_dim"])+ t[f"{e}.q_weight"] = torch.ones(hc, dim, dtype=torch.bfloat16)+ t[f"{e}.k_weight"] = torch.ones(hc, dim, dtype=torch.bfloat16)+ rows = text["engram_num_embeddings"][text["engram_layer_ids"].index(L)]+ t[f"{e}.embed.weight"] = rnd(rows, text["engram_head_dim"], scale=1.0).to(torch.float8_e4m3fn)+ t[f"{e}.embed.scale"] = torch.full((rows, text["engram_head_dim"] // 32), 127, dtype=torch.uint8).view(torch.float8_e8m0fnu)+ # the DSpark stage and the vision tower: present in a real checkpoint, dropped by the loader+ t["mtp.0.attn_norm.weight"] = torch.ones(dim, dtype=torch.bfloat16)+ t["vision.norm.weight"] = torch.ones(8, dtype=torch.bfloat16)+ return t+++def write_tiny_checkpoint(folder: str, seed: int = 0, *, quantized: bool = False, **over) -> dict[str, torch.Tensor]:+ """A loadable checkpoint directory: HF-style config.json + one safetensors shard + index."""+ from safetensors.torch import save_file++ if quantized:+ over.setdefault("moe_intermediate_size", 256)+ text = tiny_text_config(**over)+ os.makedirs(folder, exist_ok=True)+ with open(os.path.join(folder, "config.json"), "w") as f:+ config = {"architectures": ["DeepseekV41ForCausalLM"], "model_type": "deepseek_v41", "text_config": text}+ if quantized:+ config["quantization_config"] = {+ "quant_method": "fp8", "activation_scheme": "dynamic",+ "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4",+ }+ json.dump(config, f)+ tensors = checkpoint_tensors(text, seed)+ if quantized:+ tensors = _quantize_checkpoint(tensors)+ save_file(tensors, os.path.join(folder, "model-00001-of-00001.safetensors"))+ with open(os.path.join(folder, "model.safetensors.index.json"), "w") as f:+ json.dump({"metadata": {}, "weight_map": {k: "model-00001-of-00001.safetensors" for k in tensors}}, f)+ return tensors+++def _quantize_checkpoint(tensors):+ from .reference.kernel import _round_fp4++ out = dict(tensors)+ fp8_suffixes = (+ ".attn.wq_a", ".attn.wq_b", ".attn.wkv", ".attn.wo_a", ".attn.wo_b",+ ".indexer.wq_b", ".shared_experts.w1", ".shared_experts.w2", ".shared_experts.w3", ".engram.wkv",+ )+ grid = torch.tensor([0., .5, 1., 1.5, 2., 3., 4., 6.])+ for name, weight in tensors.items():+ if not name.endswith(".weight"):+ continue+ stem = name.removesuffix(".weight")+ if ".ffn.experts." in name:+ groups = weight.float().unflatten(-1, (-1, 32))+ scale = torch.exp2(torch.ceil(torch.log2(groups.abs().amax(-1).clamp_min(1e-12) / 6)))+ q = _round_fp4(groups / scale.unsqueeze(-1)).flatten(-2)+ codes = (q.abs().unsqueeze(-1) == grid).to(torch.int32).argmax(-1).to(torch.uint8)+ codes |= torch.signbit(q).to(torch.uint8) * 8+ out[name] = codes[:, 0::2] | (codes[:, 1::2] << 4)+ out[stem + ".scale"] = scale.to(torch.float8_e8m0fnu)+ elif stem.endswith(fp8_suffixes):+ n, k = weight.shape+ blocks = weight.float().reshape(n // 32, 32, k // 32, 32).permute(0, 2, 1, 3)+ scale = torch.exp2(torch.ceil(torch.log2(blocks.abs().amax((-1, -2)).clamp_min(1e-4) / 448)))+ expanded = scale.repeat_interleave(32, 0).repeat_interleave(32, 1)+ out[name] = (weight.float() / expanded).to(torch.float8_e4m3fn)+ out[stem + ".scale"] = scale.to(torch.float8_e8m0fnu)+ return outdiff --git a/tests/models/deepseek_v41/harness.py b/tests/models/deepseek_v41/harness.pynew file mode 100644index 00000000..98b19e2d--- /dev/null+++ b/tests/models/deepseek_v41/harness.py@@ -0,0 +1,182 @@+"""A minimal in-process engine for the tiny DeepSeek-V4.1 model: builds the DSV41 pool, backend and+context by hand (no scheduler), loads a synthetic bf16 or FP8/MXFP4 checkpoint with resident or offloaded experts, and drives+ragged prefill / batched decode batches through ``model.forward()``. Shared by the forward smoke test+and the reference-parity test."""++from __future__ import annotations++from dataclasses import replace++import torch++from freetoken.core import Batch, Context, Req, SamplingParams, get_global_ctx, set_global_ctx+from freetoken.distributed.info import set_tp_info, try_get_tp_info+from freetoken.engine.engine import _materialize_loaded_weight_state_dict+from freetoken.kvcache.dsv4.v41_cost_model import dsv41_pool_sizes+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache+from freetoken.layers import set_rope_device+from freetoken.layers.quantization import NoQuantConfig+from freetoken.layers.quantization.method import finalize_quant+from freetoken.models import create_model+from freetoken.models.deepseek_v41.config import parse_config+from freetoken.models.deepseek_v41.engram import ZeroEngramTable+from freetoken.models.deepseek_v41.weight import iter_weights+from freetoken.utils.hf import cached_load_hf_config+from freetoken.utils.torch_utils import torch_dtype++P = 128+++class TinyEngine:+ def __init__(self, checkpoint: str, *, max_seq_len: int = 1024, max_running_req: int = 2, swa_decoder_replay: str = "exact", engram_table=None, quantized: bool = False):+ self.device = torch.device("cuda")+ self.checkpoint = checkpoint+ if try_get_tp_info() is None:+ set_tp_info(0, 1)+ set_rope_device(self.device)+ hf_config = cached_load_hf_config(checkpoint)+ mc = parse_config(hf_config)+ args = mc.dsv41_args+ args.max_seq_len = max_seq_len+ args.max_batch_size = max_running_req + 1+ args.swa_decoder_replay = swa_decoder_replay+ self.args = args+ quant = NoQuantConfig()+ if quantized:+ from freetoken.models.register import checkpoint_quant_config, get_model_spec++ quant = checkpoint_quant_config(checkpoint, hf_config, get_model_spec("DeepseekV41ForCausalLM"))+ self.config = replace(mc, moe_strategy="offload" if quantized else "fused", decode_target="gpu", quant=quant)+ mc = self.config+ with torch.device("meta"), torch_dtype(torch.bfloat16):+ self.model = create_model(self.config)+ state = _materialize_loaded_weight_state_dict(+ self.model.state_dict(), iter_weights(checkpoint, self.device, include_moe_experts=False), device=self.device,+ )+ self.model.load_state_dict(state)+ finalize_quant(self.model)+ from freetoken.moe.expert_banks import load_expert_banks++ method = self.model.model.layers.op_list[0].ffn.experts.quant_method+ if quantized:+ from freetoken.moe.offload_cache import OffloadMoeCache, attach_offload_moe_cache++ self.banks = load_expert_banks(+ checkpoint, mc, method=method, device=self.device, dtype=torch.bfloat16, parallel=False,+ )+ self.expert_cache = OffloadMoeCache(+ num_layers=mc.num_moe_layers, num_experts=mc.num_experts,+ cache_size=mc.num_moe_layers * mc.num_experts, device=self.device,+ quant_format=self.banks.quant_format, layout=method.layout(),+ )+ self.expert_cache.set_bank_sources(self.banks.sources, layer_residency=self.banks.layer_residency)+ attach_offload_moe_cache(self.model, self.expert_cache)+ else: # resident experts load as banks, not through the state dict (as in Engine._load_resident_experts)+ from freetoken.layers import iter_moe_layers+ from freetoken.moe.expert_banks import attach_resident_banks++ attach_resident_banks(list(iter_moe_layers(self.model)), load_expert_banks(+ checkpoint, mc, method=method, device=self.device, dtype=torch.bfloat16, parallel=False, resident=True,+ ))+ for layer in self.model.engram_layers():+ layer.attach_table(engram_table if engram_table is not None else ZeroEngramTable(layer.width, self.device))++ self.max_running_req = max_running_req+ num_pages = max_seq_len // P+ self.pool = DSV41PagedKVCache(dsv41_pool_sizes(num_pages + 1, args, 1.0, P), args, self.device, n_scratch=max_running_req + 1)+ self.pool._init_paged_state(max_running_req, radix=False)+ self.page_table = torch.zeros(max_running_req + 1, max_seq_len, dtype=torch.int32, device=self.device)+ self.page_table[max_running_req].fill_(num_pages * P) # the dummy row -> the reserved tail page+ self.pool.attach_page_table(self.page_table)+ try:+ ctx = get_global_ctx()+ except AssertionError:+ ctx = Context(page_size=P)+ set_global_ctx(ctx)+ from freetoken.attention.dsv41_sparse import DSV41SparseAttnBackend++ self.ctx = ctx+ ctx.page_table = self.page_table+ ctx.kv_cache = self.pool # the backend reads the pool's device from the context+ self.backend = DSV41SparseAttnBackend(self.config)+ self._next_page = 0+ self._bind()++ def _bind(self) -> None:+ """Point the process-wide ``Context`` at this engine's page table, pool and backend. The+ context is a singleton the model reads through ``get_global_ctx()``, so a test that holds two+ engines must rebind before each forward; ``prefill`` / ``decode`` do this, and the model+ re-resolves its backend when the binding changes."""+ ctx = self.ctx+ if (+ getattr(ctx, "attn_backend", None) is self.backend+ and getattr(ctx, "kv_cache", None) is self.pool+ and getattr(ctx, "page_table", None) is self.page_table+ ):+ return+ ctx.page_table = self.page_table+ ctx.kv_cache = self.pool+ ctx.attn_backend = self.backend+ self.model.mark_for_rebind()++ # ----- requests -----------------------------------------------------------------------+ def new_request(self, table_idx: int, tokens: list[int], max_new: int = 64) -> Req:+ """A request with pages for ``len(tokens) + max_new`` positions bound on row ``table_idx``."""+ n_pages = -(-(len(tokens) + max_new) // P)+ first = self._next_page+ self._next_page += n_pages+ locs = torch.arange(first * P, (first + n_pages) * P, device=self.device)+ self.page_table[table_idx, : n_pages * P] = locs.to(torch.int32)+ self.pool.alloc_swa(locs)+ return Req(input_ids=torch.tensor(tokens, dtype=torch.int32), table_idx=table_idx, cached_len=0, output_len=max_new,+ uid=table_idx, sampling_params=SamplingParams(), cache_handle=None)++ def new_request_on_prefix(self, table_idx: int, donor: Req, tokens: list[int], max_new: int = 64) -> Req:+ """A request whose first ``hit`` positions alias ``donor``'s pages (what a radix prefix hit does+ through the page table) and whose remaining positions get pages of their own. ``hit`` is what the+ scheduler admits: the donor's prompt, capped by ``match_req`` (the last token, and the pool's+ ``prefix_replay_tokens`` under bounded replay) and page-aligned. The caller prefills the rest."""+ limit = len(tokens) - 1+ if self.pool.prefix_replay_tokens:+ limit = min(limit, max(0, len(tokens) - self.pool.prefix_replay_tokens))+ hit = min(donor.device_len, limit) // P * P+ assert tokens[:hit] == donor.input_ids[:hit].tolist(), "the prefix must match the donor's prompt"+ self.page_table[table_idx, :hit] = self.page_table[donor.table_idx, :hit]+ n_pages = -(-(len(tokens) + max_new - hit) // P)+ first = self._next_page+ self._next_page += n_pages+ locs = torch.arange(first * P, (first + n_pages) * P, device=self.device)+ self.page_table[table_idx, hit : hit + n_pages * P] = locs.to(torch.int32)+ self.pool.alloc_swa(locs)+ return Req(input_ids=torch.tensor(tokens, dtype=torch.int32), table_idx=table_idx, cached_len=hit, output_len=max_new,+ uid=table_idx, sampling_params=SamplingParams(), cache_handle=None)++ def prefill(self, reqs: list[Req]) -> torch.Tensor:+ """Run each request's ``[cached_len, device_len)`` tokens; returns ``[B, vocab]`` logits."""+ batch = Batch(reqs=reqs, phase="prefill")+ batch.padded_reqs = list(reqs)+ batch.input_ids = torch.cat([r.input_ids[r.cached_len : r.device_len] for r in reqs]).to(self.device)+ batch.positions = torch.cat([torch.arange(r.cached_len, r.device_len) for r in reqs]).to(self.device)+ self._bind()+ self.backend.prepare_metadata(batch)+ with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, False):+ return self.model.forward().float()++ def decode(self, reqs: list[Req], tokens: list[int]) -> torch.Tensor:+ """Feed one new token per request (appended at ``device_len - 1``); returns ``[B, vocab]``."""+ for r, t in zip(reqs, tokens):+ r.append_host(torch.tensor([t], dtype=torch.int32))+ r.complete_one()+ batch = Batch(reqs=reqs, phase="decode")+ batch.padded_reqs = list(reqs)+ batch.input_ids = torch.tensor(tokens, dtype=torch.int32, device=self.device)+ batch.positions = torch.tensor([r.device_len - 1 for r in reqs], device=self.device)+ batch.active_table_idx = torch.tensor([r.table_idx for r in reqs], device=self.device)+ self._bind()+ self.backend.prepare_metadata(batch)+ with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, False):+ return self.model.forward().float()++ def finish_prefill(self, reqs: list[Req]) -> None:+ for r in reqs:+ r.cached_len = r.device_lendiff --git a/tests/models/deepseek_v41/reference/LICENSE b/tests/models/deepseek_v41/reference/LICENSEnew file mode 100644index 00000000..d42fae90--- /dev/null+++ b/tests/models/deepseek_v41/reference/LICENSE@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2023 DeepSeek++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.\ No newline at end of filediff --git a/tests/models/deepseek_v41/reference/__init__.py b/tests/models/deepseek_v41/reference/__init__.pynew file mode 100644index 00000000..79d0c331--- /dev/null+++ b/tests/models/deepseek_v41/reference/__init__.py@@ -0,0 +1,2 @@+"""The DeepSeek-V4.1 reference implementation (MIT, vendored from the HF repo's ``inference/``) with its+tilelang kernels transcribed to torch -- the numerical oracle FreeToken's port is tested against."""diff --git a/tests/models/deepseek_v41/reference/engram.py b/tests/models/deepseek_v41/reference/engram.pynew file mode 100644index 00000000..62555311--- /dev/null+++ b/tests/models/deepseek_v41/reference/engram.py@@ -0,0 +1,186 @@+# Vendored from https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash (inference/engram.py), MIT License+# (see LICENSE next to this file); unmodified.+from dataclasses import dataclass++import numpy as np+import torch+from sympy import isprime+from torch import nn+++def find_next_prime(start: int, seen_primes: set[int]) -> int:+ """The smallest prime above `start` that has not been handed out yet."""+ candidate = start + 1+ while not isprime(candidate) or candidate in seen_primes:+ candidate += 1+ return candidate+++def build_compressed_token_map(tokenizer) -> tuple[list[int], int]:+ """Map every token id onto a smaller id space where tokens that normalize alike collapse together.++ N-grams are hashed over these compressed ids, so " The", "the" and "THE" all hash the same way.+ Returns the lookup plus the size of the compressed vocab -- and that size matters beyond bounds+ checking, because every hash multiplier is derived from it.+ """+ from tokenizers import Regex, normalizers++ # a private-use char, so a token that is exactly one space survives Strip() instead of+ # collapsing to the empty string and merging with unrelated tokens+ sentinel = "\ue000"+ normalizer = normalizers.Sequence(+ [+ normalizers.NFKC(),+ normalizers.NFD(),+ normalizers.StripAccents(),+ normalizers.Lowercase(),+ normalizers.Replace(Regex(r"[ \t\r\n]+"), " "),+ normalizers.Replace(Regex(r"^ $"), sentinel),+ normalizers.Strip(),+ normalizers.Replace(sentinel, " "),+ ]+ )++ # the raw Rust tokenizer, matching what training decodes with (no clean_up_tokenization_spaces)+ backend = tokenizer.backend_tokenizer+ key_to_new: dict[str, int] = {}+ lookup = [0] * len(tokenizer)+ for token_id in range(len(tokenizer)):+ text = backend.decode([token_id], skip_special_tokens=False)+ if "\ufffd" in text:+ # a partial UTF-8 byte token: nothing to normalize, so key it by its raw form+ key = backend.id_to_token(token_id)+ else:+ normalized = normalizer.normalize_str(text)+ key = normalized if normalized else text++ new_id = key_to_new.get(key)+ if new_id is None:+ new_id = len(key_to_new)+ key_to_new[key] = new_id+ lookup[token_id] = new_id++ return lookup, len(key_to_new)+++def compute_hash_multipliers(+ layer_ids: tuple[int, ...], max_ngram_size: int, tokenizer_vocab_size: int+) -> torch.Tensor:+ """One multiplier per (layer, lookback), from a per-layer RNG so layers hash differently.++ Kept odd, and bounded so that `token_id * multiplier` cannot overflow int64.+ """+ max_long = np.iinfo(np.int64).max+ multiplier_bound = max(1, (max_long // tokenizer_vocab_size) // 2)+ rows = []+ for layer_id in layer_ids:+ generator = np.random.default_rng(10007 * layer_id)+ values = generator.integers(+ low=0,+ high=multiplier_bound,+ size=(max_ngram_size,),+ dtype=np.int64,+ )+ rows.append(torch.tensor(values * 2 + 1))+ return torch.stack(rows)+++@dataclass(frozen=True)+class EngramLayout:+ """Bucket layout of the n-gram hash tables.++ A position is hashed as `max_ngram_size - 1` n-grams (2-gram .. max_ngram_size-gram), each split+ over `n_heads` heads. Every (n-gram size, head) pair owns its own prime-sized bucket range in the+ layer's table; the primes are drawn in order and never reused, which keeps the ranges disjoint.+ """++ max_ngram_size: int+ layer_ids: tuple[int, ...]+ num_embeddings: tuple[int, ...] # table rows, per engram layer+ primes: tuple[tuple[tuple[int, ...], ...], ...] # [layer][n-gram size][head] bucket modulus+ n_heads: int+ head_dim: int++ @classmethod+ def from_args(cls, args) -> "EngramLayout | None":+ layer_ids = tuple(args.engram_layer_ids)+ if not layer_ids:+ return None+ max_ngram_size, n_heads = args.engram_max_ngram_size, args.engram_n_heads+ primes, seen = [], set()+ for _ in layer_ids:+ per_ngram = []+ for _ in range(max_ngram_size - 1):+ sizes, current = [], args.engram_vocab_size - 1+ for _ in range(n_heads):+ current = find_next_prime(current, seen)+ seen.add(current)+ sizes.append(current)+ per_ngram.append(tuple(sizes))+ primes.append(tuple(per_ngram))+ return cls(+ max_ngram_size=max_ngram_size,+ layer_ids=layer_ids,+ num_embeddings=tuple(args.engram_num_embeddings),+ primes=tuple(primes),+ n_heads=n_heads,+ head_dim=args.engram_head_dim,+ )+++class NgramHashState(nn.Module):+ """Maps each position to the hash ids of the n-grams ending there.++ Ids go through the compressed table, then each position is hashed with the `max_ngram_size - 1`+ tokens before it. Look-back stops at the start of the sequence and at any dead token (an image+ span, cached as DEAD), so an n-gram never spans one. The cache carries all of this across the+ prefill/decode split.+ """++ DEAD = -1++ def __init__(self, args, layout: EngramLayout, tokenizer):+ super().__init__()+ self.layout = layout+ # every hash multiplier derives from the compressed vocab size, so a mismatch there would+ # silently rehash the whole table+ token_map, vocab_size = build_compressed_token_map(tokenizer)+ assert vocab_size == args.engram_compressed_vocab_size, (vocab_size, args.engram_compressed_vocab_size)+ self.pad_id = token_map[args.engram_pad_id]+ flat = [[p for per_ngram in layer for p in per_ngram] for layer in layout.primes]+ offsets = [np.cumsum([0, *sizes[:-1]]) for sizes in flat]+ multipliers = compute_hash_multipliers(layout.layer_ids, layout.max_ngram_size, vocab_size)+ self.register_buffer("primes", torch.tensor(layout.primes), persistent=False)+ self.register_buffer("offsets", torch.tensor(np.array(offsets)), persistent=False)+ self.register_buffer("multipliers", multipliers, persistent=False)+ self.register_buffer("token_map", torch.tensor(token_map), persistent=False)+ self.register_buffer(+ "cache", torch.empty(args.max_batch_size, args.max_seq_len, dtype=torch.int64), persistent=False+ )++ @torch.inference_mode()+ def forward(self, input_ids: torch.Tensor, start_pos: int, token_mask: torch.Tensor | None = None) -> torch.Tensor:+ """token_mask: [B, L], False for tokens that take no part in an n-gram (image spans).+ Returns the hash ids, shaped [B, L, n_engram_layers, n_hash_cols]."""+ batch, seqlen = input_ids.shape+ compressed = self.token_map[input_ids]+ if token_mask is not None:+ compressed = torch.where(token_mask, compressed, self.DEAD)+ self.cache[:batch, start_pos : start_pos + seqlen] = compressed++ positions = torch.arange(start_pos, start_pos + seqlen, device=input_ids.device).expand(batch, seqlen)+ tokens, blocked = [], torch.zeros_like(positions, dtype=torch.bool)+ for shift in range(self.layout.max_ngram_size):+ source = self.cache[:batch].gather(1, (positions - shift).clamp_min(0))+ blocked = blocked | (positions < shift) | (source == self.DEAD)+ tokens.append(torch.where(blocked, self.pad_id, source))+ tokens = torch.stack(tokens, dim=-1) # [B, L, max_ngram_size]++ # XOR the multiplied ids together one lookback at a time, so the running value after step i+ # is the hash of the (i+1)-gram; each lands in its own prime-sized bucket range+ products = tokens.unsqueeze(2) * self.multipliers # [B, L, n_engram_layers, max_ngram_size]+ rolling, hashes = products[..., 0], []+ for i in range(1, self.layout.max_ngram_size):+ rolling = torch.bitwise_xor(rolling, products[..., i])+ hashes.append(rolling.unsqueeze(-1) % self.primes[:, i - 1])+ return torch.cat(hashes, dim=-1) + self.offsetsdiff --git a/tests/models/deepseek_v41/reference/kernel.py b/tests/models/deepseek_v41/reference/kernel.pynew file mode 100644index 00000000..fab0ae08--- /dev/null+++ b/tests/models/deepseek_v41/reference/kernel.py@@ -0,0 +1,113 @@+"""Pure-torch transcriptions of the reference tilelang kernels (inference/kernel.py), for the test oracle.++Same numerics as the tilelang code: fp32 math, power-of-two (ue8m0) or e4m3 scales rounded the way+the kernels round them, the e2m1 grid with the hardware's round-to-nearest-even, and the fp8/fp4+in-place round trips that write the dequantized value back in the input dtype. Quantized GEMMs+dequantize independently in torch and round each projection's output to bf16.+"""++from __future__ import annotations++import torch++_FP4_GRID = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])+++def _pow2_ceil(x: torch.Tensor) -> torch.Tensor:+ """``fast_round_scale``: 2 ** ceil(log2(x)) for x > 0."""+ return torch.exp2(torch.ceil(torch.log2(x)))+++def _round_fp4(x: torch.Tensor) -> torch.Tensor:+ """Round-to-nearest-even onto the signed e2m1 grid (the ``float4_e2m1fn`` cast)."""+ a = x.abs()+ grid = _FP4_GRID.to(x.device)+ hi = torch.searchsorted(grid, a.reshape(-1).contiguous(), right=True).reshape(a.shape).clamp(1, 7)+ below, above = grid[hi - 1], grid[hi]+ mid = (below + above) / 2+ even_above = torch.isin(above, torch.tensor([0.0, 1.0, 2.0, 4.0], device=x.device))+ r = torch.where((a > mid) | ((a == mid) & even_above), above, below)+ r = torch.where(a >= 6.0, torch.full_like(r, 6.0), r)+ return torch.copysign(r, x)+++def act_quant(x: torch.Tensor, block_size: int = 128, scale_fmt=None, scale_dtype=torch.float32, inplace: bool = False):+ """Block-wise fp8 quantization; ``inplace=True`` writes the quant+dequant round trip back into ``x``."""+ n = x.size(-1)+ assert n % block_size == 0+ g = x.float().unflatten(-1, (-1, block_size))+ amax = g.abs().amax(dim=-1).clamp_min(1e-4)+ s = _pow2_ceil(amax / 448.0) if scale_fmt is not None else amax / 448.0+ q = (g / s.unsqueeze(-1)).clamp(-448.0, 448.0).to(torch.float8_e4m3fn)+ if inplace:+ x.copy_((q.float() * s.unsqueeze(-1)).flatten(-2).to(x.dtype))+ return x+ return q.flatten(-2), s.to(scale_dtype)+++def fp4_act_quant(x: torch.Tensor, block_size: int = 32, inplace: bool = False, scale_dtype=torch.float8_e8m0fnu):+ """Block-wise fp4 with ue8m0 (pow2) or e4m3 scales; ``inplace=True`` writes the round trip back."""+ n = x.size(-1)+ assert n % block_size == 0+ g = x.float().unflatten(-1, (-1, block_size))+ amax = g.abs().amax(dim=-1)+ if scale_dtype == torch.float8_e4m3fn:+ amax = amax.clamp_min(6 * 2**-9)+ s = (amax / 6.0).to(torch.float8_e4m3fn).float()+ else:+ amax = amax.clamp_min(6 * 2**-126)+ s = _pow2_ceil(amax / 6.0)+ q = _round_fp4((g / s.unsqueeze(-1)).clamp(-6.0, 6.0))+ if inplace:+ x.copy_((q * s.unsqueeze(-1)).flatten(-2).to(x.dtype))+ return x+ return q.flatten(-2), s.to(scale_dtype)+++def fp8_gemm(a, a_s, b, b_s, scale_dtype=torch.float32, block_size=128):+ """``C = (A * s_a) @ (B * s_b)^T`` over dequantized operands."""+ k = a.size(-1)+ a_deq = (a.float().unflatten(-1, (-1, block_size)) * a_s.float().unsqueeze(-1)).flatten(-2)+ n = b.size(0)+ bs = b_s.float().repeat_interleave(block_size, 0)[:n].repeat_interleave(block_size, 1)[:, :k]+ return (a_deq @ (b.float() * bs).t()).bfloat16()+++def fp4_gemm(a, a_s, b, b_s, scale_dtype=torch.float8_e8m0fnu, act_block_size=32):+ packed = b.view(torch.uint8)+ codes = torch.stack((packed & 15, packed >> 4), dim=-1).flatten(-2).long()+ grid = torch.tensor([0., .5, 1., 1.5, 2., 3., 4., 6., -0., -.5, -1., -1.5, -2., -3., -4., -6.], device=b.device)+ weight = grid[codes] * b_s.float().repeat_interleave(32, -1)+ x = (a.float().unflatten(-1, (-1, act_block_size)) * a_s.float().unsqueeze(-1)).flatten(-2)+ return (x @ weight.t()).bfloat16()+++def sparse_attn(q: torch.Tensor, kv: torch.Tensor, attn_sink: torch.Tensor, topk_idxs: torch.Tensor, softmax_scale: float) -> torch.Tensor:+ """Per (batch, query): softmax over the gathered ``topk_idxs`` (-1 = absent) plus the sink null key.++ ``q [b, m, h, d]``, ``kv [b, n, d]``, ``topk_idxs [b, m, topk]`` -> ``o [b, m, h, d]`` in q's dtype.+ A row with no valid index yields zeros (the finite ``-1e30`` running max of the kernel)."""+ b, m, h, d = q.shape+ valid = topk_idxs >= 0+ gathered = kv[torch.arange(b, device=q.device)[:, None, None], topk_idxs.clamp_min(0).long()] # [b, m, topk, d]+ scores = torch.einsum("bmhd,bmtd->bmht", q.float(), gathered.float()) * softmax_scale+ scores = scores.masked_fill(~valid[:, :, None, :], float("-inf"))+ mx = scores.amax(dim=-1).clamp_min(-1e30)+ p = torch.exp(scores - mx.unsqueeze(-1))+ denom = p.sum(-1) + torch.exp(attn_sink.float()[None, None, :] - mx)+ o = torch.einsum("bmht,bmtd->bmhd", p, gathered.float()) / denom.unsqueeze(-1)+ return o.to(q.dtype)+++def hc_split_sinkhorn(mixes: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor, hc_mult: int = 4, sinkhorn_iters: int = 20, eps: float = 1e-6):+ hc = hc_mult+ mixes, sc, base = mixes.float(), hc_scale.float(), hc_base.float()+ pre = torch.sigmoid(mixes[..., :hc] * sc[0] + base[:hc]) + eps+ post = 2 * torch.sigmoid(mixes[..., hc : 2 * hc] * sc[1] + base[hc : 2 * hc])+ comb = (mixes[..., 2 * hc :] * sc[2] + base[2 * hc :]).unflatten(-1, (hc, hc))+ comb = comb.softmax(dim=-1) + eps+ comb = comb / (comb.sum(dim=-2, keepdim=True) + eps)+ for _ in range(sinkhorn_iters - 1):+ comb = comb / (comb.sum(dim=-1, keepdim=True) + eps)+ comb = comb / (comb.sum(dim=-2, keepdim=True) + eps)+ return pre, post, combdiff --git a/tests/models/deepseek_v41/reference/model.py b/tests/models/deepseek_v41/reference/model.pynew file mode 100644index 00000000..593c7518--- /dev/null+++ b/tests/models/deepseek_v41/reference/model.py@@ -0,0 +1,1458 @@+# Vendored from https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash (inference/model.py), MIT License+# (see LICENSE next to this file). FreeToken keeps this copy as the numerical ORACLE its port is tested+# against. Edits, all marked "FreeToken:", are limited to (1) local imports with the tilelang kernels+# replaced by the pure-torch transcriptions in kernel.py, and (2) forward_bounded(), a Decoder SWA+# Bounded Replay oracle built from the unmodified layer math.+import math+from contextlib import contextmanager+from dataclasses import dataclass+from functools import lru_cache+from typing import Literal++import torch+import torch.distributed as dist+import torch.nn.functional as F+from torch import nn++# FreeToken: local imports.+from .engram import EngramLayout, NgramHashState+from .kernel import (+ act_quant,+ fp4_act_quant,+ fp4_gemm,+ fp8_gemm,+ hc_split_sinkhorn,+ sparse_attn,+)++from .vision import ViT, Aligner++IMAGE_START, IMAGE, IMAGE_NEW_LINE, IMAGE_END = range(4) # FreeToken: image_processor constants++# Set once by Transformer.__init__; one model per process, so layers just read them.+world_size = 1+rank = 0+default_dtype = torch.float8_e4m3fn # storage dtype for Linear weights, from ModelArgs.dtype+fp8_block_size = 32 # one fp8 scale per 32x32 weight block / 32 activations+fp4_block_size = 32 # one fp4 scale per 32 elements along K+scale_fmt = "ue8m0"+scale_dtype = torch.float8_e8m0fnu+++@contextmanager+def set_dtype(dtype):+ """Temporarily override torch's default dtype, restoring it even if the body raises."""+ prev = torch.get_default_dtype()+ torch.set_default_dtype(dtype)+ try:+ yield+ finally:+ torch.set_default_dtype(prev)+++@dataclass+class ModelArgs:+ """Field names are exactly the config JSON keys. The defaults are a small model that+ `python model.py` can run, not the released shapes -- though the scale-independent+ values (norm_eps, score_func, hc_*, engram_*) do match it."""++ # runtime limits rather than model shape: they size the KV caches+ max_batch_size: int = 4+ max_seq_len: int = 4096+ temperature: float = 1+ dtype: Literal["bf16", "fp8"] = "fp8"+ expert_dtype: Literal["fp4"] | None = "fp4"+ vocab_size: int = 129280+ dim: int = 1024+ moe_inter_dim: int = 1024+ n_layers: int = 5+ n_mtp_layers: int = 1 # extra draft layers appended after the backbone, indices n_layers..+ n_heads: int = 16+ # moe+ n_routed_experts: int = 8+ n_shared_experts: int = 1+ n_activated_experts: int = 2+ score_func: Literal["softmax", "sigmoid", "sqrtsoftplus"] = "sqrtsoftplus"+ gate_temp: float = 1.0+ norm_topk_prob: bool = True+ route_scale: float = 1.0+ swiglu_limit: float = 0.0+ # attention: latent q/kv projections, plus a LoRA-factorised output projection over o_groups+ q_lora_rank: int = 256+ head_dim: int = 128+ rope_head_dim: int = 32+ norm_eps: float = 1e-20+ o_groups: int = 8+ o_lora_rank: int = 256+ # sparse attention: every layer attends over a sliding window, and may add compressed KV on top+ window_size: int = 128+ # one entry per layer, MTP layers included: 0 = sliding window only, r = KV compressed r-to-1+ compress_ratios: tuple[int, ...] = (0, 2, 2, 1, 1, 0)+ # layers sharing a ratio also share one compressed KV and one indexer, produced by the first+ kv_source_layers: tuple[int, ...] = (1, 3)+ index_source_layers: tuple[int, ...] = (1, 3)+ # rope, with YaRN extrapolation when original_seq_len > 0. Compressed KV rotates at its own+ # theta because one latent stands for compress_ratio tokens, so its positions are further apart.+ compress_rope_theta: float = 40000.0+ original_seq_len: int = 0+ rope_theta: float = 10000.0+ rope_factor: float = 40+ beta_fast: int = 32+ beta_slow: int = 1+ # the indexer: a small extra attention that scores compressed positions, so each query can keep+ # just `index_topk` of them. Names match DeepSeek-V3.2-Exp, where this mechanism first appeared.+ index_n_heads: int = 16+ index_head_dim: int = 64+ index_topk: int = 64+ # candidate pre-filtering: candidate_source_layer < 0 turns it off and the other two are unused+ candidate_source_layer: int = -1+ candidate_topk_blocks: int = 0+ candidate_block_size: int = 0+ # hyper-connections: the residual stream is carried as hc_mult parallel copies+ hc_mult: int = 4+ hc_sinkhorn_iters: int = 20+ hc_eps: float = 1e-6+ # engram: n-gram hash lookups added into the residual stream at a few layers+ engram_layer_ids: tuple[int, ...] = ()+ engram_num_embeddings: tuple[int, ...] = () # unpadded table rows; each rank allocates ceil(rows / world_size)+ engram_max_ngram_size: int = 1+ engram_vocab_size: int = 0 # bucket size each (n-gram size, head) starts searching primes from+ engram_n_heads: int = 0+ engram_head_dim: int = 0+ engram_pad_id: int = 2 # token that fills n-gram slots with no history; matches training+ # size of the compressed tokenizer vocab; every hash multiplier is derived from it+ engram_compressed_vocab_size: int = 0+ # vision (VL); vision_n_layers == 0 disables the vision path+ vision_n_layers: int = 0+ vision_dim: int = 1024+ vision_n_heads: int = 16+ vision_inter_dim: int = 2816+ vision_patch_size: int = 14+ vision_rope_theta: float = 10000.0+ vision_downsample_ratio: int = 3+ vision_max_n_token: int = 1024+ vision_min_pixels: int = 544 * 544+ vision_max_wh_ratio: int | None = None+ # raw id of <|deepseek_image|>; every position of an image span carries this id in input_ids+ image_token_id: int = 129264+ # dspark draft head. Only the forward pass is implemented here -- nothing calls forward_spec,+ # so these are read but the speculative-decoding loop itself is out of scope for this repo.+ dspark_block_size: int = 0+ dspark_noise_token_id: int = 0+ dspark_target_layer_ids: tuple[int, ...] = ()+ dspark_markov_rank: int = 256+ dspark_n_routed_experts: int = 0+ dspark_n_activated_experts: int = 0++ @property+ def vision_enabled(self) -> bool:+ return self.vision_n_layers > 0++ def get_moe_config(self, layer_id: int) -> tuple[int, int]:+ """Return the routed/activated expert counts for a given layer."""+ if layer_id < self.n_layers:+ return self.n_routed_experts, self.n_activated_experts+ return (+ self.dspark_n_routed_experts or self.n_routed_experts,+ self.dspark_n_activated_experts or self.n_activated_experts,+ )+++class ParallelEmbedding(nn.Module):+ """Embedding sharded along the vocab dimension. Each rank holds vocab_size // world_size rows.+ Out-of-range indices are zero-masked before all_reduce to combine partial embeddings."""++ def __init__(self, vocab_size: int, dim: int):+ super().__init__()+ self.vocab_size = vocab_size+ self.dim = dim+ assert vocab_size % world_size == 0, (+ f"Vocabulary size must be divisible by world size (world_size={world_size})"+ )+ self.part_vocab_size = vocab_size // world_size+ self.vocab_start_idx = rank * self.part_vocab_size+ self.vocab_end_idx = self.vocab_start_idx + self.part_vocab_size+ self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ if world_size > 1:+ # ids off this rank read row 0 then get zeroed, so the all_reduce sums one real row+ mask = (x < self.vocab_start_idx) | (x >= self.vocab_end_idx)+ x = x - self.vocab_start_idx+ x[mask] = 0+ y = F.embedding(x, self.weight)+ if world_size > 1:+ y[mask] = 0+ dist.all_reduce(y)+ return y+++def linear(x: torch.Tensor, weight: torch.Tensor, bias: torch.Tensor | None = None) -> torch.Tensor:+ """Pick a GEMM from the weight dtype. Quantized weights need a quantized activation, and both+ fp4 and fp8 weights take an fp8 one -- for fp4 the kernel handles the mixed precision."""+ assert bias is None++ if weight.dtype == torch.float4_e2m1fn_x2:+ x, s = act_quant(x, fp8_block_size, scale_fmt, scale_dtype)+ return fp4_gemm(+ x,+ s,+ weight,+ weight.scale,+ scale_dtype,+ act_block_size=fp8_block_size,+ )+ elif weight.dtype == torch.float8_e4m3fn:+ x, s = act_quant(x, fp8_block_size, scale_fmt, scale_dtype)+ return fp8_gemm(+ x,+ s,+ weight,+ weight.scale,+ scale_dtype,+ block_size=fp8_block_size,+ )+ else:+ return F.linear(x, weight)+++class Linear(nn.Module):+ """bf16, fp8 or fp4 weights. Quantized ones get a `scale`, also attached to `.weight` so that+ `linear()` can reach it."""++ def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype=None):+ super().__init__()+ self.in_features = in_features+ self.out_features = out_features+ dtype = dtype or default_dtype+ if dtype == torch.float4_e2m1fn_x2:+ # two values per byte: [out, in] logically, [out, in//2] stored+ self.weight = nn.Parameter(torch.empty(out_features, in_features // 2, dtype=torch.float4_e2m1fn_x2))+ self.weight.scale = self.scale = nn.Parameter(+ torch.empty(out_features, in_features // fp4_block_size, dtype=torch.float8_e8m0fnu)+ )+ elif dtype == torch.float8_e4m3fn:+ self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype))+ self.weight.scale = self.scale = nn.Parameter(+ torch.empty(+ (out_features + fp8_block_size - 1) // fp8_block_size,+ (in_features + fp8_block_size - 1) // fp8_block_size,+ dtype=torch.float8_e8m0fnu,+ )+ )+ else:+ self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype))+ self.register_parameter("scale", None)+ if bias:+ self.bias = nn.Parameter(torch.empty(out_features))+ else:+ self.register_parameter("bias", None)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ return linear(x, self.weight, self.bias)+++class ColumnParallelLinear(Linear):+ """Splits the output dim across ranks; each rank's slice of the output is already complete."""++ def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype=None):+ assert out_features % world_size == 0, (+ f"Output features must be divisible by world size (world_size={world_size})"+ )+ self.part_out_features = out_features // world_size+ super().__init__(in_features, self.part_out_features, bias, dtype)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ return linear(x, self.weight, self.bias)+++class RowParallelLinear(Linear):+ """Splits the reduction dim, so each rank holds a partial sum: hence the fp32 all_reduce, with+ the bias added only after it."""++ def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype=None):+ assert in_features % world_size == 0, (+ f"Input features must be divisible by world size (world_size={world_size})"+ )+ self.part_in_features = in_features // world_size+ super().__init__(self.part_in_features, out_features, bias, dtype)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ y = linear(x, self.weight, None)+ if world_size > 1:+ y = y.float()+ dist.all_reduce(y)+ if self.bias is not None:+ y += self.bias+ return y.type_as(x)+++class RMSNorm(nn.Module):+ def __init__(self, dim: int, eps: float = 1e-6):+ super().__init__()+ self.dim = dim+ self.eps = eps+ self.weight = nn.Parameter(torch.ones(dim))++ def forward(self, x: torch.Tensor):+ dtype = x.dtype+ x = x.float()+ var = x.square().mean(-1, keepdim=True)+ x = x * torch.rsqrt(var + self.eps)+ return (self.weight * x).to(dtype)+++class ParallelEngramEmbedding(nn.Module):+ """The n-gram hash table, sharded over its rows. Stays fp8: rows are dequantized on lookup."""++ def __init__(self, num_embeddings: int, dim: int):+ super().__init__()+ self.num_embeddings = num_embeddings+ self.dim = dim+ self.part_num_embeddings = (num_embeddings + world_size - 1) // world_size+ self.vocab_start_idx = rank * self.part_num_embeddings+ self.vocab_end_idx = self.vocab_start_idx + self.part_num_embeddings++ self.block_size = fp8_block_size+ # the table stays fp8 as stored: rows are dequantized with `scale` on lookup+ self.weight = nn.Parameter(torch.empty(self.part_num_embeddings, dim, dtype=torch.float8_e4m3fn))+ self.scale = nn.Parameter(torch.empty(self.part_num_embeddings, dim // self.block_size, dtype=scale_dtype))++ def forward(self, indices: torch.Tensor) -> torch.Tensor:+ mask = (indices < self.vocab_start_idx) | (indices >= self.vocab_end_idx)+ local_indices = indices - self.vocab_start_idx+ local_indices = local_indices.masked_fill(mask, 0)++ values = F.embedding(local_indices, self.weight)+ scales = F.embedding(local_indices, self.scale)+ values = values.float().unflatten(-1, (-1, self.block_size)) * scales.float().unsqueeze(-1)+ values = values.flatten(-2).to(torch.bfloat16)+ values = values.masked_fill(mask.unsqueeze(-1), 0)++ if world_size > 1:+ dist.all_reduce(values)+ return values+++class Engram(nn.Module):+ """Writes an n-gram lookup into the residual stream, gated by how well it matches that stream.++ The hash ids fetch `n_hash_cols` rows; `wkv` turns them into one key per hc copy plus a shared+ value. The gate is a normalized dot product of stream against key.+ """++ def __init__(self, args: ModelArgs, layer_id: int, layout: EngramLayout):+ super().__init__()+ self.layer_id = layer_id+ self.layer_hash_index = layout.layer_ids.index(layer_id)+ self.dim = args.dim+ self.hc_mult = args.hc_mult+ self.clamp_value = 1e-6++ self.embed = ParallelEngramEmbedding(layout.num_embeddings[self.layer_hash_index], layout.head_dim)+ n_hash_cols = (layout.max_ngram_size - 1) * layout.n_heads+ self.wkv = Linear(n_hash_cols * layout.head_dim, args.dim * (args.hc_mult + 1))+ self.eps = args.norm_eps+ self.q_weight = nn.Parameter(torch.ones(args.hc_mult, args.dim))+ self.k_weight = nn.Parameter(torch.ones(args.hc_mult, args.dim))++ def forward(self, x: torch.Tensor, hash_ids: torch.Tensor, token_mask: torch.Tensor | None = None) -> torch.Tensor:+ """x: [B, L, hc_mult, dim]; hash_ids: [B, L, n_hash_cols]; token_mask: [B, L], False shuts+ the gate so those positions pass through untouched."""+ kv = self.wkv(self.embed(hash_ids).flatten(-2))+ key, value = kv.split([self.hc_mult * self.dim, self.dim], dim=-1)+ key = key.float().unflatten(-1, (self.hc_mult, self.dim))+ weight = self.q_weight.float() * self.k_weight.float() # only ever used as a product+ h, eps = x.float(), self.eps+ # normalized per (token, hc copy) over `dim`, NOT jointly over the copies+ rstd = torch.rsqrt(h.square().mean(-1) + eps) * torch.rsqrt(key.square().mean(-1) + eps)+ dot = (h * weight * key).sum(-1) * rstd * self.dim**-0.5+ # signed sqrt before the sigmoid, matching the training kernel+ gate = torch.sigmoid(torch.copysign(dot.abs().clamp_min(self.clamp_value).sqrt(), dot))+ if token_mask is not None:+ gate = gate.masked_fill(~token_mask.unsqueeze(-1), 0)+ return (h + gate.unsqueeze(-1) * value.float().unsqueeze(-2)).to(x.dtype)+++@lru_cache(2)+def precompute_freqs_cis(dim, seqlen, original_seq_len, base, factor, beta_fast, beta_slow) -> torch.Tensor:+ """Rotary frequencies as complex exponentials, one row per position.++ With original_seq_len > 0 this applies YaRN: dimensions whose wavelength already fits inside the+ training context keep their frequency, those far beyond it are divided by `factor`, and the+ `beta_fast`..`beta_slow` band in between is faded across with a linear ramp.+ """+ freqs = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))+ if original_seq_len > 0:+ # the dim whose wavelength completes `rotations` turns over the training context+ def corrected_dim(rotations):+ return dim * math.log(original_seq_len / (rotations * 2 * math.pi)) / (2 * math.log(base))++ low = max(math.floor(corrected_dim(beta_fast)), 0)+ high = min(math.ceil(corrected_dim(beta_slow)), dim - 1)+ ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / max(high - low, 1e-3)).clamp(0, 1)+ smooth = 1 - ramp+ freqs = freqs / factor * (1 - smooth) + freqs * smooth++ freqs = torch.outer(torch.arange(seqlen), freqs)+ return torch.polar(torch.ones_like(freqs), freqs)+++def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor, inverse: bool = False) -> torch.Tensor:+ """Rotate `x` in place, taking adjacent element pairs as complex numbers. Accepts [b, s, d] and+ [b, s, h, d]; `inverse` conjugates the rotation, which is how the attention output gets the+ query's rotation removed again so the cache can stay in one shared rotated form."""+ y = x+ x = torch.view_as_complex(x.float().unflatten(-1, (-1, 2)))+ if inverse:+ freqs_cis = freqs_cis.conj()+ if x.ndim == 3:+ freqs_cis = freqs_cis.view(1, x.size(1), x.size(-1))+ else:+ freqs_cis = freqs_cis.view(1, x.size(1), 1, x.size(-1))+ x = torch.view_as_real(x * freqs_cis).flatten(-2)+ y.copy_(x)+ return y+++@lru_cache(1)+def get_window_topk_idxs(window_size: int, bsz: int, seqlen: int, start_pos: int):+ """Which sliding-window cache slots each query attends to; -1 marks a slot holding nothing.++ The cache is a ring of `window_size` slots. Prefill needs one row per query, each seeing its own+ causal window. A decode step has a single query that sees the whole ring, listed oldest first.+ Order within a row does not matter to `sparse_attn`, which handles every slot independently.+ """+ if start_pos == 0:+ end = torch.arange(seqlen).unsqueeze(1)+ idxs = (end - window_size + 1).clamp(0) + torch.arange(min(seqlen, window_size))+ idxs = torch.where(idxs > end, -1, idxs) # before the sequence started+ else:+ oldest = start_pos % window_size + 1+ idxs = torch.cat([torch.arange(oldest, window_size), torch.arange(oldest)])+ idxs = torch.where(idxs > start_pos, -1, idxs) # ring still filling+ # sparse_attn needs real [b, m, topk] int32 memory, hence the materializing expand+ return idxs.int().unsqueeze(0).expand(bsz, -1, -1).contiguous()+++class Compressor(nn.Module):+ """Pools `compress_ratio` consecutive tokens into one KV latent with a learned softmax gate.++ Returns the latent before RoPE, or None while a group is still filling up -- so during decode it+ only yields every `compress_ratio` steps, holding the partial group in `kv_state`/`score_state`.+ Pre-RoPE is deliberate: the indexer needs the unrotated form, so Attention rotates afterwards.+ """++ def __init__(self, args: ModelArgs, layer_id: int):+ super().__init__()+ compress_ratio = args.compress_ratios[layer_id]+ head_dim = args.head_dim+ self.compress_ratio = compress_ratio+ self.head_dim = head_dim+ self.norm = RMSNorm(head_dim, args.norm_eps)+ # ratio 1 is a plain projection, so it stays in the checkpoint's bf16; the softmax pooling+ # above ratio 1 runs in fp32, so those weights are promoted to fp32 to match+ self.wkv = Linear(args.dim, head_dim, dtype=torch.float32 if compress_ratio > 1 else torch.bfloat16)+ if compress_ratio > 1:+ self.wgate = Linear(args.dim, head_dim, dtype=torch.float32)+ # tail of an incomplete group, carried across decode steps+ self.kv_state: torch.Tensor+ self.score_state: torch.Tensor+ state_shape = (args.max_batch_size, compress_ratio, head_dim)+ self.register_buffer("kv_state", torch.zeros(state_shape, dtype=torch.float32), persistent=False)+ self.register_buffer(+ "score_state", torch.full(state_shape, -torch.inf, dtype=torch.float32), persistent=False+ )++ def forward(self, x: torch.Tensor, start_pos: int) -> torch.Tensor | None:+ bsz, seqlen, _ = x.size()+ ratio, dtype = self.compress_ratio, x.dtype+ if ratio == 1: # one token per group: nothing to pool, so no gate and no fp32+ return self.norm(self.wkv(x))++ x = x.float()+ kv, score = self.wkv(x), self.wgate(x)+ if start_pos == 0:+ should_compress = seqlen >= ratio+ remainder = seqlen % ratio+ cutoff = seqlen - remainder+ if remainder: # trailing partial group waits in the state+ kv, self.kv_state[:bsz, :remainder] = kv.split([cutoff, remainder], dim=1)+ score, self.score_state[:bsz, :remainder] = score.split([cutoff, remainder], dim=1)+ kv = kv.unflatten(1, (-1, ratio))+ score = score.unflatten(1, (-1, ratio))+ kv = (kv * score.softmax(dim=2)).sum(dim=2)+ else: # one token per step: fill a slot, and pool only when the group just completed+ should_compress = (start_pos + 1) % ratio == 0+ slot = start_pos % ratio+ self.kv_state[:bsz, slot] = kv.squeeze(1)+ self.score_state[:bsz, slot] = score.squeeze(1)+ if should_compress:+ kv = (self.kv_state[:bsz] * self.score_state[:bsz].softmax(dim=1)).sum(dim=1, keepdim=True)+ if not should_compress:+ return None+ return self.norm(kv.to(dtype))+++class Indexer(torch.nn.Module):+ """Keeps the `index_topk` best compressed positions per query.++ A small side attention: fp4 query heads against one shared key per compressed position, scores+ rectified then combined by `weights_proj`. With a candidate source this is the second of two+ levels; `select_candidate_blocks` is the first.+ """++ def __init__(self, args: ModelArgs, layer_id: int):+ super().__init__()+ # the index keys are derived from the compressor's latent, so only a layer that compresses+ # its own KV can produce them; every other indexer reads them from that layer's cache+ self.owns_k = layer_id in args.kv_source_layers+ self.compress_ratio = args.compress_ratios[layer_id]+ self.is_candidate_source = layer_id == args.candidate_source_layer+ self.uses_candidates = 0 <= args.candidate_source_layer < layer_id+ self.candidate_topk_blocks = args.candidate_topk_blocks+ self.candidate_block_size = args.candidate_block_size+ self.dim = args.dim+ self.n_heads = args.index_n_heads+ self.n_local_heads = args.index_n_heads // world_size+ self.index_head_dim = args.index_head_dim+ self.rope_head_dim = args.rope_head_dim+ self.index_topk = args.index_topk+ self.q_lora_rank = args.q_lora_rank+ self.softmax_scale = self.index_head_dim**-0.5+ self.wq_b = ColumnParallelLinear(self.q_lora_rank, self.n_heads * self.index_head_dim)+ self.weights_proj = ColumnParallelLinear(self.dim, self.n_heads, dtype=torch.bfloat16)+ self.freqs_cis: torch.Tensor | None = None+ if self.owns_k:+ self.wk = Linear(args.head_dim, self.index_head_dim, dtype=torch.bfloat16)+ self.k_norm = RMSNorm(self.index_head_dim, args.norm_eps)+ self.k_cache: torch.Tensor+ self.register_buffer(+ "k_cache",+ torch.zeros(args.max_batch_size, args.max_seq_len // self.compress_ratio, args.index_head_dim),+ persistent=False,+ )++ def forward(self, x: torch.Tensor, qr: torch.Tensor, latent: torch.Tensor, start_pos: int, offset: int):+ """`latent` is this layer's RoPE-free compressed latent, None when this layer does not+ compress or when its current group is still incomplete. An index-key owner turns it into+ index keys here, which has to happen before Attention overwrites that same storage with+ the RoPE'd, quantized values."""+ assert self.freqs_cis is not None+ bsz, seqlen, _ = x.size()+ ratio, rd, end_pos = self.compress_ratio, self.rope_head_dim, start_pos + seqlen++ # latent is None while a group is still filling up, so there is nothing to publish yet+ if self.owns_k and latent is not None:+ # a latent stands for the first token of its group, so group j takes position j * ratio+ freqs = (+ self.freqs_cis[: seqlen - seqlen % ratio : ratio]+ if start_pos == 0+ else self.freqs_cis[start_pos + 1 - ratio].unsqueeze(0)+ )+ k = self.k_norm(self.wk(latent))+ apply_rotary_emb(k[..., -rd:], freqs)+ fp4_act_quant(k, fp4_block_size, True)+ self.k_cache[:bsz, start_pos // ratio : start_pos // ratio + k.size(1)] = k+ shared_attn.index_k = self.k_cache++ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.index_head_dim))+ apply_rotary_emb(q[..., -rd:], self.freqs_cis[start_pos:end_pos])+ fp4_act_quant(q, fp4_block_size, True)++ index_k = shared_attn.index_k[:bsz, : end_pos // ratio]+ weights = self.weights_proj(x) * (self.softmax_scale * self.n_heads**-0.5)+ index_score = torch.einsum("bshd,btd->bsht", q, index_k)+ index_score = (index_score.relu_() * weights.unsqueeze(-1)).sum(dim=2)+ if world_size > 1:+ dist.all_reduce(index_score)++ # how many compressed positions each query can see: a block becomes visible once the query+ # has passed its last token. One query per decode step, so there it is just a number.+ if start_pos == 0:+ compress_lens = (torch.arange(1, seqlen + 1, device=x.device) // ratio).unsqueeze(-1)+ index_score.masked_fill_(torch.arange(seqlen // ratio, device=x.device) >= compress_lens, -torch.inf)+ else:+ compress_lens = end_pos // ratio++ if self.is_candidate_source:+ shared_attn.candidates = select_candidate_blocks(+ index_score, compress_lens, self.candidate_topk_blocks, self.candidate_block_size+ )+ elif self.uses_candidates:+ # level two: score with our own weights, but only inside the source's candidate blocks+ index_score = index_score.masked_fill(~shared_attn.candidates, -torch.inf)++ # top-k by score, re-sorted into position order; unreachable -> -1, rest shifted by offset+ topk = min(self.index_topk, end_pos // ratio)+ idxs = index_score.topk(topk, dim=-1, sorted=False).indices.sort(dim=-1).values+ return torch.where(idxs < compress_lens, idxs + offset, -1).int()++ # FreeToken: Decoder SWA Bounded Replay oracle -- the indexer for a TAIL prefill: queries are the+ # positions [s, s + n) of a prompt of L_all tokens; the keys (a source's) were published for all+ # L_all tokens by forward()'s start_pos == 0 path. Same scoring / masking / top-k as forward().+ def forward_tail(self, x, qr, s: int, n: int, offset: int):+ assert self.freqs_cis is not None+ bsz = x.size(0)+ ratio, rd = self.compress_ratio, self.rope_head_dim+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.index_head_dim))+ apply_rotary_emb(q[..., -rd:], self.freqs_cis[s : s + n])+ fp4_act_quant(q, fp4_block_size, True)+ end_pos = s + n+ index_k = shared_attn.index_k[:bsz, : end_pos // ratio]+ weights = self.weights_proj(x) * (self.softmax_scale * self.n_heads**-0.5)+ index_score = torch.einsum("bshd,btd->bsht", q, index_k)+ index_score = (index_score.relu_() * weights.unsqueeze(-1)).sum(dim=2)+ compress_lens = ((s + torch.arange(1, n + 1, device=x.device)) // ratio).unsqueeze(-1)+ index_score.masked_fill_(torch.arange(end_pos // ratio, device=x.device) >= compress_lens, -torch.inf)+ if self.is_candidate_source:+ shared_attn.candidates = select_candidate_blocks(+ index_score, compress_lens, self.candidate_topk_blocks, self.candidate_block_size+ )+ elif self.uses_candidates:+ index_score = index_score.masked_fill(~shared_attn.candidates, -torch.inf)+ topk = min(self.index_topk, end_pos // ratio)+ idxs = index_score.topk(topk, dim=-1, sorted=False).indices.sort(dim=-1).values+ return torch.where(idxs < compress_lens, idxs + offset, -1).int()+++def select_candidate_blocks(+ logits: torch.Tensor,+ compress_lens: torch.Tensor | int,+ topk_blocks: int,+ block_size: int,+) -> torch.Tensor:+ """Level one of the two-level top-k: keep the `topk_blocks` highest-scoring blocks per query.++ `logits` is [..., n_positions] with positions the query cannot reach already at -inf, which is+ what makes a block score of -inf mean "not reachable yet". `compress_lens` is a plain int during+ decode, or broadcasts against logits' leading dims during prefill. Returns a bool mask shaped+ like `logits`, so the layers consuming it just mask and never think about blocks again.+ """+ width = logits.size(-1)+ # score each block by its best position; -inf pads the last one out to block_size+ scores = F.pad(logits, (0, -width % block_size), value=-torch.inf)+ scores = scores.unflatten(-1, (-1, block_size)).amax(dim=-1)+ num_blocks = scores.size(-1)++ # the block with this query's newest position is only partly filled, so pin it in: it holds the+ # most recent tokens but could otherwise be outscored by an older, full block+ last = (compress_lens - 1) // block_size+ scores = scores.masked_fill(torch.arange(num_blocks, device=logits.device) == last, torch.inf)++ top = scores.topk(min(topk_blocks, num_blocks), dim=-1)+ # fewer reachable blocks than topk_blocks means leftover picks came back -inf: drop them+ keep = torch.zeros_like(scores, dtype=torch.bool).scatter_(-1, top.indices, top.values > -torch.inf)+ return keep.repeat_interleave(block_size, dim=-1)[..., :width]+++class Attention(nn.Module):+ """Latent attention over two KV sources at once, concatenated into one `sparse_attn` call: a+ sliding window of raw KV, plus -- when compress_ratio > 0 -- `index_topk` compressed positions+ reaching further back. Q and the output projection are both low-rank, the latter grouped.++ compress_ratio > 0 does not mean the layer compresses its own KV: only kv_source_layers do,+ the rest read that same cache.+ """++ def __init__(self, layer_id: int, args: ModelArgs):+ super().__init__()+ self.layer_id = layer_id+ self.dim = args.dim+ self.n_heads = args.n_heads+ self.n_local_heads = args.n_heads // world_size+ self.q_lora_rank = args.q_lora_rank+ self.o_lora_rank = args.o_lora_rank+ self.head_dim = args.head_dim+ self.rope_head_dim = args.rope_head_dim+ self.nope_head_dim = args.head_dim - args.rope_head_dim+ self.n_groups = args.o_groups+ self.n_local_groups = self.n_groups // world_size+ self.window_size = args.window_size+ self.compress_ratio = args.compress_ratios[layer_id]+ self.eps = args.norm_eps++ self.attn_sink = nn.Parameter(torch.empty(self.n_local_heads, dtype=torch.float32))+ self.wq_a = Linear(self.dim, self.q_lora_rank)+ self.q_norm = RMSNorm(self.q_lora_rank, self.eps)+ self.wq_b = ColumnParallelLinear(self.q_lora_rank, self.n_heads * self.head_dim)+ self.wkv = Linear(self.dim, self.head_dim)+ self.kv_norm = RMSNorm(self.head_dim, self.eps)+ self.wo_a = ColumnParallelLinear(+ self.n_heads * self.head_dim // self.n_groups,+ self.n_groups * args.o_lora_rank,+ dtype=torch.bfloat16,+ )+ self.wo_b = RowParallelLinear(self.n_groups * args.o_lora_rank, self.dim)+ self.softmax_scale = self.head_dim**-0.5++ is_backbone = layer_id < args.n_layers+ self.is_kv_source = is_backbone and layer_id in args.kv_source_layers+ self.is_index_source = is_backbone and layer_id in args.index_source_layers+ self.compressor: Compressor | None = None+ self.indexer: Indexer | None = None+ if self.is_kv_source:+ self.compressor = Compressor(args, layer_id)+ if self.is_index_source:+ self.indexer = Indexer(args, layer_id)++ self.window_kv_cache: torch.Tensor+ self.register_buffer(+ "window_kv_cache",+ torch.zeros(args.max_batch_size, args.window_size, self.head_dim),+ persistent=False,+ )+ if self.is_kv_source:+ self.compress_kv_cache: torch.Tensor+ self.register_buffer(+ "compress_kv_cache",+ torch.zeros(+ args.max_batch_size,+ args.max_seq_len // self.compress_ratio,+ self.head_dim,+ ),+ persistent=False,+ )+ if self.compress_ratio:+ original_seq_len, rope_theta = (+ args.original_seq_len,+ args.compress_rope_theta,+ )+ else:+ # disable YaRN and use base rope_theta in pure sliding-window attention+ original_seq_len, rope_theta = 0, args.rope_theta+ freqs_cis = precompute_freqs_cis(+ self.rope_head_dim,+ args.max_seq_len,+ original_seq_len,+ rope_theta,+ args.rope_factor,+ args.beta_fast,+ args.beta_slow,+ )+ self.freqs_cis: torch.Tensor+ self.register_buffer("freqs_cis", freqs_cis, persistent=False)++ def _window_kv(self, x, freqs_cis, start_pos):+ """This layer's sliding-window K and the window positions every query may attend to. The K+ stays fp8, quantized over the whole post-RoPE vector, RoPE tail included."""+ bsz, seqlen, _ = x.size()+ win = self.window_size+ kv = self.kv_norm(self.wkv(x))+ apply_rotary_emb(kv[..., -self.rope_head_dim :], freqs_cis)+ act_quant(kv, fp8_block_size, scale_fmt, scale_dtype, True)+ if start_pos == 0: # prefill: attend over this chunk, seeding the ring buffer for decode+ if seqlen <= win:+ self.window_kv_cache[:bsz, :seqlen] = kv+ else:+ cutoff = seqlen % win+ self.window_kv_cache[:bsz, cutoff:win], self.window_kv_cache[:bsz, :cutoff] = kv[:, -win:].split(+ [win - cutoff, cutoff], dim=1+ )+ window_kv = kv+ else: # decode: one token into the ring buffer, attend over the whole window+ self.window_kv_cache[:bsz, start_pos % win] = kv.squeeze(1)+ window_kv = self.window_kv_cache[:bsz]+ return window_kv, get_window_topk_idxs(win, bsz, seqlen, start_pos)++ def _compress_topk_idxs(self, x, qr, latent, start_pos, offset, compress_len):+ """Which compressed positions each query attends to. Index sources run their own indexer;+ the layers in between reuse the result their source published."""+ if not self.is_index_source:+ return shared_attn.topk_idxs++ bsz, seqlen, _ = x.size()+ if compress_len == 0:+ idxs = torch.empty(bsz, seqlen, 0, dtype=torch.int32, device=x.device)+ else:+ assert self.indexer is not None+ if self.indexer.freqs_cis is None:+ self.indexer.freqs_cis = self.freqs_cis+ idxs = self.indexer(x, qr, latent, start_pos, offset)+ shared_attn.topk_idxs = idxs+ return idxs++ def _compress_kv(self, x, qr, start_pos, offset):+ """The shared compressed KV and the compressed positions every query may attend to. This+ layer compresses its own KV only when it is a source; otherwise it just reads the cache."""+ bsz, seqlen, _ = x.size()+ ratio = self.compress_ratio+ compress_len = (start_pos + seqlen) // ratio+ latent = None+ if self.is_kv_source:+ latent = self.compressor(x, start_pos)+ shared_attn.compress_kv = self.compress_kv_cache+ # the indexer needs the latent before RoPE, so it runs before the cache is written+ idxs = self._compress_topk_idxs(x, qr, latent, start_pos, offset, compress_len)+ if latent is not None:+ # a latent stands for the first token of its group, so group j takes position j * ratio+ freqs = (+ self.freqs_cis[: seqlen - seqlen % ratio : ratio]+ if start_pos == 0+ else self.freqs_cis[start_pos + 1 - ratio].unsqueeze(0)+ )+ apply_rotary_emb(latent[..., -self.rope_head_dim :], freqs)+ # Compressed KV uses groups of 16 with E4M3 scales; the indexer uses 32 with E8M0.+ fp4_act_quant(latent, 16, True, scale_dtype=torch.float8_e4m3fn)+ self.compress_kv_cache[:bsz, start_pos // ratio : start_pos // ratio + latent.size(1)] = latent+ # read after the write, so this does not depend on the slice aliasing the cache+ return shared_attn.compress_kv[:bsz, :compress_len], idxs++ def forward(self, x: torch.Tensor, start_pos: int):+ bsz, seqlen, _ = x.size()+ freqs_cis = self.freqs_cis[start_pos : start_pos + seqlen]+ rd = self.rope_head_dim++ qr = self.q_norm(self.wq_a(x))+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.head_dim))+ apply_rotary_emb(q[..., -rd:], freqs_cis)++ kv, topk_idxs = self._window_kv(x, freqs_cis, start_pos)+ if self.compress_ratio:+ compress_kv, compress_idxs = self._compress_kv(x, qr, start_pos, kv.size(1))+ kv = torch.cat([kv, compress_kv], dim=1)+ topk_idxs = torch.cat([topk_idxs, compress_idxs], dim=-1)++ o = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)+ apply_rotary_emb(o[..., -rd:], freqs_cis, True)++ # wo_a is block-diagonal over groups (each projects only its own heads), hence einsum not+ # Linear. convert.py dequantizes it to bf16; an fp8 grouped GEMM would halve the memory.+ o = o.view(bsz, seqlen, self.n_local_groups, -1)+ wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)+ o = torch.einsum("bsgd,grd->bsgr", o, wo_a)+ x = self.wo_b(o.flatten(2))+ return x++ # FreeToken: Decoder SWA Bounded Replay oracle. Prefill of the prompt TAIL [s, s + n) as its own+ # window segment (a query sees SWA keys in [max(s, i - win + 1), i]) while the global branch+ # covers the whole prompt: a kv source compresses ``x_all`` (every token, forward()'s start_pos+ # == 0 path), the indexer scores the tail queries at their absolute positions.+ def forward_tail(self, x: torch.Tensor, s: int, x_all: torch.Tensor | None = None):+ bsz, n, _ = x.size()+ win, rd, ratio = self.window_size, self.rope_head_dim, self.compress_ratio+ freqs_cis = self.freqs_cis[s : s + n]++ qr = self.q_norm(self.wq_a(x))+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.head_dim))+ apply_rotary_emb(q[..., -rd:], freqs_cis)++ kv = self.kv_norm(self.wkv(x))+ apply_rotary_emb(kv[..., -rd:], freqs_cis)+ act_quant(kv, fp8_block_size, scale_fmt, scale_dtype, True)+ # seed the decode ring with the tail's keys at their absolute slots+ for i in range(max(0, n - win), n):+ self.window_kv_cache[:bsz, (s + i) % win] = kv[:, i]+ topk_idxs = get_window_topk_idxs(win, bsz, n, 0).to(x.device)++ if ratio:+ end = s + n+ compress_len = end // ratio+ latent = None+ if self.is_kv_source:+ assert x_all is not None, "the kv source compresses every prompt token"+ latent = self.compressor(x_all, 0)+ shared_attn.compress_kv = self.compress_kv_cache+ if self.indexer is not None:+ if self.indexer.freqs_cis is None:+ self.indexer.freqs_cis = self.freqs_cis+ # publish the index keys for every group (forward()'s start_pos == 0 path)+ L = x_all.size(1)+ k = self.indexer.k_norm(self.indexer.wk(latent))+ apply_rotary_emb(k[..., -rd:], self.freqs_cis[: L - L % ratio : ratio])+ fp4_act_quant(k, fp4_block_size, True)+ self.indexer.k_cache[:bsz, : k.size(1)] = k+ shared_attn.index_k = self.indexer.k_cache+ if not self.is_index_source:+ idxs = shared_attn.topk_idxs+ elif compress_len == 0:+ idxs = torch.empty(bsz, n, 0, dtype=torch.int32, device=x.device)+ else:+ if self.indexer.freqs_cis is None:+ self.indexer.freqs_cis = self.freqs_cis+ idxs = self.indexer.forward_tail(x, qr, s, n, kv.size(1))+ shared_attn.topk_idxs = idxs+ if latent is not None:+ L = x_all.size(1)+ apply_rotary_emb(latent[..., -rd:], self.freqs_cis[: L - L % ratio : ratio])+ fp4_act_quant(latent, 16, True, scale_dtype=torch.float8_e4m3fn)+ self.compress_kv_cache[:bsz, : latent.size(1)] = latent+ kv = torch.cat([kv, shared_attn.compress_kv[:bsz, :compress_len]], dim=1)+ topk_idxs = torch.cat([topk_idxs, idxs], dim=-1)++ o = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)+ apply_rotary_emb(o[..., -rd:], freqs_cis, True)+ o = o.view(bsz, n, self.n_local_groups, -1)+ wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)+ o = torch.einsum("bsgd,grd->bsgr", o, wo_a)+ return self.wo_b(o.flatten(2))+++class Gate(nn.Module):+ """MoE gating. The correction bias steers expert selection only; the routing weights come from the+ unbiased scores. Image-span tokens use a separate bias (training `noaux_tc_for_vl`)."""++ def __init__(self, layer_id: int, args: ModelArgs):+ super().__init__()+ n_routed_experts, n_activated_experts = args.get_moe_config(layer_id)+ self.dim = args.dim+ self.topk = n_activated_experts+ self.score_func = args.score_func+ self.gate_temp = args.gate_temp+ self.norm_topk_prob = args.norm_topk_prob+ self.route_scale = args.route_scale+ self.weight = nn.Parameter(torch.empty(n_routed_experts, args.dim))+ self.bias = nn.Parameter(torch.empty(n_routed_experts, dtype=torch.float32))+ self.bias_vl = nn.Parameter(torch.empty(n_routed_experts, dtype=torch.float32)) if args.vision_enabled else None++ def forward(self, x: torch.Tensor, image_mask: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor]:+ """x: [n, dim]; image_mask: [n] bool, True for tokens inside an image span."""+ scores = linear(x.float(), self.weight.float()) / self.gate_temp+ if self.score_func == "softmax":+ scores = scores.softmax(dim=-1)+ elif self.score_func == "sigmoid":+ scores = scores.sigmoid()+ else:+ scores = F.softplus(scores).sqrt()+ bias = self.bias+ if image_mask is not None and self.bias_vl is not None:+ bias = torch.where(image_mask.unsqueeze(-1), self.bias_vl, bias)+ # the bias picks experts but does not scale them: weights come from the raw scores+ indices = (scores + bias).topk(self.topk, dim=-1)[1]+ weights = scores.gather(1, indices)+ if self.norm_topk_prob and self.topk > 1:+ weights /= weights.sum(dim=-1, keepdim=True) + 1e-20 # not norm_eps, matches training+ weights *= self.route_scale+ return weights, indices+++class Expert(nn.Module):+ """One SwiGLU FFN. The clamps come straight from training, where they keep fp8/fp4 activations in+ range: the up branch is clamped on both sides, the gate branch only from above."""++ def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0.0):+ super().__init__()+ self.w1 = Linear(dim, inter_dim, dtype=dtype)+ self.w2 = Linear(inter_dim, dim, dtype=dtype)+ self.w3 = Linear(dim, inter_dim, dtype=dtype)+ self.swiglu_limit = swiglu_limit++ def forward(self, x: torch.Tensor, weights: torch.Tensor | None = None) -> torch.Tensor:+ dtype = x.dtype+ gate = self.w1(x).float()+ up = self.w3(x).float()+ if self.swiglu_limit > 0:+ up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)+ gate = torch.clamp(gate, max=self.swiglu_limit)+ x = F.silu(gate) * up+ if weights is not None:+ x = weights * x+ return self.w2(x.to(dtype))+++class MoE(nn.Module):+ """Top-k routed experts plus one shared expert every token goes through. Experts are split+ across ranks, so `self.experts` is None for those another rank owns."""++ def __init__(self, layer_id: int, args: ModelArgs):+ super().__init__()+ n_routed_experts, n_activated_experts = args.get_moe_config(layer_id)+ self.layer_id = layer_id+ self.dim = args.dim+ assert n_routed_experts % world_size == 0, (+ f"Number of experts must be divisible by world size (world_size={world_size})"+ )+ self.n_routed_experts = n_routed_experts+ self.n_local_experts = n_routed_experts // world_size+ self.n_activated_experts = n_activated_experts+ self.experts_start_idx = rank * self.n_local_experts+ self.experts_end_idx = self.experts_start_idx + self.n_local_experts+ self.gate = Gate(layer_id, args)+ expert_dtype = torch.float4_e2m1fn_x2 if args.expert_dtype == "fp4" else None+ self.experts = nn.ModuleList(+ [+ Expert(+ args.dim,+ args.moe_inter_dim,+ dtype=expert_dtype,+ swiglu_limit=args.swiglu_limit,+ )+ if self.experts_start_idx <= i < self.experts_end_idx+ else None+ for i in range(self.n_routed_experts)+ ]+ )+ assert args.n_shared_experts == 1+ self.shared_experts = Expert(args.dim, args.moe_inter_dim, swiglu_limit=args.swiglu_limit)++ def forward(self, x: torch.Tensor, image_mask: torch.Tensor | None = None) -> torch.Tensor:+ shape = x.size()+ x = x.view(-1, self.dim)+ weights, indices = self.gate(x, None if image_mask is None else image_mask.flatten())+ y = torch.zeros_like(x, dtype=torch.float32)+ counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()+ for i in range(self.experts_start_idx, self.experts_end_idx):+ if counts[i] == 0:+ continue+ expert = self.experts[i]+ idx, top = torch.where(indices == i)+ y[idx] += expert(x[idx], weights[idx, top, None])+ if world_size > 1:+ dist.all_reduce(y)+ y += self.shared_experts(x)+ return y.type_as(x).view(shape)+++class Block(nn.Module):+ """A block whose residual stream is `hc_mult` parallel copies (Hyper-Connections).++ Attention and FFN each sit between `hc_pre` (collapse the copies into one sublayer input) and+ `hc_post` (expand back out, mixing the residual in through `comb`). `hc_mixes` derives all three+ coefficient sets from the stream itself, `comb` made doubly stochastic by Sinkhorn.++ The coefficients a sublayer computes are used by the *next* one -- see `forward`.+ """++ attention_cls = Attention++ def __init__(+ self,+ layer_id: int,+ args: ModelArgs,+ engram_layout: EngramLayout | None = None,+ ):+ super().__init__()+ self.layer_id = layer_id+ self.norm_eps = args.norm_eps+ self.attn = self.attention_cls(layer_id, args)+ self.ffn = MoE(layer_id, args)+ self.engram = None+ if engram_layout is not None and layer_id in engram_layout.layer_ids:+ self.engram = Engram(args, layer_id, engram_layout)+ self.attn_norm = RMSNorm(args.dim, self.norm_eps)+ self.ffn_norm = RMSNorm(args.dim, self.norm_eps)+ self.hc_mult = hc_mult = args.hc_mult+ self.hc_sinkhorn_iters = args.hc_sinkhorn_iters+ self.hc_eps = args.hc_eps+ mix_hc = (2 + hc_mult) * hc_mult+ hc_dim = hc_mult * args.dim+ with set_dtype(torch.float32):+ self.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))+ self.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))+ self.hc_attn_base = nn.Parameter(torch.empty(mix_hc))+ self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc))+ self.hc_attn_scale = nn.Parameter(torch.empty(3))+ self.hc_ffn_scale = nn.Parameter(torch.empty(3))++ def hc_mixes(self, x: torch.Tensor, hc_fn: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor):+ """x: [b,s,hc,d], hc_fn: [mix_hc, hc*d], hc_scale: [3], hc_base: [mix_hc]. Returns the+ pre / post / comb coefficients, split out of one projection of the flattened stream."""+ # normalized over the whole flattened hc*d stream, one statistic per token+ x = x.flatten(2).float()+ rsqrt = torch.rsqrt(x.square().mean(-1, keepdim=True) + self.norm_eps)+ mixes = F.linear(x, hc_fn) * rsqrt+ return hc_split_sinkhorn(mixes, hc_scale, hc_base, self.hc_mult, self.hc_sinkhorn_iters, self.hc_eps)++ def hc_pre(self, x: torch.Tensor, pre_mix: torch.Tensor):+ """Collapse the hc copies into one, weighted by pre_mix. [b,s,hc,d] x [b,s,hc] -> [b,s,d]"""+ y = torch.sum(pre_mix.unsqueeze(-1) * x.float(), dim=2)+ return y.to(x.dtype)++ def hc_post(self, x: torch.Tensor, residual: torch.Tensor, post: torch.Tensor, comb: torch.Tensor):+ """Expand the sublayer output back to hc copies and mix the residual in through `comb`.+ x: [b,s,d], residual: [b,s,hc,d], post: [b,s,hc], comb: [b,s,hc,hc] -> [b,s,hc,d]"""+ y = post.unsqueeze(-1) * x.unsqueeze(-2) + torch.sum(comb.unsqueeze(-1) * residual.unsqueeze(-2), dim=2)+ return y.type_as(x)++ def forward(+ self,+ x: torch.Tensor,+ start_pos: int,+ pre_mix: torch.Tensor,+ image_mask: torch.Tensor | None,+ *attn_args,+ ) -> tuple[torch.Tensor, torch.Tensor]:+ """`pre_mix` collapses the hc_mult copies down to one input for this block's attention. Each+ sub-block's own `hc_mixes` produces the mix for the *next* one, so attention uses what the+ previous layer's FFN produced and the FFN uses what this attention produced.++ image_mask: [b, s] bool, True inside image spans (selects the VL routing bias)."""+ residual = x+ attn_pre, attn_post, attn_comb = self.hc_mixes(x, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)+ x = self.hc_pre(x, pre_mix)+ x = self.attn_norm(x)+ x = self.attn(x, start_pos, *attn_args)+ x = self.hc_post(x, residual, attn_post, attn_comb)++ residual = x+ ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)+ x = self.hc_pre(x, attn_pre)+ x = self.ffn_norm(x)+ x = self.ffn(x, image_mask)+ x = self.hc_post(x, residual, ffn_post, ffn_comb)+ return x, ffn_pre++ # FreeToken: Decoder SWA Bounded Replay oracle. ``h`` / ``pre_mix`` are the tail [s, N) streams,+ # or -- at the decoder's kv source -- ``h_all`` / ``pre_mix_all`` cover every token so the+ # source compresses them all before the block narrows to the tail.+ def forward_tail(self, h, s: int, pre_mix, h_all=None, pre_mix_all=None):+ if h_all is not None:+ attn_pre, attn_post, attn_comb = self.hc_mixes(h_all, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)+ x_all = self.attn_norm(self.hc_pre(h_all, pre_mix_all))+ residual = h_all[:, s:]+ attn_pre, attn_post, attn_comb = attn_pre[:, s:], attn_post[:, s:], attn_comb[:, s:]+ x = self.attn.forward_tail(x_all[:, s:], s, x_all)+ else:+ residual = h+ attn_pre, attn_post, attn_comb = self.hc_mixes(h, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)+ x = self.attn_norm(self.hc_pre(h, pre_mix))+ x = self.attn.forward_tail(x, s)+ x = self.hc_post(x, residual, attn_post, attn_comb)++ residual = x+ ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)+ x = self.hc_pre(x, attn_pre)+ x = self.ffn_norm(x)+ x = self.ffn(x, None)+ x = self.hc_post(x, residual, ffn_post, ffn_comb)+ return x, ffn_pre+++class ParallelHead(nn.Module):+ def __init__(self, vocab_size: int, dim: int, norm_eps: float = 1e-6, hc_eps: float = 1e-6):+ super().__init__()+ self.vocab_size = vocab_size+ self.dim = dim+ self.norm_eps = norm_eps+ self.hc_eps = hc_eps+ self.part_vocab_size = vocab_size // world_size+ # bf16 in the checkpoint, kept as fp32 here so the logits come out in fp32 directly+ self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim, dtype=torch.float32))++ def forward(self, x: torch.Tensor, full_logits=False):+ """x: [b, s, d]. Generation only needs the last position, so that is the default."""+ if not full_logits:+ x = x[:, -1]+ logits = F.linear(x.float(), self.weight)+ if world_size > 1:+ all_logits = [torch.empty_like(logits) for _ in range(world_size)]+ dist.all_gather(all_logits, logits)+ logits = torch.cat(all_logits, dim=-1)+ return logits+++@lru_cache(1)+def get_dspark_topk_idxs(window_size: int, bsz: int, block_size: int, start_pos: int):+ assert start_pos > 0+ matrix = torch.cat(+ [+ torch.arange(min(window_size, start_pos + 1)),+ window_size + torch.arange(block_size),+ ]+ )+ return matrix.int().view(1, 1, -1).expand(bsz, block_size, -1).contiguous()+++class DSparkAttention(Attention):+ def forward(self, x: torch.Tensor, start_pos: int, main_x: torch.Tensor):+ assert self.compress_ratio == 0+ bsz, seqlen, _ = main_x.size()+ win = self.window_size+ rd = self.rope_head_dim++ main_freqs_cis = self.freqs_cis[start_pos : start_pos + seqlen]+ main_kv = self.kv_norm(self.wkv(main_x))+ apply_rotary_emb(main_kv[..., -rd:], main_freqs_cis)+ act_quant(main_kv, fp8_block_size, scale_fmt, scale_dtype, True)++ if start_pos == 0:+ if seqlen <= win:+ self.window_kv_cache[:bsz, :seqlen] = main_kv+ else:+ cutoff = seqlen % win+ self.window_kv_cache[:bsz, cutoff:win], self.window_kv_cache[:bsz, :cutoff] = main_kv[:, -win:].split(+ [win - cutoff, cutoff], dim=1+ )+ return x++ bsz, block_size, _ = x.size()+ freqs_cis = self.freqs_cis[start_pos + seqlen : start_pos + seqlen + block_size]++ qr = self.q_norm(self.wq_a(x))+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.head_dim))+ apply_rotary_emb(q[..., -rd:], freqs_cis)+ kv = self.kv_norm(self.wkv(x))+ apply_rotary_emb(kv[..., -rd:], freqs_cis)+ act_quant(kv, fp8_block_size, scale_fmt, scale_dtype, True)++ topk_idxs = get_dspark_topk_idxs(win, bsz, block_size, start_pos)+ self.window_kv_cache[:bsz, start_pos % win] = main_kv.squeeze(1)+ kv = torch.cat([self.window_kv_cache[:bsz], kv], dim=1)+ o = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)+ apply_rotary_emb(o[..., -rd:], freqs_cis, True)++ o = o.view(bsz, block_size, self.n_local_groups, -1)+ wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)+ o = torch.einsum("bsgd,grd->bsgr", o, wo_a)+ x = self.wo_b(o.flatten(2))+ return x+++class DSparkMarkovHead(nn.Module):+ def __init__(self, vocab_size: int, dspark_markov_rank: int):+ super().__init__()+ self.embed = ParallelEmbedding(vocab_size, dspark_markov_rank)+ self.head = ParallelHead(vocab_size, dspark_markov_rank)++ def forward(self, token_ids: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:+ embed = self.embed(token_ids)+ logits = self.head(embed, full_logits=True)+ return logits, embed+++class DSparkConfidenceHead(nn.Module):+ def __init__(self, input_dim: int):+ super().__init__()+ # proj in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for fp32 confidence score.+ self.proj = Linear(input_dim, 1, dtype=torch.float32)++ def forward(self, hidden: torch.Tensor, markov_embed: torch.Tensor):+ hidden = torch.cat([hidden, markov_embed], dim=-1)+ return self.proj(hidden.float()).squeeze(-1)+++class DSparkBlock(Block):+ """DSpark stage stored under the mtp.* checkpoint namespace."""++ attention_cls = DSparkAttention++ def __init__(self, layer_id: int, args: ModelArgs):+ super().__init__(layer_id, args)+ stage_id = layer_id - args.n_layers+ self.block_size = args.dspark_block_size+ self.noise_token_id = args.dspark_noise_token_id+ self.temperature = args.temperature+ if stage_id == 0:+ assert len(args.dspark_target_layer_ids) > 0, "DSpark needs target layers"+ self.main_proj = Linear(args.dim * len(args.dspark_target_layer_ids), args.dim)+ self.main_norm = RMSNorm(args.dim, args.norm_eps)+ if stage_id == args.n_mtp_layers - 1:+ self.norm = RMSNorm(args.dim, args.norm_eps)+ self.markov_head = DSparkMarkovHead(args.vocab_size, args.dspark_markov_rank)+ self.confidence_head = DSparkConfidenceHead(args.dim + args.dspark_markov_rank)+ self.embed: ParallelEmbedding | None = None+ self.head: ParallelHead | None = None++ def forward(self, x: torch.Tensor, start_pos: int, pre_mix: torch.Tensor, main_x: torch.Tensor):+ if start_pos == 0:+ self.attn(x, start_pos, main_x) # prefill only seeds the window KV cache+ return x, pre_mix+ return super().forward(x, start_pos, pre_mix, None, main_x) # drafts are text: no VL bias++ def forward_embed(self, main_hidden: torch.Tensor, input_ids: torch.Tensor):+ assert self.embed is not None+ main_x = self.main_norm(self.main_proj(main_hidden))+ draft_input_ids = input_ids.new_full([input_ids.size(0), self.block_size], self.noise_token_id)+ draft_input_ids[:, 0] = input_ids+ x = self.embed(draft_input_ids)+ x = x.unsqueeze(2).repeat(1, 1, self.hc_mult, 1)+ return x, main_x++ def forward_head(+ self,+ x: torch.Tensor,+ pre_mix: torch.Tensor,+ input_ids: torch.Tensor,+ ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ assert self.head is not None+ x = self.hc_pre(x, pre_mix)+ logits = self.head(self.norm(x), full_logits=True)+ output_ids = input_ids.new_empty(input_ids.size(0), self.block_size + 1)+ output_ids[:, 0] = input_ids+ markov_embeds = []+ for i in range(self.block_size):+ logits_bias, markov_embed = self.markov_head(output_ids[:, i])+ logits[:, i].add_(logits_bias)+ markov_embeds.append(markov_embed)+ output_ids[:, i + 1] = sample(logits[:, i], self.temperature)+ markov_embed = torch.stack(markov_embeds, dim=1)+ confidence = self.confidence_head(x, markov_embed)+ return output_ids, logits, confidence+++def make_identity_pre_mix(x: torch.Tensor, hc_mult: int) -> torch.Tensor:+ """initial one-hot mix"""+ pre_mix = x.new_zeros(x.size(0), x.size(1), hc_mult, dtype=torch.float32)+ pre_mix[:, :, 0] = 1.0+ return pre_mix+++class SharedAttentionRuntime:+ """What attention layers hand down the stack instead of recomputing. Layers run in order and+ every source writes before its consumers read, so one slot each is enough and nothing needs+ resetting between forwards. Sources: compress_kv and index_k from kv_source_layers, topk_idxs+ from index_source_layers, candidates from candidate_source_layer."""++ def __init__(self):+ self.compress_kv: torch.Tensor | None = None+ self.index_k: torch.Tensor | None = None+ self.topk_idxs: torch.Tensor | None = None+ self.candidates: torch.Tensor | None = None+++# Only ever one model per process, same as world_size / rank / default_dtype above.+shared_attn = SharedAttentionRuntime()+++class Transformer(nn.Module):+ """DeepSeek-V4.1: embed -> expand to hc_mult copies -> blocks -> collapse -> logits. Building+ this sets the globals at the top of the file. The tokenizer only feeds the engram token map."""++ def __init__(self, args: ModelArgs, tokenizer=None):+ global world_size, rank, default_dtype+ world_size = dist.get_world_size() if dist.is_initialized() else 1+ rank = dist.get_rank() if dist.is_initialized() else 0+ default_dtype = torch.float8_e4m3fn if args.dtype == "fp8" else torch.bfloat16+ super().__init__()+ self.max_seq_len = args.max_seq_len+ self.temperature = args.temperature+ self.norm_eps = args.norm_eps+ self.hc_eps = args.hc_eps+ self.engram_layout = EngramLayout.from_args(args)+ self.engram_hash = (+ NgramHashState(args, self.engram_layout, tokenizer) if self.engram_layout is not None else None+ )+ self.embed = ParallelEmbedding(args.vocab_size, args.dim)+ self.layers = torch.nn.ModuleList()+ for layer_id in range(args.n_layers):+ self.layers.append(Block(layer_id, args, self.engram_layout))+ self.norm = RMSNorm(args.dim, self.norm_eps)+ self.head = ParallelHead(args.vocab_size, args.dim, self.norm_eps, self.hc_eps)+ self.mtp = torch.nn.ModuleList()+ self.target_layer_ids = args.dspark_target_layer_ids+ if args.dspark_block_size:+ for layer_id in range(args.n_mtp_layers):+ self.mtp.append(DSparkBlock(args.n_layers + layer_id, args))+ self.mtp[-1].embed = self.embed+ self.mtp[-1].head = self.head+ self.hc_mult = args.hc_mult+ self.vision = None+ if args.vision_enabled:+ self.vision = ViT(args)+ self.aligner = Aligner(args)+ # learned embeddings for the image span delimiters+ self.image_start = nn.Parameter(torch.empty(args.dim))+ self.image_end = nn.Parameter(torch.empty(args.dim))+ self.image_newline = nn.Parameter(torch.empty(args.dim))++ @torch.inference_mode()+ def encode_image(self, patches: torch.Tensor, n_vit_h: int, n_vit_w: int) -> torch.Tensor:+ return self.aligner(self.vision(patches, n_vit_h, n_vit_w), n_vit_h, n_vit_w)++ def merge_image_embeddings(self, images, h: torch.Tensor):+ """Overwrite each image's token span in h with its ViT/aligner features. The IMAGE slots take+ the aligner rows in row-major order; the span delimiters take learned embeddings."""+ for i, sample in enumerate(images):+ for img in sample or ():+ types = img.types.to(h.device)+ span = h[i, img.start : img.start + types.numel()]+ span[types == IMAGE_START] = self.image_start.to(h.dtype)+ span[types == IMAGE_END] = self.image_end.to(h.dtype)+ span[types == IMAGE_NEW_LINE] = self.image_newline.to(h.dtype)+ embeds = self.encode_image(img.patches.to(h.device), img.n_vit_h, img.n_vit_w)+ span[types == IMAGE] = embeds.to(h.dtype)++ @torch.inference_mode()+ def forward(+ self, input_ids: torch.Tensor, start_pos: int = 0, images=None, token_types: torch.Tensor | None = None+ ):+ """input_ids: [b, s], every entry a real token id -- generate.py only ever passes positions it+ has already filled, so the padding it uses internally never reaches here. token_types /+ images carry the VL inputs built by image_processor.prepare_vl_inputs; image spans must lie+ inside the first (start_pos 0) chunk."""+ image_mask = None if token_types is None else token_types >= 0 # TEXT is -1+ # image tokens take no part in an n-gram and get no engram contribution; text-only needs no mask+ engram_mask = None if image_mask is None else ~image_mask+ engram_hashes = self.engram_hash(input_ids, start_pos, engram_mask) if self.engram_hash is not None else None+ h = self.embed(input_ids)+ if images is not None:+ assert start_pos == 0, "image spans must be prefilled in a single chunk"+ self.merge_image_embeddings(images, h)+ # Expand to hc_mult copies for Hyper-Connections+ h = h.unsqueeze(2).repeat(1, 1, self.hc_mult, 1)+ main_hiddens = []+ pre_mix = make_identity_pre_mix(h, self.hc_mult)+ for i, layer in enumerate(self.layers):+ if layer.engram is not None:+ h = layer.engram(h, engram_hashes[:, :, layer.engram.layer_hash_index, :], engram_mask)+ # the MTP head reads the attention input of its target layers, not their output+ if i in self.target_layer_ids:+ main_hiddens.append(h.mean(dim=2))+ h, pre_mix = layer(h, start_pos, pre_mix, image_mask)+ h = layer.hc_pre(h, pre_mix)+ logits = self.head(self.norm(h))+ output_ids = sample(logits, self.temperature)+ main_hidden = torch.cat(main_hiddens, dim=-1) if main_hiddens else None+ return output_ids, logits, main_hidden++ # FreeToken: Decoder SWA Bounded Replay oracle (tech report sec. 3.2.2): the encoder runs on+ # every prompt token, the decoder on the last ``n_win`` with its sliding window truncated+ # there; the decoder's kv source (the last kv_source layer) compresses every token. Returns+ # the last position's logits and leaves the caches ready for decode from position seqlen.+ @torch.inference_mode()+ def forward_bounded(self, input_ids: torch.Tensor, n_win: int):+ engram_hashes = self.engram_hash(input_ids, 0, None) if self.engram_hash is not None else None+ h = self.embed(input_ids).unsqueeze(2).repeat(1, 1, self.hc_mult, 1)+ pre_mix = make_identity_pre_mix(h, self.hc_mult)+ N = input_ids.size(1)+ s = max(0, N - n_win)+ dec_start = max(i for i, layer in enumerate(self.layers) if layer.attn.is_kv_source)+ for i, layer in enumerate(self.layers):+ if layer.engram is not None:+ assert i < dec_start, "Engram sits in the encoder"+ h = layer.engram(h, engram_hashes[:, :, layer.engram.layer_hash_index, :], None)+ if i < dec_start:+ h, pre_mix = layer(h, 0, pre_mix, None)+ elif i == dec_start:+ h, pre_mix = layer.forward_tail(None, s, None, h_all=h, pre_mix_all=pre_mix)+ else:+ h, pre_mix = layer.forward_tail(h, s, pre_mix)+ h = layer.hc_pre(h, pre_mix)+ return self.head(self.norm(h))++ @torch.inference_mode()+ def forward_spec(self, input_ids: torch.Tensor, main_hidden: torch.Tensor, start_pos: int = 0):+ h, main_x = self.mtp[0].forward_embed(main_hidden, input_ids)+ pre_mix = make_identity_pre_mix(h, self.hc_mult)+ for layer in self.mtp:+ h, pre_mix = layer(h, start_pos, pre_mix, main_x)+ if start_pos == 0:+ return None+ return self.mtp[-1].forward_head(h, pre_mix, input_ids)+++def sample(logits, temperature: float = 1.0):+ """Gumbel-max trick: equivalent to multinomial sampling but faster on GPU,+ since it avoids the GPU-to-CPU sync in torch.multinomial."""+ if temperature == 0:+ return logits.argmax(dim=-1)+ logits = logits / max(temperature, 1e-5)+ probs = torch.softmax(logits, dim=-1, dtype=torch.float32)+ return probs.div_(torch.empty_like(probs).exponential_(1)).argmax(dim=-1)+++if __name__ == "__main__":+ torch.set_default_dtype(torch.bfloat16)+ torch.set_default_device("cuda")+ torch.manual_seed(0)+ args = ModelArgs(dspark_block_size=6, dspark_target_layer_ids=(3, 4))+ x = torch.randint(0, args.vocab_size, (2, 150))+ model = Transformer(args)++ output_ids, logits, main_hidden = model(x[:, :128])+ model.forward_spec(output_ids, main_hidden)+ for i in range(128, 150):+ output_ids, logits, main_hidden = model(x[:, i : i + 1], i)+ result = model.forward_spec(output_ids, main_hidden, i)+ assert result is not None+ output_ids, logits, confidence = resultdiff --git a/tests/models/deepseek_v41/reference/vision.py b/tests/models/deepseek_v41/reference/vision.pynew file mode 100644index 00000000..2fd86160--- /dev/null+++ b/tests/models/deepseek_v41/reference/vision.py@@ -0,0 +1,121 @@+# Vendored from deepseek-ai/DeepSeek-V4.1-Flash, revision dba1be0a, inference/vision.py.+# MIT License; see LICENSE in this directory.+from functools import lru_cache++import torch+import torch.nn.functional as F+from torch import nn+++@lru_cache(8)+def get_vision_cos_sin(n_h: int, n_w: int, dim: int, theta: float):+ inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))+ hpos = torch.arange(n_h).unsqueeze(1).expand(n_h, n_w)+ wpos = torch.arange(n_w).unsqueeze(0).expand(n_h, n_w)+ freqs = torch.stack([hpos, wpos], dim=-1).reshape(-1, 2, 1).float() * inv_freq+ freqs = freqs.flatten(1)+ return freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1)+++def apply_rotary(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:+ dtype = x.dtype+ x1, x2 = x.float().chunk(2, dim=-1)+ return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1).to(dtype)+++class RMSNorm(nn.Module):+ def __init__(self, dim: int, eps: float = 1e-6):+ super().__init__()+ self.eps = eps+ self.weight = nn.Parameter(torch.ones(dim, dtype=torch.float32))++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ dtype = x.dtype+ x = x.float()+ x = x * torch.rsqrt(x.square().mean(-1, keepdim=True) + self.eps)+ return (self.weight * x).to(dtype)+++class PatchEmbed(nn.Module):+ def __init__(self, args):+ super().__init__()+ self.proj = nn.Linear(3 * args.vision_patch_size**2, args.vision_dim)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ return self.proj(x.flatten(1))+++class Attention(nn.Module):+ def __init__(self, args):+ super().__init__()+ self.n_heads = args.vision_n_heads+ self.head_dim = args.vision_dim // args.vision_n_heads+ self.wqkv = nn.Linear(args.vision_dim, 3 * args.vision_dim)+ self.wo = nn.Linear(args.vision_dim, args.vision_dim)++ def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:+ n = x.size(0)+ q, k, v = (t.view(n, self.n_heads, self.head_dim) for t in self.wqkv(x).chunk(3, dim=-1))+ q = apply_rotary(q, cos, sin)+ k = apply_rotary(k, cos, sin)+ o = F.scaled_dot_product_attention(q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1))+ return self.wo(o.transpose(0, 1).reshape(n, -1))+++class MLP(nn.Module):+ def __init__(self, args):+ super().__init__()+ self.w1 = nn.Linear(args.vision_dim, 2 * args.vision_inter_dim, bias=False)+ self.w2 = nn.Linear(args.vision_inter_dim, args.vision_dim, bias=False)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ gate, up = self.w1(x).chunk(2, dim=-1)+ return self.w2(F.silu(gate) * up)+++class Block(nn.Module):+ def __init__(self, args):+ super().__init__()+ self.norm1 = RMSNorm(args.vision_dim)+ self.attn = Attention(args)+ self.norm2 = RMSNorm(args.vision_dim)+ self.mlp = MLP(args)++ def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:+ x = x + self.attn(self.norm1(x), cos, sin)+ return x + self.mlp(self.norm2(x))+++class ViT(nn.Module):+ """DeepSeek ViT: full bidirectional attention over one image with 2D RoPE."""++ def __init__(self, args):+ super().__init__()+ self.rope_dim = args.vision_dim // args.vision_n_heads // 2+ self.rope_theta = args.vision_rope_theta+ self.patch_embed = PatchEmbed(args)+ self.blocks = nn.ModuleList([Block(args) for _ in range(args.vision_n_layers)])+ self.norm = RMSNorm(args.vision_dim)++ def forward(self, patches: torch.Tensor, n_h: int, n_w: int) -> torch.Tensor:+ x = self.patch_embed(patches)+ cos, sin = get_vision_cos_sin(n_h, n_w, self.rope_dim, self.rope_theta)+ for block in self.blocks:+ x = block(x, cos, sin)+ return self.norm(x)+++class Aligner(nn.Module):+ def __init__(self, args):+ super().__init__()+ self.downsample_ratio = args.vision_downsample_ratio+ in_dim = args.vision_dim * self.downsample_ratio**2+ self.w1 = nn.Linear(in_dim, args.dim)+ self.w2 = nn.Linear(args.dim, args.dim)++ def forward(self, x: torch.Tensor, n_h: int, n_w: int) -> torch.Tensor:+ r = self.downsample_ratio+ x = x.view(n_h, n_w, -1).permute(2, 0, 1)+ x = F.pad(x, (0, -n_w % r, 0, -n_h % r))+ x = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).transpose(0, 1)+ return self.w2(F.gelu(self.w1(x)))diff --git a/tests/models/deepseek_v41/test_config.py b/tests/models/deepseek_v41/test_config.pynew file mode 100644index 00000000..fab385fb--- /dev/null+++ b/tests/models/deepseek_v41/test_config.py@@ -0,0 +1,169 @@+"""Config parsing, the DSV41 layer-role table, registry dispatch and loader / model key parity."""++from __future__ import annotations++from dataclasses import replace++import pytest+import torch++from freetoken.layers import set_rope_device+from freetoken.layers.quantization import NoQuantConfig+from freetoken.distributed.info import set_tp_info, try_get_tp_info+from freetoken.models import create_model+from freetoken.models.deepseek_v41.args import DeepseekV41Args, Mode+from freetoken.models.deepseek_v41.config import parse_config+from freetoken.models.register import get_model_spec+from freetoken.utils.torch_utils import torch_dtype++from .common import ENGRAM_LAYERS, INDEX_SOURCES, KV_SOURCES, N_LAYERS, RATIOS, tiny_hf_config, write_tiny_checkpoint+++def test_roles_follow_the_dsv41_mode_table():+ args = parse_config(tiny_hf_config()).dsv41_args+ modes = [r.mode for r in args.roles]+ assert modes == [Mode.WINDOW, Mode.WINDOW, Mode.FULL, Mode.REUSE, Mode.FULL, Mode.REINDEX]+ assert [r.kv_source for r in args.roles] == [None, None, 2, 2, 4, 4]+ assert args.roles[4].is_candidate_source and args.roles[5].uses_candidates and not args.roles[2].uses_candidates+ assert args.decoder_start_layer == 4 and args.backbone_kv_sources == (2, 4)+ assert args.freqs_params(0)[1] == 0 and args.freqs_params(2)[1] == 1024 # yarn only on compressing layers+ assert args.swa_decoder_replay == "bounded"+++def test_shipping_layout_roles():+ """The real 40-layer layout: 3 encoder Full layers (ratio 2), the decoder Full at 20 building the+ candidate pool, Reindex at 24/28/32/36 inside it, Reuse elsewhere."""+ text = tiny_hf_config().text_config+ text.update(+ num_hidden_layers=40, compress_ratios=[0, 0] + [2] * 18 + [1] * 20 + [0, 0, 0],+ kv_source_layer_ids=[2, 8, 14, 20], index_source_layer_ids=[2, 8, 14, 20, 24, 28, 32, 36],+ candidate_source_layer_id=20, num_nextn_predict_layers=3, engram_layer_ids=[1, 14], engram_num_embeddings=[10, 10],+ )+ args = DeepseekV41Args.from_hf(tiny_hf_config(**text))+ full = [r.layer_id for r in args.roles if r.mode is Mode.FULL]+ reindex = [r.layer_id for r in args.roles if r.mode is Mode.REINDEX]+ assert full == [2, 8, 14, 20] and reindex == [24, 28, 32, 36]+ assert sum(r.mode is Mode.REUSE for r in args.roles) == 40 - 2 - 4 - 4+ assert [r.layer_id for r in args.roles if r.uses_candidates] == reindex+ assert args.decoder_start_layer == 20+++def test_bad_layouts_are_rejected():+ with pytest.raises(ValueError, match="no kv source"):+ DeepseekV41Args.from_hf(tiny_hf_config(kv_source_layer_ids=[4]))+ with pytest.raises(ValueError, match="reads kv source"):+ DeepseekV41Args.from_hf(tiny_hf_config(compress_ratios=[0, 0, 2, 1, 1, 1, 0]))+ with pytest.raises(ValueError, match="different kv source"):+ DeepseekV41Args.from_hf(tiny_hf_config(compress_ratios=[0, 0, 2, 2, 2, 2, 0], kv_source_layer_ids=[2, 4], candidate_source_layer_id=2, index_source_layer_ids=[2, 4, 5]))+++def test_registry_and_kv_sources():+ spec = get_model_spec("DeepseekV41ForCausalLM")+ assert spec.module == "freetoken.models.deepseek_v41"+ from freetoken.attention import AttnType+ from freetoken.models.config import DSV4AttentionGroupConfig++ mc = parse_config(tiny_hf_config())+ args = mc.dsv41_args+ assert args.backbone_kv_sources == (2, 4) and tuple(r.ratio for r in args.roles) == tuple(RATIOS[:N_LAYERS])+ (group,) = mc.attention_groups+ assert type(group) is DSV4AttentionGroupConfig and group.variant == "v41"+ assert mc.kv_cache_group_specs()[0].attn_type is AttnType.DSV41 and mc.attn_type_for_layer(0) is AttnType.DSV41+ assert mc.num_layers == N_LAYERS and mc.dsv41_args is not None+++def _meta_model(mc):+ if try_get_tp_info() is None:+ set_tp_info(0, 1)+ set_rope_device(torch.device("cpu"))+ mc = replace(mc, moe_strategy="offload", decode_target="gpu", quant=NoQuantConfig())+ with torch.device("meta"), torch_dtype(torch.bfloat16):+ return create_model(mc)+++def test_loader_names_every_model_parameter(tmp_path):+ from freetoken.models.deepseek_v41.weight import iter_weights++ write_tiny_checkpoint(str(tmp_path))+ model = _meta_model(parse_config(tiny_hf_config()))+ params = model.state_dict()+ loaded = dict(iter_weights(str(tmp_path), torch.device("cpu"), include_moe_experts=False))+ assert set(loaded) == set(params), (set(loaded) ^ set(params))+ for name, tensor in loaded.items():+ assert tuple(tensor.shape) == tuple(params[name].shape), name+ # routed experts, the DSpark stage, the vision tower and the image routing bias are not resident weights+ assert not any(".experts." in k or k.startswith(("mtp.", "vision.")) or k.endswith("bias_vl") for k in loaded)+ # per-role structure+ assert model.model.layers.op_list[3].attn.compressor is None and model.model.layers.op_list[3].attn.indexer is None+ assert model.model.layers.op_list[4].attn.compressor.ratio == 1 and model.model.layers.op_list[2].attn.compressor.ratio == 2+ assert [b.engram is not None for b in model.model.layers.op_list] == [L in ENGRAM_LAYERS for L in range(N_LAYERS)]+ assert all(L in INDEX_SOURCES for L in range(N_LAYERS) if model.model.layers.op_list[L].attn.indexer is not None)+ assert all(L in KV_SOURCES for L in range(N_LAYERS) if model.model.layers.op_list[L].attn.compressor is not None)++[email protected]("missing_weight", [False, True])+def test_loader_closes_shards_when_expert_loading_stops(tmp_path, monkeypatch, missing_weight):+ from freetoken.models.deepseek_v41 import weight++ write_tiny_checkpoint(str(tmp_path))+ closed = []+ close, get = weight._ShardReader.close, weight._ShardReader.get++ def close_reader(reader):+ close(reader)+ closed.append(True)++ def read_weight(reader, name):+ if missing_weight and name.endswith(".w3.weight"):+ raise KeyError(name)+ return get(reader, name)++ monkeypatch.setattr(weight._ShardReader, "close", close_reader)+ monkeypatch.setattr(weight._ShardReader, "get", read_weight)+ tensors = weight.iter_weights(str(tmp_path), torch.device("cpu"))+ if missing_weight:+ with pytest.raises(KeyError):+ next(tensors)+ else:+ next(tensors)+ tensors.close()+ assert closed == [True]+++def test_fp8_wo_a_layer_rejects_a_dequantized_ftw_layout_clearly():+ """An fp8-declared wo_a fed a state dict without ``wo_a_scale`` (an FTW converted before wo_a kept+ its payload) fails with a reconvert message, not a raw missing-key error; a bf16-declared layer fed+ the fp8 payload + scale dequantizes it on the way in."""+ from types import SimpleNamespace++ from freetoken.layers.quantization.scheme import fp8_block_scheme+ from freetoken.models.deepseek_v41.attention import DSV41Attention++ if try_get_tp_info() is None:+ set_tp_info(0, 1)+ args = parse_config(tiny_hf_config()).dsv41_args+ args.max_seq_len = 512+ # wo_a is fp8; the LinearReplicated modules stay unquantized (the bf16 quant method)+ plain = NoQuantConfig()+ quant = SimpleNamespace(+ scheme_for=lambda name: fp8_block_scheme("ue8m0", 32) if name.endswith(".wo_a") else None,+ get_quant_method=lambda layer, prefix: plain.get_quant_method(layer, prefix),+ )+ with torch.device("meta"), torch_dtype(torch.bfloat16):+ fp8_layer = DSV41Attention(args, args.roles[2], quant_config=quant, prefix="attn")+ bf16_layer = DSV41Attention(args, args.roles[2], quant_config=None, prefix="attn")+ assert fp8_layer.wo_a.dtype == torch.float8_e4m3fn and bf16_layer.wo_a.dtype == torch.bfloat16+ rows, cols = fp8_layer.wo_a.shape+ with torch.device("meta"):+ state = {k: torch.empty_like(v) for k, v in fp8_layer.state_dict(prefix="attn").items()}+ del state["attn.wo_a_scale"]+ state["attn.wo_a"] = torch.empty(rows, cols, dtype=torch.bfloat16, device="meta")+ with pytest.raises(RuntimeError, match="reconvert"):+ fp8_layer.load_state_dict(dict(state), prefix="attn")+ # the bf16 layer given the fp8 payload + scale dequantizes (real tensors: the dequant runs)+ w = (torch.randn(rows, cols) * 0.05).to(torch.float8_e4m3fn)+ scale = torch.full((rows // 32, cols // 32), 127, dtype=torch.uint8).view(torch.float8_e8m0fnu)+ state2 = {k: torch.zeros(v.shape, dtype=v.dtype) for k, v in bf16_layer.state_dict(prefix="attn").items()}+ state2["attn.wo_a"], state2["attn.wo_a_scale"] = w, scale+ bf16_layer.load_state_dict(state2, prefix="attn")+ assert torch.equal(bf16_layer.wo_a, w.to(torch.bfloat16))diff --git a/tests/models/deepseek_v41/test_engine_config.py b/tests/models/deepseek_v41/test_engine_config.pynew file mode 100644index 00000000..116209ea--- /dev/null+++ b/tests/models/deepseek_v41/test_engine_config.py@@ -0,0 +1,81 @@+"""Engine config resolution for DeepSeek-V4.1 on the tiny checkpoint: the DSV41 backend and pool are+picked, the page size is the window page, the replay knob lands on the model args, and the fp8+dialect (block 32, fp4 experts) reaches the quant layer with the matching activation block."""++from __future__ import annotations++import json+import os++import pytest+import torch++from freetoken.distributed import DistributedInfo+from freetoken.scheduler.config import SchedulerConfig+from freetoken.engine.engine import _adjust_config+from freetoken.kvcache import resolve_pool_class+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache++from .common import write_tiny_checkpoint+++def _engine_config(path, **over):+ return SchedulerConfig(model_path=path, tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, **over)++[email protected]("replay", ["bounded", "exact"])+def test_resolution_picks_dsv41_and_the_replay_knob(tmp_path, monkeypatch, replay):+ from freetoken.engine import engine++ monkeypatch.setattr(engine, "is_sm100_family", lambda: False)+ monkeypatch.setattr(engine, "is_sm90_family", lambda: True)+ write_tiny_checkpoint(str(tmp_path))+ config = _engine_config(str(tmp_path), attention_backend="auto", moe_strategy="offload", swa_decoder_replay=replay, max_seq_len_override=2048)+ _adjust_config(config)+ assert config.attention_backend == "dsv41_sparse"+ assert config.page_size == 128 and config.cache_type == "swa_radix"+ assert resolve_pool_class(config.model_config) is DSV41PagedKVCache+ args = config.model_config.dsv41_args+ assert args.swa_decoder_replay == replay and args.max_seq_len == 2048 and args.max_batch_size == config.max_running_req + 1+ assert config.max_extend_tokens == 8192 # the prefill chunk stays bounded (whole window pages)+ # the cache contract follows the replay mode: a bounded-mode prefix hit must leave the prompt's last+ # window to the prefill (the pool's prefix_replay_tokens); the history a resume reads stays one window+ from freetoken.kvcache.dsv4.v41_cost_model import dsv41_pool_sizes++ spec = config.model_config.kv_cache_group_specs()[0]+ pool = DSV41PagedKVCache(dsv41_pool_sizes(16, args, 1.0, 128), args, torch.device("cpu"))+ assert pool.prefix_replay_tokens == (128 if replay == "bounded" else 0)+ assert pool.sliding_window_size == 128 and spec.sliding_window == 128 and not spec.is_swa+ # bounded replay keeps the decoder's per-request window KV in private rings, off the shared pages+ assert pool.private_window_layer_ids == (tuple(range(args.decoder_start_layer, args.n_layers)) if replay == "bounded" else ())+ assert DSV41PagedKVCache.min_kv_tokens(config) // 128 == 8 + 3 * config.max_running_req + 2 * (config.max_running_req + 1) + 1+++def test_fp8_block32_dialect_reaches_the_quant_layer(tmp_path):+ """A V4.1-style quantization_config on the tiny checkpoint: dense linears get the 32-block+ e8m0 scheme, routed experts the MXFP4 scheme with a 32-wide activation block."""+ from freetoken.distributed.info import set_tp_info, try_get_tp_info+ from freetoken.layers import OffloadMoELayer+ from freetoken.layers.quantization import Fp8BlockConfig, QuantKind+ from freetoken.layers.quantization.scheme import fp8_block_size++ write_tiny_checkpoint(str(tmp_path))+ cfg_path = os.path.join(str(tmp_path), "config.json")+ with open(cfg_path) as f:+ cfg = json.load(f)+ cfg["quantization_config"] = {"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4"}+ with open(cfg_path, "w") as f:+ json.dump(cfg, f)+ config = _engine_config(str(tmp_path), moe_strategy="offload")+ quant = config.model_config.quant+ assert type(quant) is Fp8BlockConfig and quant.block == 32+ dense = quant.scheme_for("model.layers.3.attn.wq_a")+ assert dense.kind is QuantKind.FP8_BLOCK and fp8_block_size(dense) == 32+ if try_get_tp_info() is None:+ set_tp_info(0, 1)+ experts = OffloadMoELayer(3, num_experts=4, top_k=2, hidden_size=64, intermediate_size=64, limit=10.0, quant_config=quant, prefix="model.layers.3.ffn.experts")+ assert experts.quant_method.scheme.kind is QuantKind.MXFP4 and experts.quant_method.cfg.act_block == 32+ # the family's bf16 modules stay unquantized although the fp8 config lists no exceptions+ for name in ("model.layers.2.attn.compressor.wkv", "model.layers.2.attn.indexer.wk", "model.layers.5.attn.indexer.weights_proj", "head"):+ assert quant.scheme_for(name) is None, name+ assert quant.scheme_for("model.layers.1.engram.wkv").kind is QuantKind.FP8_BLOCKdiff --git a/tests/models/deepseek_v41/test_engram.py b/tests/models/deepseek_v41/test_engram.pynew file mode 100644index 00000000..dec852ec--- /dev/null+++ b/tests/models/deepseek_v41/test_engram.py@@ -0,0 +1,281 @@+"""Engram: the hash against a transcription of the reference ``NgramHashState``, the prime buckets,+the disk table against a torch oracle over the synthetic checkpoint (eager, decode with context,+CUDA-graph flag sync), and the layer's gate math against the reference ``Engram.forward``."""++from __future__ import annotations++import numpy as np+import pytest+import torch++from freetoken.models.deepseek_v41.args import DeepseekV41Args+from freetoken.models.deepseek_v41.engram import EngramHash, bucket_primes, hash_multipliers++from .common import VOCAB, requires_cuda, tiny_hf_config, write_tiny_checkpoint+++def _args(**over) -> DeepseekV41Args:+ return DeepseekV41Args.from_hf(tiny_hf_config(**over))+++def _reference_hashes(args: DeepseekV41Args, ids: torch.Tensor, start_pos: int, cache: torch.Tensor) -> torch.Tensor:+ """``NgramHashState.forward`` for one sequence with an identity token map; ``cache`` is the+ sequence's compressed-id history buffer (mutated like the reference's)."""+ layout_primes = bucket_primes(args)+ flat = [[p for per_ngram in layer for p in per_ngram] for layer in layout_primes]+ offsets = torch.tensor(np.array([np.cumsum([0, *sizes[:-1]]) for sizes in flat]))+ primes = torch.tensor(layout_primes)+ multipliers = hash_multipliers(args.engram_layer_ids, args.engram_max_ngram_size, args.engram_compressed_vocab_size)+ pad_id = args.engram_pad_id+ seqlen = ids.numel()+ cache[start_pos : start_pos + seqlen] = ids+ positions = torch.arange(start_pos, start_pos + seqlen)+ tokens, blocked = [], torch.zeros_like(positions, dtype=torch.bool)+ for shift in range(args.engram_max_ngram_size):+ source = cache[(positions - shift).clamp_min(0)]+ blocked = blocked | (positions < shift)+ tokens.append(torch.where(blocked, pad_id, source))+ tokens = torch.stack(tokens, dim=-1)+ products = tokens.unsqueeze(1) * multipliers+ rolling, hashes = products[..., 0], []+ for i in range(1, args.engram_max_ngram_size):+ rolling = torch.bitwise_xor(rolling, products[..., i])+ hashes.append(rolling.unsqueeze(-1) % primes[:, i - 1])+ return torch.cat(hashes, dim=-1) + offsets+++def test_primes_are_distinct_and_ordered():+ args = _args()+ primes = bucket_primes(args)+ flat = [p for layer in primes for per in layer for p in per]+ assert len(flat) == len(set(flat)) and flat == sorted(flat) and min(flat) > args.engram_vocab_size - 1+ from sympy import isprime++ assert all(isprime(p) for p in flat)+++def test_hash_matches_the_reference_across_a_prefill_decode_split():+ args = _args()+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)+ ids = torch.randint(3, VOCAB, (37,), generator=torch.Generator().manual_seed(0))+ cache = torch.empty(64, dtype=torch.int64)+ want = _reference_hashes(args, ids[:30], 0, cache)+ got = hash.row_ids(ids[:30])+ assert torch.equal(got, want)+ # decode continues with the three preceding tokens as context+ for step in range(30, 37):+ want = _reference_hashes(args, ids[step : step + 1], step, cache)+ got = hash.row_ids(ids[step : step + 1], context=ids[:step])+ assert torch.equal(got, want), step+ # a run resumed mid-sequence (chunked prefill) hashes like the uninterrupted one+ whole = hash.row_ids(ids)+ assert torch.equal(torch.cat([hash.row_ids(ids[:20]), hash.row_ids(ids[20:], context=ids[:20])]), whole)+ assert whole.shape == (37, 1, hash.n_cols) and int(whole.max()) < hash.rows_per_layer[0]+ with pytest.raises(ValueError, match="compressed vocab"):+ EngramHash(args, list(range(VOCAB)), VOCAB + 1)+++@requires_cuda+def test_disk_table_matches_the_dequantized_shard(tmp_path):+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, engram_row_source++ tensors = write_tiny_checkpoint(str(tmp_path))+ args = _args()+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)+ src = engram_row_source(str(tmp_path), 1)+ assert src.num_rows == tensors["layers.1.engram.embed.weight"].shape[0] and src.head_dim == 32+ table = EngramDiskTable(src, hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=16)+ fp8 = tensors["layers.1.engram.embed.weight"]+ scale = torch.exp2(tensors["layers.1.engram.embed.scale"].view(torch.uint8).float() - 127.0)++ def oracle(rows: torch.Tensor) -> torch.Tensor:+ vals = fp8[rows.flatten()].float().view(-1, 32 // 32, 32) * scale[rows.flatten()].unsqueeze(-1)+ return vals.view(rows.shape[0], -1).to(torch.bfloat16).cuda()++ ids = torch.randint(3, VOCAB, (40,), generator=torch.Generator().manual_seed(1))+ rows = hash.row_ids(ids)[:, 0]+ table.stage(rows, graph=False) # larger than the initial eager staging: grows+ table.flush(signal=False)+ assert torch.equal(table.lookup(40), oracle(rows))+ # the graph staging + flag protocol: capture a lookup, fill after the launch, replay+ from freetoken.kernel.row_store import probe_wait_sync++ table.wait_sync = probe_wait_sync("auto", torch.device("cuda"))+ dec_rows = hash.row_ids(ids[40 - 1 :], context=ids[: 40 - 1])[:, 0]+ out = torch.empty(1, hash.n_cols * 32, dtype=torch.bfloat16, device="cuda")+ graph = torch.cuda.CUDAGraph()+ stream = torch.cuda.Stream()+ with torch.cuda.stream(stream):+ if not table.wait_sync:+ table.stage(dec_rows, graph=True)+ table.flush(signal=False)+ with torch.cuda.graph(graph, stream=stream):+ out.copy_(table.lookup(1))+ torch.cuda.synchronize()+ graph.replay() # under wait-sync the replay blocks until the host signals+ table.stage(dec_rows, graph=True)+ table.flush(signal=table.wait_sync)+ torch.cuda.synchronize()+ assert torch.equal(out, oracle(dec_rows))+++@requires_cuda+def test_eager_staging_does_not_overwrite_an_in_flight_lookup(tmp_path):+ """The overlap scheduler stages batch k+1 while batch k's lookup copy is still queued behind+ the layers before the Engram layer (a sleep kernel here): batch k must read its own rows."""+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, engram_row_source++ tensors = write_tiny_checkpoint(str(tmp_path))+ hash = EngramHash(_args(), list(range(VOCAB)), VOCAB)+ table = EngramDiskTable(engram_row_source(str(tmp_path), 1), hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=64)+ fp8 = tensors["layers.1.engram.embed.weight"]+ scale = torch.exp2(tensors["layers.1.engram.embed.scale"].view(torch.uint8).float() - 127.0)++ def oracle(rows):+ vals = fp8[rows.flatten()].float().view(-1, 1, 32) * scale[rows.flatten()].unsqueeze(-1)+ return vals.view(rows.shape[0], -1).to(torch.bfloat16).cuda()++ gen = torch.Generator().manual_seed(0)+ batches = [hash.row_ids(torch.randint(3, VOCAB, (40,), generator=gen))[:, 0] for _ in range(4)]+ got = []+ for rows in batches:+ table.stage(rows, graph=False)+ table.flush(signal=False)+ torch.cuda._sleep(50_000_000)+ got.append(table.lookup(rows.shape[0]))+ torch.cuda.synchronize()+ for i, (rows, out) in enumerate(zip(batches, got)):+ assert torch.equal(out, oracle(rows)), f"batch {i} read another batch's rows"+++@requires_cuda[email protected]_mode()+def test_host_fills_a_graph_decode_step_from_a_worker_thread(tmp_path):+ """The deferred graph-decode fill must not wait for the launch call to return: a 40-layer graph+ blocks cuGraphLaunch on the host while the GPU sits on the WAIT, so the host coordinator reads the+ tokens back and signals from a worker thread. The worker hashes against the DISPATCH snapshot of+ the request state: advancing the request right after dispatch (what the engine does) must not+ change which rows are fetched."""+ from types import SimpleNamespace++ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, EngramHost, engram_row_source++ tensors = write_tiny_checkpoint(str(tmp_path))+ args = _args()+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)+ table = EngramDiskTable(engram_row_source(str(tmp_path), 1), hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=16)+ host = EngramHost(hash, [table], torch.device("cuda"))+ if not host.wait_sync:+ pytest.skip("stream memops unavailable: launch-gating mode needs no worker")+ ids = torch.randint(3, VOCAB, (20,), generator=torch.Generator().manual_seed(7))+ req = SimpleNamespace(input_ids=ids.clone(), device_len=20, cached_len=19)+ batch = SimpleNamespace(is_decode=True, is_prefill=False, padded_reqs=[req], padded_size=1, input_ids=ids[19:20].to(torch.int32).cuda())+ out = torch.empty(1, hash.n_cols * 32, dtype=torch.bfloat16, device="cuda")+ stream = torch.cuda.Stream()+ graph = torch.cuda.CUDAGraph()+ with torch.cuda.stream(stream):+ with torch.cuda.graph(graph, stream=stream):+ out.copy_(table.lookup(1))+ torch.cuda.synchronize()+ with torch.cuda.stream(stream):+ # slow the readback so the request advances (complete_one + append) before the worker runs+ torch.cuda._sleep(200_000_000)+ with host.forward_host_ctx(batch, use_graph=True):+ graph.replay()+ req.device_len += 1+ req.input_ids = torch.cat([req.input_ids, torch.tensor([5], dtype=ids.dtype)])+ torch.cuda.synchronize()+ fp8 = tensors["layers.1.engram.embed.weight"]+ rows = hash.row_ids(ids[19:20], context=ids[:19])[:, 0].flatten()+ want = fp8[rows].float().view(1, -1).to(torch.bfloat16).cuda() # scales are all 1.0 in the synthetic table+ assert torch.equal(out, want)+ host.close()+++@requires_cuda+def test_host_surfaces_a_failed_deferred_fill_at_dispatch_exit(tmp_path):+ from types import SimpleNamespace++ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, EngramHost, engram_row_source++ write_tiny_checkpoint(str(tmp_path))+ args = _args()+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)+ table = EngramDiskTable(engram_row_source(str(tmp_path), 1), hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=16)+ host = EngramHost(hash, [table], torch.device("cuda"))+ if not host.wait_sync:+ pytest.skip("stream memops unavailable")+ ids = torch.randint(3, VOCAB, (8,), generator=torch.Generator().manual_seed(8))+ req = SimpleNamespace(input_ids=ids, device_len=8, cached_len=7)+ batch = SimpleNamespace(is_decode=True, is_prefill=False, padded_reqs=[req], padded_size=1, input_ids=ids[7:8].to(torch.int32).cuda())+ stream = torch.cuda.Stream()+ graph = torch.cuda.CUDAGraph()+ out = torch.empty(1, hash.n_cols * 32, dtype=torch.bfloat16, device="cuda")+ with torch.cuda.stream(stream):+ with torch.cuda.graph(graph, stream=stream):+ out.copy_(table.lookup(1))+ torch.cuda.synchronize()+ table.stage = lambda *a, **k: (_ for _ in ()).throw(RuntimeError("disk gone")) # the fill fails+ with pytest.raises(RuntimeError, match="disk gone"):+ with torch.cuda.stream(stream):+ with host.forward_host_ctx(batch, use_graph=True):+ graph.replay()+ torch.cuda.synchronize() # the failing worker released the flag: the stream is not stuck+ host.close()+++def test_ftw_side_files_carry_the_tables_and_reopen(tmp_path):+ """An FTW conversion copies the shards holding the Engram tables next to it (the converter's+ ``ftw_side_files`` hook, which skips the HF index); the row source resolves the copies through+ their headers exactly like the HF shards, byte for byte."""+ from freetoken.kernel.row_store import RowStore+ from freetoken.models.deepseek_v41.engram_table import engram_row_source+ from freetoken.models.deepseek_v41.weight import ftw_side_files++ src, out = tmp_path / "hf", tmp_path / "ftw"+ out.mkdir()+ tensors = write_tiny_checkpoint(str(src))+ (out / "config.json").write_bytes((src / "config.json").read_bytes()) # the converter copies the metadata+ shard = "model-00001-of-00001.safetensors" # the tiny checkpoint's single shard holds the table+ assert ftw_side_files(str(src), str(out)) == [shard]+ assert (out / shard).read_bytes() == (src / shard).read_bytes() and not (out / "model.safetensors.index.json").exists()+ source = engram_row_source(str(out), 1)+ assert source.path == str(out / shard) and source.num_rows == 4096 and source.head_dim == 32+ assert torch.equal(source.scales, tensors["layers.1.engram.embed.scale"].view(torch.uint8))+ store = RowStore(paths=[source.path], extent_file=[0], extent_base=[source.base], rows_per_extent=source.num_rows,+ row_bytes=source.head_dim, row_stride=source.head_dim, use_io_uring=False)+ ids = torch.tensor([0, 7, 4095], dtype=torch.int64)+ buf = torch.zeros(3 * 32, dtype=torch.uint8)+ store.stage_rows(ids.data_ptr(), 3, buf.data_ptr(), 0)+ store.flush(0)+ assert torch.equal(buf.view(3, 32), tensors["layers.1.engram.embed.weight"][ids].view(torch.uint8))+++def test_layer_gate_matches_the_reference():+ """``EngramLayer.forward`` against the reference ``Engram.forward`` (bf16 wkv, fp32 gate math)."""+ from freetoken.distributed.info import set_tp_info, try_get_tp_info+ from freetoken.models.deepseek_v41.engram import EngramLayer++ if try_get_tp_info() is None:+ set_tp_info(0, 1)+ args = _args()+ torch.manual_seed(0)+ layer = EngramLayer(args, 1)+ T, hc, dim = 5, args.hc_mult, args.dim+ wkv = torch.randn(dim * (hc + 1), layer.width, dtype=torch.bfloat16) * 0.05+ layer.wkv.weight = wkv+ layer.q_weight = torch.rand(hc, dim) + 0.5+ layer.k_weight = torch.rand(hc, dim) + 0.5+ x = torch.randn(T, hc, dim, dtype=torch.bfloat16)+ rows = torch.randn(T, layer.width, dtype=torch.bfloat16)++ kv = torch.nn.functional.linear(rows, wkv)+ key, value = kv.split([hc * dim, dim], dim=-1)+ key = key.float().view(T, hc, dim)+ weight = layer.q_weight * layer.k_weight+ h = x.float()+ rstd = torch.rsqrt(h.square().mean(-1) + args.norm_eps) * torch.rsqrt(key.square().mean(-1) + args.norm_eps)+ dot = (h * weight * key).sum(-1) * rstd * dim**-0.5+ gate = torch.sigmoid(torch.copysign(dot.abs().clamp_min(1e-6).sqrt(), dot))+ want = (h + gate.unsqueeze(-1) * value.float().unsqueeze(-2)).to(x.dtype)+ assert torch.equal(layer.forward(x, rows), want)diff --git a/tests/models/deepseek_v41/test_forward_smoke.py b/tests/models/deepseek_v41/test_forward_smoke.pynew file mode 100644index 00000000..e0861e88--- /dev/null+++ b/tests/models/deepseek_v41/test_forward_smoke.py@@ -0,0 +1,184 @@+"""End-to-end forward smoke on the tiny model: cold prefill, batched decode, a second request, chunked+prefill and bounded replay all produce finite logits of the right shape (numerics: test_reference_parity)."""++from __future__ import annotations++import pytest+import torch++from .common import VOCAB, requires_cuda, write_tiny_checkpoint++pytestmark = requires_cuda++[email protected](scope="module")+def checkpoint(tmp_path_factory):+ folder = tmp_path_factory.mktemp("dsv41-tiny")+ write_tiny_checkpoint(str(folder))+ return str(folder)+++def _tokens(n: int, seed: int) -> list[int]:+ g = torch.Generator().manual_seed(seed)+ return torch.randint(3, VOCAB, (n,), generator=g).tolist()+++def test_prefill_and_decode(checkpoint):+ from .harness import TinyEngine++ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")+ r0 = eng.new_request(0, _tokens(300, 1))+ logits = eng.prefill([r0])+ assert logits.shape == (1, VOCAB) and torch.isfinite(logits).all()+ eng.finish_prefill([r0])+ r1 = eng.new_request(1, _tokens(50, 2))+ assert torch.isfinite(eng.prefill([r1])).all()+ eng.finish_prefill([r1])+ for step in range(5): # batched decode over two requests at different positions+ out = eng.decode([r0, r1], [10 + step, 20 + step])+ assert out.shape == (2, VOCAB) and torch.isfinite(out).all()+++def test_chunked_prefill_matches_single_shot(checkpoint):+ """Two 128-aligned chunks must reproduce the single-shot prefill's last-token logits: the+ compressor carry, the window ring and the compressed rows are all resumed through the pool. The+ chunks run their projections at other row counts, so the match is to rounding, not bitwise."""+ from .harness import TinyEngine+ from .test_reference_parity import _compare++ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")+ toks = _tokens(300, 3)+ whole = eng.new_request(0, toks)+ ref = eng.prefill([whole])+ chunked = eng.new_request(1, toks)+ chunked.device_len = 256 # first chunk [0, 256)+ eng.prefill([chunked])+ chunked.cached_len, chunked.device_len = 256, 300+ got = eng.prefill([chunked])+ _compare("chunked prefill", got, ref)+++def test_bounded_replay_is_exact_within_one_window(checkpoint):+ """A prompt no longer than the window replays every token with no floor: bounded == exact."""+ from .harness import TinyEngine+ from .test_reference_parity import _compare++ toks = _tokens(100, 5)+ exact = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")+ want = exact.prefill([exact.new_request(0, toks)])+ bounded = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")+ got = bounded.prefill([bounded.new_request(0, toks)])+ _compare("bounded within one window", got, want)++[email protected]("cut", [128, 256])+def test_bounded_chunked_prefill_matches_unchunked(checkpoint, cut):+ """A chunked bounded prefill gives each chunk its part of the prompt's last window ``[172, 300)``:+ a first chunk that ends before it (cut 128) runs no decoder row and hands the head a placeholder;+ one that reaches into it (cut 256) leaves rows in the private ring for the last chunk, which is+ shorter than the window (44 tokens). Both match the unchunked prefill."""+ from .harness import TinyEngine+ from .test_reference_parity import _compare++ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")+ toks = _tokens(300, 6)+ ref = eng.prefill([eng.new_request(0, toks)])+ chunked = eng.new_request(1, toks)+ chunked.device_len = cut+ first = eng.prefill([chunked])+ assert first.shape == (1, VOCAB) and torch.isfinite(first).all()+ chunked.cached_len, chunked.device_len = cut, 300+ _compare(f"bounded chunks cut at {cut}", eng.prefill([chunked]), ref)+++def test_bounded_batch_mixes_a_decoderless_chunk_with_a_whole_prompt(checkpoint):+ """One batch: a chunk that ends before its prompt's last window (no decoder row, a placeholder for+ the head) next to a whole prompt; the whole prompt's logits are those of its own prefill."""+ from .harness import TinyEngine+ from .test_reference_parity import _compare++ eng = TinyEngine(checkpoint, max_seq_len=2048, max_running_req=2, swa_decoder_replay="bounded")+ whole = _tokens(300, 8)+ want = eng.prefill([eng.new_request(0, whole)])+ chunk = eng.new_request(0, _tokens(600, 9))+ chunk.device_len = 256+ other = eng.new_request(1, whole)+ got = eng.prefill([chunk, other])+ assert got.shape == (2, VOCAB) and torch.isfinite(got).all()+ _compare("whole prompt next to a decoderless chunk", got[1:], want)+++def test_bounded_prefix_hit_matches_cold_prefill(checkpoint):+ """A prefix hit that would leave fewer than a window of new tokens (768 cached + 20 new) is capped+ one window before the prompt end, so the request prefills ``[640, 788)`` -- the decoder's whole+ window among it -- and produces the cold prefill's logits."""+ from .harness import TinyEngine+ from .test_reference_parity import _compare++ eng = TinyEngine(checkpoint, max_seq_len=2048, max_running_req=2, swa_decoder_replay="bounded")+ prefix = _tokens(768, 11)+ donor = eng.new_request(0, prefix + _tokens(20, 12))+ cold = eng.prefill([donor])+ eng.finish_prefill([donor])+ hit = eng.new_request_on_prefix(1, donor, prefix + _tokens(20, 12)) # the same prompt over the shared pages+ assert hit.cached_len == 640 and hit.extend_len == 148+ got = eng.prefill([hit])+ _compare("bounded prefix hit", got, cold)+++def test_shared_prefix_replays_do_not_disturb_each_other(checkpoint):+ """Two requests sharing a 768-token prefix with different suffix lengths: B's hit is capped at 640+ (a window before its prompt end), its decoder writes only B's private ring and it reads the shared+ encoder history, so A's pending decode sees the logits it would have seen without B, and neither+ the shared pages nor A's ring change."""+ from .harness import TinyEngine++ prefix = _tokens(768, 13)+ a_toks, b_toks = prefix + _tokens(20, 14), prefix + _tokens(40, 15)++ def run(interleave: bool):+ eng = TinyEngine(checkpoint, max_seq_len=2048, max_running_req=2, swa_decoder_replay="bounded")+ a = eng.new_request(0, a_toks)+ eng.prefill([a])+ eng.finish_prefill([a])+ dec_layer, enc_layer = eng.args.decoder_start_layer, 2+ shared_slots = eng.pool.translate_full_to_window(eng.page_table[0, 640:768].long())+ enc_before = eng.pool.window_pool[enc_layer][shared_slots].clone()+ P = eng.args.window_size+ ring_before = eng.pool.window_pool[dec_layer][0 * P : 1 * P].clone() # A's ring row+ if interleave:+ b = eng.new_request_on_prefix(1, a, b_toks)+ eng.prefill([b])+ eng.finish_prefill([b])+ assert torch.equal(eng.pool.window_pool[enc_layer][shared_slots], enc_before)+ assert torch.equal(eng.pool.window_pool[dec_layer][0 * P : 1 * P], ring_before)+ return eng.decode([a], [7])++ assert torch.equal(run(True), run(False))+++def test_commit_dedup_onto_a_longer_prompts_pages_keeps_the_decoder_state(checkpoint):+ """The radix commit may replace a request's freshly written pages with an existing node's (the+ reviewer's donor case): D prefilled 256 tokens, so under bounded replay its decoder never computed+ positions [0, 128); A (exactly D's first 128 tokens) is repointed onto D's pages at commit. A's+ decoder KV lives in A's own ring, so its next decode is unchanged by the repoint (D's shared rows+ came from a longer prefill, so to rounding)."""+ from .harness import TinyEngine+ from .test_reference_parity import _compare++ donor_toks = _tokens(256, 21)+ a_toks = donor_toks[:128]++ def run(repoint: bool):+ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")+ d = eng.new_request(0, donor_toks)+ eng.prefill([d])+ eng.finish_prefill([d])+ a = eng.new_request(1, a_toks)+ eng.prefill([a])+ eng.finish_prefill([a])+ if repoint: # what CacheManager.cache_req does when the tree already holds the prefix+ eng.page_table[1, :128] = eng.page_table[0, :128]+ return eng.decode([a], [3])++ _compare("decode after a commit repoint", run(True), run(False))diff --git a/tests/models/deepseek_v41/test_reference_parity.py b/tests/models/deepseek_v41/test_reference_parity.pynew file mode 100644index 00000000..839cdafb--- /dev/null+++ b/tests/models/deepseek_v41/test_reference_parity.py@@ -0,0 +1,219 @@+"""Logits parity against the vendored reference implementation on the tiny synthetic checkpoint.++The reference runs bf16 or FP8/FP4 weights with its own torch quantizers, sparse attention, engram+hash and single-pass mHC; FreeToken runs the same checkpoint through its kernels and packed pools.+Compared: exact-mode prefill, greedy decode steps, a two-request batch, and the bounded-replay path+against the reference's ``forward_bounded`` oracle.+"""++from __future__ import annotations++from dataclasses import asdict, fields++import pytest+import torch++from .common import VOCAB, requires_cuda, tiny_text_config, write_tiny_checkpoint++pytestmark = requires_cuda++# The sparse-attention implementations accumulate in different orders.+ATOL = RTOL = 1e-2+# The fp4 experts also round the routed sum to bf16 before the shared-expert add and weight each+# expert's down output; the reference keeps that sum fp32 and weights the intermediate before its fp8+# round-trip (10 seeds x 16 checks: max |err| 0.019, against 0.007 in the reference's order).+QUANT_TOL = 2e-2++[email protected](scope="module")+def checkpoint(tmp_path_factory):+ folder = tmp_path_factory.mktemp("dsv41-parity")+ tensors = write_tiny_checkpoint(str(folder), seed=11)+ return str(folder), tensors+++def _tokens(n: int, seed: int) -> torch.Tensor:+ return torch.randint(3, VOCAB, (n,), generator=torch.Generator().manual_seed(seed))+++class Reference:+ """The vendored ``Transformer`` on CUDA, loaded from the synthetic checkpoint (identity token map)."""++ def __init__(self, tensors: dict, text: dict, *, max_seq_len: int, max_batch_size: int, quantized: bool = False, vision_config: dict | None = None):+ from freetoken.models.deepseek_v41.args import DeepseekV41Args+ from types import SimpleNamespace++ from . import reference+ from .reference import engram as ref_engram+ from .reference import model as ref_model++ args = DeepseekV41Args.from_hf(SimpleNamespace(text_config=text))+ names = {f.name for f in fields(ref_model.ModelArgs)}+ kwargs = {k: v for k, v in asdict(args).items() if k in names}+ kwargs.update(dtype="fp8" if quantized else "bf16", expert_dtype="fp4" if quantized else None,+ max_seq_len=max_seq_len, max_batch_size=max_batch_size, vision_n_layers=0, temperature=0.0)+ if vision_config:+ kwargs.update(vision_n_layers=vision_config["num_hidden_layers"], vision_dim=vision_config["hidden_size"],+ vision_n_heads=vision_config["num_attention_heads"], vision_inter_dim=vision_config["intermediate_size"],+ vision_patch_size=vision_config["patch_size"], vision_downsample_ratio=vision_config["downsample_ratio"])+ self.margs = ref_model.ModelArgs(**kwargs)+ # the synthetic checkpoint's tokenizer is the identity over VOCAB ids+ ref_engram.build_compressed_token_map = lambda tokenizer: (list(range(VOCAB)), VOCAB)+ torch.set_default_dtype(torch.bfloat16)+ try:+ with torch.device("cuda"):+ self.model = ref_model.Transformer(self.margs, tokenizer=object())+ finally:+ torch.set_default_dtype(torch.float32)+ state = {k: v for k, v in tensors.items() if not k.startswith("mtp.")+ and (vision_config or (not k.startswith("vision.") and not k.endswith("bias_vl")))}+ if quantized:+ state = {k: v.view(torch.float4_e2m1fn_x2) if ".ffn.experts." in k and k.endswith(".weight") else v+ for k, v in state.items()}+ # The reference's grouped output projection holds dequantized bf16 weights.+ for name in list(state):+ if name.endswith(".wo_a.scale"):+ scale = state.pop(name).float().repeat_interleave(32, 0).repeat_interleave(32, 1)+ weight_name = name.removesuffix(".scale") + ".weight"+ state[weight_name] = (state[weight_name].float() * scale).bfloat16()+ missing, unexpected = self.model.load_state_dict({k: v.cuda() for k, v in state.items()}, strict=False)+ assert not unexpected, unexpected+ assert not [m for m in missing if "freqs_cis" not in m], missing+ self.reference = reference++ # the reference builds its index tensors on the default device (generate.py sets it to cuda)+ def prefill(self, ids: torch.Tensor) -> torch.Tensor:+ with torch.device("cuda"):+ return self.model(ids.view(1, -1).cuda(), 0)[1].float()++ def prefill_batch(self, ids: torch.Tensor) -> torch.Tensor:+ with torch.device("cuda"):+ return self.model(ids.cuda(), 0)[1].float()++ def prefill_bounded(self, ids: torch.Tensor) -> torch.Tensor:+ with torch.device("cuda"):+ return self.model.forward_bounded(ids.reshape(-1, ids.shape[-1]).cuda(), self.margs.window_size).float()++ def decode(self, tokens: torch.Tensor, pos: int) -> torch.Tensor:+ with torch.device("cuda"):+ return self.model(tokens.view(-1, 1).cuda(), pos)[1].float()+++def _engram_table_for(engine, tensors, text):+ """Our engine reads the Engram rows straight from the synthetic shard (the disk table), hashed+ with the same identity token map the reference oracle uses."""+ from freetoken.models.deepseek_v41.engram import EngramHash+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, EngramHost, engram_row_source++ hash = EngramHash(engine.args, list(range(VOCAB)), VOCAB)+ tables = [+ EngramDiskTable(engram_row_source(engine.checkpoint, layer.layer_id), hash.n_cols, engine.device, max_graph_rows=8, max_extend_tokens=64)+ for layer in engine.model.engram_layers()+ ]+ for layer, table in zip(engine.model.engram_layers(), tables):+ layer.attach_table(table)+ host = EngramHost(hash, tables, engine.device)+ engine.model.forward_host_ctx = host.forward_host_ctx+ engine.engram_host = host+++def _compare(name, got, want, *, tol=ATOL):+ err = (got - want).abs().max().item()+ assert torch.equal(got.argmax(-1), want.argmax(-1)), f"{name}: argmax differs (max abs err {err:.4f})"+ torch.testing.assert_close(got, want, atol=tol, rtol=tol, msg=lambda m: f"{name}: {m}")++[email protected]("mode", ["exact", "bounded"])[email protected]("batch_size", [1, 2])+def test_quantized_offload_prefill_and_decode_match_reference(tmp_path, mode, batch_size):+ from .harness import TinyEngine++ tensors = write_tiny_checkpoint(str(tmp_path), seed=11, quantized=True)+ text = tiny_text_config(moe_intermediate_size=256)+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3, quantized=True)+ eng = TinyEngine(str(tmp_path), swa_decoder_replay=mode, quantized=True)+ _engram_table_for(eng, tensors, text)+ ids = torch.stack([_tokens(300, 1 + i) for i in range(batch_size)])+ want = ref.prefill_batch(ids) if mode == "exact" else ref.prefill_bounded(ids)+ reqs = [eng.new_request(i, row.tolist()) for i, row in enumerate(ids)]+ got = eng.prefill(reqs)+ _compare(f"quantized {mode} prefill", got, want, tol=QUANT_TOL)+ eng.finish_prefill(reqs)+ for step in range(3):+ token = want.argmax(-1)+ want = ref.decode(token, ids.shape[-1] + step)+ got = eng.decode(reqs, token.tolist())+ _compare(f"quantized {mode} decode {step}", got, want, tol=QUANT_TOL)+++def test_exact_prefill_and_greedy_decode_match_the_reference(checkpoint):+ from .harness import TinyEngine++ folder, tensors = checkpoint+ text = tiny_text_config()+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3)+ eng = TinyEngine(folder, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")+ _engram_table_for(eng, tensors, text)++ ids = _tokens(300, 1)+ want = ref.prefill(ids)+ req = eng.new_request(0, ids.tolist())+ got = eng.prefill([req])+ _compare("prefill", got, want)+ eng.finish_prefill([req])+ # greedy decode from the reference's own picks keeps both sides on one trajectory+ pos = 300+ for step in range(12):+ nxt = int(want.argmax(-1).item())+ want = ref.decode(torch.tensor([nxt]), pos)+ got = eng.decode([req], [nxt])+ _compare(f"decode step {step}", got, want)+ pos += 1+++def test_batched_prefill_and_decode_match_the_reference(checkpoint):+ from .harness import TinyEngine++ folder, tensors = checkpoint+ text = tiny_text_config()+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3)+ eng = TinyEngine(folder, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")+ _engram_table_for(eng, tensors, text)++ ids = torch.stack([_tokens(200, 2), _tokens(200, 3)])+ want = ref.prefill_batch(ids)+ reqs = [eng.new_request(i, ids[i].tolist()) for i in range(2)]+ got = eng.prefill(reqs)+ _compare("batched prefill", got, want)+ eng.finish_prefill(reqs)+ pos = 200+ for step in range(6):+ nxt = want.argmax(-1).cpu()+ want = ref.decode(nxt, pos)+ got = eng.decode(reqs, nxt.tolist())+ _compare(f"batched decode step {step}", got, want)+ pos += 1+++def test_bounded_replay_matches_the_reference_oracle(checkpoint):+ from .harness import TinyEngine++ folder, tensors = checkpoint+ text = tiny_text_config()+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3)+ eng = TinyEngine(folder, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")+ _engram_table_for(eng, tensors, text)++ ids = _tokens(300, 4)+ want = ref.prefill_bounded(ids)+ req = eng.new_request(0, ids.tolist())+ got = eng.prefill([req])+ _compare("bounded prefill", got, want)+ eng.finish_prefill([req])+ pos = 300+ for step in range(6):+ nxt = int(want.argmax(-1).item())+ want = ref.decode(torch.tensor([nxt]), pos)+ got = eng.decode([req], [nxt])+ _compare(f"decode after bounded prefill, step {step}", got, want)+ pos += 1diff --git a/tests/models/deepseek_v41/test_vision.py b/tests/models/deepseek_v41/test_vision.pynew file mode 100644index 00000000..0e1d4a48--- /dev/null+++ b/tests/models/deepseek_v41/test_vision.py@@ -0,0 +1,214 @@+"""Image layout, official ViT/logits parity and image replay through the shared encoder cache."""++from __future__ import annotations++import io+import json+from dataclasses import asdict+from types import SimpleNamespace++import numpy as np+import pytest+import torch+from PIL import Image+from safetensors.torch import save_file++from freetoken.core import Batch+from freetoken.engine.engine import Engine+from freetoken.mm.config import MultimodalConfig+from freetoken.mm.encoder_cache import EncoderCache+from freetoken.mm.processors.deepseek_v41 import DeepseekV41MMProcessor+from freetoken.models.deepseek_v41.config import VisionConfig+from freetoken.scheduler.mm import mm_rows_after, plan_mm_batch+from freetoken.utils.torch_utils import torch_dtype++from .common import DIM, VOCAB, requires_cuda, tiny_text_config, write_tiny_checkpoint+from .reference.vision import Aligner, ViT++VC = asdict(VisionConfig(num_hidden_layers=2, hidden_size=32, num_attention_heads=4, intermediate_size=48,+ patch_size=2, min_pixels=0, max_image_tokens=64))+++def _args():+ return SimpleNamespace(vision_n_layers=2, vision_dim=32, vision_n_heads=4, vision_inter_dim=48,+ vision_patch_size=2, vision_downsample_ratio=3, vision_rope_theta=10000.0, dim=DIM)+++def _checkpoint(path, quantized=False):+ tensors = write_tiny_checkpoint(str(path), seed=11, quantized=quantized)+ torch.manual_seed(12)+ with torch_dtype(torch.bfloat16):+ vit, aligner = ViT(_args()), Aligner(_args())+ # Replace the unused dummy vision weight from the text-only fixture.+ tensors = {k: v for k, v in tensors.items() if not k.startswith("vision.")}+ for prefix, module in (("vision", vit), ("aligner", aligner)):+ tensors.update({f"{prefix}.{k}": v.detach().bfloat16().contiguous() for k, v in module.state_dict().items()})+ tensors.update({k: torch.randn(DIM, dtype=torch.bfloat16) for k in ("image_start", "image_end", "image_newline")})+ save_file(tensors, path / 'model-00001-of-00001.safetensors')+ (path / 'model.safetensors.index.json').write_text(json.dumps({'weight_map': {k: 'model-00001-of-00001.safetensors' for k in tensors}}))+ config = json.loads((path / 'config.json').read_text())+ config.update(vision_config=VC, image_token_id=VOCAB - 1)+ (path / 'config.json').write_text(json.dumps(config))+ return tensors+++def _processor():+ cfg = SimpleNamespace(vision_config=SimpleNamespace(**VC), image_token_id=VOCAB - 1)+ return DeepseekV41MMProcessor(cfg, '.', MultimodalConfig())+++def _image(color):+ buf = io.BytesIO()+ Image.new('RGB', (14, 10), color).save(buf, format='PNG')+ return buf.getvalue()+++def _inputs(length=151):+ ids = [3] * 17 + [VOCAB - 1] + [4] * (length - 19) + [VOCAB - 1]+ return _processor().apply(torch.tensor(ids, dtype=torch.int32), [_image('red'), _image('blue')])+++def _prefill(eng, reqs):+ if not hasattr(eng, 'encoder_cache'):+ eng.encoder_cache = EncoderCache()+ eng.dtype = torch.bfloat16+ eng._mm_registered = set()+ for req in reqs:+ if id(req) not in eng._mm_registered:+ for item in req.mm_items or (): # what the scheduler claims at admission+ eng.encoder_cache.register(item.hash, req.uid, mm_rows_after(item, req.cached_len))+ eng._mm_registered.add(id(req))+ batch = Batch(reqs=reqs, phase='prefill')+ batch.padded_reqs = reqs+ batch.input_ids = torch.cat([r.input_ids[r.cached_len:r.device_len] for r in reqs]).cuda()+ batch.positions = torch.cat([torch.arange(r.cached_len, r.device_len) for r in reqs]).cuda()+ jobs, plan, rows, _ = plan_mm_batch(reqs, eng.encoder_cache)+ batch.mm_encoder_jobs, batch.mm_gather_plan = jobs, plan+ if plan:+ batch.mm_rows = torch.tensor(rows, dtype=torch.long, device=eng.device)+ Engine._run_mm_encoder(eng, batch)+ eng._bind()+ eng.backend.prepare_metadata(batch)+ with eng.ctx.forward_batch(batch), eng.model.forward_host_ctx(batch, False):+ return eng.model.forward().float()+++def test_processor_patch_order_and_all_span_rows_are_embeddings():+ pixels = np.arange(10 * 14 * 3, dtype=np.uint8).reshape(10, 14, 3)+ item = _processor().process([Image.fromarray(pixels)])[0]+ assert item.grid_thw == [1, 5, 7]+ expected = ((torch.tensor(pixels[:2, :2]).permute(2, 0, 1).float() / 255 - .5) / .5).bfloat16()+ assert torch.equal(item.feature[0], expected)+ assert len(_processor().prompt_replacement(item).full) == 10+ result = _inputs()+ assert [i.offsets for i in result.mm_items] == [[[17, 27]], [[159, 169]]]+ assert result.mrope_positions is None+ assert result.mm_items[0].hash != result.mm_items[1].hash+++def test_text_only_skips_every_visual_parameter(tmp_path):+ from freetoken.models.deepseek_v41.weight import iter_weights+ from freetoken.mm.processor import get_mm_processor+ from .test_engine_config import _engine_config+ _checkpoint(tmp_path)+ enabled = _engine_config(str(tmp_path), moe_strategy='offload')+ assert enabled.model_config.is_multimodal+ disabled = MultimodalConfig(disabled_encoders=frozenset({'vision', 'audio'}))+ text = _engine_config(str(tmp_path), moe_strategy='offload', mm=disabled)+ assert not text.model_config.is_multimodal and not text.active_encoders+ assert get_mm_processor(str(tmp_path), disabled) is None+ loaded = dict(iter_weights(str(tmp_path), 'cpu', include_moe_experts=False, include_vision=False))+ assert not any(k.startswith('visual.') for k in loaded)+ assert get_mm_processor(str(tmp_path)).placeholder == [VOCAB - 1]+++@requires_cuda[email protected]('placement', ['gpu', 'host'])+def test_vision_encoder_matches_official_with_padded_grid(tmp_path, placement):+ from .harness import TinyEngine+ tensors = _checkpoint(tmp_path)+ eng = TinyEngine(str(tmp_path))+ eng.model.place_encoder_weights(placement)+ item = _processor().process([Image.new('RGB', (14, 10), 'red')])[0]+ with torch_dtype(torch.bfloat16), torch.device('cuda'):+ vit, aligner = ViT(_args()), Aligner(_args())+ for prefix, module in (('vision', vit), ('aligner', aligner)):+ module.load_state_dict({k.removeprefix(prefix + '.'): v.cuda() for k, v in tensors.items() if k.startswith(prefix + '.')})+ with torch.inference_mode(), torch.device('cuda'):+ x = aligner(vit(item.feature.cuda(), 5, 7), 5, 7)+ got = eng.model.encode(item)+ torch.testing.assert_close(got[1:-1].view(2, 4, DIM)[:, :3].reshape(-1, DIM), x, atol=1e-3, rtol=1e-3)+ assert torch.equal(got[0], tensors['image_start'].cuda())+ assert torch.equal(got[-1], tensors['image_end'].cuda())+ assert torch.equal(got[[4, 8]], tensors['image_newline'].cuda().expand(2, -1))+ eng.model.place_encoder_weights('gpu')+++@requires_cuda[email protected]('quantized', [False, True])+def test_image_logits_and_decode_match_reference(tmp_path, quantized):+ from .harness import TinyEngine+ from .test_reference_parity import ATOL, QUANT_TOL, Reference, _compare, _engram_table_for+ tol = QUANT_TOL if quantized else ATOL+ tensors = _checkpoint(tmp_path, quantized)+ text = tiny_text_config(moe_intermediate_size=256) if quantized else tiny_text_config()+ ref = Reference(tensors, text, max_seq_len=512, max_batch_size=2, quantized=quantized, vision_config=VC)+ eng = TinyEngine(str(tmp_path), max_seq_len=512, swa_decoder_replay='exact', quantized=quantized)+ _engram_table_for(eng, tensors, text)+ result = _inputs()+ ids = result.input_ids.clone()+ types = torch.full_like(ids, -1)+ images = []+ for item in result.mm_items:+ lo, hi = item.offsets[0]+ ids[lo:hi] = VOCAB - 1+ typ = torch.tensor([0] + ([1] * 3 + [2]) * 2 + [3])+ types[lo:hi] = typ+ images.append(SimpleNamespace(start=lo, patches=item.feature, n_vit_h=5, n_vit_w=7, types=typ))+ with torch.device('cuda'):+ want = ref.model(ids.cuda()[None], images=[images], token_types=types.cuda()[None])[1]+ req = eng.new_request(0, result.input_ids.tolist())+ req.mm_items = result.mm_items+ got = _prefill(eng, [req])+ _compare('image prefill', got, want, tol=tol)+ eng.finish_prefill([req])+ for step in range(3):+ token = want.argmax(-1)+ want = ref.decode(token, len(ids) + step)+ got = eng.decode([req], token.tolist())+ _compare(f'image decode {step}', got, want, tol=tol)+++@requires_cuda[email protected]('mode', ['exact', 'bounded'])+def test_images_survive_chunking_and_prefix_replay(tmp_path, mode):+ from .harness import TinyEngine+ from .test_reference_parity import _compare, _engram_table_for+ tensors = _checkpoint(tmp_path)+ eng = TinyEngine(str(tmp_path), max_seq_len=2048, swa_decoder_replay=mode)+ _engram_table_for(eng, tensors, tiny_text_config())+ # bounded mode caps a hit a window before the prompt end: trailing text keeps that cap at 128+ tail = 150 if mode == 'bounded' else 0+ def inputs():+ ids = [3] * 80 + [VOCAB - 1] + [4] * 37 + [VOCAB - 1] + [4] * 31 + [VOCAB - 1] + [4] * tail+ return _processor().apply(torch.tensor(ids, dtype=torch.int32), [_image(c) for c in ('red', 'blue', 'green')])+ whole = inputs()+ assert [i.offsets for i in whole.mm_items] == [[[80, 90]], [[127, 137]], [[168, 178]]]+ a = eng.new_request(0, whole.input_ids.tolist())+ a.mm_items = whole.mm_items+ want = _prefill(eng, [a])+ chunk = inputs()+ b = eng.new_request(1, chunk.input_ids.tolist())+ b.mm_items = chunk.mm_items+ b.device_len = 128+ _prefill(eng, [b])+ b.cached_len, b.device_len = 128, len(chunk.input_ids)+ got = _prefill(eng, [b])+ _compare('chunked image prefill', got, want)+ prefix = inputs()+ # The second image straddles the chunk / prefix boundary at 128 (in bounded mode the first chunk+ # also ends before the prompt's last window and runs no decoder row).+ hit = eng.new_request_on_prefix(1, a, prefix.input_ids.tolist())+ assert hit.cached_len == 128+ hit.mm_items = prefix.mm_items+ _compare('image prefix hit', _prefill(eng, [hit]), want)diff --git a/tests/models/test_quant_config.py b/tests/models/test_quant_config.pyindex 084a2e4a..bfcfb876 100644--- a/tests/models/test_quant_config.py+++ b/tests/models/test_quant_config.py@@ -1,6 +1,7 @@"""What each checkpoint's quant config resolves to: the dialect, the method behind every layer, and agreement with the stored tensors.-Rows need the checkpoint's config.json locally (/mnt/nvme/models or the HF cache); absent ones skip.+Rows need the checkpoint's config.json locally (/mnt/nvme/models or the HF cache); absent ones skip. The+stored-tensor scan also reads $FREETOKEN_TEST_MODELS_ROOT."""from __future__ import annotations@@ -47,6 +48,8 @@ from freetoken.utils.hf import sidecar_quantization_configfrom freetoken.utils.torch_utils import torch_dtypeMODELS = "/mnt/nvme/models"+# the stored-tensor scan takes every checkpoint under this root (the case table above matches by name)+SCAN_ROOT = os.environ.get("FREETOKEN_TEST_MODELS_ROOT", MODELS)HF_CACHE = os.path.expanduser("~/.cache/huggingface/hub")BF16, FP8B, FP8T, MXFP8, NVFP4 = UnquantizedLinearMethod, Fp8BlockLinearMethod, Fp8TensorLinearMethod, Mxfp8LinearMethod, Nvfp4LinearMethod@@ -435,6 +438,25 @@ def test_modelopt_nvfp4_reads_the_activation_quantizer_from_config_groups(extra,assert scheme.kind is QuantKind.NVFP4 and scheme.has("input_scale") is has_input_scale+def test_fp8_dialect_reads_the_block_size_and_the_nested_expert_dtype():+ """DeepSeek-V4.1 keeps the ``fp8`` dialect with 32x32 blocks and puts ``expert_dtype`` inside quantization_config."""+ from freetoken.layers.quantization.scheme import fp8_block_size++ q = {"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4"}+ quant = QuantConfig.from_hf(SimpleNamespace(quantization_config=q))+ assert type(quant) is Fp8BlockConfig and quant.block == 32+ dense = quant.scheme_for("layers.3.attn.wq_a")+ assert dense.kind is QuantKind.FP8_BLOCK and fp8_block_size(dense) == 32 and dense.weight.scale == "e8m0"+ assert quant.storage(dense)["weight_scale_inv"].name == "scale"+ assert quant.scheme_for("layers.3.ffn.experts.7.w1").kind is QuantKind.MXFP4+ # the V4 128-block export still resolves to the 128 scheme+ q128 = {"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [128, 128], "scale_fmt": "ue8m0"}+ assert fp8_block_size(QuantConfig.from_hf(SimpleNamespace(quantization_config=q128)).scheme_for("layers.3.attn.wq_a")) == 128+ for bad in ([64, 64], [32, 128], [128]):+ with pytest.raises(NotImplementedError):+ QuantConfig.from_hf(SimpleNamespace(quantization_config={**q, "weight_block_size": bad}))++def test_every_dialect_names_the_tensors_behind_its_schemes():from freetoken.layers.quantization.registry import dialects@@ -491,7 +513,7 @@ def _weight_map(p: Path) -> dict[str, str] | None:def _candidate_dirs() -> list[Path]:- roots = glob.glob(os.path.join(HF_CACHE, "models--*/snapshots/*/")) + glob.glob(os.path.join(MODELS, "*/"))+ roots = glob.glob(os.path.join(HF_CACHE, "models--*/snapshots/*/")) + glob.glob(os.path.join(SCAN_ROOT, "*/"))return [Path(d) for d in roots if (Path(d) / "config.json").exists() and _weight_map(Path(d)) is not None]diff --git a/tests/moe/test_cpu_moe.py b/tests/moe/test_cpu_moe.pyindex f496de31..547c94c6 100644--- a/tests/moe/test_cpu_moe.py+++ b/tests/moe/test_cpu_moe.py@@ -391,6 +391,7 @@ def test_cpu_decode_dsfp4_matches_gpu(bs):max_tokens=bs,device=dev,swiglu_limit=swiglu_limit,+ act_block=128,)hidden = torch.randn(bs, H, device=dev, dtype=torch.bfloat16)@@ -585,7 +586,7 @@ def test_cpu_moe_decode_cuda_graph_replay_dsfp4():ex = CpuMoeExecutor(cache, top_k=top_k, activation="silu",apply_router_weight_on_input=False, num_threads=8, max_tokens=bs, device=dev,- swiglu_limit=limit,+ swiglu_limit=limit, act_block=128,)b = cache.bank_sourceslayer_banks = [b[n][layer].to(dev) for n in (diff --git a/tests/moe/test_dsfp4_prequant.py b/tests/moe/test_dsfp4_prequant.pyindex 6d6b49d3..11519502 100644--- a/tests/moe/test_dsfp4_prequant.py+++ b/tests/moe/test_dsfp4_prequant.py@@ -46,7 +46,7 @@ def test_dsfp4_gpu_prequant_bit_parity():def run(cpu_roundtrip: bool) -> torch.Tensor:ex = CpuMoeExecutor(cache, top_k=TOPK, activation="silu", apply_router_weight_on_input=False,- num_threads=4, max_tokens=2, device=dev, swiglu_limit=10.0,+ num_threads=4, max_tokens=2, device=dev, swiglu_limit=10.0, act_block=128,)assert ex._gpu_prequant, "ds_fp4 on CUDA should enable GPU prequant"if cpu_roundtrip: # control: the original CPU-side scalar round-tripdiff --git a/tests/moe/test_dsfp4_reference_semantics.py b/tests/moe/test_dsfp4_reference_semantics.pynew file mode 100644index 00000000..2f89cf33--- /dev/null+++ b/tests/moe/test_dsfp4_reference_semantics.py@@ -0,0 +1,170 @@+"""ds_fp4 (DeepSeek W4A8) experts against an INDEPENDENT torch transcription over+``fp4_gemm(act_quant(x, block), W)`` at both fp8 activation blocks (128 on V4, 32 on V4.1):++ x_q = fp8_roundtrip(x, block) # act_quant(..., inplace)+ gate = bf16(x_q @ deq(W1)^T), up = bf16(x_q @ deq(W3)^T) # fp4_gemm outputs bf16+ h = bf16(silu(min(gate, L)) * clamp(up, -L, L))+ y_r = bf16(w_r * (fp8_roundtrip(h, block) @ deq(W2)^T)) # routing weight in the down epilogue+ y = sum_r y_r++The routing weight scales the down output (the placement main serves V4 with); the reference+``Expert.forward`` scales the intermediate before its fp8 quant instead. The GEMV (decode), the+grouped GEMM (prefill) and the CPU executor all follow this transcription.+"""++from __future__ import annotations++import pytest+import torch++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++E, H, I, TOP_K, LIMIT = 16, 512, 256, 4, 7.0+E2M1 = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0])+++def _banks(device, seed=0):+ g = torch.Generator(device="cpu").manual_seed(seed)++ def u8(*shape, low=0, high=256):+ return torch.randint(low, high, shape, dtype=torch.uint8, generator=g).to(device)++ return (u8(E, 2 * I, H // 2), u8(E, 2 * I, H // 32, low=120, high=130), u8(E, H, I // 2), u8(E, H, I // 32, low=120, high=130))+++def _dequant(packed: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:+ """[E, N, K//2] e2m1 pairs (even channel low nibble) + [E, N, K//32] e8m0 -> [E, N, K] fp32."""+ lut = E2M1.to(packed.device)+ lo, hi = lut[(packed & 0xF).long()], lut[(packed >> 4).long()]+ vals = torch.stack([lo, hi], dim=-1).flatten(-2) # [E, N, K]+ s = torch.exp2(scale.view(torch.uint8).float() - 127.0).repeat_interleave(32, dim=-1)+ return vals * s+++def _fp8_roundtrip(x: torch.Tensor, block: int) -> torch.Tensor:+ g = x.float().unflatten(-1, (-1, block))+ s = torch.exp2(torch.ceil(torch.log2(g.abs().amax(-1).clamp_min(1e-4) / 448.0)))+ q = (g / s.unsqueeze(-1)).clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float()+ return (q * s.unsqueeze(-1)).flatten(-2).to(torch.bfloat16)+++def reference(x, slots, weights, banks, block):+ """The MoE over dequantized banks: per-route bf16 expert outputs accumulated into an fp32 ``y``,+ returned in fp32 (the kernels round that sum to bf16)."""+ gu_p, gu_s, dn_p, dn_s = banks+ W13 = _dequant(gu_p, gu_s) # [E, 2I, H]+ W2 = _dequant(dn_p, dn_s) # [E, H, I]+ xq = _fp8_roundtrip(x, block).float()+ T = x.shape[0]+ y = torch.zeros(T, H, dtype=torch.float32, device=x.device)+ for t in range(T):+ for r in range(TOP_K):+ e = int(slots[t, r])+ gu = (xq[t] @ W13[e].T).to(torch.bfloat16).float()+ gate, up = gu[:I].clamp(max=LIMIT), gu[I:].clamp(-LIMIT, LIMIT)+ h = (torch.nn.functional.silu(gate) * up).to(torch.bfloat16)+ hq = _fp8_roundtrip(h.view(1, -1), block).float().view(-1)+ y[t] += (weights[t, r].float() * (hq @ W2[e].T)).to(torch.bfloat16).float()+ return y+++def _inputs(T, device, seed=1):+ g = torch.Generator(device="cpu").manual_seed(seed)+ x = (torch.randn(T, H, dtype=torch.bfloat16, generator=g) * 0.5).to(device)+ slots = torch.stack([torch.randperm(E, generator=g)[:TOP_K] for _ in range(T)]).to(device=device, dtype=torch.int32).contiguous()+ w = torch.rand(T, TOP_K, generator=g)+ w = (1.5 * w / w.sum(-1, keepdim=True)).float().to(device).contiguous() # route_scale-style weights > 1 too+ return x, slots, w++[email protected]("block", [32, 128])+def test_gemv_path_matches_the_transcription(block):+ from freetoken.moe.fused_ds_fp4 import routed_experts_fp4++ banks = _banks("cuda")+ x, slots, w = _inputs(6, "cuda")+ got = routed_experts_fp4(x, slots, w, *banks, LIMIT, act_block=block).float()+ want = reference(x, slots, w, banks, block)+ torch.testing.assert_close(got, want, atol=2e-2, rtol=2e-2)++[email protected]("block", [32, 128])+def test_cpu_executor_matches_the_transcription(block):+ """The ``ds_fp4`` CPU executor (cpu / hybrid decode) implements the same contract as the GPU path."""+ from types import SimpleNamespace++ from freetoken.kernel.pinned import alloc_pinned_tensor+ from freetoken.moe.cpu_executor import CpuMoeExecutor++ gu_p, gu_s, dn_p, dn_s = _banks("cuda")+ pinned = {}+ for name, t in (("gate_up_packed", gu_p), ("gate_up_scale", gu_s), ("down_packed", dn_p), ("down_scale", dn_s)):+ buf = alloc_pinned_tensor(*t.shape, dtype=torch.uint8)+ buf.copy_(t.cpu())+ pinned[name] = buf+ cache = SimpleNamespace(quant_format="ds_fp4", bank_sources={k: [v] for k, v in pinned.items()}, num_layers=1, num_experts=E)+ dev = torch.device("cuda", 0)+ x, slots, w = _inputs(2, dev)+ ex = CpuMoeExecutor(cache, top_k=TOP_K, activation="silu", apply_router_weight_on_input=False, num_threads=4, max_tokens=2, device=dev, swiglu_limit=LIMIT, act_block=block)+ got = ex.decode(0, x, w, slots.clone()).clone().float()+ torch.cuda.synchronize()+ del ex+ want = reference(x, slots, w, (gu_p, gu_s, dn_p, dn_s), block)+ torch.testing.assert_close(got, want, atol=2e-2, rtol=2e-2)+++def test_grouped_prefill_path_matches_the_transcription():+ from freetoken.moe import fused_ds_fp4++ banks = _banks("cuda")+ x, slots, w = _inputs(64, "cuda")+ got = fused_ds_fp4.routed_experts_fp4_prefill(x, slots, w, *banks, LIMIT, E, act_block=32).float()+ want = reference(x, slots, w, banks, 32)+ torch.testing.assert_close(got, want, atol=2e-2, rtol=2e-2)++[email protected]("block", [32, 128])+def test_inactive_routes_contribute_zero_without_reading_their_slots(block):+ """Hybrid decode hands the GPU kernel slot -1 (weight 0) for the routes the CPU computes. The kernel+ must produce exactly zero for them without touching any slot: every slot the active routes do not+ use is filled with 0xFF (an e8m0 scale of NaN), and the result still equals the reference over the+ active routes and the CPU executor's output for the same split."""+ from types import SimpleNamespace++ from freetoken.kernel.pinned import alloc_pinned_tensor+ from freetoken.moe.cpu_executor import CpuMoeExecutor+ from freetoken.moe.fused_ds_fp4 import routed_experts_fp4++ gu_p, gu_s, dn_p, dn_s = (t.clone() for t in _banks("cuda"))+ x, slots, w = _inputs(2, "cuda")+ # routes: token 0 keeps routes 0..1 on the GPU, token 1 keeps route 3; the rest go to the CPU+ on_gpu = torch.zeros_like(slots, dtype=torch.bool)+ on_gpu[0, :2] = True+ on_gpu[1, 3] = True+ gpu_slots = torch.where(on_gpu, slots, -1)+ gpu_w = torch.where(on_gpu, w, 0.0)+ used = set(slots[on_gpu].tolist())+ poison = [e for e in range(E) if e not in used]+ for t in (gu_p, gu_s, dn_p, dn_s):+ t[poison] = 0xFF+ got = routed_experts_fp4(x, gpu_slots, gpu_w, gu_p, gu_s, dn_p, dn_s, LIMIT, act_block=block)+ assert torch.isfinite(got).all()+ clean = _banks("cuda")+ # the poisoned-bank result equals the clean-bank result with the same holes (neither reads them),+ # and each share matches the reference over its own routes (a zero weight drops a route)+ got_clean = routed_experts_fp4(x, gpu_slots, gpu_w, *clean, LIMIT, act_block=block)+ assert torch.equal(got, got_clean)+ torch.testing.assert_close(got.float(), reference(x, slots, gpu_w, clean, block), atol=2e-2, rtol=2e-2)+ # the CPU executor computes the complementary split from the same raw ids (ids < 0 skipped)+ pinned = {}+ for name, t in (("gate_up_packed", clean[0]), ("gate_up_scale", clean[1]), ("down_packed", clean[2]), ("down_scale", clean[3])):+ buf = alloc_pinned_tensor(*t.shape, dtype=torch.uint8)+ buf.copy_(t.cpu())+ pinned[name] = buf+ cache = SimpleNamespace(quant_format="ds_fp4", bank_sources={k: [v] for k, v in pinned.items()}, num_layers=1, num_experts=E)+ ex = CpuMoeExecutor(cache, top_k=TOP_K, activation="silu", apply_router_weight_on_input=False, num_threads=4, max_tokens=2, device=torch.device("cuda", 0), swiglu_limit=LIMIT, act_block=block)+ cpu_ids = torch.where(on_gpu, -1, slots)+ cpu_out = ex.decode(0, x, w, cpu_ids.clone()).clone()+ torch.cuda.synchronize()+ del ex+ torch.testing.assert_close(cpu_out.float(), reference(x, slots, torch.where(on_gpu, 0.0, w), clean, block), atol=2e-2, rtol=2e-2)diff --git a/tests/scheduler/test_dsv4_generic_manager.py b/tests/scheduler/test_dsv4_generic_manager.pyindex ad69b89e..1bca10bc 100644--- a/tests/scheduler/test_dsv4_generic_manager.py+++ b/tests/scheduler/test_dsv4_generic_manager.py@@ -14,8 +14,8 @@ import pytestimport torchfrom freetoken.core import Req, SamplingParams-from freetoken.kvcache.dsv4_cost_model import dsv4_pool_sizes-from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache+from freetoken.kvcache.dsv4.v4_cost_model import dsv4_pool_sizes+from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCachefrom freetoken.models.deepseek_v4.args import DeepseekV4Argsfrom freetoken.scheduler.cache import CacheManager@@ -191,6 +191,33 @@ def test_chunk_boundaries_stay_page_aligned_under_unaligned_budget():cm.check_integrity()[email protected]("replay, new_tokens, cached", [(0, 2, 256), (P, 2, 128), (P, 200, 256)])+def test_admission_caps_the_hit_by_the_pools_prefix_replay(replay, new_tokens, cached):+ """A pool whose model recomputes the prompt's tail (DeepSeek-V4.1's bounded decoder replay) holds+ back ``prefix_replay_tokens`` at admission: the match ends at or before ``L - replay`` (page-aligned+ by the radix), a hit already that far from the prompt end is untouched, and the request's+ ``prompt_len`` is the whole prompt."""+ from freetoken.scheduler.decode import DecodeManager+ from freetoken.scheduler.prefill import PrefillManager+ from freetoken.scheduler.table import TableManager+ from freetoken.scheduler.utils import PendingReq++ cm, pool, pt = _stack(num_pages=48)+ pool.prefix_replay_tokens = replay+ prompt = torch.arange(1, 301, dtype=torch.int32)+ _lifecycle(cm, _req(MRR - 1, prompt, n_decode=310), total_len=310)+ pm = PrefillManager(cm, TableManager(max_running_reqs=MRR, page_table=pt), DecodeManager(page_size=P))+ tokens = torch.cat([prompt, torch.arange(1000, 1000 + new_tokens, dtype=torch.int32)])+ pm.pending_list = [PendingReq(uid=1, input_ids=tokens, sampling_params=SamplingParams(max_tokens=1))]+ batch = pm.schedule_next_batch(1024)+ req = batch.reqs[0]+ assert req.cached_len == cached and req.extend_len == len(tokens) - cached and req.prompt_len == len(tokens)+ cm.allocate_paged(batch.reqs)+ req.complete_one()+ cm.cache_req(req, finished=True)+ cm.check_integrity()++def test_abort_anywhere_fuzz_conserves_and_isolates():"""admit / extend / finish-or-abort at arbitrary points with shared prefixes + evictionpressure, on a deliberately small window tier. The generic conservation check (free + treediff --git a/tests/scheduler/test_scheduler_kv_usage.py b/tests/scheduler/test_scheduler_kv_usage.pyindex 6ab94970..bf0e825c 100644--- a/tests/scheduler/test_scheduler_kv_usage.py+++ b/tests/scheduler/test_scheduler_kv_usage.py@@ -2,6 +2,8 @@ from __future__ import annotationsfrom types import SimpleNamespace+from freetoken.attention import AttnType+def test_kv_usage_pages_excludes_evictable_prefix_cache():# page_usage now lives on the CacheManagerLike interface (polymorphic vs DSV4); test the@@ -53,10 +55,15 @@ def _fake_engine(swa=True, moe=True, mamba=True):config=SimpleNamespace(page_size=16,cache_type="swa_radix",- model_config=SimpleNamespace(dsv4_args=None, has_swa_attention=swa),+ max_running_req=1,+ model_config=SimpleNamespace(+ dsv4_args=None, has_swa_attention=swa,+ kv_cache_group_specs=lambda: [SimpleNamespace(attn_type=AttnType.SWA, is_swa=True, sliding_window=128)] if swa else [],+ swa_attention_group=lambda: SimpleNamespace(sliding_window=128),+ ),),kv_cache=SimpleNamespace(- swa_num_tokens=513, unit_bytes=lambda: (1 << 20, 1 << 21)+ window_pages=512, swa_num_tokens=513, unit_bytes=lambda: (1 << 20, 1 << 21)),moe_offload_cache=SimpleNamespace(cache_size=24, num_layers=8, num_experts=16,diff --git a/tests/server/test_effort_dialect.py b/tests/server/test_effort_dialect.pyindex 6a68a4ec..a25de8e7 100644--- a/tests/server/test_effort_dialect.py+++ b/tests/server/test_effort_dialect.py@@ -159,6 +159,22 @@ def test_empty_effort_is_treated_as_absent():assert state.sent.chat_template_kwargs == {}+def test_numeric_effort_rides_template_kwargs_not_the_wire_field():+ # The wire field takes named tiers only (as vLLM and SGLang type it); a numeric budget goes+ # through chat_template_kwargs, where the checkpoint's encoder validates it at render time.+ from pydantic import ValidationError+ import pytest++ for effort in (37, True, 25.5):+ with pytest.raises(ValidationError):+ chat_request(reasoning_effort=effort)+ ctk = {"enable_thinking": True, "reasoning_effort": 37}+ state = FakeState(reasoning_parser="qwen3")+ response = run(handle_chat_completion(chat_request(chat_template_kwargs=ctk), None, state, {}))+ assert not isinstance(response, JSONResponse)+ assert state.sent.chat_template_kwargs == ctk++def test_foreign_thinking_shapes_stay_ignored():# extra="allow" swallowed any thinking shape before the field existed;# a bare string, a bool, or a typeless dict must keep working unchanged.diff --git a/tests/server/test_function_call_parser.py b/tests/server/test_function_call_parser.pyindex 34c1c04d..bc222967 100644--- a/tests/server/test_function_call_parser.py+++ b/tests/server/test_function_call_parser.py@@ -337,6 +337,34 @@ def test_dsv32_streaming_multi_param_args_prefix_stable():}+V41_BLOCK = (+ "<|DSML| calls>\n"+ '<|DSML| invoke name="glob">\n'+ '<|DSML| parameter name="pattern" string="true">*.py</|DSML| parameter>\n'+ '<|DSML| parameter name="path" string="true">/src</|DSML| parameter>\n'+ "</|DSML| invoke>\n"+ "</|DSML| calls>"+)+++def test_dsv41_dsml_tags_lead_with_a_space():+ """DeepSeek-V4.1 leads every DSML tag name with a space (its encoder's ``<|DSML| calls>``):+ the V3.2 detector sees no tool call there, the V4.1 one parses it whole and streamed."""+ text = "Let me look.\n\n" + V41_BLOCK+ assert not FunctionCallParser(OPENCODE_TOOLS, tool_call_parser="deepseekv32").has_tool_call(text)+ parser = FunctionCallParser(OPENCODE_TOOLS, tool_call_parser="deepseekv41")+ assert parser.has_tool_call(text)+ result = parser.parse_non_stream(text)+ assert result.normal_text == "Let me look."+ assert [(c.name, json.loads(c.parameters)) for c in result.calls] == [("glob", {"pattern": "*.py", "path": "/src"})]++ streamed = FunctionCallParser(OPENCODE_TOOLS, tool_call_parser="deepseekv41")+ texts, calls = _feed(streamed, [text[i : i + 5] for i in range(0, len(text), 5)])+ assert "".join(texts).strip() + streamed.finish_stream() == "Let me look."+ assert [c.name for c in calls if c.name] == ["glob"]+ assert json.loads("".join(c.parameters for c in calls if c.name is None)) == {"pattern": "*.py", "path": "/src"}++def test_streaming_support_flags():# Every registered detector is incremental-safe (buffered fallback remains as# the escape hatch for future formats, exercised via monkeypatch indiff --git a/tests/server/test_parser_auto_selection.py b/tests/server/test_parser_auto_selection.pyindex 78b1663e..e1eb8798 100644--- a/tests/server/test_parser_auto_selection.py+++ b/tests/server/test_parser_auto_selection.py@@ -87,6 +87,13 @@ def test_qwen3_5_is_not_shadowed_by_the_generic_qwen_branch():assert _inferred("Qwen3MoeForCausalLM")[0] == "qwen25"+def test_deepseek_v41_is_not_shadowed_by_the_v4_branch():+ """V4.1's DSML leads every tag name with a space; the V4 arm (``"v4"`` is a substring of+ ``"v41"``) would hand it a tool parser that never sees a call."""+ assert _inferred("DeepseekV41ForCausalLM") == ("deepseekv41", "deepseekv32")+ assert _inferred("DeepseekV4ForCausalLM")[0] == "deepseekv32"++def test_an_explicit_choice_beats_inference():config = _Config({"architectures": ["DeepseekV4ForCausalLM"], "torch_dtype": "bfloat16"})with patch("freetoken.utils.cached_load_hf_config", lambda _path: config):diff --git a/tests/server/test_streaming_model_matrix.py b/tests/server/test_streaming_model_matrix.pyindex 43922021..eb6e3a2e 100644--- a/tests/server/test_streaming_model_matrix.py+++ b/tests/server/test_streaming_model_matrix.py@@ -67,6 +67,11 @@ CALL_BLOCKS = {'<|DSML|parameter name="filePath" string="true">/tmp/test_calc.py</|DSML|parameter>'"</|DSML|invoke></|DSML|function_calls>"),+ "deepseekv41": (+ '<|DSML| calls>\n<|DSML| invoke name="read">\n'+ '<|DSML| parameter name="filePath" string="true">/tmp/test_calc.py</|DSML| parameter>\n'+ "</|DSML| invoke>\n</|DSML| calls>"+ ),"qwen25": '<tool_call>\n{"name": "read", "arguments": {"filePath": "/tmp/test_calc.py"}}\n</tool_call>',"qwen3_coder": ("<tool_call><function=read><parameter=filePath>/tmp/test_calc.py</parameter>"@@ -104,6 +109,7 @@ CALL_BLOCKS = {# Substrings that must never leak into user-visible content.MARKUP_MARKERS = {"deepseekv32": ["|DSML|"],+ "deepseekv41": ["|DSML|"],"qwen25": ["<tool_call>", "</tool_call>"],"qwen3_coder": ["<tool_call>", "<function="],"glm47": ["<arg_key>", "<arg_value>", "</tool_call>"],@@ -121,6 +127,7 @@ MARKUP_MARKERS = {# opens the block implicitly (model emits only the closing marker).REASONING_FAMILIES = {"dsv4": ("deepseekv32", "deepseekv32", "", "</think>"),+ "dsv4.1": ("deepseekv41", "deepseekv32", "", "</think>"),"qwen3.5": ("qwen3_coder", "qwen3", "", "</think>"),"qwen": ("qwen25", "qwen3", "", "</think>"),"glm4.7": ("glm47", "glm", "", "</think>"),diff --git a/tests/tokenizer/test_effort.py b/tests/tokenizer/test_effort.pyindex e56df5fe..e0cf7a81 100644--- a/tests/tokenizer/test_effort.py+++ b/tests/tokenizer/test_effort.py@@ -1,6 +1,8 @@"""Unit tests for the reasoning-effort dialect layer (tokenizer/effort.py)."""from __future__ import annotations+import pytest+from freetoken.tokenizer.effort import (EFFORT_SCALE,EffortProfile,@@ -131,3 +133,37 @@ def test_probe_skips_rounds_whose_baseline_fails():profile = probe_effort_profile(render)assert profile.supported == frozenset({"xhigh", "medium", "low"})assert profile.consumes_effort+++def _numeric_render(kwargs, tools):+ if not tools and not kwargs.get("enable_thinking"):+ return "chat"+ effort = kwargs.get("reasoning_effort", "high")+ if isinstance(effort, str):+ effort = {"low": 50, "high": 75, "max": 100}[effort]+ if type(effort) is not int or not 1 <= effort <= 100:+ raise ValueError("invalid budget")+ return f"Reasoning Effort: {effort}"+++def test_probe_preserves_validated_numeric_budgets():+ profile = probe_effort_profile(_numeric_render)+ assert profile.integer_range == (1, 100)+ assert profile.default == "high"+ for budget in range(1, 101):+ assert quantize_effort(budget, profile) == budget+ for bad in (0, 101, -1, True, False, 1.0, 1.5):+ with pytest.raises(ValueError, match="integer"):+ quantize_effort(bad, profile)+++def test_numeric_capability_requires_validation_and_the_whole_range():+ assert probe_effort_profile(_dsv4_render).integer_range is None+ assert probe_effort_profile(lambda kw, tools: str(kw)).integer_range is None++ def holes(kwargs, tools):+ if kwargs.get("reasoning_effort") == 42:+ raise ValueError("unsupported")+ return _numeric_render(kwargs, tools)++ assert probe_effort_profile(holes).integer_range is Nonediff --git a/tests/tokenizer/test_tokenize.py b/tests/tokenizer/test_tokenize.pyindex 16ef6d74..fb73fd31 100644--- a/tests/tokenizer/test_tokenize.py+++ b/tests/tokenizer/test_tokenize.py@@ -405,6 +405,36 @@ class FakeDsv4Tokenizer:return torch.tensor([[4, 5, 6]], dtype=torch.long)+def test_dsv41_encoder_gets_image_sources_before_and_after_fetch(tmp_path):+ import copy+ from freetoken.mm.media import collect_image_refs+ encoding_dir = tmp_path / "encoding"+ encoding_dir.mkdir()+ (encoding_dir / "encoding_dsv4.py").write_text('''+def encode_messages(messages, thinking_mode, reasoning_effort=None):+ parts = messages[0]["content"]+ assert parts[1] == {"type": "text", "text": "compare"}+ assert parts[0].get("url") == "https://example.com/image.png" or parts[0].get("data") == b"first image bytes"+ assert parts[2].get("data") in ("aW1hZ2U=", b"second image bytes")+ return "two images"+''')+ messages = [{"role": "user", "content": [+ {"type": "image", "freetoken_ref": {"kind": "url", "data": "https://example.com/image.png"}},+ {"type": "text", "text": "compare"},+ {"type": "image", "freetoken_ref": {"kind": "b64", "data": "aW1hZ2U="}},+ ]}]+ manager = TokenizeManager(FakeDsv4Tokenizer(tmp_path))+ msg = TokenizeMsg(uid=1, text=messages, sampling_params=SamplingParams())+ original = copy.deepcopy(messages)+ assert manager.render_prompt(msg) == "two images"+ assert messages == original+ assert len(collect_image_refs(messages)) == 2+ msg.images = [b"first image bytes", b"second image bytes"]+ stripped = copy.deepcopy(messages)+ assert manager.render_prompt(msg) == "two images"+ assert messages == stripped++def test_tokenize_manager_uses_dsv4_encoder_when_chat_template_is_missing(tmp_path):encoding_dir = tmp_path / "encoding"encoding_dir.mkdir()@@ -761,3 +791,28 @@ def test_tokenize_survives_an_unhashable_effort():manager.tokenize([msg])assert "reasoning_effort" not in tokenizer.chat_template_kwargs++[email protected]("budget", [1, 25, 50, 75, 100])+def test_numeric_encoder_effort_survives_prompt_rendering(tmp_path, budget):+ encoding_dir = tmp_path / "encoding"+ encoding_dir.mkdir()+ (encoding_dir / "encoding.py").write_text('''+def encode_messages(messages, thinking_mode, reasoning_effort=None):+ if thinking_mode != "thinking":+ return "chat"+ effort = "high" if reasoning_effort is None else reasoning_effort+ if isinstance(effort, str):+ effort = {"low": 50, "high": 75, "max": 100}[effort]+ if type(effort) is not int or not 1 <= effort <= 100:+ raise ValueError("invalid budget")+ return f"Reasoning Effort: {effort}"+''')+ manager = TokenizeManager(FakeDsv4Tokenizer(tmp_path))+ msg = TokenizeMsg(+ uid=1, text=[{"role": "user", "content": "hello"}],+ sampling_params=SamplingParams(),+ chat_template_kwargs={"enable_thinking": True, "reasoning_effort": budget},+ )+ assert manager.render_prompt(msg) == f"Reasoning Effort: {budget}"+ assert msg.chat_template_kwargs["reasoning_effort"] == budget