Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

feat(deepseek-v41): support text and vision serving (#603)

Xiaoze Fan127c85665b49
diff --git a/docs/models.md b/docs/models.md
index 2cd3baa3..b999d29e 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -5,6 +5,7 @@ for them; other checkpoints of the same architectures work too.
| Model | HF checkpoints |
|---|---|
+| DeepSeek-V4.1-Flash | [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) |
| DeepSeek-V4 | [deepseek-ai/DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) |
| GLM-5.3-Flash | [RedHatAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4) |
| GLM-5.2 | [nvidia/GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) |
@@ -27,6 +28,7 @@ These families accept image input by default; pass `--text-model-only` to skip t
| Family | Image tokens | `--image-min-tokens` / `--image-max-tokens` | `--mm-processor-kwargs` example |
| --- | --- | --- | --- |
+| DeepSeek-V4.1-Flash (ViT tower, streamed under `--mm-encoder-weights host`) | one feature per 42x42 pixels, plus a learned row break per row and two delimiters; checkpoint cap 1024 total span tokens | maximum caps the entire span; minimum maps to a pixel-area floor before the cap; checkpoint minimum area is 544x544 | `{"max_image_tokens": 512, "min_pixels": 295936}` |
| Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next, Qwen3-VL | one token per 32x32 pixels of the resized image, dynamic resolution | pixel areas in `size.shortest_edge` / `longest_edge`; checkpoint defaults 64 to 16384 tokens | `{"size": {"longest_edge": 1048576}}` |
| Gemma-4 26B-A4B, 31B (`gemma4`: ViT tower, streamed under `--mm-encoder-weights host`) | one of the soft-token budgets 70 / 140 / 280 / 560 / 1120, every image scaled to its budget as far as the aspect ratio allows | the maximum picks the largest budget within it, below 70 is refused at start-up; the minimum has no effect | `{"max_soft_tokens": 1120}` |
| Gemma-4 12B (`gemma4_unified`: linear patch embedder, resident under either placement) | same budgets, one 48x48 super-patch per soft token | same as the tower releases | same |
@@ -86,4 +88,34 @@ unset on PCIe GPUs: wider grids add no bandwidth there.
with `ft checkpoint`, or add the encoder in place with [scripts/ftw_hotfix.py](ftw-hotfix.md).
- DeepSeek-V4 checkpoints must keep the `inference/config.json` subdir — the
authoritative model args are read from there.
+- DeepSeek-V4.1-Flash serves text and images from the HF checkpoint as shipped (fp8 block-32
+ dense weights, fp4 experts on the offload cache, 890 B/token global KV in packed
+ fp4/fp8 pools). Its two 98 GiB Engram tables stream from the checkpoint shards on
+ demand (keep them on a fast NVMe; the 6 GiB of table scales stay in host RAM), using
+ bounded pinned staging buffers; an FTW conversion copies the shards that hold them next
+ to the checkpoint.
+ SWA bounded replay (DeepSeek_V41_Tech_Report.pdf, shipped in the checkpoint, §3.2.2): a replayed
+ segment recomputes only the SWA KV of its last `n_win` tokens and truncates each query's window to
+ the segment. *Encoder* replay rebuilds the encoder SWA KV behind a prefix hit from the global KV
+ alone (approximate; the report's fallback when the SWA KV of a hit has been evicted). *Decoder*
+ replay runs the decoder layers on each prompt's last `n_win` tokens only; their SWA KV is never
+ prefix-cached and post-training simulated it. FreeToken keeps encoder SWA KV in the radix cache (no
+ encoder replay); `--swa-decoder-replay bounded` (default) is the report's decoder replay, with a
+ prefix hit stopping at least `n_win` tokens before the prompt end so those tokens are prefilled;
+ `exact` runs the decoder on every prompt token (reference numerics).
+ Bounded output differs from exact by construction and does not depend on the prefill chunk size.
+ Under expert offload the prefill time is bounded by streaming each layer's experts, so decoder
+ replay saves decoder-layer compute, not prefill time.
+ Run `ft bench bw --model dsv4.1-flash` to measure local PCIe and CPU bandwidth before
+ choosing a MoE strategy; `--moe-strategy hybrid` splits expert misses using those
+ measurements. With `--moe-cache-auto`, `--kv-reserve-tokens` sets the token capacity reserved
+ before the expert cache is allocated, subject to the attention pool's structural minimum; size it
+ for the intended context and concurrency. `--max-extend-length` controls the prefill chunk size
+ and its workspace. Image input uses the shared multimodal processor and encoder cache, including
+ chunked prefill and prefix replay; `--text-model-only` skips the vision weights. DSpark
+ speculative decoding is not served.
+ `reasoning_effort` takes `low`, `high` or `max`; the 1-100 integer budget goes through the
+ template kwargs with thinking on, `"chat_template_kwargs": {"enable_thinking": true,
+ "reasoning_effort": 37}`, and the checkpoint's encoder validates it. Runtime window-cache
+ controls use 128-token pages.
- Qwen3.8-Flash-Next keeps a 47.7 GiB PLE n-gram table pinned in host RAM.
diff --git a/python/freetoken/attention/__init__.py b/python/freetoken/attention/__init__.py
index 8a410ffe..eda3b61f 100644
--- a/python/freetoken/attention/__init__.py
+++ b/python/freetoken/attention/__init__.py
@@ -102,6 +102,16 @@ def create_dsv4_sparse_backend(config: ModelConfig):
return DSV4SparseAttnBackend(config)
+@SUPPORTED_ATTENTION_BACKENDS.register(
+ "dsv41_sparse",
+ BackendInfo(supported_types=frozenset({AttnType.DSV41})),
+)
+def create_dsv41_sparse_backend(config: ModelConfig):
+ from .dsv41_sparse import DSV41SparseAttnBackend
+
+ return DSV41SparseAttnBackend(config)
+
+
@SUPPORTED_ATTENTION_BACKENDS.register(
"dsa",
BackendInfo(supported_types=frozenset({AttnType.MLA, AttnType.DSA})),
diff --git a/python/freetoken/attention/base.py b/python/freetoken/attention/base.py
index 9115b809..846e9944 100644
--- a/python/freetoken/attention/base.py
+++ b/python/freetoken/attention/base.py
@@ -20,6 +20,9 @@ class AttnType(str, Enum):
MLA = "mla" # plain latent-KV MLA -> MLAKVCache
DSA = "dsa" # latent-KV MLA + DSA sparse indexer -> DSAKVCache
DSV4 = "dsv4" # DSV4 window+compressed sparse -> DSV4PagedKVCache
+ # DSV41 (DeepSeek-V4.1): window + cross-layer-shared compressed KV in packed fp8/fp4
+ # rows, Full/Reindex/Reuse layer modes -> DSV41PagedKVCache
+ DSV41 = "dsv41"
LINEAR = "linear" # GDN/mamba state layers -> LinearStatePool
# GQA block-sparse (MiniMax-M3): paged GQA K/V + a per-sparse-layer index-key
# slab; the indexer picks top-k 128-token blocks per query -> BSAKVCache
diff --git a/python/freetoken/attention/dsv41_sparse.py b/python/freetoken/attention/dsv41_sparse.py
new file mode 100644
index 00000000..baaa2cf0
--- /dev/null
+++ b/python/freetoken/attention/dsv41_sparse.py
@@ -0,0 +1,438 @@
+"""DeepSeek-V4.1 sparse-attention backend.
+
+Same contract and division of labour as ``dsv4_sparse``: the backend owns the per-forward KV
+ADDRESSING over the shared page table (window ring slots, compressed rows, compress-state carry,
+the decode snapshot and the CUDA-graph staging buffers) plus the selection helpers that are pure
+index arithmetic (causal top-k, the hierarchical candidate pool); the model computes projections
+and hands the picks back to be resolved into pool rows.
+
+What DSV41 adds on top of DSV4's vocabulary:
+
+* **cross-layer sharing** -- the compressed tiers belong to the kv-source layer; a consumer layer
+ addresses ``source_of(layer)``'s pools. The Top-K rows and the candidate pool the Full / Reindex
+ layers produce ride on the per-forward ``SharedSelection`` for the Reuse layers to read.
+* **packed rows** -- writes quantize into the pool (``pool.store_*``), reads dequantize in-kernel.
+* **prefill passes with a window floor** -- a ``PrefillSegment`` carries ``window_floor``: the
+ absolute position a query's sliding window may not reach below (Decoder SWA Bounded Replay
+ truncates the decoder's window at ``L - W``, the start of the prompt's last window). The metadata
+ carries the encoder pass (every new token) and the decoder pass (the new tokens inside the
+ prompt's last window, or every new token in exact mode).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING, List
+
+import torch
+from freetoken.core import Batch, get_global_ctx
+
+from .base import AttentionSpec, BaseAttnBackend, BaseAttnMetadata
+
+if TYPE_CHECKING:
+ from freetoken.models import ModelConfig
+
+
+@dataclass(frozen=True)
+class PrefillSegment:
+ """One request's slice of a flat prefill token stream.
+
+ ``offset``/``n`` tile the stream; ``start_pos`` is the absolute position of the first token;
+ ``window_floor`` bounds every query's sliding window from below (0 = the exact window; bounded
+ replay sets it to the start of the prompt's last window).
+ """
+
+ offset: int
+ n: int
+ table_idx: int
+ start_pos: int
+ window_floor: int = 0
+
+ @property
+ def end(self) -> int:
+ return self.start_pos + self.n
+
+
+@dataclass
+class SharedSelection:
+ """What the index-producing layers hand down the stack within ONE forward.
+
+ ``topk_rows`` are GLOBAL main-pool rows (``-1`` masked) for the current kv source, laid out
+ exactly as the attention kernel consumes them; ``cmp_counts`` is the decode-only per-row live
+ count that bounds the kernel's loop; ``candidates`` is the Hierarchical Sparse Indexer's
+ compressed-position pool (``-1`` = empty slot). Layers run in order and every producer writes
+ before its consumers read, so one slot each suffices.
+ """
+
+ source: int | None = None
+ topk_rows: torch.Tensor | None = None
+ cmp_counts: torch.Tensor | None = None
+ candidates: torch.Tensor | None = None
+
+
+@dataclass
+class DSV41AttnMetadata(BaseAttnMetadata):
+ last_indices: torch.Tensor
+ # Prefill: the encoder pass tiles the batch's new tokens; the decoder pass tiles the tokens the
+ # decoder layers process (``decoder_rows`` gathers them out of the encoder stream; None = all of
+ # it). ``decoder_pad``: some request has no decoder row this chunk, and its ``last_indices`` entry
+ # points at one placeholder row the model appends after the decoder stream (its logits are unused).
+ segments: List[PrefillSegment] | None = None
+ decoder_segments: List[PrefillSegment] | None = None
+ decoder_rows: torch.Tensor | None = None
+ decoder_pad: bool = False
+ # Decode: the whole-history full-loc snapshot (captured buffer under a replay, lazy eager copy).
+ full_snap: torch.Tensor | None = None
+ table_rows: torch.Tensor | None = None
+ window_ar: torch.Tensor | None = None
+ selection: SharedSelection = field(default_factory=SharedSelection)
+
+ def get_last_indices(self, bs: int) -> torch.Tensor:
+ return self.last_indices[:bs]
+
+ @property
+ def stage_width(self) -> int:
+ assert self.full_snap is not None, "stage_width is capture/replay-only"
+ return self.full_snap.shape[1]
+
+ def full_snapshot(self) -> torch.Tensor:
+ if self.full_snap is None:
+ assert self.table_rows is not None, "snapshot is decode-only"
+ pool = get_global_ctx().kv_cache
+ self.full_snap = pool.full_loc_map.index_select(0, self.table_rows).to(torch.int64)
+ return self.full_snap
+
+ def window_ctx(self, pos: torch.Tensor, rows: torch.Tensor):
+ """Layer-invariant decode ring context ``(window_slots, prev_window_slots,
+ window_slots_topk [B, 1, win])``; computed fresh on every call (never cached -- a capture
+ must record these gathers), off the snapshot so a concurrent allocation cannot redirect
+ an in-flight replay."""
+ snap = self.full_snapshot()
+ translate = get_global_ctx().kv_cache.translate_full_to_window
+ bs = pos.shape[0]
+ j = self.window_ar
+ assert j is not None, "window_ctx is decode-only"
+ win = j.shape[0]
+ window_slots = translate(snap[rows, pos])
+ prev_window_slots = translate(snap[rows, (pos - 1).clamp_min(0)])
+ p = pos[:, None] - ((pos[:, None] - j[None, :]) % win)
+ ws = translate(snap[rows[:, None], p.clamp(min=0)])
+ window_slots_topk = torch.where((p >= 0) & (j[None, :] <= pos[:, None]), ws, -1).view(bs, 1, win)
+ return window_slots, prev_window_slots, window_slots_topk
+
+ def private_window_ctx(self, pos: torch.Tensor, rows: torch.Tensor):
+ """The private-ring counterpart of ``window_ctx`` for request-private window layers:
+ ``(window_slots [B], window_slots_topk [B, 1, win])`` in the layer's ring, where a request's
+ row holds positions by ``pos % win`` (candidate ``j`` is the position congruent to ``j``)."""
+ assert self.table_rows is not None and self.window_ar is not None, "private_window_ctx is decode-only"
+ pool = get_global_ctx().kv_cache
+ bs = pos.shape[0]
+ j = self.window_ar
+ win = j.shape[0]
+ table = self.table_rows[rows]
+ window_slots = pool.ring_slots(table, pos)
+ p = pos[:, None] - ((pos[:, None] - j[None, :]) % win) # the position at ring index j (< 0: none yet)
+ window_slots_topk = pool.ring_slots(table[:, None], p).view(bs, 1, win)
+ return window_slots, window_slots_topk
+
+
+@dataclass
+class DSV41CaptureData:
+ full_snap: torch.Tensor
+ last_indices: torch.Tensor
+ table_rows: torch.Tensor # each batch row's page-table row (the private window rings key on it)
+
+ @classmethod
+ def create(cls, max_bs: int, width: int, device: torch.device) -> "DSV41CaptureData":
+ return cls(
+ full_snap=torch.full((max_bs, width), -1, dtype=torch.int64, device=device),
+ last_indices=torch.arange(max_bs, dtype=torch.int32, device=device),
+ table_rows=torch.zeros(max_bs, dtype=torch.int64, device=device),
+ )
+
+
+class DSV41SparseAttnBackend(BaseAttnBackend):
+ def __init__(self, config: ModelConfig):
+ self.config = config
+ self.device = get_global_ctx().kv_cache.device
+ args = config.dsv41_args
+ self.window_size = args.window_size
+ self.swa_decoder_replay: str = args.swa_decoder_replay
+ self.capture: DSV41CaptureData | None = None
+ self.capture_bs: List[int] = []
+ self.max_graph_bs = 0
+ self._window_ar = torch.arange(self.window_size, device=self.device)
+
+ @property
+ def pool(self):
+ return get_global_ctx().kv_cache
+
+ # ----- generic contract -------------------------------------------------------------
+ def forward(self, q, k, v, layer_id, batch, attn_spec: AttentionSpec | None = None):
+ raise NotImplementedError("DSV41 attention is driven per-tier from the model module; use DSV41SparseAttnBackend.attend().")
+
+ def prepare_metadata(self, batch: Batch) -> None:
+ if not batch.is_decode:
+ batch.attn_metadata = self._prefill_metadata(batch)
+ return
+ last = torch.tensor([r.extend_len for r in batch.padded_reqs], dtype=torch.int32, device=self.device).cumsum_(0) - 1
+ batch.attn_metadata = DSV41AttnMetadata(last_indices=last, table_rows=self._table_rows(batch), window_ar=self._window_ar)
+
+ def _prefill_metadata(self, batch: Batch) -> DSV41AttnMetadata:
+ """The encoder pass runs every request's new tokens ``[cached_len, device_len)``. Exact mode runs
+ the decoder on the same stream. Decoder SWA Bounded Replay runs it on the new tokens inside the
+ prompt's last window ``[L - W, L)`` (``L = prompt_len``) with every query's window floored at
+ ``L - W``: the chunks of a chunked prefill each take their part of that window (earlier rows
+ come from the request's private ring), and a chunk that ends before it runs no decoder row."""
+ segments: List[PrefillSegment] = []
+ off = 0
+ for r in batch.reqs:
+ segments.append(PrefillSegment(off, r.extend_len, r.table_idx, r.cached_len))
+ off += r.extend_len
+ if self.swa_decoder_replay == "exact":
+ last = torch.tensor([s.n for s in segments], dtype=torch.int32, device=self.device).cumsum_(0) - 1
+ return DSV41AttnMetadata(last_indices=last, segments=segments, decoder_segments=segments)
+ win = self.window_size
+ dec: List[PrefillSegment] = []
+ rows: list[torch.Tensor] = []
+ last: list[int] = []
+ doff = 0
+ for r, s in zip(batch.reqs, segments):
+ floor = max(0, r.prompt_len - win)
+ lo = max(s.start_pos, floor)
+ n = max(0, s.end - lo)
+ if n:
+ # the decoder's window KV goes to the request's private rings, never to radix-shared pages
+ dec.append(PrefillSegment(doff, n, s.table_idx, lo, window_floor=floor))
+ rows.append(torch.arange(s.offset + lo - s.start_pos, s.offset + s.n))
+ doff += n
+ last.append(doff - 1 if n else -1)
+ pad = -1 in last
+ last_indices = torch.tensor([doff if i < 0 else i for i in last], dtype=torch.int32, device=self.device)
+ decoder_rows = torch.cat(rows).to(self.device, non_blocking=True) if rows else torch.empty(0, dtype=torch.int64, device=self.device)
+ return DSV41AttnMetadata(last_indices=last_indices, segments=segments, decoder_segments=dec, decoder_rows=decoder_rows, decoder_pad=pad)
+
+ def init_capture_graph(self, max_seq_len: int, bs_list: List[int]) -> None:
+ assert self.capture is None, "Capture already initialized."
+ self.max_graph_bs = max(bs_list)
+ self.capture = DSV41CaptureData.create(self.max_graph_bs, max_seq_len, self.device)
+ self.capture_bs = sorted(bs_list)
+
+ def prepare_for_capture(self, batch: Batch) -> None:
+ bs = batch.size
+ rows = torch.full((bs,), batch.padded_reqs[0].table_idx, dtype=torch.int64, device=self.device)
+ self._point_to_capture(batch, bs, rows)
+
+ def prepare_for_replay(self, batch: Batch) -> None:
+ self._point_to_capture(batch, batch.padded_size, self._table_rows(batch))
+
+ def _table_rows(self, batch: Batch) -> torch.Tensor:
+ assert batch.active_table_idx is not None, "decode batch is missing its page-table rows"
+ return batch.active_table_idx.to(torch.int64)
+
+ def _point_to_capture(self, batch: Batch, bs: int, rows_ti: torch.Tensor) -> None:
+ assert self.capture is not None and bs <= self.max_graph_bs
+ cap = self.capture
+ src = self.pool.full_loc_map.index_select(0, rows_ti)
+ w = min(src.shape[1], cap.full_snap.shape[1])
+ cap.full_snap[:bs, :w].copy_(src[:bs, :w])
+ cap.table_rows[:bs].copy_(rows_ti[:bs])
+ batch.attn_metadata = DSV41AttnMetadata(
+ last_indices=cap.last_indices[:bs], full_snap=cap.full_snap[:bs], table_rows=cap.table_rows[:bs], window_ar=self._window_ar,
+ )
+
+ # ----- metadata / selection access -------------------------------------------------
+ @property
+ def metadata(self) -> DSV41AttnMetadata:
+ md = get_global_ctx().batch.attn_metadata
+ assert isinstance(md, DSV41AttnMetadata)
+ return md
+
+ @property
+ def selection(self) -> SharedSelection:
+ return self.metadata.selection
+
+ def snapshot(self) -> torch.Tensor:
+ return self.metadata.full_snapshot()
+
+ # ----- window tier -------------------------------------------------------------------
+ def window_slots_of(self, ti: int, lo: int, hi: int) -> torch.Tensor:
+ """SHARED window slots of positions ``[lo, hi)`` off the request's LIVE full locs (slid-out -> -1)."""
+ return self.pool.translate_full_to_window(self.pool.full_loc_map[ti, lo:hi])
+
+ def ring_slots_of(self, ti: int, lo: int, hi: int) -> torch.Tensor:
+ """PRIVATE-ring slots of positions ``[lo, hi)`` of page-table row ``ti`` (``hi - lo <= win``)."""
+ assert hi - lo <= self.window_size, f"a private ring holds one window, not {hi - lo} positions"
+ return self.pool.ring_slots(torch.tensor(ti, device=self.device), torch.arange(lo, hi, device=self.device))
+
+ def layer_window_slots_of(self, layer_id: int, ti: int, lo: int, hi: int) -> torch.Tensor:
+ """Where ``layer_id`` keeps the window KV of positions ``[lo, hi)``: its private ring or the shared pool."""
+ if self.pool.is_private_window(layer_id):
+ return self.ring_slots_of(ti, lo, hi)
+ return self.window_slots_of(ti, lo, hi)
+
+ def store_window(self, kv: torch.Tensor, layer_id: int, window_slots: torch.Tensor) -> None:
+ self.pool.store_window(kv, layer_id, window_slots)
+
+ def window_topk_prefill(self, seg: PrefillSegment, layer_id: int | None = None) -> torch.Tensor:
+ """Per-query window candidates for a prefill segment as GLOBAL window slots ``[1, n, win]``
+ (``-1`` where empty) in the tier ``layer_id`` reads (shared pool by default). Query at
+ absolute ``p`` sees ``[max(floor, first_retained, p - win + 1), p]``."""
+ win, device = self.window_size, self.device
+ lo = max(0, seg.start_pos - win + 1, seg.window_floor)
+ private = layer_id is not None and self.pool.is_private_window(layer_id)
+ ws_pool = (self.ring_slots_of if private else self.window_slots_of)(seg.table_idx, lo, seg.end) # [end - lo]
+ abs_p = seg.start_pos + torch.arange(seg.n, device=device).unsqueeze(1)
+ cand = (abs_p - win + 1).clamp(min=lo) + torch.arange(win, device=device)
+ cols = torch.where(cand > abs_p, -1, cand - lo)
+ g = ws_pool[cols.clamp_min(0)]
+ return torch.where(cols < 0, -1, g).unsqueeze(0)
+
+ # ----- compressed tiers (per kv source) ----------------------------------------------
+ def source_of(self, layer_id: int) -> int:
+ return self.pool.source_of(layer_id)
+
+ def compressed_rows_of(self, ti: int, group_starts: torch.Tensor, ratio: int) -> torch.Tensor:
+ """Main / index rows of the compressed groups whose ABSOLUTE first positions are
+ ``group_starts``, off the request's LIVE full locs (a page is ratio-divisible, so every
+ position of a group shares one row)."""
+ return self.pool.cmp_rows(self.pool.full_loc_map[ti, group_starts], ratio)
+
+ def locs_prefill(self, ti: int, end: int) -> torch.Tensor:
+ """``[1, end]`` int32: the request's live full locs for positions ``[0, end)`` -- what the indexer
+ derives compressed rows from (``loc // ratio``) for the positions its queries may see."""
+ return self.pool.full_loc_map[ti : ti + 1, :end]
+
+ def store_main(self, latent: torch.Tensor, source: int, rows: torch.Tensor) -> None:
+ self.pool.store_main(latent, source, rows)
+
+ def store_index(self, k: torch.Tensor, source: int, rows: torch.Tensor) -> None:
+ self.pool.store_index(k, source, rows)
+
+ def decode_store_rows(self, rows: torch.Tensor, pos: torch.Tensor, ratio: int, source: int, completed: torch.Tensor) -> torch.Tensor:
+ """Per-row decode destination: the completed group's arithmetic row, or the row's OWN scratch
+ row when this step did not complete a group (graph-safe, collision-free masked store)."""
+ row_of_group = self.pool.cmp_rows(self.snapshot()[rows, pos], ratio)
+ scratch = rows + self.pool.scratch_base[source]
+ return torch.where(completed, row_of_group, scratch)
+
+ # ----- compress-state ring (ratio > 1 sources) ---------------------------------------
+ def ring_page_base(self, window_slots: torch.Tensor, ring_size: int) -> torch.Tensor:
+ return torch.div(window_slots, self.window_size, rounding_mode="floor") * ring_size
+
+ def carry_state_loc(self, window_slot: int, ring_size: int) -> torch.Tensor:
+ base = (window_slot // self.window_size) * ring_size
+ return torch.arange(base, base + ring_size, device=self.device, dtype=torch.int64)
+
+ def read_carry(self, source: int, window_slot: int) -> torch.Tensor:
+ """The ``[ring_size, 2 * head_dim]`` carry block at ``window_slot``'s page (kv | score)."""
+ ring = self.pool.state_ring[source]
+ return ring.get(self.carry_state_loc(window_slot, ring.ring_size))
+
+ def write_carry(self, source: int, window_slot: int, kv_score: torch.Tensor) -> None:
+ ring = self.pool.state_ring[source]
+ ring.set(self.carry_state_loc(window_slot, ring.ring_size), kv_score)
+
+ def read_carry_blocks(self, source: int, window_slots: torch.Tensor) -> torch.Tensor:
+ ring = self.pool.state_ring[source]
+ return ring.get_blocks(self.ring_page_base(window_slots, ring.ring_size))
+
+ def write_carry_blocks(self, source: int, window_slots: torch.Tensor, blocks: torch.Tensor) -> None:
+ ring = self.pool.state_ring[source]
+ ring.set_blocks(self.ring_page_base(window_slots, ring.ring_size), blocks)
+
+ def write_boundary_carries(self, source: int, *, lo: int, hi: int, window_slots: torch.Tensor) -> None:
+ """Persist the compressor carry at every window-page boundary ``B`` in ``(lo, hi]`` so a
+ page-aligned radix match can resume by value. A page holds whole groups (``P % ratio == 0``),
+ so the carry at a boundary is the empty group -- the reset block is written so a resume never
+ reads a stale one. ``window_slots`` is indexed by ``pos - lo``. One batched ring write, no host syncs."""
+ ring = self.pool.state_ring.get(source)
+ if ring is None:
+ return
+ P = self.window_size
+ first = (lo // P + 1) * P
+ if first > hi:
+ return
+ bounds = torch.arange(first, hi + 1, P, device=self.device)
+ page_slots = window_slots[bounds - 1 - lo].to(torch.int64) # the last slot of each page
+ locs = (torch.div(page_slots, P, rounding_mode="floor") * ring.ring_size)[:, None] + torch.arange(ring.ring_size, device=self.device)
+ empty = torch.cat([
+ torch.zeros(ring.ring_size, ring.item_size, dtype=torch.float32, device=self.device),
+ torch.full((ring.ring_size, ring.item_size), float("-inf"), dtype=torch.float32, device=self.device),
+ ], dim=-1)
+ ring.set(locs.flatten(), empty.repeat(bounds.numel(), 1))
+
+ # ----- indexer ----------------------------------------------------------------------
+ def indexer_logits(
+ self, q: torch.Tensor, weights: torch.Tensor, source: int, locs: torch.Tensor, ratio: int, live: torch.Tensor,
+ T: int | None = None, candidates: torch.Tensor | None = None,
+ ) -> torch.Tensor:
+ """``[B, S, T]`` full-range scores (live columns written) or ``[B, S, NC]`` candidate scores; the
+ row of compressed position ``t`` is ``locs[b, t * ratio] // ratio``."""
+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed
+
+ return indexer_logits_packed(q, weights, self.pool.idx_pool[source], self.pool.idx_fmt, locs, ratio, live, T=T, candidates=candidates)
+
+ @staticmethod
+ def select_topk(scores: torch.Tensor, live: torch.Tensor, topk: int) -> torch.Tensor:
+ """Exact causal top-k over ``[B, S, T]`` full-range scores (columns at or past ``live`` unread):
+ compressed positions ``[B, S, topk]`` int32 in ascending order, ``-1`` in the tail for picks
+ that do not exist (fewer live or finite positions than ``topk``). Ties: lowest position wins."""
+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk
+
+ b, s, t = scores.shape
+ return dsv41_topk(scores.reshape(b * s, t), live.reshape(b * s), topk).view(b, s, topk)
+
+ @staticmethod
+ def select_topk_in_candidates(scores: torch.Tensor, candidates: torch.Tensor, topk: int) -> torch.Tensor:
+ """Top-k over ``[B, S, NC]`` candidate-aligned scores -> the picked compressed positions
+ ``[B, S, topk]`` int32. The candidate list is a sorted valid prefix (``-1`` tail) and empty /
+ unreachable slots score ``-inf``, so the ascending winning slots map to ascending positions."""
+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk
+
+ b, s, nc = scores.shape
+ live = torch.full((b * s,), nc, dtype=torch.int32, device=scores.device)
+ slots = dsv41_topk(scores.reshape(b * s, nc), live, topk).view(b, s, topk)
+ pos = candidates.gather(-1, slots.clamp_min(0).to(torch.int64)).to(torch.int32)
+ return torch.where(slots < 0, -1, pos)
+
+ @staticmethod
+ def select_candidate_blocks(scores: torch.Tensor, live: torch.Tensor, topk_blocks: int, block_size: int) -> torch.Tensor:
+ """Level one of the Hierarchical Sparse Indexer: the ``topk_blocks`` best live ``block_size``
+ blocks by their best position, the block holding the newest live position always kept,
+ expanded to the compact candidate list ``[B, S, topk_blocks * block_size]`` int32 of
+ compressed positions (ascending, ``-1`` tail). Mirrors the reference ``select_candidate_blocks``
+ (which returns the equivalent boolean mask)."""
+ from freetoken.kernel.triton.dsv41.topk import dsv41_candidate_blocks
+
+ b, s, t = scores.shape
+ return dsv41_candidate_blocks(scores.reshape(b * s, t), live.reshape(b * s), topk_blocks, block_size).view(b, s, -1)
+
+ @staticmethod
+ def positions_to_rows(positions: torch.Tensor, locs: torch.Tensor, ratio: int) -> torch.Tensor:
+ """Compressed positions ``[B, S, K]`` (``-1`` masked) -> GLOBAL main rows ``locs[b, p * ratio] // ratio``."""
+ b = positions.shape[0]
+ idx = (positions.to(torch.int64) * ratio).clamp_min(0).flatten(1)
+ full = locs.to(torch.int64).gather(1, idx.clamp(max=locs.shape[1] - 1)).view_as(positions)
+ rows = torch.div(full, ratio, rounding_mode="floor")
+ return torch.where((positions < 0) | (full < 0), -1, rows)
+
+ # ----- attention -----------------------------------------------------------------------
+ def attend(
+ self, q: torch.Tensor, layer_id: int, topk_idxs: torch.Tensor, n_window: int, attn_sink: torch.Tensor,
+ softmax_scale: float, cmp_counts: torch.Tensor | None = None,
+ ) -> torch.Tensor:
+ """Paged sparse attention over ``[window | compressed]`` global rows; the compressed half reads
+ the layer's kv source (window-only layers pass ``n_window == topk``)."""
+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed
+
+ pool, args = self.pool, self.config.dsv41_args
+ src = args.roles[layer_id].kv_source
+ cmp = pool.main_pool[src] if src is not None else pool.main_pool[args.backbone_kv_sources[0]]
+ return sparse_attn_packed(
+ q, pool.window_pool[layer_id], pool.win_fmt, cmp, pool.main_fmt, attn_sink,
+ topk_idxs, n_window, softmax_scale, cmp_counts=cmp_counts,
+ )
+
+
+__all__ = ["DSV41SparseAttnBackend", "DSV41AttnMetadata", "PrefillSegment", "SharedSelection"]
diff --git a/python/freetoken/core.py b/python/freetoken/core.py
index fd691d18..45a763c0 100644
--- a/python/freetoken/core.py
+++ b/python/freetoken/core.py
@@ -43,6 +43,8 @@ class Req:
mm_items: list | None = None
mrope_positions_full: torch.Tensor | None = None # [3, prompt_len] int32, CPU
mrope_delta: int = 0
+ # The whole prompt's length; 0 = this request's input_ids (a ChunkedReq's end at its chunk).
+ prompt_len: int = 0
# --- hybrid-radix (GDN linear-state) per-request slots; None for non-hybrid models or
# until allocated from LinearStatePool. Set by the scheduler (P2). ---
@@ -69,6 +71,7 @@ class Req:
assert self.input_ids.is_cpu
self.device_len = len(self.input_ids)
self.max_device_len = len(self.input_ids) + self.output_len
+ self.prompt_len = self.prompt_len or self.device_len
assert 0 <= self.cached_len < self.device_len <= self.max_device_len
self._alloc_ids_buf()
diff --git a/python/freetoken/engine/cache_budget.py b/python/freetoken/engine/cache_budget.py
index d522be49..b5f07ee8 100644
--- a/python/freetoken/engine/cache_budget.py
+++ b/python/freetoken/engine/cache_budget.py
@@ -24,7 +24,7 @@ def expert_bytes_per_slot(sources: dict[str, "list[torch.Tensor]"]) -> int:
# marlin/b12x gate_up/down alpha scales are fixed [L*E] residency (do not scale
# with cache_size), so they are intentionally excluded from the per-slot growth term.
# tensor[0].numel() is the per-row element count (one expert slot); see the matching
- # slot-byte idiom in kvcache/linear_state_pool.py and kvcache/dsv4_paged_pool.py.
+ # slot-byte idiom in kvcache/linear_state_pool.py and kvcache/dsv4/v4_pool.py.
return sum(t[0][0].numel() * t[0].element_size() for t in sources.values())
diff --git a/python/freetoken/engine/config.py b/python/freetoken/engine/config.py
index 3e38cb21..b6fa8b89 100644
--- a/python/freetoken/engine/config.py
+++ b/python/freetoken/engine/config.py
@@ -34,6 +34,9 @@ class EngineConfig:
quant_backend: str | None = None
# PLE table backend: "disk" (default) reads rows from the checkpoint files per fill, "pinned" preloads the table into page-locked host RAM.
ple_backend: str = "disk"
+ # DeepSeek-V4.1 Decoder SWA Bounded Replay: "bounded" (default) runs the decoder layers on each
+ # prompt's last window; "exact" runs them on every token.
+ swa_decoder_replay: str = "bounded"
# Expert-bank host load (--expert-load): auto|serial|parallel. "auto" reads scattered
# experts in parallel but falls back to serial when free RAM can't cover the banks + the
# parallel reader's extra (non-reclaimable) whole-shard buffer; "serial" forces the
diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.py
index e5ccf710..bf3960ab 100644
--- a/python/freetoken/engine/engine.py
+++ b/python/freetoken/engine/engine.py
@@ -129,6 +129,8 @@ def _resolve_auto_attention_backend(required: frozenset[AttnType]) -> str:
candidates: list[tuple[str, bool]] = []
if AttnType.DSV4 in required:
candidates.append(("dsv4_sparse", True))
+ if AttnType.DSV41 in required:
+ candidates.append(("dsv41_sparse", True))
if required & {AttnType.MLA, AttnType.DSA}:
candidates.append(("dsa", True))
if AttnType.BSA in required:
@@ -812,6 +814,7 @@ class Engine:
# Decode batches never exceed max_running_req, but CUDA-graph padding can
# round a batch up to the largest captured size; cover both.
max_tokens = max(config.max_running_req, config.cuda_graph_max_bs or 0, 1)
+ method = sample.quant_method
executor = CpuMoeExecutor(
cache,
top_k=sample.top_k,
@@ -823,7 +826,8 @@ class Engine:
swiglu_alpha=float(sample.alpha),
swiglu_limit=sample.limit,
# FIXME: the None branch serves GGUF q4_0 banks, which have no quant method yet; drop it once GGUF joins the quant path
- fmt=sample.quant_method.cpu_format if sample.quant_method is not None else None,
+ fmt=method.cpu_format if method is not None else None,
+ act_block=method.cfg.act_block if method is not None else None,
)
cache.set_cpu_executor(executor)
self.cpu_moe_executor = executor
@@ -1200,6 +1204,39 @@ def _resolve_cache_type(has_linear_attention: bool, requested: str) -> str:
return requested
+def _adjust_dsv41_config(config: EngineConfig, override) -> None:
+ """DeepSeek-V4.1 config reconciliation, the DSV4 policy with the replay knob: sync the
+ runtime into ``dsv41_args``, page_size = the window page P, radix -> swa_radix, decode graph
+ batches <= max_running_req. Unlike DSV4 the prefill chunk keeps ``max_extend_tokens`` (whole
+ window pages): at a 1M ceiling the window pool alone would admit ~40K-token chunks whose
+ activations (64 x 512 latent queries per token) do not fit next to the expert cache."""
+ model_config = config.model_config
+ args = model_config.dsv41_args
+ args.max_seq_len = config.max_seq_len
+ args.max_batch_size = config.max_running_req + 1 # +1 dummy
+ args.swa_decoder_replay = config.swa_decoder_replay
+ P = args.window_size
+ override("page_size", P)
+ logger.info_rank0(
+ f"DSV41 KV pages are {P}-token window pages; page_size set to {P}; SWA decoder replay: {args.swa_decoder_replay}"
+ + ("" if args.swa_decoder_replay == "exact" else f" (prefix hits stop {P} tokens before the prompt end)")
+ )
+ if getattr(config, "cache_type", "radix") != "naive":
+ override("cache_type", "swa_radix")
+ current = getattr(config, "max_extend_tokens", None) # a SchedulerConfig field
+ if current is not None and (extend := max(P, current // P * P)) != current:
+ override("max_extend_tokens", extend)
+ mr = config.max_running_req
+ if config.cuda_graph_max_bs is not None and config.cuda_graph_max_bs > mr:
+ logger.warning_rank0(f"cuda_graph_max_bs {config.cuda_graph_max_bs} exceeds max_running_req {mr}; clamping (larger decode batches never occur).")
+ override("cuda_graph_max_bs", mr)
+ if config.cuda_graph_bs is not None:
+ kept = [bs for bs in config.cuda_graph_bs if bs <= mr]
+ if kept != list(config.cuda_graph_bs):
+ logger.warning_rank0(f"dropping cuda_graph_bs entries above max_running_req {mr}: {[bs for bs in config.cuda_graph_bs if bs > mr]}")
+ override("cuda_graph_bs", kept)
+
+
def _adjust_dsv4_config(config: EngineConfig, override) -> None:
"""DSV4 engine-config reconciliation at config-resolution time (before the pool exists).
Syncs the resolved runtime config into the opaque ``dsv4_args`` payload, sets
@@ -1477,6 +1514,7 @@ def _adjust_config(config: EngineConfig):
model_config = config.model_config
single_stream_only = getattr(model_config, "single_stream_only", False)
is_dsv4 = getattr(model_config, "dsv4_args", None) is not None
+ is_dsv41 = getattr(model_config, "dsv41_args", None) is not None
has_swa_attention = getattr(model_config, "has_swa_attention", False)
has_linear_attention = getattr(model_config, "has_linear_attention", False)
is_moe = getattr(model_config, "is_moe", False)
@@ -1520,6 +1558,8 @@ def _adjust_config(config: EngineConfig):
if is_dsv4:
_adjust_dsv4_config(config, override)
+ if is_dsv41:
+ _adjust_dsv41_config(config, override)
if has_swa_attention:
# Both SWA cache paths use the global-paged swa pool (page_size==1 only for now).
@@ -1806,7 +1846,7 @@ def _adjust_config(config: EngineConfig):
# DSV4 is exempt: it sizes its own table from the resolved max_seq_len (_adjust_dsv4_config).
rotary = getattr(model_config, "rotary_config", None)
seq_override = getattr(config, "max_seq_len_override", None)
- if seq_override is not None and rotary is not None and not is_dsv4:
+ if seq_override is not None and rotary is not None and not (is_dsv4 or is_dsv41):
if seq_override > rotary.table_positions:
raise ValueError(
f"--max-seq-len-override {seq_override} exceeds the model's "
diff --git a/python/freetoken/kernel/aot_models.py b/python/freetoken/kernel/aot_models.py
index a8ef85d4..ecd1f850 100644
--- a/python/freetoken/kernel/aot_models.py
+++ b/python/freetoken/kernel/aot_models.py
@@ -332,6 +332,17 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (
expert_formats=("ds_fp4",),
embed_indexing=False, # plain nn.Embedding
),
+ AotModel(
+ # DSV41 pools are packed byte rows written by the dsv41 pack kernel, not store_cache;
+ # the routed experts share DSV4's ds_fp4 bank layout (hidden 5120, intermediate 2304).
+ name="deepseek-ai/DeepSeek-V4.1-Flash",
+ architecture="DeepseekV41ForCausalLM",
+ hidden_size=5120,
+ kv_groups=(),
+ top_k=6,
+ moe_intermediate_size=2304,
+ expert_formats=("ds_fp4",),
+ ),
# ---- dense checkpoints (store/index only, no expert banks) ----
AotModel(
name="Qwen/Qwen3.6-27B",
diff --git a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp
index fb6a33f4..86765013 100644
--- a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp
+++ b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp
@@ -1063,11 +1063,12 @@ inline void deinterleave_bf16_f32(const bf16_t* src, float* even, float* odd, in
}
}
-// DeepSeek-V4 activation FP8 round-trip (bf16 in/out): per 128-block,
-// s = 2^ceil(log2(max(|x|,1e-4)/448)); y = round_e4m3(clamp(x/s,+-448)) * s.
-void fp8_roundtrip_bf16(const bf16_t* src, bf16_t* dst, int K) {
- for (int b0 = 0; b0 < K; b0 += 128) {
- const int b1 = std::min(K, b0 + 128);
+// DeepSeek-V4 / V4.1 activation FP8 round-trip (bf16 in/out): per `block` (the checkpoint's
+// fp8 block: 128 on V4, 32 on V4.1), s = 2^ceil(log2(max(|x|,1e-4)/448));
+// y = round_e4m3(clamp(x/s,+-448)) * s.
+void fp8_roundtrip_bf16(const bf16_t* src, bf16_t* dst, int K, int block) {
+ for (int b0 = 0; b0 < K; b0 += block) {
+ const int b1 = std::min(K, b0 + block);
float amax = 1e-4f;
for (int i = b0; i < b1; ++i) amax = std::max(amax, std::fabs(bf16_to_f32(src[i])));
const float s = std::ldexp(1.0f, ceil_log2_pos(amax * (1.0f / 448.0f)));
@@ -1275,6 +1276,8 @@ struct CpuMoeExecutor {
// it to a captured GPU elementwise kernel removes it while keeping the official
// W4A8 numerics bit-exact. Set via set_input_prequant (see cpu_executor.py).
bool input_prequant = false;
+ // ds_fp4 activation round-trip block (the checkpoint's fp8 block); set via set_act_block.
+ int act_block = 128;
// Q4_0 packed-row byte strides (H/32*18 for gate_up over K=H, I/32*18 for down over K=I).
int q4_gu_row_bytes = 0, q4_dn_row_bytes = 0;
float e2m1_lut[16];
@@ -1817,7 +1820,7 @@ struct CpuMoeExecutor {
gas_scratch.data() + (size_t)r * (I / 32));
return;
}
- if (fmt == WF_DSFP4) fp8_roundtrip_bf16(g, g, I);
+ if (fmt == WF_DSFP4) fp8_roundtrip_bf16(g, g, I, act_block);
float* ge = ge_scratch.data() + (size_t)r * (I / 2);
float* go = go_scratch.data() + (size_t)r * (I / 2);
deinterleave_bf16_f32(g, ge, go, I);
@@ -1957,7 +1960,7 @@ struct CpuMoeExecutor {
const bf16_t* src = t->x + (size_t)tok * H;
if (ds) { // DSV4 FP8-round-trips the input before the gate_up GEMV
bf16_t* xq = xq_scratch.data() + (size_t)tok * H;
- fp8_roundtrip_bf16(src, xq, H);
+ fp8_roundtrip_bf16(src, xq, H, act_block);
src = xq;
}
float* xe = xe_scratch.data() + (size_t)tok * (H / 2);
@@ -2185,6 +2188,9 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
.def("set_input_prequant",
[](CpuMoeExecutor& e, bool v) { e.input_prequant = v; },
py::arg("value"))
+ .def("set_act_block",
+ [](CpuMoeExecutor& e, int v) { e.act_block = v; },
+ py::arg("value"))
.def("isa_name", &CpuMoeExecutor::isa_name)
.def("set_spin_wait",
[](CpuMoeExecutor& e, bool v) { e.spin_wait.store(v, std::memory_order_relaxed); },
diff --git a/python/freetoken/kernel/triton/dsv4/fp8_linear.py b/python/freetoken/kernel/triton/dsv4/fp8_linear.py
index 47e3c9f5..fb17a351 100644
--- a/python/freetoken/kernel/triton/dsv4/fp8_linear.py
+++ b/python/freetoken/kernel/triton/dsv4/fp8_linear.py
@@ -1,23 +1,25 @@
-"""Block-scaled FP8 (e4m3) linear for DeepSeek-V4, matching the reference numerics.
+"""Block-scaled FP8 (e4m3) linear for DeepSeek-V4 / V4.1, matching the reference numerics.
The reference (``inference/model.py`` ``linear`` + ``inference/kernel.py``
-``act_quant``/``fp8_gemm``) quantizes the *activation* to FP8 with a per-128 block
+``act_quant``/``fp8_gemm``) quantizes the *activation* to FP8 with a per-``block``
power-of-two (ue8m0) scale, then runs an FP8xFP8 block-scaled GEMM against the FP8
-weight (which carries its own 128x128 ue8m0 block scale). Both operands' scales are
-applied per 128-K block to a separate FP32 accumulator. This module reproduces that:
+weight (which carries its own ``block x block`` ue8m0 scale). Both operands' scales are
+applied per ``block``-K slab to a separate FP32 accumulator. This module reproduces that:
- ``y = fp8_gemm(act_quant(x, 128, ue8m0), weight_fp8, weight_scale_e8m0)``
+ ``y = fp8_gemm(act_quant(x, block, ue8m0), weight_fp8, weight_scale_e8m0)``
+
+``block`` is the checkpoint's ``weight_block_size``: 128 for DeepSeek-V4, 32 for V4.1.
``act_quant`` (reference): per block ``s = 2**ceil(log2(max(|x|,1e-4)/448))`` (exact
via IEEE bit ops -> matches ``fast_round_scale``), ``x_fp8 = round_e4m3(clamp(x/s,
+-448))``, scale stored e8m0. The GEMM accumulates ``sum_k (A_fp8 @ B_fp8) * s_a * s_b``
-per 128-K block in FP32.
+per ``block``-K slab in FP32.
Also provides ``act_quant_fp8_inplace`` -- the fused FP8 quant+dequant round-trip the
reference applies in-place to the window / compressor KV (``act_quant(..., 64, ...,
inplace=True)``), returning BF16.
-Assumes ``K % 128 == 0`` and ``N % 128 == 0`` (true for every DeepSeek-V4 projection).
+Assumes ``K % block == 0`` and ``N % block == 0`` (true for every DeepSeek-V4 / V4.1 projection).
"""
from __future__ import annotations
@@ -218,8 +220,8 @@ def fp4_act_quant_inplace(x: torch.Tensor, block: int = 32) -> torch.Tensor:
def _fp8_act_gemm_kernel(
a_ptr, # [M, K] float8_e4m3fn (quantized activation)
w_ptr, # [N, K] float8_e4m3fn
- sa_ptr, # [M, K//128] uint8 (e8m0 act codes)
- sb_ptr, # [N//128, K//128] uint8 (e8m0 weight codes)
+ sa_ptr, # [M, K//BLOCK_K] uint8 (e8m0 act codes)
+ sb_ptr, # [N//BLOCK_K, K//BLOCK_K] uint8 (e8m0 weight codes; square BLOCK_K blocks)
c_ptr, # [M, N] compute dtype
M, N, K,
stride_am, stride_ak, stride_wn, stride_wk,
@@ -228,6 +230,11 @@ def _fp8_act_gemm_kernel(
BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr,
compute_type: tl.constexpr,
):
+ """``BLOCK_K`` is the weight's square block edge, so each K step maps to exactly one scale
+ column. When the N tile IS one scale row (``BLOCK_N == BLOCK_K``, the DeepSeek-V4 block-128
+ case) the weight scale is a scalar per step and the N masks vanish -- the original kernel;
+ otherwise (block 32) the tile spans ``BLOCK_N // BLOCK_K`` scale rows and N is masked."""
+ SCALAR_SCALE: tl.constexpr = BLOCK_N == BLOCK_K
pid_m = tl.program_id(0)
pid_n = tl.program_id(1)
offs_m = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)
@@ -238,112 +245,217 @@ def _fp8_act_gemm_kernel(
w_ptrs = w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :] * stride_wk
acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)
num_k = tl.cdiv(K, BLOCK_K)
- for k in range(num_k):
- a = tl.load(a_ptrs, mask=m_mask[:, None], other=0.0)
- w = tl.load(w_ptrs)
- if e4m3_native_cx():
- p = tl.dot(a, tl.trans(w), out_dtype=tl.float32)
- else:
- # bf16 dot on the same e4m3 grid: operands exact in bf16, fp32 acc
- p = tl.dot(a, tl.trans(e4m3_u8_to_f32(w).to(tl.bfloat16)), out_dtype=tl.float32)
- sa_code = tl.load(sa_ptr + offs_m * stride_sam + k * stride_sak, mask=m_mask, other=0)
- sca = tl.exp2(sa_code.to(tl.float32) - 127.0) # [BLOCK_M]
- sb_code = tl.load(sb_ptr + pid_n * stride_sbn + k * stride_sbk)
- scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # scalar (one 128-N block)
- acc += p * sca[:, None] * scb
- a_ptrs += BLOCK_K * stride_ak
- w_ptrs += BLOCK_K * stride_wk
- c_ptrs = c_ptr + offs_m[:, None] * stride_cm + offs_n[None, :] * stride_cn
- tl.store(c_ptrs, acc.to(compute_type), mask=m_mask[:, None])
+ if SCALAR_SCALE:
+ for k in range(num_k):
+ a = tl.load(a_ptrs, mask=m_mask[:, None], other=0.0)
+ w = tl.load(w_ptrs)
+ if e4m3_native_cx():
+ p = tl.dot(a, tl.trans(w), out_dtype=tl.float32)
+ else:
+ # bf16 dot on the same e4m3 grid: operands exact in bf16, fp32 acc
+ p = tl.dot(a, tl.trans(e4m3_u8_to_f32(w).to(tl.bfloat16)), out_dtype=tl.float32)
+ sa_code = tl.load(sa_ptr + offs_m * stride_sam + k * stride_sak, mask=m_mask, other=0)
+ sca = tl.exp2(sa_code.to(tl.float32) - 127.0) # [BLOCK_M]
+ sb_code = tl.load(sb_ptr + pid_n * stride_sbn + k * stride_sbk)
+ scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # scalar (one N block)
+ acc += p * sca[:, None] * scb
+ a_ptrs += BLOCK_K * stride_ak
+ w_ptrs += BLOCK_K * stride_wk
+ c_ptrs = c_ptr + offs_m[:, None] * stride_cm + offs_n[None, :] * stride_cn
+ tl.store(c_ptrs, acc.to(compute_type), mask=m_mask[:, None])
+ else:
+ n_mask = offs_n < N # N is a multiple of BLOCK_K, not necessarily of BLOCK_N
+ sn = offs_n // BLOCK_K
+ for k in range(num_k):
+ a = tl.load(a_ptrs, mask=m_mask[:, None], other=0.0)
+ w = tl.load(w_ptrs, mask=n_mask[:, None], other=0.0)
+ if e4m3_native_cx():
+ p = tl.dot(a, tl.trans(w), out_dtype=tl.float32)
+ else:
+ p = tl.dot(a, tl.trans(e4m3_u8_to_f32(w).to(tl.bfloat16)), out_dtype=tl.float32)
+ sa_code = tl.load(sa_ptr + offs_m * stride_sam + k * stride_sak, mask=m_mask, other=0)
+ sca = tl.exp2(sa_code.to(tl.float32) - 127.0) # [BLOCK_M]
+ sb_code = tl.load(sb_ptr + sn * stride_sbn + k * stride_sbk, mask=n_mask, other=0)
+ scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # [BLOCK_N]
+ acc += p * sca[:, None] * scb[None, :]
+ a_ptrs += BLOCK_K * stride_ak
+ w_ptrs += BLOCK_K * stride_wk
+ c_ptrs = c_ptr + offs_m[:, None] * stride_cm + offs_n[None, :] * stride_cn
+ tl.store(c_ptrs, acc.to(compute_type), mask=m_mask[:, None] & n_mask[None, :])
@triton.jit
-def _fp8_act_gemv_splitk_kernel(
- a_ptr, # [K] float8_e4m3fn
- sa_ptr, # [K//128] uint8 (e8m0 act codes)
- w_ptr, # [N, K] float8_e4m3fn
- sb_ptr, # [N//128, K//128] uint8 (e8m0 weight codes)
- part_ptr, # [SPLIT_K, N] fp32
- N, K,
- stride_ak, stride_wn, stride_wk, stride_sbn, stride_sbk, stride_pk, stride_pn,
- BLOCK_N: tl.constexpr, SPLIT_K: tl.constexpr,
+def _fp8_gemv_kernel(
+ x_ptr, # [M, G, K] activation in the compute dtype (bf16); QUANT_ACT: quantized in-kernel per SLAB
+ w_ptr, # [G * N, K] float8_e4m3fn (or its uint8 view on pre-Ada parts)
+ sb_ptr, # [G * N // SLAB, K // SLAB] uint8 (e8m0 weight codes)
+ out_ptr, # [M, G * N] OUT dtype (SPLIT_K == 1)
+ part_ptr, # [SPLIT_K, M, G * N] fp32 partials (SPLIT_K > 1)
+ M, N, K,
+ stride_xm, stride_xg, stride_wn, stride_wk, stride_sbn, stride_sbk, stride_om, stride_pk, stride_pm,
+ BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr, SLAB: tl.constexpr, SPLIT_K: tl.constexpr,
+ QUANT_ACT: tl.constexpr, OUT: tl.constexpr,
):
+ """Small-M (decode) block-scaled fp8 GEMV/GEMM: one pass over the weight at near HBM bandwidth.
+
+ Each program owns ``BLOCK_N`` rows and ``K // SPLIT_K`` contiguous K, walked ``BLOCK_K`` (several
+ ``SLAB``-wide scale slabs) at a time: a wide tile keeps enough bytes in flight per SM to saturate
+ the memory system, which a slab-at-a-time walk (512 B per load at the V4.1 slab of 32) cannot.
+ Every weight tile is applied to all ``M`` (<= ``BLOCK_M``) activation rows, so a decode batch
+ costs one weight read like a single token does (the tensor-core GEMM path is 3-6x slower at
+ M = 2..8: too few tiles, tiny K steps). ``QUANT_ACT`` quantizes the activation here rather than
+ by a separate ``act_quant`` launch: every program recomputes the ue8m0 scale and the e4m3 rounding
+ of the slabs it touches (the reference formula, bit-identical to ``act_quant_fp8``) from the
+ L2-resident bf16 rows, so the path is one kernel (plus a small reduce when ``SPLIT_K > 1``).
+ Without it the activation stays bf16 (W8A16: the reference's bf16 einsum over a dequantized
+ weight, e.g. ``wo_a``). The third grid axis is an independent group (block-diagonal weights:
+ group ``g`` reads ``x[:, g]`` and rows ``[g * N, (g + 1) * N)``)."""
+ tl.static_assert(BLOCK_K % SLAB == 0, "the K tile holds whole scale slabs")
+ NSLAB: tl.constexpr = BLOCK_K // SLAB
pid_n = tl.program_id(0)
pid_k = tl.program_id(1)
+ g = tl.program_id(2).to(tl.int64)
+ x_ptr += g * stride_xg
offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
n_mask = offs_n < N
- sn = offs_n // 128
+ sn = (g * N + offs_n) // SLAB
+ offs_n = g * N + offs_n
+ rows = tl.arange(0, BLOCK_M)
k_per = K // SPLIT_K
k_start = pid_k * k_per
- acc = tl.zeros((BLOCK_N,), dtype=tl.float32)
- for k0 in range(0, k_per, 128):
- offs_k = k_start + k0 + tl.arange(0, 128)
- a = tl.load(a_ptr + offs_k * stride_ak).to(tl.float32)
+ acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)
+ for k0 in range(0, k_per, BLOCK_K):
+ offs_k = k_start + k0 + tl.arange(0, BLOCK_K)
+ # --- the weight tile, dequantized per slab (read once for every activation row) ---
w_ptrs = w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :] * stride_wk
if e4m3_native_cx():
w = tl.load(w_ptrs, mask=n_mask[:, None], other=0.0).to(tl.float32)
else:
w = e4m3_u8_to_f32(tl.load(w_ptrs, mask=n_mask[:, None], other=0))
- kb = (k_start + k0) // 128
- sb_code = tl.load(sb_ptr + sn * stride_sbn + kb * stride_sbk, mask=n_mask, other=0)
- scb = tl.exp2(sb_code.to(tl.float32) - 127.0)
- sa_code = tl.load(sa_ptr + kb)
- sca = tl.exp2(sa_code.to(tl.float32) - 127.0)
- acc += tl.sum(w * a[None, :], axis=1) * scb * sca
- tl.store(part_ptr + pid_k * stride_pk + offs_n * stride_pn, acc, mask=n_mask)
+ kb = (k_start + k0) // SLAB + tl.arange(0, NSLAB)
+ sb_code = tl.load(sb_ptr + sn[:, None] * stride_sbn + kb[None, :] * stride_sbk, mask=n_mask[:, None], other=0)
+ scb = tl.exp2(sb_code.to(tl.float32) - 127.0) # [BLOCK_N, NSLAB]
+ for m in tl.static_range(BLOCK_M):
+ if m < M:
+ xs = tl.reshape(tl.load(x_ptr + m * stride_xm + offs_k).to(tl.float32), (NSLAB, SLAB))
+ if QUANT_ACT:
+ # --- reference act_quant on this row's slabs: s = 2**ceil(log2(max(|x|, 1e-4) / 448)) ---
+ amax = tl.maximum(tl.max(tl.abs(xs), axis=1), 1e-4)
+ sa = tl.exp2(_log2_ceil(amax * (1.0 / 448.0)).to(tl.float32)) # [NSLAB]
+ q = tl.clamp(xs / sa[:, None], -448.0, 448.0)
+ if e4m3_native_cx():
+ q = q.to(tl.float8e4nv).to(tl.float32)
+ else:
+ q = round_e4m3(q).to(tl.float32)
+ else:
+ sa = tl.full((NSLAB,), 1.0, tl.float32)
+ q = xs
+ a = tl.reshape(q, (BLOCK_K,))
+ slab = tl.sum(tl.reshape(w * a[None, :], (BLOCK_N, NSLAB, SLAB)), axis=2) # [BLOCK_N, NSLAB]
+ contrib = tl.sum(slab * scb * sa[None, :], axis=1) # [BLOCK_N]
+ acc = tl.where(rows[:, None] == m, acc + contrib[None, :], acc)
+ row_mask = rows < M
+ if SPLIT_K == 1:
+ tl.store(out_ptr + rows[:, None] * stride_om + offs_n[None, :], acc.to(OUT), mask=row_mask[:, None] & n_mask[None, :])
+ else:
+ tl.store(part_ptr + pid_k * stride_pk + rows[:, None] * stride_pm + offs_n[None, :], acc, mask=row_mask[:, None] & n_mask[None, :])
@triton.jit
-def _splitk_reduce_kernel(part_ptr, out_ptr, N, SPLIT_K: tl.constexpr,
- stride_pk, stride_pn, BLOCK: tl.constexpr, OUT: tl.constexpr):
+def _splitk_reduce_kernel(part_ptr, out_ptr, M, N, SPLIT_K: tl.constexpr,
+ stride_pk, stride_pm, stride_om, BLOCK: tl.constexpr, OUT: tl.constexpr):
+ m = tl.program_id(1)
offs = tl.program_id(0) * BLOCK + tl.arange(0, BLOCK)
mask = offs < N
acc = tl.zeros((BLOCK,), dtype=tl.float32)
for k in tl.static_range(SPLIT_K):
- acc += tl.load(part_ptr + k * stride_pk + offs * stride_pn, mask=mask, other=0.0)
- tl.store(out_ptr + offs, acc.to(OUT), mask=mask)
-
-
-# Swept-best decode GEMV config per (N, K) -> (BLOCK_N, SPLIT_K, num_warps).
-_DECODE_FP8_CFG = {
- (1024, 4096): (16, 16, 1), # wq_a
- (32768, 1024): (16, 1, 2), # wq_b
- (512, 4096): (16, 16, 1), # attn wkv
- (4096, 8192): (16, 8, 1), # wo_b
- (2048, 4096): (16, 8, 1), # shared w1 / w3
- (4096, 2048): (16, 8, 1), # shared w2
- (8192, 1024): (16, 2, 4), # indexer wq_b
-}
-
-
-def _decode_cfg(N: int, K: int) -> tuple[int, int, int]:
- cfg = _DECODE_FP8_CFG.get((N, K))
- if cfg is not None:
- bn, sk, nw = cfg
- return bn, max(1, min(sk, K // 128)), nw
- bn = 16
- n_tiles = triton.cdiv(N, bn)
- sk = max(1, 1536 // n_tiles)
- sk = 1 << (sk.bit_length() - 1)
- return bn, max(1, min(sk, K // 128)), 1
-
-
-def _fp8_act_gemv(a_fp8: torch.Tensor, sa: torch.Tensor, weight: torch.Tensor,
- sb: torch.Tensor, out_dtype: torch.dtype) -> torch.Tensor:
- N, K = weight.shape
- BLOCK_N, split_k, num_warps = _decode_cfg(N, K)
- n_tiles = triton.cdiv(N, BLOCK_N)
- part = torch.empty((split_k, N), dtype=torch.float32, device=a_fp8.device)
- _fp8_act_gemv_splitk_kernel[(n_tiles, split_k)](
- a_fp8, sa, weight, sb, part, N, K,
- a_fp8.stride(0), weight.stride(0), weight.stride(1),
- sb.stride(0), sb.stride(1), part.stride(0), part.stride(1),
- BLOCK_N=BLOCK_N, SPLIT_K=split_k, num_warps=num_warps,
+ acc += tl.load(part_ptr + k * stride_pk + m * stride_pm + offs, mask=mask, other=0.0)
+ tl.store(out_ptr + m * stride_om + offs, acc.to(OUT), mask=mask)
+
+
+_GEMV_BLOCK_N = 32
+_GEMV_NARROW_BLOCK_N = 8
+def _gemv_cfg(N: int, K: int, slab: int, sm_count: int) -> tuple[int, int, int, int]:
+ """Choose whole scale tiles and enough independent programs to cover the device."""
+ block_k = next((bk for bk in (512, 256, 128) if bk >= slab and K % bk == 0), slab)
+ if triton.cdiv(N, _GEMV_BLOCK_N) >= sm_count:
+ return _GEMV_BLOCK_N, block_k, 1, 4
+ n_tiles = triton.cdiv(N, _GEMV_NARROW_BLOCK_N)
+ split = 1
+ while n_tiles * split < sm_count and (K // block_k) % (split * 2) == 0:
+ split *= 2
+ return _GEMV_NARROW_BLOCK_N, block_k, split, 4
+
+
+GEMV_MAX_M = 4 # bound the unrolled row loop's register footprint
+
+
+def _fp8_gemv(
+ x: torch.Tensor, weight: torch.Tensor, sb: torch.Tensor, out_dtype: torch.dtype, block: int, *, groups: int = 1, quant_act: bool = True,
+) -> torch.Tensor:
+ """``x [M, groups, K]`` bf16 x block-diagonal ``weight [groups * N, K]`` fp8 -> ``[M, groups * N]``
+ in ``out_dtype`` (``M <= GEMV_MAX_M``). ``quant_act`` applies the reference fp8 activation quant
+ (W8A8); off, the activation stays bf16 (W8A16). ``groups == 1`` is the plain GEMV."""
+ GN, K = weight.shape
+ N = GN // groups
+ x = x.reshape(-1, groups, K)
+ M = x.shape[0]
+ assert 1 <= M <= GEMV_MAX_M, M
+ block_m = triton.next_power_of_2(M)
+ block_n, block_k, split_k, num_warps = _gemv_cfg(GN, K, block, torch.cuda.get_device_properties(x.device).multi_processor_count)
+ out = torch.empty((M, GN), dtype=out_dtype, device=x.device)
+ part = out if split_k == 1 else torch.empty((split_k, M, GN), dtype=torch.float32, device=x.device)
+ _fp8_gemv_kernel[(triton.cdiv(N, block_n), split_k, groups)](
+ x, weight, sb, out, part, M, N, K,
+ x.stride(0), x.stride(1), weight.stride(0), weight.stride(1), sb.stride(0), sb.stride(1), out.stride(0),
+ part.stride(0) if split_k > 1 else 0, part.stride(1) if split_k > 1 else 0,
+ BLOCK_M=block_m, BLOCK_N=block_n, BLOCK_K=block_k, SLAB=block, SPLIT_K=split_k, QUANT_ACT=quant_act, OUT=_TL_DTYPE[out_dtype],
+ num_warps=num_warps,
)
- out = torch.empty(N, dtype=out_dtype, device=a_fp8.device)
- _splitk_reduce_kernel[(triton.cdiv(N, 256),)](
- part, out, N, split_k, part.stride(0), part.stride(1),
- BLOCK=256, OUT=_TL_DTYPE[out_dtype], num_warps=2,
+ if split_k > 1:
+ _splitk_reduce_kernel[(triton.cdiv(GN, 256), M)](
+ part, out, M, GN, split_k, part.stride(0), part.stride(1), out.stride(0),
+ BLOCK=256, OUT=_TL_DTYPE[out_dtype], num_warps=2,
+ )
+ return out
+
+
+def grouped_w8a16_gemv(x: torch.Tensor, weight: torch.Tensor, scale: torch.Tensor, *, block: int) -> torch.Tensor:
+ """Decode form of the reference's grouped bf16 einsum ``tgd,grd->tgr`` over a block-diagonal fp8
+ weight: ``x [t, G, d]`` bf16 (``t <= GEMV_MAX_M``), ``weight [G * r, d]`` fp8 with ``[G * r //
+ block, d // block]`` e8m0 ``scale`` -> ``[t, G * r]`` bf16. The weight is dequantized in-kernel
+ (exact in bf16: e4m3 x 2^k), the activation is not quantized, so this is the einsum's math with a
+ different fp32 summation order."""
+ assert x.ndim == 3 and weight.dtype == FP8
+ G = x.shape[1]
+ sb = (scale.view(torch.uint8) if scale.dtype == torch.float8_e8m0fnu else scale).contiguous()
+ return _fp8_gemv(x.contiguous(), e4m3_kernel_view(weight), sb, x.dtype, block, groups=G, quant_act=False)
+
+
+def _dequant_block_fp8_kernel(w_ptr, sb_ptr, out_ptr, N, K, stride_wn, stride_sbn, stride_on, SLAB: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr):
+ pid_n = tl.program_id(0)
+ pid_k = tl.program_id(1)
+ offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
+ offs_k = pid_k * BLOCK_K + tl.arange(0, BLOCK_K)
+ mask = (offs_n < N)[:, None] & (offs_k < K)[None, :]
+ if e4m3_native_cx():
+ w = tl.load(w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :], mask=mask, other=0.0).to(tl.float32)
+ else:
+ w = e4m3_u8_to_f32(tl.load(w_ptr + offs_n[:, None] * stride_wn + offs_k[None, :], mask=mask, other=0))
+ code = tl.load(sb_ptr + (offs_n // SLAB)[:, None] * stride_sbn + (offs_k // SLAB)[None, :], mask=mask, other=0)
+ tl.store(out_ptr + offs_n[:, None] * stride_on + offs_k[None, :], (w * tl.exp2(code.to(tl.float32) - 127.0)).to(tl.bfloat16), mask=mask)
+
+
+def dequant_block_fp8(weight: torch.Tensor, scale: torch.Tensor, *, block: int) -> torch.Tensor:
+ """``[N, K]`` block-scaled e4m3 -> bf16 (exact), one pass; the prefill-side companion of
+ :func:`grouped_w8a16_gemv` for weights the reference consumes through a bf16 einsum."""
+ N, K = weight.shape
+ sb = (scale.view(torch.uint8) if scale.dtype == torch.float8_e8m0fnu else scale).contiguous()
+ out = torch.empty((N, K), dtype=torch.bfloat16, device=weight.device)
+ _dequant_block_fp8_kernel[(triton.cdiv(N, 64), triton.cdiv(K, 256))](
+ e4m3_kernel_view(weight), sb, out, N, K, weight.stride(0), sb.stride(0), out.stride(0),
+ SLAB=block, BLOCK_N=64, BLOCK_K=256, num_warps=4,
)
return out
@@ -353,36 +465,41 @@ def block_fp8_linear(
weight: torch.Tensor,
scale: torch.Tensor,
bias: torch.Tensor | None = None,
+ *,
+ block: int = 128,
) -> torch.Tensor:
"""``y = act_quant(x) @ weight^T`` (reference FP8 path).
``x``: ``[..., K]`` bf16; ``weight``: ``[N, K]`` float8_e4m3fn; ``scale``:
- ``[N//128, K//128]`` float8_e8m0fnu (weight block scale). Activation is quantized
- to FP8 with a per-128 ue8m0 scale; the GEMM applies both scales per 128-K block.
+ ``[N//block, K//block]`` float8_e8m0fnu (weight block scale). Activation is quantized
+ to FP8 with a per-``block`` ue8m0 scale; the GEMM applies both scales per ``block``-K slab.
"""
assert weight.dtype == FP8
*lead, K = x.shape
N = weight.shape[0]
assert weight.shape[1] == K
- assert K % 128 == 0 and N % 128 == 0, (N, K)
+ assert K % block == 0 and N % block == 0, (N, K, block)
+ assert tuple(scale.shape) == (N // block, K // block), (tuple(scale.shape), N, K, block)
compute_dtype = x.dtype if x.dtype in _TL_DTYPE else torch.bfloat16
sb = scale.view(torch.uint8) if scale.dtype == torch.float8_e8m0fnu else scale
sb = sb.contiguous()
w = e4m3_kernel_view(weight)
- a_fp8, sa = act_quant_fp8(x, 128) # [M,K] fp8, [M,K//128] e8m0 codes
- M = a_fp8.shape[0]
-
- if M == 1:
- out = _fp8_act_gemv(a_fp8[0], sa[0], w, sb, compute_dtype).reshape(*lead, N)
+ x2d = x.reshape(-1, K)
+ if x2d.shape[0] == 1:
+ # decode: one kernel quantizes the activation rows and streams the weight once
+ out = _fp8_gemv(x2d.contiguous(), w, sb, compute_dtype, block).reshape(*lead, N)
if bias is not None:
out = out + bias.to(out.dtype)
return out
+ a_fp8, sa = act_quant_fp8(x, block) # [M,K] fp8, [M,K//block] e8m0 codes
+ M = a_fp8.shape[0]
+
out = torch.empty((M, N), dtype=compute_dtype, device=x.device)
BLOCK_M = 32
BLOCK_N = 128
- BLOCK_K = 128
+ BLOCK_K = block
grid = (triton.cdiv(M, BLOCK_M), triton.cdiv(N, BLOCK_N))
_fp8_act_gemm_kernel[grid](
a_fp8, w, sa, sb, out,
@@ -399,4 +516,4 @@ def block_fp8_linear(
return out
-__all__ = ["block_fp8_linear", "act_quant_fp8", "act_quant_fp8_inplace", "fp4_act_quant_inplace"]
+__all__ = ["block_fp8_linear", "act_quant_fp8", "act_quant_fp8_inplace", "fp4_act_quant_inplace", "grouped_w8a16_gemv", "dequant_block_fp8", "GEMV_MAX_M"]
diff --git a/python/freetoken/kernel/triton/dsv4/fused_moe.py b/python/freetoken/kernel/triton/dsv4/fused_moe.py
index 6eea606c..0adfbdd3 100644
--- a/python/freetoken/kernel/triton/dsv4/fused_moe.py
+++ b/python/freetoken/kernel/triton/dsv4/fused_moe.py
@@ -24,6 +24,9 @@ import torch
import triton
import triton.language as tl
+from freetoken.kernel.triton.dsv4.fp8_linear import _log2_ceil
+from freetoken.kernel.triton.e4m3_compat import e4m3_native_cx, round_e4m3
+
_E2M1_VALUES = [
0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0,
-0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0,
@@ -71,6 +74,12 @@ def _decode_dsfp4_moe_kernel(
n_mask = offs_n < N
slot = tl.load(topk_ids_ptr + token_id * stride_tid_m + route_k * stride_tid_k).to(tl.int64)
+ c_ptrs = c_ptr + token_id * stride_cm + route_k * stride_ck + offs_n * stride_cn
+ if slot < 0:
+ # an inactive route (hybrid decode computes it on the CPU): zero contribution, and no read of
+ # any slot -- a slot's storage may be uninitialized, and 0 * NaN is not 0
+ tl.store(c_ptrs, tl.zeros((BLOCK_SIZE_N,), dtype=compute_type), mask=(route_id < total_routes) & n_mask)
+ return
a_row = route_id if A_ROW_IS_ROUTE else token_id
a_base = a_ptr + a_row * stride_am
@@ -113,7 +122,6 @@ def _decode_dsfp4_moe_kernel(
weight = tl.load(topk_weights_ptr + token_id * stride_tw_m + route_k * stride_tw_k)
accumulator = accumulator * weight
- c_ptrs = c_ptr + token_id * stride_cm + route_k * stride_ck + offs_n * stride_cn
tl.store(c_ptrs, accumulator.to(compute_type), mask=(route_id < total_routes) & n_mask)
@@ -124,11 +132,15 @@ def _swiglu_kernel(
R, I, limit,
stride_gr, stride_gi, stride_or, stride_oi,
BLOCK: tl.constexpr, HAS_LIMIT: tl.constexpr, compute_type: tl.constexpr,
+ ACT_BLOCK: tl.constexpr, # > 0: fp8 (e4m3, ue8m0 per ACT_BLOCK) round-trip of the stored value, fused
):
"""Fused SwiGLU: ``out = silu(min(gate, limit)) * clamp(up, -limit, limit)``.
Collapses the gate/up split + two clamps + silu + mul + the fp32 round-trip
- (6 elementwise launches over [R, I]) between the two FP4 GEMVs into one pass."""
+ (6 elementwise launches over [R, I]) between the two FP4 GEMVs into one pass. With
+ ``ACT_BLOCK`` the down projection's activation quant (``act_quant_fp8_inplace`` on the bf16
+ value: ``s = 2**ceil(log2(max(|x|, 1e-4) / 448))``, e4m3 rounding, dequant) is applied to the
+ same tile before the store -- one launch and one memory pass fewer, bit-identical."""
row = tl.program_id(0)
cb = tl.program_id(1)
offs = cb * BLOCK + tl.arange(0, BLOCK)
@@ -139,6 +151,17 @@ def _swiglu_kernel(
g = tl.minimum(g, limit)
u = tl.minimum(tl.maximum(u, -limit), limit)
act = (g * tl.sigmoid(g)) * u
+ if ACT_BLOCK > 0:
+ tl.static_assert(BLOCK % ACT_BLOCK == 0, "the tile holds whole quant blocks")
+ xs = tl.reshape(act.to(compute_type).to(tl.float32), (BLOCK // ACT_BLOCK, ACT_BLOCK)) # the value the store would hold
+ amax = tl.maximum(tl.max(tl.abs(xs), axis=1), 1e-4)
+ sc = tl.exp2(_log2_ceil(amax * (1.0 / 448.0)).to(tl.float32))
+ q = tl.clamp(xs / sc[:, None], -448.0, 448.0)
+ if e4m3_native_cx():
+ q = q.to(tl.float8e4nv).to(tl.float32)
+ else:
+ q = round_e4m3(q)
+ act = tl.reshape(q * sc[:, None], (BLOCK,))
tl.store(out_ptr + row * stride_or + offs * stride_oi, act.to(compute_type), mask=mask)
@@ -274,20 +297,22 @@ def _compute_type(dtype: torch.dtype):
return {torch.bfloat16: tl.bfloat16, torch.float16: tl.float16, torch.float32: tl.float32}[dtype]
-def fused_swiglu(gate_up: torch.Tensor, limit: float) -> torch.Tensor:
- """``[..., 2I] -> [..., I]`` SwiGLU in a single kernel (see ``_swiglu_kernel``)."""
+def fused_swiglu(gate_up: torch.Tensor, limit: float, *, act_block: int = 0) -> torch.Tensor:
+ """``[..., 2I] -> [..., I]`` SwiGLU in a single kernel (see ``_swiglu_kernel``). ``act_block > 0``
+ also applies the down projection's fp8 activation round-trip (block ``act_block``) in the same pass."""
*lead, two_I = gate_up.shape
I = two_I // 2
gu = gate_up.reshape(-1, two_I)
R = gu.shape[0]
out = torch.empty((R, I), dtype=gate_up.dtype, device=gate_up.device)
BLOCK = 1024
+ assert act_block == 0 or (I % act_block == 0 and BLOCK % act_block == 0), (I, act_block)
grid = (R, triton.cdiv(I, BLOCK))
_swiglu_kernel[grid](
gu, out, R, I, float(limit),
gu.stride(0), gu.stride(1), out.stride(0), out.stride(1),
BLOCK=BLOCK, HAS_LIMIT=limit > 0,
- compute_type=_compute_type(gate_up.dtype), num_warps=4,
+ compute_type=_compute_type(gate_up.dtype), ACT_BLOCK=act_block, num_warps=4,
)
return out.reshape(*lead, I)
diff --git a/python/freetoken/kernel/triton/dsv4/router.py b/python/freetoken/kernel/triton/dsv4/router.py
new file mode 100644
index 00000000..17e3506a
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv4/router.py
@@ -0,0 +1,83 @@
+"""Fused MoE gate selection (DeepSeek-V4 / V4.1 ``Gate``): score activation, selection bias, top-k,
+weight normalization and route scale in one program per token.
+
+ s = act(scores) act: softmax | sigmoid | sqrt(softplus)
+ ids = topk(s + bias) selection is biased, the weights are not
+ w = s[ids] [/ (sum + 1e-20)] * route_scale
+
+One launch replaces the eight or so elementwise / top-k / gather launches of the torch expression --
+at decode each of those moves a few hundred floats and costs its launch floor. Ties in ``s + bias``
+go to the lowest expert id (torch.topk leaves them unspecified).
+"""
+
+from __future__ import annotations
+
+import torch
+import triton
+import triton.language as tl
+from triton.language.extra import libdevice
+
+_SCORE = {"softmax": 0, "sigmoid": 1, "sqrtsoftplus": 2}
+
+
+def _router_select_kernel(
+ scores_ptr, bias_ptr, alternate_bias_ptr, alternate_mask_ptr, w_ptr, id_ptr, E, route_scale,
+ stride_s, stride_w, stride_id,
+ SCORE: tl.constexpr, TOPK: tl.constexpr, BLOCK_K: tl.constexpr, NORMALIZE: tl.constexpr, BLOCK_E: tl.constexpr, ALTERNATE: tl.constexpr,
+):
+ t = tl.program_id(0).to(tl.int64)
+ offs = tl.arange(0, BLOCK_E)
+ mask = offs < E
+ s = tl.load(scores_ptr + t * stride_s + offs, mask=mask, other=float("-inf")).to(tl.float32)
+ if SCORE == 0:
+ s = tl.exp(s - tl.max(s))
+ s = s / tl.sum(tl.where(mask, s, 0.0))
+ elif SCORE == 1:
+ s = tl.sigmoid(s)
+ else:
+ s = tl.sqrt(tl.where(s > 20.0, s, libdevice.log1p(tl.exp(s)))) # torch softplus (threshold 20)
+ if ALTERNATE:
+ bias_ptr = tl.where(tl.load(alternate_mask_ptr + t), alternate_bias_ptr, bias_ptr)
+ bias = tl.load(bias_ptr + offs, mask=mask, other=0.0).to(tl.float32)
+ key = tl.where(mask, s + bias, float("-inf"))
+ slots = tl.arange(0, BLOCK_K)
+ picked_w = tl.zeros((BLOCK_K,), dtype=tl.float32)
+ picked_id = tl.zeros((BLOCK_K,), dtype=tl.int32)
+ for k in tl.static_range(TOPK):
+ best = tl.max(key)
+ idx = tl.min(tl.where(key == best, offs, BLOCK_E)) # lowest id among ties
+ w = tl.sum(tl.where(offs == idx, s, 0.0))
+ picked_w = tl.where(slots == k, w, picked_w)
+ picked_id = tl.where(slots == k, idx.to(tl.int32), picked_id)
+ key = tl.where(offs == idx, float("-inf"), key)
+ if NORMALIZE:
+ picked_w = picked_w / (tl.sum(picked_w) + 1e-20)
+ picked_w = picked_w * route_scale
+ tl.store(w_ptr + t * stride_w + slots, picked_w, mask=slots < TOPK)
+ tl.store(id_ptr + t * stride_id + slots, picked_id, mask=slots < TOPK)
+
+
+def router_select(
+ scores: torch.Tensor, bias: torch.Tensor, topk: int, *, score_func: str, normalize: bool, route_scale: float,
+ alternate_bias: torch.Tensor | None = None, alternate_mask: torch.Tensor | None = None,
+) -> tuple[torch.Tensor, torch.Tensor]:
+ """``scores [T, E]`` fp32 (pre-activation gate logits), ``bias [E]`` -> ``(weights [T, topk] fp32,
+ ids [T, topk] int32)`` in selection order (best first)."""
+ T, E = scores.shape
+ assert scores.dtype == torch.float32 and scores.stride(1) == 1, (scores.dtype, scores.stride())
+ weights = torch.empty((T, topk), dtype=torch.float32, device=scores.device)
+ ids = torch.empty((T, topk), dtype=torch.int32, device=scores.device)
+ if T == 0:
+ return weights, ids
+ _router_select_kernel[(T,)](
+ scores, bias, alternate_bias if alternate_bias is not None else bias,
+ alternate_mask if alternate_mask is not None else scores, weights, ids, E, float(route_scale),
+ scores.stride(0), weights.stride(0), ids.stride(0),
+ SCORE=_SCORE[score_func], TOPK=topk, BLOCK_K=triton.next_power_of_2(topk), NORMALIZE=normalize and topk > 1,
+ BLOCK_E=triton.next_power_of_2(E), ALTERNATE=alternate_mask is not None, num_warps=4,
+ )
+ return weights, ids
+
+
+__all__ = ["router_select"]
diff --git a/python/freetoken/kernel/triton/dsv41/__init__.py b/python/freetoken/kernel/triton/dsv41/__init__.py
new file mode 100644
index 00000000..00ec72da
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv41/__init__.py
@@ -0,0 +1,9 @@
+"""Triton kernels for DeepSeek-V4.1 Compressed Sparse Attention 2 over packed KV pools.
+
+``row_format`` device helpers: load a tile of packed rows (bf16 / fp8+e8m0 / fp4+e4m3 / fp4+e8m0)
+ and dequantize it to fp32, encode / decode e2m1 nibbles.
+``pack`` host wrappers: quantize rows into a packed pool (``pack_rows``) and read them back
+ (``unpack_rows``), matching the reference ``act_quant`` / ``fp4_act_quant`` numerics.
+``sparse_attn`` paged sparse attention gathering window (fp8) and compressed (fp4) rows.
+``indexer`` Lightning-indexer logits over fp4 index keys, full-range or candidate-restricted.
+"""
diff --git a/python/freetoken/kernel/triton/dsv41/indexer.py b/python/freetoken/kernel/triton/dsv41/indexer.py
new file mode 100644
index 00000000..37213158
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv41/indexer.py
@@ -0,0 +1,176 @@
+"""Lightning-Indexer logits over PACKED index keys (DeepSeek-V4.1, Full and Reindex modes).
+
+ logits[b, s, j] = sum_h relu(q[b, s, h, :] . k(b, pos_j)) * weights[b, s, h]
+
+``q`` carries the ``index_n_heads`` fp4-baked index queries, ``k`` is the single shared index key of
+a compressed position, gathered from the kv-source's packed pool and dequantized in-kernel. The
+row of compressed position ``t`` is derived in-kernel from the request's full-token locs (the page
+table row at prefill, the decode snapshot): ``locs[b, t * ratio] // ratio`` (negative = no row) --
+no ``[B, T]`` row table is materialized per layer. ``weights`` folds ``softmax_scale *
+n_heads**-0.5`` in.
+
+Two position sets, one kernel:
+
+* **full range** (Full mode): position ``j`` is the compressed position ``j``; the output is
+ ``[B, S, T]`` over the staged width ``T``, but only the live tiles (columns below ``live[b, s]``
+ rounded up to ``BLOCK_T``, the rest of the last tile ``-inf``) are WRITTEN. A fixed grid of worker
+ programs strides over the live tiles, so both the work and the launch geometry follow the live
+ history rather than the staged width; consumers (``dsv41/topk.py``) read below ``live`` only, so
+ the dead area needs no fill.
+* **candidate pool** (Reindex mode, Hierarchical Sparse Indexer): position ``j`` is
+ ``candidates[b, s, j]`` (a compressed position, ``-1`` = empty slot); the output is ``[B, S, NC]``
+ aligned with the candidate list and fully written (``-inf`` for empty / unreachable slots), so
+ a top-k over it indexes straight back into the list.
+
+Causality lives in the kernel: ``live[b, s]`` is the number of compressed positions query ``s`` may
+see, and every position at or past it (or with no row) scores ``-inf``. The dot product, weighted
+scores and head reduction round to bf16 at the same boundaries as the reference expression.
+"""
+
+from __future__ import annotations
+
+import torch
+import triton
+import triton.language as tl
+
+from freetoken.kvcache.dsv4.v41_row_format import RowFormat
+
+from .row_format import load_rows
+
+BLOCK_T = 64
+
+
+def _score_tile(
+ q, w, pool_ptr, locs_row, live, pos, store_mask, out_ptrs,
+ n_locs, RATIO: tl.constexpr, D: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr,
+):
+ """Score one tile of compressed positions ``pos`` against the query (``q [BLOCK_H, D]`` bf16,
+ ``w [BLOCK_H]`` fp32) and store ``[BLOCK_T]`` logits (``-inf`` where unreachable)."""
+ pos_ok = store_mask & (pos >= 0) & (pos < live)
+ full_idx = pos * RATIO
+ full = tl.load(locs_row + tl.maximum(full_idx, 0), mask=pos_ok & (full_idx < n_locs), other=-1)
+ rows = tl.where(full >= 0, full // RATIO, -1).to(tl.int32)
+ valid = pos_ok & (rows >= 0)
+ offs_d = tl.arange(0, D)
+ k = load_rows(pool_ptr, rows, valid, offs_d, D, ROW_BYTES, FMT).to(tl.bfloat16) # [BLOCK_T, D]
+ score = tl.dot(q, tl.trans(k)).to(tl.bfloat16).to(tl.float32)
+ score = (tl.maximum(score, 0.0) * w[:, None]).to(tl.bfloat16).to(tl.float32)
+ logits = tl.sum(score, axis=0).to(tl.bfloat16).to(tl.float32)
+ tl.store(out_ptrs, tl.where(valid, logits, float("-inf")), mask=store_mask)
+
+
+def _indexer_logits_packed_kernel(
+ q_ptr, w_ptr, pool_ptr, locs_ptr, live_ptr, cand_ptr, out_ptr,
+ T, NC, n_locs,
+ stride_qb, stride_qs, stride_qh, stride_qd,
+ stride_wb, stride_ws, stride_wh,
+ stride_lb, stride_lt,
+ stride_vb, stride_vs,
+ stride_cb, stride_cs, stride_cn,
+ stride_ob, stride_os, stride_on,
+ RATIO: tl.constexpr,
+ H: tl.constexpr,
+ D: tl.constexpr,
+ ROW_BYTES: tl.constexpr,
+ FMT: tl.constexpr,
+ BLOCK_H: tl.constexpr,
+ BLOCK_T: tl.constexpr,
+ HAS_CAND: tl.constexpr,
+ N_WORKERS: tl.constexpr,
+):
+ pid_s = tl.program_id(0)
+ pid_b = tl.program_id(1)
+ pid_t = tl.program_id(2)
+
+ live = tl.load(live_ptr + pid_b * stride_vb + pid_s * stride_vs)
+ locs_row = locs_ptr + pid_b * stride_lb
+ out_row = out_ptr + pid_b * stride_ob + pid_s * stride_os
+ offs_d = tl.arange(0, D)
+ offs_h = tl.arange(0, BLOCK_H)
+ h_mask = offs_h < H
+ q = tl.load(
+ q_ptr + pid_b * stride_qb + pid_s * stride_qs + offs_h[:, None] * stride_qh + offs_d[None, :] * stride_qd,
+ mask=h_mask[:, None], other=0.0,
+ ).to(tl.bfloat16)
+ w = tl.load(w_ptr + pid_b * stride_wb + pid_s * stride_ws + offs_h * stride_wh, mask=h_mask, other=0.0).to(tl.bfloat16).to(tl.float32)
+ offs = tl.arange(0, BLOCK_T)
+
+ if HAS_CAND:
+ # one program per candidate tile; the list is fixed-width and fully written
+ offs_t = pid_t * BLOCK_T + offs
+ store_mask = offs_t < NC
+ pos = tl.load(cand_ptr + pid_b * stride_cb + pid_s * stride_cs + offs_t * stride_cn, mask=store_mask, other=-1)
+ _score_tile(q, w, pool_ptr, locs_row, live, pos, store_mask, out_row + offs_t * stride_on, n_locs, RATIO, D, ROW_BYTES, FMT)
+ else:
+ # worker pid_t scores live tiles pid_t, pid_t + N_WORKERS, ...; nothing past live is written
+ for tile in range(pid_t, tl.cdiv(tl.minimum(live, T), BLOCK_T), N_WORKERS):
+ offs_t = tile * BLOCK_T + offs
+ store_mask = offs_t < T
+ _score_tile(q, w, pool_ptr, locs_row, live, offs_t, store_mask, out_row + offs_t * stride_on, n_locs, RATIO, D, ROW_BYTES, FMT)
+
+
+def indexer_logits_packed(
+ q: torch.Tensor, # [B, S, H, D] bf16 (fp4 round-tripped index queries)
+ weights: torch.Tensor, # [B, S, H] (softmax_scale * n_heads**-0.5 folded in)
+ k_pool: torch.Tensor, # [R, fmt.row_bytes(D)] uint8, the kv-source's packed index keys
+ fmt: RowFormat,
+ locs: torch.Tensor, # [B, W] int32/int64 full-token locs of each request (page-table row / decode snapshot)
+ ratio: int, # compressed position t -> full loc locs[b, t * ratio]; its row is loc // ratio
+ live: torch.Tensor, # [B, S] int32: compressed positions query (b, s) may see
+ T: int | None = None, # staged width of the full-range output (default: W // ratio)
+ candidates: torch.Tensor | None = None, # [B, S, NC] int32 compressed positions (-1 = empty)
+ out: torch.Tensor | None = None,
+) -> torch.Tensor:
+ """Head-reduced indexer logits: ``[B, S, T]`` over the full staged range (columns below ``live``
+ written, the rest untouched), or ``[B, S, NC]`` aligned with ``candidates`` (fully written).
+ Unreachable / empty positions are ``-inf``."""
+ B, S, H, D = q.shape
+ assert locs.ndim == 2 and locs.shape[0] == B and live.shape == (B, S), (locs.shape, live.shape, (B, S))
+ assert weights.shape == (B, S, H), (weights.shape, (B, S, H))
+ assert D == triton.next_power_of_2(D), f"index_head_dim must be pow2, got {D}"
+ assert k_pool.dtype == torch.uint8 and k_pool.shape[1] == fmt.row_bytes(D) and k_pool.is_contiguous()
+ assert locs.dtype in (torch.int32, torch.int64) and locs.stride(1) == 1, (locs.dtype, locs.stride())
+ n_locs = locs.shape[1]
+ if T is None:
+ T = n_locs // ratio
+ live = live.to(torch.int32)
+ has_cand = candidates is not None
+ if has_cand:
+ assert candidates.shape[:2] == (B, S), (candidates.shape, (B, S))
+ cand = candidates.to(torch.int32)
+ NC = cand.shape[2]
+ cs = cand.stride()
+ else:
+ cand, NC, cs = live, 0, (0, 0, 0)
+ n_out = NC if has_cand else T
+ if out is None:
+ out = torch.empty((B, S, n_out), dtype=torch.float32, device=q.device)
+ assert out.shape == (B, S, n_out) and out.dtype == torch.float32
+ if n_out == 0 or S == 0:
+ return out
+
+ n_tiles = triton.cdiv(n_out, BLOCK_T)
+ # A device-sized worker grid strides over live history without launching a
+ # program for every reserved tile. Independent queries share the SM budget.
+ sm_count = torch.cuda.get_device_properties(q.device).multi_processor_count
+ workers = n_tiles if has_cand else min(n_tiles, max(1, triton.cdiv(sm_count, B * S)))
+ grid = (S, B, workers)
+ _indexer_logits_packed_kernel[grid](
+ q, weights, k_pool, locs, live, cand, out,
+ T, NC, n_locs,
+ q.stride(0), q.stride(1), q.stride(2), q.stride(3),
+ weights.stride(0), weights.stride(1), weights.stride(2),
+ locs.stride(0), locs.stride(1),
+ live.stride(0), live.stride(1),
+ cs[0], cs[1], cs[2],
+ out.stride(0), out.stride(1), out.stride(2),
+ RATIO=ratio, H=H, D=D, ROW_BYTES=fmt.row_bytes(D), FMT=fmt.code,
+ BLOCK_H=triton.next_power_of_2(H), BLOCK_T=BLOCK_T, HAS_CAND=has_cand, N_WORKERS=workers,
+ num_warps=4, num_stages=2,
+ )
+ return out
+
+
+__all__ = ["indexer_logits_packed", "BLOCK_T"]
diff --git a/python/freetoken/kernel/triton/dsv41/pack.py b/python/freetoken/kernel/triton/dsv41/pack.py
new file mode 100644
index 00000000..47a9164e
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv41/pack.py
@@ -0,0 +1,183 @@
+"""Quantize rows into a packed pool and read them back (DeepSeek-V4.1 KV storage formats).
+
+``pack_rows`` reproduces the reference quantizers (``inference/kernel.py``) exactly:
+
+* fp8_e8m0_b32 ``act_quant(x, 32, "ue8m0", e8m0)``: per 32-group ``s = 2**ceil(log2(max(amax, 1e-4) / 448))``,
+ ``q = round_e4m3(clamp(x / s, +-448))``
+* fp4_e4m3_b16 ``fp4_act_quant(x, 16, scale_dtype=e4m3)``: ``s = e4m3(max(amax, 6 * 2^-9) / 6)`` (round to
+ nearest e4m3), ``q = round_e2m1(clamp(x / s, +-6))``
+* fp4_e8m0_b32 ``fp4_act_quant(x, 32)``: ``s = 2**ceil(log2(max(amax, 6 * 2^-126) / 6))``, same e2m1 rounding
+* bf16 a plain byte copy
+
+Rows land at ``pool[row_ids[m]]`` (or ``pool[m]`` without ids), so the same launch serves the KV write
+path (scatter into slots) and plain buffer conversion. ``unpack_rows`` is the inverse gather + dequant
+(bf16-rounded like the reference cache); the attention / indexer kernels use the same device helper
+(``row_format.load_rows``) in-line instead of materializing rows.
+"""
+
+from __future__ import annotations
+
+import torch
+import triton
+import triton.language as tl
+
+from freetoken.kernel.triton.dsv4.fp8_linear import _log2_ceil, _round_fp4
+from freetoken.kernel.triton.e4m3_compat import e4m3_f32_to_u8, e4m3_native_cx, round_e4m3
+from freetoken.kvcache.dsv4.v41_row_format import BF16, RowFormat
+
+from .row_format import fp4_f32_to_nibble, load_rows
+
+
+def _e4m3_codes(q):
+ """fp32 (|q| <= 448) -> e4m3 bit pattern, RNE."""
+ if e4m3_native_cx():
+ return q.to(tl.float8e4nv).to(tl.uint8, bitcast=True)
+ else:
+ return e4m3_f32_to_u8(round_e4m3(q))
+
+
+def _e4m3_round(q):
+ """fp32 -> nearest e4m3 value, as fp32."""
+ if e4m3_native_cx():
+ return q.to(tl.float8e4nv).to(tl.float32)
+ else:
+ return round_e4m3(q)
+
+
+def _pack_rows_kernel(
+ x_ptr, out_ptr, idx_ptr, stride_xm,
+ D: tl.constexpr, GROUP: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr, HAS_IDX: tl.constexpr,
+):
+ """One program per input row ``m``; writes packed bytes to pool row ``idx[m]`` (or ``m``)."""
+ m = tl.program_id(0).to(tl.int64)
+ if HAS_IDX:
+ row = tl.load(idx_ptr + m).to(tl.int64)
+ else:
+ row = m
+ dst = out_ptr + row * ROW_BYTES
+ src = x_ptr + m * stride_xm
+
+ if FMT == 0:
+ offs_d = tl.arange(0, D)
+ bits = tl.load(src + offs_d).to(tl.bfloat16).to(tl.uint16, bitcast=True)
+ tl.store(dst + 2 * offs_d, (bits & 0xFF).to(tl.uint8))
+ tl.store(dst + 2 * offs_d + 1, (bits >> 8).to(tl.uint8))
+ elif FMT == 1:
+ NG: tl.constexpr = D // GROUP
+ offs_d = tl.arange(0, D)
+ offs_g = tl.arange(0, NG)
+ x = tl.reshape(tl.load(src + offs_d).to(tl.float32), [NG, GROUP])
+ amax = tl.maximum(tl.max(tl.abs(x), axis=1), 1e-4)
+ e = _log2_ceil(amax * (1.0 / 448.0))
+ s = tl.exp2(e.to(tl.float32))
+ q = tl.clamp(tl.math.div_rn(x, s[:, None]), -448.0, 448.0)
+ tl.store(dst + offs_d, tl.reshape(_e4m3_codes(q), [D]))
+ tl.store(dst + D + offs_g, (e + 127).to(tl.uint8))
+ else:
+ # e2m1 pairs: even channels from one strided load, odd from another, so a byte is (lo | hi << 4)
+ NG: tl.constexpr = D // GROUP
+ HALF: tl.constexpr = GROUP // 2
+ offs_h = tl.arange(0, D // 2)
+ offs_g = tl.arange(0, NG)
+ xe = tl.reshape(tl.load(src + 2 * offs_h).to(tl.float32), [NG, HALF])
+ xo = tl.reshape(tl.load(src + 2 * offs_h + 1).to(tl.float32), [NG, HALF])
+ amax = tl.maximum(tl.max(tl.abs(xe), axis=1), tl.max(tl.abs(xo), axis=1))
+ if FMT == 2:
+ amax = tl.maximum(amax, 6.0 * (2.0 ** -9))
+ s = _e4m3_round(amax * (1.0 / 6.0))
+ tl.store(dst + D // 2 + offs_g, _e4m3_codes(s))
+ else:
+ amax = tl.maximum(amax, 6.0 * (2.0 ** -126))
+ e = _log2_ceil(amax * (1.0 / 6.0))
+ s = tl.exp2(e.to(tl.float32))
+ tl.store(dst + D // 2 + offs_g, (e + 127).to(tl.uint8))
+ # IEEE division: the reference's CUDA ``x / s`` is correctly rounded, and with an e4m3
+ # (non power-of-two) scale triton's default approximate div would land a hair off the
+ # e2m1 midpoints and flip their round-to-even
+ qe = fp4_f32_to_nibble(_round_fp4(tl.clamp(tl.math.div_rn(xe, s[:, None]), -6.0, 6.0)))
+ qo = fp4_f32_to_nibble(_round_fp4(tl.clamp(tl.math.div_rn(xo, s[:, None]), -6.0, 6.0)))
+ tl.store(dst + offs_h, tl.reshape(qe | (qo << 4), [D // 2]))
+
+
+def _unpack_rows_kernel(
+ pool_ptr, idx_ptr, out_ptr, M,
+ D: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr, BLOCK_T: tl.constexpr, HAS_IDX: tl.constexpr,
+):
+ pid = tl.program_id(0)
+ offs_t = pid * BLOCK_T + tl.arange(0, BLOCK_T)
+ valid = offs_t < M
+ if HAS_IDX:
+ rows = tl.load(idx_ptr + offs_t, mask=valid, other=0)
+ else:
+ rows = offs_t
+ offs_d = tl.arange(0, D)
+ vals = load_rows(pool_ptr, rows, valid, offs_d, D, ROW_BYTES, FMT)
+ tl.store(
+ out_ptr + offs_t[:, None].to(tl.int64) * D + offs_d[None, :],
+ vals.to(out_ptr.dtype.element_ty), mask=valid[:, None],
+ )
+
+
+def pack_rows(
+ x: torch.Tensor, fmt: RowFormat, pool: torch.Tensor | None = None, row_ids: torch.Tensor | None = None,
+) -> torch.Tensor:
+ """Quantize ``x [M, D]`` (bf16 / fp16 / fp32) into packed rows.
+
+ With ``pool [rows, row_bytes] uint8`` and ``row_ids [M]``, row ``m`` is written to ``pool[row_ids[m]]``
+ (the KV write path). Without a pool a fresh ``[M, row_bytes]`` uint8 tensor is returned.
+ """
+ assert x.dim() == 2, x.shape
+ m, d = x.shape
+ fmt.validate_dim(d)
+ row_bytes = fmt.row_bytes(d)
+ assert d & (d - 1) == 0 and d >= 16, f"pack_rows wants a power-of-two D >= 16, got {d}"
+ if pool is None:
+ assert row_ids is None
+ pool = torch.empty((m, row_bytes), dtype=torch.uint8, device=x.device)
+ else:
+ assert pool.dtype == torch.uint8 and pool.shape[1] == row_bytes and pool.is_contiguous(), (pool.shape, row_bytes)
+ assert row_ids is not None and row_ids.shape == (m,), (None if row_ids is None else row_ids.shape, m)
+ if m == 0:
+ return pool
+ if x.stride(1) != 1:
+ x = x.contiguous()
+ _pack_rows_kernel[(m,)](
+ x, pool, row_ids if row_ids is not None else pool, x.stride(0),
+ D=d, GROUP=fmt.group or d, ROW_BYTES=row_bytes, FMT=fmt.code, HAS_IDX=row_ids is not None,
+ num_warps=4,
+ )
+ return pool
+
+
+def unpack_rows(
+ pool: torch.Tensor, fmt: RowFormat, dim: int, row_ids: torch.Tensor | None = None,
+ out_dtype: torch.dtype = torch.bfloat16,
+) -> torch.Tensor:
+ """Gather ``pool[row_ids]`` (or every row) and dequantize to ``[M, dim]`` in ``out_dtype``."""
+ fmt.validate_dim(dim)
+ assert pool.dtype == torch.uint8 and pool.shape[1] == fmt.row_bytes(dim) and pool.is_contiguous(), pool.shape
+ m = pool.shape[0] if row_ids is None else row_ids.numel()
+ out = torch.empty((m, dim), dtype=out_dtype, device=pool.device)
+ if m == 0:
+ return out
+ block_t = 16
+ _unpack_rows_kernel[(triton.cdiv(m, block_t),)](
+ pool, row_ids if row_ids is not None else pool, out, m,
+ D=dim, ROW_BYTES=fmt.row_bytes(dim), FMT=fmt.code, BLOCK_T=block_t, HAS_IDX=row_ids is not None,
+ num_warps=4,
+ )
+ return out
+
+
+def quant_roundtrip(x: torch.Tensor, fmt: RowFormat) -> torch.Tensor:
+ """``unpack(pack(x))`` -- the value the cache will hand back; the reference's ``inplace=True`` semantics."""
+ if fmt is BF16:
+ return x.to(torch.bfloat16)
+ return unpack_rows(pack_rows(x, fmt), fmt, x.shape[1], out_dtype=torch.bfloat16)
+
+
+__all__ = ["pack_rows", "unpack_rows", "quant_roundtrip"]
diff --git a/python/freetoken/kernel/triton/dsv41/row_format.py b/python/freetoken/kernel/triton/dsv41/row_format.py
new file mode 100644
index 00000000..032d77e9
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv41/row_format.py
@@ -0,0 +1,92 @@
+"""Device-side row-format helpers shared by the DSV41 kernels.
+
+The Python-side descriptors live in ``freetoken.kvcache.dsv4.v41_row_format``; the ``FMT`` constexpr a
+kernel receives is ``RowFormat.code``:
+
+ 0 bf16 2 B / channel, no scale
+ 1 fp8_e8m0_b32 e4m3 byte / channel, ue8m0 scale byte per 32 channels
+ 2 fp4_e4m3_b16 e2m1 nibble / channel (even channel low nibble), e4m3 scale byte per 16
+ 3 fp4_e8m0_b32 e2m1 nibble / channel, ue8m0 scale byte per 32
+
+A row is ``[values | scales]``; ``ROW_BYTES`` is the pool's row stride. Every dequantized value
+is rounded through bf16 before use -- the reference bakes its quant round-trip into a bf16 cache,
+so this is what its attention / indexer kernels actually read.
+"""
+
+from __future__ import annotations
+
+import triton
+import triton.language as tl
+
+from freetoken.kernel.triton.e4m3_compat import e4m3_u8_to_f32
+
+FMT_BF16 = 0
+FMT_FP8_E8M0_B32 = 1
+FMT_FP4_E4M3_B16 = 2
+FMT_FP4_E8M0_B32 = 3
+
+
+def fp4_nibble_to_f32(nib):
+ """e2m1 code (uint8, 0..15) -> fp32: bit 3 sign, bits 2-1 exponent, bit 0 mantissa;
+ codes 0/1 are the subnormals 0 and 0.5, the rest ``(1 + m/2) * 2^(e-1)``."""
+ c = (nib & 7).to(tl.int32)
+ neg = (nib & 8) != 0
+ sub = c.to(tl.float32) * 0.5
+ normal = (1.0 + 0.5 * (c & 1).to(tl.float32)) * tl.exp2(((c >> 1) - 1).to(tl.float32))
+ mag = tl.where(c < 2, sub, normal)
+ return tl.where(neg, -mag, mag)
+
+
+def fp4_f32_to_nibble(q):
+ """fp32 value already on the e2m1 grid ({0, .5, 1, 1.5, 2, 3, 4, 6}, signed) -> e2m1 code.
+ Thresholds sit between grid points, so a value a hair off the grid still maps to its point."""
+ a = tl.abs(q)
+ c = tl.where(
+ a < 1.75,
+ tl.where(a < 0.75, tl.where(a < 0.25, 0, 1), tl.where(a < 1.25, 2, 3)),
+ tl.where(a < 3.5, tl.where(a < 2.5, 4, 5), tl.where(a < 5.0, 6, 7)),
+ )
+ return (c + tl.where(q < 0, 8, 0)).to(tl.uint8)
+
+
+def load_rows(pool_ptr, rows, valid, offs_d, D: tl.constexpr, ROW_BYTES: tl.constexpr, FMT: tl.constexpr):
+ """Gather + dequantize ``[T, D]`` fp32 from a packed pool.
+
+ ``pool_ptr`` uint8 pool base; ``rows`` [T] row ids (any int width; masked-out rows may be -1);
+ ``valid`` [T] bool; ``offs_d`` [D] = ``tl.arange(0, D)``. Invalid rows read as 0.
+ """
+ row_i64 = tl.maximum(rows, 0).to(tl.int64)
+ base = pool_ptr + row_i64[:, None] * ROW_BYTES
+ m = valid[:, None] & (offs_d >= 0)[None, :]
+ if FMT == 0:
+ lo = tl.load(base + 2 * offs_d[None, :], mask=m, other=0).to(tl.uint16)
+ hi = tl.load(base + 2 * offs_d[None, :] + 1, mask=m, other=0).to(tl.uint16)
+ val = (lo | (hi << 8)).to(tl.bfloat16, bitcast=True).to(tl.float32)
+ elif FMT == 1:
+ v = tl.load(base + offs_d[None, :], mask=m, other=0)
+ s = tl.load(base + D + (offs_d // 32)[None, :], mask=m, other=127)
+ val = e4m3_u8_to_f32(v) * tl.exp2(s.to(tl.float32) - 127.0)
+ else:
+ b = tl.load(base + (offs_d // 2)[None, :], mask=m, other=0)
+ nib = tl.where((offs_d % 2 == 0)[None, :], b & 0xF, b >> 4)
+ q = fp4_nibble_to_f32(nib)
+ if FMT == 2:
+ s = e4m3_u8_to_f32(tl.load(base + D // 2 + (offs_d // 16)[None, :], mask=m, other=0))
+ else:
+ s = tl.exp2(tl.load(base + D // 2 + (offs_d // 32)[None, :], mask=m, other=127).to(tl.float32) - 127.0)
+ val = q * s
+ return val.to(tl.bfloat16).to(tl.float32)
+
+
+__all__ = [
+ "FMT_BF16",
+ "FMT_FP8_E8M0_B32",
+ "FMT_FP4_E4M3_B16",
+ "FMT_FP4_E8M0_B32",
+ "fp4_nibble_to_f32",
+ "fp4_f32_to_nibble",
+ "load_rows",
+]
diff --git a/python/freetoken/kernel/triton/dsv41/sparse_attn.py b/python/freetoken/kernel/triton/dsv41/sparse_attn.py
new file mode 100644
index 00000000..9d6b0816
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv41/sparse_attn.py
@@ -0,0 +1,323 @@
+"""Sparse gathered-KV flash attention over PACKED pools (DeepSeek-V4.1).
+
+Same contract as ``kernel/triton/dsv4/sparse_attn.py`` -- each query attends to a per-query list of
+GLOBAL slots laid out ``[window part | compressed part]`` plus a per-head attention sink, K == V is one
+shared latent head -- but the two pools are byte pools in (possibly different) ``RowFormat``s: the
+window ring holds fp8 rows, the compressed KV holds fp4 rows, and every gathered row is dequantized
+in-kernel (``row_format.load_rows``) to the bf16 value the reference cache would have held.
+
+Because the two pools dequantize differently, a tile never mixes them: the window part is
+``n_window`` columns with ``n_window % BLOCK_T == 0`` (the caller pads it with ``-1``), so the kernel
+walks the window tiles and then the compressed tiles in two loops, each with a single load site.
+Both dots take bf16 operands with fp32 accumulation, like the reference ``sparse_attn`` kernel.
+
+Two implementations, picked from the launch shape (no knob):
+ * prefill (``m`` > 1): one program per (query, request, head block)
+ * decode (``m`` == 1): flash-decoding split over the candidate axis, merged through log-sum-exp
+ (the sink joins in the merge). Not bit-identical to the single-program path.
+
+``cmp_counts`` (device ``[b, m]`` int32) bounds the compressed columns a CUDA-graph replay visits;
+without it the full ``topk_idxs`` width is walked.
+"""
+
+from __future__ import annotations
+
+import torch
+import triton
+import triton.language as tl
+
+from freetoken.kvcache.dsv4.v41_row_format import RowFormat
+
+from .row_format import load_rows
+
+BLOCK_H = 16
+BLOCK_T = 32
+MAX_SPLITS = 32
+# Decode splits at tile boundaries, bounded by the device's SM count below.
+MIN_TILES_PER_SPLIT = 1
+
+
+def _attend_tile(q, kv, valid, scale, m_i, l_i, acc):
+ """One online-softmax step over a ``[BLOCK_T, D]`` fp32 KV tile (bf16-exact values)."""
+ kv16 = kv.to(tl.bfloat16)
+ scores = tl.dot(q, tl.trans(kv16)) * scale # [BLOCK_H, BLOCK_T] fp32
+ scores = tl.where(valid[None, :], scores, -float("inf"))
+ m_new = tl.maximum(m_i, tl.max(scores, axis=1))
+ # an all-masked tile keeps m_new at -inf; short-circuit so -inf - -inf never forms
+ alpha = tl.where(m_new == -float("inf"), 1.0, tl.exp(m_i - m_new))
+ p = tl.where(valid[None, :], tl.exp(scores - m_new[:, None]), 0.0)
+ l_i = l_i * alpha + tl.sum(p, axis=1)
+ acc = acc * alpha[:, None] + tl.dot(p.to(tl.bfloat16), kv16)
+ return m_new, l_i, acc
+
+
+def _sparse_attn_packed_kernel(
+ q_ptr, win_ptr, cmp_ptr, o_ptr, sink_ptr, idx_ptr, cnt_ptr,
+ scale,
+ H, TOPK, N_WINDOW,
+ stride_qb, stride_qm, stride_qh, stride_qd,
+ stride_ob, stride_om, stride_oh, stride_od,
+ stride_ib, stride_im, stride_it,
+ stride_nb, stride_nm,
+ D: tl.constexpr,
+ WIN_FMT: tl.constexpr, WIN_ROW_BYTES: tl.constexpr,
+ CMP_FMT: tl.constexpr, CMP_ROW_BYTES: tl.constexpr,
+ BLOCK_H: tl.constexpr,
+ BLOCK_T: tl.constexpr,
+ HAS_COUNTS: tl.constexpr,
+):
+ pid_m = tl.program_id(0)
+ pid_b = tl.program_id(1)
+ pid_h = tl.program_id(2)
+
+ offs_h = pid_h * BLOCK_H + tl.arange(0, BLOCK_H)
+ h_mask = offs_h < H
+ offs_d = tl.arange(0, D)
+
+ q_ptrs = q_ptr + pid_b * stride_qb + pid_m * stride_qm + offs_h[:, None] * stride_qh + offs_d[None, :] * stride_qd
+ q = tl.load(q_ptrs, mask=h_mask[:, None], other=0.0).to(tl.bfloat16) # [BLOCK_H, D]
+
+ m_i = tl.full((BLOCK_H,), -float("inf"), dtype=tl.float32)
+ l_i = tl.zeros((BLOCK_H,), dtype=tl.float32)
+ acc = tl.zeros((BLOCK_H, D), dtype=tl.float32)
+
+ n_active = TOPK
+ if HAS_COUNTS:
+ n_active = N_WINDOW + tl.load(cnt_ptr + pid_b * stride_nb + pid_m * stride_nm)
+
+ idx_base = idx_ptr + pid_b * stride_ib + pid_m * stride_im
+ for start in range(0, N_WINDOW, BLOCK_T):
+ offs_t = start + tl.arange(0, BLOCK_T)
+ idxs = tl.load(idx_base + offs_t * stride_it)
+ valid = idxs >= 0
+ kv = load_rows(win_ptr, idxs, valid, offs_d, D, WIN_ROW_BYTES, WIN_FMT)
+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)
+ for start in range(N_WINDOW, n_active, BLOCK_T):
+ offs_t = start + tl.arange(0, BLOCK_T)
+ t_mask = offs_t < n_active
+ idxs = tl.load(idx_base + offs_t * stride_it, mask=t_mask, other=-1)
+ valid = idxs >= 0
+ kv = load_rows(cmp_ptr, idxs, valid, offs_d, D, CMP_ROW_BYTES, CMP_FMT)
+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)
+
+ sink = tl.load(sink_ptr + offs_h, mask=h_mask, other=0.0).to(tl.float32)
+ l_i = l_i + tl.exp(sink - m_i)
+ o = acc / l_i[:, None]
+
+ o_ptrs = o_ptr + pid_b * stride_ob + pid_m * stride_om + offs_h[:, None] * stride_oh + offs_d[None, :] * stride_od
+ tl.store(o_ptrs, o.to(o_ptr.dtype.element_ty), mask=h_mask[:, None])
+
+
+def _sparse_attn_packed_splitk_kernel(
+ q_ptr, win_ptr, cmp_ptr, mid_o_ptr, mid_lse_ptr, idx_ptr, cnt_ptr,
+ scale,
+ H, TOPK, N_WINDOW,
+ stride_qb, stride_qm, stride_qh, stride_qd,
+ stride_mb, stride_mm, stride_mh, stride_ms, stride_md,
+ stride_lb, stride_lm, stride_lh, stride_ls,
+ stride_ib, stride_im, stride_it,
+ stride_nb, stride_nm,
+ D: tl.constexpr,
+ WIN_FMT: tl.constexpr, WIN_ROW_BYTES: tl.constexpr,
+ CMP_FMT: tl.constexpr, CMP_ROW_BYTES: tl.constexpr,
+ BLOCK_H: tl.constexpr,
+ BLOCK_T: tl.constexpr,
+ HAS_COUNTS: tl.constexpr,
+ NUM_SPLITS: tl.constexpr,
+):
+ """Stage 1: each program reduces one BLOCK_T-aligned slice of the candidate list and writes
+ its normalized partial output + log-sum-exp. A slice may straddle the window / compressed
+ boundary; it is walked as (window tiles, compressed tiles) since N_WINDOW is tile-aligned."""
+ pid_ms = tl.program_id(0)
+ pid_m = pid_ms // NUM_SPLITS
+ split_id = pid_ms % NUM_SPLITS
+ pid_b = tl.program_id(1)
+ pid_h = tl.program_id(2)
+
+ offs_h = pid_h * BLOCK_H + tl.arange(0, BLOCK_H)
+ h_mask = offs_h < H
+ offs_d = tl.arange(0, D)
+
+ n_active = TOPK
+ if HAS_COUNTS:
+ n_active = N_WINDOW + tl.load(cnt_ptr + pid_b * stride_nb + pid_m * stride_nm)
+
+ per_split = tl.cdiv(tl.cdiv(n_active, NUM_SPLITS), BLOCK_T) * BLOCK_T
+ split_start = per_split * split_id
+ split_end = tl.minimum(split_start + per_split, n_active)
+
+ m_i = tl.full((BLOCK_H,), -float("inf"), dtype=tl.float32)
+ l_i = tl.zeros((BLOCK_H,), dtype=tl.float32)
+ acc = tl.zeros((BLOCK_H, D), dtype=tl.float32)
+
+ if split_end > split_start:
+ q_ptrs = (
+ q_ptr + pid_b * stride_qb + pid_m * stride_qm
+ + offs_h[:, None] * stride_qh + offs_d[None, :] * stride_qd
+ )
+ q = tl.load(q_ptrs, mask=h_mask[:, None], other=0.0).to(tl.bfloat16)
+ idx_base = idx_ptr + pid_b * stride_ib + pid_m * stride_im
+
+ win_end = tl.minimum(split_end, N_WINDOW)
+ for start in range(split_start, win_end, BLOCK_T):
+ offs_t = start + tl.arange(0, BLOCK_T)
+ idxs = tl.load(idx_base + offs_t * stride_it)
+ valid = idxs >= 0
+ kv = load_rows(win_ptr, idxs, valid, offs_d, D, WIN_ROW_BYTES, WIN_FMT)
+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)
+ cmp_start = tl.maximum(split_start, N_WINDOW)
+ for start in range(cmp_start, split_end, BLOCK_T):
+ offs_t = start + tl.arange(0, BLOCK_T)
+ t_mask = offs_t < split_end
+ idxs = tl.load(idx_base + offs_t * stride_it, mask=t_mask, other=-1)
+ valid = idxs >= 0
+ kv = load_rows(cmp_ptr, idxs, valid, offs_d, D, CMP_ROW_BYTES, CMP_FMT)
+ m_i, l_i, acc = _attend_tile(q, kv, valid, scale, m_i, l_i, acc)
+
+ out = tl.where(l_i[:, None] == 0.0, 0.0, acc / l_i[:, None])
+ lse = tl.where(l_i == 0.0, -float("inf"), m_i + tl.log(l_i))
+
+ mid_base = (
+ mid_o_ptr + pid_b * stride_mb + pid_m * stride_mm
+ + offs_h[:, None] * stride_mh + split_id * stride_ms + offs_d[None, :] * stride_md
+ )
+ tl.store(mid_base, out, mask=h_mask[:, None])
+ lse_base = (
+ mid_lse_ptr + pid_b * stride_lb + pid_m * stride_lm
+ + offs_h * stride_lh + split_id * stride_ls
+ )
+ tl.store(lse_base, lse, mask=h_mask)
+
+
+def _sparse_attn_splitk_merge_kernel(
+ mid_o_ptr, mid_lse_ptr, o_ptr, sink_ptr,
+ stride_mb, stride_mm, stride_mh, stride_ms, stride_md,
+ stride_lb, stride_lm, stride_lh, stride_ls,
+ stride_ob, stride_om, stride_oh, stride_od,
+ D: tl.constexpr,
+ NUM_SPLITS: tl.constexpr,
+):
+ """Stage 2: log-sum-exp merge over the splits; the attention sink joins here once, as a null
+ key with logit ``attn_sink[h]`` and zero value. One program per (query, request, head)."""
+ pid_m = tl.program_id(0)
+ pid_b = tl.program_id(1)
+ pid_h = tl.program_id(2)
+ offs_d = tl.arange(0, D)
+
+ m_i = tl.load(sink_ptr + pid_h).to(tl.float32)
+ l_i = 1.0
+ acc = tl.zeros((D,), dtype=tl.float32)
+
+ mid_base = mid_o_ptr + pid_b * stride_mb + pid_m * stride_mm + pid_h * stride_mh + offs_d * stride_md
+ lse_base = mid_lse_ptr + pid_b * stride_lb + pid_m * stride_lm + pid_h * stride_lh
+
+ for split_id in tl.range(0, NUM_SPLITS, num_stages=2):
+ partial = tl.load(mid_base + split_id * stride_ms)
+ lse = tl.load(lse_base + split_id * stride_ls)
+ m_new = tl.maximum(m_i, lse)
+ alpha = tl.exp(m_i - m_new)
+ beta = tl.where(lse == -float("inf"), 0.0, tl.exp(lse - m_new))
+ acc = acc * alpha + partial * beta
+ l_i = l_i * alpha + beta
+ m_i = m_new
+
+ o = acc / l_i
+ o_ptrs = o_ptr + pid_b * stride_ob + pid_m * stride_om + pid_h * stride_oh + offs_d * stride_od
+ tl.store(o_ptrs, o.to(o_ptr.dtype.element_ty))
+
+
+def split_count(b: int, m: int, h: int, topk: int, device) -> int:
+ """How many ways to split the candidate axis; 0 means run the single-program kernel. Decode
+ splits until every SM has a program or every split is down to one tile."""
+ if m != 1:
+ return 0
+ sm_count = torch.cuda.get_device_properties(device).multi_processor_count
+ n_splits = min(
+ MAX_SPLITS,
+ triton.cdiv(topk, MIN_TILES_PER_SPLIT * BLOCK_T),
+ max(1, sm_count // (b * triton.cdiv(h, BLOCK_H))),
+ )
+ return n_splits if n_splits > 1 else 0
+
+
+def sparse_attn_packed(
+ q: torch.Tensor, # [b, m, h, d] bf16
+ window_pool: torch.Tensor, # [n_win_slots, win_fmt.row_bytes(d)] uint8
+ win_fmt: RowFormat,
+ cmp_pool: torch.Tensor, # [n_cmp, cmp_fmt.row_bytes(d)] uint8
+ cmp_fmt: RowFormat,
+ attn_sink: torch.Tensor, # [h]
+ topk_idxs: torch.Tensor, # [b, m, topk] int32, GLOBAL rows, layout [window | compressed]
+ n_window: int, # window columns (a multiple of BLOCK_T; pad with -1)
+ softmax_scale: float,
+ cmp_counts: torch.Tensor | None = None, # [b, m] int32 live compressed columns per query
+) -> torch.Tensor:
+ """Paged sparse MLA attention over packed KV pools; see the module docstring for the contract."""
+ b, m, h, d = q.shape
+ topk = topk_idxs.shape[-1]
+ assert n_window % BLOCK_T == 0 and 0 <= n_window <= topk, (n_window, topk)
+ assert window_pool.dtype == torch.uint8 and window_pool.shape[1] == win_fmt.row_bytes(d), (window_pool.shape, win_fmt)
+ assert cmp_pool.dtype == torch.uint8 and cmp_pool.shape[1] == cmp_fmt.row_bytes(d), (cmp_pool.shape, cmp_fmt)
+ assert window_pool.is_contiguous() and cmp_pool.is_contiguous()
+ q = q.contiguous()
+ idx = topk_idxs.contiguous().to(torch.int32)
+ sink = attn_sink.contiguous().to(torch.float32)
+ o = torch.empty_like(q)
+
+ has_counts = cmp_counts is not None
+ if has_counts:
+ cnt = cmp_counts.contiguous().to(torch.int32).view(b, m)
+ stride_nb, stride_nm = cnt.stride()
+ else:
+ cnt, stride_nb, stride_nm = idx, 0, 0
+
+ fmt_args = dict(
+ WIN_FMT=win_fmt.code, WIN_ROW_BYTES=win_fmt.row_bytes(d),
+ CMP_FMT=cmp_fmt.code, CMP_ROW_BYTES=cmp_fmt.row_bytes(d),
+ )
+ n_splits = split_count(b, m, h, topk, q.device)
+ if n_splits:
+ head_blocks = triton.cdiv(h, BLOCK_H)
+ mid_o = torch.empty((b, m, h, n_splits, d), dtype=torch.float32, device=q.device)
+ mid_lse = torch.empty((b, m, h, n_splits), dtype=torch.float32, device=q.device)
+ _sparse_attn_packed_splitk_kernel[(m * n_splits, b, head_blocks)](
+ q, window_pool, cmp_pool, mid_o, mid_lse, idx, cnt,
+ float(softmax_scale),
+ h, topk, int(n_window),
+ q.stride(0), q.stride(1), q.stride(2), q.stride(3),
+ mid_o.stride(0), mid_o.stride(1), mid_o.stride(2), mid_o.stride(3), mid_o.stride(4),
+ mid_lse.stride(0), mid_lse.stride(1), mid_lse.stride(2), mid_lse.stride(3),
+ idx.stride(0), idx.stride(1), idx.stride(2),
+ stride_nb, stride_nm,
+ D=d, BLOCK_H=BLOCK_H, BLOCK_T=BLOCK_T, HAS_COUNTS=has_counts, NUM_SPLITS=n_splits,
+ **fmt_args, num_warps=8, num_stages=2,
+ )
+ _sparse_attn_splitk_merge_kernel[(m, b, h)](
+ mid_o, mid_lse, o, sink,
+ mid_o.stride(0), mid_o.stride(1), mid_o.stride(2), mid_o.stride(3), mid_o.stride(4),
+ mid_lse.stride(0), mid_lse.stride(1), mid_lse.stride(2), mid_lse.stride(3),
+ o.stride(0), o.stride(1), o.stride(2), o.stride(3),
+ D=d, NUM_SPLITS=n_splits, num_warps=4,
+ )
+ return o
+
+ grid = (m, b, triton.cdiv(h, BLOCK_H))
+ _sparse_attn_packed_kernel[grid](
+ q, window_pool, cmp_pool, o, sink, idx, cnt,
+ float(softmax_scale),
+ h, topk, int(n_window),
+ q.stride(0), q.stride(1), q.stride(2), q.stride(3),
+ o.stride(0), o.stride(1), o.stride(2), o.stride(3),
+ idx.stride(0), idx.stride(1), idx.stride(2),
+ stride_nb, stride_nm,
+ D=d, BLOCK_H=BLOCK_H, BLOCK_T=BLOCK_T, HAS_COUNTS=has_counts,
+ **fmt_args, num_warps=8, num_stages=2,
+ )
+ return o
+
+
+__all__ = ["sparse_attn_packed", "split_count", "BLOCK_T"]
diff --git a/python/freetoken/kernel/triton/dsv41/topk.py b/python/freetoken/kernel/triton/dsv41/topk.py
new file mode 100644
index 00000000..4d638e4d
--- /dev/null
+++ b/python/freetoken/kernel/triton/dsv41/topk.py
@@ -0,0 +1,265 @@
+"""Exact top-k and block-candidate selection whose work follows the live history (DeepSeek-V4.1).
+
+The indexer scores a request's compressed positions into a buffer as wide as the staged history
+(``max_seq_len / ratio``, fixed while a CUDA graph is captured), but only the first ``live[row]``
+columns are meaningful -- and at 1M staged positions a 4K-token request must not pay for the
+other 99.6%. Every kernel here takes the live count per row on the device, keeps its output
+addresses and shapes fixed, and restricts what it READS and reduces to the live prefix; nothing
+past ``live`` is ever read, so the score buffer's dead area needs no initialization.
+
+Selection is a fixed pipeline of small kernels with explicit workspace contracts:
+
+* ``dsv41_topk``: exact top-k of ``[R, T]`` fp32 scores below ``live``. Level 0 gives every
+ ``CHUNK``-wide slice of a row its own program, which radix-selects the slice's top-k into a
+ ``(key, column)`` candidate workspace (a global winner beats at most k-1 columns, so it is
+ inside its own slice's top-k: the union is exact). Further levels apply the same program to the
+ candidates until one register-resident tile remains; the final program ranks it and emits the
+ winners in ascending column order, ``-1`` padded. Slices past ``live`` exit at once and a
+ slice with no more entries than ``k`` is copied through, so a short history costs a handful of
+ near-empty launches. The radix primitives are the QSA block top-k's (``kernel/triton/qsa/topk``).
+* ``dsv41_block_scores``: per-``block_size`` block maxima over the live prefix (the Hierarchical
+ Sparse Indexer's level one), the block holding the newest live position pinned to ``+inf`` so it
+ is always kept; a fixed worker grid strides over the live blocks.
+* ``dsv41_candidate_blocks``: the two above plus a fixed-width expansion into the compact candidate
+ list ``[R, topk_blocks * block_size]`` of compressed positions, ascending with ``-1`` in the tail.
+
+Tie rule (documented, deterministic): scores order descending, equal scores by ascending column,
+so the lowest positions win among exact ties. ``torch.topk`` leaves ties unspecified.
+"""
+
+from __future__ import annotations
+
+import torch
+import triton
+import triton.language as tl
+
+from freetoken.kernel.triton.qsa.topk import _BINS, _PASSES, _RADIX, _monotone_key, _resident_prefix, _tile_ranks
+
+CHUNK = 4096 # register-resident slice per level program (fp32 keys + int32 columns)
+FINAL_MAX = 4096 # the final program's resident candidate tile
+BLOCK_SCORE_TILE = 512 # blocks per block-max program tile
+
+
+def _load_entries(src_ptr, col_ptr, base, offsets, limit, SRC_LOGITS: tl.constexpr):
+ """One slice of a level's input as ``(monotone key, column)``; key 0 = dead (past ``limit``,
+ ``-inf``, or a padding slot of the previous level)."""
+ live = offsets < limit
+ if SRC_LOGITS:
+ value = tl.load(src_ptr + base + offsets, mask=live, other=-float("inf"))
+ key = tl.where(live & (value > -float("inf")), _monotone_key(value), 0)
+ cols = (base + offsets).to(tl.int32)
+ else:
+ key = tl.load(src_ptr + base + offsets, mask=live, other=0).to(tl.uint32, bitcast=True)
+ cols = tl.load(col_ptr + base + offsets, mask=live, other=-1)
+ return key, cols
+
+
+def _dsv41_topk_level_kernel(
+ src_ptr, src_col_ptr, live_in_ptr, key_out_ptr, col_out_ptr, live_out_ptr,
+ stride_src_row, stride_out_row, num_columns,
+ TOP_K: tl.constexpr, PAD_K: tl.constexpr, CHUNK: tl.constexpr, N_SPLITS: tl.constexpr,
+ SRC_LOGITS: tl.constexpr, BINS: tl.constexpr, RADIX: tl.constexpr, PASSES: tl.constexpr,
+):
+ """One program per (row, slice): the slice's top-k into candidate slot ``split * TOP_K``. Slot
+ ``split`` is read downstream only when ``split < cdiv(live, CHUNK)``; program 0 publishes that
+ live candidate count for the next level."""
+ tl.static_assert(CHUNK <= 0xFFFF, "packed cumsum keeps 16 bits per half")
+ tl.static_assert(TOP_K <= CHUNK, "a slice must hold its own top-k")
+ row = tl.program_id(0)
+ split = tl.program_id(1)
+ visible = tl.maximum(tl.minimum(tl.load(live_in_ptr + row), num_columns), 0)
+ if split == 0:
+ tl.store(live_out_ptr + row, tl.minimum(tl.cdiv(visible, CHUNK), N_SPLITS) * TOP_K)
+ base = split * CHUNK
+ limit = tl.minimum(tl.maximum(visible - base, 0), CHUNK)
+ if limit <= 0:
+ return # a dead slice: nothing downstream reads its slot
+ slot = row.to(tl.int64) * stride_out_row + split * TOP_K
+ offsets = tl.arange(0, CHUNK)
+ key, cols = _load_entries(src_ptr + row.to(tl.int64) * stride_src_row, src_col_ptr + row.to(tl.int64) * stride_src_row, base, offsets, limit, SRC_LOGITS)
+ if limit <= TOP_K:
+ # every live entry is a winner: copy in order (dead keys stay 0 for the next level)
+ inside = offsets < TOP_K
+ tl.store(key_out_ptr + slot + offsets, key.to(tl.int32, bitcast=True), mask=inside)
+ tl.store(col_out_ptr + slot + offsets, cols, mask=inside)
+ return
+ prefix, ties = _resident_prefix(key, TOP_K, BINS, RADIX, PASSES)
+ rank, take, above, equal = _tile_ranks(key, prefix, ties, 0, 0)
+ write = take & (rank < TOP_K)
+ tl.store(key_out_ptr + slot + rank, key.to(tl.int32, bitcast=True), mask=write)
+ tl.store(col_out_ptr + slot + rank, cols, mask=write)
+ emitted = above + tl.minimum(equal, ties)
+ pad = tl.arange(0, PAD_K)
+ tl.store(key_out_ptr + slot + pad, 0, mask=(pad >= emitted) & (pad < TOP_K))
+
+
+def _final_tile(src_row, col_row, out_row, limit, TOP_K: tl.constexpr, BLOCK: tl.constexpr, SRC_LOGITS: tl.constexpr, BINS: tl.constexpr, RADIX: tl.constexpr, PASSES: tl.constexpr):
+ tl.static_assert(BLOCK <= 0xFFFF, "packed cumsum keeps 16 bits per half")
+ offsets = tl.arange(0, BLOCK)
+ key, cols = _load_entries(src_row, col_row, 0, offsets, limit, SRC_LOGITS)
+ prefix, ties = _resident_prefix(key, tl.minimum(limit, TOP_K), BINS, RADIX, PASSES)
+ rank, take, above, equal = _tile_ranks(key, prefix, ties, 0, 0)
+ tl.store(out_row + rank, cols, mask=take & (rank < TOP_K))
+ return above + tl.minimum(equal, ties)
+
+
+def _dsv41_topk_final_kernel(
+ src_ptr, src_col_ptr, live_ptr, out_ptr,
+ stride_src_row, stride_out_row, num_columns,
+ TOP_K: tl.constexpr, PAD_K: tl.constexpr,
+ BLOCK_SMALL: tl.constexpr, BLOCK_MID: tl.constexpr, BLOCK_FULL: tl.constexpr,
+ SRC_LOGITS: tl.constexpr, BINS: tl.constexpr, RADIX: tl.constexpr, PASSES: tl.constexpr,
+):
+ """One program per row over the last level's live candidates (or the scores themselves when the
+ row fits one tile): winners at their ascending-column ranks, ``-1`` from ``emitted`` on."""
+ row = tl.program_id(0)
+ limit = tl.maximum(tl.minimum(tl.load(live_ptr + row), num_columns), 0)
+ src_row = src_ptr + row.to(tl.int64) * stride_src_row
+ col_row = src_col_ptr + row.to(tl.int64) * stride_src_row
+ out_row = out_ptr + row.to(tl.int64) * stride_out_row
+ emitted = 0
+ if limit > 0:
+ # register residency costs the whole tile, so a short row takes a narrower one
+ if limit <= BLOCK_SMALL:
+ emitted = _final_tile(src_row, col_row, out_row, limit, TOP_K, BLOCK_SMALL, SRC_LOGITS, BINS, RADIX, PASSES)
+ elif limit <= BLOCK_MID:
+ emitted = _final_tile(src_row, col_row, out_row, limit, TOP_K, BLOCK_MID, SRC_LOGITS, BINS, RADIX, PASSES)
+ else:
+ emitted = _final_tile(src_row, col_row, out_row, limit, TOP_K, BLOCK_FULL, SRC_LOGITS, BINS, RADIX, PASSES)
+ pad = tl.arange(0, PAD_K)
+ tl.store(out_row + pad, -1, mask=(pad >= emitted) & (pad < TOP_K))
+
+
+MAX_K = CHUNK // 2 # a slice must shed at least half its entries per level, or the plan cannot converge
+
+
+def topk_plan(columns: int, k: int) -> list[tuple[int, int]]:
+ """``[(input width, n_splits), ...]`` of the reduction levels before the final tile; empty when
+ the row already fits the final tile. Derived from the buffer width alone (graph-static). Each
+ level maps ``width`` to ``cdiv(width, CHUNK) * k`` candidates, which shrinks the row by at least
+ ``CHUNK / k >= 2`` per level for ``k <= MAX_K``; larger ``k`` would stall (``k == CHUNK`` never
+ shrinks) and is rejected here, before any launch."""
+ if not 0 < k <= MAX_K:
+ raise ValueError(f"dsv41_topk supports 1 <= k <= {MAX_K}, got {k}")
+ levels: list[tuple[int, int]] = []
+ width = columns
+ while width > FINAL_MAX:
+ n_splits = -(-width // CHUNK)
+ levels.append((width, n_splits))
+ nxt = n_splits * k
+ assert nxt < width, (width, k) # guaranteed by k <= CHUNK // 2 for width > FINAL_MAX == CHUNK
+ width = nxt
+ return levels
+
+
+def dsv41_topk(scores: torch.Tensor, live: torch.Tensor, k: int, out: torch.Tensor | None = None) -> torch.Tensor:
+ """Exact top-``k`` columns of every ``scores`` row among its first ``live[row]`` columns.
+
+ ``scores [R, T]`` fp32 row-contiguous (columns at or past ``live`` are never read; ``-inf``
+ columns are dead), ``live [R]`` int32. Returns ``out [R, k]`` int32: the winners in ascending
+ column order packed at the front, ``-1`` from the ``emitted`` count on (fewer live or finite
+ columns than ``k`` leave a ``-1`` tail). Ties: lower column wins."""
+ assert scores.ndim == 2 and scores.dtype == torch.float32 and scores.stride(1) == 1, (scores.shape, scores.dtype)
+ R, T = scores.shape
+ assert live.shape == (R,), (live.shape, R)
+ live = live.to(torch.int32).contiguous()
+ if out is None:
+ out = torch.empty((R, k), dtype=torch.int32, device=scores.device)
+ assert out.shape == (R, k) and out.dtype == torch.int32 and out.stride(1) == 1
+ if R == 0 or k == 0:
+ return out
+ if T == 0:
+ out.fill_(-1)
+ return out
+ pad_k = triton.next_power_of_2(k)
+ src, src_col, src_live, width, src_logits = scores, scores, live, T, True
+ for level_width, n_splits in topk_plan(T, k):
+ keys = torch.empty((R, n_splits * k), dtype=torch.int32, device=scores.device)
+ cols = torch.empty((R, n_splits * k), dtype=torch.int32, device=scores.device)
+ live_out = torch.empty((R,), dtype=torch.int32, device=scores.device)
+ _dsv41_topk_level_kernel[(R, n_splits)](
+ src, src_col, src_live, keys, cols, live_out,
+ src.stride(0), keys.stride(0), level_width,
+ TOP_K=k, PAD_K=pad_k, CHUNK=CHUNK, N_SPLITS=n_splits, SRC_LOGITS=src_logits,
+ BINS=_BINS, RADIX=_RADIX, PASSES=_PASSES, num_warps=8, num_stages=1,
+ )
+ src, src_col, src_live, width, src_logits = keys, cols, live_out, n_splits * k, False
+ block = triton.next_power_of_2(max(width, 1))
+ _dsv41_topk_final_kernel[(R,)](
+ src, src_col, src_live, out,
+ src.stride(0), out.stride(0), width,
+ TOP_K=k, PAD_K=pad_k,
+ BLOCK_SMALL=min(1024, block), BLOCK_MID=min(4096, block), BLOCK_FULL=block,
+ SRC_LOGITS=src_logits, BINS=_BINS, RADIX=_RADIX, PASSES=_PASSES, num_warps=8, num_stages=1,
+ )
+ return out
+
+
+def _dsv41_block_scores_kernel(
+ scores_ptr, live_ptr, blk_ptr,
+ stride_s_row, stride_b_row, num_blocks,
+ BLOCK_SIZE: tl.constexpr, TILE: tl.constexpr, N_WORKERS: tl.constexpr,
+):
+ """Worker ``w`` of a row scores block tiles ``w, w + N_WORKERS, ...`` below the live block count:
+ the block's best live position, ``+inf`` for the block holding the newest live position."""
+ row = tl.program_id(0)
+ worker = tl.program_id(1)
+ live = tl.maximum(tl.load(live_ptr + row), 0)
+ nb_live = tl.minimum(tl.cdiv(live, BLOCK_SIZE), num_blocks)
+ s_row = scores_ptr + row.to(tl.int64) * stride_s_row
+ b_row = blk_ptr + row.to(tl.int64) * stride_b_row
+ offs_b = tl.arange(0, TILE)
+ offs_p = tl.arange(0, BLOCK_SIZE)
+ for tile in range(worker, tl.cdiv(nb_live, TILE), N_WORKERS):
+ blocks = tile * TILE + offs_b
+ b_mask = blocks < nb_live
+ pos = blocks[:, None] * BLOCK_SIZE + offs_p[None, :]
+ v = tl.load(s_row + pos, mask=b_mask[:, None] & (pos < live), other=-float("inf"))
+ best = tl.max(v, axis=1)
+ best = tl.where(blocks == nb_live - 1, float("inf"), best)
+ tl.store(b_row + blocks, best, mask=b_mask)
+
+
+def dsv41_block_scores(scores: torch.Tensor, live: torch.Tensor, block_size: int, out: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor]:
+ """``(block scores [R, cdiv(T, block_size)] fp32, live blocks [R] int32)``: each live block's best
+ live position (``+inf`` for the newest block); blocks at or past the live count are not written."""
+ R, T = scores.shape
+ nb = -(-T // block_size)
+ live = live.to(torch.int32).contiguous()
+ if out is None:
+ out = torch.empty((R, nb), dtype=torch.float32, device=scores.device)
+ live_blocks = torch.div(live + block_size - 1, block_size, rounding_mode="floor").clamp_(min=0).to(torch.int32)
+ if R == 0 or nb == 0:
+ return out, live_blocks
+ assert block_size == triton.next_power_of_2(block_size), block_size
+ sm_count = torch.cuda.get_device_properties(scores.device).multi_processor_count
+ workers = min(triton.cdiv(nb, BLOCK_SCORE_TILE), max(1, triton.cdiv(sm_count, R)))
+ _dsv41_block_scores_kernel[(R, workers)](
+ scores, live, out, scores.stride(0), out.stride(0), nb,
+ BLOCK_SIZE=block_size, TILE=BLOCK_SCORE_TILE, N_WORKERS=workers, num_warps=4,
+ )
+ return out, live_blocks
+
+
+def dsv41_candidate_blocks(scores: torch.Tensor, live: torch.Tensor, topk_blocks: int, block_size: int) -> torch.Tensor:
+ """Level one of the Hierarchical Sparse Indexer over ``[R, T]`` scores: the ``topk_blocks`` best
+ live blocks (the newest always among them) expanded to the compact candidate list ``[R,
+ topk_blocks * block_size]`` of compressed positions -- ascending, ``-1`` for positions at or past
+ ``live`` and for unfilled blocks, all in the tail (a sorted valid prefix)."""
+ R, T = scores.shape
+ blk, live_blocks = dsv41_block_scores(scores, live, block_size)
+ keep = dsv41_topk(blk, live_blocks, min(topk_blocks, blk.shape[1])) # [R, KB] ascending, -1 tail
+ pos = keep.to(torch.int64).unsqueeze(-1) * block_size + torch.arange(block_size, device=scores.device)
+ dead = (keep < 0).unsqueeze(-1) | (pos >= live.to(torch.int64).view(R, 1, 1))
+ out = torch.where(dead, -1, pos).flatten(1).to(torch.int32)
+ if out.shape[1] < topk_blocks * block_size:
+ out = torch.nn.functional.pad(out, (0, topk_blocks * block_size - out.shape[1]), value=-1)
+ return out
+
+
+__all__ = ["dsv41_topk", "dsv41_block_scores", "dsv41_candidate_blocks", "topk_plan", "CHUNK", "FINAL_MAX", "MAX_K"]
diff --git a/python/freetoken/kernel/triton/e4m3_compat.py b/python/freetoken/kernel/triton/e4m3_compat.py
index 6ea0c753..14a7acd8 100644
--- a/python/freetoken/kernel/triton/e4m3_compat.py
+++ b/python/freetoken/kernel/triton/e4m3_compat.py
@@ -109,6 +109,20 @@ def e4m3_u8_to_f16_x128(v):
return h.to(tl.float16, bitcast=True) * 32768.0
+@jit
+def e4m3_f32_to_u8(x):
+ """Encode an fp32 value that already sits on the e4m3 grid (see :func:`round_e4m3`, |x| <= 448)
+ into its e4m3 bit pattern (uint8). Normals: fp32 exponent rebased 127 -> 7 with the top three
+ mantissa bits; subnormals (|x| < 2^-6, grid 2^-9): mantissa = |x| / 2^-9, exponent field 0."""
+ b = x.to(tl.uint32, bitcast=True)
+ sign = ((b >> 31) & 1).to(tl.uint8) << 7
+ exp = ((b >> 23) & 0xFF).to(tl.int32) - 127 + 7
+ man = ((b >> 20) & 0x7).to(tl.uint8)
+ normal = (exp.to(tl.uint8) << 3) | man
+ sub = (tl.abs(x) * 512.0).to(tl.uint8) # 0..7 on the 2^-9 grid
+ return sign | tl.where(tl.abs(x) >= 0.015625, normal, sub)
+
+
@jit
def round_e4m3(x):
"""Round fp32 onto the e4m3 value grid (RNE), fp32 -> fp32, in a SINGLE
diff --git a/python/freetoken/kernel/triton/mhc.py b/python/freetoken/kernel/triton/mhc.py
index 0782efe3..876643c1 100644
--- a/python/freetoken/kernel/triton/mhc.py
+++ b/python/freetoken/kernel/triton/mhc.py
@@ -27,7 +27,7 @@ def _mhc_stage1_kernel(
H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,
SPLIT: tl.constexpr, # hidden elems per split (multiple of BLOCK_H)
BLOCK_H: tl.constexpr,
- NS: tl.constexpr,
+ NS: tl.constexpr, # splits per token (the grid's second axis; need not be a power of two)
HAS_POST: tl.constexpr,
):
"""Split-K stage of the fused mHC: each program owns one hidden slice of one
@@ -79,24 +79,24 @@ def _mhc_stage1_kernel(
@triton.jit
-def _mhc_stage2_kernel(
- sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,
+def _mhc_gates(
+ t, sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,
post_out_ptr, comb_out_ptr, pre_out_ptr,
rms_eps, hc_eps, post_mult,
SINKHORN: tl.constexpr,
H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,
- NS: tl.constexpr,
+ NS: tl.constexpr, BLK_NS: tl.constexpr,
):
- """Reduce the split partials and run the tiny gate math (sigmoid gates,
- row-softmax + in-register 4x4 Sinkhorn); emits pre gates for stage 3."""
- t = tl.program_id(0).to(tl.int64)
+ """Reduce token ``t``'s split partials and run the tiny gate math (sigmoid gates,
+ row-softmax + in-register 4x4 Sinkhorn); writes post / comb / pre for the next sublayer."""
offs_mix = tl.arange(0, BLK_MIX)
mix_mask = offs_mix < MIX
- offs_s = tl.arange(0, NS)
+ offs_s = tl.arange(0, BLK_NS)
+ s_mask = offs_s < NS
- sqsum = tl.sum(tl.load(sq_part_ptr + t * NS + offs_s))
+ sqsum = tl.sum(tl.load(sq_part_ptr + t * NS + offs_s, mask=s_mask, other=0.0))
acc = tl.sum(
- tl.load(mix_part_ptr + (t * NS + offs_s)[:, None] * BLK_MIX + offs_mix[None, :]),
+ tl.load(mix_part_ptr + (t * NS + offs_s)[:, None] * BLK_MIX + offs_mix[None, :], mask=s_mask[:, None], other=0.0),
axis=0,
)
inv_rms = tl.math.rsqrt(sqsum / (N * H) + rms_eps)
@@ -142,6 +142,23 @@ def _mhc_stage2_kernel(
tl.store(comb_out_ptr + t * N * N + offs_n2[:, None] * N + offs_n2[None, :], comb)
+
+def _mhc_stage2_kernel(
+ sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,
+ post_out_ptr, comb_out_ptr, pre_out_ptr,
+ rms_eps, hc_eps, post_mult,
+ SINKHORN: tl.constexpr,
+ H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,
+ NS: tl.constexpr, BLK_NS: tl.constexpr,
+):
+ """Gate math alone (the two-pass variant, whose stage 3 consumes these pre gates)."""
+ _mhc_gates(
+ tl.program_id(0).to(tl.int64), sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr, post_out_ptr, comb_out_ptr, pre_out_ptr,
+ rms_eps, hc_eps, post_mult, SINKHORN, H, N, MIX, BLK_MIX, NS, BLK_NS,
+ )
+
+
@triton.jit
def _mhc_stage3_kernel(
res_out_ptr, pre_ptr, li_out_ptr,
@@ -161,7 +178,46 @@ def _mhc_stage3_kernel(
tl.store(li_out_ptr + t * H + offs_h, li.to(li_out_ptr.dtype.element_ty), mask=h_mask)
-def mhc_fused_post_pre_triton(
+def _mhc_stage23_kernel(
+ res_out_ptr, pre_ptr, li_out_ptr,
+ sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr,
+ post_out_ptr, comb_out_ptr, pre_out_ptr,
+ rms_eps, hc_eps, post_mult,
+ SINKHORN: tl.constexpr,
+ H: tl.constexpr, N: tl.constexpr, MIX: tl.constexpr, BLK_MIX: tl.constexpr,
+ NS: tl.constexpr, BLK_NS: tl.constexpr, BLOCK_H: tl.constexpr,
+):
+ """Single-pass mHC: the layer input mixes with the PREVIOUS sublayer's pre gates, so the
+ gate math for the next sublayer (stage 2) does not gate stage 3 -- one launch does both:
+ every program mixes one hidden chunk, program 0 of each token also reduces the split
+ partials and writes the next sublayer's gates."""
+ t = tl.program_id(0).to(tl.int64)
+ hb = tl.program_id(1)
+ offs_h = hb * BLOCK_H + tl.arange(0, BLOCK_H)
+ h_mask = offs_h < H
+ offs_n = tl.arange(0, N)
+ pre = tl.load(pre_ptr + t * N + offs_n)
+ li = tl.zeros([BLOCK_H], dtype=tl.float32)
+ for n in tl.static_range(N):
+ r = tl.load(res_out_ptr + (t * N + n) * H + offs_h, mask=h_mask, other=0.0).to(tl.float32)
+ li += tl.sum(tl.where(offs_n == n, pre, 0.0)) * r
+ tl.store(li_out_ptr + t * H + offs_h, li.to(li_out_ptr.dtype.element_ty), mask=h_mask)
+ if hb == 0:
+ _mhc_gates(
+ t, sq_part_ptr, mix_part_ptr, scale_ptr, base_ptr, post_out_ptr, comb_out_ptr, pre_out_ptr,
+ rms_eps, hc_eps, post_mult, SINKHORN, H, N, MIX, BLK_MIX, NS, BLK_NS,
+ )
+
+
+def _stage1_hidden_tile(h: int) -> int:
+ """The stage-1 hidden split. It fixes the fp32 reduction order over the hidden width, so it
+ depends on the width alone and results reproduce across processes, machines and chunk sizes.
+ 128 gives 32 / 40 programs per token at widths 4096 / 5120."""
+ return min(128, triton.next_power_of_2(h))
+
+
+def _mhc_fused_launch(
x: torch.Tensor,
residual: torch.Tensor,
post_mix: torch.Tensor | None,
@@ -173,11 +229,12 @@ def mhc_fused_post_pre_triton(
hc_eps: float,
post_mult: float,
sinkhorn_repeat: int,
-) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
- """Fused hc_post (skipped when ``post_mix is None``) + hc_pre. Three-stage
- split-K: the GEMV/sq-sum reduction fans out over NS hidden slices. Returns
- (residual_new [T,N,H] bf16, post [T,N,1] fp32, comb [T,N,N] fp32,
- layer_input [T,H] bf16)."""
+ pre_mix_in: torch.Tensor | None,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+ """The three stages behind both variants. ``pre_mix_in`` selects single-pass: the
+ layer input is mixed with it instead of with the pre gates stage 2 just computed
+ (those are still returned, for the next sublayer). Returns (residual_new, post,
+ comb, pre_new [T,N] fp32, layer_input)."""
t, n, h = residual.shape
mix = 2 * n + n * n
assert fn.shape == (mix, n * h) and fn.dtype == torch.float32
@@ -190,12 +247,7 @@ def mhc_fused_post_pre_triton(
comb_out = torch.empty(t, n, n, dtype=torch.float32, device=dev)
li_out = torch.empty(t, h, dtype=residual.dtype, device=dev)
- block_h = min(512, triton.next_power_of_2(h))
- # NS feeds a tl.arange in stage 2 -> keep it a power of two.
- ns = 1
- while ns * 2 <= min(16, h // block_h):
- ns *= 2
- split = triton.cdiv(triton.cdiv(h, ns), block_h) * block_h
+ block_h = split = _stage1_hidden_tile(h)
ns = triton.cdiv(h, split)
blk_mix = triton.next_power_of_2(mix)
sq_part = torch.empty(t, ns, dtype=torch.float32, device=dev)
@@ -213,20 +265,83 @@ def mhc_fused_post_pre_triton(
HAS_POST=has_post,
num_warps=4, num_stages=2,
)
- _mhc_stage2_kernel[(t,)](
- sq_part, mix_part, hc_scale, hc_base,
- post_out, comb_out, pre_out,
- rms_eps, hc_eps, post_mult,
- SINKHORN=sinkhorn_repeat,
- H=h, N=n, MIX=mix, BLK_MIX=blk_mix, NS=ns,
- num_warps=1,
+ block_h3 = min(1024, triton.next_power_of_2(h))
+ if pre_mix_in is None:
+ # two-pass: stage 3 mixes with the gates stage 2 just produced
+ _mhc_stage2_kernel[(t,)](
+ sq_part, mix_part, hc_scale, hc_base,
+ post_out, comb_out, pre_out,
+ rms_eps, hc_eps, post_mult,
+ SINKHORN=sinkhorn_repeat,
+ H=h, N=n, MIX=mix, BLK_MIX=blk_mix, NS=ns, BLK_NS=triton.next_power_of_2(ns),
+ num_warps=1,
+ )
+ _mhc_stage3_kernel[(t, triton.cdiv(h, block_h3))](
+ res_out, pre_out, li_out,
+ H=h, N=n, BLOCK_H=block_h3,
+ num_warps=4,
+ )
+ else:
+ # single-pass: the mix uses the previous sublayer's gates, so stages 2 and 3 share a launch
+ assert pre_mix_in.shape == (t, n), (pre_mix_in.shape, (t, n))
+ _mhc_stage23_kernel[(t, triton.cdiv(h, block_h3))](
+ res_out, pre_mix_in.to(torch.float32).contiguous(), li_out,
+ sq_part, mix_part, hc_scale, hc_base,
+ post_out, comb_out, pre_out,
+ rms_eps, hc_eps, post_mult,
+ SINKHORN=sinkhorn_repeat,
+ H=h, N=n, MIX=mix, BLK_MIX=blk_mix, NS=ns, BLK_NS=triton.next_power_of_2(ns), BLOCK_H=block_h3,
+ num_warps=4,
+ )
+ return res_out, post_out.view(t, n, 1), comb_out, pre_out, li_out
+
+
+def mhc_fused_post_pre_triton(
+ x: torch.Tensor,
+ residual: torch.Tensor,
+ post_mix: torch.Tensor | None,
+ comb_mix: torch.Tensor | None,
+ fn: torch.Tensor,
+ hc_scale: torch.Tensor,
+ hc_base: torch.Tensor,
+ rms_eps: float,
+ hc_eps: float,
+ post_mult: float,
+ sinkhorn_repeat: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Fused hc_post (skipped when ``post_mix is None``) + hc_pre. Three-stage
+ split-K: the GEMV/sq-sum reduction fans out over NS hidden slices. Returns
+ (residual_new [T,N,H] bf16, post [T,N,1] fp32, comb [T,N,N] fp32,
+ layer_input [T,H] bf16)."""
+ res_out, post_out, comb_out, _pre_out, li_out = _mhc_fused_launch(
+ x, residual, post_mix, comb_mix, fn, hc_scale, hc_base,
+ rms_eps, hc_eps, post_mult, sinkhorn_repeat, pre_mix_in=None,
)
- _mhc_stage3_kernel[(t, triton.cdiv(h, 1024))](
- res_out, pre_out, li_out,
- H=h, N=n, BLOCK_H=min(1024, triton.next_power_of_2(h)),
- num_warps=4,
+ return res_out, post_out, comb_out, li_out
+
+
+def mhc_fused_post_pre_single_pass_triton(
+ x: torch.Tensor,
+ residual: torch.Tensor,
+ post_mix: torch.Tensor | None,
+ comb_mix: torch.Tensor | None,
+ pre_mix_in: torch.Tensor,
+ fn: torch.Tensor,
+ hc_scale: torch.Tensor,
+ hc_base: torch.Tensor,
+ rms_eps: float,
+ hc_eps: float,
+ post_mult: float,
+ sinkhorn_repeat: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Single-pass mHC (DeepSeek-V4.1): same fused stages, but the layer input is mixed
+ with the PREVIOUS sublayer's ``pre_mix_in`` and this sublayer's pre gates come back
+ as ``pre_new`` for the next one. Returns (residual_new, post [T,N,1], comb [T,N,N],
+ pre_new [T,N] fp32, layer_input)."""
+ return _mhc_fused_launch(
+ x, residual, post_mix, comb_mix, fn, hc_scale, hc_base,
+ rms_eps, hc_eps, post_mult, sinkhorn_repeat, pre_mix_in=pre_mix_in,
)
- return res_out, post_out.view(t, n, 1), comb_out, li_out
-__all__ = ["mhc_fused_post_pre_triton"]
+__all__ = ["mhc_fused_post_pre_triton", "mhc_fused_post_pre_single_pass_triton"]
diff --git a/python/freetoken/kvcache/__init__.py b/python/freetoken/kvcache/__init__.py
index 1792da2b..e0cf606d 100644
--- a/python/freetoken/kvcache/__init__.py
+++ b/python/freetoken/kvcache/__init__.py
@@ -34,7 +34,7 @@ def resolve_pool_class(model_config: ModelConfig) -> type[BaseKVCachePool]:
specs_fn = getattr(model_config, "kv_cache_group_specs", None)
if specs_fn is None:
if getattr(model_config, "dsv4_args", None) is not None:
- from .dsv4_paged_pool import DSV4PagedKVCache
+ from .dsv4.v4_pool import DSV4PagedKVCache
return DSV4PagedKVCache
from .mha_pool import MHAKVCache
@@ -43,9 +43,13 @@ def resolve_pool_class(model_config: ModelConfig) -> type[BaseKVCachePool]:
specs = list(specs_fn())
types = {spec.attn_type for spec in specs}
if AttnType.DSV4 in types:
- from .dsv4_paged_pool import DSV4PagedKVCache
+ from .dsv4.v4_pool import DSV4PagedKVCache
return DSV4PagedKVCache
+ if AttnType.DSV41 in types:
+ from .dsv4.v41_pool import DSV41PagedKVCache
+
+ return DSV41PagedKVCache
if AttnType.SWA in types:
from .hybrid_swa_pool import HybridSWAKVCache
@@ -80,9 +84,9 @@ def create_kv_pool(config, num_pages: int, device: torch.device, dtype: torch.dt
"""Build the engine's KV pool for ``num_pages`` USABLE pages (the dummy page and every
secondary tier -- window pool, index slab, state rings -- are derived here or inside
the pool). Single factory entry for all pool families, DSV4 included."""
- from .dsv4_cost_model import _dsv4_pool_sizes
+ from .dsv4.v4_cost_model import _dsv4_pool_sizes
from .hybrid_swa_pool import _naive_swa_num_tokens, _swa_paged_num_tokens
- from .dsv4_paged_pool import DSV4PagedKVCache
+ from .dsv4.v4_pool import DSV4PagedKVCache
model_config = config.model_config
if resolve_pool_class(model_config) is DSV4PagedKVCache:
@@ -99,6 +103,21 @@ def create_kv_pool(config, num_pages: int, device: torch.device, dtype: torch.dt
)
pool._init_paged_state(config.max_running_req, config.cache_type != "naive")
return pool
+ if getattr(model_config, "dsv41_args", None) is not None:
+ from .dsv4.v41_cost_model import _dsv41_pool_sizes
+ from .dsv4.v41_pool import DSV41PagedKVCache
+
+ # Same route as DSV4: the generic CacheManager over the shared page table; the pool is the
+ # swa_pool plug-in (window tier + per-source packed main / index pools + state rings).
+ pool = DSV41PagedKVCache(
+ sizes=_dsv41_pool_sizes(config, num_pages + 1), # +1 for dummy page
+ args=model_config.dsv41_args,
+ device=device,
+ dtype=dtype,
+ n_scratch=config.max_running_req + 1,
+ )
+ pool._init_paged_state(config.max_running_req, config.cache_type != "naive")
+ return pool
num_swa_tokens = None
# Both the naive and radix SWA paths share the global-paged swa pool; radix sizes it by
diff --git a/python/freetoken/kvcache/base.py b/python/freetoken/kvcache/base.py
index 95669e8c..569da1cb 100644
--- a/python/freetoken/kvcache/base.py
+++ b/python/freetoken/kvcache/base.py
@@ -35,6 +35,14 @@ def spec_kv_bytes_per_token(spec, config) -> int:
return per_token + spec.index_head_dim * spec.num_index_layers * 2 // spec.index_ratio
+@dataclass(frozen=True)
+class WindowPoolSpec:
+ """Window resize units and minimum usable capacity, excluding reserved pages."""
+
+ page_size: int
+ min_pages: int
+
+
class BaseKVCachePool(ABC):
"""
Base class for key-value caches.
@@ -45,6 +53,15 @@ class BaseKVCachePool(ABC):
# model re-bound after a rebuild; the engine asks before it resizes.
needs_rebind_on_rebuild: ClassVar[bool] = False
+ @classmethod
+ def window_spec(cls, config) -> WindowPoolSpec | None:
+ return None
+
+ @property
+ def window_pages(self) -> int:
+ """Current usable window pages in the units declared by window_spec."""
+ return 0
+
# ---- sizing/cost classmethods: run BEFORE the pool exists (startup budget solve,
# --moe-cache-auto). The engine measures memory and passes bytes in; each pool family
# implements kv_cost for ITS OWN buffers only (the engine sums families, e.g. adds
diff --git a/python/freetoken/kvcache/cache_status.py b/python/freetoken/kvcache/cache_status.py
index 6aa13e7f..5b531ce2 100644
--- a/python/freetoken/kvcache/cache_status.py
+++ b/python/freetoken/kvcache/cache_status.py
@@ -3,13 +3,14 @@ from __future__ import annotations
from typing import Any, Dict
+def window_pool_spec(config):
+ from . import resolve_pool_class
+
+ return resolve_pool_class(config.model_config).window_spec(config)
+
+
def _supports_swa_ratio(config) -> bool:
- """Whether ``swa_full_tokens_ratio`` sizes a separate window pool for this model -- DSV4
- (always) or a radix-SWA model (Gemma). Gates the ratio in telemetry and rebuild."""
- mc = config.model_config
- if mc.dsv4_args is not None:
- return True
- return mc.has_swa_attention and config.cache_type == "swa_radix"
+ return window_pool_spec(config) is not None
def compute_cache_unit_bytes(engine: "Engine") -> Dict[str, int]:
@@ -120,18 +121,8 @@ def compute_cache_floors(engine: "Engine") -> Dict[str, int]:
return int(_linear_pool_min_slots(config) - 1)
def _swa() -> int:
- # Report usable tokens, matching num_swa_pages in rebuild requests.
- # DSV4's physical floor includes a dummy page; the SWA floor already excludes slot 0.
- from .dsv4_cost_model import _dsv4_window_floor_pages
- from .hybrid_swa_pool import _swa_pool_floor
-
- mc = config.model_config
- if mc.dsv4_args is not None:
- P = mc.dsv4_args.window_size
- return int((_dsv4_window_floor_pages(config, P) - 1) * P)
- if not (mc.has_swa_attention and config.cache_type == "swa_radix"):
- return 0
- return int(_swa_pool_floor(config))
+ spec = window_pool_spec(config)
+ return spec.page_size * spec.min_pages if spec is not None else 0
for key, fn in (("kv_tokens", _kv), ("moe_experts", _moe), ("mamba_slots", _mamba),
("swa_tokens", _swa)):
@@ -157,19 +148,10 @@ def compute_cache_pools(engine: "Engine") -> Dict[str, int]:
pools["num_pages"] = int(engine.num_pages or 0)
if config is not None:
pools["page_size"] = int(config.page_size or 0)
- # Window pool: its own page unit (swa_page_size -- DSV4 windows are P-token pages,
- # radix-SWA is token-granular page_size 1) and the concrete current size in that unit
- # (num_swa_pages, usable count). Same source as the scheduler's _current_cache_geometry.
- # Both 0 for models without a window pool. Lets a client denominate the swa control.
- mc = config.model_config
- if mc.dsv4_args is not None:
- pools["swa_page_size"] = int(mc.dsv4_args.window_size or 0)
- sizes = getattr(engine.kv_cache, "sizes", None) # usable = physical minus dummy
- if sizes is not None:
- pools["num_swa_pages"] = max(0, int(sizes.n_win_pages) - 1)
- elif mc.has_swa_attention and config.cache_type == "swa_radix":
- pools["swa_page_size"] = 1 # usable = pool tokens minus the slot-0 sentinel
- pools["num_swa_pages"] = max(0, int(getattr(engine.kv_cache, "swa_num_tokens", 0) or 0) - 1)
+ spec = window_pool_spec(config)
+ if spec is not None:
+ pools["swa_page_size"] = spec.page_size
+ pools["num_swa_pages"] = max(0, int(engine.kv_cache.window_pages))
moe = engine.moe_offload_cache
if moe is not None:
pools["moe_cache_size"] = int(moe.cache_size or 0)
diff --git a/python/freetoken/kvcache/dsa_pool.py b/python/freetoken/kvcache/dsa_pool.py
index 13225cec..d7c36fd7 100644
--- a/python/freetoken/kvcache/dsa_pool.py
+++ b/python/freetoken/kvcache/dsa_pool.py
@@ -2,7 +2,7 @@
``MLAKVCache`` is the MHA pool's sibling for latent-KV models: ONE slab holding the
per-token latent ``ckv (kv_lora_rank) | kpe (qk_rope_head_dim)`` -- there is no
-separate V (``v_cache`` aliases ``k_cache``, same convention as dsv4_paged_pool's
+separate V (``v_cache`` aliases ``k_cache``, same convention as dsv4/v4_pool's
single-latent tiers). ``DSAKVCache`` extends it with the DeepSeek-Sparse-Attention
index-key slab: one ``index_head_dim``-wide bf16 row per token per full-indexer
layer, addressed by the SAME physical rows as the latent slab (GLM-5.2, page 1;
@@ -75,7 +75,7 @@ class MLAKVCache(BaseKVCachePool):
)
def v_cache(self, layer_id: int) -> torch.Tensor:
- # MLA: K == V (single latent); same buffer, dsv4_paged_pool precedent.
+ # MLA: K == V (single latent); same buffer, dsv4/v4_pool precedent.
return self.k_cache(layer_id)
def latent_rows(self, layer_id: int) -> torch.Tensor:
diff --git a/python/freetoken/kvcache/dsv4/__init__.py b/python/freetoken/kvcache/dsv4/__init__.py
new file mode 100644
index 00000000..7531470f
--- /dev/null
+++ b/python/freetoken/kvcache/dsv4/__init__.py
@@ -0,0 +1 @@
+"""Paged KV pools for the DeepSeek-V4 family."""
diff --git a/python/freetoken/kvcache/dsv4/v41_cost_model.py b/python/freetoken/kvcache/dsv4/v41_cost_model.py
new file mode 100644
index 00000000..4db1fb18
--- /dev/null
+++ b/python/freetoken/kvcache/dsv4/v41_cost_model.py
@@ -0,0 +1,216 @@
+"""Cost model + per-tier sizing for the DSV41 paged KV pool (DeepSeek-V4.1).
+
+Same shape as ``v4_cost_model``: one bytes-per-P-token number for the budget division, and
+independent per-tier sizing from the anchor ``full_token = num_pages * P``. The tiers:
+
+* window pool, every layer -- ``swa_ratio`` of the full history, packed fp8 rows
+* main KV pool, per kv source -- ``full_token // ratio`` packed fp4 rows
+* index-key pool, per kv source -- ``full_token // ratio`` packed fp4 rows
+* compress-state ring, ratio>1 sources -- ``ring_size`` fp32 ``kv|score`` slots per WINDOW page
+
+Sizing reads ``ModelConfig.dsv41_args`` by attribute, like ``v4_cost_model`` reads ``dsv4_args``:
+the kvcache package never imports the model package. All byte widths come from the tier row formats
+below, so there is exactly one place that knows what a row costs.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from .v41_row_format import FP4_E4M3_B16, FP4_E8M0_B32, FP8_E8M0_B32
+from .window_tier import reserved_window_pages
+
+_INT64_BYTES = 8
+_FP32_BYTES = 4
+
+WIN_FMT = FP8_E8M0_B32
+MAIN_FMT = FP4_E4M3_B16
+IDX_FMT = FP4_E8M0_B32
+
+
+def _win_row_bytes(args) -> int:
+ return WIN_FMT.row_bytes(args.head_dim)
+
+
+def _main_row_bytes(args) -> int:
+ return MAIN_FMT.row_bytes(args.head_dim)
+
+
+def _idx_row_bytes(args) -> int:
+ return IDX_FMT.row_bytes(args.index_head_dim)
+
+
+def _state_bytes(args) -> int:
+ """One compress-state ring slot: fp32 ``kv | score`` of the latent width."""
+ return 2 * args.head_dim * _FP32_BYTES
+
+
+def private_window_layer_ids(args) -> tuple[int, ...]:
+ """Layers whose window KV is REQUEST-PRIVATE: kept in a per-page-table-row ring of ``window``
+ slots (slot = row * window + pos % window) instead of the shared, page-bound window pool, so
+ it never lands in radix-shared pages. The decoder layers under Decoder SWA Bounded Replay:
+ their KV is recomputed per request over the prompt's last window (positions before it are
+ never computed), so a shared page could hold rows another request never wrote. Exact mode
+ computes every position and keeps the decoder on the shared pool."""
+ if args.swa_decoder_replay == "exact":
+ return ()
+ return tuple(range(args.decoder_start_layer, args.n_layers))
+
+
+@dataclass
+class DSV41PoolSizes:
+ """Per-tier row counts derived from the budget anchor ``full_token``; the global tiers are
+ keyed by kv-source layer id."""
+
+ P: int
+ swa_ratio: float
+ full_token: int
+ n_win_slots: int
+ n_win_pages: int
+ main_rows: dict[int, int] = field(default_factory=dict)
+ idx_rows: dict[int, int] = field(default_factory=dict)
+ state_slots: dict[int, int] = field(default_factory=dict) # ratio>1 sources only
+
+
+def dsv41_pool_sizes(
+ num_pages: int, args, swa_ratio: float, P: int, n_win_pages: int | None = None
+) -> DSV41PoolSizes:
+ full_token = num_pages * P
+ if n_win_pages is None:
+ n_win_pages = (round(swa_ratio * full_token) + P - 1) // P
+ n_win_pages = min(n_win_pages, num_pages)
+ sizes = DSV41PoolSizes(P=P, swa_ratio=swa_ratio, full_token=full_token, n_win_slots=n_win_pages * P, n_win_pages=n_win_pages)
+ for src in args.backbone_kv_sources:
+ ratio = args.compress_ratios[src]
+ sizes.main_rows[src] = full_token // ratio
+ sizes.idx_rows[src] = full_token // ratio
+ if ratio > 1:
+ sizes.state_slots[src] = n_win_pages * ratio # ring size == ratio
+ return sizes
+
+
+def dsv41_pool_bytes(sizes: DSV41PoolSizes, args, n_scratch: int = 1) -> int:
+ """Exact bytes a ``DSV41PagedKVCache`` built from ``sizes`` allocates (mirror of ``total_bytes``)."""
+ n_private = len(private_window_layer_ids(args))
+ total = (args.n_layers - n_private) * sizes.n_win_slots * _win_row_bytes(args)
+ total += n_private * n_scratch * sizes.P * _win_row_bytes(args) # per-request rings
+ total += (sizes.full_token + 1) * _INT64_BYTES # full_to_window (+ sentinel row)
+ for src in args.backbone_kv_sources:
+ total += (sizes.main_rows[src] + n_scratch) * _main_row_bytes(args)
+ total += (sizes.idx_rows[src] + n_scratch) * _idx_row_bytes(args)
+ if src in sizes.state_slots:
+ total += (sizes.state_slots[src] + 1) * _state_bytes(args)
+ return int(total)
+
+
+def dsv41_cache_per_page(args, swa_ratio: float, P: int) -> int:
+ """Marginal bytes per P-token page across all tiers (window scaled by ``swa_ratio``)."""
+ n_shared = args.n_layers - len(private_window_layer_ids(args))
+ total = n_shared * round(swa_ratio * P) * _win_row_bytes(args)
+ for src in args.backbone_kv_sources:
+ ratio = args.compress_ratios[src]
+ total += (P // ratio) * (_main_row_bytes(args) + _idx_row_bytes(args))
+ if ratio > 1:
+ total += round(swa_ratio * ratio) * _state_bytes(args)
+ return int(total)
+
+
+def dsv41_kv_unit_bytes(args, P: int) -> int:
+ """FULL-tier bytes per full-history token (main + index pools + the mapping); the slider's
+ ``kv_bytes_per_token``. 890 B/token for DeepSeek-V4.1-Flash before the mapping."""
+ per_page = P * _INT64_BYTES
+ for src in args.backbone_kv_sources:
+ per_page += (P // args.compress_ratios[src]) * (_main_row_bytes(args) + _idx_row_bytes(args))
+ return -(-per_page // P)
+
+
+def dsv41_window_unit_bytes(args, P: int) -> int:
+ """WINDOW-tier bytes per window token (sliding KV on every shared-window layer + the state rings)."""
+ per_page = (args.n_layers - len(private_window_layer_ids(args))) * P * _win_row_bytes(args)
+ for src in args.backbone_kv_sources:
+ ratio = args.compress_ratios[src]
+ if ratio > 1:
+ per_page += ratio * _state_bytes(args)
+ return -(-per_page // P)
+
+
+def dsv41_solve_num_pages(
+ available_bytes: int, args, swa_ratio: float, floor_win_pages: int, P: int, n_scratch: int = 1
+) -> DSV41PoolSizes:
+ """Largest budget-respecting pool; the window floor is honored in pages and the total is
+ byte-checked. Raises ``ValueError`` when even the minimal pool does not fit."""
+
+ def _sizes(num: int) -> DSV41PoolSizes:
+ win = max(floor_win_pages, (round(swa_ratio * num * P) + P - 1) // P)
+ return dsv41_pool_sizes(num, args, swa_ratio, P, n_win_pages=win)
+
+ lo = max(floor_win_pages, 2)
+ if dsv41_pool_bytes(_sizes(lo), args, n_scratch) > available_bytes:
+ raise ValueError(
+ f"DSV41 KV budget {available_bytes} bytes cannot fit the minimal pool ({lo} pages incl. the "
+ f"window working-set floor {floor_win_pages}); raise memory_ratio or lower max_running_req/max_seq_len"
+ )
+ hi = max(lo, available_bytes // max(1, dsv41_cache_per_page(args, 0.0, P)))
+ while dsv41_pool_bytes(_sizes(hi), args, n_scratch) <= available_bytes:
+ hi *= 2
+ while lo < hi - 1:
+ mid = (lo + hi) // 2
+ if dsv41_pool_bytes(_sizes(mid), args, n_scratch) <= available_bytes:
+ lo = mid
+ else:
+ hi = mid
+ return _sizes(lo)
+
+
+def dsv41_auto_cost_model(args, swa_ratio: float, floor_win_pages: int, P: int, n_scratch: int = 1):
+ """Affine ``(cache_per_page, fixed_cache_size, min_reserve_tokens)`` for the MoE-first auto planner."""
+ per_page = dsv41_cache_per_page(args, swa_ratio, P) + P * _INT64_BYTES
+ n0 = max(floor_win_pages, 2)
+ win0 = max(floor_win_pages, (round(swa_ratio * n0 * P) + P - 1) // P)
+ base = dsv41_pool_bytes(dsv41_pool_sizes(n0, args, swa_ratio, P, n_win_pages=win0), args, n_scratch)
+ # The engine combines this structural floor with the configured KV reserve.
+ return per_page, max(0, base - n0 * per_page), n0 * P
+
+
+# ---- config-facing sizing (EngineConfig in, sizes out) ----
+
+
+def _dsv41_swa_ratio(config) -> float:
+ return float(config.swa_full_tokens_ratio)
+
+
+def _dsv41_window_floor_pages(config, args) -> int:
+ """Minimum window pages: one prefill chunk's reach (capped at 8 pages) + the running requests'
+ working set (see ``reserved_window_pages``)."""
+ P = args.window_size
+ prefill_reach_pages = (config.max_seq_len + P - 1) // P
+ radix = config.cache_type != "naive"
+ return min(prefill_reach_pages, 8) + reserved_window_pages(config.max_running_req, radix)
+
+
+def _dsv41_pool_sizes(config, num_pages: int, num_swa_pages: int | None = None) -> DSV41PoolSizes:
+ """Sizes for ``num_pages`` PHYSICAL pages (dummy included). Window precedence: an explicit
+ ``num_swa_pages`` > ``config.swa_num_pages_override`` > ``swa_ratio`` x full, floored ONCE in pages."""
+ args = config.model_config.dsv41_args
+ P = args.window_size
+ swa_ratio = _dsv41_swa_ratio(config)
+ floor_pages = _dsv41_window_floor_pages(config, args)
+ target = num_swa_pages if num_swa_pages is not None else config.swa_num_pages_override
+ if target is not None:
+ win = min(num_pages, max(floor_pages, int(target) + 1))
+ else:
+ win = max(floor_pages, (round(swa_ratio * num_pages * P) + P - 1) // P)
+ return dsv41_pool_sizes(num_pages, args, swa_ratio, P, n_win_pages=win)
+
+
+__all__ = [
+ "DSV41PoolSizes",
+ "dsv41_auto_cost_model",
+ "dsv41_cache_per_page",
+ "dsv41_kv_unit_bytes",
+ "dsv41_pool_bytes",
+ "dsv41_pool_sizes",
+ "dsv41_solve_num_pages",
+ "dsv41_window_unit_bytes",
+ "private_window_layer_ids",
+]
diff --git a/python/freetoken/kvcache/dsv4/v41_pool.py b/python/freetoken/kvcache/dsv4/v41_pool.py
new file mode 100644
index 00000000..36d3986a
--- /dev/null
+++ b/python/freetoken/kvcache/dsv4/v41_pool.py
@@ -0,0 +1,292 @@
+"""DSV41 paged KV pool (DeepSeek-V4.1): packed byte tiers over the shared page table.
+
+Tiers, sized from a budget (``v41_cost_model``) not from ``num_requests``:
+
+* ``window_pool[L]`` -- every layer; the P-sliding KV ring in ``win_fmt`` (fp8) rows, page-granular.
+* ``main_pool[src]`` -- per kv-source layer; compressed KV latents in ``main_fmt`` (fp4) rows.
+* ``idx_pool[src]`` -- per kv-source layer; indexer keys in ``idx_fmt`` (fp4) rows.
+* ``state_ring[src]`` -- ratio>1 sources; the per-window-page compress-state ring (fp32 ``kv|score``).
+
+Consumer layers (Reindex / Reuse modes) alias their source's pools: ``main_pool_of(layer)``.
+
+Addressing is the shared ``WindowTierPagedPool``'s (``kvcache/dsv4/window_tier.py``, also under the DSV4
+pool): the page table maps ``(table_idx, pos)`` to a full-token loc; ``full_to_window`` maps that to a
+window slot (page-bound by the free list); the main / index rows are pure arithmetic ``full_loc //
+ratio``; the ring slot derives from the window slot. Scratch rows past every main / index pool's
+arithmetic range give a graph-safe destination for the decode rows whose compressed group did not
+complete this step.
+
+Rows are written through ``pack_rows`` (quantize + scatter) and read inside the attention /
+indexer kernels; ``read_*`` helpers dequantize for tests and torch reference paths.
+"""
+
+from __future__ import annotations
+
+import torch
+
+from freetoken.utils import init_logger
+
+from .v41_cost_model import (
+ IDX_FMT,
+ MAIN_FMT,
+ WIN_FMT,
+ DSV41PoolSizes,
+ _idx_row_bytes,
+ _main_row_bytes,
+ _win_row_bytes,
+ dsv41_kv_unit_bytes,
+ dsv41_window_unit_bytes,
+ private_window_layer_ids,
+)
+from .window_tier import CompressStateRing, WindowTierPagedPool
+
+logger = init_logger(__name__)
+
+
+class DSV41PagedKVCache(WindowTierPagedPool):
+ win_fmt = WIN_FMT
+ main_fmt = MAIN_FMT
+ idx_fmt = IDX_FMT
+
+ def __init__(
+ self,
+ sizes: DSV41PoolSizes,
+ args,
+ device: torch.device,
+ dtype: torch.dtype = torch.bfloat16,
+ n_scratch: int = 1,
+ ) -> None:
+ for fmt, dim in ((self.win_fmt, args.head_dim), (self.main_fmt, args.head_dim), (self.idx_fmt, args.index_head_dim)):
+ fmt.validate_dim(dim)
+ for src in args.backbone_kv_sources:
+ if args.window_size % args.compress_ratios[src]:
+ raise ValueError(f"window {args.window_size} must be a multiple of compress ratio {args.compress_ratios[src]}")
+ self.args = args
+ self.private_window_layer_ids = private_window_layer_ids(args)
+ self.sizes = sizes
+ self._device = device
+ self._dtype = dtype # the model's compute dtype; the tiers themselves are packed bytes
+ self.P = args.window_size
+ self.head_dim = args.head_dim
+ self.index_head_dim = args.index_head_dim
+ self.n_scratch = int(n_scratch)
+ self._paged_params: tuple[int, bool] | None = None
+ self.full_loc_map: torch.Tensor | None = None
+ self._alloc_buffers()
+
+ # ----- buffers -----
+ def _alloc_buffers(self) -> None:
+ sizes, args, dev = self.sizes, self.args, self._device
+ self.full_to_window = torch.full((sizes.full_token + 1,), -1, dtype=torch.int64, device=dev)
+ # shared layers: the page-bound window pool; private layers: one ring of P slots per
+ # page-table row (n_scratch rows, the dummy included), addressed row * P + pos % P
+ self.window_pool: list[torch.Tensor] = [
+ torch.zeros(self.n_scratch * self.P if self.is_private_window(l) else sizes.n_win_slots, _win_row_bytes(args), dtype=torch.uint8, device=dev)
+ for l in range(args.n_layers)
+ ]
+ self.main_pool: dict[int, torch.Tensor] = {}
+ self.idx_pool: dict[int, torch.Tensor] = {}
+ self.state_ring: dict[int, CompressStateRing] = {}
+ self.scratch_base: dict[int, int] = {}
+ for src in args.backbone_kv_sources:
+ rows = sizes.main_rows[src]
+ self.scratch_base[src] = rows
+ self.main_pool[src] = torch.zeros(rows + self.n_scratch, _main_row_bytes(args), dtype=torch.uint8, device=dev)
+ self.idx_pool[src] = torch.zeros(sizes.idx_rows[src] + self.n_scratch, _idx_row_bytes(args), dtype=torch.uint8, device=dev)
+ if src in sizes.state_slots:
+ self.state_ring[src] = CompressStateRing(
+ n_slots=sizes.state_slots[src], ring_size=args.compress_ratios[src], overlap=False,
+ head_dim=args.head_dim, device=dev,
+ )
+
+ def total_bytes(self) -> int:
+ n = self.full_to_window.numel() * self.full_to_window.element_size()
+ n += sum(t.numel() for t in self.window_pool)
+ n += sum(t.numel() for t in self.main_pool.values())
+ n += sum(t.numel() for t in self.idx_pool.values())
+ n += sum(r.buffer.numel() * r.buffer.element_size() for r in self.state_ring.values())
+ return int(n)
+
+ @property
+ def prefix_replay_tokens(self) -> int:
+ """Prompt-tail tokens a prefix hit must leave to the prefill: bounded replay runs the decoder on
+ the prompt's last window from that window's encoder outputs, which no cache keeps."""
+ return self.P if self.private_window_layer_ids else 0
+
+ # ----- private window rings -----
+ def is_private_window(self, layer_id: int) -> bool:
+ return layer_id in self.private_window_layer_ids
+
+ def ring_slots(self, table_rows: torch.Tensor, positions: torch.Tensor) -> torch.Tensor:
+ """Ring slots of ``positions`` (int64, broadcastable with ``table_rows``) for private window layers;
+ negative positions map to ``-1``."""
+ slots = table_rows.to(torch.int64) * self.P + positions % self.P
+ return torch.where(positions < 0, -1, slots)
+
+ # ----- source aliasing -----
+ def source_of(self, layer_id: int) -> int:
+ src = self.args.roles[layer_id].kv_source
+ assert src is not None, f"layer {layer_id} is window-only"
+ return src
+
+ def main_pool_of(self, layer_id: int) -> torch.Tensor:
+ return self.main_pool[self.source_of(layer_id)]
+
+ def idx_pool_of(self, layer_id: int) -> torch.Tensor:
+ return self.idx_pool[self.source_of(layer_id)]
+
+ def ratio_of(self, layer_id: int) -> int:
+ return self.args.roles[layer_id].ratio
+
+ # ----- engine-facing sizing / rebuild surface -----
+ @classmethod
+ def kv_cost(cls, config) -> tuple[int, int, int, int]:
+ from .v41_cost_model import _dsv41_swa_ratio, _dsv41_window_floor_pages, dsv41_auto_cost_model
+
+ args = config.model_config.dsv41_args
+ P = args.window_size
+ per_page, fixed, min_reserve = dsv41_auto_cost_model(
+ args, _dsv41_swa_ratio(config), _dsv41_window_floor_pages(config, args), P, n_scratch=config.max_running_req + 1
+ )
+ return per_page, fixed, config.page_size, min_reserve
+
+ @classmethod
+ def solve_num_pages(cls, config, available_memory: int) -> int:
+ from freetoken.utils import mem_GB
+
+ from .v41_cost_model import (
+ _dsv41_pool_sizes,
+ _dsv41_swa_ratio,
+ _dsv41_window_floor_pages,
+ dsv41_pool_bytes,
+ dsv41_solve_num_pages,
+ )
+
+ args = config.model_config.dsv41_args
+ P = args.window_size
+ num_pages = config.num_page_override
+ if num_pages is None:
+ sizes = dsv41_solve_num_pages(
+ available_memory, args, _dsv41_swa_ratio(config), _dsv41_window_floor_pages(config, args), P,
+ n_scratch=config.max_running_req + 1,
+ )
+ num_pages = sizes.full_token // P - 1 # one physical page is the dummy
+ else:
+ floor = _dsv41_window_floor_pages(config, args)
+ if num_pages < floor:
+ raise ValueError(
+ f"--num-pages {num_pages} ({num_pages * P} tokens) is below the DSV41 window working-set "
+ f"floor {floor} pages ({floor * P} tokens); raise --num-pages or lower max_running_req/max_seq_len"
+ )
+ sizes = _dsv41_pool_sizes(config, num_pages + 1)
+ assert num_pages > 1, "Not enough memory for KV cache, try reducing --num-pages"
+ real = dsv41_pool_bytes(sizes, args, config.max_running_req + 1)
+ logger.info(
+ f"Allocating {num_pages * P} tokens for DSV41 KV cache ({sizes.n_win_pages} window pages), total = {mem_GB(real)}"
+ )
+ return num_pages
+
+ @classmethod
+ def window_spec(cls, config):
+ from ..base import WindowPoolSpec
+ from .v41_cost_model import _dsv41_window_floor_pages
+
+ args = config.model_config.dsv41_args
+ return WindowPoolSpec(args.window_size, _dsv41_window_floor_pages(config, args) - 1)
+
+ @classmethod
+ def min_kv_tokens(cls, config) -> int:
+ from .v41_cost_model import _dsv41_window_floor_pages
+
+ args = config.model_config.dsv41_args
+ return _dsv41_window_floor_pages(config, args) * args.window_size
+
+ def validate_rebuild(
+ self, config, *, num_pages: int | None, target_moe: int, per_expert_bytes: int,
+ baseline_free: int, weights_bytes: int, current_num_pages: int,
+ extra_fixed_bytes: int = 0, extra_note: str = "",
+ num_swa_pages: int | None = None, **targets,
+ ) -> None:
+ from freetoken.engine.cache_budget import net_cache_budget_bytes
+ from freetoken.utils import mem_GB
+
+ from ..base import CacheRebuildRejected
+ from .v41_cost_model import _dsv41_pool_sizes, _dsv41_window_floor_pages, dsv41_pool_bytes
+
+ if num_pages is not None:
+ floor = _dsv41_window_floor_pages(config, self.args)
+ if num_pages < floor:
+ raise CacheRebuildRejected(
+ f"num_pages {num_pages} is below the DSV41 window working-set floor {floor} "
+ f"(max_running_req={config.max_running_req}); admission would deadlock"
+ )
+ if num_pages is not None or num_swa_pages is not None:
+ target_pages = num_pages if num_pages is not None else current_num_pages
+ kv_sizes = _dsv41_pool_sizes(config, target_pages + 1, num_swa_pages=num_swa_pages)
+ else:
+ kv_sizes = self.sizes
+ budget = net_cache_budget_bytes(config.memory_ratio, baseline_free, weights_bytes, 0)
+ need = target_moe * per_expert_bytes + dsv41_pool_bytes(kv_sizes, self.args, config.max_running_req + 1)
+ if need > budget:
+ kv_part = f"kv={num_pages} P-pages" if num_pages is not None else "kv=current pool"
+ raise CacheRebuildRejected(
+ f"requested cache (moe={target_moe} slots, {kv_part}) needs {mem_GB(need)} > budget {mem_GB(budget)}; "
+ "old cache kept, still serving"
+ )
+
+ def rebuild_from_config(self, config, num_pages: int, *, num_swa_pages: int | None = None) -> None:
+ from .v41_cost_model import _dsv41_pool_sizes
+
+ self.rebuild(_dsv41_pool_sizes(config, num_pages + 1, num_swa_pages=num_swa_pages))
+
+ def _drop_buffers(self) -> None:
+ self.window_pool = self.main_pool = self.idx_pool = self.state_ring = None # type: ignore[assignment]
+
+ def unit_bytes(self) -> tuple[int, int]:
+ return dsv41_kv_unit_bytes(self.args, self.P), dsv41_window_unit_bytes(self.args, self.P)
+
+ # ----- writes (quantize + scatter) -----
+ def store_window(self, kv: torch.Tensor, layer_id: int, window_slot: torch.Tensor) -> None:
+ from freetoken.kernel.triton.dsv41.pack import pack_rows
+
+ pack_rows(kv, self.win_fmt, pool=self.window_pool[layer_id], row_ids=window_slot)
+
+ def store_main(self, latent: torch.Tensor, source: int, rows: torch.Tensor) -> None:
+ from freetoken.kernel.triton.dsv41.pack import pack_rows
+
+ pack_rows(latent, self.main_fmt, pool=self.main_pool[source], row_ids=rows)
+
+ def store_index(self, k: torch.Tensor, source: int, rows: torch.Tensor) -> None:
+ from freetoken.kernel.triton.dsv41.pack import pack_rows
+
+ pack_rows(k, self.idx_fmt, pool=self.idx_pool[source], row_ids=rows)
+
+ # ----- reads (gather + dequantize; tests and torch reference paths) -----
+ def read_window(self, layer_id: int, window_slot: torch.Tensor) -> torch.Tensor:
+ from freetoken.kernel.triton.dsv41.pack import unpack_rows
+
+ return unpack_rows(self.window_pool[layer_id], self.win_fmt, self.head_dim, window_slot)
+
+ def read_main(self, source: int, rows: torch.Tensor) -> torch.Tensor:
+ from freetoken.kernel.triton.dsv41.pack import unpack_rows
+
+ return unpack_rows(self.main_pool[source], self.main_fmt, self.head_dim, rows)
+
+ def read_index(self, source: int, rows: torch.Tensor) -> torch.Tensor:
+ from freetoken.kernel.triton.dsv41.pack import unpack_rows
+
+ return unpack_rows(self.idx_pool[source], self.idx_fmt, self.index_head_dim, rows)
+
+ # ----- compress-state ring accessors -----
+ def get_state(self, source: int, state_loc: torch.Tensor) -> torch.Tensor:
+ return self.state_ring[source].get(state_loc)
+
+ def set_state(self, source: int, state_loc: torch.Tensor, kv_score: torch.Tensor) -> None:
+ self.state_ring[source].set(state_loc, kv_score)
+
+ @property
+ def num_layers(self) -> int:
+ return self.args.n_layers
+
+
+__all__ = ["DSV41PagedKVCache"]
diff --git a/python/freetoken/kvcache/dsv4/v41_row_format.py b/python/freetoken/kvcache/dsv4/v41_row_format.py
new file mode 100644
index 00000000..f273be4f
--- /dev/null
+++ b/python/freetoken/kvcache/dsv4/v41_row_format.py
@@ -0,0 +1,69 @@
+"""RowFormat: how one DeepSeek-V4.1 KV row (a head_dim-wide vector) is stored in a packed byte pool.
+
+DeepSeek-V4.1 keeps three cache tiers in three storage formats (tech report sec. 2.4.4 and
+``inference/model.py``):
+
+* SWA KV -- ``fp8_e8m0_b32``: e4m3 values, one ue8m0 (power-of-two) scale per 32 channels
+* main KV -- ``fp4_e4m3_b16``: e2m1 values, one e4m3 scale per 16 channels (NVFP4 without the global scale)
+* indexer K -- ``fp4_e8m0_b32``: e2m1 values, one ue8m0 scale per 32 channels (OCP MXFP4)
+
+A row is laid out as ``[values | scales]``: ``value_bytes(dim)`` bytes of (packed) values followed by
+``scale_bytes(dim)`` scale bytes, so one gather fetches both. e2m1 pairs pack the even channel into the low
+nibble and the odd channel into the high nibble (``torch.float4_e2m1fn_x2`` / ``convert.py`` order).
+
+The Triton side (``kernel/triton/dsv41/row_format.py``) selects the dequant path from ``RowFormat.code``
+(a constexpr); the pack / unpack wrappers in ``kernel/triton/dsv41/pack.py`` reproduce the reference
+``act_quant`` / ``fp4_act_quant`` numerics exactly, including the bf16 rounding of the dequantized value
+the reference bakes into its cache.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class RowFormat:
+ name: str
+ elem_bits: int # 16 (bf16), 8 (e4m3) or 4 (e2m1)
+ group: int # channels per scale; 0 = unscaled
+ scale: str | None # None | "e8m0" | "e4m3"
+ code: int # Triton constexpr selector
+
+ def value_bytes(self, dim: int) -> int:
+ assert (dim * self.elem_bits) % 8 == 0, (dim, self.elem_bits)
+ return dim * self.elem_bits // 8
+
+ def scale_bytes(self, dim: int) -> int:
+ if not self.group:
+ return 0
+ assert dim % self.group == 0, (dim, self.group)
+ return dim // self.group
+
+ def row_bytes(self, dim: int) -> int:
+ return self.value_bytes(dim) + self.scale_bytes(dim)
+
+ def num_groups(self, dim: int) -> int:
+ return dim // self.group if self.group else 0
+
+ def validate_dim(self, dim: int) -> None:
+ if dim <= 0 or (dim * self.elem_bits) % 8 or (self.group and dim % self.group):
+ raise ValueError(f"row format {self.name} cannot hold a {dim}-wide row")
+
+
+BF16 = RowFormat("bf16", elem_bits=16, group=0, scale=None, code=0)
+FP8_E8M0_B32 = RowFormat("fp8_e8m0_b32", elem_bits=8, group=32, scale="e8m0", code=1)
+FP4_E4M3_B16 = RowFormat("fp4_e4m3_b16", elem_bits=4, group=16, scale="e4m3", code=2)
+FP4_E8M0_B32 = RowFormat("fp4_e8m0_b32", elem_bits=4, group=32, scale="e8m0", code=3)
+
+ROW_FORMATS: dict[str, RowFormat] = {f.name: f for f in (BF16, FP8_E8M0_B32, FP4_E4M3_B16, FP4_E8M0_B32)}
+
+
+def row_format(name: str) -> RowFormat:
+ try:
+ return ROW_FORMATS[name]
+ except KeyError:
+ raise ValueError(f"unknown row format {name!r}; one of {sorted(ROW_FORMATS)}") from None
+
+
+__all__ = ["RowFormat", "BF16", "FP8_E8M0_B32", "FP4_E4M3_B16", "FP4_E8M0_B32", "ROW_FORMATS", "row_format"]
diff --git a/python/freetoken/kvcache/dsv4_cost_model.py b/python/freetoken/kvcache/dsv4/v4_cost_model.py
similarity index 96%
rename from python/freetoken/kvcache/dsv4_cost_model.py
rename to python/freetoken/kvcache/dsv4/v4_cost_model.py
index 2c084aff..c6f5b7ce 100644
--- a/python/freetoken/kvcache/dsv4_cost_model.py
+++ b/python/freetoken/kvcache/dsv4/v4_cost_model.py
@@ -28,14 +28,12 @@ _FP32_BYTES = 4
def dsv4_reserved_window_pages(max_running_req: int, radix: bool) -> int:
- """Window pages the sliding pool must always keep for the concurrent working set: each
- running request's decode transients (2 per req + dummy) plus, in radix mode, PER concurrent
- request one locked live-tail page AND a retained (soft-pinned) prompt-end window -- the
- window is 2 pages here because the retention gap page-aligns to a whole extra page at
- P==window==128, so a distinct follow-up per running request can re-lock 2 pages each.
- Shared by the engine's window-floor and the manager's prefill_chunk_budget so both
- reserve the same set."""
- return 2 * (max_running_req + 1) + (3 * max_running_req if radix else 0) + 1
+ """Window pages the sliding pool must always keep for the concurrent working set; the
+ formula is shared with every sliding-window paged pool (``window_tier.reserved_window_pages``)
+ and by the engine's window-floor and the manager's prefill_chunk_budget."""
+ from .window_tier import reserved_window_pages
+
+ return reserved_window_pages(max_running_req, radix)
def ring_size_for_ratio(ratio: int) -> int:
diff --git a/python/freetoken/kvcache/dsv4_paged_pool.py b/python/freetoken/kvcache/dsv4/v4_pool.py
similarity index 52%
rename from python/freetoken/kvcache/dsv4_paged_pool.py
rename to python/freetoken/kvcache/dsv4/v4_pool.py
index 3ce23eea..d5eae6d0 100644
--- a/python/freetoken/kvcache/dsv4_paged_pool.py
+++ b/python/freetoken/kvcache/dsv4/v4_pool.py
@@ -23,135 +23,22 @@ import torch
from freetoken.utils import init_logger
-from .base import BaseKVCachePool
-from .dsv4_cost_model import (
+from .v4_cost_model import (
DSV4PoolSizes,
dsv4_kv_unit_bytes,
dsv4_window_unit_bytes,
ring_size_for_ratio,
)
+# The window-tier building blocks and the pool base live in window_tier.py (shared with the DSV41
+# pool); the first two are re-exported here for the DSV4 callers and tests that import them from
+# this module.
+from .window_tier import CompressStateRing, FreeListAllocator, WindowTierPagedPool # noqa: F401
logger = init_logger(__name__)
-# LIFO free-list allocator for the window tier: one allocated unit spans ``page_unit = P``
-# slots, so a unit base is always a multiple of ``P`` (the spec's ``G*P`` page-base
-# invariant). The cmp/idx tiers have no allocator -- their rows are ``full_loc // ratio``
-# pure arithmetic.
-class FreeListAllocator:
- def __init__(
- self,
- capacity: int,
- device: torch.device,
- page_unit: int = 1,
- ) -> None:
- assert capacity % page_unit == 0, (
- f"capacity {capacity} must be a multiple of page_unit {page_unit}"
- )
- self._capacity = int(capacity)
- self._page_unit = int(page_unit)
- self._device = device
- # Unit base slots: [0, page_unit, 2*page_unit, ...]. LIFO -> pop the tail.
- self._free = self._fresh_free()
-
- def _fresh_free(self) -> torch.Tensor:
- n_units = self._capacity // self._page_unit
- return torch.arange(n_units, dtype=torch.int64, device=self._device) * self._page_unit
-
- def alloc(self, n_units: int) -> torch.Tensor:
- """Return ``n_units`` unit base slots (each a multiple of ``page_unit``)."""
- if n_units < 0:
- raise ValueError(f"n_units must be non-negative, got {n_units}")
- if n_units > self._free.numel():
- raise RuntimeError(
- f"FreeListAllocator out of slots: requested {n_units} units, "
- f"have {self._free.numel()} (capacity {self._capacity}, unit {self._page_unit})"
- )
- if n_units == 0:
- return self._free[:0].clone()
- taken = self._free[-n_units:].clone()
- self._free = self._free[:-n_units]
- return taken
-
- def free(self, units: torch.Tensor) -> None:
- """Return previously-allocated unit base slots to the pool (LIFO recycle)."""
- if units.numel() == 0:
- return
- units = units.to(device=self._device, dtype=torch.int64).reshape(-1)
- self._free = torch.cat([self._free, units])
-
- def available(self) -> int:
- """Free capacity in slots (units * page_unit)."""
- return int(self._free.numel()) * self._page_unit
-
- @property
- def capacity(self) -> int:
- return self._capacity
-
- def reset(self) -> None:
- self._free = self._fresh_free()
-
-
-
-class CompressStateRing:
- """Per-layer fp32 compress-state ring: ``[n_slots + 1, 2*(1+overlap)*head_dim]``.
-
- Last row (index ``-1``) is a permanent scratch slot, re-cleared on every
- write. Last dim is split ``kv | score``; ``set_state`` writes both halves.
- """
-
- def __init__(
- self,
- n_slots: int,
- ring_size: int,
- overlap: bool,
- head_dim: int,
- device: torch.device,
- dtype: torch.dtype = torch.float32,
- ) -> None:
- self.ring_size = ring_size
- self.head_dim = head_dim
- self._item_size = (1 + int(overlap)) * head_dim # width of kv (== width of score)
- last_dim = 2 * self._item_size
- # +1 trailing scratch row at index -1.
- self.buffer = torch.zeros((n_slots + 1, last_dim), dtype=dtype, device=device)
- self.n_slots = n_slots
- self._clear_scratch()
-
- def _clear_scratch(self) -> None:
- self.buffer[-1, : self._item_size].zero_()
- self.buffer[-1, self._item_size :].fill_(float("-inf"))
-
- @property
- def item_size(self) -> int:
- return self._item_size
-
- def get(self, state_loc: torch.Tensor) -> torch.Tensor:
- """Gather ``kv_score`` rows at ``state_loc`` (``-1`` -> scratch row)."""
- return self.buffer[state_loc]
-
- def set(self, state_loc: torch.Tensor, kv_score: torch.Tensor) -> None:
- """Scatter ``kv_score`` rows to ``state_loc`` then re-clear the scratch row."""
- self.buffer[state_loc] = kv_score
- self._clear_scratch()
-
- def get_blocks(self, page_base: torch.Tensor) -> torch.Tensor:
- """Batched per-row carry-block read. ``page_base`` is ``[B]`` (the ring block base
- row per row, == ``(window_slot // P) * ring_size``). Returns ``[B, ring_size, 2*item]``
- — each row's whole rolling carry block. Distinct pages -> disjoint blocks (isolation)."""
- rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)
- return self.buffer[rows]
-
- def set_blocks(self, page_base: torch.Tensor, blocks: torch.Tensor) -> None:
- """Batched per-row carry-block write: ``buffer[base+arange(ring_size)] = blocks[row]``.
- ``page_base`` ``[B]``, ``blocks`` ``[B, ring_size, 2*item]``. Re-clears scratch."""
- rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)
- self.buffer[rows] = blocks
- self._clear_scratch()
-
-
-class DSV4PagedKVCache(BaseKVCachePool):
+class DSV4PagedKVCache(WindowTierPagedPool):
def __init__(
self,
sizes: DSV4PoolSizes,
@@ -273,50 +160,11 @@ class DSV4PagedKVCache(BaseKVCachePool):
)
)
- # ----- full-loc translation (gather-only -1 safety; see field comment) -----
- def translate_full_to_window(self, full_locs: torch.Tensor) -> torch.Tensor:
- # int64 gather indices: the shared page_table stores full locs as int32.
- return self.full_to_window[full_locs.to(dtype=torch.int64)]
-
- @staticmethod
- def cmp_rows(full_locs: torch.Tensor, ratio: int) -> torch.Tensor:
- return torch.div(full_locs.to(dtype=torch.int64), ratio, rounding_mode="floor")
-
- def bind_window_pages(self, full_page_base: int, window_page_base: int) -> None:
- assert full_page_base % self.P == 0 and window_page_base % self.P == 0
- self.full_to_window[full_page_base : full_page_base + self.P] = torch.arange(
- window_page_base, window_page_base + self.P, dtype=torch.int64, device=self._device
- )
-
- def unbind_window_pages(self, full_locs: torch.Tensor) -> None:
- self.full_to_window[full_locs[full_locs >= 0]] = -1
-
- # ----- generic swa_pool duck-type (the CacheManager plug-in surface) -----
- # ShadowRadix layering: the shared page_table is the virtual full-token coordinate; this pool
- # projects it into the physical tiers. The window tier is the managed "second currency" --
- # token-face signatures (what the generic CacheManager speaks), PAGE-ATOMIC internals (window
- # pages are 1:1 page-bound to full pages; the per-page state ring requires it). alloc_swa
- # receives whole ascending pages (allocate_paged's _page_to_token expansion) and free paths
- # are page-complete by construction (padded finish tails, align_down frontiers, page-aligned
- # tree nodes) -- asserted here, not assumed.
- swa_paged = True
-
- @property
- def sliding_window_size(self) -> int:
- return self.P
-
- @property
- def prefill_chunk_budget(self) -> int:
- return self._chunk_budget
-
# ----- engine-facing rebuild surface -----
- # The tier buffers are bound into per-forward model scratch, invalid after a realloc.
- needs_rebind_on_rebuild = True
-
@classmethod
def kv_cost(cls, config) -> tuple[int, int, int, int]:
- from .dsv4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages
- from .dsv4_cost_model import dsv4_auto_cost_model
+ from .v4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages
+ from .v4_cost_model import dsv4_auto_cost_model
dsv4_args = config.model_config.dsv4_args
P = dsv4_args.window_size
@@ -335,8 +183,8 @@ class DSV4PagedKVCache(BaseKVCachePool):
# graceful config error, not a late OOM.
from freetoken.utils import mem_GB
- from .dsv4_cost_model import _dsv4_pool_sizes, _dsv4_swa_ratio, _dsv4_window_floor_pages
- from .dsv4_cost_model import dsv4_pool_bytes, dsv4_solve_num_pages
+ from .v4_cost_model import _dsv4_pool_sizes, _dsv4_swa_ratio, _dsv4_window_floor_pages
+ from .v4_cost_model import dsv4_pool_bytes, dsv4_solve_num_pages
dsv4_args = config.model_config.dsv4_args
P = dsv4_args.window_size
@@ -369,11 +217,19 @@ class DSV4PagedKVCache(BaseKVCachePool):
)
return num_pages
+ @classmethod
+ def window_spec(cls, config):
+ from ..base import WindowPoolSpec
+ from .v4_cost_model import _dsv4_window_floor_pages
+
+ P = config.model_config.dsv4_args.window_size
+ return WindowPoolSpec(P, _dsv4_window_floor_pages(config, P) - 1)
+
@classmethod
def min_kv_tokens(cls, config) -> int:
# The full anchor must cover the window working-set floor (full >= window always), so
# that floor -- the value validate_rebuild enforces -- is the pool's floor in tokens.
- from .dsv4_cost_model import _dsv4_window_floor_pages
+ from .v4_cost_model import _dsv4_window_floor_pages
P = config.model_config.dsv4_args.window_size
return _dsv4_window_floor_pages(config, P) * P
@@ -387,9 +243,9 @@ class DSV4PagedKVCache(BaseKVCachePool):
from freetoken.engine.cache_budget import net_cache_budget_bytes
from freetoken.utils import mem_GB
- from .base import CacheRebuildRejected
- from .dsv4_cost_model import _dsv4_pool_sizes, _dsv4_window_floor_pages
- from .dsv4_cost_model import dsv4_pool_bytes
+ from ..base import CacheRebuildRejected
+ from .v4_cost_model import _dsv4_pool_sizes, _dsv4_window_floor_pages
+ from .v4_cost_model import dsv4_pool_bytes
dsv4_args = config.model_config.dsv4_args
if num_pages is not None:
@@ -426,120 +282,19 @@ class DSV4PagedKVCache(BaseKVCachePool):
def rebuild_from_config(
self, config, num_pages: int, *, num_swa_pages: int | None = None
) -> None:
- from .dsv4_cost_model import _dsv4_pool_sizes
+ from .v4_cost_model import _dsv4_pool_sizes
# +1 for the dummy page
self.rebuild(_dsv4_pool_sizes(config, num_pages + 1, num_swa_pages=num_swa_pages))
- def attach_page_table(self, page_table: torch.Tensor) -> None:
- # The model reads full locs through full_loc_map; under the shared route that IS the
- # page table. The attention backend re-allocates its decode snapshot on re-capture.
- self.full_loc_map = page_table
-
def unit_bytes(self) -> tuple[int, int]:
# No measurable flat buffer (owned paged pool): the full (cmp/idx + mapping) and window
# (sliding KV + state rings) per-token costs come from the per-tier cost model.
return dsv4_kv_unit_bytes(self.args, self.P), dsv4_window_unit_bytes(self.args, self.P)
- def rebuild(self, sizes) -> None:
- """In-place resize to ``sizes`` (identity-preserving; free-before-alloc). The manager's
- tree/page bookkeeping reset is the scheduler's generic cache_manager.rebuild; the engine
- re-attaches the page table via attach_page_table afterwards."""
- import gc
-
- assert self._paged_params is not None, "rebuild before _init_paged_state"
- self.sizes = sizes
+ def _drop_buffers(self) -> None:
self.window_pool = self.cmp_pool = self.idx_pool = None
self.state_ring = self.indexer_state_ring = None
- self.full_to_window = None
- gc.collect()
- if torch.cuda.is_available():
- torch.cuda.empty_cache()
- self._alloc_buffers()
- self._init_paged_state(*self._paged_params)
-
- def _init_paged_state(self, max_running_req: int, radix: bool) -> None:
- """Build the pool-owned window free-list + the tail dummy binding. The LAST full page and
- LAST window page are the reserved dummy region: page_table's dummy row points at
- ``full_token - P`` (== the generic ``fill_(num_tokens)`` convention with num_tokens = the
- allocatable token count), permanently bound so graph-padded rows scatter to a real slot."""
- from .dsv4_cost_model import dsv4_reserved_window_pages
- P = self.P
- self._paged_params = (int(max_running_req), bool(radix))
- self.full_to_window.fill_(-1)
- self._win_alloc = FreeListAllocator(self.sizes.n_win_slots - P, self._device, page_unit=P)
- self.bind_window_pages(self.sizes.full_token - P, self.sizes.n_win_slots - P)
- # Chunk cap: a batched prefill holds the whole chunk's window live at once (sliding frees
- # only between chunks; peak ~2x the chunk), so reserve the concurrent working set and
- # halve the rest -- the same formula the bespoke manager used.
- n_win_pages = (self.sizes.n_win_slots // P) - 1
- reserved = dsv4_reserved_window_pages(max_running_req, radix)
- self._chunk_budget = max(P, (n_win_pages - reserved) // 2 * P)
-
- @property
- def swa_num_tokens(self) -> int:
- # Allocatable window slots + 1: the generic capacity convention reserves slot 0 as a
- # sentinel (cap == swa_num_tokens - 1); DSV4's reserved unit is the tail dummy page,
- # already excluded from the free-list, so +1 re-encodes the same cap.
- return (self.sizes.n_win_slots - self.P) + 1
-
- def swa_available_size(self) -> int:
- return int(self._win_alloc.available())
-
- def alloc_swa(self, full_indices: torch.Tensor) -> None:
- """Bind one window page per incoming FULL page. ``full_indices`` must be whole ascending
- pages (the ``_page_to_token`` expansion); the in-page offsets are preserved
- (``window_slot = wbase + pos % P``), which the state ring's page-block layout requires."""
- n = int(full_indices.numel())
- if n == 0:
- return
- P = self.P
- assert n % P == 0, f"alloc_swa needs whole pages, got {n} slots"
- fi = full_indices.to(device=self._device, dtype=torch.int64).view(-1, P)
- fbases = fi[:, 0]
- assert torch.equal(fi, fbases[:, None] + torch.arange(P, device=self._device)), (
- "alloc_swa pages must be contiguous ascending"
- )
- wbases = self._win_alloc.alloc(fbases.numel()) # raises when exhausted (caller gated)
- offsets = torch.arange(P, dtype=torch.int64, device=self._device)
- self.full_to_window[(fbases[:, None] + offsets).flatten()] = (
- wbases[:, None] + offsets
- ).flatten()
-
- def free_swa(self, full_indices: torch.Tensor) -> None:
- """Return the window pages backing these FULL locs and unbind the mapping. Page-atomic:
- the incoming locs must cover each touched page completely (guaranteed by the padded
- finish tails / aligned frontiers / page-aligned tree values). Idempotent over already
- unbound (slid/tombstoned) pages."""
- if full_indices.numel() == 0:
- return
- P = self.P
- fi = full_indices.to(device=self._device, dtype=torch.int64)
- fi = fi[fi >= 0]
- if fi.numel() == 0:
- return
- fbases, counts = torch.unique(
- torch.div(fi, P, rounding_mode="floor") * P, return_counts=True
- )
- assert bool((counts == P).all()), (
- f"free_swa got partial pages (counts {counts[counts != P].tolist()[:4]})"
- )
- ws = self.full_to_window[fbases]
- live = ws[ws >= 0]
- offsets = torch.arange(P, dtype=torch.int64, device=self._device)
- self.full_to_window[(fbases[:, None] + offsets).flatten()] = -1
- if live.numel():
- self._win_alloc.free(torch.div(live, P, rounding_mode="floor") * P)
-
- def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor) -> torch.Tensor:
- return self.full_to_window[kv_indices.to(dtype=torch.int64)]
-
- # ----- state_loc derivation (vectorized, LongTensor in/out) -----
- @staticmethod
- def state_loc(window_slot: torch.Tensor, ring_size: int, P: int) -> torch.Tensor:
- pages = torch.div(window_slot, P, rounding_mode="floor")
- loc = pages * ring_size + (window_slot % ring_size)
- return torch.where(window_slot < 0, torch.full_like(loc, -1), loc)
def ring_size(self, layer_id: int) -> int:
return ring_size_for_ratio(self.compress_ratios[layer_id])
@@ -586,26 +341,6 @@ class DSV4PagedKVCache(BaseKVCachePool):
)
return int(n)
- # ----- BaseKVCachePool interface -----
- def k_cache(self, index: int) -> torch.Tensor:
- return self.window_pool[index]
-
- def v_cache(self, index: int) -> torch.Tensor: # MLA: K == V (single latent)
- return self.window_pool[index]
-
- def store_kv(self, k, v, out_loc, layer_id) -> None:
- # Thin window-write shim for ABC compat; DSV4 writes via the specialized
- # setters above. K == V (single latent), so the window slot is out_loc.
- self.store_window(k, layer_id, out_loc)
-
- @property
- def device(self) -> torch.device:
- return self._device
-
- @property
- def dtype(self) -> torch.dtype:
- return self._dtype
-
@property
def num_layers(self) -> int:
return self._n_layers
diff --git a/python/freetoken/kvcache/dsv4/window_tier.py b/python/freetoken/kvcache/dsv4/window_tier.py
new file mode 100644
index 00000000..0a25c058
--- /dev/null
+++ b/python/freetoken/kvcache/dsv4/window_tier.py
@@ -0,0 +1,331 @@
+"""Shared pieces of the sliding-window paged pools (DSV4, DSV41).
+
+* ``FreeListAllocator`` -- LIFO page allocator for the window tier (unit = one P-slot page).
+* ``CompressStateRing`` -- per-window-page fp32 ``kv | score`` carry ring with a scratch row.
+* ``reserved_window_pages`` -- the window working set every such pool keeps for its running requests.
+* ``window_state_loc`` -- the derived (never stored) ring slot of a window slot.
+* ``WindowTierPagedPool`` -- the pool base: the full-loc -> window-slot mapping over the shared page
+ table, the page-atomic window allocator behind the CacheManager's ``swa_pool`` duck-type, the
+ in-place rebuild, and the ABC glue every such pool shares. Subclasses allocate their own tiers
+ (``_alloc_buffers`` / ``_drop_buffers``), size themselves (the ``kv_cost`` family) and write rows.
+
+``DSV4PagedKVCache`` (the origin of this code) and ``DSV41PagedKVCache`` both build on these, so a
+fix to allocation, freeing or the mapping lands in one place.
+"""
+
+from __future__ import annotations
+
+import gc
+from abc import abstractmethod
+
+import torch
+
+from ..base import BaseKVCachePool
+
+
+class FreeListAllocator:
+ """LIFO free list over ``capacity // page_unit`` units; a unit's base slot is a multiple of
+ ``page_unit`` (the window tier's ``G*P`` page-base invariant)."""
+
+ def __init__(self, capacity: int, device: torch.device, page_unit: int = 1) -> None:
+ assert capacity % page_unit == 0, f"capacity {capacity} must be a multiple of page_unit {page_unit}"
+ self._capacity = int(capacity)
+ self._page_unit = int(page_unit)
+ self._device = device
+ self._free = self._fresh_free()
+
+ def _fresh_free(self) -> torch.Tensor:
+ n_units = self._capacity // self._page_unit
+ return torch.arange(n_units, dtype=torch.int64, device=self._device) * self._page_unit
+
+ def alloc(self, n_units: int) -> torch.Tensor:
+ """Return ``n_units`` unit base slots (each a multiple of ``page_unit``)."""
+ if n_units < 0:
+ raise ValueError(f"n_units must be non-negative, got {n_units}")
+ if n_units > self._free.numel():
+ raise RuntimeError(
+ f"FreeListAllocator out of slots: requested {n_units} units, "
+ f"have {self._free.numel()} (capacity {self._capacity}, unit {self._page_unit})"
+ )
+ if n_units == 0:
+ return self._free[:0].clone()
+ taken = self._free[-n_units:].clone()
+ self._free = self._free[:-n_units]
+ return taken
+
+ def free(self, units: torch.Tensor) -> None:
+ """Return previously-allocated unit base slots to the pool (LIFO recycle)."""
+ if units.numel() == 0:
+ return
+ units = units.to(device=self._device, dtype=torch.int64).reshape(-1)
+ self._free = torch.cat([self._free, units])
+
+ def available(self) -> int:
+ """Free capacity in slots (units * page_unit)."""
+ return int(self._free.numel()) * self._page_unit
+
+ @property
+ def capacity(self) -> int:
+ return self._capacity
+
+ def reset(self) -> None:
+ self._free = self._fresh_free()
+
+
+class CompressStateRing:
+ """Per-layer fp32 compress-state ring: ``[n_slots + 1, 2*(1+overlap)*head_dim]``.
+
+ Last row (index ``-1``) is a permanent scratch slot, re-cleared on every write. Last dim is
+ split ``kv | score``; ``set`` writes both halves.
+ """
+
+ def __init__(
+ self,
+ n_slots: int,
+ ring_size: int,
+ overlap: bool,
+ head_dim: int,
+ device: torch.device,
+ dtype: torch.dtype = torch.float32,
+ ) -> None:
+ self.ring_size = ring_size
+ self.head_dim = head_dim
+ self._item_size = (1 + int(overlap)) * head_dim # width of kv (== width of score)
+ last_dim = 2 * self._item_size
+ self.buffer = torch.zeros((n_slots + 1, last_dim), dtype=dtype, device=device)
+ self.n_slots = n_slots
+ self._clear_scratch()
+
+ def _clear_scratch(self) -> None:
+ self.buffer[-1, : self._item_size].zero_()
+ self.buffer[-1, self._item_size :].fill_(float("-inf"))
+
+ @property
+ def item_size(self) -> int:
+ return self._item_size
+
+ def get(self, state_loc: torch.Tensor) -> torch.Tensor:
+ """Gather ``kv_score`` rows at ``state_loc`` (``-1`` -> scratch row)."""
+ return self.buffer[state_loc]
+
+ def set(self, state_loc: torch.Tensor, kv_score: torch.Tensor) -> None:
+ """Scatter ``kv_score`` rows to ``state_loc`` then re-clear the scratch row."""
+ self.buffer[state_loc] = kv_score
+ self._clear_scratch()
+
+ def get_blocks(self, page_base: torch.Tensor) -> torch.Tensor:
+ """Batched per-row carry-block read: ``page_base`` ``[B]`` (``(window_slot // P) * ring_size``)
+ -> ``[B, ring_size, 2*item]``. Distinct pages -> disjoint blocks."""
+ rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)
+ return self.buffer[rows]
+
+ def set_blocks(self, page_base: torch.Tensor, blocks: torch.Tensor) -> None:
+ """Batched per-row carry-block write of ``[B, ring_size, 2*item]``; re-clears scratch."""
+ rows = page_base[:, None] + torch.arange(self.ring_size, device=page_base.device)
+ self.buffer[rows] = blocks
+ self._clear_scratch()
+
+
+def reserved_window_pages(max_running_req: int, radix: bool) -> int:
+ """Window pages the sliding pool must always keep for the concurrent working set: each
+ running request's decode transients (2 per req + dummy) plus, in radix mode, PER concurrent
+ request one locked live-tail page AND a retained (soft-pinned) prompt-end window -- the
+ window is 2 pages here because the retention gap page-aligns to a whole extra page at
+ P == window."""
+ return 2 * (max_running_req + 1) + (3 * max_running_req if radix else 0) + 1
+
+
+def window_state_loc(window_slot: torch.Tensor, ring_size: int, P: int) -> torch.Tensor:
+ """Ring slot of a window slot: ``(ws // P) * ring_size + ws % ring_size``; ``-1`` stays ``-1``
+ (the scratch row). ``ring_size | P`` keeps distinct pages on disjoint blocks."""
+ pages = torch.div(window_slot, P, rounding_mode="floor")
+ loc = pages * ring_size + (window_slot % ring_size)
+ return torch.where(window_slot < 0, torch.full_like(loc, -1), loc)
+
+
+class WindowTierPagedPool(BaseKVCachePool):
+ """A paged KV pool whose window tier is page-bound to the shared full-token page table.
+
+ Subclasses set ``P`` (the window page), ``_device``, ``_dtype`` and ``sizes`` (with
+ ``full_token`` and ``n_win_slots``) before calling ``_alloc_buffers``, which must create
+ ``self.full_to_window`` (``[full_token + 1]`` int64, ``-1`` = unmapped, the trailing row a
+ permanent ``-1`` sentinel so a gather at ``-1`` is safe) and ``self.window_pool`` (one tensor
+ per layer, indexed by window slot). Scatter paths must never see a negative slot.
+
+ ShadowRadix layering: the shared page table is the virtual full-token coordinate; the pool
+ projects it into physical tiers. The window tier is the managed "second currency" -- token-face
+ signatures (what the generic CacheManager speaks), PAGE-ATOMIC internals (window pages are 1:1
+ page-bound to full pages; the per-page state ring requires it). ``alloc_swa`` receives whole
+ ascending pages and free paths are page-complete by construction -- asserted here, not assumed.
+ """
+
+ swa_paged = True
+ # the tier buffers are bound into per-forward model scratch, invalid after a realloc
+ needs_rebind_on_rebuild = True
+
+ P: int
+ _device: torch.device
+ _dtype: torch.dtype
+ sizes: object
+ full_to_window: torch.Tensor
+ window_pool: list
+
+ # ----- tiers (subclass) -----
+ @abstractmethod
+ def _alloc_buffers(self) -> None: ...
+
+ @abstractmethod
+ def _drop_buffers(self) -> None:
+ """Release every tier buffer before a rebuild reallocates them."""
+
+ # ----- full-loc translation (gather-only -1 safety; see the class docstring) -----
+ def translate_full_to_window(self, full_locs: torch.Tensor) -> torch.Tensor:
+ # int64 gather indices: the shared page_table stores full locs as int32
+ return self.full_to_window[full_locs.to(dtype=torch.int64)]
+
+ @staticmethod
+ def cmp_rows(full_locs: torch.Tensor, ratio: int) -> torch.Tensor:
+ """The compressed row of a full loc: ``full_loc // ratio`` (negative stays negative)."""
+ return torch.div(full_locs.to(dtype=torch.int64), ratio, rounding_mode="floor")
+
+ def bind_window_pages(self, full_page_base: int, window_page_base: int) -> None:
+ assert full_page_base % self.P == 0 and window_page_base % self.P == 0
+ self.full_to_window[full_page_base : full_page_base + self.P] = torch.arange(
+ window_page_base, window_page_base + self.P, dtype=torch.int64, device=self._device
+ )
+
+ def unbind_window_pages(self, full_locs: torch.Tensor) -> None:
+ self.full_to_window[full_locs[full_locs >= 0]] = -1
+
+ @staticmethod
+ def state_loc(window_slot: torch.Tensor, ring_size: int, P: int) -> torch.Tensor:
+ return window_state_loc(window_slot, ring_size, P)
+
+ # ----- generic swa_pool duck-type (the CacheManager plug-in surface) -----
+ @property
+ def sliding_window_size(self) -> int:
+ return self.P
+
+ @property
+ def prefill_chunk_budget(self) -> int:
+ return self._chunk_budget
+
+ def _init_paged_state(self, max_running_req: int, radix: bool) -> None:
+ """Build the pool-owned window free list + the tail dummy binding. The LAST full page and
+ LAST window page are the reserved dummy region: page_table's dummy row points at
+ ``full_token - P`` (the generic ``fill_(num_tokens)`` convention), permanently bound so
+ graph-padded rows scatter to a real slot."""
+ P = self.P
+ self._paged_params = (int(max_running_req), bool(radix))
+ self.full_to_window.fill_(-1)
+ self._win_alloc = FreeListAllocator(self.sizes.n_win_slots - P, self._device, page_unit=P)
+ self.bind_window_pages(self.sizes.full_token - P, self.sizes.n_win_slots - P)
+ # Chunk cap: a batched prefill holds the whole chunk's window live at once (sliding frees
+ # only between chunks; peak ~2x the chunk), so reserve the concurrent working set and
+ # halve the rest.
+ n_win_pages = (self.sizes.n_win_slots // P) - 1
+ reserved = reserved_window_pages(max_running_req, radix)
+ self._chunk_budget = max(P, (n_win_pages - reserved) // 2 * P)
+
+ @property
+ def window_pages(self) -> int:
+ return self.sizes.n_win_pages - 1
+
+ @property
+ def swa_num_tokens(self) -> int:
+ # Allocatable window slots + 1: the generic capacity convention reserves slot 0 as a
+ # sentinel (cap == swa_num_tokens - 1); here the reserved unit is the tail dummy page,
+ # already excluded from the free list, so +1 re-encodes the same cap.
+ return (self.sizes.n_win_slots - self.P) + 1
+
+ def swa_available_size(self) -> int:
+ return int(self._win_alloc.available())
+
+ def alloc_swa(self, full_indices: torch.Tensor) -> None:
+ """Bind one window page per incoming FULL page. ``full_indices`` must be whole ascending
+ pages (the ``_page_to_token`` expansion); the in-page offsets are preserved
+ (``window_slot = wbase + pos % P``), which the state ring's page-block layout requires."""
+ n = int(full_indices.numel())
+ if n == 0:
+ return
+ P = self.P
+ assert n % P == 0, f"alloc_swa needs whole pages, got {n} slots"
+ fi = full_indices.to(device=self._device, dtype=torch.int64).view(-1, P)
+ fbases = fi[:, 0]
+ assert torch.equal(fi, fbases[:, None] + torch.arange(P, device=self._device)), "alloc_swa pages must be contiguous ascending"
+ wbases = self._win_alloc.alloc(fbases.numel()) # raises when exhausted (caller gated)
+ offsets = torch.arange(P, dtype=torch.int64, device=self._device)
+ self.full_to_window[(fbases[:, None] + offsets).flatten()] = (wbases[:, None] + offsets).flatten()
+
+ def free_swa(self, full_indices: torch.Tensor) -> None:
+ """Return the window pages backing these FULL locs and unbind the mapping. Page-atomic: the
+ incoming locs must cover each touched page completely (guaranteed by the padded finish
+ tails / aligned frontiers / page-aligned tree values). Idempotent over already unbound
+ (slid / tombstoned) pages."""
+ if full_indices.numel() == 0:
+ return
+ P = self.P
+ fi = full_indices.to(device=self._device, dtype=torch.int64)
+ fi = fi[fi >= 0]
+ if fi.numel() == 0:
+ return
+ fbases, counts = torch.unique(torch.div(fi, P, rounding_mode="floor") * P, return_counts=True)
+ assert bool((counts == P).all()), f"free_swa got partial pages (counts {counts[counts != P].tolist()[:4]})"
+ ws = self.full_to_window[fbases]
+ live = ws[ws >= 0]
+ offsets = torch.arange(P, dtype=torch.int64, device=self._device)
+ self.full_to_window[(fbases[:, None] + offsets).flatten()] = -1
+ if live.numel():
+ self._win_alloc.free(torch.div(live, P, rounding_mode="floor") * P)
+
+ def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor) -> torch.Tensor:
+ return self.full_to_window[kv_indices.to(dtype=torch.int64)]
+
+ # ----- rebuild / page table -----
+ def rebuild(self, sizes) -> None:
+ """In-place resize to ``sizes`` (identity-preserving; free-before-alloc). The manager's
+ tree / page bookkeeping reset is the scheduler's generic cache_manager.rebuild; the engine
+ re-attaches the page table via attach_page_table afterwards."""
+ assert self._paged_params is not None, "rebuild before _init_paged_state"
+ self.sizes = sizes
+ self._drop_buffers()
+ self.full_to_window = None # type: ignore[assignment]
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ self._alloc_buffers()
+ self._init_paged_state(*self._paged_params)
+
+ def attach_page_table(self, page_table: torch.Tensor) -> None:
+ # the model reads full locs through full_loc_map; under the shared route that IS the page table
+ self.full_loc_map = page_table
+
+ # ----- BaseKVCachePool glue: the window tier is the K == V latent cache -----
+ def k_cache(self, index: int) -> torch.Tensor:
+ return self.window_pool[index]
+
+ def v_cache(self, index: int) -> torch.Tensor:
+ return self.window_pool[index]
+
+ def store_kv(self, k, v, out_loc, layer_id) -> None:
+ self.store_window(k, layer_id, out_loc)
+
+ @abstractmethod
+ def store_window(self, kv: torch.Tensor, layer_id: int, window_slot: torch.Tensor) -> None: ...
+
+ @property
+ def device(self) -> torch.device:
+ return self._device
+
+ @property
+ def dtype(self) -> torch.dtype:
+ return self._dtype
+
+
+__all__ = [
+ "FreeListAllocator",
+ "CompressStateRing",
+ "WindowTierPagedPool",
+ "reserved_window_pages",
+ "window_state_loc",
+]
diff --git a/python/freetoken/kvcache/hybrid_swa_pool.py b/python/freetoken/kvcache/hybrid_swa_pool.py
index 41c3880e..022eb499 100644
--- a/python/freetoken/kvcache/hybrid_swa_pool.py
+++ b/python/freetoken/kvcache/hybrid_swa_pool.py
@@ -8,7 +8,7 @@ from freetoken.distributed import get_tp_info
from freetoken.models.config import KVCacheGroupSpec
from freetoken.utils import align_ceil, div_even
-from .base import BaseKVCachePool
+from .base import BaseKVCachePool, WindowPoolSpec
@dataclass(frozen=True)
@@ -295,6 +295,16 @@ class HybridSWAKVCache(BaseKVCachePool):
# by CacheManager.rebuild is then consistent by construction (idle-only).
self._init_swa_paged_state()
+ @classmethod
+ def window_spec(cls, config) -> WindowPoolSpec | None:
+ if config.cache_type != "swa_radix":
+ return None
+ return WindowPoolSpec(1, _swa_pool_floor(config))
+
+ @property
+ def window_pages(self) -> int:
+ return self.swa_num_tokens - 1
+
@classmethod
def kv_cost(cls, config, *, num_swa_pages: int | None = None) -> tuple[int, int, int, int]:
"""Full groups ride cache_per_page; the window pool is either a pinned absolute
diff --git a/python/freetoken/layers/mhc.py b/python/freetoken/layers/mhc.py
index 1cd6459d..dcf49151 100644
--- a/python/freetoken/layers/mhc.py
+++ b/python/freetoken/layers/mhc.py
@@ -19,6 +19,16 @@ PR #53906) bit-for-bit in fp32; the per-layer weights are ``hc_{attn,ffn}_fn`` /
correctness baseline; the fused triton kernel (``kernel/triton/mhc.py``,
dispatched by ``mhc_fused_post_pre`` below) replaces it on CUDA and is
validated against this file.
+
+Two variants share the gate math:
+
+* **multi-pass** (GLM-5.3, DeepSeek-V4): ``pre`` is computed from the residual the
+ sublayer is about to read and applied immediately -- ``mhc_pre`` / ``mhc_fused_post_pre``.
+* **single-pass** (DeepSeek-V4.1 "Single-Pass mHC", tech report sec. 2.4.1): every
+ sublayer consumes the ``pre`` gates the *previous* sublayer produced, so the input
+ mix no longer waits on this sublayer's own reduction -- ``mhc_pre_single_pass`` /
+ ``mhc_fused_post_pre_single_pass``. The returned ``pre_new`` is what the next
+ sublayer applies; the model seeds the chain with a one-hot pre on stream 0.
"""
from __future__ import annotations
@@ -36,8 +46,8 @@ def hc_contract(x: torch.Tensor) -> torch.Tensor:
return x.mean(dim=1)
-def mhc_pre(
- residual: torch.Tensor, # [T, n, hidden] bf16
+def mhc_gates(
+ residual: torch.Tensor, # [T, n, hidden]
fn: torch.Tensor, # [2n + n^2, n*hidden] fp32
hc_scale: torch.Tensor, # [3] fp32
hc_base: torch.Tensor, # [2n + n^2] fp32
@@ -46,8 +56,8 @@ def mhc_pre(
post_mult: float,
sinkhorn_repeat: int,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
- """Returns (post_mix [T, n, 1] fp32, comb_mix [T, n, n] fp32,
- layer_input [T, hidden] bf16)."""
+ """The coefficient predictor H(X): returns (pre_mix [T, n] fp32, post_mix [T, n, 1]
+ fp32, comb_mix [T, n, n] fp32) from one fp32 GEMM over the flattened streams."""
n, hidden = residual.shape[-2], residual.shape[-1]
t = residual.shape[0]
@@ -68,11 +78,46 @@ def mhc_pre(
for _ in range(sinkhorn_repeat - 1):
comb = comb / (comb.sum(dim=-1, keepdim=True) + hc_eps)
comb = comb / (comb.sum(dim=-2, keepdim=True) + hc_eps)
+ return pre_mix, post_mix.view(t, n, 1), comb
- layer_input = (
- (pre_mix.unsqueeze(-1) * residual.to(torch.float32)).sum(dim=1).to(residual.dtype)
- )
- return post_mix.view(t, n, 1), comb, layer_input
+
+def mhc_mix_input(residual: torch.Tensor, pre_mix: torch.Tensor) -> torch.Tensor:
+ """The sublayer input ``sum_i pre_i * res_i``: [T, n, hidden] x [T, n] -> [T, hidden]."""
+ return (pre_mix.unsqueeze(-1) * residual.to(torch.float32)).sum(dim=1).to(residual.dtype)
+
+
+def mhc_pre(
+ residual: torch.Tensor, # [T, n, hidden] bf16
+ fn: torch.Tensor, # [2n + n^2, n*hidden] fp32
+ hc_scale: torch.Tensor, # [3] fp32
+ hc_base: torch.Tensor, # [2n + n^2] fp32
+ rms_eps: float,
+ hc_eps: float,
+ post_mult: float,
+ sinkhorn_repeat: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Multi-pass: gates from ``residual``, pre applied to it right away. Returns
+ (post_mix [T, n, 1] fp32, comb_mix [T, n, n] fp32, layer_input [T, hidden] bf16)."""
+ pre_mix, post_mix, comb = mhc_gates(residual, fn, hc_scale, hc_base, rms_eps, hc_eps, post_mult, sinkhorn_repeat)
+ return post_mix, comb, mhc_mix_input(residual, pre_mix)
+
+
+def mhc_pre_single_pass(
+ residual: torch.Tensor, # [T, n, hidden]
+ pre_mix_in: torch.Tensor, # [T, n] fp32 -- the previous sublayer's pre gates
+ fn: torch.Tensor,
+ hc_scale: torch.Tensor,
+ hc_base: torch.Tensor,
+ rms_eps: float,
+ hc_eps: float,
+ post_mult: float,
+ sinkhorn_repeat: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Single-pass: gates from ``residual`` for THIS sublayer's post/comb and the NEXT
+ sublayer's pre; the input is mixed with ``pre_mix_in``. Returns (pre_new [T, n] fp32,
+ post_mix [T, n, 1] fp32, comb_mix [T, n, n] fp32, layer_input [T, hidden])."""
+ pre_new, post_mix, comb = mhc_gates(residual, fn, hc_scale, hc_base, rms_eps, hc_eps, post_mult, sinkhorn_repeat)
+ return pre_new, post_mix, comb, mhc_mix_input(residual, pre_mix_in.to(torch.float32))
def mhc_post(
@@ -140,11 +185,69 @@ def mhc_fused_post_pre(
)
+def mhc_fused_post_pre_single_pass_torch(
+ x: torch.Tensor,
+ residual: torch.Tensor,
+ post_mix: torch.Tensor | None,
+ comb_mix: torch.Tensor | None,
+ pre_mix_in: torch.Tensor,
+ fn: torch.Tensor,
+ hc_scale: torch.Tensor,
+ hc_base: torch.Tensor,
+ rms_eps: float,
+ hc_eps: float,
+ post_mult: float,
+ sinkhorn_repeat: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Decomposed single-pass reference: the previous sublayer's hc_post (skipped when
+ ``post_mix is None`` -- the model's first sublayer), then this sublayer's gates with
+ the input mixed by ``pre_mix_in``. Returns (residual_new, post_new [T, n, 1],
+ comb_new [T, n, n], pre_new [T, n], layer_input)."""
+ residual_new = residual if post_mix is None else mhc_post(x, residual, post_mix, comb_mix)
+ pre_new, post_new, comb_new, layer_input = mhc_pre_single_pass(
+ residual_new, pre_mix_in, fn, hc_scale, hc_base, rms_eps, hc_eps, post_mult, sinkhorn_repeat
+ )
+ return residual_new, post_new, comb_new, pre_new, layer_input
+
+
+def mhc_fused_post_pre_single_pass(
+ x: torch.Tensor,
+ residual: torch.Tensor,
+ post_mix: torch.Tensor | None,
+ comb_mix: torch.Tensor | None,
+ pre_mix_in: torch.Tensor,
+ fn: torch.Tensor,
+ hc_scale: torch.Tensor,
+ hc_base: torch.Tensor,
+ rms_eps: float,
+ hc_eps: float,
+ post_mult: float,
+ sinkhorn_repeat: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Single-pass sublayer boundary (DeepSeek-V4.1): fused triton on CUDA, torch elsewhere."""
+ if residual.is_cuda:
+ from freetoken.kernel.triton.mhc import mhc_fused_post_pre_single_pass_triton
+
+ return mhc_fused_post_pre_single_pass_triton(
+ x, residual, post_mix, comb_mix, pre_mix_in, fn, hc_scale, hc_base,
+ rms_eps, hc_eps, post_mult, sinkhorn_repeat,
+ )
+ return mhc_fused_post_pre_single_pass_torch(
+ x, residual, post_mix, comb_mix, pre_mix_in, fn, hc_scale, hc_base,
+ rms_eps, hc_eps, post_mult, sinkhorn_repeat,
+ )
+
+
__all__ = [
"hc_expand",
"hc_contract",
+ "mhc_gates",
+ "mhc_mix_input",
"mhc_pre",
+ "mhc_pre_single_pass",
"mhc_post",
"mhc_fused_post_pre",
"mhc_fused_post_pre_torch",
+ "mhc_fused_post_pre_single_pass",
+ "mhc_fused_post_pre_single_pass_torch",
]
diff --git a/python/freetoken/layers/moe.py b/python/freetoken/layers/moe.py
index 996beaa7..4918a032 100644
--- a/python/freetoken/layers/moe.py
+++ b/python/freetoken/layers/moe.py
@@ -308,18 +308,21 @@ class OffloadMoELayer(MoELayer):
The CPU pool is kicked off (``decode_submit``) before the GPU PCIe fetch + GEMM so
the CPU overflow GEMV runs concurrently with the GPU work. Capture-safe: the
routing split is device-side elementwise and the CPU submit/sync are host nodes.
- Each route is computed exactly once -- the GPU weights are zeroed for CPU-assigned
- routes and the CPU ids are -1 for GPU-assigned routes (the C++ kernel skips id<0).
+ Each route is computed exactly once -- CPU-assigned routes reach the GPU as slot -1 with
+ weight 0 (a kernel with ``supports_inactive_slots`` skips them without a read; any other reads
+ slot 0, whose storage is a payload or the bank's zero fill) and the CPU ids are -1 for
+ GPU-assigned routes (the C++ kernel skips id<0).
"""
executor = cache.cpu_executor
assert executor is not None, "CPU MoE executor was not initialized"
- raw = topk_ids.clone() # raw expert ids for the CPU partial
- cache.ensure_experts_hybrid(self.layer_id, topk_ids) # -> slot (hit/fetched) or -1
+ raw_ids = topk_ids.clone()
+ cache.ensure_experts_hybrid(self.layer_id, topk_ids)
+ on_gpu = topk_ids >= 0
+ cpu_ids = torch.where(on_gpu, -1, raw_ids).contiguous()
+ gpu_w = torch.where(on_gpu, topk_weights, 0.0).contiguous()
+ gpu_slots = topk_ids
if cache.collect_stats:
cache.record_decode_stats_hybrid(self.layer_id)
- on_gpu = topk_ids >= 0
-
- cpu_ids = torch.where(on_gpu, raw.new_full((), -1), raw).contiguous()
pending = executor.decode_submit(self.layer_id, hidden_states, topk_weights, cpu_ids)
# Measurement knob: FREETOKEN_HYBRID_OVERLAP=0 syncs the CPU pool *before* the
@@ -329,8 +332,10 @@ class OffloadMoELayer(MoELayer):
)
cache.copy_missing()
- gpu_slots = topk_ids.clamp_min(0) # -1 -> slot 0 (zero-weighted below)
- gpu_w = torch.where(on_gpu, topk_weights, topk_weights.new_zeros(())).contiguous()
+ if not (self.quant_method is not None and self.quant_method.kernel.supports_inactive_slots):
+ # the kernel reads every route's slot: give the inactive ones slot 0, which holds a
+ # complete payload or the bank's zero fill (never uninitialized storage), times weight 0
+ gpu_slots = gpu_slots.clamp_min(0)
gpu_routed = self._expert_gemm(
cache,
hidden_states,
diff --git a/python/freetoken/layers/quantization/configs/base.py b/python/freetoken/layers/quantization/configs/base.py
index e84dcbcb..a6413610 100644
--- a/python/freetoken/layers/quantization/configs/base.py
+++ b/python/freetoken/layers/quantization/configs/base.py
@@ -89,9 +89,13 @@ class QuantConfig(ABC):
layer_kind = layer.quant_layer_kind
kind = scheme.kind if scheme else QuantKind.NONE
cls = method_class(kind, layer_kind)
- cfg = LAYER_CONFIGS[layer_kind].from_layer(layer, scheme)
+ cfg = self.layer_config(layer, layer_kind, scheme)
return cls(cfg, get_quant_backend().select(layer_kind, kind))
+ def layer_config(self, layer: Any, layer_kind: LayerKind, scheme: QuantScheme | None) -> LinearConfig | MoEConfig:
+ """The layer's Linear / MoE config; a dialect overrides it to add what only its checkpoint knows."""
+ return LAYER_CONFIGS[layer_kind].from_layer(layer, scheme)
+
@staticmethod
def from_hf(
hf_config: Any,
diff --git a/python/freetoken/layers/quantization/configs/fp8.py b/python/freetoken/layers/quantization/configs/fp8.py
index c23cbfde..3e74f0c4 100644
--- a/python/freetoken/layers/quantization/configs/fp8.py
+++ b/python/freetoken/layers/quantization/configs/fp8.py
@@ -1,20 +1,25 @@
from __future__ import annotations
+import dataclasses
from typing import Any, ClassVar
+from ..linear import LinearConfig
+from ..moe import MoEConfig
from ..names import is_routed_expert, name_set, substr_set
-from ..registry import register_dialect
+from ..registry import LayerKind, register_dialect
from ..scheme import QuantKind, QuantScheme
-from ..scheme import FP8_BLOCK, fp8_block_scheme, fp8_tensor_scheme, mxfp4_scheme
+from ..scheme import FP8_BLOCK_SIZES, fp8_block_scheme, fp8_tensor_scheme, mxfp4_scheme
from .base import QuantConfig, Stored, cfg_get
@register_dialect
class Fp8BlockConfig(QuantConfig):
- """HF ``quant_method: fp8`` (DeepSeek-V3 style 128x128 block scales) plus the DeepSeek-V4 e8m0 / fp4-expert variant."""
+ """HF ``quant_method: fp8`` (DeepSeek-V3 style 128x128 block scales) plus the DeepSeek-V4 e8m0 / fp4-expert
+ variant; DeepSeek-V4.1 keeps that dialect with 32x32 blocks (``weight_block_size: [32, 32]``)."""
dialect = "fp8"
+ # the 128-block schemes every dialect table names; ``block_scheme`` carries the checkpoint's actual block edge
SCHEMES: ClassVar[dict[str, QuantScheme]] = {
"BLOCK": fp8_block_scheme("float"),
"BLOCK_E8M0": fp8_block_scheme("e8m0"),
@@ -32,15 +37,23 @@ class Fp8BlockConfig(QuantConfig):
def __init__(self, q: dict[str, Any], hf_config: Any = None, *, name_map=None, unquantized=()):
super().__init__(name_map, unquantized)
block = tuple(int(x) for x in (q.get("weight_block_size") or ()))
- if q.get("weight_per_tensor") or block != (FP8_BLOCK, FP8_BLOCK):
- raise NotImplementedError(f"fp8 checkpoint with weight_block_size={block} per_tensor={q.get('weight_per_tensor')} is not supported; only 128x128 blocks are")
+ square = len(block) == 2 and block[0] == block[1] and block[0] in FP8_BLOCK_SIZES
+ if q.get("weight_per_tensor") or not square:
+ raise NotImplementedError(
+ f"fp8 checkpoint with weight_block_size={block} per_tensor={q.get('weight_per_tensor')} is not supported; "
+ f"only square blocks of {FP8_BLOCK_SIZES} are"
+ )
+ self.block = block[0]
# transformers skips lm_head when the checkpoint gives no list
not_convert = tuple(q.get("modules_to_not_convert") or ("lm_head",))
self.not_convert = name_set(not_convert)
self.not_convert_substr = substr_set(not_convert)
self.convert_tables = name_set(tuple(q.get("modules_to_convert") or ()))
self.e8m0 = str(q.get("scale_fmt") or "").lower() == "ue8m0"
- self.expert_fp4 = str(cfg_get(hf_config, "expert_dtype") or "").lower() == "fp4"
+ # ``expert_dtype`` sits at the config top level (V4) or inside quantization_config (V4.1)
+ expert_dtype = q.get("expert_dtype") or cfg_get(hf_config, "expert_dtype")
+ self.expert_fp4 = str(expert_dtype or "").lower() == "fp4"
+ self.block_scheme = fp8_block_scheme("e8m0" if self.e8m0 else "float", self.block)
def storage(self, scheme: QuantScheme) -> dict[str, Stored]:
names = super().storage(scheme)
@@ -55,4 +68,11 @@ class Fp8BlockConfig(QuantConfig):
return None
if self.expert_fp4 and is_routed_expert(name):
return self.SCHEMES["EXPERT_MXFP4"]
- return self.SCHEMES["BLOCK_E8M0" if self.e8m0 else "BLOCK"]
+ return self.block_scheme
+
+ def layer_config(self, layer: Any, layer_kind: LayerKind, scheme: QuantScheme | None) -> LinearConfig | MoEConfig:
+ cfg = super().layer_config(layer, layer_kind, scheme)
+ if layer_kind is LayerKind.MOE and scheme is not None and scheme.kind is QuantKind.MXFP4:
+ # the fp4 experts quantize their activations at the same block as the dense linears
+ cfg = dataclasses.replace(cfg, act_block=self.block)
+ return cfg
diff --git a/python/freetoken/layers/quantization/linear/fp8_block.py b/python/freetoken/layers/quantization/linear/fp8_block.py
index 3b78e977..3eafdfdf 100644
--- a/python/freetoken/layers/quantization/linear/fp8_block.py
+++ b/python/freetoken/layers/quantization/linear/fp8_block.py
@@ -1,4 +1,4 @@
-"""fp8 e4m3 weight with 128x128 block scales (DeepSeek-V3 style)."""
+"""fp8 e4m3 weight with square block scales: 128x128 (DeepSeek-V3 style) or 32x32 (DeepSeek-V4.1)."""
from __future__ import annotations
@@ -7,7 +7,7 @@ from typing import Any
import torch
from ..registry import LayerKind, register_method
-from ..scheme import FP8_BLOCK as BLOCK, QuantKind
+from ..scheme import FP8_BLOCK, QuantKind, fp8_block_size
from .base import LinearConfig, LinearKernel, LinearMethod
FP8 = torch.float8_e4m3fn
@@ -18,6 +18,11 @@ def _e8m0(cfg: LinearConfig) -> bool:
return cfg.scheme is not None and cfg.scheme.weight.scale == "e8m0"
+def layer_block_size(layer: Any) -> int:
+ """The block edge a finalized layer was declared with: ``K / (scale columns)``, exact by construction."""
+ return layer.weight.shape[1] // layer.weight_scale_inv.shape[1]
+
+
class Dsv4Fp8BlockLinearKernel(LinearKernel):
"""DeepSeek-V4's reference path: activations quantized to fp8 with power-of-two block scales, e8m0 weight scales read as codes."""
@@ -29,7 +34,7 @@ class Dsv4Fp8BlockLinearKernel(LinearKernel):
def apply(self, layer: Any, x: torch.Tensor) -> torch.Tensor:
from freetoken.kernel.triton.dsv4.fp8_linear import block_fp8_linear
- return block_fp8_linear(x, layer.weight, layer.weight_scale_inv, layer.bias)
+ return block_fp8_linear(x, layer.weight, layer.weight_scale_inv, layer.bias, block=layer_block_size(layer))
class TritonFp8BlockLinearKernel(LinearKernel):
@@ -38,7 +43,11 @@ class TritonFp8BlockLinearKernel(LinearKernel):
name = "triton"
def unusable_reason(self, cfg: LinearConfig) -> str | None:
- return "reads float block scales; e8m0 codes go to the dsv4 kernel" if _e8m0(cfg) else None
+ if _e8m0(cfg):
+ return "reads float block scales; e8m0 codes go to the dsv4 kernel"
+ if fp8_block_size(cfg.scheme) != FP8_BLOCK:
+ return f"float-scale kernel serves {FP8_BLOCK}x{FP8_BLOCK} blocks only"
+ return None
def apply(self, layer: Any, x: torch.Tensor) -> torch.Tensor:
from freetoken.kernel.triton.fp8_block_linear import block_fp8_linear
@@ -52,9 +61,10 @@ class Fp8BlockLinearMethod(LinearMethod):
def create_weights(self, layer: Any) -> None:
g = self.cfg
- if g.in_features % BLOCK or any(o % BLOCK for o in g.output_sizes):
- raise ValueError(f"block-fp8 needs in/out sizes divisible by {BLOCK}, got K={g.in_features} N={g.output_sizes}")
+ block = fp8_block_size(g.scheme)
+ if g.in_features % block or any(o % block for o in g.output_sizes):
+ raise ValueError(f"block-fp8 needs in/out sizes divisible by {block}, got K={g.in_features} N={g.output_sizes}")
layer.weight = torch.empty(g.out_features, g.in_features, dtype=FP8)
# e8m0 codes stay codes for the dsv4 kernel; float scales are bf16 as the readers push them today
scale_dtype = E8M0 if _e8m0(g) else torch.bfloat16
- layer.weight_scale_inv = torch.empty(g.out_features // BLOCK, g.in_features // BLOCK, dtype=scale_dtype)
+ layer.weight_scale_inv = torch.empty(g.out_features // block, g.in_features // block, dtype=scale_dtype)
diff --git a/python/freetoken/layers/quantization/moe/base.py b/python/freetoken/layers/quantization/moe/base.py
index e4a428d3..de1ce004 100644
--- a/python/freetoken/layers/quantization/moe/base.py
+++ b/python/freetoken/layers/quantization/moe/base.py
@@ -28,6 +28,8 @@ class MoEConfig:
beta: float = 0.0
limit: float | None = None
interleaved: bool = False
+ # dynamic fp8 activation quant block of a W4A8 expert kernel, set by the quant dialect; None = no activation quant
+ act_block: int | None = None
has_bias: bool = False
apply_router_weight_on_input: bool = False
strategy: str = "resident"
@@ -138,6 +140,10 @@ class MoEKernel(ABC):
name: ClassVar[str]
cpu_format: ClassVar[str | None] = None
max_slots: ClassVar[int | None] = None
+ # Whether ``apply`` treats a slot of -1 as an inactive route (zero output, no read of any slot).
+ # Hybrid decode hands the GPU a route split with such holes; kernels without this get the holes
+ # pointed at slot 0 with weight 0 (slot storage is zero-filled at allocation, so that is exact).
+ supports_inactive_slots: ClassVar[bool] = False
def unusable_reason(self, cfg: MoEConfig) -> str | None:
return None
diff --git a/python/freetoken/layers/quantization/moe/fp8_block.py b/python/freetoken/layers/quantization/moe/fp8_block.py
index 20c3dde9..7552978d 100644
--- a/python/freetoken/layers/quantization/moe/fp8_block.py
+++ b/python/freetoken/layers/quantization/moe/fp8_block.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import torch
from ..registry import LayerKind, register_method
-from ..scheme import FP8_BLOCK as BLOCK, QuantKind
+from ..scheme import FP8_BLOCK as BLOCK, QuantKind, fp8_block_size
from .base import BankSpec, ExpertView, fused_piece, gated_epilogue_reason, limit_or_inf, MoEConfig, MoEKernel, MoEMethod
FP8 = torch.float8_e4m3fn
@@ -24,6 +24,8 @@ class TritonFp8BlockMoEKernel(MoEKernel):
reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=False)
if reason:
return reason
+ if cfg.scheme is not None and fp8_block_size(cfg.scheme) != BLOCK:
+ return f"fp8 block MoE kernel serves {BLOCK}x{BLOCK} expert blocks only"
reason = gated_epilogue_reason(cfg)
if reason:
return f"fp8 block MoE kernel: {reason}"
diff --git a/python/freetoken/layers/quantization/moe/mxfp4.py b/python/freetoken/layers/quantization/moe/mxfp4.py
index 13cd9746..1af6d9f4 100644
--- a/python/freetoken/layers/quantization/moe/mxfp4.py
+++ b/python/freetoken/layers/quantization/moe/mxfp4.py
@@ -12,16 +12,20 @@ E8M0 = torch.float8_e8m0fnu
class TritonMxfp4MoEKernel(MoEKernel):
- """Standard OCP MXFP4 experts (DeepSeek-V4 ds_fp4): e2m1 pairs + e8m0 scales, no bias."""
+ """Standard OCP MXFP4 experts (DeepSeek-V4 / V4.1 ds_fp4): e2m1 pairs + e8m0 scales, no bias;
+ W4A8 with the fp8 activation round-trips at the checkpoint's block."""
name = "triton"
cpu_format = "ds_fp4"
+ supports_inactive_slots = True
def unusable_reason(self, cfg: MoEConfig) -> str | None:
if cfg.interleaved:
return "standard MXFP4 kernel reads the concatenated gate|up row order"
if (cfg.alpha, cfg.beta) != (1.0, 0.0):
return "standard MXFP4 kernel has no alpha / beta in its swiglu"
+ if cfg.act_block is None:
+ return "W4A8 ds_fp4 experts need the fp8 activation quant block"
return self._common_reject(cfg, tp_ok=False, cpu_ok=True, plain_silu_only=False)
def layout(self, cfg: MoEConfig) -> dict[str, BankSpec]:
@@ -44,13 +48,14 @@ class TritonMxfp4MoEKernel(MoEKernel):
t = view.tensors
banks = (t["gate_up"], t["gate_up_scale"], t["down"], t["down_scale"])
limit = limit_or_inf(layer)
+ act_block = layer.quant_method.cfg.act_block
if is_prefill and view.n is not None:
from freetoken.moe.fused_ds_fp4 import routed_experts_fp4_prefill
- return routed_experts_fp4_prefill(x, topk_ids, topk_weights, *banks, limit, view.n)
+ return routed_experts_fp4_prefill(x, topk_ids, topk_weights, *banks, limit, view.n, act_block=act_block)
from freetoken.moe.fused_ds_fp4 import routed_experts_fp4
- return routed_experts_fp4(x, topk_ids, topk_weights, *banks, limit)
+ return routed_experts_fp4(x, topk_ids, topk_weights, *banks, limit, act_block=act_block)
class TritonGptossMxfp4MoEKernel(MoEKernel):
diff --git a/python/freetoken/layers/quantization/scheme.py b/python/freetoken/layers/quantization/scheme.py
index 668eb6fb..89f968af 100644
--- a/python/freetoken/layers/quantization/scheme.py
+++ b/python/freetoken/layers/quantization/scheme.py
@@ -25,7 +25,8 @@ class QuantKind(Enum):
return self.value
-FP8_BLOCK = 128
+FP8_BLOCK = 128 # DeepSeek-V3 style square block; V4.1 exports 32x32
+FP8_BLOCK_SIZES = (32, 128)
NVFP4_GROUP = 16
MX_GROUP = 32
@@ -80,8 +81,17 @@ def fp8_tensor_scheme(scale: str, *, per_row: bool = False, input_scale: bool =
return QuantScheme(QuantKind.FP8_TENSOR, WeightDesc("e4m3", (1, -1) if per_row else (-1, -1), scale), roles)
-def fp8_block_scheme(scale: str) -> QuantScheme:
- return QuantScheme(QuantKind.FP8_BLOCK, WeightDesc("e4m3", (FP8_BLOCK, FP8_BLOCK), scale), {"weight", "weight_scale_inv"})
+def fp8_block_scheme(scale: str, block: int = FP8_BLOCK) -> QuantScheme:
+ """Square ``block x block`` e4m3 weight blocks, one scale each; ``block`` is the checkpoint's ``weight_block_size``."""
+ if block not in FP8_BLOCK_SIZES:
+ raise NotImplementedError(f"fp8 block size {block} is not supported; one of {FP8_BLOCK_SIZES}")
+ return QuantScheme(QuantKind.FP8_BLOCK, WeightDesc("e4m3", (block, block), scale), {"weight", "weight_scale_inv"})
+
+
+def fp8_block_size(scheme: QuantScheme) -> int:
+ """The square block edge of an FP8_BLOCK scheme (128 for DeepSeek-V3 style exports, 32 for DeepSeek-V4.1)."""
+ assert scheme.kind is QuantKind.FP8_BLOCK, scheme
+ return scheme.weight.group[0]
def mxfp8_scheme() -> QuantScheme:
diff --git a/python/freetoken/mm/processors/deepseek_v41.py b/python/freetoken/mm/processors/deepseek_v41.py
new file mode 100644
index 00000000..e40c9615
--- /dev/null
+++ b/python/freetoken/mm/processors/deepseek_v41.py
@@ -0,0 +1,88 @@
+"""DeepSeek-V4.1 image resize/patch layout and learned image-span delimiters."""
+
+from __future__ import annotations
+
+import math
+import struct
+
+import numpy as np
+import torch
+from PIL import ImageOps
+
+from freetoken.message import MMItem
+from freetoken.mm import mm_pad_value
+from freetoken.mm.processor import MMProcessor, PromptReplacement, content_hash
+
+
+def image_grid(width, height, patch, ratio, min_pixels, max_tokens, max_wh_ratio=None):
+ if max_wh_ratio is not None:
+ width = min(width, height * max_wh_ratio)
+ if width * height < min_pixels:
+ scale = (min_pixels / (width * height)) ** 0.5
+ width, height = int(width * scale), int(height * scale)
+ w, h = math.ceil(width / patch) * patch, math.ceil(height / patch) * patch
+ if math.ceil(h / (patch * ratio)) * (math.ceil(w / (patch * ratio)) + 1) + 2 > max_tokens:
+ aspect, cell = height / width, patch * ratio
+ wf = math.sqrt((max_tokens - 2) / aspect + 0.25) - 0.5
+ hf = wf * aspect
+ if wf < 1:
+ h, w = (max_tokens - 2) // 2 * cell, cell
+ elif hf < 1:
+ h, w = cell, (max_tokens - 3) * cell
+ else:
+ scale = min(math.floor(wf) * cell / width, math.floor(hf) * cell / height)
+ h, w = math.floor(height * scale / patch) * patch, math.floor(width * scale / patch) * patch
+ return h, w
+
+
+class DeepseekV41MMProcessor(MMProcessor):
+ def __init__(self, hf_config, model_path, mm):
+ super().__init__(model_path, mm)
+ self.vc = hf_config.vision_config
+ self.placeholder = [hf_config.image_token_id]
+ self.min_pixels = self.vc.min_pixels
+ self.max_tokens = mm.image_max_tokens if mm.image_max_tokens is not None else self.vc.max_image_tokens
+ if mm.image_min_tokens is not None:
+ # Delimiters and row breaks count against the total span budget.
+ self.min_pixels = max(1, mm.image_min_tokens - 2) * (self.vc.patch_size * self.vc.downsample_ratio)**2
+ allowed = {"min_pixels", "max_image_tokens", "max_wh_ratio"}
+ unknown = set(mm.processor_kwargs) - allowed
+ if unknown:
+ raise ValueError(f"unsupported DeepSeek image processor options: {sorted(unknown)}")
+ self.min_pixels = mm.processor_kwargs.get("min_pixels", self.min_pixels)
+ self.max_tokens = mm.processor_kwargs.get("max_image_tokens", self.max_tokens)
+ self.max_wh_ratio = mm.processor_kwargs.get("max_wh_ratio", self.vc.max_wh_ratio)
+ if self.max_tokens < 4 or self.min_pixels < 0:
+ raise ValueError("image token budget must be >= 4 and min_pixels must be nonnegative")
+ if self.max_wh_ratio is not None and self.max_wh_ratio <= 0:
+ raise ValueError("max_wh_ratio must be positive")
+
+ def process(self, images):
+ p, r = self.vc.patch_size, self.vc.downsample_ratio
+ items = []
+ for image in images:
+ h, w = image_grid(image.width, image.height, p, r, self.min_pixels, self.max_tokens, self.max_wh_ratio)
+ if self.max_wh_ratio is not None and image.width >= self.max_wh_ratio * image.height:
+ image = image.resize((w, h))
+ else:
+ image = ImageOps.pad(image, (w, h), color=(127, 127, 127))
+ x = torch.from_numpy(np.asarray(image, dtype=np.float32)).permute(2, 0, 1) / 255
+ x = ((x - 0.5) / 0.5).to(torch.bfloat16)
+ nh, nw = h // p, w // p
+ patches = x.reshape(3, nh, p, nw, p).permute(1, 3, 0, 2, 4).reshape(nh * nw, 3, p, p).contiguous()
+ item_hash = content_hash(patches, struct.pack("<3i", 1, nh, nw))
+ items.append(MMItem(modality="image", hash=item_hash, pad_value=mm_pad_value(item_hash), offsets=[], feature=patches,
+ model_specific_data={"grid_thw": [1, nh, nw]}))
+ return items
+
+ def prompt_replacement(self, item):
+ _, h, w = item.grid_thw
+ r = self.vc.downsample_ratio
+ n = ((h + r - 1) // r) * ((w + r - 1) // r + 1) + 2
+ # Every span row, including delimiters/newlines, has a learned visual embedding.
+ return PromptReplacement([self.placeholder[0]] * n)
+
+ def dummy_items(self, dtype, device):
+ p = self.vc.patch_size
+ return [MMItem(modality="image", hash=0, pad_value=0, offsets=[[0, 4]],
+ feature=torch.zeros(1, 3, p, p, dtype=dtype, device=device), model_specific_data={"grid_thw": [1, 1, 1]})]
diff --git a/python/freetoken/models/config.py b/python/freetoken/models/config.py
index c5943dce..55b90bb5 100644
--- a/python/freetoken/models/config.py
+++ b/python/freetoken/models/config.py
@@ -205,7 +205,8 @@ class DSV4AttentionGroupConfig(BaseAttentionGroupConfig):
"""DSV4 sparse attention (window + compressed tiers + Lightning Indexer). Standalone
class on purpose: subclassing SWAAttentionGroupConfig would flip has_swa_attention
and reroute DSV4 through the SWA gates. Geometry beyond the latent width lives in
- dsv4_args; the pool prices itself from there, not from this spec."""
+ dsv4_args (``dsv41_args`` for the ``v41`` variant); the pool prices itself from there,
+ not from this spec."""
kind: ClassVar[Literal["dsv4"]] = "dsv4"
cache_kind: ClassVar[Literal["dsv4_paged"]] = "dsv4_paged"
@@ -213,6 +214,8 @@ class DSV4AttentionGroupConfig(BaseAttentionGroupConfig):
num_kv_heads: int
head_dim: int
sliding_window: int # the P-token window page
+ # "v41": DeepSeek-V4.1's cross-layer-shared packed tiers (AttnType.DSV41, its own pool and backend)
+ variant: Literal["v4", "v41"] = "v4"
AttentionGroupConfig: TypeAlias = (
@@ -327,6 +330,9 @@ class ModelConfig:
# CSA/HCA compressors, Lightning Indexer, manifold-constrained Hyper-Connections,
# hash routing). Opaque to model-agnostic engine code; None for non-DSV4 models.
dsv4_args: Any | None = None
+ # DeepSeek-V4.1 payload (DeepseekV41Args): DSV41 layer roles, Engram, replay policy. Opaque
+ # to model-agnostic engine code; None for every other model.
+ dsv41_args: Any | None = None
# GLM-5.2 (glm_moe_dsa) MLA/DSA payload (GlmMoeDsaArgs): the MLA low-rank dims and the
# DSA indexer geometry the model module needs. Opaque to model-agnostic engine code;
# None for every other model.
@@ -448,7 +454,7 @@ class ModelConfig:
if isinstance(group, SWAAttentionGroupConfig):
return AttnType.SWA
if isinstance(group, DSV4AttentionGroupConfig):
- return AttnType.DSV4
+ return AttnType.DSV41 if group.variant == "v41" else AttnType.DSV4
return _full_group_attn_type(group)
def kv_cache_group_specs(self) -> Tuple[KVCacheGroupSpec, ...]:
@@ -502,7 +508,7 @@ class ModelConfig:
num_kv_heads=group.num_kv_heads,
head_dim=group.head_dim,
sliding_window=group.sliding_window,
- attn_type=AttnType.DSV4,
+ attn_type=AttnType.DSV41 if group.variant == "v41" else AttnType.DSV4,
)
)
return tuple(specs)
diff --git a/python/freetoken/models/deepseek_v4/__init__.py b/python/freetoken/models/deepseek_v4/__init__.py
index 37f5076b..e19438d1 100644
--- a/python/freetoken/models/deepseek_v4/__init__.py
+++ b/python/freetoken/models/deepseek_v4/__init__.py
@@ -10,7 +10,7 @@ of experts is resident on the GPU (the framework's core acceleration).
DeepSeek-V4-Flash is a first-class registered model on the shared paged-KV engine:
its window / compressed-attention / compressed-index KV live in DSV4-owned pools
addressed by page tables, and sparse attention is a physical-slot gather (see
-:mod:`freetoken.attention.dsv4_sparse` and :mod:`freetoken.kvcache.dsv4_paged_pool`).
+:mod:`freetoken.attention.dsv4_sparse` and :mod:`freetoken.kvcache.dsv4.v4_pool`).
"""
from .args import DeepseekV4Args, load_args
diff --git a/python/freetoken/models/deepseek_v4/model.py b/python/freetoken/models/deepseek_v4/model.py
index a2d4387a..9faefb83 100644
--- a/python/freetoken/models/deepseek_v4/model.py
+++ b/python/freetoken/models/deepseek_v4/model.py
@@ -4,7 +4,7 @@ A faithful single-stream port of the reference (MLA attention with a sliding win
+ stateful KV compressors / Lightning Indexer, manifold-constrained Hyper-Connections,
sqrtsoftplus / hash MoE), wired onto FreeToken's shared paged engine:
- - KV lives in DSV4-owned paged pools (:class:`~freetoken.kvcache.dsv4_paged_pool.DSV4PagedKVCache`):
+ - KV lives in DSV4-owned paged pools (:class:`~freetoken.kvcache.dsv4.v4_pool.DSV4PagedKVCache`):
each layer's window-ring + compressed KV is a region of a shared global pool, addressed by
per-layer slot maps. Sparse attention is a PAGED physical-slot gather:
``sparse_attn_paged`` reads KV directly from the two global pools (window / compressed) at
diff --git a/python/freetoken/models/deepseek_v41/__init__.py b/python/freetoken/models/deepseek_v41/__init__.py
new file mode 100644
index 00000000..bcdef078
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/__init__.py
@@ -0,0 +1,14 @@
+"""DeepSeek-V4.1-Flash: Causal Encoder-Decoder MoE with Compressed Sparse Attention 2.
+
+40 layers (20 encoder + 20 decoder), every layer a sliding window (128) plus -- from layer 2 -- global
+sparse attention over cross-layer-shared compressed KV in packed fp4 (Full / Reindex / Reuse
+modes, a hierarchical candidate pool in the decoder), single-pass mHC residual streams, Engram n-gram
+memory at layers 1 and 14 streamed from disk, one shared + 384 routed MXFP4 experts per layer served
+from the offload cache. Images use the DeepSeek ViT; DSpark speculative decoding is not served.
+"""
+
+from .config import parse_config
+from .model import DeepseekV41ForCausalLM
+from .weight import ftw_side_files, iter_expert_pieces, iter_weights, iter_vision_weights
+
+__all__ = ["DeepseekV41ForCausalLM", "parse_config", "iter_weights", "iter_vision_weights", "iter_expert_pieces", "ftw_side_files"]
diff --git a/python/freetoken/models/deepseek_v41/args.py b/python/freetoken/models/deepseek_v41/args.py
new file mode 100644
index 00000000..867dcb21
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/args.py
@@ -0,0 +1,225 @@
+"""DeepSeek-V4.1-Flash hyperparameters and the static per-layer attention roles.
+
+``DeepseekV41Args`` mirrors the reference ``ModelArgs`` (``inference/model.py``) field for field,
+read from the HF ``config.json``'s ``text_config`` (the ``inference/config.json`` dialect is also
+accepted). Everything the engine reconciles at runtime (``max_seq_len``, the replay policy) is
+attached by ``parse_config`` / the engine, not read from the checkpoint.
+
+``LayerRole`` is the attention mode table (tech report sec. 2.3.1): every compressing layer is Full
+(owns the compressed KV + indexer K), Reindex (own indexer over a shared K) or Reuse (shared
+Top-K); the candidate pool of the Hierarchical Sparse Indexer (sec. 2.3.2) is produced by the
+first decoder Full layer and consumed by the decoder Reindex layers after it.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any, Literal
+
+# text_config key -> ModelArgs key, where the two dialects differ
+_HF_TO_ARGS = {
+ "hidden_size": "dim",
+ "moe_intermediate_size": "moe_inter_dim",
+ "num_hidden_layers": "n_layers",
+ "num_nextn_predict_layers": "n_mtp_layers",
+ "num_attention_heads": "n_heads",
+ "num_experts_per_tok": "n_activated_experts",
+ "scoring_func": "score_func",
+ "routed_scaling_factor": "route_scale",
+ "qk_rope_head_dim": "rope_head_dim",
+ "rms_norm_eps": "norm_eps",
+ "sliding_window": "window_size",
+ "kv_source_layer_ids": "kv_source_layers",
+ "index_source_layer_ids": "index_source_layers",
+ "candidate_source_layer_id": "candidate_source_layer",
+ "engram_pad_token_id": "engram_pad_id",
+ "dspark_num_experts_per_tok": "dspark_n_activated_experts",
+}
+
+
+class Mode(str, Enum):
+ WINDOW = "window" # sliding window only (compress_ratio == 0)
+ FULL = "full" # computes main KV + indexer K, runs its indexer
+ REINDEX = "reindex" # reuses main KV + indexer K, runs its own indexer
+ REUSE = "reuse" # reuses main KV and the latest Top-K
+
+
+@dataclass(frozen=True)
+class LayerRole:
+ layer_id: int
+ ratio: int
+ mode: Mode
+ kv_source: int | None # the layer whose main KV / indexer K this layer reads (itself when Full)
+ is_candidate_source: bool # builds the hierarchical candidate pool
+ uses_candidates: bool # indexes only inside the candidate pool
+
+ @property
+ def compresses(self) -> bool:
+ return self.ratio > 0
+
+ @property
+ def has_indexer(self) -> bool:
+ return self.mode in (Mode.FULL, Mode.REINDEX)
+
+
+@dataclass
+class DeepseekV41Args:
+ vocab_size: int = 129280
+ dim: int = 5120
+ moe_inter_dim: int = 2304
+ n_layers: int = 40
+ n_mtp_layers: int = 3
+ n_heads: int = 64
+ # moe
+ n_routed_experts: int = 384
+ n_shared_experts: int = 1
+ n_activated_experts: int = 6
+ score_func: Literal["softmax", "sigmoid", "sqrtsoftplus"] = "sqrtsoftplus"
+ norm_topk_prob: bool = True
+ route_scale: float = 1.5
+ swiglu_limit: float = 10.0
+ # attention: latent q, one shared latent kv head, grouped low-rank output projection
+ q_lora_rank: int = 1280
+ head_dim: int = 512
+ rope_head_dim: int = 64
+ norm_eps: float = 1e-20
+ o_groups: int = 8
+ o_lora_rank: int = 1024
+ # sparse attention
+ window_size: int = 128
+ compress_ratios: tuple[int, ...] = () # one per layer, MTP layers included
+ kv_source_layers: tuple[int, ...] = ()
+ index_source_layers: tuple[int, ...] = ()
+ compress_rope_theta: float = 160000.0
+ original_seq_len: int = 65536
+ rope_theta: float = 10000.0
+ rope_factor: float = 16.0
+ beta_fast: int = 32
+ beta_slow: int = 1
+ index_n_heads: int = 32
+ index_head_dim: int = 128
+ index_topk: int = 512
+ candidate_source_layer: int = -1
+ candidate_topk_blocks: int = 0
+ candidate_block_size: int = 0
+ # hyper-connections (single-pass mHC)
+ hc_mult: int = 4
+ hc_sinkhorn_iters: int = 20
+ hc_eps: float = 1e-6
+ # engram
+ engram_layer_ids: tuple[int, ...] = ()
+ engram_num_embeddings: tuple[int, ...] = ()
+ engram_max_ngram_size: int = 4
+ engram_vocab_size: int = 0
+ engram_n_heads: int = 0
+ engram_head_dim: int = 0
+ engram_pad_id: int = 2
+ engram_compressed_vocab_size: int = 0
+ # dspark (weights are not served in v1; kept so the config round-trips)
+ dspark_block_size: int = 0
+ dspark_target_layer_ids: tuple[int, ...] = ()
+ # ---- runtime (engine-reconciled, never read from the checkpoint) ----
+ max_seq_len: int = 4096
+ max_batch_size: int = 1
+ # "bounded": the decoder runs on the prompt's last window_size tokens with the sliding window
+ # truncated there (Decoder SWA Bounded Replay, tech report sec. 3.2.2); "exact": on every token.
+ swa_decoder_replay: Literal["bounded", "exact"] = "bounded"
+ roles: tuple[LayerRole, ...] = field(default_factory=tuple, repr=False)
+
+ def __post_init__(self) -> None:
+ for name in ("compress_ratios", "kv_source_layers", "index_source_layers", "engram_layer_ids",
+ "engram_num_embeddings", "dspark_target_layer_ids"):
+ setattr(self, name, tuple(int(v) for v in getattr(self, name)))
+ if len(self.compress_ratios) < self.n_layers:
+ raise ValueError(f"compress_ratios has {len(self.compress_ratios)} entries for {self.n_layers} layers")
+ if self.n_shared_experts != 1:
+ raise ValueError("DeepSeek-V4.1 has exactly one shared expert")
+ if self.hc_mult < 1:
+ raise ValueError("hc_mult must be positive")
+ self.roles = tuple(self._role(layer) for layer in range(self.n_layers))
+ if len(self.engram_layer_ids) != len(self.engram_num_embeddings):
+ raise ValueError("engram_layer_ids and engram_num_embeddings differ in length")
+
+ # ---- derived ----
+ def _role(self, layer: int) -> LayerRole:
+ ratio = self.compress_ratios[layer]
+ if ratio == 0:
+ return LayerRole(layer, 0, Mode.WINDOW, None, False, False)
+ sources = [s for s in self.kv_source_layers if s <= layer]
+ if not sources:
+ raise ValueError(f"layer {layer} compresses but no kv source precedes it")
+ source = max(sources)
+ if self.compress_ratios[source] != ratio:
+ raise ValueError(f"layer {layer} (ratio {ratio}) reads kv source {source} (ratio {self.compress_ratios[source]})")
+ if layer == source:
+ mode = Mode.FULL
+ elif layer in self.index_source_layers:
+ mode = Mode.REINDEX
+ else:
+ mode = Mode.REUSE
+ cand = self.candidate_source_layer
+ is_cand_src = cand >= 0 and layer == cand
+ uses = cand >= 0 and cand < layer and mode is Mode.REINDEX
+ if uses and self.kv_source_layers and max(s for s in self.kv_source_layers if s <= cand) != source:
+ raise ValueError(f"layer {layer} would index the candidate pool of layer {cand} over a different kv source")
+ return LayerRole(layer, ratio, mode, source, is_cand_src, uses)
+
+ @property
+ def nope_head_dim(self) -> int:
+ return self.head_dim - self.rope_head_dim
+
+ @property
+ def backbone_kv_sources(self) -> tuple[int, ...]:
+ return tuple(s for s in self.kv_source_layers if s < self.n_layers)
+
+ @property
+ def decoder_start_layer(self) -> int:
+ """The first decoder layer: the last kv source (CED projects the decoder's global KV from
+ its input, the final encoder hidden state)."""
+ return max(self.backbone_kv_sources) if self.backbone_kv_sources else self.n_layers
+
+ @property
+ def engram_hash_cols(self) -> int:
+ return (self.engram_max_ngram_size - 1) * self.engram_n_heads
+
+ @property
+ def index_softmax_scale(self) -> float:
+ return self.index_head_dim**-0.5
+
+ def freqs_params(self, layer: int) -> tuple[int, int, float, float, int, int]:
+ """``(rotary_dim, original_seq_len, theta, factor, beta_fast, beta_slow)``: compressing layers
+ rotate at the compressed theta under YaRN, window-only layers at the base theta without it."""
+ if self.compress_ratios[layer]:
+ return self.rope_head_dim, self.original_seq_len, self.compress_rope_theta, self.rope_factor, self.beta_fast, self.beta_slow
+ return self.rope_head_dim, 0, self.rope_theta, self.rope_factor, self.beta_fast, self.beta_slow
+
+ @classmethod
+ def from_hf(cls, hf_config: Any) -> "DeepseekV41Args":
+ """From the HF config (``text_config`` of the multimodal wrapper, or the text config itself,
+ or the ``inference/config.json`` dialect)."""
+ text = getattr(hf_config, "text_config", None) or hf_config
+ if isinstance(text, dict):
+ raw = text
+ elif hasattr(text, "to_dict"): # PretrainedConfig / RawConfigShim
+ raw = text.to_dict()
+ else:
+ raw = vars(text)
+ names = {f.name for f in cls.__dataclass_fields__.values()} - {"roles", "max_seq_len", "max_batch_size", "swa_decoder_replay"}
+ kwargs: dict[str, Any] = {}
+ for key, value in raw.items():
+ key = _HF_TO_ARGS.get(key, key)
+ if key in names:
+ kwargs[key] = value
+ # HF nests the YaRN parameters; the inference dialect keeps them flat
+ scaling = raw.get("rope_scaling") or raw.get("rope_parameters")
+ if isinstance(scaling, dict):
+ if "factor" in scaling:
+ kwargs["rope_factor"] = scaling["factor"]
+ for src, dst in (("beta_fast", "beta_fast"), ("beta_slow", "beta_slow"), ("original_max_position_embeddings", "original_seq_len")):
+ if src in scaling:
+ kwargs[dst] = scaling[src]
+ return cls(**kwargs)
+
+
+__all__ = ["DeepseekV41Args", "LayerRole", "Mode"]
diff --git a/python/freetoken/models/deepseek_v41/attention.py b/python/freetoken/models/deepseek_v41/attention.py
new file mode 100644
index 00000000..1643ddb6
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/attention.py
@@ -0,0 +1,305 @@
+"""DSV41 attention layer (reference ``Attention``): latent MLA over a sliding window plus, on
+compressing layers, ``index_topk`` compressed positions -- one paged sparse-attention call.
+
+Per layer mode (``LayerRole``):
+
+* Full -- ``publish_*`` compresses its input into the kv source's main KV (fp4) and index keys
+ (fp4), then its indexer selects fresh Top-K rows and publishes them on the forward's
+ ``SharedSelection``; the decoder's first Full layer also builds the candidate pool.
+* Reindex -- selects fresh Top-K rows with its own indexer over the shared keys (inside the
+ candidate pool in the decoder) and publishes them.
+* Reuse -- attends over the latest published Top-K rows.
+* Window -- the sliding window only.
+
+All modes compute their own query and sliding-window KV (fp8 into the layer's window ring). The
+attention output is rotated back (K == V share one rotated latent) and projected through the
+grouped low-rank ``wo_a`` (block-diagonal over ``o_groups``; fp8 payload kept, W8A16 at decode) and ``wo_b``.
+
+Prefill runs ragged over a flat token stream tiled by ``PrefillSegment``s; the compressor / indexer
+state is per request, so those run per segment. Decode is batched (one token per row) and
+CUDA-graph safe: every gather reads the backend's snapshot, never the live page table.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import List
+
+import torch
+from freetoken.attention.dsv41_sparse import PrefillSegment
+from freetoken.core import get_global_ctx
+from freetoken.kernel.triton.dsv4.fp8_linear import GEMV_MAX_M, dequant_block_fp8, grouped_w8a16_gemv
+from freetoken.layers import BaseOP, LinearReplicated, RMSNorm
+from freetoken.layers.quantization import QuantKind
+from freetoken.layers.quantization.scheme import fp8_block_size
+
+from .args import DeepseekV41Args, LayerRole, Mode
+from .compressor import Compressor
+from .indexer import Indexer
+from .rope import apply_rotary_emb, apply_rotary_emb_decode, get_freqs_cis
+
+
+class DSV41Attention(BaseOP):
+ def __init__(self, args: DeepseekV41Args, role: LayerRole, *, quant_config=None, prefix: str = ""):
+ self.args = args
+ self.role = role
+ self.layer_id = role.layer_id
+ self.n_heads = args.n_heads
+ self.head_dim = args.head_dim
+ self.rope_dim = args.rope_head_dim
+ self.n_groups = args.o_groups
+ self.o_lora_rank = args.o_lora_rank
+ self.window = args.window_size
+ self.softmax_scale = args.head_dim**-0.5
+
+ self.wq_a = LinearReplicated(args.dim, args.q_lora_rank, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wq_a")
+ self.q_norm = RMSNorm(args.q_lora_rank, args.norm_eps)
+ self.wq_b = LinearReplicated(args.q_lora_rank, args.n_heads * args.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wq_b")
+ self.wkv = LinearReplicated(args.dim, args.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wkv")
+ self.kv_norm = RMSNorm(args.head_dim, args.norm_eps)
+ # Block-diagonal over groups. The reference dequantizes it to bf16 and runs a grouped einsum
+ # (the activation stays bf16); a quantized checkpoint keeps the e4m3 payload + e8m0 scale
+ # here and dequantizes in-kernel -- the same math, half the bytes per decode step, and no
+ # bf16 copy in VRAM. An unquantized checkpoint ships it bf16.
+ wo_a_scheme = quant_config.scheme_for(f"{prefix}.wo_a") if quant_config is not None else None
+ self._wo_a_block = fp8_block_size(wo_a_scheme) if wo_a_scheme is not None and wo_a_scheme.kind is QuantKind.FP8_BLOCK else None
+ rows, cols = args.o_groups * args.o_lora_rank, args.n_heads * args.head_dim // args.o_groups
+ if self._wo_a_block is not None:
+ self.wo_a = torch.empty(rows, cols, dtype=torch.float8_e4m3fn)
+ self.wo_a_scale = torch.empty(rows // self._wo_a_block, cols // self._wo_a_block, dtype=torch.float8_e8m0fnu)
+ else:
+ self.wo_a = torch.empty(rows, cols, dtype=torch.bfloat16)
+ self.wo_b = LinearReplicated(args.o_groups * args.o_lora_rank, args.dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wo_b")
+ self.attn_sink = torch.empty(args.n_heads, dtype=torch.float32)
+
+ self.compressor = Compressor(args, self.layer_id, role.ratio, quant_config=quant_config, prefix=f"{prefix}.compressor") if role.mode is Mode.FULL else None
+ self.indexer = Indexer(args, role, quant_config=quant_config, prefix=f"{prefix}.indexer") if role.has_indexer else None
+ self._freqs_params = args.freqs_params(self.layer_id)
+ self._freqs: torch.Tensor | None = None
+
+ # addressing lives on the live backend, pool buffers on the live pool (both read per access)
+ @property
+ def attn(self):
+ return get_global_ctx().attn_backend
+
+ @property
+ def freqs_key(self) -> tuple:
+ """The RoPE table this layer rotates with (window layers: base theta; compressing layers: the
+ compressed theta under YaRN). Layers with the same key share one table."""
+ return self._freqs_params
+
+ def bind(self, device: torch.device, tables: dict | None = None) -> None:
+ """Attach the RoPE table; ``tables`` (key -> table) lets the model share one table per key
+ instead of every layer holding its own ``[max_seq_len, rd // 2]`` copy."""
+ if tables is not None and self.freqs_key in tables:
+ self._freqs = tables[self.freqs_key]
+ return
+ rd, original_seq_len, theta, factor, beta_fast, beta_slow = self._freqs_params
+ self._freqs = get_freqs_cis(rd, self.args.max_seq_len, original_seq_len, theta, factor, beta_fast, beta_slow, device)
+ if tables is not None:
+ tables[self.freqs_key] = self._freqs
+
+ @property
+ def freqs(self) -> torch.Tensor:
+ assert self._freqs is not None, "attention layer is not bound (call bind first)"
+ return self._freqs
+
+ # ----- shared pieces -------------------------------------------------------------------
+ def _q(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ qr = self.q_norm.forward(self.wq_a.forward(x))
+ q = self.wq_b.forward(qr).unflatten(-1, (self.n_heads, self.head_dim))
+ return qr, q
+
+ def _kv(self, x: torch.Tensor) -> torch.Tensor:
+ return self.kv_norm.forward(self.wkv.forward(x))
+
+ def _wo(self, o: torch.Tensor) -> torch.Tensor:
+ """``wo_b(wo_a(o))``: up to ``GEMV_MAX_M`` rows the grouped W8A16 GEMV over the fp8 payload, else the
+ reference's grouped einsum over the bf16 (or dequantized) ``wo_a``."""
+ t = o.shape[0]
+ o = o.reshape(t, self.n_groups, -1)
+ if self._wo_a_block is not None and t <= GEMV_MAX_M:
+ return self.wo_b.forward(grouped_w8a16_gemv(o, self.wo_a, self.wo_a_scale, block=self._wo_a_block))
+ wo_a = self.wo_a if self._wo_a_block is None else dequant_block_fp8(self.wo_a, self.wo_a_scale, block=self._wo_a_block)
+ return self.wo_b.forward(torch.einsum("tgd,grd->tgr", o, wo_a.view(self.n_groups, self.o_lora_rank, -1)).flatten(1))
+
+ def load_state_dict(self, state_dict, *, prefix: str = "", _internal: bool = False) -> None:
+ key = f"{prefix}.wo_a" if prefix else "wo_a"
+ scale_key = f"{key}_scale"
+ if self._wo_a_block is None and scale_key in state_dict:
+ # a bf16 layer (no fp8 scheme) fed a quantized checkpoint dequantizes wo_a on the way in
+ from .weight import dequant_fp8_block
+
+ state_dict[key] = dequant_fp8_block(state_dict[key], state_dict.pop(scale_key))
+ elif self._wo_a_block is not None and scale_key not in state_dict:
+ # an FTW conversion made before wo_a kept its fp8 payload carries a dequantized bf16
+ # tensor and no scale: say so instead of failing on the missing key
+ raise RuntimeError(
+ f"{key}: the checkpoint has no {scale_key} for the fp8 wo_a layout; an FTW checkpoint "
+ "converted by an earlier build stores wo_a dequantized -- reconvert it with `ft checkpoint`"
+ )
+ super().load_state_dict(state_dict, prefix=prefix, _internal=_internal)
+
+ # ----- prefill -----------------------------------------------------------------------------
+ def publish_prefill(self, x: torch.Tensor, segments: List[PrefillSegment]) -> None:
+ """Full mode: compress this layer's input for every segment into the source's main KV and
+ index keys. Runs on EVERY new token (under bounded replay the decoder source publishes for
+ the whole prompt even though its attention runs on the prompt's last window only)."""
+ assert self.compressor is not None and self.indexer is not None
+ attn, ratio, src = self.attn, self.role.ratio, self.layer_id
+ for seg in segments:
+ slots = attn.window_slots_of(seg.table_idx, seg.start_pos, seg.end)
+ tail = None
+ if self.compressor.needs_tail_carry(seg.start_pos):
+ tail = int(attn.window_slots_of(seg.table_idx, seg.start_pos - 1, seg.start_pos).item())
+ latent, starts = self.compressor.forward_prefill(x[seg.offset : seg.offset + seg.n], seg.start_pos, slots, tail)
+ if not latent.shape[0]:
+ continue
+ rows = attn.compressed_rows_of(seg.table_idx, starts, ratio)
+ freqs = self.freqs.index_select(0, starts)
+ attn.store_index(self.indexer.index_keys(latent, freqs), src, rows)
+ apply_rotary_emb(latent[..., -self.rope_dim :], freqs)
+ attn.store_main(latent, src, rows)
+
+ def forward_prefill(self, x: torch.Tensor, segments: List[PrefillSegment], positions: torch.Tensor) -> torch.Tensor:
+ """Ragged prefill over ``x [T, dim]`` tiled by ``segments`` (``positions [T]`` absolute)."""
+ attn, role = self.attn, self.role
+ freqs = self.freqs.index_select(0, positions)
+ qr, q = self._q(x)
+ apply_rotary_emb(q[..., -self.rope_dim :], freqs)
+ kv = self._kv(x)
+ apply_rotary_emb(kv[..., -self.rope_dim :], freqs)
+
+ sel = attn.selection
+ win_parts, cmp_parts, pools = [], [], []
+ for seg in segments:
+ lo, hi = seg.offset, seg.offset + seg.n
+ attn.store_window(kv[lo:hi], self.layer_id, attn.layer_window_slots_of(self.layer_id, seg.table_idx, seg.start_pos, seg.end))
+ win_parts.append(attn.window_topk_prefill(seg, self.layer_id))
+ if not role.compresses:
+ continue
+ if role.has_indexer:
+ locs = attn.locs_prefill(seg.table_idx, seg.end // role.ratio * role.ratio)
+ cand = sel.candidates[:, lo:hi] if role.uses_candidates else None
+ rows, pool = self.indexer.select_prefill(
+ x[lo:hi], qr[lo:hi], freqs[lo:hi], start_pos=seg.start_pos, ratio=role.ratio, locs=locs, candidates=cand,
+ )
+ cmp_parts.append(rows)
+ if pool is not None:
+ pools.append(pool)
+ else:
+ assert sel.source == role.kv_source, "Reuse layer without a published selection for its source"
+ cmp_parts.append(sel.topk_rows[:, lo:hi])
+ if role.has_indexer:
+ sel.source, sel.topk_rows = role.kv_source, torch.cat(cmp_parts, dim=1)
+ if pools:
+ sel.candidates = torch.cat(pools, dim=1)
+ win = torch.cat(win_parts, dim=1)
+ topk = torch.cat([win, torch.cat(cmp_parts, dim=1)], dim=-1) if role.compresses else win
+
+ o = attn.attend(q.unsqueeze(0), self.layer_id, topk.to(torch.int32), self.window, self.attn_sink, self.softmax_scale)[0]
+ apply_rotary_emb(o[..., -self.rope_dim :], freqs, inverse=True)
+ return self._wo(o)
+
+ # ----- decode ------------------------------------------------------------------------------
+ def forward_decode(self, x: torch.Tensor, pos: torch.Tensor, rows: torch.Tensor, dctx: "DecodeStepContext", cmp_stage_cap: int) -> torch.Tensor:
+ """Batched single-token attention. ``pos [B]`` device positions, ``rows [B]`` local rows into
+ the decode snapshot, ``dctx`` the step's layer-invariant context (ring slots, window
+ candidates, RoPE rows), ``cmp_stage_cap`` the static compressed staging bound (max position
+ any row reaches; the capture width under a graph)."""
+ attn, role = self.attn, self.role
+ B = x.shape[0]
+ freqs_t = dctx.freqs[self.freqs_key]
+ qr, q = self._q(x)
+ apply_rotary_emb_decode(q[..., -self.rope_dim :].unsqueeze(1), freqs_t)
+ kv = self._kv(x)
+ apply_rotary_emb_decode(kv[..., -self.rope_dim :].unsqueeze(1), freqs_t)
+ # the shared ring context feeds the compressor's carry ring on every layer; a request-private
+ # window layer stores and reads its own KV through its ring context instead
+ window_slots, prev_window_slots = dctx.window_slots, dctx.prev_window_slots
+ private = dctx.private is not None and attn.pool.is_private_window(self.layer_id)
+ tier = dctx.private if private else dctx.shared
+ attn.store_window(kv, self.layer_id, tier.slots if private else window_slots)
+
+ cmp_counts = None
+ if role.compresses:
+ ratio, src = role.ratio, role.kv_source
+ n_stage = (cmp_stage_cap + 1) // ratio
+ sel = attn.selection
+ if role.mode is Mode.FULL:
+ latent, completed = self.compressor.forward_decode(x, pos, prev_window_slots, window_slots)
+ dst = attn.decode_store_rows(rows, pos, ratio, src, completed)
+ freqs_g = self.freqs.index_select(0, (pos + 1 - ratio).clamp_min(0))
+ k = self.indexer.k_norm.forward(self.indexer.wk.forward(latent))
+ apply_rotary_emb_decode(k[..., -self.rope_dim :].unsqueeze(1), freqs_g)
+ attn.store_index(k, src, dst)
+ apply_rotary_emb_decode(latent[..., -self.rope_dim :].unsqueeze(1), freqs_g)
+ attn.store_main(latent, src, dst)
+ if role.has_indexer:
+ cand = sel.candidates if role.uses_candidates else None
+ cmp_rows, counts, pool = self.indexer.select_decode(x, qr, freqs_t, pos=pos, ratio=ratio, locs=attn.snapshot(), T=n_stage, candidates=cand)
+ sel.source, sel.topk_rows, sel.cmp_counts = src, cmp_rows, counts
+ if pool is not None:
+ sel.candidates = pool
+ # the step's [window | compressed] candidate buffer of this tier takes the new rows once;
+ # the Reuse layers that follow read them in place (no per-layer cat / cast)
+ tier.topk[..., self.window :].copy_(cmp_rows)
+ else:
+ assert sel.source == src, "Reuse layer without a published selection for its source"
+ counts = sel.cmp_counts
+ topk = tier.topk
+ cmp_counts = counts
+ else:
+ topk = tier.window_topk
+
+ o = attn.attend(q.view(B, 1, self.n_heads, self.head_dim), self.layer_id, topk, self.window, self.attn_sink, self.softmax_scale, cmp_counts=cmp_counts)
+ apply_rotary_emb_decode(o[..., -self.rope_dim :], freqs_t, inverse=True)
+ return self._wo(o.view(B, self.n_heads, self.head_dim))
+
+
+@dataclass
+class TierCandidates:
+ """One window tier's per-step decode candidates: the request's store slot and the int32
+ ``[B, 1, win]`` window candidates, plus the ``[B, 1, win + topk]`` ``[window | compressed]``
+ buffer whose compressed half the index layers of that tier fill as they publish."""
+
+ slots: torch.Tensor
+ window_topk: torch.Tensor
+ topk: torch.Tensor
+
+
+@dataclass
+class DecodeStepContext:
+ """Everything a decode step resolves once for all layers (a capture records it once): the shared
+ pool's ring slots (the compressor's carry ring keys on them), the shared / private tier
+ candidates, and each RoPE table gathered at the step's positions."""
+
+ window_slots: torch.Tensor
+ prev_window_slots: torch.Tensor
+ shared: TierCandidates
+ private: TierCandidates | None
+ freqs: dict
+
+ @classmethod
+ def build(cls, layers, md, pos: torch.Tensor, rows: torch.Tensor, window: int, topk: int) -> "DecodeStepContext":
+ B = pos.shape[0]
+
+ def tier(slots, window_topk):
+ win32 = window_topk.to(torch.int32)
+ buf = torch.empty((B, 1, window + topk), dtype=torch.int32, device=pos.device)
+ buf[..., :window].copy_(window_topk)
+ return TierCandidates(slots, win32, buf)
+
+ window_slots, prev_window_slots, window_topk = md.window_ctx(pos, rows)
+ private = None
+ if get_global_ctx().kv_cache.private_window_layer_ids:
+ private = tier(*md.private_window_ctx(pos, rows))
+ tables: dict = {}
+ for layer in layers:
+ key = layer.freqs_key
+ if key not in tables:
+ tables[key] = layer.freqs.index_select(0, pos)
+ return cls(window_slots, prev_window_slots, tier(window_slots, window_topk), private, tables)
+
+
+__all__ = ["DSV41Attention", "DecodeStepContext", "TierCandidates"]
diff --git a/python/freetoken/models/deepseek_v41/compressor.py b/python/freetoken/models/deepseek_v41/compressor.py
new file mode 100644
index 00000000..5d5d07fe
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/compressor.py
@@ -0,0 +1,131 @@
+"""DSV41 KV compressor: ``compress_ratio`` consecutive tokens -> one latent (reference ``Compressor``).
+
+* ratio 1 (the CED decoder source): a plain bf16 projection + RMSNorm, one latent per token.
+* ratio > 1 (the encoder sources): ``wkv`` / ``wgate`` promoted to fp32, a softmax over the group's
+ gate scores pools the group's ``wkv`` rows; the trailing partial group is carried across chunks
+ and decode steps in the pool's compress-state ring (per window page, so a page-aligned radix hit
+ resumes it by value -- see ``DSV41SparseAttnBackend`` for the addressing).
+
+Returns the latent BEFORE RoPE: the indexer key is projected from the unrotated latent, the
+attention layer rotates and quantizes it afterwards.
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn.functional as F
+from freetoken.core import get_global_ctx
+from freetoken.layers import BaseOP, LinearReplicated, RMSNorm
+
+from .args import DeepseekV41Args
+
+
+class Compressor(BaseOP):
+ def __init__(self, args: DeepseekV41Args, layer_id: int, ratio: int, *, quant_config=None, prefix: str = ""):
+ assert ratio >= 1
+ self.layer_id = layer_id
+ self.ratio = ratio
+ self.dim = args.dim
+ self.head_dim = args.head_dim
+ self.window = args.window_size
+ if ratio > 1:
+ # fp32 like the reference (the softmax pooling runs in fp32); the checkpoint ships bf16
+ self.wkv = torch.empty(args.head_dim, args.dim, dtype=torch.float32)
+ self.wgate = torch.empty(args.head_dim, args.dim, dtype=torch.float32)
+ else:
+ self.wkv = LinearReplicated(args.dim, args.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wkv")
+ self.norm = RMSNorm(args.head_dim, args.norm_eps)
+
+ @property
+ def attn(self):
+ return get_global_ctx().attn_backend
+
+ def _project(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ """fp32 ``wkv`` / ``wgate`` projections."""
+ xf = x.float()
+ return F.linear(xf, self.wkv), F.linear(xf, self.wgate)
+
+ # ----- prefill -----------------------------------------------------------------------
+ def needs_tail_carry(self, start_pos: int) -> bool:
+ """Whether a prefill from ``start_pos`` resumes a partial group (never at a page-aligned start)."""
+ return self.ratio > 1 and start_pos % self.ratio != 0
+
+ def forward_prefill(
+ self, x: torch.Tensor, start_pos: int, window_slots: torch.Tensor, tail_window_slot: int | None = None,
+ ) -> tuple[torch.Tensor, torch.Tensor]:
+ """Compress the tokens ``[start_pos, start_pos + n)`` of one request.
+
+ ``window_slots`` are the tokens' window slots (the ring block of each window page carries the
+ partial group); ``tail_window_slot`` is the previous token's slot, needed only when
+ ``needs_tail_carry(start_pos)``. Returns ``(latent [G, head_dim] bf16 pre-RoPE, group_starts [G]
+ int64)`` for every group that completed in this call.
+ """
+ n = x.shape[0]
+ device = x.device
+ if self.ratio == 1:
+ latent = self.norm.forward(self.wkv.forward(x))
+ return latent, torch.arange(start_pos, start_pos + n, device=device)
+
+ ratio, d = self.ratio, self.head_dim
+ kv, score = self._project(x)
+ # the partial group in progress at start_pos: its rows live in the previous token's ring block
+ offset = start_pos % ratio
+ if offset:
+ assert tail_window_slot is not None, "resuming mid-group needs the previous token's window slot"
+ carry = self.attn.read_carry(self.layer_id, tail_window_slot) # [ratio, 2d]
+ kv = torch.cat([carry[:offset, :d], kv], dim=0)
+ score = torch.cat([carry[:offset, d:], score], dim=0)
+ total = kv.shape[0]
+ groups = total // ratio
+ remainder = total % ratio
+ cut = groups * ratio
+ if groups:
+ kv_g = kv[:cut].view(groups, ratio, d)
+ score_g = score[:cut].view(groups, ratio, d)
+ pooled = (kv_g * score_g.softmax(dim=1)).sum(dim=1)
+ latent = self.norm.forward(pooled.to(x.dtype))
+ else:
+ latent = x.new_empty(0, d)
+ group_starts = (start_pos - offset) + ratio * torch.arange(groups, device=device)
+ # ring: the reset block at every page boundary crossed (a page holds whole groups), and the
+ # in-progress tail (the remainder rows) on the last page so decode / the next chunk resume
+ end = start_pos + n
+ self.attn.write_boundary_carries(self.layer_id, lo=start_pos, hi=end, window_slots=window_slots)
+ if end % self.window != 0:
+ block = self._carry_block(kv[cut:], score[cut:], remainder)
+ self.attn.write_carry(self.layer_id, int(window_slots[-1].item()), block)
+ return latent, group_starts
+
+ def _carry_block(self, kv_tail: torch.Tensor, score_tail: torch.Tensor, filled: int) -> torch.Tensor:
+ ratio, d = self.ratio, self.head_dim
+ ks = kv_tail.new_zeros(ratio, d)
+ ss = kv_tail.new_full((ratio, d), float("-inf"))
+ if filled:
+ ks[:filled] = kv_tail
+ ss[:filled] = score_tail
+ return torch.cat([ks, ss], dim=-1)
+
+ # ----- decode ------------------------------------------------------------------------
+ def forward_decode(self, x: torch.Tensor, pos: torch.Tensor, prev_window_slots: torch.Tensor, window_slots: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ """One token per row. Returns ``(latent [B, head_dim] bf16 pre-RoPE, completed [B] bool)``;
+ a row whose group did not complete carries a meaningless latent the caller discards."""
+ B = x.shape[0]
+ if self.ratio == 1:
+ latent = self.norm.forward(self.wkv.forward(x))
+ return latent, torch.ones(B, dtype=torch.bool, device=x.device)
+
+ from freetoken.kernel.triton.dsv4.compress import gated_pool
+
+ ratio, d = self.ratio, self.head_dim
+ kv, score = self._project(x)
+ idx = (pos % ratio).view(B, 1, 1).expand(B, 1, d)
+ block = self.attn.read_carry_blocks(self.layer_id, prev_window_slots) # [B, ratio, 2d]
+ ks = block[..., :d].clone().scatter_(1, idx, kv.unsqueeze(1))
+ ss = block[..., d:].clone().scatter_(1, idx, score.unsqueeze(1))
+ self.attn.write_carry_blocks(self.layer_id, window_slots, torch.cat([ks, ss], dim=-1))
+ pooled = gated_pool(ks, ss, x.dtype).view(B, d)
+ completed = (pos + 1) % ratio == 0
+ return self.norm.forward(pooled), completed
+
+
+__all__ = ["Compressor"]
diff --git a/python/freetoken/models/deepseek_v41/config.py b/python/freetoken/models/deepseek_v41/config.py
new file mode 100644
index 00000000..9e738939
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/config.py
@@ -0,0 +1,95 @@
+"""Engine-facing config for DeepSeek-V4.1-Flash.
+
+``parse_config`` maps the standard transformer fields into :class:`ModelConfig`, carries the full
+:class:`DeepseekV41Args` in ``ModelConfig.dsv41_args`` for the model module and the KV pool (which
+sizes its tiers from it), and declares the ``v41`` variant of the DSV4 attention group.
+The engine reconciles the runtime knobs (``max_seq_len``, ``swa_decoder_replay``) onto ``dsv41_args``
+at config resolution.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, fields
+from typing import Any
+
+from freetoken.models.config import DSV4AttentionGroupConfig, ModelConfig, RotaryConfig
+
+from .args import DeepseekV41Args
+
+
+@dataclass(frozen=True)
+class VisionConfig:
+ num_hidden_layers: int = 32
+ hidden_size: int = 1024
+ num_attention_heads: int = 16
+ intermediate_size: int = 2816
+ patch_size: int = 14
+ rope_theta: float = 10000.0
+ downsample_ratio: int = 3
+ max_image_tokens: int = 1024
+ min_pixels: int = 544 * 544
+ max_wh_ratio: float | None = None
+
+
+def parse_config(hf_config: Any) -> ModelConfig:
+ args = DeepseekV41Args.from_hf(hf_config)
+ text = getattr(hf_config, "text_config", None) or hf_config
+ max_position = int(getattr(text, "max_position_embeddings", 0) or getattr(hf_config, "max_position_embeddings", 0) or 0)
+ if max_position <= 0:
+ max_position = int(args.original_seq_len * args.rope_factor)
+ vc = getattr(hf_config, "vision_config", None)
+ vision = VisionConfig(**{f.name: getattr(vc, f.name, f.default) for f in fields(VisionConfig)}) if vc is not None else None
+
+ return ModelConfig(
+ num_layers=args.n_layers,
+ num_qo_heads=args.n_heads,
+ num_kv_heads=1, # one shared latent KV head (K == V)
+ head_dim=args.head_dim,
+ hidden_size=args.dim,
+ vocab_size=args.vocab_size,
+ intermediate_size=args.moe_inter_dim,
+ hidden_act="silu",
+ rms_norm_eps=args.norm_eps,
+ tie_word_embeddings=False,
+ rotary_config=RotaryConfig(
+ head_dim=args.head_dim,
+ rotary_dim=args.rope_head_dim,
+ max_position=max_position,
+ base=args.rope_theta,
+ scaling={
+ "rope_type": "yarn",
+ "factor": args.rope_factor,
+ "beta_fast": args.beta_fast,
+ "beta_slow": args.beta_slow,
+ "original_max_position_embeddings": args.original_seq_len,
+ },
+ ),
+ num_experts=args.n_routed_experts,
+ num_experts_per_tok=args.n_activated_experts,
+ moe_intermediate_size=args.moe_inter_dim,
+ norm_topk_prob=args.norm_topk_prob,
+ n_shared_experts=args.n_shared_experts,
+ routed_scaling_factor=args.route_scale,
+ swiglu_limit=args.swiglu_limit,
+ model_type="deepseek_v41",
+ architectures=["DeepseekV41ForCausalLM"],
+ moe_enabled=True,
+ expert_quant="ds_fp4",
+ attn_sm_scale=args.head_dim**-0.5,
+ dsv41_args=args,
+ vision_config=vision,
+ image_token_id=getattr(hf_config, "image_token_id", None),
+ attention_groups=(
+ DSV4AttentionGroupConfig(
+ name="dsv41",
+ layer_ids=tuple(range(args.n_layers)),
+ num_kv_heads=1,
+ head_dim=args.head_dim,
+ sliding_window=args.window_size,
+ variant="v41",
+ ),
+ ),
+ )
+
+
+__all__ = ["parse_config"]
diff --git a/python/freetoken/models/deepseek_v41/engram.py b/python/freetoken/models/deepseek_v41/engram.py
new file mode 100644
index 00000000..1c997a7c
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/engram.py
@@ -0,0 +1,262 @@
+"""Engram conditional memory (reference ``inference/engram.py`` + ``Engram``): n-gram hash lookups
+added into the residual streams before layers 1 and 14.
+
+Hashing (host side, ``EngramHash``): token ids are first mapped through a compressed vocabulary
+(tokens that normalize alike collapse together), then each position is hashed with the
+``max_ngram_size - 1`` tokens before it -- a rolling XOR of ``id * multiplier`` over the lookback,
+one bucket range (a distinct prime modulus) per (n-gram order, head). Lookback stops at the sequence
+start (pad id) and never crosses an image span. Addresses depend
+only on the token ids, so the rows can be fetched before the forward starts.
+
+Lookup (``EngramTable`` protocol): the 384M-row fp8 tables live on disk; a backend gathers the
+``n_hash_cols`` rows of every token into a device tensor ``[T, n_hash_cols * head_dim]`` in bf16.
+
+Gate (``EngramLayer``): ``wkv`` (fp8 linear) turns the gathered rows into one key per residual
+stream plus a shared value; the gate is a normalized dot product of stream and key, signed-sqrt
+then sigmoid; ``stream += gate * value``.
+"""
+
+from __future__ import annotations
+
+from typing import Protocol
+
+import numpy as np
+import torch
+from freetoken.layers import BaseOP, LinearReplicated
+
+from .args import DeepseekV41Args
+
+
+# ----- hashing ---------------------------------------------------------------------------------
+
+
+def _is_prime(n: int) -> bool:
+ """Deterministic Miller-Rabin for n < 3.3e24 (the bases below are a proven set), matching
+ ``sympy.isprime`` on every 64-bit input."""
+ if n < 2:
+ return False
+ small = (2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41)
+ for p in small:
+ if n % p == 0:
+ return n == p
+ d, s = n - 1, 0
+ while d % 2 == 0:
+ d //= 2
+ s += 1
+ for a in small:
+ x = pow(a, d, n)
+ if x in (1, n - 1):
+ continue
+ for _ in range(s - 1):
+ x = pow(x, 2, n)
+ if x == n - 1:
+ break
+ else:
+ return False
+ return True
+
+
+def find_next_prime(start: int, seen: set[int]) -> int:
+ """The smallest prime above ``start`` not handed out yet (reference ``find_next_prime``)."""
+ candidate = start + 1
+ while not _is_prime(candidate) or candidate in seen:
+ candidate += 1
+ return candidate
+
+
+def bucket_primes(args: DeepseekV41Args) -> list[list[list[int]]]:
+ """``[layer][ngram order - 2][head]`` bucket moduli: distinct primes drawn in order from
+ ``engram_vocab_size - 1`` upward, never reused across layers / orders / heads."""
+ primes: list[list[list[int]]] = []
+ seen: set[int] = set()
+ for _ in args.engram_layer_ids:
+ per_ngram = []
+ for _ in range(args.engram_max_ngram_size - 1):
+ sizes, current = [], args.engram_vocab_size - 1
+ for _ in range(args.engram_n_heads):
+ current = find_next_prime(current, seen)
+ seen.add(current)
+ sizes.append(current)
+ per_ngram.append(sizes)
+ primes.append(per_ngram)
+ return primes
+
+
+def hash_multipliers(layer_ids: tuple[int, ...], max_ngram_size: int, compressed_vocab_size: int) -> torch.Tensor:
+ """``[n_layers, max_ngram_size]`` odd int64 multipliers from a per-layer RNG (reference
+ ``compute_hash_multipliers``), bounded so ``id * multiplier`` cannot overflow int64."""
+ max_long = np.iinfo(np.int64).max
+ bound = max(1, (max_long // compressed_vocab_size) // 2)
+ rows = []
+ for layer_id in layer_ids:
+ gen = np.random.default_rng(10007 * layer_id)
+ values = gen.integers(low=0, high=bound, size=(max_ngram_size,), dtype=np.int64)
+ rows.append(torch.tensor(values * 2 + 1))
+ return torch.stack(rows)
+
+
+def build_compressed_token_map(tokenizer) -> tuple[list[int], int]:
+ """Token id -> compressed id: tokens that normalize alike (NFKC, accents stripped, lowercased,
+ whitespace collapsed) share one id. Reference ``build_compressed_token_map``; the size feeds the
+ multipliers, so it is asserted against ``engram_compressed_vocab_size``."""
+ from tokenizers import Regex, normalizers
+
+ sentinel = "\ue000"
+ normalizer = normalizers.Sequence([
+ normalizers.NFKC(),
+ normalizers.NFD(),
+ normalizers.StripAccents(),
+ normalizers.Lowercase(),
+ normalizers.Replace(Regex(r"[ \t\r\n]+"), " "),
+ normalizers.Replace(Regex(r"^ $"), sentinel),
+ normalizers.Strip(),
+ normalizers.Replace(sentinel, " "),
+ ])
+ backend = tokenizer.backend_tokenizer
+ key_to_new: dict[str, int] = {}
+ lookup = [0] * len(tokenizer)
+ for token_id in range(len(tokenizer)):
+ text = backend.decode([token_id], skip_special_tokens=False)
+ if "\ufffd" in text:
+ key = backend.id_to_token(token_id)
+ else:
+ normalized = normalizer.normalize_str(text)
+ key = normalized if normalized else text
+ new_id = key_to_new.get(key)
+ if new_id is None:
+ new_id = len(key_to_new)
+ key_to_new[key] = new_id
+ lookup[token_id] = new_id
+ return lookup, len(key_to_new)
+
+
+class EngramHash:
+ """Maps token runs to table rows: ``row_ids(tokens) -> [T, n_layers, n_hash_cols]`` int64.
+
+ ``token_map`` is the compressed vocabulary (``build_compressed_token_map``); ``pad_id`` is the
+ compressed id of ``engram_pad_id``. Runs on the host (numpy / CPU torch) since the addresses
+ are needed before the forward launches.
+ """
+
+ def __init__(self, args: DeepseekV41Args, token_map: list[int] | torch.Tensor, compressed_vocab_size: int):
+ if compressed_vocab_size != args.engram_compressed_vocab_size:
+ raise ValueError(
+ f"compressed vocab has {compressed_vocab_size} ids, config says {args.engram_compressed_vocab_size}; "
+ "the hash multipliers derive from it, so the tokenizer does not match the checkpoint"
+ )
+ self.args = args
+ self.max_ngram = args.engram_max_ngram_size
+ self.token_map = torch.as_tensor(token_map, dtype=torch.int64)
+ self.pad_id = int(self.token_map[args.engram_pad_id])
+ primes = bucket_primes(args)
+ flat = [[p for per_ngram in layer for p in per_ngram] for layer in primes]
+ self.primes = torch.tensor(primes, dtype=torch.int64) # [L, ngram-1, heads]
+ self.offsets = torch.tensor(np.array([np.cumsum([0, *sizes[:-1]]) for sizes in flat]), dtype=torch.int64) # [L, cols]
+ self.multipliers = hash_multipliers(args.engram_layer_ids, self.max_ngram, compressed_vocab_size) # [L, ngram]
+ self.rows_per_layer = [int(o[-1] + s[-1]) for o, s in zip(self.offsets.tolist(), [[p for pn in layer for p in pn] for layer in primes])]
+
+ @property
+ def n_cols(self) -> int:
+ return (self.max_ngram - 1) * self.args.engram_n_heads
+
+ def row_ids(self, tokens: torch.Tensor, context: torch.Tensor | None = None) -> torch.Tensor:
+ """Rows for ``tokens [T]`` (raw ids, one contiguous run). ``context`` holds the
+ ``max_ngram_size - 1`` raw ids immediately before the run (fewer -> the run starts a
+ sequence and the missing lookback is the pad id). Returns ``[T, n_layers, n_hash_cols]``."""
+ tokens = tokens.to(torch.int64).cpu()
+ ctx_len = self.max_ngram - 1
+ if context is None:
+ context = tokens.new_empty(0)
+ context = context.to(torch.int64).cpu()[-ctx_len:] if ctx_len else tokens.new_empty(0)
+ raw = torch.cat([context, tokens])
+ # Content-pad ids identify image spans; n-grams stop at them, including across chunks.
+ dead = (raw < 0) | (raw >= self.token_map.numel())
+ compressed = self.token_map[raw.clamp(0, self.token_map.numel() - 1)]
+ n_ctx = context.numel()
+ T = tokens.numel()
+ positions = torch.arange(n_ctx, n_ctx + T)
+ cols = []
+ blocked = torch.zeros(T, dtype=torch.bool)
+ for shift in range(self.max_ngram):
+ src = positions - shift
+ blocked |= (src < 0) | dead[src.clamp_min(0)]
+ cols.append(torch.where(blocked, torch.full_like(src, self.pad_id), compressed[src.clamp_min(0)]))
+ stack = torch.stack(cols, dim=-1) # [T, max_ngram]; column s is the token s places back
+ products = stack.unsqueeze(1) * self.multipliers # [T, L, max_ngram]
+ rolling, hashes = products[..., 0], []
+ for i in range(1, self.max_ngram):
+ rolling = torch.bitwise_xor(rolling, products[..., i])
+ hashes.append(rolling.unsqueeze(-1) % self.primes[:, i - 1]) # [T, L, heads]
+ return torch.cat(hashes, dim=-1) + self.offsets # [T, L, cols]
+
+
+# ----- lookup ------------------------------------------------------------------------------------
+
+
+class EngramTable(Protocol):
+ """Row store of one Engram layer: ``lookup`` returns the dequantized rows of the current
+ forward's tokens as ``[T, n_hash_cols * head_dim]`` bf16 (staged by the host before launch)."""
+
+ def lookup(self, num_tokens: int) -> torch.Tensor: ...
+
+
+class ZeroEngramTable:
+ """Stand-in when no table is attached (dummy weights / unit tests): every lookup reads zeros."""
+
+ def __init__(self, width: int, device: torch.device) -> None:
+ self.width = width
+ self.device = device
+
+ def lookup(self, num_tokens: int) -> torch.Tensor:
+ return torch.zeros(num_tokens, self.width, dtype=torch.bfloat16, device=self.device)
+
+
+# ----- the layer -----------------------------------------------------------------------------------
+
+
+class EngramLayer(BaseOP):
+ """``streams [T, hc, dim] += gate * value`` with key / value from the gathered n-gram rows."""
+
+ def __init__(self, args: DeepseekV41Args, layer_id: int, *, quant_config=None, prefix: str = ""):
+ self.layer_id = layer_id
+ self.dim = args.dim
+ self.hc_mult = args.hc_mult
+ self.eps = args.norm_eps
+ self.clamp_value = 1e-6
+ self.width = args.engram_hash_cols * args.engram_head_dim
+ self.wkv = LinearReplicated(self.width, args.dim * (args.hc_mult + 1), has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wkv")
+ self.q_weight = torch.empty(args.hc_mult, args.dim, dtype=torch.float32)
+ self.k_weight = torch.empty(args.hc_mult, args.dim, dtype=torch.float32)
+ self._table: EngramTable | None = None
+
+ def attach_table(self, table: EngramTable) -> None:
+ self._table = table
+
+ def forward(self, streams: torch.Tensor, rows: torch.Tensor | None = None, *, image_mask: torch.Tensor | None = None) -> torch.Tensor:
+ """``streams [T, hc, dim]``; ``rows`` overrides the table lookup (tests / reference parity)."""
+ T = streams.shape[0]
+ if rows is None:
+ assert self._table is not None, "Engram table was never attached"
+ rows = self._table.lookup(T)
+ kv = self.wkv.forward(rows.to(streams.dtype))
+ key, value = kv.split([self.hc_mult * self.dim, self.dim], dim=-1)
+ key = key.float().view(T, self.hc_mult, self.dim)
+ weight = self.q_weight * self.k_weight # only ever used as a product
+ h = streams.float()
+ # normalized per (token, stream) over dim, not jointly over the streams
+ rstd = torch.rsqrt(h.square().mean(-1) + self.eps) * torch.rsqrt(key.square().mean(-1) + self.eps)
+ dot = (h * weight * key).sum(-1) * rstd * self.dim**-0.5
+ gate = torch.sigmoid(torch.copysign(dot.abs().clamp_min(self.clamp_value).sqrt(), dot))
+ out = (h + gate.unsqueeze(-1) * value.float().unsqueeze(1)).to(streams.dtype)
+ return out if image_mask is None else torch.where(image_mask[:, None, None], streams, out)
+
+
+__all__ = [
+ "EngramHash",
+ "EngramLayer",
+ "EngramTable",
+ "ZeroEngramTable",
+ "build_compressed_token_map",
+ "bucket_primes",
+ "hash_multipliers",
+]
diff --git a/python/freetoken/models/deepseek_v41/engram_table.py b/python/freetoken/models/deepseek_v41/engram_table.py
new file mode 100644
index 00000000..55450bbd
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/engram_table.py
@@ -0,0 +1,296 @@
+"""Engram tables streamed directly from checkpoint shards.
+
+Each Engram layer owns a ``[384M, 256]`` fp8 table (98 GB) plus a ``[384M, 8]`` ue8m0 scale
+tensor (3 GB). The tables stay where the checkpoint put them: ``EngramDiskTable`` maps the shard
+tensor as one ``RowStore`` extent and reads the 24 rows a token needs per fill (io_uring, O_DIRECT,
+duplicates deduplicated), while the scales are host-resident and gathered next to the values, so
+the staged row is exactly the pool row format ``fp8_e8m0_b32`` and the device dequantizes it with
+the same kernel the KV pools use.
+
+Protocol (the Qwen3.8 PLE disk table's, ``models/qwen4_exp/ple_disk.py``): the engine wraps every
+dispatch in ``EngramHost.forward_host_ctx``. Eager: hash + stage + read before the launch. CUDA-graph
+decode: the sampled token lives on the device under overlap scheduling, so the host reads it back,
+fills the graph's pinned staging and releases a per-table flag the captured ``lookup`` WAITs on (or,
+without stream memops, fills before the replay). The fill runs on a worker thread that is started
+BEFORE the graph is launched: with a graph this large (40 layers) the driver's launch call itself
+blocks once the GPU sits on the WAIT with the rest of the graph queued behind it, so a fill scheduled
+after the launch returns would deadlock. The worker works from an immutable dispatch snapshot plus
+the readback event (which the GPU completes on its own), and the dispatch context waits for it after
+the launch so its failure surfaces before the step's output is consumed.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import struct
+from concurrent.futures import Future, ThreadPoolExecutor
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Sequence
+
+import torch
+from freetoken.core import Batch
+from freetoken.kernel.pinned import alloc_pinned_tensor
+from freetoken.kvcache.dsv4.v41_row_format import FP8_E8M0_B32
+from freetoken.utils import init_logger
+
+from .engram import EngramHash
+
+logger = init_logger(__name__)
+
+
+@dataclass(frozen=True)
+class _TensorExtent:
+ """Where a safetensors tensor's bytes sit in its shard: a contiguous ``[offset, offset + nbytes)``."""
+
+ path: str
+ offset: int
+ nbytes: int
+ shape: tuple[int, ...]
+ dtype: str # the safetensors dtype string (e.g. "F8_E4M3")
+
+
+def _tensor_extent(folder: str, name: str) -> _TensorExtent:
+ """Locate ``name`` in the checkpoint directory without reading it (the table is streamed in place)."""
+ from freetoken.models.loader import safetensors_weight_map
+
+ path = os.path.join(folder, safetensors_weight_map(folder)[name])
+ with open(path, "rb") as fh:
+ n = struct.unpack("<Q", fh.read(8))[0]
+ header = json.loads(fh.read(n))
+ meta = header[name]
+ start, end = meta["data_offsets"]
+ return _TensorExtent(path, 8 + n + start, end - start, tuple(meta["shape"]), meta["dtype"])
+
+
+@dataclass(frozen=True)
+class EngramRowSource:
+ """The fp8 table tensor in its shard (one extent) and its host-resident scales."""
+
+ path: str
+ base: int
+ num_rows: int
+ head_dim: int
+ scales: torch.Tensor # [num_rows, head_dim // 32] uint8 (e8m0 codes), CPU
+
+
+def engram_row_source(folder: str, layer_id: int) -> EngramRowSource:
+ """Locate a layer's table in whatever safetensors files the checkpoint directory holds: the HF
+ shards (through the index) or the shards an FTW conversion copied next to it (through their headers)."""
+ from safetensors import safe_open
+
+ weight = _tensor_extent(folder, f"layers.{layer_id}.engram.embed.weight")
+ if weight.dtype != "F8_E4M3":
+ raise ValueError(f"engram table of layer {layer_id} is {weight.dtype}, expected F8_E4M3")
+ rows, head_dim = weight.shape
+ scale = _tensor_extent(folder, f"layers.{layer_id}.engram.embed.scale")
+ with safe_open(scale.path, framework="pt", device="cpu") as f:
+ scales = f.get_tensor(f"layers.{layer_id}.engram.embed.scale").view(torch.uint8).contiguous()
+ if tuple(scales.shape) != (rows, head_dim // 32):
+ raise ValueError(f"engram scales of layer {layer_id} are {tuple(scales.shape)}, expected {(rows, head_dim // 32)}")
+ return EngramRowSource(weight.path, weight.offset, rows, head_dim, scales)
+
+
+class EngramDiskTable:
+ """One layer's table: stages ``[T, n_cols]`` rows into pinned memory on the host, dequantizes on
+ the device. Implements the model's ``EngramTable`` protocol (``lookup``)."""
+
+ def __init__(
+ self, source: EngramRowSource, n_cols: int, device: torch.device, *, max_graph_rows: int, max_extend_tokens: int,
+ use_io_uring: bool = True,
+ ) -> None:
+ from freetoken.kernel.row_store import RowStore
+
+ self.head_dim = source.head_dim
+ self.n_cols = n_cols
+ self.num_rows = source.num_rows
+ self.row_bytes = FP8_E8M0_B32.row_bytes(source.head_dim) # values | scales
+ self.scales = source.scales
+ self.device = device
+ self.store = RowStore(
+ paths=[source.path], extent_file=[0], extent_base=[source.base], rows_per_extent=source.num_rows,
+ row_bytes=source.head_dim, row_stride=source.head_dim, use_io_uring=use_io_uring,
+ )
+ # the graph staging is allocated up front (pinned alloc inside capture is illegal) and
+ # outlives any one capture so a re-capture keeps the same pointers; the eager staging
+ # grows to the largest prefill chunk seen (never inside a capture)
+ self.max_graph_rows = max_graph_rows
+ self._graph_pinned = alloc_pinned_tensor(max_graph_rows * n_cols * self.row_bytes, dtype=torch.uint8)
+ self._graph_pinned.zero_()
+ self._graph_dev = torch.empty(max_graph_rows * n_cols * self.row_bytes, dtype=torch.uint8, device=device)
+ # completes once the last graph launch that read ``_graph_pinned`` has run (recorded by the
+ # host coordinator after the launch: a record inside the capture would not be observable)
+ self.graph_consumed = torch.cuda.Event()
+ # the overlap scheduler stages batch k+1 while batch k's lookup copy may still be queued: the
+ # eager staging alternates two pinned buffers, and each is rewritten only once the H2D copy
+ # that last read it has run (``_eager_read[i]``, recorded right after that copy)
+ self._eager_pinned: list[torch.Tensor] = []
+ self._eager_read = [torch.cuda.Event(), torch.cuda.Event()]
+ self._eager_slot = 0
+ self._eager_dev = None
+ self._grow_eager(max_extend_tokens)
+ self.flag = alloc_pinned_tensor(1, dtype=torch.int64)
+ self.flag.zero_()
+ self.wait_sync = False # set by the host coordinator after its probe
+
+ def _grow_eager(self, num_tokens: int) -> None:
+ nbytes = num_tokens * self.n_cols * self.row_bytes
+ if self._eager_pinned and self._eager_pinned[0].numel() >= nbytes:
+ return
+ for event in self._eager_read:
+ event.synchronize()
+ self._eager_pinned = [alloc_pinned_tensor(nbytes, dtype=torch.uint8) for _ in self._eager_read]
+ for buf in self._eager_pinned:
+ buf.zero_() # a warmup prefill stages nothing and reads whatever sits here
+ self._eager_dev = torch.empty(nbytes, dtype=torch.uint8, device=self.device)
+
+ # ----- host -----
+ def stage(self, row_ids: torch.Tensor, *, graph: bool) -> None:
+ """Queue ``row_ids [T, n_cols]`` (int64, host): values through the store, scales from the resident tensor."""
+ n = row_ids.numel()
+ if graph:
+ self.graph_consumed.synchronize()
+ pinned = self._graph_pinned
+ assert n <= self.max_graph_rows * self.n_cols, f"graph staging holds {self.max_graph_rows} rows, batch has {n // self.n_cols}"
+ else:
+ self._grow_eager(n // self.n_cols)
+ self._eager_slot ^= 1
+ self._eager_read[self._eager_slot].synchronize()
+ pinned = self._eager_pinned[self._eager_slot]
+ ids = row_ids.reshape(-1).to(torch.int64).contiguous()
+ rows = pinned[: n * self.row_bytes].view(n, self.row_bytes)
+ self.store.stage_rows(ids.data_ptr(), n, rows.data_ptr(), self.row_bytes)
+ rows[:, self.head_dim :].copy_(self.scales.index_select(0, ids))
+
+ def flush(self, *, signal: bool) -> None:
+ self.store.flush(self.flag.data_ptr() if signal else 0)
+
+ # ----- device (EngramTable protocol) -----
+ def lookup(self, num_tokens: int) -> torch.Tensor:
+ from freetoken.kernel.row_store import wait_reset
+ from freetoken.kernel.triton.dsv41.pack import unpack_rows
+
+ stream = torch.cuda.current_stream(self.device)
+ capturing = torch.cuda.is_current_stream_capturing()
+ if capturing and self.wait_sync:
+ wait_reset(stream, self.flag)
+ pinned, dev = (self._graph_pinned, self._graph_dev) if capturing else (self._eager_pinned[self._eager_slot], self._eager_dev)
+ n = num_tokens * self.n_cols
+ nbytes = n * self.row_bytes
+ dev[:nbytes].copy_(pinned[:nbytes], non_blocking=True)
+ if not capturing:
+ self._eager_read[self._eager_slot].record(stream)
+ values = unpack_rows(dev[:nbytes].view(n, self.row_bytes), FP8_E8M0_B32, self.head_dim)
+ return values.view(num_tokens, self.n_cols * self.head_dim)
+
+
+def _context(ids: torch.Tensor, position: int, width: int) -> torch.Tensor:
+ """The up-to-``width`` raw ids before ``position`` (fewer at the sequence start)."""
+ return ids[max(0, position - width) : position]
+
+
+class EngramHost:
+ """The per-model coordinator: hashes each batch's token runs and fills every layer's table.
+
+ The token range a prefill hashes is the attention metadata's encoder segments -- the one
+ execution plan the scheduler, the attention layers and the model already agreed on. A graph
+ decode's fill runs on a worker thread from an immutable dispatch snapshot (position + the
+ ``max_ngram - 1`` preceding ids per row) plus the readback of the sampled token; the request
+ objects themselves advance on the main thread right after dispatch and are never read late.
+ """
+
+ def __init__(self, hash: EngramHash, tables: Sequence[EngramDiskTable], device: torch.device, *, sync_mode: str = "auto") -> None:
+ from freetoken.kernel.row_store import probe_wait_sync
+
+ self.hash = hash
+ self.tables = list(tables)
+ self.device = device
+ self.wait_sync = probe_wait_sync(sync_mode, device)
+ for t in self.tables:
+ t.wait_sync = self.wait_sync
+ self._token_readback = alloc_pinned_tensor(min(t.max_graph_rows for t in self.tables), dtype=torch.int32)
+ self._readback_event = torch.cuda.Event()
+ self._worker = ThreadPoolExecutor(max_workers=1, thread_name_prefix="engram-fill")
+ self._pending: Future | None = None
+ logger.info_rank0(f"Engram disk backend: {self.tables[0].store.io_backend()}, {'wait-sync' if self.wait_sync else 'launch-gating'}")
+
+ def close(self) -> None:
+ self._worker.shutdown(wait=True)
+
+ def __del__(self) -> None:
+ worker = getattr(self, "_worker", None)
+ if worker is not None:
+ worker.shutdown(wait=False)
+
+ def fill(self, runs: Sequence[tuple[torch.Tensor, torch.Tensor]], *, graph: bool) -> None:
+ """``runs``: per request ``(context ids, token ids)`` in batch order; stages every layer's rows."""
+ ids = torch.cat([self.hash.row_ids(tokens, ctx) for ctx, tokens in runs]) if runs else torch.empty(0, len(self.tables), self.hash.n_cols, dtype=torch.int64)
+ for l, table in enumerate(self.tables):
+ table.stage(ids[:, l], graph=graph)
+ table.flush(signal=graph and self.wait_sync)
+
+ @torch.inference_mode()
+ def _deferred_fill(self, bs: int, histories: list[tuple[int, ...]]) -> None:
+ """Worker side of a graph decode: wait for the sampled tokens, hash them against the dispatch
+ snapshot, stage, and release the tables' flags. Reads nothing that the main thread mutates.
+ Inference mode is thread-local; staging buffers were created under it by the server."""
+ try:
+ self._readback_event.synchronize()
+ tokens = self._token_readback[:bs].to(torch.int64)
+ runs = [(torch.tensor(hist, dtype=torch.int64), tokens[i : i + 1]) for i, hist in enumerate(histories)]
+ self.fill(runs, graph=True)
+ except BaseException:
+ from freetoken.kernel.row_store import signal
+
+ for t in self.tables: # unblock the stream before surfacing; the step's output is discarded
+ signal(t.flag)
+ raise
+
+ def _await_pending(self) -> None:
+ """Surface the deferred fill's outcome (it depends only on GPU work already enqueued, so this
+ cannot wait on the host)."""
+ if self._pending is not None:
+ pending, self._pending = self._pending, None
+ pending.result()
+
+ def host_fill_batch(self, batch: Batch, use_graph: bool) -> None:
+ """Stage this batch's rows: eagerly, or -- a graph decode under flag-sync -- on the worker
+ thread once the sampled tokens have been read back."""
+ self._await_pending()
+ width = self.hash.max_ngram - 1
+ if batch.is_decode:
+ reqs = list(batch.padded_reqs)
+ # the dispatch snapshot: each row's position context, taken before anything advances
+ histories = [tuple(_context(r.input_ids, r.device_len - 1, width).tolist()) for r in reqs]
+ if use_graph and self.wait_sync:
+ bs = batch.padded_size
+ self._token_readback[:bs].copy_(batch.input_ids, non_blocking=True)
+ self._readback_event.record(torch.cuda.current_stream(self.device))
+ self._pending = self._worker.submit(self._deferred_fill, bs, histories)
+ return
+ tokens = batch.input_ids.to("cpu").to(torch.int64)
+ runs = [(torch.tensor(hist, dtype=torch.int64), tokens[i : i + 1]) for i, hist in enumerate(histories)]
+ self.fill(runs, graph=use_graph)
+ return
+ # prefill: the encoder segments are the token range every consumer of this forward executes
+ segments = batch.attn_metadata.segments
+ assert segments is not None and len(segments) == len(batch.reqs)
+ runs = [(_context(r.input_ids, seg.start_pos, width), r.input_ids[seg.start_pos : seg.end]) for r, seg in zip(batch.reqs, segments)]
+ self.fill(runs, graph=False)
+
+ @contextmanager
+ def forward_host_ctx(self, batch: Batch, use_graph: bool):
+ self.host_fill_batch(batch, use_graph)
+ yield
+ # the launch has been enqueued: wait for the deferred fill here so a failure surfaces before
+ # this step's output is consumed (the fill only waits on the GPU readback, never on the host)
+ self._await_pending()
+ if use_graph:
+ # only after this step's fill: the next fill waits on this launch, not the fill on it
+ stream = torch.cuda.current_stream(self.device)
+ for t in self.tables:
+ t.graph_consumed.record(stream)
+
+
+__all__ = ["EngramDiskTable", "EngramHost", "EngramRowSource", "engram_row_source"]
diff --git a/python/freetoken/models/deepseek_v41/indexer.py b/python/freetoken/models/deepseek_v41/indexer.py
new file mode 100644
index 00000000..4ab3da2b
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/indexer.py
@@ -0,0 +1,127 @@
+"""DSV41 Lightning Indexer (reference ``Indexer``): scores the compressed positions and keeps the
+``index_topk`` best per query.
+
+A Full-mode layer owns the index keys: ``k = k_norm(wk(latent))`` from its compressor's unrotated
+latent, RoPE'd at the group position, fp4-quantized (ue8m0 per 32) into the kv source's index pool.
+Every Full / Reindex layer scores its own fp4-baked queries ``wq_b(qr)`` against those keys,
+``sum_h relu(q_h . k) * w_h`` with ``w = weights_proj(x) * softmax_scale * n_heads**-0.5``.
+
+The Hierarchical Sparse Indexer (sec. 2.3.2): the candidate-source layer (the decoder's first Full
+layer) additionally keeps the ``candidate_topk_blocks`` best ``candidate_block_size``-position blocks
+as a candidate pool; the decoder's Reindex layers score only that pool. Logits are produced in query
+blocks bounded by ``LOGITS_BUDGET_BYTES`` so a long-context prefill never materializes ``[T, 1M]``.
+"""
+
+from __future__ import annotations
+
+import torch
+from freetoken.core import get_global_ctx
+from freetoken.kernel.triton.dsv4.fp8_linear import fp4_act_quant_inplace
+from freetoken.layers import BaseOP, LinearReplicated, RMSNorm
+
+from .args import DeepseekV41Args, LayerRole, Mode
+from .rope import apply_rotary_emb, apply_rotary_emb_decode
+
+LOGITS_BUDGET_BYTES = 512 << 20
+
+
+class Indexer(BaseOP):
+ def __init__(self, args: DeepseekV41Args, role: LayerRole, *, quant_config=None, prefix: str = ""):
+ self.role = role
+ self.n_heads = args.index_n_heads
+ self.head_dim = args.index_head_dim
+ self.rope_dim = args.rope_head_dim
+ self.topk = args.index_topk
+ self.weights_scale = args.index_softmax_scale * args.index_n_heads**-0.5
+ self.candidate_topk_blocks = args.candidate_topk_blocks
+ self.candidate_block_size = args.candidate_block_size
+ self.wq_b = LinearReplicated(args.q_lora_rank, self.n_heads * self.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wq_b")
+ self.weights_proj = LinearReplicated(args.dim, self.n_heads, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.weights_proj")
+ if role.mode is Mode.FULL:
+ self.wk = LinearReplicated(args.head_dim, self.head_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.wk")
+ self.k_norm = RMSNorm(self.head_dim, args.norm_eps)
+
+ @property
+ def attn(self):
+ return get_global_ctx().attn_backend
+
+ # ----- keys (Full mode) ----------------------------------------------------------------
+ def index_keys(self, latent: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
+ """``[G, index_head_dim]`` keys from the unrotated latents, rotated at their group positions. The
+ fp4 quantization is the pool's packing."""
+ k = self.k_norm.forward(self.wk.forward(latent))
+ apply_rotary_emb(k[..., -self.rope_dim :], freqs)
+ return k
+
+ # ----- queries ---------------------------------------------------------------------------
+ def _queries(self, x: torch.Tensor, qr: torch.Tensor, rotate) -> tuple[torch.Tensor, torch.Tensor]:
+ q = self.wq_b.forward(qr).unflatten(-1, (self.n_heads, self.head_dim))
+ rotate(q[..., -self.rope_dim :])
+ fp4_act_quant_inplace(q, 32)
+ weights = self.weights_proj.forward(x) * self.weights_scale
+ return q, weights
+
+ # ----- selection -------------------------------------------------------------------------
+ def select_prefill(
+ self, x: torch.Tensor, qr: torch.Tensor, freqs: torch.Tensor, *, start_pos: int, ratio: int, locs: torch.Tensor,
+ candidates: torch.Tensor | None,
+ ) -> tuple[torch.Tensor, torch.Tensor | None]:
+ """Per-query picks for one request's ``n`` new tokens at ``[start_pos, start_pos + n)``.
+
+ ``locs [1, W]`` are the request's live full locs for the positions its queries may see (the
+ source's row of compressed position ``t`` is ``locs[t * ratio] // ratio``); ``candidates [1, n,
+ NC]`` restricts a Reindex layer to the pool. Returns ``(rows [1, n, topk] int32 global main rows,
+ -1 tail; candidates [1, n, NC] | None -- the pool this layer built as candidate source)``.
+ """
+ n = x.shape[0]
+ T = locs.shape[1] // ratio
+ if T == 0: # a prompt shorter than one compression block: no source row exists yet
+ rows = torch.full((1, n, self.topk), -1, dtype=torch.int32, device=x.device)
+ pool = None
+ if self.role.is_candidate_source:
+ pool = torch.full((1, n, self.candidate_topk_blocks * self.candidate_block_size), -1, dtype=torch.int32, device=x.device)
+ return rows, pool
+ q, w = self._queries(x, qr, lambda t: apply_rotary_emb(t, freqs))
+ live = ((start_pos + torch.arange(1, n + 1, device=x.device)) // ratio).to(torch.int32).view(1, n)
+ width = candidates.shape[-1] if candidates is not None else T
+ qb = max(1, LOGITS_BUDGET_BYTES // max(1, width * 4))
+ picks, pools = [], []
+ for s0 in range(0, n, qb):
+ s1 = min(n, s0 + qb)
+ cand = candidates[:, s0:s1] if candidates is not None else None
+ logits = self.attn.indexer_logits(q[None, s0:s1], w[None, s0:s1], self.role.kv_source, locs, ratio, live[:, s0:s1], T=T, candidates=cand)
+ if cand is not None:
+ pos = self.attn.select_topk_in_candidates(logits, cand, self.topk)
+ else:
+ pos = self.attn.select_topk(logits, live[:, s0:s1], self.topk)
+ if self.role.is_candidate_source:
+ pools.append(self.attn.select_candidate_blocks(logits, live[:, s0:s1], self.candidate_topk_blocks, self.candidate_block_size))
+ picks.append(self.attn.positions_to_rows(pos, locs, ratio))
+ rows = torch.cat(picks, dim=1)
+ pool = torch.cat(pools, dim=1) if pools else None
+ return rows.to(torch.int32), pool
+
+ def select_decode(
+ self, x: torch.Tensor, qr: torch.Tensor, freqs: torch.Tensor, *, pos: torch.Tensor, ratio: int, locs: torch.Tensor,
+ T: int, candidates: torch.Tensor | None,
+ ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]:
+ """Batched one-token selection. ``locs [B, W]`` is the decode snapshot (graph-static width),
+ ``T`` the staged compressed width, ``pos [B]``. Returns ``(rows [B, 1, topk] int32, counts [B, 1]
+ int32 valid picks, candidates [B, 1, NC] | None)``."""
+ B = x.shape[0]
+ q, w = self._queries(x, qr, lambda t: apply_rotary_emb_decode(t.unsqueeze(1), freqs))
+ live = ((pos + 1) // ratio).to(torch.int32).view(B, 1)
+ logits = self.attn.indexer_logits(q.view(B, 1, self.n_heads, self.head_dim), w.view(B, 1, -1), self.role.kv_source, locs, ratio, live, T=T, candidates=candidates)
+ pool = None
+ if candidates is not None:
+ picks = self.attn.select_topk_in_candidates(logits, candidates, self.topk)
+ else:
+ picks = self.attn.select_topk(logits, live, self.topk)
+ if self.role.is_candidate_source:
+ pool = self.attn.select_candidate_blocks(logits, live, self.candidate_topk_blocks, self.candidate_block_size)
+ rows = self.attn.positions_to_rows(picks, locs, ratio)
+ counts = (picks >= 0).sum(dim=-1, dtype=torch.int32) # valid picks form a prefix
+ return rows.to(torch.int32), counts, pool
+
+
+__all__ = ["Indexer", "LOGITS_BUDGET_BYTES"]
diff --git a/python/freetoken/models/deepseek_v41/model.py b/python/freetoken/models/deepseek_v41/model.py
new file mode 100644
index 00000000..8112022f
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/model.py
@@ -0,0 +1,289 @@
+"""DeepSeek-V4.1-Flash model (engine-native port of ``inference/model.py``).
+
+ embed -> hc_mult residual streams -> [Engram at 1, 14] -> 40 blocks (single-pass mHC around
+ DSV41 attention and MoE) -> collapse with the last block's pre gates -> norm -> head
+
+Single-pass mHC (``layers/mhc.py``): each sublayer boundary applies the PREVIOUS sublayer's post /
+comb to the streams, predicts this sublayer's (pre, post, comb) from the updated streams, and mixes
+the sublayer input with the pre gates the previous sublayer predicted. The chain state
+``(y, residual, post, comb, pre)`` threads through the layer loop; layer 0's attention starts it
+with a one-hot pre on stream 0 and no pending post.
+
+Causal Encoder-Decoder: the decoder's kv source (layer 20) compresses ITS INPUT -- the final encoder
+hidden state -- into the global KV every decoder layer reads. Under Decoder SWA Bounded Replay the
+decoder layers run on each request's last ``window_size`` prompt tokens only (their sliding window
+truncated at the start of that window, see the backend; a prefix hit stops before it, so those tokens
+are always prefilled); layer 20 still publishes global KV for every prompt token.
+``--swa-decoder-replay exact`` runs the decoder on every token (the reference numerics).
+
+KV addressing is the attention backend's; pool buffers are read off the live pool per access, so a
+runtime cache rebuild needs no unbind. Decode is batched and CUDA-graph safe (the DSV4 precedent).
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List, NamedTuple
+
+import torch
+from freetoken.attention.dsv41_sparse import DSV41AttnMetadata, PrefillSegment
+from freetoken.core import get_global_ctx
+from freetoken.layers import BaseOP, OPList, ParallelLMHead, RMSNorm, VocabParallelEmbedding
+from freetoken.layers.mhc import mhc_fused_post_pre_single_pass, mhc_mix_input, mhc_post
+from freetoken.models.blocks import BaseLLMModel, embed_input_ids
+
+from .args import DeepseekV41Args
+from .attention import DSV41Attention, DecodeStepContext
+from .engram import EngramLayer
+from .moe import MoE
+
+if TYPE_CHECKING:
+ from freetoken.models.config import ModelConfig
+
+POST_MULT = 2.0 # post gates are 2 * sigmoid (reference hc_split_sinkhorn)
+
+
+class Streams(NamedTuple):
+ """The single-pass mHC chain state between sublayers: the last sublayer output ``y`` with its
+ not-yet-applied ``post`` / ``comb``, the residual streams ``[T, hc, dim]`` they apply to, and
+ the ``pre`` gates ``[T, hc]`` the next sublayer mixes its input with."""
+
+ y: torch.Tensor | None
+ residual: torch.Tensor
+ post: torch.Tensor | None
+ comb: torch.Tensor | None
+ pre: torch.Tensor
+
+ def materialize(self) -> torch.Tensor:
+ """The streams with the pending post applied."""
+ if self.post is None:
+ return self.residual
+ return mhc_post(self.y, self.residual, self.post, self.comb)
+
+
+class Block(BaseOP):
+ def __init__(self, config: ModelConfig, layer_id: int, *, prefix: str = ""):
+ args: DeepseekV41Args = config.dsv41_args
+ self.layer_id = layer_id
+ self.role = args.roles[layer_id]
+ self.norm_eps = args.norm_eps
+ self.hc_eps = args.hc_eps
+ self.sinkhorn = args.hc_sinkhorn_iters
+ hc, dim = args.hc_mult, args.dim
+ mix = (2 + hc) * hc
+ self.attn = DSV41Attention(args, self.role, quant_config=config.quant, prefix=f"{prefix}.attn")
+ self.ffn = MoE(config, layer_id, prefix=f"{prefix}.ffn")
+ self.attn_norm = RMSNorm(dim, args.norm_eps)
+ self.ffn_norm = RMSNorm(dim, args.norm_eps)
+ self.engram = EngramLayer(args, layer_id, quant_config=config.quant, prefix=f"{prefix}.engram") if layer_id in args.engram_layer_ids else None
+ self.hc_attn_fn = torch.empty(mix, hc * dim, dtype=torch.float32)
+ self.hc_ffn_fn = torch.empty(mix, hc * dim, dtype=torch.float32)
+ self.hc_attn_base = torch.empty(mix, dtype=torch.float32)
+ self.hc_ffn_base = torch.empty(mix, dtype=torch.float32)
+ self.hc_attn_scale = torch.empty(3, dtype=torch.float32)
+ self.hc_ffn_scale = torch.empty(3, dtype=torch.float32)
+
+ def _boundary(self, s: Streams, fn, scale, base):
+ residual, post, comb, pre_next, x = mhc_fused_post_pre_single_pass(
+ s.y if s.y is not None else s.residual[:, 0], s.residual, s.post, s.comb, s.pre, fn, scale, base,
+ self.norm_eps, self.hc_eps, POST_MULT, self.sinkhorn,
+ )
+ return residual, post, comb, pre_next, x
+
+ def attn_input(self, s: Streams, *, image_mask: torch.Tensor | None = None):
+ """Apply the pending post, predict the attention gates, mix and norm the attention input.
+ Returns ``(residual, post, comb, pre_next, x)``."""
+ if self.engram is not None:
+ s = Streams(None, self.engram.forward(s.materialize(), image_mask=image_mask), None, None, s.pre)
+ residual, post, comb, pre_next, x = self._boundary(s, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
+ return residual, post, comb, pre_next, self.attn_norm.forward(x)
+
+ def ffn_step(self, y: torch.Tensor, residual, post, comb, pre, *, image_mask: torch.Tensor | None = None) -> Streams:
+ """The FFN sublayer after the attention output ``y``: boundary, norm, MoE; returns the chain state."""
+ residual, post, comb, pre_next, x = self._boundary(Streams(y, residual, post, comb, pre), self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
+ return Streams(self.ffn.forward(self.ffn_norm.forward(x), image_mask=image_mask), residual, post, comb, pre_next)
+
+ def forward_prefill(self, s: Streams, segments: List[PrefillSegment], positions: torch.Tensor, image_mask: torch.Tensor | None = None) -> Streams:
+ residual, post, comb, pre_next, x = self.attn_input(s, image_mask=image_mask)
+ if self.attn.compressor is not None:
+ self.attn.publish_prefill(x, segments)
+ y = self.attn.forward_prefill(x, segments, positions)
+ return self.ffn_step(y, residual, post, comb, pre_next, image_mask=image_mask)
+
+ def forward_prefill_narrowed(
+ self, s: Streams, segments: List[PrefillSegment], rows: torch.Tensor, query_segments: List[PrefillSegment],
+ query_positions: torch.Tensor, image_mask: torch.Tensor | None = None, query_image_mask: torch.Tensor | None = None,
+ ) -> Streams | None:
+ """``forward_prefill`` with a row cut after the global KV is published: the attention input and
+ ``publish_prefill`` cover every token of ``segments``, the attention and the FFN only ``rows``
+ (tiled by ``query_segments``) -- the CED decoder's kv source under bounded replay. None when no
+ query segment is left (every chunk in the batch ends before its prompt's last window)."""
+ residual, post, comb, pre_next, x = self.attn_input(s, image_mask=image_mask)
+ if self.attn.compressor is not None:
+ self.attn.publish_prefill(x, segments)
+ if not query_segments:
+ return None
+ y = self.attn.forward_prefill(x[rows], query_segments, query_positions)
+ return self.ffn_step(y, residual[rows], post[rows], comb[rows], pre_next[rows], image_mask=query_image_mask)
+
+ def forward_decode(self, s: Streams, pos: torch.Tensor, rows: torch.Tensor, dctx, cmp_stage_cap: int) -> Streams:
+ residual, post, comb, pre_next, x = self.attn_input(s)
+ y = self.attn.forward_decode(x, pos, rows, dctx, cmp_stage_cap)
+ return self.ffn_step(y, residual, post, comb, pre_next)
+
+
+class Transformer(BaseOP):
+ def __init__(self, config: ModelConfig, *, prefix: str = "model"):
+ args: DeepseekV41Args = config.dsv41_args
+ self.args = args
+ self.embed = VocabParallelEmbedding(num_embeddings=args.vocab_size, embedding_dim=args.dim)
+ self.layers = OPList([Block(config, i, prefix=f"{prefix}.layers.{i}") for i in range(args.n_layers)])
+ self.norm = RMSNorm(args.dim, args.norm_eps)
+
+ def bind(self, device: torch.device) -> None:
+ tables: dict = {} # one RoPE table per distinct parameter set, shared by the layers using it
+ for block in self.layers.op_list:
+ block.attn.bind(device, tables)
+
+ def _entry(self, input_ids: torch.Tensor) -> Streams:
+ h = embed_input_ids(self.embed, input_ids, get_global_ctx().batch)
+ residual = h.unsqueeze(1).expand(-1, self.args.hc_mult, -1).contiguous()
+ pre = torch.zeros(h.shape[0], self.args.hc_mult, dtype=torch.float32, device=h.device)
+ pre[:, 0] = 1.0
+ return Streams(None, residual, None, None, pre)
+
+ def _exit(self, s: Streams) -> torch.Tensor:
+ return self.norm.forward(mhc_mix_input(s.materialize(), s.pre))
+
+ def prefill(self, input_ids: torch.Tensor, positions: torch.Tensor, md: DSV41AttnMetadata) -> torch.Tensor:
+ """Ragged prefill over the flat new-token stream; returns the hidden states the head reads (the
+ decoder stream under bounded replay, where ``md.last_indices`` also point, plus the placeholder
+ row ``md.decoder_pad`` asks for)."""
+ s = self._entry(input_ids)
+ dtype = s.residual.dtype
+ image_mask = input_ids >= self.args.vocab_size if get_global_ctx().batch.mm_embeds is not None else None
+ segments, dec_segments, rows = md.segments, md.decoder_segments, md.decoder_rows
+ decoder_start = self.args.decoder_start_layer
+ if rows is not None:
+ dec_positions = positions[rows]
+ dec_image_mask = None if image_mask is None else image_mask[rows]
+ for block in self.layers.op_list:
+ if rows is None or block.layer_id < decoder_start:
+ s = block.forward_prefill(s, segments, positions, image_mask)
+ elif block.layer_id == decoder_start:
+ s = block.forward_prefill_narrowed(s, segments, rows, dec_segments, dec_positions, image_mask, dec_image_mask)
+ if s is None:
+ break
+ else:
+ s = block.forward_prefill(s, dec_segments, dec_positions, dec_image_mask)
+ pad = [torch.zeros(1, self.args.dim, dtype=dtype, device=input_ids.device)] if md.decoder_pad else []
+ return torch.cat([self._exit(s)] + pad) if s is not None else pad[0]
+
+ def decode(self, input_ids: torch.Tensor, pos: torch.Tensor, md: DSV41AttnMetadata, cmp_stage_cap: int) -> torch.Tensor:
+ B = input_ids.shape[0]
+ rows = torch.arange(B, device=input_ids.device)
+ s = self._entry(input_ids)
+ # the layer-invariant step context, resolved once (a capture records it once): ring slots and
+ # window candidates of the shared pool -- and of the private rings when bounded replay keeps
+ # the decoder's window KV per request -- plus each RoPE table at the step's positions
+ dctx = DecodeStepContext.build([b.attn for b in self.layers.op_list], md, pos, rows, self.args.window_size, self.args.index_topk)
+ for block in self.layers.op_list:
+ s = block.forward_decode(s, pos, rows, dctx, cmp_stage_cap)
+ return self._exit(s)
+
+
+class DeepseekV41ForCausalLM(BaseLLMModel):
+ def __init__(self, config: ModelConfig):
+ self._config = config
+ self._args: DeepseekV41Args = config.dsv41_args
+ self.model = Transformer(config)
+ self.head = ParallelLMHead(num_embeddings=self._args.vocab_size, embedding_dim=self._args.dim, quant_config=config.quant, prefix="head")
+ self._bound = False
+ if config.is_multimodal:
+ from .vision import DeepseekV41Vision
+
+ self.visual = DeepseekV41Vision(config.vision_config, config.hidden_size,
+ num_layers=self._args.n_layers, num_experts=self._args.n_routed_experts)
+
+ def encode(self, item):
+ return self.visual.forward(item)
+
+ def place_encoder_weights(self, mode: str) -> None:
+ self.visual.place_weights(mode)
+
+ def _ensure_bound(self) -> None:
+ if not self._bound:
+ self.model.bind(get_global_ctx().kv_cache.device)
+ if self._config.is_multimodal:
+ for i, block in enumerate(self.model.layers.op_list):
+ block.ffn.gate._bias_vl = self.visual.routing_bias[i]
+ self._bound = True
+
+ def mark_for_rebind(self) -> None:
+ """A runtime pool rebuild changes nothing the model caches (rope tables are pool-independent),
+ but the engine asks; re-bind is cheap and keeps the contract with the DSV4 sibling."""
+ self._bound = False
+
+ def engram_layers(self) -> List[EngramLayer]:
+ return [b.engram for b in self.model.layers.op_list if b.engram is not None]
+
+ def load_host_tables(self, engine_config) -> int:
+ """Attach the Engram tables (streamed from the checkpoint shards; zeros for dummy weights)
+ and take over ``forward_host_ctx`` so every dispatch stages its rows. Returns the pinned
+ host bytes the engine reserves from its pin budget."""
+ import os
+
+ layers = self.engram_layers()
+ if not layers:
+ return 0
+ device = torch.device("cuda", torch.cuda.current_device())
+ if getattr(engine_config, "use_dummy_weight", False):
+ from .engram import ZeroEngramTable
+
+ for layer in layers:
+ layer.attach_table(ZeroEngramTable(layer.width, device))
+ return 0
+ from freetoken.utils import download_hf_weight
+ from transformers import AutoTokenizer
+
+ from .engram import EngramHash, build_compressed_token_map
+ from .engram_table import EngramDiskTable, EngramHost, engram_row_source
+
+ folder = download_hf_weight(engine_config.model_path)
+ token_map, compressed_vocab = build_compressed_token_map(AutoTokenizer.from_pretrained(folder, trust_remote_code=True))
+ hash = EngramHash(self._args, token_map, compressed_vocab)
+ graph_rows = max(engine_config.max_running_req, engine_config.cuda_graph_max_bs or 0, 1)
+ tables = [
+ EngramDiskTable(
+ engram_row_source(folder, layer.layer_id), hash.n_cols, device,
+ max_graph_rows=graph_rows,
+ max_extend_tokens=engine_config.max_extend_tokens,
+ use_io_uring=os.getenv("FREETOKEN_ENGRAM_IO_URING", "1") != "0",
+ )
+ for layer in layers
+ ]
+ for layer, table in zip(layers, tables):
+ layer.attach_table(table)
+ self._engram_host = EngramHost(hash, tables, device, sync_mode=os.getenv("FREETOKEN_ENGRAM_SYNC", "auto"))
+ self.forward_host_ctx = self._engram_host.forward_host_ctx
+ return sum(t._graph_pinned.numel() + sum(b.numel() for b in t._eager_pinned) for t in tables)
+
+ def forward(self) -> torch.Tensor:
+ self._ensure_bound()
+ batch = get_global_ctx().batch
+ md = batch.attn_metadata
+ assert isinstance(md, DSV41AttnMetadata)
+ if batch.is_prefill:
+ hidden = self.model.prefill(batch.input_ids, batch.positions.long(), md)
+ else:
+ input_ids = batch.input_ids
+ B = batch.padded_size
+ pos = batch.positions.long().view(-1)[:B]
+ if torch.cuda.is_current_stream_capturing():
+ cmp_stage_cap = md.stage_width - 1
+ else:
+ cmp_stage_cap = int(pos.max().item())
+ hidden = self.model.decode(input_ids.view(B), pos, md, cmp_stage_cap)
+ return self.head.forward(hidden)
+
+
+__all__ = ["DeepseekV41ForCausalLM", "Transformer", "Block", "Streams"]
diff --git a/python/freetoken/models/deepseek_v41/moe.py b/python/freetoken/models/deepseek_v41/moe.py
new file mode 100644
index 00000000..91c62ecb
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/moe.py
@@ -0,0 +1,90 @@
+"""DeepSeek-V4.1 MoE block: sqrtsoftplus router with a selection-only correction bias, one shared
+clamped-SwiGLU expert, and the routed MXFP4 experts on the shared offload cache.
+
+Router (reference ``Gate``): ``scores = sqrt(softplus(x @ W^T))`` in fp32; the bias picks the experts,
+the unbiased scores weight them (renormalized, ``+1e-20`` as in training), times ``route_scale``.
+Image-span tokens select experts using the checkpoint's separate ``bias_vl``.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Tuple
+
+import torch
+from freetoken.kernel.triton.dsv4.bf16_linear import bf16_linear_fp32
+from freetoken.kernel.triton.dsv4.router import router_select
+from freetoken.kernel.triton.dsv4.swiglu import fused_swiglu
+from freetoken.layers import BaseOP, LinearReplicated, make_moe_layer
+
+from .args import DeepseekV41Args
+
+if TYPE_CHECKING:
+ from freetoken.models.config import ModelConfig
+
+TopK = Tuple[torch.Tensor, torch.Tensor]
+
+
+class Router(BaseOP):
+ def __init__(self, args: DeepseekV41Args):
+ self.topk = args.n_activated_experts
+ self.score_func = args.score_func
+ self.norm_topk_prob = args.norm_topk_prob
+ self.route_scale = args.route_scale
+ self.weight = torch.empty(args.n_routed_experts, args.dim, dtype=torch.bfloat16)
+ self.bias = torch.empty(args.n_routed_experts, dtype=torch.float32)
+ self._bias_vl = None # bound to the visual stack's per-layer routing bias after loading
+
+ def forward(self, x: torch.Tensor, *, image_mask: torch.Tensor | None = None) -> TopK:
+ if image_mask is not None:
+ assert self._bias_vl is not None, "image routing bias was not loaded"
+ scores = bf16_linear_fp32(x, self.weight)
+ return router_select(
+ scores.view(-1, scores.shape[-1]), self.bias, self.topk,
+ score_func=self.score_func, normalize=self.norm_topk_prob, route_scale=self.route_scale,
+ alternate_bias=self._bias_vl, alternate_mask=image_mask,
+ )
+
+
+class SharedExpert(BaseOP):
+ """The dense clamped-SwiGLU expert every token passes through (fp8 block-32 linears)."""
+
+ def __init__(self, args: DeepseekV41Args, *, quant_config=None, prefix: str = ""):
+ self.w1 = LinearReplicated(args.dim, args.moe_inter_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.w1")
+ self.w2 = LinearReplicated(args.moe_inter_dim, args.dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.w2")
+ self.w3 = LinearReplicated(args.dim, args.moe_inter_dim, has_bias=False, quant_config=quant_config, prefix=f"{prefix}.w3")
+ self.limit = args.swiglu_limit
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ h = fused_swiglu(self.w1.forward(x), self.w3.forward(x), self.limit, x.dtype)
+ return self.w2.forward(h)
+
+
+class MoE(BaseOP):
+ def __init__(self, config: ModelConfig, layer_id: int, *, prefix: str = ""):
+ args: DeepseekV41Args = config.dsv41_args
+ self.dim = args.dim
+ self.gate = Router(args)
+ self.shared_experts = SharedExpert(args, quant_config=config.quant, prefix=f"{prefix}.shared_experts")
+ # every layer is MoE: the offload cache indexes experts by layer id directly
+ self.experts = make_moe_layer(
+ config,
+ layer_id=layer_id,
+ activation="swiglu_clamp",
+ renormalize=args.norm_topk_prob,
+ limit=args.swiglu_limit,
+ quant_config=config.quant,
+ prefix=f"{prefix}.experts",
+ )
+
+ def forward(self, x: torch.Tensor, *, image_mask: torch.Tensor | None = None) -> torch.Tensor:
+ shape = x.shape
+ x = x.view(-1, self.dim)
+ weights, indices = self.gate.forward(x, image_mask=image_mask)
+ # the shared expert runs before the routed experts (same stream, so it does not overlap the
+ # hybrid decode's CPU work, whose submit follows it -- reordering that is follow-up work)
+ shared = self.shared_experts.forward(x)
+ routed = self.experts.routed_forward(x, weights, indices)
+ return (routed + shared).view(shape)
+
+
+__all__ = ["MoE", "Router", "SharedExpert"]
diff --git a/python/freetoken/models/deepseek_v41/rope.py b/python/freetoken/models/deepseek_v41/rope.py
new file mode 100644
index 00000000..6d7df9f8
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/rope.py
@@ -0,0 +1,58 @@
+"""Rotary embeddings for DeepSeek-V4.1 (verbatim reference math, ``inference/model.py``).
+
+Interleaved complex rotation on the last ``rope_head_dim`` channels, with YaRN when
+``original_seq_len > 0``. Two regimes share two cached tables: window-only layers at the base
+theta without YaRN, compressing layers at ``compress_rope_theta`` under YaRN. A compressed latent
+stands for the first token of its group, so group ``j`` rotates at position ``j * ratio``; the
+attention output is rotated back (``inverse=True``) because K == V share one rotated latent.
+"""
+
+from __future__ import annotations
+
+import math
+from functools import lru_cache
+
+import torch
+
+
+def precompute_freqs_cis(dim: int, seqlen: int, original_seq_len: int, base: float, factor: float, beta_fast: int, beta_slow: int) -> torch.Tensor:
+ freqs = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
+ if original_seq_len > 0:
+
+ def corrected_dim(rotations):
+ return dim * math.log(original_seq_len / (rotations * 2 * math.pi)) / (2 * math.log(base))
+
+ low = max(math.floor(corrected_dim(beta_fast)), 0)
+ high = min(math.ceil(corrected_dim(beta_slow)), dim - 1)
+ ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / max(high - low, 1e-3)).clamp(0, 1)
+ smooth = 1 - ramp
+ freqs = freqs / factor * (1 - smooth) + freqs * smooth
+ freqs = torch.outer(torch.arange(seqlen), freqs)
+ return torch.polar(torch.ones_like(freqs), freqs)
+
+
+@lru_cache(2) # the model's two rope regimes; a 1M table is ~256 MB, so never one per layer
+def get_freqs_cis(dim: int, seqlen: int, original_seq_len: int, base: float, factor: float, beta_fast: int, beta_slow: int, device: torch.device) -> torch.Tensor:
+ return precompute_freqs_cis(dim, seqlen, original_seq_len, base, factor, beta_fast, beta_slow).to(device)
+
+
+def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor, inverse: bool = False) -> torch.Tensor:
+ """In-place rotation of ``x[..., :]`` (``[T, rd]`` or ``[T, H, rd]``) by per-token ``freqs_cis [T, rd//2]``."""
+ xc = torch.view_as_complex(x.float().unflatten(-1, (-1, 2)))
+ if inverse:
+ freqs_cis = freqs_cis.conj()
+ if xc.ndim == 3:
+ freqs_cis = freqs_cis.unsqueeze(1)
+ x.copy_(torch.view_as_real(xc * freqs_cis).flatten(-2))
+ return x
+
+
+def apply_rotary_emb_decode(x: torch.Tensor, freqs_cis: torch.Tensor, inverse: bool = False) -> torch.Tensor:
+ """Batched decode rotation with PER-ROW freqs: ``x [B, 1, ..., rd]``, ``freqs_cis [B, rd//2]``
+ (the fused dsv4 triton kernel; bit-identical to ``apply_rotary_emb``)."""
+ from freetoken.kernel.triton.dsv4.rope import rope_decode_inplace
+
+ return rope_decode_inplace(x, freqs_cis, inverse)
+
+
+__all__ = ["precompute_freqs_cis", "get_freqs_cis", "apply_rotary_emb", "apply_rotary_emb_decode"]
diff --git a/python/freetoken/models/deepseek_v41/vision.py b/python/freetoken/models/deepseek_v41/vision.py
new file mode 100644
index 00000000..f5fea969
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/vision.py
@@ -0,0 +1,139 @@
+"""DeepSeek ViT and 3x3 pixel-unshuffle aligner, with the shared encoder weight streamer."""
+
+from __future__ import annotations
+
+import torch
+import torch.nn.functional as F
+from freetoken.layers import BaseOP, LinearReplicated, OPList
+from freetoken.models.weight_stream import BlockWeightStreamer
+
+
+class VisionNorm(BaseOP):
+ def __init__(self, dim: int):
+ self.weight = torch.empty(dim, dtype=torch.float32)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ y = x.float()
+ return (y * torch.rsqrt(y.square().mean(-1, keepdim=True) + 1e-6) * self.weight).to(x.dtype)
+
+
+class VisionAttention(BaseOP):
+ def __init__(self, vc):
+ self.n_heads = vc.num_attention_heads
+ self.head_dim = vc.hidden_size // self.n_heads
+ self.wqkv = LinearReplicated(vc.hidden_size, 3 * vc.hidden_size, has_bias=True)
+ self.wo = LinearReplicated(vc.hidden_size, vc.hidden_size, has_bias=True)
+
+ def forward(self, x, cos, sin):
+ n = x.shape[0]
+ q, k, v = (t.reshape(n, self.n_heads, self.head_dim) for t in self.wqkv.forward(x).chunk(3, -1))
+ def rotary(t):
+ a, b = t.float().chunk(2, -1)
+ return torch.cat((a * cos - b * sin, b * cos + a * sin), -1).to(t.dtype)
+ q, k = rotary(q), rotary(k)
+ q, k, v = (t.transpose(0, 1).unsqueeze(0) for t in (q, k, v))
+ y = F.scaled_dot_product_attention(q, k, v)
+ return self.wo.forward(y[0].transpose(0, 1).reshape(n, -1))
+
+
+class VisionMLP(BaseOP):
+ def __init__(self, vc):
+ self.w1 = LinearReplicated(vc.hidden_size, 2 * vc.intermediate_size, has_bias=False)
+ self.w2 = LinearReplicated(vc.intermediate_size, vc.hidden_size, has_bias=False)
+
+ def forward(self, x):
+ gate, up = self.w1.forward(x).chunk(2, -1)
+ return self.w2.forward(F.silu(gate) * up)
+
+
+class VisionBlock(BaseOP):
+ def __init__(self, vc):
+ self.norm1 = VisionNorm(vc.hidden_size)
+ self.attn = VisionAttention(vc)
+ self.norm2 = VisionNorm(vc.hidden_size)
+ self.mlp = VisionMLP(vc)
+
+ def forward(self, x, cos, sin):
+ x = x + self.attn.forward(self.norm1.forward(x), cos, sin)
+ return x + self.mlp.forward(self.norm2.forward(x))
+
+
+class PatchEmbed(BaseOP):
+ def __init__(self, vc):
+ self.proj = LinearReplicated(3 * vc.patch_size**2, vc.hidden_size, has_bias=True)
+
+ def forward(self, x):
+ return self.proj.forward(x.flatten(1))
+
+
+class VisionTransformer(BaseOP):
+ def __init__(self, vc):
+ self.patch_embed = PatchEmbed(vc)
+ self.blocks = OPList([VisionBlock(vc) for _ in range(vc.num_hidden_layers)])
+ self.norm = VisionNorm(vc.hidden_size)
+ self._vc = vc
+ self._streamer = None
+
+ def place_weights(self, mode: str) -> None:
+ if mode == "host" and self._streamer is None:
+ self._streamer = BlockWeightStreamer(self.blocks.op_list, self.patch_embed.proj.weight.device)
+ elif mode == "gpu" and self._streamer is not None:
+ self._streamer.unstream()
+ self._streamer = None
+ elif mode not in ("gpu", "host"):
+ raise ValueError(f"unknown vision weight placement {mode!r}")
+
+ def forward(self, patches, h: int, w: int):
+ x = self.patch_embed.forward(patches)
+ dim = self._vc.hidden_size // self._vc.num_attention_heads // 2
+ inv = 1.0 / (self._vc.rope_theta ** (torch.arange(0, dim, 2, dtype=torch.float32, device=x.device) / dim))
+ rows = torch.arange(h, device=x.device)[:, None].expand(h, w)
+ cols = torch.arange(w, device=x.device)[None, :].expand(h, w)
+ angles = (torch.stack((rows, cols), -1).reshape(-1, 2, 1).float() * inv).flatten(1)
+ cos, sin = angles.cos().unsqueeze(1), angles.sin().unsqueeze(1)
+ blocks = enumerate(self.blocks.op_list) if self._streamer is None else self._streamer.blocks(self.blocks.op_list)
+ for _, block in blocks:
+ x = block.forward(x, cos, sin)
+ return self.norm.forward(x)
+
+
+class VisionAligner(BaseOP):
+ def __init__(self, vc, dim: int):
+ self.ratio = vc.downsample_ratio
+ self.w1 = LinearReplicated(vc.hidden_size * self.ratio**2, dim, has_bias=True)
+ self.w2 = LinearReplicated(dim, dim, has_bias=True)
+
+ def forward(self, x, h: int, w: int):
+ r = self.ratio
+ x = x.view(h, w, -1).permute(2, 0, 1)
+ x = F.pad(x, (0, -w % r, 0, -h % r))
+ x = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).T
+ return self.w2.forward(F.gelu(self.w1.forward(x)))
+
+
+class DeepseekV41Vision(BaseOP):
+ def __init__(self, vc, dim: int, *, num_layers: int, num_experts: int):
+ self.vision = VisionTransformer(vc)
+ self.aligner = VisionAligner(vc, dim)
+ self.image_start = torch.empty(dim)
+ self.image_end = torch.empty(dim)
+ self.image_newline = torch.empty(dim)
+ # Keep all image-only parameters under visual, including FTW/text-only filtering.
+ self.routing_bias = torch.empty(num_layers, num_experts, dtype=torch.float32)
+
+ def place_weights(self, mode: str) -> None:
+ self.vision.place_weights(mode)
+
+ @torch.inference_mode()
+ def forward(self, item):
+ _, h, w = item.grid_thw
+ patches = item.feature.to(device=self.image_start.device, dtype=self.image_start.dtype)
+ x = self.aligner.forward(self.vision.forward(patches, h, w), h, w)
+ r = self.aligner.ratio
+ nh, nw = (h + r - 1) // r, (w + r - 1) // r
+ out = x.new_empty(nh * (nw + 1) + 2, x.shape[-1])
+ out[0], out[-1] = self.image_start, self.image_end
+ rows = out[1:-1].view(nh, nw + 1, -1)
+ rows[:, :nw] = x.view(nh, nw, -1)
+ rows[:, nw] = self.image_newline
+ return out
diff --git a/python/freetoken/models/deepseek_v41/weight.py b/python/freetoken/models/deepseek_v41/weight.py
new file mode 100644
index 00000000..1ccd23ea
--- /dev/null
+++ b/python/freetoken/models/deepseek_v41/weight.py
@@ -0,0 +1,242 @@
+"""Weight loading for DeepSeek-V4.1-Flash.
+
+* :func:`iter_weights` streams the resident (non-expert) tensors keyed by the model's attribute
+ paths: checkpoint ``layers.N.*`` -> ``model.layers.N.*``, ``embed`` / ``norm`` -> ``model.*``,
+ ``head`` -> ``head``. fp8 linears keep their e4m3 payload and e8m0 ``scale`` (declared by the quant
+ method as ``weight_scale_inv``); ``wo_a`` keeps its e4m3 payload and ``wo_a_scale`` too (the layer
+ dequantizes in-kernel for the reference's bf16 grouped einsum). The engine casts everything else to
+ the declared parameter dtype (fp32 compressor weights, hc gates, engram gate weights).
+* :func:`iter_expert_pieces` streams the routed MXFP4 experts (e2m1 pairs + e8m0 per-32 scales) per
+ expert for the offload banks.
+
+Dropped: the DSpark ``mtp.*`` stage; vision weights and ``gate.bias_vl`` when serving text-only.
+The Engram tables (``layers.{1,14}.engram.embed.*``) are
+not state-dict tensors: the disk row store maps them in place (``engram_table.py``); an FTW conversion
+carries them as side files (:func:`ftw_side_files`).
+"""
+
+from __future__ import annotations
+
+from contextlib import ExitStack
+import json
+import os
+import re
+
+import safetensors
+import torch
+from freetoken.distributed import get_tp_info
+from freetoken.layers.quantization import QuantKind
+from freetoken.models.loader import drop_page_cache
+from tqdm import tqdm
+
+from .args import DeepseekV41Args, Mode
+
+
+class _ShardReader:
+ def __init__(self, folder: str, weight_map: dict, device):
+ self._folder = folder
+ self._weight_map = weight_map
+ self._device = str(device)
+ self._handles: dict[str, object] = {}
+ self._stack = ExitStack()
+
+ def has(self, name: str) -> bool:
+ return name in self._weight_map
+
+ def get(self, name: str) -> torch.Tensor:
+ shard = self._weight_map[name]
+ handle = self._handles.get(shard)
+ if handle is None:
+ handle = self._stack.enter_context(safetensors.safe_open(os.path.join(self._folder, shard), framework="pt", device=self._device))
+ self._handles[shard] = handle
+ return handle.get_tensor(name)
+
+ def close(self) -> None:
+ try:
+ self._stack.close()
+ finally:
+ for shard in self._handles:
+ drop_page_cache(os.path.join(self._folder, shard))
+ self._handles.clear()
+
+
+def weight_map(model_path: str) -> dict:
+ with open(os.path.join(model_path, "model.safetensors.index.json")) as f:
+ return json.load(f)["weight_map"]
+
+
+def load_args(model_path: str) -> DeepseekV41Args:
+ from freetoken.utils.hf import cached_load_hf_config
+
+ return DeepseekV41Args.from_hf(cached_load_hf_config(model_path))
+
+
+def dequant_fp8_block(weight: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:
+ """Square block-scaled e4m3 -> bf16; ``scale`` holds e8m0 codes (``2^(code-127)``), the block
+ edge follows from the shapes."""
+ n, k = weight.shape
+ block = n // scale.shape[0]
+ assert (n // block, k // block) == tuple(scale.shape), (weight.shape, scale.shape)
+ s = torch.exp2(scale.view(torch.uint8).to(torch.float32) - 127.0)
+ s = s.repeat_interleave(block, dim=0).repeat_interleave(block, dim=1)
+ return (weight.to(torch.float32) * s).to(torch.bfloat16)
+
+
+def iter_weights(model_path: str, device, *, include_moe_experts: bool = True, include_non_moe: bool = True, include_vision: bool = True):
+ args = load_args(model_path)
+ reader = _ShardReader(model_path, weight_map(model_path), device)
+ get = reader.get
+ def linear(src: str, dst: str):
+ yield f"{dst}.weight", get(f"{src}.weight")
+ if reader.has(f"{src}.scale"):
+ yield f"{dst}.weight_scale_inv", get(f"{src}.scale")
+
+ try:
+ if include_moe_experts:
+ # Resident experts support unquantized checkpoints; FP4 uses the offload cache.
+ if reader.has("layers.0.ffn.experts.0.w1.scale"):
+ raise ValueError(
+ "DeepSeek-V4.1's fp4 routed experts are served from the offload cache; run with "
+ "--moe-strategy offload (include_moe_experts must be False)."
+ )
+ for L in range(args.n_layers):
+ p = f"layers.{L}.ffn.experts"
+ gate_up = torch.stack([torch.cat([get(f"{p}.{e}.w1.weight"), get(f"{p}.{e}.w3.weight")]) for e in range(args.n_routed_experts)])
+ down = torch.stack([get(f"{p}.{e}.w2.weight") for e in range(args.n_routed_experts)])
+ yield f"model.layers.{L}.ffn.experts.gate_up_proj", gate_up
+ yield f"model.layers.{L}.ffn.experts.down_proj", down
+ if not include_non_moe:
+ return
+ if include_vision and reader.has("vision.patch_embed.proj.weight"):
+ yield from iter_vision_weights(model_path, device)
+ yield "model.embed.weight", get("embed.weight")
+ yield "model.norm.weight", get("norm.weight")
+ yield "head.weight", get("head.weight")
+ for L in range(args.n_layers):
+ role = args.roles[L]
+ a, m = f"layers.{L}.attn", f"model.layers.{L}.attn"
+ yield from linear(f"{a}.wq_a", f"{m}.wq_a")
+ yield f"{m}.q_norm.weight", get(f"{a}.q_norm.weight")
+ yield from linear(f"{a}.wq_b", f"{m}.wq_b")
+ yield from linear(f"{a}.wkv", f"{m}.wkv")
+ yield f"{m}.kv_norm.weight", get(f"{a}.kv_norm.weight")
+ yield f"{m}.wo_a", get(f"{a}.wo_a.weight")
+ if reader.has(f"{a}.wo_a.scale"): # the fp8 payload + scale; the layer dequantizes in-kernel
+ yield f"{m}.wo_a_scale", get(f"{a}.wo_a.scale")
+ yield from linear(f"{a}.wo_b", f"{m}.wo_b")
+ yield f"{m}.attn_sink", get(f"{a}.attn_sink")
+ if role.mode is Mode.FULL:
+ c = f"{a}.compressor"
+ yield f"{m}.compressor.norm.weight", get(f"{c}.norm.weight")
+ if role.ratio > 1:
+ yield f"{m}.compressor.wkv", get(f"{c}.wkv.weight")
+ yield f"{m}.compressor.wgate", get(f"{c}.wgate.weight")
+ else:
+ yield from linear(f"{c}.wkv", f"{m}.compressor.wkv")
+ yield from linear(f"{a}.indexer.wk", f"{m}.indexer.wk")
+ yield f"{m}.indexer.k_norm.weight", get(f"{a}.indexer.k_norm.weight")
+ if role.has_indexer:
+ yield from linear(f"{a}.indexer.wq_b", f"{m}.indexer.wq_b")
+ yield from linear(f"{a}.indexer.weights_proj", f"{m}.indexer.weights_proj")
+
+ p = f"layers.{L}"
+ yield f"model.{p}.attn_norm.weight", get(f"{p}.attn_norm.weight")
+ yield f"model.{p}.ffn_norm.weight", get(f"{p}.ffn_norm.weight")
+ yield f"model.{p}.ffn.gate.weight", get(f"{p}.ffn.gate.weight")
+ yield f"model.{p}.ffn.gate.bias", get(f"{p}.ffn.gate.bias")
+ for proj in ("w1", "w2", "w3"):
+ yield from linear(f"{p}.ffn.shared_experts.{proj}", f"model.{p}.ffn.shared_experts.{proj}")
+ for nm in ("hc_attn_fn", "hc_ffn_fn", "hc_attn_base", "hc_ffn_base", "hc_attn_scale", "hc_ffn_scale"):
+ yield f"model.{p}.{nm}", get(f"{p}.{nm}")
+ if L in args.engram_layer_ids:
+ e = f"{p}.engram"
+ yield from linear(f"{e}.wkv", f"model.{e}.wkv")
+ yield f"model.{e}.q_weight", get(f"{e}.q_weight")
+ yield f"model.{e}.k_weight", get(f"{e}.k_weight")
+ finally:
+ reader.close()
+
+
+def iter_vision_weights(model_path: str, device):
+ names = weight_map(model_path)
+ reader = _ShardReader(model_path, names, device)
+ try:
+ for name in names:
+ if name.startswith(("vision.", "aligner.")) or name in ("image_start", "image_end", "image_newline"):
+ yield f"visual.{name}", reader.get(name)
+ args = load_args(model_path)
+ yield "visual.routing_bias", torch.stack([reader.get(f"layers.{i}.ffn.gate.bias_vl") for i in range(args.n_layers)])
+ finally:
+ reader.close()
+
+
+# ----- FTW side files: the Engram tables ----------------------------------------------------------
+def engram_table_names(layer_id: int) -> tuple[str, str]:
+ return f"layers.{layer_id}.engram.embed.weight", f"layers.{layer_id}.engram.embed.scale"
+
+
+def ftw_side_files(model_path: str, out_dir: str) -> list[str]:
+ """Copy the checkpoint shards holding the Engram tables (fp8 rows + ue8m0 scales) next to an FTW
+ checkpoint. The tables are never FTW entries: ``load_host_tables`` maps them in place from whatever
+ safetensors file holds them (``engram_table.engram_row_source`` reads the headers), and the FTW
+ loader never reads ``.safetensors`` as weights, so the shard's other tensors ride along unused."""
+ import shutil
+
+ from freetoken.models.loader import safetensors_weight_map
+ from freetoken.utils import download_hf_weight
+
+ folder = download_hf_weight(model_path)
+ args = load_args(folder)
+ shard_of = safetensors_weight_map(folder)
+ shards = sorted({shard_of[n] for layer_id in args.engram_layer_ids if layer_id < args.n_layers for n in engram_table_names(layer_id)})
+ for shard in shards:
+ shutil.copyfile(os.path.join(folder, shard), os.path.join(out_dir, shard))
+ return shards
+
+
+# ----- routed MXFP4 expert pieces ---------------------------------------------------------------
+_EXPERT_RE = re.compile(r"^layers\.(?P<layer>\d+)\.ffn\.experts\.(?P<expert>\d+)\.(?P<proj>w1|w2|w3)\.(?P<kind>weight|scale)$")
+_PROJ_ROLE = {"w1": "gate", "w3": "up", "w2": "down"}
+_KIND_SUFFIX = {"weight": "", "scale": "_scale"}
+
+
+def iter_expert_pieces(model_path: str, config, kind: QuantKind, *, parallel: bool | None = False, workers: int = 8, chunk: int = 8 << 20):
+ """One piece per expert: ``{gate, up, down}`` e2m1 pairs and their e8m0 ``_scale`` companions.
+ The DSpark stage's experts (``mtp.*``) never match."""
+ if kind is not QuantKind.MXFP4:
+ return None
+ if get_tp_info().size > 1:
+ raise NotImplementedError("DeepSeek-V4.1 expert banks support TP=1 only")
+ from freetoken.models.weight import iter_expert_tensors_parallel
+ from freetoken.moe.expert_pieces import per_expert_pieces
+
+ args = load_args(model_path)
+ L, E = args.n_layers, args.n_routed_experts
+
+ def locate(raw_name: str):
+ m = _EXPERT_RE.match(raw_name)
+ if m is None or int(m["layer"]) >= L:
+ return None
+ return int(m["layer"]), int(m["expert"]), _PROJ_ROLE[m["proj"]] + _KIND_SUFFIX[m["kind"]]
+
+ if parallel:
+ tensors = iter_expert_tensors_parallel(model_path, lambda n: locate(n) is not None, workers=workers, chunk=chunk)
+ return per_expert_pieces(tensors, locate, tensors_per_expert=6)
+
+ def _serial():
+ reader = _ShardReader(model_path, weight_map(model_path), torch.device("cpu"))
+ try:
+ for li in tqdm(range(L), desc="Loading DeepSeek-V4.1 experts (serial)", disable=not get_tp_info().is_primary()):
+ for e in range(E):
+ base = f"layers.{li}.ffn.experts.{e}"
+ for proj in ("w1", "w3", "w2"):
+ for kind_ in ("weight", "scale"):
+ name = f"{base}.{proj}.{kind_}"
+ yield name, reader.get(name)
+ finally:
+ reader.close()
+
+ return per_expert_pieces(_serial(), locate, tensors_per_expert=6)
+
+
+__all__ = ["iter_weights", "iter_vision_weights", "iter_expert_pieces", "ftw_side_files", "load_args", "weight_map", "dequant_fp8_block"]
diff --git a/python/freetoken/models/register.py b/python/freetoken/models/register.py
index 2a4ced10..fe150ddc 100644
--- a/python/freetoken/models/register.py
+++ b/python/freetoken/models/register.py
@@ -156,6 +156,20 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {
# the head, the KV compressors and the indexer's scorer ship bf16; the fp8 config has no modules_to_not_convert
unquantized_modules=("head", "*.compressor.wkv", "*.compressor.wgate", "*.indexer.weights_proj"),
),
+ # DeepSeek-V4.1-Flash (model_type deepseek_v41): multimodal wrapper config (text tower in
+ # text_config). The checkpoint has no ``model.`` root; the compressors, the indexer's key projection /
+ # scorer, the head and the vision tower + aligner ship bf16 (the fp8 config has no modules_to_not_convert).
+ "DeepseekV41ForCausalLM": ModelSpec(
+ "freetoken.models.deepseek_v41",
+ "DeepseekV41ForCausalLM",
+ mm_processor="freetoken.mm.processors.deepseek_v41:DeepseekV41MMProcessor",
+ encoders=(EncoderSpec("vision", "vision_config", ("image",)),),
+ checkpoint_roots=(("model.layers", "layers"), ("model.embed", "embed"), ("model.norm", "norm")),
+ packed_modules_mapping=_EXPERTS_W123_PACKED,
+ unquantized_modules=(
+ "head", "*.compressor.wkv", "*.compressor.wgate", "*.indexer.wk", "*.indexer.weights_proj", "vision.*", "aligner.*",
+ ),
+ ),
"Qwen3_5MoeForConditionalGeneration": ModelSpec(
"freetoken.models.qwen3_5_moe",
"Qwen3_5MoeForConditionalGeneration",
diff --git a/python/freetoken/moe/benchbw.py b/python/freetoken/moe/benchbw.py
index 83b3d196..e9d6ffe5 100644
--- a/python/freetoken/moe/benchbw.py
+++ b/python/freetoken/moe/benchbw.py
@@ -108,6 +108,7 @@ class Workload:
activation: str = "silu"
swiglu_alpha: float = 1.702
swiglu_limit: float | None = None
+ act_block: int | None = None # ds_fp4 W4A8 activation quant block
# Preset workloads. Dims from the model configs / benchmarks/bench_offload_cache_copy.py.
@@ -120,7 +121,8 @@ WORKLOADS: dict[str, Workload] = {
activation="gpt_oss_swiglu", swiglu_limit=7.0),
"gpt-oss-20b": Workload("gpt-oss-20b", 2880, 2880, 32, 4, ("mxfp4_triton",),
activation="gpt_oss_swiglu", swiglu_limit=7.0),
- "dsv4": Workload("dsv4", 4096, 2048, 256, 6, ("ds_fp4",), swiglu_limit=7.0),
+ "dsv4": Workload("dsv4", 4096, 2048, 256, 6, ("ds_fp4",), swiglu_limit=7.0, act_block=128),
+ "dsv4.1-flash": Workload("dsv4.1-flash", 5120, 2304, 384, 6, ("ds_fp4",), swiglu_limit=10.0, act_block=32),
"glm4.7-nvfp4": Workload("glm4.7-nvfp4", 5120, 1536, 160, 8, ("nvfp4",)),
"glm5.3-flash-nvfp4": Workload(
"glm5.3-flash-nvfp4", 4096, 2048, 288, 8, ("nvfp4",),
@@ -142,7 +144,7 @@ DTYPE_WORKLOADS: dict[str, Workload] = {
"fp8_block": Workload("dtype:fp8", 2048, 512, 128, 8, ("fp8_block",)),
"mxfp4_triton": Workload("dtype:mxfp4", 2880, 2880, 128, 4, ("mxfp4_triton",),
activation="gpt_oss_swiglu", swiglu_limit=7.0),
- "ds_fp4": Workload("dtype:ds_fp4", 4096, 2048, 128, 6, ("ds_fp4",), swiglu_limit=7.0),
+ "ds_fp4": Workload("dtype:ds_fp4", 4096, 2048, 128, 6, ("ds_fp4",), swiglu_limit=7.0, act_block=128),
}
@@ -460,7 +462,7 @@ def _build_cpu_moe_executor(fmt: str, wl: Workload, banks: dict, num_threads: in
cache, top_k=wl.top_k, activation=wl.activation,
apply_router_weight_on_input=False, num_threads=num_threads, max_tokens=1,
device=torch.device("cuda"), swiglu_alpha=wl.swiglu_alpha,
- swiglu_limit=wl.swiglu_limit,
+ swiglu_limit=wl.swiglu_limit, act_block=wl.act_block,
)
diff --git a/python/freetoken/moe/cpu_executor.py b/python/freetoken/moe/cpu_executor.py
index a4680674..c4809075 100644
--- a/python/freetoken/moe/cpu_executor.py
+++ b/python/freetoken/moe/cpu_executor.py
@@ -171,6 +171,7 @@ class CpuMoeExecutor:
swiglu_alpha: float = 1.702,
swiglu_limit: float | None = None,
fmt: str | None = None,
+ act_block: int | None = None,
) -> None:
from freetoken.kernel import _cpu_moe
from freetoken.moe.legacy_format import canonical_role
@@ -323,6 +324,11 @@ class CpuMoeExecutor:
# the C++ side to skip its own. Measured on DeepSeek-V4-Flash bs=1 decode:
# 12.85 -> 15.65 tok/s, output bit-identical (tests/moe/test_dsfp4_prequant.py).
self._gpu_prequant = fmt == "ds_fp4" and device.type == "cuda"
+ # the W4A8 activation round-trip block follows the checkpoint's fp8 block (128 on V4, 32 on V4.1)
+ self._act_block = act_block
+ if fmt == "ds_fp4":
+ assert act_block is not None, "ds_fp4 experts need act_block, the checkpoint's fp8 activation quant block"
+ self._ext.set_act_block(act_block)
if self._gpu_prequant:
self._ext.set_input_prequant(True)
logger.info_rank0(
@@ -482,7 +488,7 @@ class CpuMoeExecutor:
"""DeepSeek-V4 ``ds_fp4`` schema: row-major e2m1 (2/byte) + e8m0 per-32 block
scales, no global, no bias. Layout matches nvfp4 (K contiguous per output row),
so the C++ GEMV reads it in place. The kernel additionally FP8-round-trips the
- activations (block 128) to match DSV4's W4A8 reference, hence the %128 dims."""
+ activations (the checkpoint's fp8 block) to match the DeepSeek W4A8 reference."""
gup, gus = banks["gate_up"], banks["gate_up_scale"]
dnp, dns = banks["down"], banks["down_scale"]
assert gup[0].dtype == torch.uint8 and dnp[0].dtype == torch.uint8, (gup[0].dtype, dnp[0].dtype)
@@ -490,7 +496,7 @@ class CpuMoeExecutor:
I = int(gup[0].shape[1] // 2)
H = int(gup[0].shape[2] * 2)
assert gup[0].shape[1] == 2 * I
- assert H % 128 == 0 and I % 128 == 0, (H, I) # FP8 activation round-trip block=128
+ assert H % 32 == 0 and I % 32 == 0, (H, I) # e8m0 scales per 32 along K
assert tuple(dnp[0].shape[1:]) == (H, I // 2), (dnp[0].shape, H, I)
assert tuple(gus[0].shape[1:]) == (2 * I, H // 32), (gus[0].shape, I, H)
assert tuple(dns[0].shape[1:]) == (H, I // 32), (dns[0].shape, H, I)
@@ -580,7 +586,7 @@ class CpuMoeExecutor:
# pre-quantized activations and skips its serial scalar pass.
from freetoken.kernel.triton.dsv4.fp8_linear import act_quant_fp8_roundtrip
- hidden_states = act_quant_fp8_roundtrip(hidden_states, block=128)
+ hidden_states = act_quant_fp8_roundtrip(hidden_states, block=self._act_block)
# D2H: ship this step's activations + routing to pinned host memory.
io["x"].copy_(hidden_states, non_blocking=True)
diff --git a/python/freetoken/moe/fused_ds_fp4.py b/python/freetoken/moe/fused_ds_fp4.py
index 65fffb5c..b4dc5cd0 100644
--- a/python/freetoken/moe/fused_ds_fp4.py
+++ b/python/freetoken/moe/fused_ds_fp4.py
@@ -12,10 +12,7 @@ from __future__ import annotations
import torch
import triton
-from freetoken.kernel.triton.dsv4.fp8_linear import (
- act_quant_fp8_inplace,
- act_quant_fp8_roundtrip,
-)
+from freetoken.kernel.triton.dsv4.fp8_linear import act_quant_fp8_roundtrip
from freetoken.kernel.triton.dsv4.fused_moe import (
_decode_dsfp4_moe_kernel,
_e2m1_lut,
@@ -50,7 +47,8 @@ def _grouped_decode(
"""Grouped per-route GEMM ``c[t,r,:] = a[row] @ dequant(W[slot])^T``.
Returns ``[T, top_k, N]``. Used for both gate_up (a_row=token) and down
- (a_row=route). FP4 weights are dequantized inline from the slot cache.
+ (a_row=route). FP4 weights are dequantized inline from the slot cache. A slot
+ of ``-1`` is an inactive route: its output is zero and its storage is never read.
"""
T, top_k = slots.shape
N = packed_cache.shape[1]
@@ -62,14 +60,13 @@ def _grouped_decode(
topk_weights = out.new_empty((1, 1), dtype=torch.float32)
scale_u8 = scale_cache.view(torch.uint8)
- # Decode is a per-route GEMV, bound by the inline FP4 dequant (LUT gather + block
- # scale), not weight HBM bandwidth -- so it tops out ~29% of HBM peak. BN=16/
- # BKB=128/1 warp maximizes memory-level parallelism (many single-warp CTAs, no
- # cross-warp reduction) -> ~950 GB/s on H100 (was ~830 at BN=8/BKB=256/2). K_BYTES
- # must be a multiple of BLOCK_SIZE_KB (gate_up 2048, down 1024 -- both /128).
+ # One fixed launch geometry, never tuned per machine: the row / warp split changes the per-row
+ # K reduction layout (and so the output bits) on some Triton versions. 16 rows on 2 warps is at
+ # or near the fastest split for the V4 and V4.1 decode shapes. Each K tile contains whole
+ # packed FP4 scale groups.
BLOCK_SIZE_N = 16
BLOCK_SIZE_KB = 128
- _NW = 1
+ _NW = 2
assert (K // 2) % BLOCK_SIZE_KB == 0, (K, BLOCK_SIZE_KB)
grid = (total_routes, triton.cdiv(N, BLOCK_SIZE_N))
_decode_dsfp4_moe_kernel[grid](
@@ -103,28 +100,28 @@ def routed_experts_fp4(
down_packed: torch.Tensor, # [S, H, I//2] uint8
down_scale: torch.Tensor, # [S, H, I//32] e8m0
swiglu_limit: float,
+ act_block: int = 128,
) -> torch.Tensor:
"""Full routed-expert output (summed over the top-k routes), excludes shared expert.
- Precision matches the reference ``fp4_gemm(act_quant(x, 128), W_fp4)``: the gate_up
- and down activations are FP8-round-tripped (block 128, ue8m0) before each GEMM. Since
- an fp8 value x pow2 scale is exact in bf16, the round-tripped activation entering the
- bf16 decode kernel is bit-identical to the reference's dequantized FP8 activation
- (validated max diff = 0 vs the tilelang ``fp4_gemm`` reference)."""
+ Precision matches the reference ``fp4_gemm(act_quant(x, act_block), W_fp4)``: the gate_up
+ and down activations are FP8-round-tripped (block ``act_block`` -- the checkpoint's fp8 block,
+ 128 on V4 and 32 on V4.1 -- ue8m0) before each GEMM. Since an fp8 value x pow2 scale is exact
+ in bf16, the round-tripped activation entering the bf16 decode kernel is bit-identical to the
+ reference's dequantized FP8 activation (validated max diff = 0 vs the tilelang ``fp4_gemm``
+ reference). The routing weight scales each route's down output in the GEMM epilogue."""
T, top_k = slots.shape
H = x.shape[1]
two_I = gate_up_packed.shape[1]
I = two_I // 2
- x = act_quant_fp8_roundtrip(x, 128) # gate_up activation -> FP8 round-trip (no clone)
+ x = act_quant_fp8_roundtrip(x, act_block) # gate_up activation -> FP8 round-trip (no clone)
gate_up = _grouped_decode(
x, gate_up_packed, gate_up_scale, slots, None,
a_row_is_route=False, mul_routed_weight=False,
) # [T, top_k, 2I]
- act = fused_swiglu(gate_up, swiglu_limit) # [T, top_k, I]
-
- act = act.reshape(T * top_k, I)
- act_quant_fp8_inplace(act, 128) # down activation -> FP8 round-trip
+ # [T, top_k, I] with the down activation's FP8 round-trip in the same pass
+ act = fused_swiglu(gate_up, swiglu_limit, act_block=act_block).reshape(T * top_k, I)
down = _grouped_decode(
act, down_packed, down_scale, slots, topk_weights,
a_row_is_route=True, mul_routed_weight=True,
@@ -191,6 +188,7 @@ def routed_experts_fp4_prefill(
down_scale: torch.Tensor, # [S, H, I//32] e8m0
swiglu_limit: float,
num_rows: int,
+ act_block: int = 128,
) -> torch.Tensor:
"""Grouped-GEMM counterpart of :func:`routed_experts_fp4` for dense prefill
chunks: one moe_align sort shared by both GEMMs, each expert's weights
@@ -201,7 +199,7 @@ def routed_experts_fp4_prefill(
if T * top_k < _GROUPED_MIN_ROUTES:
return routed_experts_fp4(
x, slots, topk_weights,
- gate_up_packed, gate_up_scale, down_packed, down_scale, swiglu_limit,
+ gate_up_packed, gate_up_scale, down_packed, down_scale, swiglu_limit, act_block=act_block,
)
H = x.shape[1]
two_I = gate_up_packed.shape[1]
@@ -217,16 +215,14 @@ def routed_experts_fp4_prefill(
sorted_ids, expert_ids, ntpp = moe_align_block_size(slots, cfg["BLOCK_SIZE_M"], num_rows)
tw = topk_weights.reshape(-1).contiguous()
- x = act_quant_fp8_roundtrip(x, 128) # gate_up activation -> FP8 round-trip (no clone)
+ x = act_quant_fp8_roundtrip(x, act_block) # gate_up activation -> FP8 round-trip (no clone)
gate_up = torch.empty((T, top_k, two_I), dtype=x.dtype, device=x.device)
_grouped_prefill(
x, gate_up_packed, gate_up_scale, gate_up, tw,
sorted_ids, expert_ids, ntpp, routes, top_k, False, cfg,
)
- act = fused_swiglu(gate_up, swiglu_limit) # [T, top_k, I]
-
- act = act.reshape(routes, I)
- act_quant_fp8_inplace(act, 128) # down activation -> FP8 round-trip
+ # [T, top_k, I] with the down activation's FP8 round-trip in the same pass
+ act = fused_swiglu(gate_up, swiglu_limit, act_block=act_block).reshape(routes, I)
down = torch.empty((T, top_k, H), dtype=x.dtype, device=x.device)
_grouped_prefill(
act, down_packed, down_scale, down, tw,
diff --git a/python/freetoken/scheduler/cache.py b/python/freetoken/scheduler/cache.py
index 8f77be93..772297e0 100644
--- a/python/freetoken/scheduler/cache.py
+++ b/python/freetoken/scheduler/cache.py
@@ -66,6 +66,11 @@ class CacheManager:
supports_runtime_rebuild = True
prefill_chunk_budget = None # generic shared page pool: no per-model prefill chunk cap
+ @property
+ def prefix_replay_tokens(self) -> int:
+ """Prompt-tail tokens a prefix hit must leave to the prefill (a pool whose model recomputes them)."""
+ return getattr(self.swa_pool, "prefix_replay_tokens", 0) if self.swa_pool is not None else 0
+
@property
def prefill_chunk_align(self) -> int:
"""Granularity a non-final prefill chunk should end on. A hybrid snapshot is donated only
@@ -90,10 +95,10 @@ class CacheManager:
return SWARadixCache(device, page_size, self.sliding_window_size)
return create_prefix_cache(device=device, type=type, page_size=page_size)
- def match_req(self, req: PendingReq) -> MatchResult:
+ def match_req(self, req: PendingReq, max_len: int | None = None) -> MatchResult:
input_len = req.input_len
assert input_len > 0, "Input length must be greater than 0."
- ids = req.input_ids[: input_len - 1]
+ ids = req.input_ids[: input_len - 1 if max_len is None else max(0, min(input_len - 1, max_len))]
if self.is_swa:
from freetoken.kvcache.swa_radix_cache import SWACacheHandle
m = self.prefix_cache.match_prefix(ids)
diff --git a/python/freetoken/scheduler/prefill.py b/python/freetoken/scheduler/prefill.py
index bd8129aa..de4bddfb 100644
--- a/python/freetoken/scheduler/prefill.py
+++ b/python/freetoken/scheduler/prefill.py
@@ -71,8 +71,9 @@ class PrefillAdder:
if self.table_manager.available_size == 0:
return None
- # TODO: consider host cache match case
- mr = self.cache_manager.match_req(req)
+ # TODO: consider host cache match case (it needs the same replay cap)
+ replay = self.cache_manager.prefix_replay_tokens
+ mr = self.cache_manager.match_req(req, max_len=max(0, req.input_len - replay) if replay else None)
handle = mr.cuda_handle
cached_len = handle.cached_len
# TODO: better estimate policy
@@ -214,6 +215,7 @@ class PrefillAdder:
uid=pending_req.uid,
cache_handle=cache_handle,
sampling_params=pending_req.sampling_params,
+ prompt_len=pending_req.input_len,
)
req.mm_items = pending_req.mm_items
req.mrope_positions_full = pending_req.mrope_positions_full
diff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.py
index fa0d96a7..d057d3b9 100644
--- a/python/freetoken/scheduler/scheduler.py
+++ b/python/freetoken/scheduler/scheduler.py
@@ -349,7 +349,7 @@ class Scheduler(SchedulerIOMixin):
next_token = int(next_token.item())
# EOS / stop-string -> "stop", output budget exhausted -> "length";
# EOS and stop strings win over length.
- # Overlap can advance device_len ahead of the token delivered to the host.
+ # Overlap can advance device_len one token ahead of this host reply.
hit_length = req.input_ids.numel() >= req.max_device_len
hit_eos = (
not req.sampling_params.ignore_eos and next_token in self.eos_token_ids
@@ -737,17 +737,10 @@ class Scheduler(SchedulerIOMixin):
the CONCRETE current window (usable pages) so a rollback restores it byte-for-byte,
whether it was pinned or ratio-derived."""
eng = self.engine
- config = self.config
- mc = config.model_config
- num_swa_pages = None
- if getattr(mc, "dsv4_args", None) is not None:
- sizes = getattr(eng.kv_cache, "sizes", None)
- if sizes is not None: # usable window pages = physical n_win_pages minus the dummy page
- num_swa_pages = max(0, sizes.n_win_pages - 1)
- elif getattr(mc, "has_swa_attention", False) and (
- getattr(config, "cache_type", None) == "swa_radix"
- ): # usable window tokens = pool tokens minus the slot-0 sentinel
- num_swa_pages = max(0, int(getattr(eng.kv_cache, "swa_num_tokens", 0) or 0) - 1)
+ from freetoken.kvcache.cache_status import window_pool_spec
+
+ spec = window_pool_spec(self.config)
+ num_swa_pages = eng.kv_cache.window_pages if spec is not None else None
return dict(
num_pages=eng.num_pages,
moe_cache_size=eng.moe_offload_cache.cache_size if eng.moe_offload_cache is not None else None,
diff --git a/python/freetoken/server/api_server.py b/python/freetoken/server/api_server.py
index 5592a2b0..34221cfc 100644
--- a/python/freetoken/server/api_server.py
+++ b/python/freetoken/server/api_server.py
@@ -563,8 +563,12 @@ def _resolve_num_swa_pages(state: FrontendManager, req: CacheRebuildRequest) ->
or pools.get("num_pages", 0) or 0
)
page_size = int(pools.get("page_size", 0) or getattr(config, "page_size", 1) or 1)
- is_dsv4 = getattr(getattr(config, "model_config", None), "dsv4_args", None) is not None
- swa_page_size = page_size if is_dsv4 else 1
+ from freetoken.kvcache.cache_status import window_pool_spec
+
+ spec = window_pool_spec(config)
+ if spec is None:
+ raise ValueError("swa_full_tokens_ratio requested but this model has no resizable window pool")
+ swa_page_size = spec.page_size
window_tokens = int(round(req.swa_full_tokens_ratio * num_pages * page_size))
return max(1, -(-window_tokens // swa_page_size)) # ceil-div to the pool's page unit
@@ -604,12 +608,16 @@ async def cache_rebuild(req: CacheRebuildRequest):
{"status": "failed", "error": "swa_full_tokens_ratio must be in (0, 1]"},
status_code=422,
)
+ try:
+ num_swa_pages = _resolve_num_swa_pages(state, req)
+ except ValueError as exc:
+ return JSONResponse({"status": "failed", "error": str(exc)}, status_code=422)
result = await dispatch_rebuild(
state,
moe_cache_size=req.moe_cache_size,
num_pages=req.num_pages,
num_mamba_slots=req.num_mamba_slots,
- num_swa_pages=_resolve_num_swa_pages(state, req),
+ num_swa_pages=num_swa_pages,
mode=req.mode,
timeout=req.timeout,
)
@@ -739,11 +747,11 @@ def cache_geometry(state: Any) -> dict:
# the internal currency), else the load-time meta value. 0.0 for models without a window pool.
swa_full_tokens_ratio = float(getattr(state, "swa_full_tokens_ratio", 0.0) or 0.0)
last_swa_pages = last.get("num_swa_pages")
- if last_swa_pages:
- is_dsv4 = getattr(getattr(config, "model_config", None), "dsv4_args", None) is not None
- full = num_pages if is_dsv4 else num_pages * page_size
+ swa_page_size = int(pools.get("swa_page_size", 0) or 0)
+ if last_swa_pages and swa_page_size:
+ full = num_pages * page_size
if full > 0:
- swa_full_tokens_ratio = min(1.0, last_swa_pages / full)
+ swa_full_tokens_ratio = min(1.0, last_swa_pages * swa_page_size / full)
try:
model_config = config.model_config
num_experts = int(getattr(model_config, "num_experts", 0) or 0)
@@ -785,9 +793,8 @@ def cache_geometry(state: Any) -> dict:
"moe_cache_policy": getattr(config, "moe_cache_policy", None),
"unit_bytes": unit_bytes,
"swa_full_tokens_ratio": swa_full_tokens_ratio,
- # The window pool's page unit for num_swa_pages: DSV4 = P (== page_size), radix-SWA = 1
- # token, 0 for models without a window pool. Lets a client denominate the swa control.
- "swa_page_size": int(pools.get("swa_page_size", 0) or 0),
+ # The pool's window page unit; 0 for models without a resizable window pool.
+ "swa_page_size": swa_page_size,
# Concrete current window size in that unit (usable pages): the last rebuild's value if it
# pinned/derived one, else the load-time pool size. 0 for models without a window pool.
"num_swa_pages": int(last.get("num_swa_pages") or pools.get("num_swa_pages", 0) or 0),
diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.py
index 2194b82e..b8d06018 100644
--- a/python/freetoken/server/args.py
+++ b/python/freetoken/server/args.py
@@ -204,6 +204,8 @@ def parse_args(
return "qwen3_coder"
if "qwen" in marker:
return "qwen25"
+ if "deepseek_v41" in marker or "deepseekv41" in marker:
+ return "deepseekv41"
if "deepseek" in marker and ("v4" in marker or "deepseek_v4" in marker):
return "deepseekv32"
if "deepseek" in marker and ("v3.2" in marker or "v32" in marker):
@@ -576,6 +578,7 @@ def parse_args(
"qwen3_coder",
"mistral",
"deepseekv32",
+ "deepseekv41",
"gemma4",
"glm47",
"minimax",
@@ -645,6 +648,17 @@ def parse_args(
),
)
+ parser.add_argument(
+ "--swa-decoder-replay",
+ default=ServerArgs.swa_decoder_replay,
+ choices=["bounded", "exact"],
+ help=(
+ "DeepSeek-V4.1 Decoder SWA Bounded Replay. 'bounded' (default) runs the 20 decoder layers "
+ "on each prompt's last 128 tokens with their sliding window truncated there, as in the "
+ "tech report; 'exact' runs them on every prompt token (the reference numerics)."
+ ),
+ )
+
parser.add_argument(
"--nvfp4-backend",
action=_DeprecatedAlias,
diff --git a/python/freetoken/server/function_call_parser.py b/python/freetoken/server/function_call_parser.py
index 0804dcc3..296cd560 100644
--- a/python/freetoken/server/function_call_parser.py
+++ b/python/freetoken/server/function_call_parser.py
@@ -78,6 +78,8 @@ TOOLS_TAG_LIST = [
"<|DSML|function_calls>",
"<|DSML|tool_calls>",
"<|DSML|invoke",
+ "<|DSML| calls>",
+ "<|DSML| invoke",
"<atem:function_calls>",
]
@@ -1563,37 +1565,45 @@ class DeepSeekV32Detector(BaseFormatDetector):
Reference: https://huggingface.co/deepseek-ai/DeepSeek-V3.2
"""
+ # the DSML tag names: (block, alternative block), invoke, parameter
+ block_tags = ("function_calls", "tool_calls")
+ invoke_tag = "invoke"
+ param_tag = "parameter"
+
def __init__(self):
super().__init__()
self.dsml_token = "|DSML|"
- self.bot_token = f"<{self.dsml_token}function_calls>"
- self.eot_token = f"</{self.dsml_token}function_calls>"
- self.alt_bot_token = f"<{self.dsml_token}tool_calls>"
- self.alt_eot_token = f"</{self.dsml_token}tool_calls>"
- self.invoke_start_prefix = f"<{self.dsml_token}invoke"
- self.invoke_end_token = f"</{self.dsml_token}invoke>"
- self.param_end_token = f"</{self.dsml_token}parameter>"
+ block, alt_block = self.block_tags
+ self.bot_token = f"<{self.dsml_token}{block}>"
+ self.eot_token = f"</{self.dsml_token}{block}>"
+ self.alt_bot_token = f"<{self.dsml_token}{alt_block}>"
+ self.alt_eot_token = f"</{self.dsml_token}{alt_block}>"
+ self.invoke_start_prefix = f"<{self.dsml_token}{self.invoke_tag}"
+ self.invoke_end_token = f"</{self.dsml_token}{self.invoke_tag}>"
+ self.param_start_prefix = f"<{self.dsml_token}{self.param_tag}"
+ self.param_end_token = f"</{self.dsml_token}{self.param_tag}>"
# Regex for complete invoke extraction
- _de = re.escape(self.dsml_token)
+ _inv = re.escape(self.dsml_token + self.invoke_tag)
+ _par = re.escape(self.dsml_token + self.param_tag)
self.invoke_regex = re.compile(
- rf'<{_de}invoke\s+name="([^"]+)"\s*>(.*?)</{_de}invoke>',
+ rf'<{_inv}\s+name="([^"]+)"\s*>(.*?)</{_inv}>',
re.DOTALL,
)
# Regex for parameter extraction
self.param_regex = re.compile(
- rf'<{_de}parameter\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>(.*?)</{_de}parameter>',
+ rf'<{_par}\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>(.*?)</{_par}>',
re.DOTALL,
)
# Regex for partial invoke (name known, body still streaming)
self.partial_invoke_regex = re.compile(
- rf'<{_de}invoke\s+name="([^"]+)"\s*>(.*)',
+ rf'<{_inv}\s+name="([^"]+)"\s*>(.*)',
re.DOTALL,
)
# Streaming state machine tag regexes (anchored matches over the buffer).
- self.invoke_open_regex = re.compile(rf'<{_de}invoke\s+name="([^"]+)"\s*>')
+ self.invoke_open_regex = re.compile(rf'<{_inv}\s+name="([^"]+)"\s*>')
self.param_open_regex = re.compile(
- rf'<{_de}parameter\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>'
+ rf'<{_par}\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>'
)
self._last_arguments = ""
@@ -1687,7 +1697,7 @@ class DeepSeekV32Detector(BaseFormatDetector):
normal_parts: List[str] = []
calls: List[ToolCallItem] = []
- param_open_token = f"<{self.dsml_token}parameter"
+ param_open_token = self.param_start_prefix
def _emit_args(fragment: str) -> None:
if fragment:
@@ -3512,6 +3522,16 @@ class MuseGlimmerDetector(InvokeParamStreamMixin, BaseFormatDetector):
return StreamingParseResult(normal_text="".join(normal_parts).strip(), calls=calls)
+class DeepSeekV41Detector(DeepSeekV32Detector):
+ """DeepSeek-V4.1's DSML: the V3.2 grammar with every tag name led by a space and the block
+ named ``calls`` (``<|DSML| calls>`` / ``<|DSML| invoke name=...>`` / ``<|DSML| parameter
+ ...>``, the checkpoint's ``encoding/encoding.py``)."""
+
+ block_tags = (" calls", " calls")
+ invoke_tag = " invoke"
+ param_tag = " parameter"
+
+
class FunctionCallParser:
"""
Parser for function/tool calls in model outputs.
@@ -3523,6 +3543,7 @@ class FunctionCallParser:
ToolCallParserEnum: Dict[str, Type[BaseFormatDetector]] = {
"deepseekv32": DeepSeekV32Detector,
+ "deepseekv41": DeepSeekV41Detector,
"gemma4": Gemma4Detector,
"gpt-oss": GptOssDetector,
"gpt_oss": GptOssDetector,
diff --git a/python/freetoken/server/stats.py b/python/freetoken/server/stats.py
index d01828e2..3d48ee96 100644
--- a/python/freetoken/server/stats.py
+++ b/python/freetoken/server/stats.py
@@ -120,12 +120,10 @@ def derive_model_card(config: Any) -> dict:
def _swa_page_size(config: Any) -> int:
- """The window pool's own page unit: P (window_size) for DSV4, 1 token for radix-SWA.
- Mirrors compute_cache_pools' swa_page_size."""
- dsv4 = getattr(getattr(config, "model_config", None), "dsv4_args", None)
- if dsv4 is not None:
- return int(getattr(dsv4, "window_size", 0) or 1)
- return 1
+ from freetoken.kvcache.cache_status import window_pool_spec
+
+ spec = window_pool_spec(config)
+ return spec.page_size if spec is not None else 1
def build_stats(state: Any, p95_ms: int, ttft_mean_ms: int) -> dict:
diff --git a/python/freetoken/tokenizer/effort.py b/python/freetoken/tokenizer/effort.py
index c81cd7be..2e9e3cd2 100644
--- a/python/freetoken/tokenizer/effort.py
+++ b/python/freetoken/tokenizer/effort.py
@@ -84,7 +84,8 @@ class EffortProfile:
template interpolates anything". ``strength_dialect`` True means the
template reads the ``reasoning_strength`` spelling (muse-glimmer's family,
whose documented ladder tops out at xhigh) rather than only
- ``reasoning_effort``.
+ ``reasoning_effort``. ``integer_range`` records a validated 1-100 budget
+ accepted independently of the named effort vocabulary.
"""
supported: frozenset[str]
@@ -92,6 +93,7 @@ class EffortProfile:
consumes_effort: bool
validates: bool = False
strength_dialect: bool = False
+ integer_range: tuple[int, int] | None = None
@dataclass(frozen=True)
@@ -118,7 +120,7 @@ class ThinkingProfile:
_MAX_QUANTIZE_DISTANCE = 0.15
-def quantize_effort(value: Any, profile: EffortProfile) -> str | None:
+def quantize_effort(value: Any, profile: EffortProfile) -> str | int | None:
"""Map a client's effort onto ``profile``; ``None`` means "send nothing".
In-vocabulary values pass through untouched. Other named levels land on the
@@ -128,6 +130,11 @@ def quantize_effort(value: Any, profile: EffortProfile) -> str | None:
the template default. With "max" excluded the remaining scale values are
unique, so quantization is deterministic across processes.
"""
+ if profile.integer_range is not None and isinstance(value, (int, float)):
+ low, high = profile.integer_range
+ if type(value) is not int or not low <= value <= high:
+ raise ValueError(f"reasoning_effort must be an integer in [{low}, {high}]")
+ return value
supported = effective_efforts(profile)
if not supported:
return None
@@ -181,6 +188,7 @@ def probe_effort_profile(
diverged: set[str] = set()
matches_baseline: dict[str, bool] = {name: True for name in KNOWN_REASONING_EFFORTS}
ran_rounds = 0
+ integer_range = None
for base_kwargs, tools in _PROBE_ROUNDS:
try:
@@ -198,6 +206,10 @@ def probe_effort_profile(
if _renderings_differ(rendering, baseline):
diverged.add(name)
matches_baseline[name] = False
+ # Only a validated budget range is a numeric capability; arbitrary Jinja
+ # interpolation must not advertise support for numeric reasoning effort.
+ if _accepts_integer_budget(render, base_kwargs, tools):
+ integer_range = (1, 100)
if ran_rounds == 0:
# Nothing learnable: sending no effort is the only safe rendering.
@@ -214,7 +226,7 @@ def probe_effort_profile(
strength_dialect = False
supported = frozenset(name for name in KNOWN_REASONING_EFFORTS if name not in rejected)
- consumes = bool(rejected or diverged)
+ consumes = bool(rejected or diverged or integer_range)
default = None
if consumes:
defaults = [name for name in supported if matches_baseline[name]]
@@ -226,9 +238,26 @@ def probe_effort_profile(
consumes_effort=consumes,
validates=bool(rejected),
strength_dialect=strength_dialect,
+ integer_range=integer_range,
)
+def _accepts_integer_budget(render, base_kwargs, tools) -> bool:
+ for value in (0, 101):
+ try:
+ render({**base_kwargs, "reasoning_effort": value}, tools)
+ except Exception: # noqa: BLE001 -- the encoder owns validation
+ continue
+ return False
+ outputs = []
+ for value in range(1, 101):
+ try:
+ outputs.append(render({**base_kwargs, "reasoning_effort": value}, tools))
+ except Exception: # noqa: BLE001
+ return False
+ return _renderings_differ(outputs[0], outputs[-1])
+
+
def probe_thinking_profile(
render: Callable[[dict[str, Any], list[dict[str, Any]] | None], Any],
efforts: EffortProfile,
diff --git a/python/freetoken/tokenizer/tokenize.py b/python/freetoken/tokenizer/tokenize.py
index f4736652..77247257 100644
--- a/python/freetoken/tokenizer/tokenize.py
+++ b/python/freetoken/tokenizer/tokenize.py
@@ -59,7 +59,7 @@ class TokenizeManager:
self._effort_profile: EffortProfile | None = None
self._thinking_profile: ThinkingProfile | None = None
self._effort_lock = threading.Lock()
- self._logged_effort_maps: set[tuple[Any, str | None]] = set()
+ self._logged_effort_maps: set[tuple[Any, str | int | None]] = set()
self._inline_system_lock = threading.Lock()
self._inline_system_modes: dict[str, str] = {}
@@ -117,7 +117,7 @@ class TokenizeManager:
if mode == "auto":
mode = self.inline_system_mode(msg.tools, kwargs)
messages = normalize_inline_system(messages, mode)
- return self._render(messages, msg.tools, kwargs)
+ return self._render(messages, msg.tools, kwargs, images=msg.images)
def inline_system_mode(self, tools: list[dict] | None, kwargs: dict[str, Any]) -> str:
key = json.dumps([tools, kwargs], sort_keys=True)
@@ -137,12 +137,13 @@ class TokenizeManager:
messages: list[dict[str, Any]],
tools: list[dict[str, Any]] | None,
chat_template_kwargs: dict[str, Any],
+ *, images: list[bytes] | None = None,
) -> str:
"""Raw render, no effort sanitation — the probe needs unsupported values
to actually reach the template so rejection is observable."""
if self._dsv4_encoder is not None:
return _apply_dsv4_chat_encoder(
- self._dsv4_encoder, messages, tools, chat_template_kwargs
+ self._dsv4_encoder, messages, tools, chat_template_kwargs, images=images
)
# Broadcast the effort in every spelling the ecosystem's templates read
# (muse-glimmer grades ``reasoning_strength``; Jinja ignores undeclared
@@ -222,8 +223,12 @@ def _load_dsv4_encoder_if_needed(tokenizer: PreTrainedTokenizerBase) -> ModuleTy
model_path = getattr(tokenizer, "name_or_path", None) or getattr(tokenizer, "_name_or_path", "")
if not model_path:
return None
- encoder_path = os.path.join(str(model_path), "encoding", "encoding_dsv4.py")
- if not os.path.isfile(encoder_path):
+ # DeepSeek-V4 ships encoding/encoding_dsv4.py, V4.1 encoding/encoding.py; same encode_messages contract
+ encoder_path = next(
+ (p for p in (os.path.join(str(model_path), "encoding", name) for name in ("encoding_dsv4.py", "encoding.py")) if os.path.isfile(p)),
+ None,
+ )
+ if encoder_path is None:
return None
spec = importlib.util.spec_from_file_location("encoding_dsv4", encoder_path)
if spec is None or spec.loader is None:
@@ -240,9 +245,28 @@ def _apply_dsv4_chat_encoder(
messages: list[dict],
tools: list[dict] | None,
chat_template_kwargs: dict,
+ *, images: list[bytes] | None = None,
) -> str:
rendered_messages = [dict(message) for message in messages]
+ image_data = iter(images) if images is not None else None
for message in rendered_messages:
+ if isinstance(message.get("content"), list):
+ content = []
+ for part in message["content"]:
+ if isinstance(part, dict) and part.get("type") == "image":
+ # Prerender still has refs; the tokenizer worker receives fetched bytes.
+ # The checkpoint encoder validates sources even when rendering only text.
+ ref = part.get("freetoken_ref")
+ if image_data is not None:
+ try:
+ part = {"type": "image", "data": next(image_data)}
+ except StopIteration:
+ raise ValueError("image blocks outnumber fetched images") from None
+ elif ref is not None:
+ key = "data" if ref["kind"] == "b64" else "url"
+ part = {"type": "image", key: ref["data"]}
+ content.append(part)
+ message["content"] = content
if message.get("tool_calls"):
message["tool_calls"] = _dsv4_tool_calls(message["tool_calls"])
if tools:
diff --git a/scripts/ftw_hotfix.py b/scripts/ftw_hotfix.py
index f9cb62d1..9640c7b7 100644
--- a/scripts/ftw_hotfix.py
+++ b/scripts/ftw_hotfix.py
@@ -263,7 +263,9 @@ def is_checkpoint_tower_name(name: str) -> bool:
head = name.split(".")
if head[0] == "model" and len(head) > 1:
head = head[1:]
- return "vision" in head[0] or "visual" in head[0] or head[0] in _TOWER_SEGMENTS
+ return ("vision" in head[0] or "visual" in head[0] or head[0] in _TOWER_SEGMENTS
+ or head[0] in ("aligner", "image_start", "image_end", "image_newline")
+ or name.endswith(".ffn.gate.bias_vl"))
def read_tower(source: TensorSource, ftw_dir: str, checkpoint_names: list[str]) -> dict[str, torch.Tensor]:
diff --git a/tests/README.md b/tests/README.md
index be22c1fb..76b2b1fb 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -62,6 +62,7 @@ checkpoint is set:
| `FREETOKEN_TEST_MEM_RATIO` | `e2e/test_aime.py` — offload-mode memory_ratio (default `0.9`) |
| `FREETOKEN_REBUILD_TEST_MODEL` | `e2e/test_cache_rebuild.py` — a SMALL local model dir; boots a real server (falls back to `FREETOKEN_TEST_MODEL`) |
| `FREETOKEN_GEMMA4_GGUF_GLOB` | `models/test_gemma4_gguf_rope.py` — glob matching a local gemma-4 GGUF file |
+| `FREETOKEN_TEST_MODELS_ROOT` | `models/test_quant_config.py` — directory whose checkpoints the stored-tensor scheme scan checks (default `/mnt/nvme/models`) |
`test_aime.py` takes its sampling protocol from the checkpoint's own
`generation_config.json` (pass@N at the recommended temperature, or a single greedy
diff --git a/tests/attention/test_dsv41_backend.py b/tests/attention/test_dsv41_backend.py
new file mode 100644
index 00000000..387742eb
--- /dev/null
+++ b/tests/attention/test_dsv41_backend.py
@@ -0,0 +1,208 @@
+"""DSV41 attention backend: addressing and selection contracts, CPU-only (kernels live in tests/kernels).
+
+* prefill metadata: encoder segments, the decoder pass under exact vs bounded replay;
+* window candidates with a floor (bounded replay truncates the window at the prompt's last window);
+* decode snapshot staging and the layer-invariant ring context;
+* the selection helpers' reshape, candidate gather and position-to-row mapping.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from freetoken.core import Batch, Context, Req, SamplingParams, get_global_ctx, set_global_ctx
+from freetoken.kvcache.dsv4.v41_cost_model import dsv41_pool_sizes
+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache
+from freetoken.models.deepseek_v41.args import DeepseekV41Args
+
+P, MRR, DEVICE = 128, 4, torch.device("cpu")
+RATIOS = (0, 2, 2, 1, 1)
+SOURCES = (1, 3)
+
+
+def _ctx(pool):
+ try:
+ ctx = get_global_ctx()
+ except AssertionError:
+ ctx = Context(page_size=P)
+ set_global_ctx(ctx)
+ ctx.kv_cache = pool
+ return ctx
+
+
+def _stack(swa_decoder_replay="exact", num_pages=32, max_seq_len=8192):
+ from freetoken.attention.dsv41_sparse import DSV41SparseAttnBackend
+
+ args = DeepseekV41Args(n_layers=len(RATIOS), head_dim=512, index_head_dim=128, window_size=P, compress_ratios=RATIOS,
+ kv_source_layers=SOURCES, swa_decoder_replay=swa_decoder_replay)
+ pool = DSV41PagedKVCache(dsv41_pool_sizes(num_pages + 1, args, 1.0, P), args, DEVICE, n_scratch=MRR + 1)
+ pool._init_paged_state(MRR, True)
+ pt = torch.zeros(MRR + 1, max_seq_len, dtype=torch.int32)
+ pt[MRR].fill_(num_pages * P)
+ pt[2, :300] = torch.arange(300, dtype=torch.int32)
+ for page in range(3):
+ pool.bind_window_pages(page * P, page * P)
+ pool.full_loc_map = pt
+ _ctx(pool)
+ backend = DSV41SparseAttnBackend(SimpleNamespace(dsv41_args=args))
+ return backend, pool, pt
+
+
+def _req(table_idx, cached_len, n_new, output_len=16, uid=0, prompt_len=0):
+ return Req(input_ids=torch.zeros(cached_len + n_new, dtype=torch.int32), table_idx=table_idx, cached_len=cached_len,
+ output_len=output_len, uid=uid, sampling_params=SamplingParams(), cache_handle=None, prompt_len=prompt_len)
+
+
+def _prefill_batch(reqs):
+ batch = Batch(reqs=reqs, phase="prefill")
+ batch.padded_reqs = reqs
+ return batch
+
+
+def test_prefill_metadata_exact_mode_runs_the_decoder_on_every_token():
+ backend, pool, _ = _stack("exact")
+ assert pool.prefix_replay_tokens == 0
+ batch = _prefill_batch([_req(0, 0, 300), _req(2, 256, 44, uid=1)])
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ assert [(s.offset, s.n, s.table_idx, s.start_pos, s.window_floor) for s in md.segments] == [(0, 300, 0, 0, 0), (300, 44, 2, 256, 0)]
+ assert md.decoder_segments is md.segments and md.decoder_rows is None
+ assert md.last_indices.tolist() == [299, 343]
+
+
+def test_prefill_metadata_bounded_mode_runs_the_decoder_on_the_last_window():
+ """The encoder runs every new token; the decoder runs each request's part of its prompt's last
+ window ``[L - P, L)``, floored at ``L - P``. A chunk ending before that window runs no decoder row
+ and its ``last_indices`` entry points at the placeholder row the model appends."""
+ backend, pool, _ = _stack("bounded")
+ assert pool.prefix_replay_tokens == P and pool.sliding_window_size == P
+ cold = _req(0, 0, 300) # a whole prompt: L - P = 172
+ early = _req(1, 256, 128, uid=1, prompt_len=600) # chunk [256, 384) of a 600-token prompt: before 472
+ straddle = _req(2, 384, 128, uid=2, prompt_len=600) # chunk [384, 512): its rows [472, 512) are in the window
+ batch = _prefill_batch([cold, early, straddle])
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ assert [(s.offset, s.n, s.start_pos, s.window_floor) for s in md.segments] == [(0, 300, 0, 0), (300, 128, 256, 0), (428, 128, 384, 0)]
+ assert [(d.offset, d.n, d.table_idx, d.start_pos, d.window_floor) for d in md.decoder_segments] == [(0, 128, 0, 172, 172), (128, 40, 2, 472, 472)]
+ assert md.decoder_rows.tolist() == list(range(172, 300)) + list(range(428 + 88, 556))
+ assert md.decoder_pad and md.last_indices.tolist() == [127, 168, 167] # 168: the placeholder after 168 decoder rows
+
+
+def test_prefill_metadata_bounded_mode_without_any_decoder_row():
+ backend, _, _ = _stack("bounded")
+ batch = _prefill_batch([_req(0, 0, 256, prompt_len=1024)])
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ assert md.decoder_segments == [] and md.decoder_rows.numel() == 0
+ assert md.decoder_pad and md.last_indices.tolist() == [0]
+
+
+def test_prefill_metadata_short_prompt_runs_the_decoder_everywhere():
+ """A prompt shorter than the window floors the decoder's window at 0: it sees what exact mode sees."""
+ backend, _, _ = _stack("bounded")
+ batch = _prefill_batch([_req(0, 0, 50)])
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ assert [(d.offset, d.n, d.start_pos, d.window_floor) for d in md.decoder_segments] == [(0, 50, 0, 0)]
+ assert not md.decoder_pad and md.last_indices.tolist() == [49]
+
+
+def test_private_window_layers_address_per_request_rings():
+ """Under bounded replay the decoder layers keep their window KV in per-request rings (slot =
+ table row * P + pos % P), never in the radix-shared window pages; the encoder layers stay on
+ the shared pool."""
+ from freetoken.attention.dsv41_sparse import PrefillSegment
+
+ backend, pool, _ = _stack("bounded")
+ assert pool.private_window_layer_ids == (3, 4) and not any(pool.is_private_window(l) for l in (0, 1, 2))
+ assert pool.window_pool[3].shape[0] == (MRR + 1) * P and pool.window_pool[0].shape[0] == pool.sizes.n_win_slots
+ # prefill: a decoder segment's slots are its ring row; an encoder segment's are page-bound
+ seg = PrefillSegment(0, 44, 2, 256, window_floor=256)
+ assert backend.layer_window_slots_of(3, 2, 256, 300).tolist() == [2 * P + p % P for p in range(256, 300)]
+ assert backend.layer_window_slots_of(0, 2, 256, 300).tolist() == list(range(256, 300))
+ g = backend.window_topk_prefill(seg, layer_id=3)[0]
+ assert g.shape == (44, P) and g[0, 0].item() == 2 * P + 0 and (g[0, 1:] == -1).all() # position 256 sees itself only (floor)
+ assert set(g[43].tolist()) - {-1} == {2 * P + p % P for p in range(256, 300)}
+ # decode: the private ring context keys on the batch row's page-table row
+ batch = _decode_batch([2], [300])
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ slots, topk = md.private_window_ctx(torch.tensor([300]), torch.arange(1))
+ assert slots.tolist() == [2 * P + 300 % P] and topk.shape == (1, 1, P)
+ assert set(topk[0, 0].tolist()) == {2 * P + j for j in range(P)} # every ring index holds a position <= 300
+
+
+def test_window_candidates_honor_the_floor_and_the_retained_prefix():
+ from freetoken.attention.dsv41_sparse import PrefillSegment
+
+ backend, pool, _ = _stack()
+ # a cold segment at positions [0, 10): query p sees [0, p]
+ g = backend.window_topk_prefill(PrefillSegment(0, 10, 2, 0))
+ assert g.shape == (1, 10, P)
+ assert g[0, 3, :4].tolist() == [0, 1, 2, 3] and (g[0, 3, 4:] == -1).all()
+ # an extend at [256, 300) sees back into the retained prefix, but not below the floor
+ g = backend.window_topk_prefill(PrefillSegment(0, 44, 2, 256, window_floor=200))
+ row = g[0, 0] # query at 256 -> [200, 256]
+ assert row[0].item() == 200 and row[56].item() == 256 and (row[57:] == -1).all()
+ # without a floor the query at 256 sees [129, 256]
+ g = backend.window_topk_prefill(PrefillSegment(0, 44, 2, 256))
+ assert g[0, 0, 0].item() == 129 and g[0, 0, 127].item() == 256
+
+
+def _decode_batch(rows, positions):
+ reqs = [_req(int(t), 0, 1, uid=i) for i, t in enumerate(rows)]
+ batch = Batch(reqs=reqs, phase="decode")
+ batch.padded_reqs = reqs
+ batch.active_table_idx = torch.tensor(rows, dtype=torch.int64)
+ batch.positions = torch.tensor(positions, dtype=torch.int64)
+ return batch
+
+
+def test_decode_snapshot_and_ring_context():
+ backend, pool, pt = _stack()
+ batch = _decode_batch([2, MRR], [259, 0])
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ assert md.full_snap is None
+ snap = md.full_snapshot()
+ assert torch.equal(snap[0, :300], torch.arange(300))
+ pt[2, :10] = 7 # a later mutation must not reach the snapshot
+ assert torch.equal(snap[0, :300], torch.arange(300))
+ pos, rows = batch.positions, torch.arange(2)
+ ws, prev, topk = md.window_ctx(pos, rows)
+ dummy_ws = pool.sizes.n_win_slots - P # the dummy row's full loc binds to the dummy window page
+ assert ws.tolist() == [259, dummy_ws] and prev.tolist() == [258, dummy_ws]
+ assert topk.shape == (2, 1, P) and (topk[0] >= 0).all()
+ assert set(topk[0, 0].tolist()) == set(range(259 - 127, 260))
+ assert topk[1, 0, 0].item() == dummy_ws and (topk[1, 0, 1:] == -1).all()
+ # the indexer derives compressed rows from the live locs: prefill off the page table row,
+ # decode off the snapshot (position t -> full_loc(t * ratio) // ratio)
+ assert torch.equal(backend.locs_prefill(2, 20), pt[2:3, :20])
+ assert snap[0, :16].tolist() == list(range(16)) # taken before the mutation above
+ # capture staging reuses one buffer
+ backend.init_capture_graph(512, [1, 2])
+ backend.prepare_for_replay(batch)
+ assert batch.attn_metadata.full_snap.data_ptr() == backend.capture.full_snap.data_ptr()
+ assert batch.attn_metadata.stage_width == 512
+
+
[email protected](not torch.cuda.is_available(), reason="the selection kernels need CUDA")
+def test_topk_helpers():
+ from freetoken.attention.dsv41_sparse import DSV41SparseAttnBackend as B
+
+ scores = torch.tensor([[[0.1, 5.0, 3.0, 7.0, 9.0]]], device="cuda") # columns past live are never read
+ live = torch.tensor([[3]], device="cuda", dtype=torch.int32)
+ assert B.select_topk(scores, live, 4).tolist() == [[[0, 1, 2, -1]]]
+ assert B.select_topk(scores, live, 2).tolist() == [[[1, 2]]]
+ # a candidate list is a sorted valid prefix; empty slots score -inf
+ cand = torch.tensor([[[2, 7, 9, -1]]], device="cuda", dtype=torch.int32)
+ cscores = torch.tensor([[[1.0, 4.0, -torch.inf, -torch.inf]]], device="cuda")
+ assert B.select_topk_in_candidates(cscores, cand, 3).tolist() == [[[2, 7, -1]]]
+ # rows come from the live locs: compressed position p -> locs[p * ratio] // ratio
+ locs = torch.arange(20, 40, device="cuda", dtype=torch.int32).view(1, 20)
+ locs[0, 14:16] = -1 # position 7 (ratio 2) has slid out
+ rows = B.positions_to_rows(torch.tensor([[[2, 7, 9, -1]]], device="cuda", dtype=torch.int32), locs, 2)
+ assert rows.tolist() == [[[12, -1, 19, -1]]]
diff --git a/tests/checkpoint/test_ftw_hotfix_vision.py b/tests/checkpoint/test_ftw_hotfix_vision.py
index d4a3eac6..eb0c592f 100644
--- a/tests/checkpoint/test_ftw_hotfix_vision.py
+++ b/tests/checkpoint/test_ftw_hotfix_vision.py
@@ -14,7 +14,7 @@ _SCRIPT = pathlib.Path(__file__).resolve().parents[2] / "scripts" / "ftw_hotfix.
# env var -> local checkpoint, as tests/README.md lists them
_CHECKPOINTS = ("FREETOKEN_QWEN3VL_MODEL", "FREETOKEN_GEMMA4_MODEL", "FREETOKEN_GEMMA4_UNIFIED_MODEL",
- "FREETOKEN_GLM53_MODEL", "FREETOKEN_MUSE_MODEL", "FREETOKEN_MINIMAX_M3_MODEL")
+ "FREETOKEN_GLM53_MODEL", "FREETOKEN_MUSE_MODEL", "FREETOKEN_MINIMAX_M3_MODEL", "FREETOKEN_DSV41_MODEL")
@pytest.fixture(scope="module")
@@ -34,6 +34,10 @@ def hotfix():
("vision_tower.vision_model.embeddings.patch_embedding.weight", True),
("multi_modal_projector.linear_1.weight", True),
("patch_merge_mlp.linear_2.bias", True),
+ ("vision.blocks.0.norm1.weight", True),
+ ("aligner.w1.weight", True),
+ ("image_newline", True),
+ ("layers.3.ffn.gate.bias_vl", True),
("model.embed_audio.embedding_projection.weight", False),
("model.language_model.layers.0.self_attn.q_proj.weight", False),
("model.layers.3.mlp.experts.0.gate_proj.weight", False),
diff --git a/tests/dsv4/test_dsv4_attn_backend.py b/tests/dsv4/test_dsv4_attn_backend.py
index 8ffc6fd4..7bc72210 100644
--- a/tests/dsv4/test_dsv4_attn_backend.py
+++ b/tests/dsv4/test_dsv4_attn_backend.py
@@ -11,8 +11,8 @@ from __future__ import annotations
import torch
from freetoken.core import Batch, Context, Req, SamplingParams, get_global_ctx, set_global_ctx
-from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
-from freetoken.kvcache.dsv4_cost_model import dsv4_pool_sizes
+from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
+from freetoken.kvcache.dsv4.v4_cost_model import dsv4_pool_sizes
from freetoken.models.deepseek_v4.args import DeepseekV4Args
P, MRR, DEVICE = 128, 4, torch.device("cpu")
diff --git a/tests/kernels/test_dsv41_indexer.py b/tests/kernels/test_dsv41_indexer.py
new file mode 100644
index 00000000..19b45ab8
--- /dev/null
+++ b/tests/kernels/test_dsv41_indexer.py
@@ -0,0 +1,139 @@
+"""DSV41 indexer logits over packed fp4 index keys: full-range (Full mode) and candidate-restricted
+(Reindex mode) against a torch reference on the dequantized keys, with causality from ``live``. The
+row of a compressed position comes from the request's full-token locs (``locs[t * ratio] // ratio``);
+in Full mode only the live columns are written."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+from freetoken.kvcache.dsv4.v41_row_format import FP4_E8M0_B32
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+D, H = 128, 32
+ROWS = 4200
+
+
+RATIO = 2
+
+
+def _setup(b, s, t, seed):
+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows
+
+ torch.manual_seed(seed)
+ keys = torch.randn(ROWS, D, device="cuda", dtype=torch.bfloat16)
+ pool = pack_rows(keys, FP4_E8M0_B32)
+ deq = unpack_rows(pool, FP4_E8M0_B32, D)
+ q = torch.randn(b, s, H, D, device="cuda", dtype=torch.bfloat16)
+ w = (torch.randn(b, s, H, device="cuda") * 0.3).bfloat16()
+ # each request owns a random permutation of pool rows for its compressed positions; a few holes.
+ # The kernel derives rows from full-token locs: compressed position p -> loc = row * RATIO (+ any
+ # in-group offset) at locs[p * RATIO]; a hole is a negative loc.
+ k_rows = torch.stack([torch.randperm(ROWS, device="cuda")[:t] for _ in range(b)]).to(torch.int32)
+ k_rows[:, 5::11] = -1
+ locs = torch.full((b, t * RATIO), -1, device="cuda", dtype=torch.int32)
+ for i in range(b):
+ for p in range(t):
+ r = int(k_rows[i, p])
+ if r >= 0:
+ locs[i, p * RATIO : (p + 1) * RATIO] = torch.arange(r * RATIO, (r + 1) * RATIO, device="cuda", dtype=torch.int32)
+ return pool, deq, q, w, k_rows, locs
+
+
+def _reference(q, w, deq, k_rows, live, positions):
+ """positions: [B, S, N] compressed positions (-1 empty) -> logits [B, S, N]."""
+ b, s, n = positions.shape
+ out = torch.full((b, s, n), float("-inf"), device="cuda")
+ for i in range(b):
+ for j in range(s):
+ for c in range(n):
+ p = int(positions[i, j, c])
+ if p < 0 or p >= int(live[i, j]):
+ continue
+ r = int(k_rows[i, p])
+ if r < 0:
+ continue
+ score = (q[i, j] @ deq[r]).relu() * w[i, j]
+ out[i, j, c] = score.sum()
+ return out
+
+
+def test_full_range_matches_reference_with_causal_live():
+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed
+
+ b, s, t = 2, 5, 300
+ pool, deq, q, w, k_rows, locs = _setup(b, s, t, seed=1)
+ live = torch.tensor([[10, 64, 65, 200, 300], [0, 1, 128, 129, 250]], device="cuda", dtype=torch.int32)
+ out = torch.full((b, s, t), 12345.0, device="cuda") # a sentinel: the dead area must stay untouched
+ got = indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, RATIO, live, out=out)
+ positions = torch.arange(t, device="cuda").view(1, 1, t).expand(b, s, t)
+ want = _reference(q, w, deq, k_rows, live, positions)
+ assert got.shape == (b, s, t)
+ written = positions < live.unsqueeze(-1)
+ touched = positions < (torch.div(live + 63, 64, rounding_mode="floor") * 64).unsqueeze(-1) # whole live tiles
+ assert (got[~touched] == 12345.0).all() and torch.isneginf(got[touched & ~written]).all()
+ assert torch.equal(torch.isinf(got[written]), torch.isinf(want[written]))
+ finite = written & ~torch.isinf(want)
+ torch.testing.assert_close(got[finite], want[finite], atol=2e-2, rtol=2e-2)
+
+
+def test_quantized_queries_preserve_reference_score_rounding():
+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed
+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows
+
+ torch.manual_seed(23)
+ t, s, topk = 4096, 32, 512
+ pool = pack_rows(torch.randn(t, D, device="cuda", dtype=torch.bfloat16), FP4_E8M0_B32)
+ keys = unpack_rows(pool, FP4_E8M0_B32, D)
+ q = torch.randn(s * H, D, device="cuda", dtype=torch.bfloat16)
+ q = unpack_rows(pack_rows(q, FP4_E8M0_B32), FP4_E8M0_B32, D).view(1, s, H, D)
+ w = torch.randn(1, s, H, device="cuda", dtype=torch.bfloat16) * (D * H) ** -0.5
+ locs = torch.arange(t, device="cuda", dtype=torch.int32).view(1, t)
+ live = torch.full((1, s), t, device="cuda", dtype=torch.int32)
+ got = indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, 1, live)
+ reference = torch.einsum("bshd,btd->bsht", q, keys.unsqueeze(0))
+ reference = (reference.relu() * w.unsqueeze(-1)).sum(dim=2).float()
+ torch.testing.assert_close(got, reference, atol=0, rtol=0)
+ # BF16 scores can tie at the cutoff; compare selected scores, not arbitrary tie order.
+ torch.testing.assert_close(got.topk(topk).values, reference.topk(topk).values, atol=0, rtol=0)
+
+
+def test_full_range_worker_grid_covers_wide_histories(monkeypatch):
+ """More live tiles than worker programs: each worker strides over its share."""
+ from freetoken.kernel.triton.dsv41 import indexer as mod
+
+ b, s, t = 1, 2, 4096 # 64 tiles
+ pool, deq, q, w, k_rows, locs = _setup(b, s, t, seed=4)
+ live = torch.tensor([[4096, 4000]], device="cuda", dtype=torch.int32)
+ from types import SimpleNamespace
+
+ properties = torch.cuda.get_device_properties(q.device)
+ with monkeypatch.context() as patch:
+ patch.setattr(torch.cuda, "get_device_properties", lambda _: SimpleNamespace(
+ multi_processor_count=5, major=properties.major, minor=properties.minor,
+ ))
+ got = mod.indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, RATIO, live)
+ positions = torch.arange(t, device="cuda").view(1, 1, t).expand(b, s, t)
+ want = _reference(q, w, deq, k_rows, live, positions)
+ written = positions < live.unsqueeze(-1)
+ assert torch.equal(torch.isinf(got[written]), torch.isinf(want[written]))
+ finite = written & ~torch.isinf(want)
+ torch.testing.assert_close(got[finite], want[finite], atol=2e-2, rtol=2e-2)
+
+
+def test_candidate_pool_matches_reference():
+ from freetoken.kernel.triton.dsv41.indexer import indexer_logits_packed
+
+ b, s, t, nc = 2, 3, 400, 96
+ pool, deq, q, w, k_rows, locs = _setup(b, s, t, seed=2)
+ live = torch.tensor([[400, 350, 33], [1, 400, 200]], device="cuda", dtype=torch.int32)
+ cand = torch.randint(0, t, (b, s, nc), device="cuda", dtype=torch.int32)
+ cand[..., -10:] = -1 # empty tail of the pool
+ got = indexer_logits_packed(q, w, pool, FP4_E8M0_B32, locs, RATIO, live, candidates=cand)
+ want = _reference(q, w, deq, k_rows, live, cand)
+ assert got.shape == (b, s, nc)
+ assert torch.equal(torch.isinf(got), torch.isinf(want))
+ finite = ~torch.isinf(want)
+ torch.testing.assert_close(got[finite], want[finite], atol=2e-2, rtol=2e-2)
diff --git a/tests/kernels/test_dsv41_pack.py b/tests/kernels/test_dsv41_pack.py
new file mode 100644
index 00000000..176889dc
--- /dev/null
+++ b/tests/kernels/test_dsv41_pack.py
@@ -0,0 +1,112 @@
+"""Packed KV row formats (``kvcache/dsv4/v41_row_format.py`` + ``kernel/triton/dsv41/pack.py``) against a torch
+transcription of the reference quantizers in DeepSeek-V4.1's ``inference/kernel.py``: the round-tripped
+value the pool hands back must equal what the reference bakes into its bf16 cache, bit for bit."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+from freetoken.kvcache.dsv4.v41_row_format import BF16, FP4_E4M3_B16, FP4_E8M0_B32, FP8_E8M0_B32, RowFormat
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+FP4_GRID = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])
+
+
+def _round_fp4(x: torch.Tensor) -> torch.Tensor:
+ """Round-to-nearest-even onto the signed e2m1 grid (the ``float4_e2m1fn`` cast)."""
+ a = x.abs()
+ grid = FP4_GRID.to(x.device)
+ # nearest grid point; ties go to the even-mantissa neighbour (0, 1, 2, 4 are even)
+ dist = (a[..., None] - grid).abs()
+ lo = torch.searchsorted(grid, a.reshape(-1).contiguous(), right=True).reshape(a.shape).clamp(1, 7)
+ below, above = grid[lo - 1], grid[lo]
+ mid = (below + above) / 2
+ even_above = torch.isin(above, torch.tensor([0.0, 1.0, 2.0, 4.0], device=x.device))
+ pick_above = (a > mid) | ((a == mid) & even_above)
+ r = torch.where(pick_above, above, below)
+ r = torch.where(a >= 6.0, torch.full_like(r, 6.0), r)
+ del dist
+ return torch.copysign(r, x)
+
+
+def reference_roundtrip(x: torch.Tensor, fmt: RowFormat) -> torch.Tensor:
+ """``act_quant`` / ``fp4_act_quant`` with ``inplace=True``: quant + dequant back to bf16."""
+ m, d = x.shape
+ if fmt is BF16:
+ return x.to(torch.bfloat16)
+ g = x.float().view(m, d // fmt.group, fmt.group)
+ amax = g.abs().amax(dim=-1)
+ if fmt is FP8_E8M0_B32:
+ s = torch.exp2(torch.ceil(torch.log2(amax.clamp_min(1e-4) / 448.0)))
+ q = (g / s[..., None]).clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float()
+ elif fmt is FP4_E8M0_B32:
+ s = torch.exp2(torch.ceil(torch.log2(amax.clamp_min(6.0 * 2.0**-126) / 6.0)))
+ q = _round_fp4((g / s[..., None]).clamp(-6.0, 6.0))
+ elif fmt is FP4_E4M3_B16:
+ s = (amax.clamp_min(6.0 * 2.0**-9) / 6.0).to(torch.float8_e4m3fn).float()
+ q = _round_fp4((g / s[..., None]).clamp(-6.0, 6.0))
+ else:
+ raise AssertionError(fmt)
+ return (q * s[..., None]).view(m, d).to(torch.bfloat16)
+
+
[email protected]("fmt", [BF16, FP8_E8M0_B32, FP4_E4M3_B16, FP4_E8M0_B32], ids=lambda f: f.name)
[email protected]("dim", [128, 512])
+def test_roundtrip_matches_the_reference_quantizer(fmt: RowFormat, dim: int):
+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows
+
+ torch.manual_seed(0)
+ m = 300
+ # mixed magnitudes, exact zeros, a tiny group and a huge one
+ x = torch.randn(m, dim, device="cuda") * torch.exp2(torch.randint(-8, 6, (m, 1), device="cuda").float())
+ x[7] = 0
+ x[11, : fmt.group or 32] *= 1e-6
+ x[13, -32:] = 400.0
+ x = x.to(torch.bfloat16)
+
+ packed = pack_rows(x, fmt)
+ assert packed.shape == (m, fmt.row_bytes(dim)) and packed.dtype == torch.uint8
+ got = unpack_rows(packed, fmt, dim)
+ want = reference_roundtrip(x, fmt)
+ assert torch.equal(got, want), f"{fmt.name}: {(got.float() - want.float()).abs().max().item()}"
+
+
+def test_scatter_into_pool_rows_and_gather_back():
+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows
+
+ torch.manual_seed(1)
+ fmt, dim, rows = FP4_E4M3_B16, 512, 64
+ pool = torch.zeros((rows, fmt.row_bytes(dim)), dtype=torch.uint8, device="cuda")
+ x = torch.randn(5, dim, device="cuda", dtype=torch.bfloat16)
+ ids = torch.tensor([3, 60, 0, 17, 42], device="cuda", dtype=torch.int64)
+ assert pack_rows(x, fmt, pool=pool, row_ids=ids) is pool
+ got = unpack_rows(pool, fmt, dim, row_ids=ids.to(torch.int32))
+ assert torch.equal(got, reference_roundtrip(x, fmt))
+ untouched = torch.ones(rows, dtype=torch.bool, device="cuda")
+ untouched[ids] = False
+ assert int(pool[untouched].sum()) == 0
+
+
+def test_e4m3_encode_helper_matches_torch_bits():
+ """``e4m3_f32_to_u8`` (the pre-sm_89 scale encoder) on every finite e4m3 code."""
+ import triton
+ import triton.language as tl
+
+ from freetoken.kernel.triton.e4m3_compat import e4m3_f32_to_u8
+
+ @triton.jit
+ def _enc(x_ptr, o_ptr, N: tl.constexpr):
+ offs = tl.arange(0, N)
+ tl.store(o_ptr + offs, e4m3_f32_to_u8(tl.load(x_ptr + offs)))
+
+ codes = torch.arange(256, dtype=torch.uint8, device="cuda")
+ finite = (codes & 0x7F) != 0x7F # skip the NaN codes
+ vals = codes.view(torch.float8_e4m3fn).float()
+ vals = torch.where(finite, vals, torch.zeros_like(vals))
+ out = torch.empty(256, dtype=torch.uint8, device="cuda")
+ _enc[(1,)](vals, out, N=256)
+ want = torch.where(finite, codes, torch.zeros_like(codes))
+ # -0.0 encodes to 0x80 in torch and 0x80 here; +0.0 -> 0x00
+ assert torch.equal(out, want)
diff --git a/tests/kernels/test_dsv41_sparse_attn.py b/tests/kernels/test_dsv41_sparse_attn.py
new file mode 100644
index 00000000..8a4f5f2a
--- /dev/null
+++ b/tests/kernels/test_dsv41_sparse_attn.py
@@ -0,0 +1,107 @@
+"""DSV41 sparse attention over packed pools against a torch reference on the DEQUANTIZED pools.
+
+``m == 1`` takes the split-k decode path, ``m > 1`` the single-program prefill one. Tolerance follows
+the dsv4 sparse-attention tests (5e-2 on fp32-cast bf16): the online-softmax accumulation order differs
+from the reference and split-k merges through log-sum-exp on top of that."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+from freetoken.kvcache.dsv4.v41_row_format import BF16, FP4_E4M3_B16, FP8_E8M0_B32
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+D, H = 128, 8
+N_WINDOW = 128 # a multiple of BLOCK_T
+N_WIN_SLOTS, N_CMP = 512, 1024
+TOL = dict(atol=5e-2, rtol=5e-2)
+
+
+def _reference(q, win, cmp, sink, idx, n_window, scale, counts):
+ """Per (request, query, head) softmax over the live columns; ``sink`` is a null key."""
+ b, m, h, d = q.shape
+ out = torch.zeros_like(q, dtype=torch.float32)
+ for i in range(b):
+ for j in range(m):
+ width = idx.shape[-1] if counts is None else n_window + int(counts[i, j])
+ kv = []
+ for c in range(min(width, idx.shape[-1])):
+ slot = int(idx[i, j, c])
+ if slot >= 0:
+ kv.append((win if c < n_window else cmp)[slot].float())
+ if not kv:
+ continue
+ kv_t = torch.stack(kv)
+ logits = q[i, j].float() @ kv_t.T * scale
+ mx = torch.maximum(logits.max(dim=-1).values, sink.float())
+ probs = torch.exp(logits - mx[:, None])
+ denom = probs.sum(dim=-1) + torch.exp(sink.float() - mx)
+ out[i, j] = (probs @ kv_t) / denom[:, None]
+ return out
+
+
+def _pools(win_fmt, cmp_fmt, seed):
+ from freetoken.kernel.triton.dsv41.pack import pack_rows, unpack_rows
+
+ torch.manual_seed(seed)
+ win_vals = torch.randn(N_WIN_SLOTS, D, device="cuda", dtype=torch.bfloat16)
+ cmp_vals = torch.randn(N_CMP, D, device="cuda", dtype=torch.bfloat16)
+ win_pool = pack_rows(win_vals, win_fmt)
+ cmp_pool = pack_rows(cmp_vals, cmp_fmt)
+ return win_pool, cmp_pool, unpack_rows(win_pool, win_fmt, D), unpack_rows(cmp_pool, cmp_fmt, D)
+
+
+def _topk(b, m, n_cmp_cols, seed, ring_fill=None):
+ torch.manual_seed(seed)
+ win = torch.randint(0, N_WIN_SLOTS, (b, m, N_WINDOW), device="cuda", dtype=torch.int32)
+ if ring_fill is not None: # a ring still filling: trailing window columns empty
+ win[..., ring_fill:] = -1
+ cmp = torch.randint(0, N_CMP, (b, m, n_cmp_cols), device="cuda", dtype=torch.int32)
+ cmp[..., ::7] = -1 # scattered unreachable picks
+ return torch.cat([win, cmp], dim=-1)
+
+
[email protected]("m", [1, 3])
[email protected]("fmts", [(FP8_E8M0_B32, FP4_E4M3_B16), (BF16, BF16)], ids=["fp8+fp4", "bf16"])
+def test_matches_reference_on_dequantized_pools(m, fmts):
+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed
+
+ win_fmt, cmp_fmt = fmts
+ b, n_cmp_cols = 2, 256
+ win_pool, cmp_pool, win_deq, cmp_deq = _pools(win_fmt, cmp_fmt, seed=1)
+ q = torch.randn(b, m, H, D, device="cuda", dtype=torch.bfloat16)
+ sink = torch.randn(H, device="cuda") * 0.5
+ idx = _topk(b, m, n_cmp_cols, seed=2, ring_fill=100)
+ scale = D**-0.5
+
+ got = sparse_attn_packed(q, win_pool, win_fmt, cmp_pool, cmp_fmt, sink, idx, N_WINDOW, scale)
+ want = _reference(q, win_deq, cmp_deq, sink, idx, N_WINDOW, scale, None)
+ torch.testing.assert_close(got.float(), want, **TOL)
+
+
+def test_cmp_counts_bound_the_visited_columns():
+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed
+
+ b, m, n_cmp_cols = 3, 1, 512
+ win_pool, cmp_pool, win_deq, cmp_deq = _pools(FP8_E8M0_B32, FP4_E4M3_B16, seed=3)
+ q = torch.randn(b, m, H, D, device="cuda", dtype=torch.bfloat16)
+ sink = torch.zeros(H, device="cuda")
+ idx = _topk(b, m, n_cmp_cols, seed=4)
+ counts = torch.tensor([[0], [37], [512]], device="cuda", dtype=torch.int32)
+ scale = D**-0.5
+
+ got = sparse_attn_packed(q, win_pool, FP8_E8M0_B32, cmp_pool, FP4_E4M3_B16, sink, idx, N_WINDOW, scale, cmp_counts=counts)
+ want = _reference(q, win_deq, cmp_deq, sink, idx, N_WINDOW, scale, counts)
+ torch.testing.assert_close(got.float(), want, **TOL)
+
+
+def test_all_masked_query_yields_zeros():
+ from freetoken.kernel.triton.dsv41.sparse_attn import sparse_attn_packed
+
+ win_pool, cmp_pool, _, _ = _pools(FP8_E8M0_B32, FP4_E4M3_B16, seed=5)
+ q = torch.randn(1, 2, H, D, device="cuda", dtype=torch.bfloat16)
+ idx = torch.full((1, 2, N_WINDOW + 64), -1, device="cuda", dtype=torch.int32)
+ out = sparse_attn_packed(q, win_pool, FP8_E8M0_B32, cmp_pool, FP4_E4M3_B16, torch.zeros(H, device="cuda"), idx, N_WINDOW, 0.1)
+ assert torch.equal(out, torch.zeros_like(out))
diff --git a/tests/kernels/test_dsv41_topk.py b/tests/kernels/test_dsv41_topk.py
new file mode 100644
index 00000000..e7fc66f8
--- /dev/null
+++ b/tests/kernels/test_dsv41_topk.py
@@ -0,0 +1,159 @@
+"""dsv41_topk / dsv41_candidate_blocks against an independent torch reference with the documented tie
+rule, over the contract's edge cases: empty and short histories, unequal live counts in one batch,
+-inf holes, widths crossing the slice and level boundaries, exact ties, newest-block inclusion,
+the sorted-valid-prefix / -1-tail output layout, and CUDA-graph replay across changing live lengths
+(stale workspace reads would surface as wrong picks)."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+
+def ref_topk(scores: torch.Tensor, live: torch.Tensor, k: int) -> torch.Tensor:
+ """(score desc, column asc) over the live, finite columns; ascending columns, -1 padded."""
+ R, T = scores.shape
+ out = torch.full((R, k), -1, dtype=torch.int32)
+ for r in range(R):
+ n = int(live[r])
+ s = scores[r, :n].double().cpu()
+ cols = torch.arange(n)
+ alive = torch.isfinite(s) | (s == float("inf"))
+ cand = [(float(-s[c]), int(c)) for c in cols[alive]]
+ cand.sort()
+ picked = sorted(c for _, c in cand[:k])
+ out[r, : len(picked)] = torch.tensor(picked, dtype=torch.int32)
+ return out.to(scores.device)
+
+
+def ref_candidates(scores, live, topk_blocks, block_size):
+ R, T = scores.shape
+ nb = -(-T // block_size)
+ out = torch.full((R, topk_blocks * block_size), -1, dtype=torch.int32)
+ for r in range(R):
+ n = int(live[r])
+ if n == 0:
+ continue
+ nbl = -(-n // block_size)
+ s = torch.nn.functional.pad(scores[r, :n].float().cpu(), (0, nbl * block_size - n), value=float("-inf")).view(nbl, block_size).amax(-1)
+ s[nbl - 1] = float("inf")
+ cand = sorted((float(-s[b]), b) for b in range(nbl) if s[b] > float("-inf"))
+ keep = sorted(b for _, b in cand[:topk_blocks])
+ pos = [p for b in keep for p in range(b * block_size, (b + 1) * block_size) if p < n]
+ out[r, : len(pos)] = torch.tensor(pos, dtype=torch.int32)
+ return out.to(scores.device)
+
+
+def _scores(R, T, seed=0, ties=False):
+ g = torch.Generator(device="cpu").manual_seed(seed)
+ s = torch.randn(R, T, generator=g)
+ if ties:
+ s = torch.randint(0, 5, (R, T), generator=g).float() # heavy exact ties
+ return s.cuda()
+
+
[email protected]("T,k", [(64, 8), (4096, 512), (4097, 512), (8192, 512), (100_000, 512), (262_144, 512), (1_048_576, 512), (131_072, 2048)])
+def test_topk_matches_reference_across_widths(T, k):
+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk
+
+ R = 3
+ scores = _scores(R, T, seed=T)
+ live = torch.tensor([0, min(T, 5000), T], dtype=torch.int32, device="cuda")[:R]
+ # holes inside the live range
+ scores[2, ::7] = float("-inf")
+ got = dsv41_topk(scores, live, k)
+ want = ref_topk(scores, live, k)
+ assert torch.equal(got, want), (got[:, :8], want[:, :8])
+
+
+def test_topk_short_and_unequal_rows_and_exact_ties():
+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk
+
+ T, k = 20_000, 512
+ scores = _scores(5, T, seed=3, ties=True)
+ live = torch.tensor([0, 1, 300, 4096, 20_000], dtype=torch.int32, device="cuda")
+ got = dsv41_topk(scores, live, k)
+ want = ref_topk(scores, live, k)
+ assert torch.equal(got, want)
+ # the layout contract: a sorted valid prefix, then -1
+ for r in range(5):
+ row = got[r].tolist()
+ n = sum(1 for v in row if v >= 0)
+ assert row[n:] == [-1] * (k - n) and row[:n] == sorted(row[:n]) and n == min(int(live[r]), k)
+
+
+def test_topk_all_dead_row_and_fewer_finite_than_k():
+ from freetoken.kernel.triton.dsv41.topk import dsv41_topk
+
+ scores = torch.full((2, 10_000), float("-inf"), device="cuda")
+ scores[1, [10, 4095, 4096, 9000]] = torch.tensor([1.0, 2.0, 3.0, 0.5], device="cuda")
+ live = torch.tensor([10_000, 9001], dtype=torch.int32, device="cuda")
+ got = dsv41_topk(scores, live, 8)
+ assert (got[0] == -1).all()
+ assert got[1].tolist() == [10, 4095, 4096, 9000, -1, -1, -1, -1]
+
+
[email protected]("T", [4000, 200_000])
+def test_candidate_blocks_match_reference_and_pin_the_newest_block(T):
+ from freetoken.kernel.triton.dsv41.topk import dsv41_candidate_blocks
+
+ R, KB, BS = 4, 2048, 8
+ scores = _scores(R, T, seed=T + 1)
+ scores[0, :] = -1000.0 # the newest block is kept even when every score is poor
+ live = torch.tensor([13, 8, min(T, 16_000), T], dtype=torch.int32, device="cuda")
+ got = dsv41_candidate_blocks(scores, live, KB, BS)
+ want = ref_candidates(scores, live, KB, BS)
+ assert got.shape == (R, KB * BS)
+ assert torch.equal(got, want)
+ # short histories: every live position is a candidate (the implicit fast path)
+ assert got[0].tolist()[:13] == list(range(13)) and (got[0, 13:] == -1).all()
+ assert got[1].tolist()[:8] == list(range(8)) and (got[1, 8:] == -1).all()
+
+
+def test_topk_under_graph_replay_across_live_lengths():
+ """Capture once at the staged width, replay with live lengths that move up and down: every replay
+ must equal the reference (a stale candidate slot or live count from a previous replay would not)."""
+ from freetoken.kernel.triton.dsv41.topk import dsv41_candidate_blocks, dsv41_topk
+
+ T, k, R = 300_000, 512, 2
+ scores = torch.empty((R, T), device="cuda")
+ live = torch.empty((R,), dtype=torch.int32, device="cuda")
+ out = torch.empty((R, k), dtype=torch.int32, device="cuda")
+ cand = torch.empty((R, 2048 * 8), dtype=torch.int32, device="cuda")
+ stream = torch.cuda.Stream()
+ with torch.cuda.stream(stream):
+ scores.copy_(_scores(R, T, seed=9))
+ live.fill_(T)
+ dsv41_topk(scores, live, k, out) # warm-up compiles
+ cand.copy_(dsv41_candidate_blocks(scores, live, 2048, 8))
+ torch.cuda.synchronize()
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(graph, stream=stream):
+ dsv41_topk(scores, live, k, out)
+ cand.copy_(dsv41_candidate_blocks(scores, live, 2048, 8))
+ torch.cuda.synchronize()
+ for step, (l0, l1) in enumerate([(T, 7), (4096, 4097), (0, 1), (250_000, 12_000), (3, T), (16_384, 16_385)]):
+ scores.copy_(_scores(R, T, seed=100 + step))
+ live.copy_(torch.tensor([l0, l1], dtype=torch.int32))
+ graph.replay()
+ torch.cuda.synchronize()
+ assert torch.equal(out, ref_topk(scores, live, k)), (step, l0, l1)
+ assert torch.equal(cand, ref_candidates(scores, live, 2048, 8)), (step, l0, l1)
+
+
+def test_topk_plan_terminates_or_rejects():
+ """Every accepted k converges (each level at least halves the row); k past CHUNK // 2 -- which
+ would stall the planner (k == CHUNK never shrinks, 3072 stalls at 6144) -- is rejected up front."""
+ from freetoken.kernel.triton.dsv41.topk import CHUNK, FINAL_MAX, MAX_K, topk_plan
+
+ for k in (1, 8, 512, 2048, MAX_K):
+ for columns in (1, 4096, 4097, 1 << 20, (1 << 20) + 3):
+ levels = topk_plan(columns, k)
+ widths = [w for w, _ in levels] + [(levels[-1][1] * k) if levels else columns]
+ assert all(a > b for a, b in zip(widths, widths[1:])) and widths[-1] <= FINAL_MAX
+ assert len(levels) <= 12
+ for k in (MAX_K + 1, 2049, 3072, CHUNK, 0, -1):
+ with pytest.raises(ValueError):
+ topk_plan(1 << 20, k)
diff --git a/tests/kernels/test_dsv4_fp8_block_linear.py b/tests/kernels/test_dsv4_fp8_block_linear.py
new file mode 100644
index 00000000..2194a65e
--- /dev/null
+++ b/tests/kernels/test_dsv4_fp8_block_linear.py
@@ -0,0 +1,101 @@
+"""``kernel/triton/dsv4/fp8_linear.block_fp8_linear`` against a torch transcription of the reference
+``act_quant`` + ``fp8_gemm`` (ue8m0 activation scales, ``block x block`` e8m0 weight scales), for the
+DeepSeek-V4 block (128) and the DeepSeek-V4.1 block (32), on the GEMV (M=1) and GEMM (M>1) paths."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+FP8 = torch.float8_e4m3fn
+E8M0 = torch.float8_e8m0fnu
+
+
+def _pow2_ceil_scale(amax: torch.Tensor, qmax: float) -> torch.Tensor:
+ """``2 ** ceil(log2(amax / qmax))`` -- the reference ``fast_round_scale``."""
+ return torch.exp2(torch.ceil(torch.log2(amax / qmax)))
+
+
+def quantize_weight(w: torch.Tensor, block: int) -> tuple[torch.Tensor, torch.Tensor]:
+ """Per ``block x block`` ue8m0-scaled e4m3 weight, scale as e8m0 codes."""
+ n, k = w.shape
+ blocks = w.float().view(n // block, block, k // block, block).permute(0, 2, 1, 3)
+ amax = blocks.abs().amax(dim=(-1, -2)).clamp_min(1e-4)
+ scale = _pow2_ceil_scale(amax, 448.0) # [n//block, k//block]
+ q = (blocks / scale[..., None, None]).clamp(-448.0, 448.0).to(FP8)
+ w_fp8 = q.permute(0, 2, 1, 3).reshape(n, k)
+ return w_fp8, scale.to(E8M0)
+
+
+def act_quant_reference(x: torch.Tensor, block: int) -> tuple[torch.Tensor, torch.Tensor]:
+ m, k = x.shape
+ groups = x.float().view(m, k // block, block)
+ amax = groups.abs().amax(dim=-1).clamp_min(1e-4)
+ scale = _pow2_ceil_scale(amax, 448.0)
+ q = (groups / scale[..., None]).clamp(-448.0, 448.0).to(FP8)
+ return q.view(m, k), scale
+
+
+def linear_reference(x: torch.Tensor, w_fp8: torch.Tensor, w_scale: torch.Tensor, block: int) -> torch.Tensor:
+ m, k = x.shape
+ n = w_fp8.shape[0]
+ a_fp8, a_scale = act_quant_reference(x, block)
+ a = a_fp8.float().view(m, k // block, block)
+ w = w_fp8.float().view(n // block, block, k // block, block)
+ ws = w_scale.float() # [n//block, k//block]
+ out = torch.zeros(m, n, dtype=torch.float32, device=x.device)
+ for kb in range(k // block):
+ wk = w[:, :, kb, :].reshape(n, block) # [n, block]
+ partial = a[:, kb, :] @ wk.t() # [m, n]
+ out += partial * a_scale[:, kb, None] * ws[:, kb].repeat_interleave(block)[None, :]
+ return out
+
+
[email protected]("block", [32, 128])
[email protected]("m", [1, 2, 5, 8, 9, 64])
+def test_block_fp8_linear_matches_reference(block: int, m: int):
+ from freetoken.kernel.triton.dsv4.fp8_linear import block_fp8_linear
+
+ torch.manual_seed(0)
+ # K // block = 5 slabs exercises the GEMV split that must divide the slab count; N spans two GEMM N-tiles
+ k, n = 5 * block, 256 if block == 128 else 224
+ x = torch.randn(m, k, device="cuda", dtype=torch.bfloat16) * 3
+ w_fp8, w_scale = quantize_weight(torch.randn(n, k, device="cuda", dtype=torch.bfloat16), block)
+
+ got = block_fp8_linear(x, w_fp8, w_scale, block=block).float()
+ want = linear_reference(x, w_fp8, w_scale, block)
+ # bf16 output; the fp32 reduction order inside the MMA may differ from the slab loop
+ torch.testing.assert_close(got, want, rtol=2e-2, atol=2e-2 * want.abs().max().item())
+
+
+def test_block_argument_must_match_the_scale_shape():
+ from freetoken.kernel.triton.dsv4.fp8_linear import block_fp8_linear
+
+ x = torch.randn(4, 256, device="cuda", dtype=torch.bfloat16)
+ w_fp8, w_scale = quantize_weight(torch.randn(256, 256, device="cuda", dtype=torch.bfloat16), 32)
+ with pytest.raises(AssertionError):
+ block_fp8_linear(x, w_fp8, w_scale, block=128)
+
+
[email protected]("block", [32, 128])
+def test_grouped_w8a16_gemv_matches_the_dequantized_einsum(block):
+ """``wo_a`` at decode: the fp8 payload dequantized in-kernel against a bf16 activation must equal
+ the reference's bf16 einsum over the dequantized weight (fp32 accumulation, order aside)."""
+ from freetoken.kernel.triton.dsv4.fp8_linear import dequant_block_fp8, grouped_w8a16_gemv
+
+ G, r, d = 4, 256, 1024
+ torch.manual_seed(block)
+ w = torch.randn(G * r, d, device="cuda") * 0.05
+ w_fp8, scale = quantize_weight(w, block)
+ x = torch.randn(G, d, device="cuda", dtype=torch.bfloat16)
+ deq = dequant_block_fp8(w_fp8, scale, block=block)
+ ref_deq = (w_fp8.float() * torch.exp2(scale.view(torch.uint8).float() - 127.0).repeat_interleave(block, 0).repeat_interleave(block, 1)).to(torch.bfloat16)
+ assert torch.equal(deq, ref_deq)
+ for t in (1, 3):
+ xt = torch.randn(t, G, d, device="cuda", dtype=torch.bfloat16) if t > 1 else x.unsqueeze(0)
+ want = torch.einsum("tgd,grd->tgr", xt, ref_deq.view(G, r, d)).flatten(1)
+ got = grouped_w8a16_gemv(xt, w_fp8, scale, block=block)
+ assert got.shape == (t, G * r) and got.dtype == torch.bfloat16
+ torch.testing.assert_close(got.float(), want.float(), atol=2e-2, rtol=2e-2)
diff --git a/tests/kernels/test_dsv4_router_select.py b/tests/kernels/test_dsv4_router_select.py
new file mode 100644
index 00000000..9829ed03
--- /dev/null
+++ b/tests/kernels/test_dsv4_router_select.py
@@ -0,0 +1,52 @@
+"""Fused gate selection (``kernel/triton/dsv4/router``) against the torch expression it replaces, for
+every score function, with and without top-k weight normalization, at decode and prefill widths."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+import torch.nn.functional as F
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+
+def reference(scores, bias, topk, score_func, normalize, route_scale):
+ if score_func == "softmax":
+ s = scores.softmax(dim=-1)
+ elif score_func == "sigmoid":
+ s = scores.sigmoid()
+ else:
+ s = F.softplus(scores).sqrt()
+ ids = (s + bias).topk(topk, dim=-1)[1]
+ w = s.gather(1, ids)
+ if normalize and topk > 1:
+ w = w / (w.sum(dim=-1, keepdim=True) + 1e-20)
+ return w * route_scale, ids
+
+
[email protected]("score_func", ["sqrtsoftplus", "sigmoid", "softmax"])
[email protected]("normalize", [True, False])
[email protected]("T", [1, 3, 300])
+def test_router_select_matches_torch(score_func, normalize, T):
+ from freetoken.kernel.triton.dsv4.router import router_select
+
+ g = torch.Generator(device="cpu").manual_seed(T)
+ E, K = 384, 6
+ scores = (torch.randn(T, E, generator=g) * 3).cuda()
+ bias = (torch.randn(E, generator=g) * 0.1).cuda()
+ w, ids = router_select(scores, bias, K, score_func=score_func, normalize=normalize, route_scale=2.5)
+ w_ref, ids_ref = reference(scores, bias, K, score_func, normalize, 2.5)
+ assert torch.equal(ids.sort(dim=-1).values, ids_ref.to(torch.int32).sort(dim=-1).values)
+ # same expert set; compare the weights per expert (selection order may differ only on exact ties)
+ order = ids.argsort(dim=-1)
+ order_ref = ids_ref.argsort(dim=-1)
+ torch.testing.assert_close(w.gather(1, order), w_ref.gather(1, order_ref), atol=1e-6, rtol=1e-5)
+
+
+def test_router_select_breaks_ties_toward_the_lowest_expert():
+ from freetoken.kernel.triton.dsv4.router import router_select
+
+ scores = torch.zeros(1, 384, device="cuda") # every expert ties
+ bias = torch.zeros(384, device="cuda")
+ _, ids = router_select(scores, bias, 6, score_func="sigmoid", normalize=True, route_scale=1.0)
+ assert ids.tolist() == [[0, 1, 2, 3, 4, 5]]
diff --git a/tests/kvcache/test_cache_unit_bytes.py b/tests/kvcache/test_cache_unit_bytes.py
index 496622f9..cce5df62 100644
--- a/tests/kvcache/test_cache_unit_bytes.py
+++ b/tests/kvcache/test_cache_unit_bytes.py
@@ -16,7 +16,7 @@ from freetoken.kvcache.cache_status import (
compute_cache_status_meta,
compute_cache_unit_bytes,
)
-from freetoken.kvcache.dsv4_cost_model import _dsv4_window_floor_pages
+from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_window_floor_pages
def _init_tp() -> None:
@@ -204,7 +204,7 @@ def test_floors_dsv4_reports_real_window_floor():
dsv4_args=SimpleNamespace(window_size=P), has_swa_attention=False
),
)
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
eng = SimpleNamespace(config=cfg, moe_offload_cache=None, linear_state_pool=None)
eng.kv_cache = object.__new__(DSV4PagedKVCache) # min_kv_tokens is a classmethod
diff --git a/tests/kvcache/test_dsv41_pool.py b/tests/kvcache/test_dsv41_pool.py
new file mode 100644
index 00000000..f41df851
--- /dev/null
+++ b/tests/kvcache/test_dsv41_pool.py
@@ -0,0 +1,205 @@
+"""DSV41 paged KV pool + cost model (CPU, no model): args validation, per-source tiers, sizing /
+byte accounting, the window free-list duck-type, full-loc translation; packed writes on CUDA."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+from freetoken.kvcache.dsv4.v41_cost_model import (
+ dsv41_cache_per_page,
+ dsv41_kv_unit_bytes,
+ dsv41_pool_bytes,
+ dsv41_pool_sizes,
+ dsv41_solve_num_pages,
+ dsv41_window_unit_bytes,
+)
+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache
+from freetoken.kvcache.dsv4.v41_row_format import FP4_E4M3_B16, FP4_E8M0_B32, FP8_E8M0_B32
+from freetoken.models.deepseek_v41.args import DeepseekV41Args
+
+DEVICE = torch.device("cpu")
+P = 128
+# a 10-layer miniature of V4.1's layout: 2 window-only, ratio-2 encoder (source 2, source 5), ratio-1 decoder (source 7)
+RATIOS = (0, 0, 2, 2, 2, 2, 2, 1, 1, 1)
+SOURCES = (2, 5, 7)
+
+
+def _args(**over) -> DeepseekV41Args:
+ base = dict(n_layers=10, head_dim=512, index_head_dim=128, window_size=P, compress_ratios=RATIOS,
+ kv_source_layers=SOURCES, swa_decoder_replay="exact")
+ base.update(over)
+ return DeepseekV41Args(**base)
+
+
+def _pool(num_pages=8, swa_ratio=0.5, n_scratch=1, **over):
+ args = _args(**over)
+ sizes = dsv41_pool_sizes(num_pages, args, swa_ratio, P)
+ return DSV41PagedKVCache(sizes, args, DEVICE, n_scratch=n_scratch), sizes, args
+
+
+def test_window_control_uses_pool_units_and_restores_concrete_capacity():
+ from types import SimpleNamespace
+
+ from freetoken.attention import AttnType
+ from freetoken.kvcache.cache_status import (
+ _supports_swa_ratio, compute_cache_floors, compute_cache_pools,
+ )
+ from freetoken.kvcache.dsv4.v41_cost_model import _dsv41_pool_sizes
+ from freetoken.scheduler.scheduler import Scheduler
+ from freetoken.server.api_server import CacheRebuildRequest, _resolve_num_swa_pages, cache_geometry
+ from freetoken.server.stats import _swa_page_size
+
+ pool, _, args = _pool(num_pages=64)
+ config = SimpleNamespace(
+ page_size=P, max_running_req=1, max_seq_len=1024, cache_type="swa_radix",
+ swa_full_tokens_ratio=0.5, swa_num_pages_override=None,
+ model_config=SimpleNamespace(
+ dsv4_args=None, dsv41_args=_args(), has_swa_attention=False,
+ kv_cache_group_specs=lambda: [SimpleNamespace(attn_type=AttnType.DSV41)],
+ ),
+ )
+ engine = SimpleNamespace(
+ config=config, kv_cache=pool, num_pages=63,
+ moe_offload_cache=None, linear_state_pool=None,
+ )
+ assert _supports_swa_ratio(config)
+ assert compute_cache_pools(engine)["swa_page_size"] == P
+ assert compute_cache_pools(engine)["num_swa_pages"] == 31
+ floor = _dsv41_pool_sizes(config, 64, num_swa_pages=1).n_win_pages - 1
+ assert compute_cache_floors(engine)["swa_tokens"] == floor * P
+ prior = Scheduler._current_cache_geometry(SimpleNamespace(engine=engine, config=config))
+ assert prior["num_swa_pages"] == 31
+ state = SimpleNamespace(config=config, cache_pools=compute_cache_pools(engine))
+ req = CacheRebuildRequest(num_pages=64, swa_full_tokens_ratio=0.5)
+ assert _resolve_num_swa_pages(state, req) == 32
+ assert _swa_page_size(config) == P
+ state.stats = SimpleNamespace(kv_total_pages=63, mamba_total_slots=0)
+ state.last_rebuild = {"num_pages": 64, "num_swa_pages": 32}
+ assert cache_geometry(state)["swa_full_tokens_ratio"] == 0.5
+ pool._init_paged_state(1, True)
+ pool.rebuild_from_config(config, 63, num_swa_pages=24)
+ assert pool.window_pages == 24
+ pool.rebuild_from_config(config, prior["num_pages"], num_swa_pages=prior["num_swa_pages"])
+ assert pool.window_pages == 31
+
+
+def test_pool_reads_sources_and_rings_from_args():
+ pool, _, _ = _pool()
+ assert [pool.source_of(l) for l in range(2, 10)] == [2, 2, 2, 5, 5, 7, 7, 7]
+ assert set(pool.state_ring) == {2, 5} and pool.state_ring[2].ring_size == 2
+ with pytest.raises(ValueError): # the window is not a multiple of a compress ratio
+ _pool(window_size=P - 1)
+ with pytest.raises(ValueError): # an indexer key the fp4 row format cannot hold
+ _pool(index_head_dim=48)
+
+
[email protected]("replay", ["bounded", "exact"])
+def test_v41_flash_row_bytes_and_private_window_layers(replay):
+ """The tech report's headline: 3 ratio-2 encoder sources + 1 ratio-1 decoder source = 890 B/token."""
+ ratios = (0, 0) + (2,) * 18 + (1,) * 20
+ args = DeepseekV41Args(n_layers=40, head_dim=512, index_head_dim=128, window_size=128, compress_ratios=ratios,
+ kv_source_layers=(2, 8, 14, 20), swa_decoder_replay=replay)
+ pool = DSV41PagedKVCache(dsv41_pool_sizes(4, args, 1.0, 128), args, DEVICE)
+ win, main, idx = pool.window_pool[0].shape[1], pool.main_pool[2].shape[1], pool.idx_pool[2].shape[1]
+ assert (win, main, idx) == (528, 288, 68)
+ assert pool.state_ring[2].buffer.shape[1] * 4 == 4096
+ assert sum((main + idx) // pool.ratio_of(s) for s in args.backbone_kv_sources) == 890
+ assert dsv41_kv_unit_bytes(args, 128) == 890 + 8 # + the int64 full->window map slot
+ assert pool.private_window_layer_ids == (tuple(range(20, 40)) if replay == "bounded" else ())
+
+
+def test_pool_tiers_per_source_and_aliasing():
+ pool, sizes, args = _pool()
+ assert len(pool.window_pool) == 10 and pool.window_pool[0].shape == (sizes.n_win_slots, 528)
+ assert set(pool.main_pool) == set(SOURCES) == set(pool.idx_pool)
+ assert set(pool.state_ring) == {2, 5} # the ratio-1 decoder source carries no partial group
+ assert pool.main_pool[2].shape == (sizes.full_token // 2 + 1, 288)
+ assert pool.main_pool[7].shape == (sizes.full_token + 1, 288)
+ assert pool.idx_pool[5].shape == (sizes.full_token // 2 + 1, 68)
+ assert pool.main_pool_of(4) is pool.main_pool[2] and pool.idx_pool_of(9) is pool.idx_pool[7]
+ assert pool.scratch_base[2] == sizes.full_token // 2
+ with pytest.raises(AssertionError):
+ pool.main_pool_of(0)
+ ring = pool.state_ring[2]
+ assert ring.ring_size == 2 and ring.item_size == 512 and ring.buffer.shape == (sizes.state_slots[2] + 1, 1024)
+ assert torch.all(ring.buffer[-1, :512] == 0) and torch.all(torch.isneginf(ring.buffer[-1, 512:]))
+
+
+def test_pool_bytes_match_allocation_and_solver_respects_budget():
+ pool, sizes, args = _pool(num_pages=16, swa_ratio=0.25)
+ assert pool.total_bytes() == dsv41_pool_bytes(sizes, args, n_scratch=1)
+ assert dsv41_cache_per_page(args, 0.25, P) > 0
+ assert dsv41_window_unit_bytes(args, P) == -(-(10 * P * 528 + 2 * 2 * 4096) // P)
+ budget = 64 << 20
+ solved = dsv41_solve_num_pages(budget, args, 0.25, floor_win_pages=4, P=P, n_scratch=3)
+ assert dsv41_pool_bytes(solved, args, 3) <= budget
+ # one more page, sized the way the solver sizes (window = max(floor, ceil(ratio * pages))), overflows
+ more = solved.full_token // P + 1
+ bigger = dsv41_pool_sizes(more, args, 0.25, P, n_win_pages=max(4, (round(0.25 * more * P) + P - 1) // P))
+ assert dsv41_pool_bytes(bigger, args, 3) > budget
+ assert solved.n_win_pages >= 4
+ with pytest.raises(ValueError):
+ dsv41_solve_num_pages(1 << 10, args, 0.25, floor_win_pages=4, P=P)
+
+
+def test_translation_state_loc_and_cmp_rows():
+ pool, sizes, args = _pool(num_pages=16)
+ pool.bind_window_pages(full_page_base=0, window_page_base=2 * P)
+ pool.bind_window_pages(full_page_base=3 * P, window_page_base=0)
+ assert pool.translate_full_to_window(torch.tensor([0, 1, 127])).tolist() == [2 * P, 2 * P + 1, 2 * P + 127]
+ assert pool.translate_full_to_window(torch.tensor([3 * P + 5])).item() == 5
+ assert pool.translate_full_to_window(torch.tensor([P + 7, -1])).tolist() == [-1, -1]
+ full = torch.tensor([0, 1, 2, 127, 128, 4 * P - 1])
+ assert pool.cmp_rows(full, 2).tolist() == [0, 0, 1, 63, 64, 2 * P - 1]
+ assert pool.cmp_rows(full, 1).tolist() == full.tolist()
+ assert pool.cmp_rows(torch.tensor([-1]), 2).item() < 0
+ top = pool.cmp_rows(torch.tensor([sizes.full_token - 1]), 2).item()
+ assert top < pool.scratch_base[2]
+ ws = pool.translate_full_to_window(torch.tensor([127, 3 * P]))
+ assert DSV41PagedKVCache.state_loc(ws, 2, P).tolist() == [2 * 2 + 1, 0]
+ assert DSV41PagedKVCache.state_loc(torch.tensor([-1]), 2, P).item() == -1
+
+
+def _expand(bases):
+ return (torch.tensor(bases, dtype=torch.int64)[:, None] + torch.arange(P)).flatten()
+
+
+def test_swa_duck_type_alloc_free_and_dummy():
+ pool, sizes, _ = _pool(num_pages=8)
+ pool._init_paged_state(max_running_req=2, radix=True)
+ cap = sizes.n_win_slots - P
+ assert pool.swa_available_size() == cap and pool.swa_num_tokens - 1 == cap
+ assert pool.sliding_window_size == P and pool.prefill_chunk_budget >= P
+ pool.alloc_swa(_expand([0, 2 * P]))
+ assert pool.swa_available_size() == cap - 2 * P
+ ws = pool.translate_loc_from_full_to_swa(torch.arange(2 * P, 3 * P))
+ assert (ws >= 0).all() and int(ws[0]) % P == 0 and torch.equal(ws - ws[0], torch.arange(P))
+ pool.free_swa(_expand([0]))
+ pool.free_swa(_expand([0])) # idempotent
+ assert pool.swa_available_size() == cap - P
+ with pytest.raises(AssertionError):
+ pool.free_swa(torch.arange(2 * P, 2 * P + 5))
+ dummy = pool.translate_loc_from_full_to_swa(torch.arange(sizes.full_token - P, sizes.full_token))
+ assert int(dummy[0]) == sizes.n_win_slots - P
+ assert pool.k_cache(3) is pool.window_pool[3] and pool.num_layers == 10 and pool.unit_bytes()[0] > 0
+
+
[email protected](not torch.cuda.is_available(), reason="packed writes run through triton")
+def test_packed_writes_round_trip_on_cuda():
+ from kernels.test_dsv41_pack import reference_roundtrip # tests/ is on sys.path under pytest
+
+ args = _args()
+ sizes = dsv41_pool_sizes(4, args, 0.5, P)
+ pool = DSV41PagedKVCache(sizes, args, torch.device("cuda"), n_scratch=2)
+ kv = torch.randn(3, 512, device="cuda", dtype=torch.bfloat16)
+ slots = torch.tensor([0, 130, 5], device="cuda")
+ pool.store_window(kv, 4, slots)
+ assert torch.equal(pool.read_window(4, slots), reference_roundtrip(kv, FP8_E8M0_B32))
+ pool.store_main(kv, 2, slots)
+ assert torch.equal(pool.read_main(2, slots), reference_roundtrip(kv, FP4_E4M3_B16))
+ k = torch.randn(3, 128, device="cuda", dtype=torch.bfloat16)
+ pool.store_index(k, 7, slots)
+ assert torch.equal(pool.read_index(7, slots), reference_roundtrip(k, FP4_E8M0_B32))
+ pool.store_kv(kv, kv, slots, 0)
+ assert torch.equal(pool.read_window(0, slots), reference_roundtrip(kv, FP8_E8M0_B32))
diff --git a/tests/kvcache/test_dsv4_free_list.py b/tests/kvcache/test_dsv4_free_list.py
index a3ae8d0c..7b0dfb55 100644
--- a/tests/kvcache/test_dsv4_free_list.py
+++ b/tests/kvcache/test_dsv4_free_list.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import pytest
import torch
-from freetoken.kvcache.dsv4_paged_pool import FreeListAllocator
+from freetoken.kvcache.dsv4.v4_pool import FreeListAllocator
DEVICE = torch.device("cpu")
P = 128
diff --git a/tests/kvcache/test_dsv4_pool.py b/tests/kvcache/test_dsv4_pool.py
index c345b67c..c8e21997 100644
--- a/tests/kvcache/test_dsv4_pool.py
+++ b/tests/kvcache/test_dsv4_pool.py
@@ -10,8 +10,8 @@ from __future__ import annotations
import pytest
import torch
-from freetoken.kvcache.dsv4_cost_model import dsv4_pool_sizes
-from freetoken.kvcache.dsv4_paged_pool import CompressStateRing, DSV4PagedKVCache
+from freetoken.kvcache.dsv4.v4_cost_model import dsv4_pool_sizes
+from freetoken.kvcache.dsv4.v4_pool import CompressStateRing, DSV4PagedKVCache
from freetoken.models.deepseek_v4.args import DeepseekV4Args
DEVICE = torch.device("cpu")
diff --git a/tests/kvcache/test_kv_cache_rebuild.py b/tests/kvcache/test_kv_cache_rebuild.py
index a5ec94a0..0e3b7050 100644
--- a/tests/kvcache/test_kv_cache_rebuild.py
+++ b/tests/kvcache/test_kv_cache_rebuild.py
@@ -178,13 +178,13 @@ def test_dsv4_rebuild_from_config_builds_the_pool_sizes_and_attaches_the_table()
the engine used to do -- and re-points full_loc_map at the shared page table."""
from types import SimpleNamespace
- from freetoken.kvcache.dsv4_cost_model import _dsv4_pool_sizes
- from freetoken.kvcache.dsv4_cost_model import (
+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_pool_sizes
+ from freetoken.kvcache.dsv4.v4_cost_model import (
dsv4_kv_unit_bytes,
dsv4_pool_sizes,
dsv4_window_unit_bytes,
)
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
from freetoken.models.deepseek_v4.args import DeepseekV4Args
P, mrr = 128, 1
@@ -222,7 +222,7 @@ def test_dsv4_refresh_seq_state_tracks_page_table_width():
from freetoken.engine.engine import Engine
from freetoken.utils import align_ceil
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
from freetoken.scheduler.table import TableManager
P, mrr = 128, 4
@@ -264,7 +264,7 @@ def test_every_kv_pool_answers_the_sizing_surface():
the model resolved to. DSV4 must carry its OWN overrides (getattr would resolve to the
base class and silently misprice the window floor if an override vanished)."""
from freetoken.kvcache.dsa_pool import DSAKVCache, MLAKVCache
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
from freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCache
from freetoken.kvcache.mha_pool import MHAKVCache
@@ -279,7 +279,7 @@ def test_every_kv_pool_answers_the_rebuild_surface():
"""The engine drives every pool through this one surface; a pool that skips an
implementation must fail loudly at import/instantiation, not at rebuild time."""
from freetoken.kvcache.dsa_pool import DSAKVCache, MLAKVCache
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
from freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCache
from freetoken.kvcache.mha_pool import MHAKVCache
diff --git a/tests/kvcache/test_pool_sizing_surface.py b/tests/kvcache/test_pool_sizing_surface.py
index 516090b0..33f27983 100644
--- a/tests/kvcache/test_pool_sizing_surface.py
+++ b/tests/kvcache/test_pool_sizing_surface.py
@@ -30,7 +30,7 @@ def _model_config(specs, **attrs):
def test_resolve_pool_class_follows_attn_type():
from freetoken.kvcache.dsa_pool import DSAKVCache, MLAKVCache
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
from freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCache
from freetoken.kvcache.mha_pool import MHAKVCache
@@ -91,11 +91,11 @@ def _dsv4_config(num_page_override=None):
def test_dsv4_kv_cost_and_floor_parity():
- from freetoken.kvcache.dsv4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_swa_ratio, _dsv4_window_floor_pages
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
- pytest.importorskip("freetoken.kvcache.dsv4_cost_model")
- from freetoken.kvcache.dsv4_cost_model import dsv4_auto_cost_model
+ pytest.importorskip("freetoken.kvcache.dsv4.v4_cost_model")
+ from freetoken.kvcache.dsv4.v4_cost_model import dsv4_auto_cost_model
config = _dsv4_config()
P = 128
@@ -147,8 +147,8 @@ def test_generic_validate_rebuild_budget_check():
def test_dsv4_validate_rebuild_floor():
from freetoken.kvcache.base import CacheRebuildRejected
- from freetoken.kvcache.dsv4_cost_model import _dsv4_window_floor_pages
- from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_window_floor_pages
+ from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
config = _dsv4_config()
pool = object.__new__(DSV4PagedKVCache) # floor path reads no instance state
@@ -165,7 +165,7 @@ def test_rebuild_from_config_explicit_num_swa_pages_wins():
# The explicit parameter must take precedence over config.swa_num_pages_override --
# the engine's override write would otherwise mask a dropped parameter forever.
from freetoken.distributed import set_tp_info, try_get_tp_info
- from freetoken.kvcache.dsv4_cost_model import _dsv4_pool_sizes
+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_pool_sizes
from freetoken.kvcache.hybrid_swa_pool import _swa_paged_num_tokens, _swa_pool_floor
if try_get_tp_info() is None:
@@ -183,7 +183,7 @@ def test_rebuild_from_config_explicit_num_swa_pages_wins():
assert _swa_paged_num_tokens(swa_cfg, 11, num_swa_pages=explicit) == explicit + 1
assert _swa_paged_num_tokens(swa_cfg, 11) == 999_999 + 1 # override path unchanged
- from freetoken.kvcache.dsv4_cost_model import _dsv4_window_floor_pages
+ from freetoken.kvcache.dsv4.v4_cost_model import _dsv4_window_floor_pages
d_cfg = _dsv4_config()
d_floor = _dsv4_window_floor_pages(d_cfg, d_cfg.page_size)
diff --git a/tests/layers/test_mhc.py b/tests/layers/test_mhc.py
index 4cf144e7..0359bed9 100644
--- a/tests/layers/test_mhc.py
+++ b/tests/layers/test_mhc.py
@@ -152,6 +152,75 @@ def test_triton_fused_respects_input_dtype(dtype, tol):
assert err < tol, f"{name} [{dtype}]: max abs err {err}"
+def test_single_pass_applies_the_previous_pre_and_hands_on_its_own():
+ """Single-pass mHC (DeepSeek-V4.1): the input mix uses ``pre_mix_in``, the gates
+ computed here are the next sublayer's pre; post/comb are unchanged from multi-pass."""
+ from freetoken.layers.mhc import (
+ mhc_fused_post_pre_single_pass,
+ mhc_gates,
+ mhc_mix_input,
+ mhc_pre_single_pass,
+ )
+
+ fn, scale, base = _weights(seed=21)
+ res = _residual(seed=22)
+ x = torch.randn(T, HIDDEN, dtype=torch.bfloat16)
+ pre_in = torch.rand(T, N, dtype=torch.float32)
+ post0, comb0, _ = mhc_pre(res, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN)
+
+ r1, p1, c1, pre1, li1 = mhc_fused_post_pre_single_pass(
+ x, res, post0, comb0, pre_in, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN
+ )
+ r_ref = mhc_post(x, res, post0, comb0)
+ pre_ref, p_ref, c_ref = mhc_gates(r_ref, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN)
+ assert torch.equal(r1, r_ref) and torch.equal(p1, p_ref) and torch.equal(c1, c_ref)
+ assert torch.equal(pre1, pre_ref)
+ assert torch.equal(li1, mhc_mix_input(r_ref, pre_in))
+ # the multi-pass input would have used pre_ref, not pre_in
+ assert not torch.equal(li1, mhc_mix_input(r_ref, pre_ref))
+ # a one-hot pre picks one stream (the model's seed for the chain)
+ one_hot = torch.zeros(T, N)
+ one_hot[:, 0] = 1.0
+ _, _, _, li0 = mhc_pre_single_pass(res, one_hot, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN)
+ assert torch.equal(li0, res[:, 0])
+ # no post on the first sublayer: the streams pass through untouched
+ r0, *_ = mhc_fused_post_pre_single_pass(
+ x, res, None, None, one_hot, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN
+ )
+ assert torch.equal(r0, res)
+
+
[email protected](not torch.cuda.is_available(), reason="needs CUDA")
[email protected]("t,hidden,with_post", [(1, 64, True), (9, 64, True), (3, 5120, True), (3, 5120, False)])
+def test_triton_single_pass_matches_torch(t, hidden, with_post):
+ from freetoken.layers.mhc import mhc_fused_post_pre_single_pass_torch
+ from freetoken.kernel.triton.mhc import mhc_fused_post_pre_single_pass_triton
+
+ torch.manual_seed(23)
+ mix = 2 * N + N * N
+ fn = torch.randn(mix, N * hidden, dtype=torch.float32, device="cuda") * 0.05
+ scale = torch.rand(3, dtype=torch.float32, device="cuda") + 0.5
+ base = torch.randn(mix, dtype=torch.float32, device="cuda") * 0.3
+ res = torch.randn(t, N, hidden, dtype=torch.bfloat16, device="cuda")
+ x = torch.randn(t, hidden, dtype=torch.bfloat16, device="cuda")
+ pre_in = torch.rand(t, N, dtype=torch.float32, device="cuda")
+ post0 = torch.rand(t, N, 1, dtype=torch.float32, device="cuda") * POST_MULT if with_post else None
+ comb0 = torch.softmax(torch.randn(t, N, N, device="cuda"), dim=-1) if with_post else None
+
+ ref = mhc_fused_post_pre_single_pass_torch(
+ x, res, post0, comb0, pre_in, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN
+ )
+ got = mhc_fused_post_pre_single_pass_triton(
+ x, res, post0, comb0, pre_in, fn, scale, base, RMS_EPS, EPS, POST_MULT, SINKHORN
+ )
+ names = ("residual", "post", "comb", "pre_new", "layer_input")
+ tols = (2e-2, 2e-3, 2e-3, 2e-3, 2e-2)
+ for name, r, g, tol in zip(names, ref, got, tols):
+ assert g.shape == r.shape, (name, g.shape, r.shape)
+ err = (g.float() - r.float()).abs().max().item()
+ assert err < tol, f"{name}: max abs err {err}"
+
+
@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
def test_triton_pre_only_matches_torch():
"""HAS_POST=False path (layer 0's standalone hc_pre through the fused kernel)."""
@@ -176,3 +245,31 @@ def test_triton_pre_only_matches_torch():
assert (got_post.float() - ref_post.float()).abs().max().item() < 2e-3
assert (got_comb.float() - ref_comb.float()).abs().max().item() < 2e-3
assert (got_li.float() - ref_li.float()).abs().max().item() < 2e-2
+
+
[email protected](not torch.cuda.is_available(), reason="needs CUDA")
+def test_mhc_split_preserves_chunked_results_and_graph_replay():
+ from freetoken.kernel.triton.mhc import mhc_fused_post_pre_single_pass_triton as fused
+
+ torch.manual_seed(26)
+ hidden, tokens = 5120, 7
+ res = torch.randn(tokens, N, hidden, device="cuda", dtype=torch.bfloat16)
+ x = torch.randn(tokens, hidden, device="cuda", dtype=torch.bfloat16)
+ post = torch.rand(tokens, N, 1, device="cuda")
+ comb = torch.softmax(torch.randn(tokens, N, N, device="cuda"), -1)
+ pre = torch.rand(tokens, N, device="cuda")
+ fn = torch.randn(2 * N + N * N, N * hidden, device="cuda") * 0.05
+ scale, base = torch.ones(3, device="cuda"), torch.zeros(2 * N + N * N, device="cuda")
+ def run(lo, hi):
+ return fused(x[lo:hi], res[lo:hi], post[lo:hi], comb[lo:hi], pre[lo:hi], fn, scale, base,
+ RMS_EPS, EPS, POST_MULT, SINKHORN)
+ whole = run(0, tokens)
+ chunks = [run(0, 2), run(2, tokens)]
+ for i, expected in enumerate(whole):
+ assert torch.equal(expected, torch.cat([chunk[i] for chunk in chunks]))
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(graph):
+ captured = run(0, tokens)
+ graph.replay()
+ for got, expected in zip(captured, whole):
+ assert torch.equal(got, expected)
diff --git a/tests/models/deepseek_v41/__init__.py b/tests/models/deepseek_v41/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/models/deepseek_v41/common.py b/tests/models/deepseek_v41/common.py
new file mode 100644
index 00000000..d0f8efee
--- /dev/null
+++ b/tests/models/deepseek_v41/common.py
@@ -0,0 +1,176 @@
+"""Shared fixtures for the deepseek_v41 tests: a tiny V4.1-shaped config (2 window layers, a ratio-2
+encoder group with one Full + one Reuse layer, a ratio-1 decoder group with a Full candidate source and
+a Reindex layer) and synthetic bf16 and FP8/MXFP4 checkpoints in the real tensor naming. Holds no tests itself."""
+
+from __future__ import annotations
+
+import json
+import os
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+requires_cuda = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU")
+
+VOCAB = 512
+DIM = 256
+N_LAYERS = 6
+RATIOS = [0, 0, 2, 2, 1, 1, 0] # + one trailing MTP layer the model ignores
+KV_SOURCES = [2, 4]
+INDEX_SOURCES = [2, 4, 5]
+ENGRAM_LAYERS = [1]
+ENGRAM_VOCAB = 500 # bucket prime start; 3 orders x 2 heads ~= 3000 rows
+ENGRAM_ROWS = 4096
+
+
+def tiny_text_config(**over) -> dict:
+ text = dict(
+ model_type="deepseek_v41_text",
+ vocab_size=VOCAB, hidden_size=DIM, moe_intermediate_size=64, num_hidden_layers=N_LAYERS, num_attention_heads=4,
+ num_key_value_heads=1, head_dim=64, qk_rope_head_dim=16, q_lora_rank=64, o_lora_rank=32, o_groups=2,
+ hidden_act="silu", swiglu_limit=10.0, rms_norm_eps=1e-20, max_position_embeddings=16384,
+ rope_theta=10000, rope_scaling=dict(rope_type="yarn", factor=16, beta_fast=32, beta_slow=1, original_max_position_embeddings=1024),
+ n_routed_experts=8, n_shared_experts=1, num_experts_per_tok=2, scoring_func="sqrtsoftplus", topk_method="noaux_tc",
+ norm_topk_prob=True, routed_scaling_factor=1.5, sliding_window=128,
+ compress_ratios=list(RATIOS), compress_rope_theta=160000, kv_source_layer_ids=list(KV_SOURCES),
+ index_source_layer_ids=list(INDEX_SOURCES), index_n_heads=2, index_head_dim=32, index_topk=8,
+ candidate_source_layer_id=4, candidate_topk_blocks=2, candidate_block_size=4,
+ hc_mult=4, hc_sinkhorn_iters=20, hc_eps=1e-6,
+ engram_layer_ids=list(ENGRAM_LAYERS), engram_num_embeddings=[ENGRAM_ROWS], engram_max_ngram_size=4,
+ engram_vocab_size=ENGRAM_VOCAB, engram_n_heads=2, engram_head_dim=32, engram_pad_token_id=2,
+ engram_compressed_vocab_size=VOCAB, # the synthetic tokenizer maps every id to itself
+ num_nextn_predict_layers=1, dspark_block_size=0, dspark_target_layer_ids=[],
+ )
+ text.update(over)
+ return text
+
+
+def tiny_hf_config(**over) -> SimpleNamespace:
+ return SimpleNamespace(architectures=["DeepseekV41ForCausalLM"], model_type="deepseek_v41", text_config=tiny_text_config(**over))
+
+
+def checkpoint_tensors(text: dict, seed: int = 0) -> dict[str, torch.Tensor]:
+ """Random bf16 weights under the checkpoint's names (the reference ``convert.py`` dialect without
+ quantization: no ``.scale`` tensors). hc / sink / router-bias / engram gates in fp32 as shipped."""
+ g = torch.Generator().manual_seed(seed)
+
+ def rnd(*shape, scale=0.02, dtype=torch.bfloat16):
+ return (torch.randn(*shape, generator=g) * scale).to(dtype)
+
+ dim, heads, hd, rd = text["hidden_size"], text["num_attention_heads"], text["head_dim"], text["qk_rope_head_dim"]
+ qr, ol, og = text["q_lora_rank"], text["o_lora_rank"], text["o_groups"]
+ inter, E, hc = text["moe_intermediate_size"], text["n_routed_experts"], text["hc_mult"]
+ ih, ihd = text["index_n_heads"], text["index_head_dim"]
+ mix = (2 + hc) * hc
+ t: dict[str, torch.Tensor] = {
+ "embed.weight": rnd(text["vocab_size"], dim, scale=1.0),
+ "norm.weight": torch.ones(dim, dtype=torch.bfloat16),
+ "head.weight": rnd(text["vocab_size"], dim),
+ }
+ for L in range(text["num_hidden_layers"]):
+ a = f"layers.{L}.attn"
+ t[f"{a}.wq_a.weight"] = rnd(qr, dim)
+ t[f"{a}.q_norm.weight"] = torch.ones(qr, dtype=torch.bfloat16)
+ t[f"{a}.wq_b.weight"] = rnd(heads * hd, qr)
+ t[f"{a}.wkv.weight"] = rnd(hd, dim)
+ t[f"{a}.kv_norm.weight"] = torch.ones(hd, dtype=torch.bfloat16)
+ t[f"{a}.wo_a.weight"] = rnd(og * ol, heads * hd // og)
+ t[f"{a}.wo_b.weight"] = rnd(dim, og * ol)
+ t[f"{a}.attn_sink"] = rnd(heads, scale=0.5, dtype=torch.float32)
+ ratio = text["compress_ratios"][L]
+ if L in text["kv_source_layer_ids"]:
+ t[f"{a}.compressor.wkv.weight"] = rnd(hd, dim)
+ if ratio > 1:
+ t[f"{a}.compressor.wgate.weight"] = rnd(hd, dim)
+ t[f"{a}.compressor.norm.weight"] = torch.ones(hd, dtype=torch.bfloat16)
+ t[f"{a}.indexer.wk.weight"] = rnd(ihd, hd)
+ t[f"{a}.indexer.k_norm.weight"] = torch.ones(ihd, dtype=torch.bfloat16)
+ if L in text["index_source_layer_ids"]:
+ t[f"{a}.indexer.wq_b.weight"] = rnd(ih * ihd, qr)
+ t[f"{a}.indexer.weights_proj.weight"] = rnd(ih, dim)
+ p = f"layers.{L}"
+ t[f"{p}.attn_norm.weight"] = torch.ones(dim, dtype=torch.bfloat16)
+ t[f"{p}.ffn_norm.weight"] = torch.ones(dim, dtype=torch.bfloat16)
+ t[f"{p}.ffn.gate.weight"] = rnd(E, dim)
+ t[f"{p}.ffn.gate.bias"] = rnd(E, scale=0.1, dtype=torch.float32)
+ t[f"{p}.ffn.gate.bias_vl"] = rnd(E, scale=0.1, dtype=torch.float32)
+ for w, shape in (("w1", (inter, dim)), ("w2", (dim, inter)), ("w3", (inter, dim))):
+ t[f"{p}.ffn.shared_experts.{w}.weight"] = rnd(*shape)
+ for e in range(E):
+ t[f"{p}.ffn.experts.{e}.{w}.weight"] = rnd(*shape)
+ for nm in ("hc_attn_fn", "hc_ffn_fn"):
+ t[f"{p}.{nm}"] = rnd(mix, hc * dim, scale=0.05, dtype=torch.float32)
+ for nm in ("hc_attn_base", "hc_ffn_base"):
+ t[f"{p}.{nm}"] = rnd(mix, scale=0.3, dtype=torch.float32)
+ for nm in ("hc_attn_scale", "hc_ffn_scale"):
+ t[f"{p}.{nm}"] = (torch.rand(3, generator=g) + 0.5).to(torch.float32)
+ if L in text["engram_layer_ids"]:
+ e = f"{p}.engram"
+ cols = (text["engram_max_ngram_size"] - 1) * text["engram_n_heads"]
+ t[f"{e}.wkv.weight"] = rnd(dim * (hc + 1), cols * text["engram_head_dim"])
+ t[f"{e}.q_weight"] = torch.ones(hc, dim, dtype=torch.bfloat16)
+ t[f"{e}.k_weight"] = torch.ones(hc, dim, dtype=torch.bfloat16)
+ rows = text["engram_num_embeddings"][text["engram_layer_ids"].index(L)]
+ t[f"{e}.embed.weight"] = rnd(rows, text["engram_head_dim"], scale=1.0).to(torch.float8_e4m3fn)
+ t[f"{e}.embed.scale"] = torch.full((rows, text["engram_head_dim"] // 32), 127, dtype=torch.uint8).view(torch.float8_e8m0fnu)
+ # the DSpark stage and the vision tower: present in a real checkpoint, dropped by the loader
+ t["mtp.0.attn_norm.weight"] = torch.ones(dim, dtype=torch.bfloat16)
+ t["vision.norm.weight"] = torch.ones(8, dtype=torch.bfloat16)
+ return t
+
+
+def write_tiny_checkpoint(folder: str, seed: int = 0, *, quantized: bool = False, **over) -> dict[str, torch.Tensor]:
+ """A loadable checkpoint directory: HF-style config.json + one safetensors shard + index."""
+ from safetensors.torch import save_file
+
+ if quantized:
+ over.setdefault("moe_intermediate_size", 256)
+ text = tiny_text_config(**over)
+ os.makedirs(folder, exist_ok=True)
+ with open(os.path.join(folder, "config.json"), "w") as f:
+ config = {"architectures": ["DeepseekV41ForCausalLM"], "model_type": "deepseek_v41", "text_config": text}
+ if quantized:
+ config["quantization_config"] = {
+ "quant_method": "fp8", "activation_scheme": "dynamic",
+ "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4",
+ }
+ json.dump(config, f)
+ tensors = checkpoint_tensors(text, seed)
+ if quantized:
+ tensors = _quantize_checkpoint(tensors)
+ save_file(tensors, os.path.join(folder, "model-00001-of-00001.safetensors"))
+ with open(os.path.join(folder, "model.safetensors.index.json"), "w") as f:
+ json.dump({"metadata": {}, "weight_map": {k: "model-00001-of-00001.safetensors" for k in tensors}}, f)
+ return tensors
+
+
+def _quantize_checkpoint(tensors):
+ from .reference.kernel import _round_fp4
+
+ out = dict(tensors)
+ fp8_suffixes = (
+ ".attn.wq_a", ".attn.wq_b", ".attn.wkv", ".attn.wo_a", ".attn.wo_b",
+ ".indexer.wq_b", ".shared_experts.w1", ".shared_experts.w2", ".shared_experts.w3", ".engram.wkv",
+ )
+ grid = torch.tensor([0., .5, 1., 1.5, 2., 3., 4., 6.])
+ for name, weight in tensors.items():
+ if not name.endswith(".weight"):
+ continue
+ stem = name.removesuffix(".weight")
+ if ".ffn.experts." in name:
+ groups = weight.float().unflatten(-1, (-1, 32))
+ scale = torch.exp2(torch.ceil(torch.log2(groups.abs().amax(-1).clamp_min(1e-12) / 6)))
+ q = _round_fp4(groups / scale.unsqueeze(-1)).flatten(-2)
+ codes = (q.abs().unsqueeze(-1) == grid).to(torch.int32).argmax(-1).to(torch.uint8)
+ codes |= torch.signbit(q).to(torch.uint8) * 8
+ out[name] = codes[:, 0::2] | (codes[:, 1::2] << 4)
+ out[stem + ".scale"] = scale.to(torch.float8_e8m0fnu)
+ elif stem.endswith(fp8_suffixes):
+ n, k = weight.shape
+ blocks = weight.float().reshape(n // 32, 32, k // 32, 32).permute(0, 2, 1, 3)
+ scale = torch.exp2(torch.ceil(torch.log2(blocks.abs().amax((-1, -2)).clamp_min(1e-4) / 448)))
+ expanded = scale.repeat_interleave(32, 0).repeat_interleave(32, 1)
+ out[name] = (weight.float() / expanded).to(torch.float8_e4m3fn)
+ out[stem + ".scale"] = scale.to(torch.float8_e8m0fnu)
+ return out
diff --git a/tests/models/deepseek_v41/harness.py b/tests/models/deepseek_v41/harness.py
new file mode 100644
index 00000000..98b19e2d
--- /dev/null
+++ b/tests/models/deepseek_v41/harness.py
@@ -0,0 +1,182 @@
+"""A minimal in-process engine for the tiny DeepSeek-V4.1 model: builds the DSV41 pool, backend and
+context by hand (no scheduler), loads a synthetic bf16 or FP8/MXFP4 checkpoint with resident or offloaded experts, and drives
+ragged prefill / batched decode batches through ``model.forward()``. Shared by the forward smoke test
+and the reference-parity test."""
+
+from __future__ import annotations
+
+from dataclasses import replace
+
+import torch
+
+from freetoken.core import Batch, Context, Req, SamplingParams, get_global_ctx, set_global_ctx
+from freetoken.distributed.info import set_tp_info, try_get_tp_info
+from freetoken.engine.engine import _materialize_loaded_weight_state_dict
+from freetoken.kvcache.dsv4.v41_cost_model import dsv41_pool_sizes
+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache
+from freetoken.layers import set_rope_device
+from freetoken.layers.quantization import NoQuantConfig
+from freetoken.layers.quantization.method import finalize_quant
+from freetoken.models import create_model
+from freetoken.models.deepseek_v41.config import parse_config
+from freetoken.models.deepseek_v41.engram import ZeroEngramTable
+from freetoken.models.deepseek_v41.weight import iter_weights
+from freetoken.utils.hf import cached_load_hf_config
+from freetoken.utils.torch_utils import torch_dtype
+
+P = 128
+
+
+class TinyEngine:
+ def __init__(self, checkpoint: str, *, max_seq_len: int = 1024, max_running_req: int = 2, swa_decoder_replay: str = "exact", engram_table=None, quantized: bool = False):
+ self.device = torch.device("cuda")
+ self.checkpoint = checkpoint
+ if try_get_tp_info() is None:
+ set_tp_info(0, 1)
+ set_rope_device(self.device)
+ hf_config = cached_load_hf_config(checkpoint)
+ mc = parse_config(hf_config)
+ args = mc.dsv41_args
+ args.max_seq_len = max_seq_len
+ args.max_batch_size = max_running_req + 1
+ args.swa_decoder_replay = swa_decoder_replay
+ self.args = args
+ quant = NoQuantConfig()
+ if quantized:
+ from freetoken.models.register import checkpoint_quant_config, get_model_spec
+
+ quant = checkpoint_quant_config(checkpoint, hf_config, get_model_spec("DeepseekV41ForCausalLM"))
+ self.config = replace(mc, moe_strategy="offload" if quantized else "fused", decode_target="gpu", quant=quant)
+ mc = self.config
+ with torch.device("meta"), torch_dtype(torch.bfloat16):
+ self.model = create_model(self.config)
+ state = _materialize_loaded_weight_state_dict(
+ self.model.state_dict(), iter_weights(checkpoint, self.device, include_moe_experts=False), device=self.device,
+ )
+ self.model.load_state_dict(state)
+ finalize_quant(self.model)
+ from freetoken.moe.expert_banks import load_expert_banks
+
+ method = self.model.model.layers.op_list[0].ffn.experts.quant_method
+ if quantized:
+ from freetoken.moe.offload_cache import OffloadMoeCache, attach_offload_moe_cache
+
+ self.banks = load_expert_banks(
+ checkpoint, mc, method=method, device=self.device, dtype=torch.bfloat16, parallel=False,
+ )
+ self.expert_cache = OffloadMoeCache(
+ num_layers=mc.num_moe_layers, num_experts=mc.num_experts,
+ cache_size=mc.num_moe_layers * mc.num_experts, device=self.device,
+ quant_format=self.banks.quant_format, layout=method.layout(),
+ )
+ self.expert_cache.set_bank_sources(self.banks.sources, layer_residency=self.banks.layer_residency)
+ attach_offload_moe_cache(self.model, self.expert_cache)
+ else: # resident experts load as banks, not through the state dict (as in Engine._load_resident_experts)
+ from freetoken.layers import iter_moe_layers
+ from freetoken.moe.expert_banks import attach_resident_banks
+
+ attach_resident_banks(list(iter_moe_layers(self.model)), load_expert_banks(
+ checkpoint, mc, method=method, device=self.device, dtype=torch.bfloat16, parallel=False, resident=True,
+ ))
+ for layer in self.model.engram_layers():
+ layer.attach_table(engram_table if engram_table is not None else ZeroEngramTable(layer.width, self.device))
+
+ self.max_running_req = max_running_req
+ num_pages = max_seq_len // P
+ self.pool = DSV41PagedKVCache(dsv41_pool_sizes(num_pages + 1, args, 1.0, P), args, self.device, n_scratch=max_running_req + 1)
+ self.pool._init_paged_state(max_running_req, radix=False)
+ self.page_table = torch.zeros(max_running_req + 1, max_seq_len, dtype=torch.int32, device=self.device)
+ self.page_table[max_running_req].fill_(num_pages * P) # the dummy row -> the reserved tail page
+ self.pool.attach_page_table(self.page_table)
+ try:
+ ctx = get_global_ctx()
+ except AssertionError:
+ ctx = Context(page_size=P)
+ set_global_ctx(ctx)
+ from freetoken.attention.dsv41_sparse import DSV41SparseAttnBackend
+
+ self.ctx = ctx
+ ctx.page_table = self.page_table
+ ctx.kv_cache = self.pool # the backend reads the pool's device from the context
+ self.backend = DSV41SparseAttnBackend(self.config)
+ self._next_page = 0
+ self._bind()
+
+ def _bind(self) -> None:
+ """Point the process-wide ``Context`` at this engine's page table, pool and backend. The
+ context is a singleton the model reads through ``get_global_ctx()``, so a test that holds two
+ engines must rebind before each forward; ``prefill`` / ``decode`` do this, and the model
+ re-resolves its backend when the binding changes."""
+ ctx = self.ctx
+ if (
+ getattr(ctx, "attn_backend", None) is self.backend
+ and getattr(ctx, "kv_cache", None) is self.pool
+ and getattr(ctx, "page_table", None) is self.page_table
+ ):
+ return
+ ctx.page_table = self.page_table
+ ctx.kv_cache = self.pool
+ ctx.attn_backend = self.backend
+ self.model.mark_for_rebind()
+
+ # ----- requests -----------------------------------------------------------------------
+ def new_request(self, table_idx: int, tokens: list[int], max_new: int = 64) -> Req:
+ """A request with pages for ``len(tokens) + max_new`` positions bound on row ``table_idx``."""
+ n_pages = -(-(len(tokens) + max_new) // P)
+ first = self._next_page
+ self._next_page += n_pages
+ locs = torch.arange(first * P, (first + n_pages) * P, device=self.device)
+ self.page_table[table_idx, : n_pages * P] = locs.to(torch.int32)
+ self.pool.alloc_swa(locs)
+ return Req(input_ids=torch.tensor(tokens, dtype=torch.int32), table_idx=table_idx, cached_len=0, output_len=max_new,
+ uid=table_idx, sampling_params=SamplingParams(), cache_handle=None)
+
+ def new_request_on_prefix(self, table_idx: int, donor: Req, tokens: list[int], max_new: int = 64) -> Req:
+ """A request whose first ``hit`` positions alias ``donor``'s pages (what a radix prefix hit does
+ through the page table) and whose remaining positions get pages of their own. ``hit`` is what the
+ scheduler admits: the donor's prompt, capped by ``match_req`` (the last token, and the pool's
+ ``prefix_replay_tokens`` under bounded replay) and page-aligned. The caller prefills the rest."""
+ limit = len(tokens) - 1
+ if self.pool.prefix_replay_tokens:
+ limit = min(limit, max(0, len(tokens) - self.pool.prefix_replay_tokens))
+ hit = min(donor.device_len, limit) // P * P
+ assert tokens[:hit] == donor.input_ids[:hit].tolist(), "the prefix must match the donor's prompt"
+ self.page_table[table_idx, :hit] = self.page_table[donor.table_idx, :hit]
+ n_pages = -(-(len(tokens) + max_new - hit) // P)
+ first = self._next_page
+ self._next_page += n_pages
+ locs = torch.arange(first * P, (first + n_pages) * P, device=self.device)
+ self.page_table[table_idx, hit : hit + n_pages * P] = locs.to(torch.int32)
+ self.pool.alloc_swa(locs)
+ return Req(input_ids=torch.tensor(tokens, dtype=torch.int32), table_idx=table_idx, cached_len=hit, output_len=max_new,
+ uid=table_idx, sampling_params=SamplingParams(), cache_handle=None)
+
+ def prefill(self, reqs: list[Req]) -> torch.Tensor:
+ """Run each request's ``[cached_len, device_len)`` tokens; returns ``[B, vocab]`` logits."""
+ batch = Batch(reqs=reqs, phase="prefill")
+ batch.padded_reqs = list(reqs)
+ batch.input_ids = torch.cat([r.input_ids[r.cached_len : r.device_len] for r in reqs]).to(self.device)
+ batch.positions = torch.cat([torch.arange(r.cached_len, r.device_len) for r in reqs]).to(self.device)
+ self._bind()
+ self.backend.prepare_metadata(batch)
+ with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, False):
+ return self.model.forward().float()
+
+ def decode(self, reqs: list[Req], tokens: list[int]) -> torch.Tensor:
+ """Feed one new token per request (appended at ``device_len - 1``); returns ``[B, vocab]``."""
+ for r, t in zip(reqs, tokens):
+ r.append_host(torch.tensor([t], dtype=torch.int32))
+ r.complete_one()
+ batch = Batch(reqs=reqs, phase="decode")
+ batch.padded_reqs = list(reqs)
+ batch.input_ids = torch.tensor(tokens, dtype=torch.int32, device=self.device)
+ batch.positions = torch.tensor([r.device_len - 1 for r in reqs], device=self.device)
+ batch.active_table_idx = torch.tensor([r.table_idx for r in reqs], device=self.device)
+ self._bind()
+ self.backend.prepare_metadata(batch)
+ with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, False):
+ return self.model.forward().float()
+
+ def finish_prefill(self, reqs: list[Req]) -> None:
+ for r in reqs:
+ r.cached_len = r.device_len
diff --git a/tests/models/deepseek_v41/reference/LICENSE b/tests/models/deepseek_v41/reference/LICENSE
new file mode 100644
index 00000000..d42fae90
--- /dev/null
+++ b/tests/models/deepseek_v41/reference/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 DeepSeek
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
\ No newline at end of file
diff --git a/tests/models/deepseek_v41/reference/__init__.py b/tests/models/deepseek_v41/reference/__init__.py
new file mode 100644
index 00000000..79d0c331
--- /dev/null
+++ b/tests/models/deepseek_v41/reference/__init__.py
@@ -0,0 +1,2 @@
+"""The DeepSeek-V4.1 reference implementation (MIT, vendored from the HF repo's ``inference/``) with its
+tilelang kernels transcribed to torch -- the numerical oracle FreeToken's port is tested against."""
diff --git a/tests/models/deepseek_v41/reference/engram.py b/tests/models/deepseek_v41/reference/engram.py
new file mode 100644
index 00000000..62555311
--- /dev/null
+++ b/tests/models/deepseek_v41/reference/engram.py
@@ -0,0 +1,186 @@
+# Vendored from https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash (inference/engram.py), MIT License
+# (see LICENSE next to this file); unmodified.
+from dataclasses import dataclass
+
+import numpy as np
+import torch
+from sympy import isprime
+from torch import nn
+
+
+def find_next_prime(start: int, seen_primes: set[int]) -> int:
+ """The smallest prime above `start` that has not been handed out yet."""
+ candidate = start + 1
+ while not isprime(candidate) or candidate in seen_primes:
+ candidate += 1
+ return candidate
+
+
+def build_compressed_token_map(tokenizer) -> tuple[list[int], int]:
+ """Map every token id onto a smaller id space where tokens that normalize alike collapse together.
+
+ N-grams are hashed over these compressed ids, so " The", "the" and "THE" all hash the same way.
+ Returns the lookup plus the size of the compressed vocab -- and that size matters beyond bounds
+ checking, because every hash multiplier is derived from it.
+ """
+ from tokenizers import Regex, normalizers
+
+ # a private-use char, so a token that is exactly one space survives Strip() instead of
+ # collapsing to the empty string and merging with unrelated tokens
+ sentinel = "\ue000"
+ normalizer = normalizers.Sequence(
+ [
+ normalizers.NFKC(),
+ normalizers.NFD(),
+ normalizers.StripAccents(),
+ normalizers.Lowercase(),
+ normalizers.Replace(Regex(r"[ \t\r\n]+"), " "),
+ normalizers.Replace(Regex(r"^ $"), sentinel),
+ normalizers.Strip(),
+ normalizers.Replace(sentinel, " "),
+ ]
+ )
+
+ # the raw Rust tokenizer, matching what training decodes with (no clean_up_tokenization_spaces)
+ backend = tokenizer.backend_tokenizer
+ key_to_new: dict[str, int] = {}
+ lookup = [0] * len(tokenizer)
+ for token_id in range(len(tokenizer)):
+ text = backend.decode([token_id], skip_special_tokens=False)
+ if "\ufffd" in text:
+ # a partial UTF-8 byte token: nothing to normalize, so key it by its raw form
+ key = backend.id_to_token(token_id)
+ else:
+ normalized = normalizer.normalize_str(text)
+ key = normalized if normalized else text
+
+ new_id = key_to_new.get(key)
+ if new_id is None:
+ new_id = len(key_to_new)
+ key_to_new[key] = new_id
+ lookup[token_id] = new_id
+
+ return lookup, len(key_to_new)
+
+
+def compute_hash_multipliers(
+ layer_ids: tuple[int, ...], max_ngram_size: int, tokenizer_vocab_size: int
+) -> torch.Tensor:
+ """One multiplier per (layer, lookback), from a per-layer RNG so layers hash differently.
+
+ Kept odd, and bounded so that `token_id * multiplier` cannot overflow int64.
+ """
+ max_long = np.iinfo(np.int64).max
+ multiplier_bound = max(1, (max_long // tokenizer_vocab_size) // 2)
+ rows = []
+ for layer_id in layer_ids:
+ generator = np.random.default_rng(10007 * layer_id)
+ values = generator.integers(
+ low=0,
+ high=multiplier_bound,
+ size=(max_ngram_size,),
+ dtype=np.int64,
+ )
+ rows.append(torch.tensor(values * 2 + 1))
+ return torch.stack(rows)
+
+
+@dataclass(frozen=True)
+class EngramLayout:
+ """Bucket layout of the n-gram hash tables.
+
+ A position is hashed as `max_ngram_size - 1` n-grams (2-gram .. max_ngram_size-gram), each split
+ over `n_heads` heads. Every (n-gram size, head) pair owns its own prime-sized bucket range in the
+ layer's table; the primes are drawn in order and never reused, which keeps the ranges disjoint.
+ """
+
+ max_ngram_size: int
+ layer_ids: tuple[int, ...]
+ num_embeddings: tuple[int, ...] # table rows, per engram layer
+ primes: tuple[tuple[tuple[int, ...], ...], ...] # [layer][n-gram size][head] bucket modulus
+ n_heads: int
+ head_dim: int
+
+ @classmethod
+ def from_args(cls, args) -> "EngramLayout | None":
+ layer_ids = tuple(args.engram_layer_ids)
+ if not layer_ids:
+ return None
+ max_ngram_size, n_heads = args.engram_max_ngram_size, args.engram_n_heads
+ primes, seen = [], set()
+ for _ in layer_ids:
+ per_ngram = []
+ for _ in range(max_ngram_size - 1):
+ sizes, current = [], args.engram_vocab_size - 1
+ for _ in range(n_heads):
+ current = find_next_prime(current, seen)
+ seen.add(current)
+ sizes.append(current)
+ per_ngram.append(tuple(sizes))
+ primes.append(tuple(per_ngram))
+ return cls(
+ max_ngram_size=max_ngram_size,
+ layer_ids=layer_ids,
+ num_embeddings=tuple(args.engram_num_embeddings),
+ primes=tuple(primes),
+ n_heads=n_heads,
+ head_dim=args.engram_head_dim,
+ )
+
+
+class NgramHashState(nn.Module):
+ """Maps each position to the hash ids of the n-grams ending there.
+
+ Ids go through the compressed table, then each position is hashed with the `max_ngram_size - 1`
+ tokens before it. Look-back stops at the start of the sequence and at any dead token (an image
+ span, cached as DEAD), so an n-gram never spans one. The cache carries all of this across the
+ prefill/decode split.
+ """
+
+ DEAD = -1
+
+ def __init__(self, args, layout: EngramLayout, tokenizer):
+ super().__init__()
+ self.layout = layout
+ # every hash multiplier derives from the compressed vocab size, so a mismatch there would
+ # silently rehash the whole table
+ token_map, vocab_size = build_compressed_token_map(tokenizer)
+ assert vocab_size == args.engram_compressed_vocab_size, (vocab_size, args.engram_compressed_vocab_size)
+ self.pad_id = token_map[args.engram_pad_id]
+ flat = [[p for per_ngram in layer for p in per_ngram] for layer in layout.primes]
+ offsets = [np.cumsum([0, *sizes[:-1]]) for sizes in flat]
+ multipliers = compute_hash_multipliers(layout.layer_ids, layout.max_ngram_size, vocab_size)
+ self.register_buffer("primes", torch.tensor(layout.primes), persistent=False)
+ self.register_buffer("offsets", torch.tensor(np.array(offsets)), persistent=False)
+ self.register_buffer("multipliers", multipliers, persistent=False)
+ self.register_buffer("token_map", torch.tensor(token_map), persistent=False)
+ self.register_buffer(
+ "cache", torch.empty(args.max_batch_size, args.max_seq_len, dtype=torch.int64), persistent=False
+ )
+
+ @torch.inference_mode()
+ def forward(self, input_ids: torch.Tensor, start_pos: int, token_mask: torch.Tensor | None = None) -> torch.Tensor:
+ """token_mask: [B, L], False for tokens that take no part in an n-gram (image spans).
+ Returns the hash ids, shaped [B, L, n_engram_layers, n_hash_cols]."""
+ batch, seqlen = input_ids.shape
+ compressed = self.token_map[input_ids]
+ if token_mask is not None:
+ compressed = torch.where(token_mask, compressed, self.DEAD)
+ self.cache[:batch, start_pos : start_pos + seqlen] = compressed
+
+ positions = torch.arange(start_pos, start_pos + seqlen, device=input_ids.device).expand(batch, seqlen)
+ tokens, blocked = [], torch.zeros_like(positions, dtype=torch.bool)
+ for shift in range(self.layout.max_ngram_size):
+ source = self.cache[:batch].gather(1, (positions - shift).clamp_min(0))
+ blocked = blocked | (positions < shift) | (source == self.DEAD)
+ tokens.append(torch.where(blocked, self.pad_id, source))
+ tokens = torch.stack(tokens, dim=-1) # [B, L, max_ngram_size]
+
+ # XOR the multiplied ids together one lookback at a time, so the running value after step i
+ # is the hash of the (i+1)-gram; each lands in its own prime-sized bucket range
+ products = tokens.unsqueeze(2) * self.multipliers # [B, L, n_engram_layers, max_ngram_size]
+ rolling, hashes = products[..., 0], []
+ for i in range(1, self.layout.max_ngram_size):
+ rolling = torch.bitwise_xor(rolling, products[..., i])
+ hashes.append(rolling.unsqueeze(-1) % self.primes[:, i - 1])
+ return torch.cat(hashes, dim=-1) + self.offsets
diff --git a/tests/models/deepseek_v41/reference/kernel.py b/tests/models/deepseek_v41/reference/kernel.py
new file mode 100644
index 00000000..fab0ae08
--- /dev/null
+++ b/tests/models/deepseek_v41/reference/kernel.py
@@ -0,0 +1,113 @@
+"""Pure-torch transcriptions of the reference tilelang kernels (inference/kernel.py), for the test oracle.
+
+Same numerics as the tilelang code: fp32 math, power-of-two (ue8m0) or e4m3 scales rounded the way
+the kernels round them, the e2m1 grid with the hardware's round-to-nearest-even, and the fp8/fp4
+in-place round trips that write the dequantized value back in the input dtype. Quantized GEMMs
+dequantize independently in torch and round each projection's output to bf16.
+"""
+
+from __future__ import annotations
+
+import torch
+
+_FP4_GRID = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])
+
+
+def _pow2_ceil(x: torch.Tensor) -> torch.Tensor:
+ """``fast_round_scale``: 2 ** ceil(log2(x)) for x > 0."""
+ return torch.exp2(torch.ceil(torch.log2(x)))
+
+
+def _round_fp4(x: torch.Tensor) -> torch.Tensor:
+ """Round-to-nearest-even onto the signed e2m1 grid (the ``float4_e2m1fn`` cast)."""
+ a = x.abs()
+ grid = _FP4_GRID.to(x.device)
+ hi = torch.searchsorted(grid, a.reshape(-1).contiguous(), right=True).reshape(a.shape).clamp(1, 7)
+ below, above = grid[hi - 1], grid[hi]
+ mid = (below + above) / 2
+ even_above = torch.isin(above, torch.tensor([0.0, 1.0, 2.0, 4.0], device=x.device))
+ r = torch.where((a > mid) | ((a == mid) & even_above), above, below)
+ r = torch.where(a >= 6.0, torch.full_like(r, 6.0), r)
+ return torch.copysign(r, x)
+
+
+def act_quant(x: torch.Tensor, block_size: int = 128, scale_fmt=None, scale_dtype=torch.float32, inplace: bool = False):
+ """Block-wise fp8 quantization; ``inplace=True`` writes the quant+dequant round trip back into ``x``."""
+ n = x.size(-1)
+ assert n % block_size == 0
+ g = x.float().unflatten(-1, (-1, block_size))
+ amax = g.abs().amax(dim=-1).clamp_min(1e-4)
+ s = _pow2_ceil(amax / 448.0) if scale_fmt is not None else amax / 448.0
+ q = (g / s.unsqueeze(-1)).clamp(-448.0, 448.0).to(torch.float8_e4m3fn)
+ if inplace:
+ x.copy_((q.float() * s.unsqueeze(-1)).flatten(-2).to(x.dtype))
+ return x
+ return q.flatten(-2), s.to(scale_dtype)
+
+
+def fp4_act_quant(x: torch.Tensor, block_size: int = 32, inplace: bool = False, scale_dtype=torch.float8_e8m0fnu):
+ """Block-wise fp4 with ue8m0 (pow2) or e4m3 scales; ``inplace=True`` writes the round trip back."""
+ n = x.size(-1)
+ assert n % block_size == 0
+ g = x.float().unflatten(-1, (-1, block_size))
+ amax = g.abs().amax(dim=-1)
+ if scale_dtype == torch.float8_e4m3fn:
+ amax = amax.clamp_min(6 * 2**-9)
+ s = (amax / 6.0).to(torch.float8_e4m3fn).float()
+ else:
+ amax = amax.clamp_min(6 * 2**-126)
+ s = _pow2_ceil(amax / 6.0)
+ q = _round_fp4((g / s.unsqueeze(-1)).clamp(-6.0, 6.0))
+ if inplace:
+ x.copy_((q * s.unsqueeze(-1)).flatten(-2).to(x.dtype))
+ return x
+ return q.flatten(-2), s.to(scale_dtype)
+
+
+def fp8_gemm(a, a_s, b, b_s, scale_dtype=torch.float32, block_size=128):
+ """``C = (A * s_a) @ (B * s_b)^T`` over dequantized operands."""
+ k = a.size(-1)
+ a_deq = (a.float().unflatten(-1, (-1, block_size)) * a_s.float().unsqueeze(-1)).flatten(-2)
+ n = b.size(0)
+ bs = b_s.float().repeat_interleave(block_size, 0)[:n].repeat_interleave(block_size, 1)[:, :k]
+ return (a_deq @ (b.float() * bs).t()).bfloat16()
+
+
+def fp4_gemm(a, a_s, b, b_s, scale_dtype=torch.float8_e8m0fnu, act_block_size=32):
+ packed = b.view(torch.uint8)
+ codes = torch.stack((packed & 15, packed >> 4), dim=-1).flatten(-2).long()
+ grid = torch.tensor([0., .5, 1., 1.5, 2., 3., 4., 6., -0., -.5, -1., -1.5, -2., -3., -4., -6.], device=b.device)
+ weight = grid[codes] * b_s.float().repeat_interleave(32, -1)
+ x = (a.float().unflatten(-1, (-1, act_block_size)) * a_s.float().unsqueeze(-1)).flatten(-2)
+ return (x @ weight.t()).bfloat16()
+
+
+def sparse_attn(q: torch.Tensor, kv: torch.Tensor, attn_sink: torch.Tensor, topk_idxs: torch.Tensor, softmax_scale: float) -> torch.Tensor:
+ """Per (batch, query): softmax over the gathered ``topk_idxs`` (-1 = absent) plus the sink null key.
+
+ ``q [b, m, h, d]``, ``kv [b, n, d]``, ``topk_idxs [b, m, topk]`` -> ``o [b, m, h, d]`` in q's dtype.
+ A row with no valid index yields zeros (the finite ``-1e30`` running max of the kernel)."""
+ b, m, h, d = q.shape
+ valid = topk_idxs >= 0
+ gathered = kv[torch.arange(b, device=q.device)[:, None, None], topk_idxs.clamp_min(0).long()] # [b, m, topk, d]
+ scores = torch.einsum("bmhd,bmtd->bmht", q.float(), gathered.float()) * softmax_scale
+ scores = scores.masked_fill(~valid[:, :, None, :], float("-inf"))
+ mx = scores.amax(dim=-1).clamp_min(-1e30)
+ p = torch.exp(scores - mx.unsqueeze(-1))
+ denom = p.sum(-1) + torch.exp(attn_sink.float()[None, None, :] - mx)
+ o = torch.einsum("bmht,bmtd->bmhd", p, gathered.float()) / denom.unsqueeze(-1)
+ return o.to(q.dtype)
+
+
+def hc_split_sinkhorn(mixes: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor, hc_mult: int = 4, sinkhorn_iters: int = 20, eps: float = 1e-6):
+ hc = hc_mult
+ mixes, sc, base = mixes.float(), hc_scale.float(), hc_base.float()
+ pre = torch.sigmoid(mixes[..., :hc] * sc[0] + base[:hc]) + eps
+ post = 2 * torch.sigmoid(mixes[..., hc : 2 * hc] * sc[1] + base[hc : 2 * hc])
+ comb = (mixes[..., 2 * hc :] * sc[2] + base[2 * hc :]).unflatten(-1, (hc, hc))
+ comb = comb.softmax(dim=-1) + eps
+ comb = comb / (comb.sum(dim=-2, keepdim=True) + eps)
+ for _ in range(sinkhorn_iters - 1):
+ comb = comb / (comb.sum(dim=-1, keepdim=True) + eps)
+ comb = comb / (comb.sum(dim=-2, keepdim=True) + eps)
+ return pre, post, comb
diff --git a/tests/models/deepseek_v41/reference/model.py b/tests/models/deepseek_v41/reference/model.py
new file mode 100644
index 00000000..593c7518
--- /dev/null
+++ b/tests/models/deepseek_v41/reference/model.py
@@ -0,0 +1,1458 @@
+# Vendored from https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash (inference/model.py), MIT License
+# (see LICENSE next to this file). FreeToken keeps this copy as the numerical ORACLE its port is tested
+# against. Edits, all marked "FreeToken:", are limited to (1) local imports with the tilelang kernels
+# replaced by the pure-torch transcriptions in kernel.py, and (2) forward_bounded(), a Decoder SWA
+# Bounded Replay oracle built from the unmodified layer math.
+import math
+from contextlib import contextmanager
+from dataclasses import dataclass
+from functools import lru_cache
+from typing import Literal
+
+import torch
+import torch.distributed as dist
+import torch.nn.functional as F
+from torch import nn
+
+# FreeToken: local imports.
+from .engram import EngramLayout, NgramHashState
+from .kernel import (
+ act_quant,
+ fp4_act_quant,
+ fp4_gemm,
+ fp8_gemm,
+ hc_split_sinkhorn,
+ sparse_attn,
+)
+
+from .vision import ViT, Aligner
+
+IMAGE_START, IMAGE, IMAGE_NEW_LINE, IMAGE_END = range(4) # FreeToken: image_processor constants
+
+# Set once by Transformer.__init__; one model per process, so layers just read them.
+world_size = 1
+rank = 0
+default_dtype = torch.float8_e4m3fn # storage dtype for Linear weights, from ModelArgs.dtype
+fp8_block_size = 32 # one fp8 scale per 32x32 weight block / 32 activations
+fp4_block_size = 32 # one fp4 scale per 32 elements along K
+scale_fmt = "ue8m0"
+scale_dtype = torch.float8_e8m0fnu
+
+
+@contextmanager
+def set_dtype(dtype):
+ """Temporarily override torch's default dtype, restoring it even if the body raises."""
+ prev = torch.get_default_dtype()
+ torch.set_default_dtype(dtype)
+ try:
+ yield
+ finally:
+ torch.set_default_dtype(prev)
+
+
+@dataclass
+class ModelArgs:
+ """Field names are exactly the config JSON keys. The defaults are a small model that
+ `python model.py` can run, not the released shapes -- though the scale-independent
+ values (norm_eps, score_func, hc_*, engram_*) do match it."""
+
+ # runtime limits rather than model shape: they size the KV caches
+ max_batch_size: int = 4
+ max_seq_len: int = 4096
+ temperature: float = 1
+ dtype: Literal["bf16", "fp8"] = "fp8"
+ expert_dtype: Literal["fp4"] | None = "fp4"
+ vocab_size: int = 129280
+ dim: int = 1024
+ moe_inter_dim: int = 1024
+ n_layers: int = 5
+ n_mtp_layers: int = 1 # extra draft layers appended after the backbone, indices n_layers..
+ n_heads: int = 16
+ # moe
+ n_routed_experts: int = 8
+ n_shared_experts: int = 1
+ n_activated_experts: int = 2
+ score_func: Literal["softmax", "sigmoid", "sqrtsoftplus"] = "sqrtsoftplus"
+ gate_temp: float = 1.0
+ norm_topk_prob: bool = True
+ route_scale: float = 1.0
+ swiglu_limit: float = 0.0
+ # attention: latent q/kv projections, plus a LoRA-factorised output projection over o_groups
+ q_lora_rank: int = 256
+ head_dim: int = 128
+ rope_head_dim: int = 32
+ norm_eps: float = 1e-20
+ o_groups: int = 8
+ o_lora_rank: int = 256
+ # sparse attention: every layer attends over a sliding window, and may add compressed KV on top
+ window_size: int = 128
+ # one entry per layer, MTP layers included: 0 = sliding window only, r = KV compressed r-to-1
+ compress_ratios: tuple[int, ...] = (0, 2, 2, 1, 1, 0)
+ # layers sharing a ratio also share one compressed KV and one indexer, produced by the first
+ kv_source_layers: tuple[int, ...] = (1, 3)
+ index_source_layers: tuple[int, ...] = (1, 3)
+ # rope, with YaRN extrapolation when original_seq_len > 0. Compressed KV rotates at its own
+ # theta because one latent stands for compress_ratio tokens, so its positions are further apart.
+ compress_rope_theta: float = 40000.0
+ original_seq_len: int = 0
+ rope_theta: float = 10000.0
+ rope_factor: float = 40
+ beta_fast: int = 32
+ beta_slow: int = 1
+ # the indexer: a small extra attention that scores compressed positions, so each query can keep
+ # just `index_topk` of them. Names match DeepSeek-V3.2-Exp, where this mechanism first appeared.
+ index_n_heads: int = 16
+ index_head_dim: int = 64
+ index_topk: int = 64
+ # candidate pre-filtering: candidate_source_layer < 0 turns it off and the other two are unused
+ candidate_source_layer: int = -1
+ candidate_topk_blocks: int = 0
+ candidate_block_size: int = 0
+ # hyper-connections: the residual stream is carried as hc_mult parallel copies
+ hc_mult: int = 4
+ hc_sinkhorn_iters: int = 20
+ hc_eps: float = 1e-6
+ # engram: n-gram hash lookups added into the residual stream at a few layers
+ engram_layer_ids: tuple[int, ...] = ()
+ engram_num_embeddings: tuple[int, ...] = () # unpadded table rows; each rank allocates ceil(rows / world_size)
+ engram_max_ngram_size: int = 1
+ engram_vocab_size: int = 0 # bucket size each (n-gram size, head) starts searching primes from
+ engram_n_heads: int = 0
+ engram_head_dim: int = 0
+ engram_pad_id: int = 2 # token that fills n-gram slots with no history; matches training
+ # size of the compressed tokenizer vocab; every hash multiplier is derived from it
+ engram_compressed_vocab_size: int = 0
+ # vision (VL); vision_n_layers == 0 disables the vision path
+ vision_n_layers: int = 0
+ vision_dim: int = 1024
+ vision_n_heads: int = 16
+ vision_inter_dim: int = 2816
+ vision_patch_size: int = 14
+ vision_rope_theta: float = 10000.0
+ vision_downsample_ratio: int = 3
+ vision_max_n_token: int = 1024
+ vision_min_pixels: int = 544 * 544
+ vision_max_wh_ratio: int | None = None
+ # raw id of <|deepseek_image|>; every position of an image span carries this id in input_ids
+ image_token_id: int = 129264
+ # dspark draft head. Only the forward pass is implemented here -- nothing calls forward_spec,
+ # so these are read but the speculative-decoding loop itself is out of scope for this repo.
+ dspark_block_size: int = 0
+ dspark_noise_token_id: int = 0
+ dspark_target_layer_ids: tuple[int, ...] = ()
+ dspark_markov_rank: int = 256
+ dspark_n_routed_experts: int = 0
+ dspark_n_activated_experts: int = 0
+
+ @property
+ def vision_enabled(self) -> bool:
+ return self.vision_n_layers > 0
+
+ def get_moe_config(self, layer_id: int) -> tuple[int, int]:
+ """Return the routed/activated expert counts for a given layer."""
+ if layer_id < self.n_layers:
+ return self.n_routed_experts, self.n_activated_experts
+ return (
+ self.dspark_n_routed_experts or self.n_routed_experts,
+ self.dspark_n_activated_experts or self.n_activated_experts,
+ )
+
+
+class ParallelEmbedding(nn.Module):
+ """Embedding sharded along the vocab dimension. Each rank holds vocab_size // world_size rows.
+ Out-of-range indices are zero-masked before all_reduce to combine partial embeddings."""
+
+ def __init__(self, vocab_size: int, dim: int):
+ super().__init__()
+ self.vocab_size = vocab_size
+ self.dim = dim
+ assert vocab_size % world_size == 0, (
+ f"Vocabulary size must be divisible by world size (world_size={world_size})"
+ )
+ self.part_vocab_size = vocab_size // world_size
+ self.vocab_start_idx = rank * self.part_vocab_size
+ self.vocab_end_idx = self.vocab_start_idx + self.part_vocab_size
+ self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ if world_size > 1:
+ # ids off this rank read row 0 then get zeroed, so the all_reduce sums one real row
+ mask = (x < self.vocab_start_idx) | (x >= self.vocab_end_idx)
+ x = x - self.vocab_start_idx
+ x[mask] = 0
+ y = F.embedding(x, self.weight)
+ if world_size > 1:
+ y[mask] = 0
+ dist.all_reduce(y)
+ return y
+
+
+def linear(x: torch.Tensor, weight: torch.Tensor, bias: torch.Tensor | None = None) -> torch.Tensor:
+ """Pick a GEMM from the weight dtype. Quantized weights need a quantized activation, and both
+ fp4 and fp8 weights take an fp8 one -- for fp4 the kernel handles the mixed precision."""
+ assert bias is None
+
+ if weight.dtype == torch.float4_e2m1fn_x2:
+ x, s = act_quant(x, fp8_block_size, scale_fmt, scale_dtype)
+ return fp4_gemm(
+ x,
+ s,
+ weight,
+ weight.scale,
+ scale_dtype,
+ act_block_size=fp8_block_size,
+ )
+ elif weight.dtype == torch.float8_e4m3fn:
+ x, s = act_quant(x, fp8_block_size, scale_fmt, scale_dtype)
+ return fp8_gemm(
+ x,
+ s,
+ weight,
+ weight.scale,
+ scale_dtype,
+ block_size=fp8_block_size,
+ )
+ else:
+ return F.linear(x, weight)
+
+
+class Linear(nn.Module):
+ """bf16, fp8 or fp4 weights. Quantized ones get a `scale`, also attached to `.weight` so that
+ `linear()` can reach it."""
+
+ def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype=None):
+ super().__init__()
+ self.in_features = in_features
+ self.out_features = out_features
+ dtype = dtype or default_dtype
+ if dtype == torch.float4_e2m1fn_x2:
+ # two values per byte: [out, in] logically, [out, in//2] stored
+ self.weight = nn.Parameter(torch.empty(out_features, in_features // 2, dtype=torch.float4_e2m1fn_x2))
+ self.weight.scale = self.scale = nn.Parameter(
+ torch.empty(out_features, in_features // fp4_block_size, dtype=torch.float8_e8m0fnu)
+ )
+ elif dtype == torch.float8_e4m3fn:
+ self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype))
+ self.weight.scale = self.scale = nn.Parameter(
+ torch.empty(
+ (out_features + fp8_block_size - 1) // fp8_block_size,
+ (in_features + fp8_block_size - 1) // fp8_block_size,
+ dtype=torch.float8_e8m0fnu,
+ )
+ )
+ else:
+ self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype))
+ self.register_parameter("scale", None)
+ if bias:
+ self.bias = nn.Parameter(torch.empty(out_features))
+ else:
+ self.register_parameter("bias", None)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ return linear(x, self.weight, self.bias)
+
+
+class ColumnParallelLinear(Linear):
+ """Splits the output dim across ranks; each rank's slice of the output is already complete."""
+
+ def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype=None):
+ assert out_features % world_size == 0, (
+ f"Output features must be divisible by world size (world_size={world_size})"
+ )
+ self.part_out_features = out_features // world_size
+ super().__init__(in_features, self.part_out_features, bias, dtype)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ return linear(x, self.weight, self.bias)
+
+
+class RowParallelLinear(Linear):
+ """Splits the reduction dim, so each rank holds a partial sum: hence the fp32 all_reduce, with
+ the bias added only after it."""
+
+ def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype=None):
+ assert in_features % world_size == 0, (
+ f"Input features must be divisible by world size (world_size={world_size})"
+ )
+ self.part_in_features = in_features // world_size
+ super().__init__(self.part_in_features, out_features, bias, dtype)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ y = linear(x, self.weight, None)
+ if world_size > 1:
+ y = y.float()
+ dist.all_reduce(y)
+ if self.bias is not None:
+ y += self.bias
+ return y.type_as(x)
+
+
+class RMSNorm(nn.Module):
+ def __init__(self, dim: int, eps: float = 1e-6):
+ super().__init__()
+ self.dim = dim
+ self.eps = eps
+ self.weight = nn.Parameter(torch.ones(dim))
+
+ def forward(self, x: torch.Tensor):
+ dtype = x.dtype
+ x = x.float()
+ var = x.square().mean(-1, keepdim=True)
+ x = x * torch.rsqrt(var + self.eps)
+ return (self.weight * x).to(dtype)
+
+
+class ParallelEngramEmbedding(nn.Module):
+ """The n-gram hash table, sharded over its rows. Stays fp8: rows are dequantized on lookup."""
+
+ def __init__(self, num_embeddings: int, dim: int):
+ super().__init__()
+ self.num_embeddings = num_embeddings
+ self.dim = dim
+ self.part_num_embeddings = (num_embeddings + world_size - 1) // world_size
+ self.vocab_start_idx = rank * self.part_num_embeddings
+ self.vocab_end_idx = self.vocab_start_idx + self.part_num_embeddings
+
+ self.block_size = fp8_block_size
+ # the table stays fp8 as stored: rows are dequantized with `scale` on lookup
+ self.weight = nn.Parameter(torch.empty(self.part_num_embeddings, dim, dtype=torch.float8_e4m3fn))
+ self.scale = nn.Parameter(torch.empty(self.part_num_embeddings, dim // self.block_size, dtype=scale_dtype))
+
+ def forward(self, indices: torch.Tensor) -> torch.Tensor:
+ mask = (indices < self.vocab_start_idx) | (indices >= self.vocab_end_idx)
+ local_indices = indices - self.vocab_start_idx
+ local_indices = local_indices.masked_fill(mask, 0)
+
+ values = F.embedding(local_indices, self.weight)
+ scales = F.embedding(local_indices, self.scale)
+ values = values.float().unflatten(-1, (-1, self.block_size)) * scales.float().unsqueeze(-1)
+ values = values.flatten(-2).to(torch.bfloat16)
+ values = values.masked_fill(mask.unsqueeze(-1), 0)
+
+ if world_size > 1:
+ dist.all_reduce(values)
+ return values
+
+
+class Engram(nn.Module):
+ """Writes an n-gram lookup into the residual stream, gated by how well it matches that stream.
+
+ The hash ids fetch `n_hash_cols` rows; `wkv` turns them into one key per hc copy plus a shared
+ value. The gate is a normalized dot product of stream against key.
+ """
+
+ def __init__(self, args: ModelArgs, layer_id: int, layout: EngramLayout):
+ super().__init__()
+ self.layer_id = layer_id
+ self.layer_hash_index = layout.layer_ids.index(layer_id)
+ self.dim = args.dim
+ self.hc_mult = args.hc_mult
+ self.clamp_value = 1e-6
+
+ self.embed = ParallelEngramEmbedding(layout.num_embeddings[self.layer_hash_index], layout.head_dim)
+ n_hash_cols = (layout.max_ngram_size - 1) * layout.n_heads
+ self.wkv = Linear(n_hash_cols * layout.head_dim, args.dim * (args.hc_mult + 1))
+ self.eps = args.norm_eps
+ self.q_weight = nn.Parameter(torch.ones(args.hc_mult, args.dim))
+ self.k_weight = nn.Parameter(torch.ones(args.hc_mult, args.dim))
+
+ def forward(self, x: torch.Tensor, hash_ids: torch.Tensor, token_mask: torch.Tensor | None = None) -> torch.Tensor:
+ """x: [B, L, hc_mult, dim]; hash_ids: [B, L, n_hash_cols]; token_mask: [B, L], False shuts
+ the gate so those positions pass through untouched."""
+ kv = self.wkv(self.embed(hash_ids).flatten(-2))
+ key, value = kv.split([self.hc_mult * self.dim, self.dim], dim=-1)
+ key = key.float().unflatten(-1, (self.hc_mult, self.dim))
+ weight = self.q_weight.float() * self.k_weight.float() # only ever used as a product
+ h, eps = x.float(), self.eps
+ # normalized per (token, hc copy) over `dim`, NOT jointly over the copies
+ rstd = torch.rsqrt(h.square().mean(-1) + eps) * torch.rsqrt(key.square().mean(-1) + eps)
+ dot = (h * weight * key).sum(-1) * rstd * self.dim**-0.5
+ # signed sqrt before the sigmoid, matching the training kernel
+ gate = torch.sigmoid(torch.copysign(dot.abs().clamp_min(self.clamp_value).sqrt(), dot))
+ if token_mask is not None:
+ gate = gate.masked_fill(~token_mask.unsqueeze(-1), 0)
+ return (h + gate.unsqueeze(-1) * value.float().unsqueeze(-2)).to(x.dtype)
+
+
+@lru_cache(2)
+def precompute_freqs_cis(dim, seqlen, original_seq_len, base, factor, beta_fast, beta_slow) -> torch.Tensor:
+ """Rotary frequencies as complex exponentials, one row per position.
+
+ With original_seq_len > 0 this applies YaRN: dimensions whose wavelength already fits inside the
+ training context keep their frequency, those far beyond it are divided by `factor`, and the
+ `beta_fast`..`beta_slow` band in between is faded across with a linear ramp.
+ """
+ freqs = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
+ if original_seq_len > 0:
+ # the dim whose wavelength completes `rotations` turns over the training context
+ def corrected_dim(rotations):
+ return dim * math.log(original_seq_len / (rotations * 2 * math.pi)) / (2 * math.log(base))
+
+ low = max(math.floor(corrected_dim(beta_fast)), 0)
+ high = min(math.ceil(corrected_dim(beta_slow)), dim - 1)
+ ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / max(high - low, 1e-3)).clamp(0, 1)
+ smooth = 1 - ramp
+ freqs = freqs / factor * (1 - smooth) + freqs * smooth
+
+ freqs = torch.outer(torch.arange(seqlen), freqs)
+ return torch.polar(torch.ones_like(freqs), freqs)
+
+
+def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor, inverse: bool = False) -> torch.Tensor:
+ """Rotate `x` in place, taking adjacent element pairs as complex numbers. Accepts [b, s, d] and
+ [b, s, h, d]; `inverse` conjugates the rotation, which is how the attention output gets the
+ query's rotation removed again so the cache can stay in one shared rotated form."""
+ y = x
+ x = torch.view_as_complex(x.float().unflatten(-1, (-1, 2)))
+ if inverse:
+ freqs_cis = freqs_cis.conj()
+ if x.ndim == 3:
+ freqs_cis = freqs_cis.view(1, x.size(1), x.size(-1))
+ else:
+ freqs_cis = freqs_cis.view(1, x.size(1), 1, x.size(-1))
+ x = torch.view_as_real(x * freqs_cis).flatten(-2)
+ y.copy_(x)
+ return y
+
+
+@lru_cache(1)
+def get_window_topk_idxs(window_size: int, bsz: int, seqlen: int, start_pos: int):
+ """Which sliding-window cache slots each query attends to; -1 marks a slot holding nothing.
+
+ The cache is a ring of `window_size` slots. Prefill needs one row per query, each seeing its own
+ causal window. A decode step has a single query that sees the whole ring, listed oldest first.
+ Order within a row does not matter to `sparse_attn`, which handles every slot independently.
+ """
+ if start_pos == 0:
+ end = torch.arange(seqlen).unsqueeze(1)
+ idxs = (end - window_size + 1).clamp(0) + torch.arange(min(seqlen, window_size))
+ idxs = torch.where(idxs > end, -1, idxs) # before the sequence started
+ else:
+ oldest = start_pos % window_size + 1
+ idxs = torch.cat([torch.arange(oldest, window_size), torch.arange(oldest)])
+ idxs = torch.where(idxs > start_pos, -1, idxs) # ring still filling
+ # sparse_attn needs real [b, m, topk] int32 memory, hence the materializing expand
+ return idxs.int().unsqueeze(0).expand(bsz, -1, -1).contiguous()
+
+
+class Compressor(nn.Module):
+ """Pools `compress_ratio` consecutive tokens into one KV latent with a learned softmax gate.
+
+ Returns the latent before RoPE, or None while a group is still filling up -- so during decode it
+ only yields every `compress_ratio` steps, holding the partial group in `kv_state`/`score_state`.
+ Pre-RoPE is deliberate: the indexer needs the unrotated form, so Attention rotates afterwards.
+ """
+
+ def __init__(self, args: ModelArgs, layer_id: int):
+ super().__init__()
+ compress_ratio = args.compress_ratios[layer_id]
+ head_dim = args.head_dim
+ self.compress_ratio = compress_ratio
+ self.head_dim = head_dim
+ self.norm = RMSNorm(head_dim, args.norm_eps)
+ # ratio 1 is a plain projection, so it stays in the checkpoint's bf16; the softmax pooling
+ # above ratio 1 runs in fp32, so those weights are promoted to fp32 to match
+ self.wkv = Linear(args.dim, head_dim, dtype=torch.float32 if compress_ratio > 1 else torch.bfloat16)
+ if compress_ratio > 1:
+ self.wgate = Linear(args.dim, head_dim, dtype=torch.float32)
+ # tail of an incomplete group, carried across decode steps
+ self.kv_state: torch.Tensor
+ self.score_state: torch.Tensor
+ state_shape = (args.max_batch_size, compress_ratio, head_dim)
+ self.register_buffer("kv_state", torch.zeros(state_shape, dtype=torch.float32), persistent=False)
+ self.register_buffer(
+ "score_state", torch.full(state_shape, -torch.inf, dtype=torch.float32), persistent=False
+ )
+
+ def forward(self, x: torch.Tensor, start_pos: int) -> torch.Tensor | None:
+ bsz, seqlen, _ = x.size()
+ ratio, dtype = self.compress_ratio, x.dtype
+ if ratio == 1: # one token per group: nothing to pool, so no gate and no fp32
+ return self.norm(self.wkv(x))
+
+ x = x.float()
+ kv, score = self.wkv(x), self.wgate(x)
+ if start_pos == 0:
+ should_compress = seqlen >= ratio
+ remainder = seqlen % ratio
+ cutoff = seqlen - remainder
+ if remainder: # trailing partial group waits in the state
+ kv, self.kv_state[:bsz, :remainder] = kv.split([cutoff, remainder], dim=1)
+ score, self.score_state[:bsz, :remainder] = score.split([cutoff, remainder], dim=1)
+ kv = kv.unflatten(1, (-1, ratio))
+ score = score.unflatten(1, (-1, ratio))
+ kv = (kv * score.softmax(dim=2)).sum(dim=2)
+ else: # one token per step: fill a slot, and pool only when the group just completed
+ should_compress = (start_pos + 1) % ratio == 0
+ slot = start_pos % ratio
+ self.kv_state[:bsz, slot] = kv.squeeze(1)
+ self.score_state[:bsz, slot] = score.squeeze(1)
+ if should_compress:
+ kv = (self.kv_state[:bsz] * self.score_state[:bsz].softmax(dim=1)).sum(dim=1, keepdim=True)
+ if not should_compress:
+ return None
+ return self.norm(kv.to(dtype))
+
+
+class Indexer(torch.nn.Module):
+ """Keeps the `index_topk` best compressed positions per query.
+
+ A small side attention: fp4 query heads against one shared key per compressed position, scores
+ rectified then combined by `weights_proj`. With a candidate source this is the second of two
+ levels; `select_candidate_blocks` is the first.
+ """
+
+ def __init__(self, args: ModelArgs, layer_id: int):
+ super().__init__()
+ # the index keys are derived from the compressor's latent, so only a layer that compresses
+ # its own KV can produce them; every other indexer reads them from that layer's cache
+ self.owns_k = layer_id in args.kv_source_layers
+ self.compress_ratio = args.compress_ratios[layer_id]
+ self.is_candidate_source = layer_id == args.candidate_source_layer
+ self.uses_candidates = 0 <= args.candidate_source_layer < layer_id
+ self.candidate_topk_blocks = args.candidate_topk_blocks
+ self.candidate_block_size = args.candidate_block_size
+ self.dim = args.dim
+ self.n_heads = args.index_n_heads
+ self.n_local_heads = args.index_n_heads // world_size
+ self.index_head_dim = args.index_head_dim
+ self.rope_head_dim = args.rope_head_dim
+ self.index_topk = args.index_topk
+ self.q_lora_rank = args.q_lora_rank
+ self.softmax_scale = self.index_head_dim**-0.5
+ self.wq_b = ColumnParallelLinear(self.q_lora_rank, self.n_heads * self.index_head_dim)
+ self.weights_proj = ColumnParallelLinear(self.dim, self.n_heads, dtype=torch.bfloat16)
+ self.freqs_cis: torch.Tensor | None = None
+ if self.owns_k:
+ self.wk = Linear(args.head_dim, self.index_head_dim, dtype=torch.bfloat16)
+ self.k_norm = RMSNorm(self.index_head_dim, args.norm_eps)
+ self.k_cache: torch.Tensor
+ self.register_buffer(
+ "k_cache",
+ torch.zeros(args.max_batch_size, args.max_seq_len // self.compress_ratio, args.index_head_dim),
+ persistent=False,
+ )
+
+ def forward(self, x: torch.Tensor, qr: torch.Tensor, latent: torch.Tensor, start_pos: int, offset: int):
+ """`latent` is this layer's RoPE-free compressed latent, None when this layer does not
+ compress or when its current group is still incomplete. An index-key owner turns it into
+ index keys here, which has to happen before Attention overwrites that same storage with
+ the RoPE'd, quantized values."""
+ assert self.freqs_cis is not None
+ bsz, seqlen, _ = x.size()
+ ratio, rd, end_pos = self.compress_ratio, self.rope_head_dim, start_pos + seqlen
+
+ # latent is None while a group is still filling up, so there is nothing to publish yet
+ if self.owns_k and latent is not None:
+ # a latent stands for the first token of its group, so group j takes position j * ratio
+ freqs = (
+ self.freqs_cis[: seqlen - seqlen % ratio : ratio]
+ if start_pos == 0
+ else self.freqs_cis[start_pos + 1 - ratio].unsqueeze(0)
+ )
+ k = self.k_norm(self.wk(latent))
+ apply_rotary_emb(k[..., -rd:], freqs)
+ fp4_act_quant(k, fp4_block_size, True)
+ self.k_cache[:bsz, start_pos // ratio : start_pos // ratio + k.size(1)] = k
+ shared_attn.index_k = self.k_cache
+
+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.index_head_dim))
+ apply_rotary_emb(q[..., -rd:], self.freqs_cis[start_pos:end_pos])
+ fp4_act_quant(q, fp4_block_size, True)
+
+ index_k = shared_attn.index_k[:bsz, : end_pos // ratio]
+ weights = self.weights_proj(x) * (self.softmax_scale * self.n_heads**-0.5)
+ index_score = torch.einsum("bshd,btd->bsht", q, index_k)
+ index_score = (index_score.relu_() * weights.unsqueeze(-1)).sum(dim=2)
+ if world_size > 1:
+ dist.all_reduce(index_score)
+
+ # how many compressed positions each query can see: a block becomes visible once the query
+ # has passed its last token. One query per decode step, so there it is just a number.
+ if start_pos == 0:
+ compress_lens = (torch.arange(1, seqlen + 1, device=x.device) // ratio).unsqueeze(-1)
+ index_score.masked_fill_(torch.arange(seqlen // ratio, device=x.device) >= compress_lens, -torch.inf)
+ else:
+ compress_lens = end_pos // ratio
+
+ if self.is_candidate_source:
+ shared_attn.candidates = select_candidate_blocks(
+ index_score, compress_lens, self.candidate_topk_blocks, self.candidate_block_size
+ )
+ elif self.uses_candidates:
+ # level two: score with our own weights, but only inside the source's candidate blocks
+ index_score = index_score.masked_fill(~shared_attn.candidates, -torch.inf)
+
+ # top-k by score, re-sorted into position order; unreachable -> -1, rest shifted by offset
+ topk = min(self.index_topk, end_pos // ratio)
+ idxs = index_score.topk(topk, dim=-1, sorted=False).indices.sort(dim=-1).values
+ return torch.where(idxs < compress_lens, idxs + offset, -1).int()
+
+ # FreeToken: Decoder SWA Bounded Replay oracle -- the indexer for a TAIL prefill: queries are the
+ # positions [s, s + n) of a prompt of L_all tokens; the keys (a source's) were published for all
+ # L_all tokens by forward()'s start_pos == 0 path. Same scoring / masking / top-k as forward().
+ def forward_tail(self, x, qr, s: int, n: int, offset: int):
+ assert self.freqs_cis is not None
+ bsz = x.size(0)
+ ratio, rd = self.compress_ratio, self.rope_head_dim
+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.index_head_dim))
+ apply_rotary_emb(q[..., -rd:], self.freqs_cis[s : s + n])
+ fp4_act_quant(q, fp4_block_size, True)
+ end_pos = s + n
+ index_k = shared_attn.index_k[:bsz, : end_pos // ratio]
+ weights = self.weights_proj(x) * (self.softmax_scale * self.n_heads**-0.5)
+ index_score = torch.einsum("bshd,btd->bsht", q, index_k)
+ index_score = (index_score.relu_() * weights.unsqueeze(-1)).sum(dim=2)
+ compress_lens = ((s + torch.arange(1, n + 1, device=x.device)) // ratio).unsqueeze(-1)
+ index_score.masked_fill_(torch.arange(end_pos // ratio, device=x.device) >= compress_lens, -torch.inf)
+ if self.is_candidate_source:
+ shared_attn.candidates = select_candidate_blocks(
+ index_score, compress_lens, self.candidate_topk_blocks, self.candidate_block_size
+ )
+ elif self.uses_candidates:
+ index_score = index_score.masked_fill(~shared_attn.candidates, -torch.inf)
+ topk = min(self.index_topk, end_pos // ratio)
+ idxs = index_score.topk(topk, dim=-1, sorted=False).indices.sort(dim=-1).values
+ return torch.where(idxs < compress_lens, idxs + offset, -1).int()
+
+
+def select_candidate_blocks(
+ logits: torch.Tensor,
+ compress_lens: torch.Tensor | int,
+ topk_blocks: int,
+ block_size: int,
+) -> torch.Tensor:
+ """Level one of the two-level top-k: keep the `topk_blocks` highest-scoring blocks per query.
+
+ `logits` is [..., n_positions] with positions the query cannot reach already at -inf, which is
+ what makes a block score of -inf mean "not reachable yet". `compress_lens` is a plain int during
+ decode, or broadcasts against logits' leading dims during prefill. Returns a bool mask shaped
+ like `logits`, so the layers consuming it just mask and never think about blocks again.
+ """
+ width = logits.size(-1)
+ # score each block by its best position; -inf pads the last one out to block_size
+ scores = F.pad(logits, (0, -width % block_size), value=-torch.inf)
+ scores = scores.unflatten(-1, (-1, block_size)).amax(dim=-1)
+ num_blocks = scores.size(-1)
+
+ # the block with this query's newest position is only partly filled, so pin it in: it holds the
+ # most recent tokens but could otherwise be outscored by an older, full block
+ last = (compress_lens - 1) // block_size
+ scores = scores.masked_fill(torch.arange(num_blocks, device=logits.device) == last, torch.inf)
+
+ top = scores.topk(min(topk_blocks, num_blocks), dim=-1)
+ # fewer reachable blocks than topk_blocks means leftover picks came back -inf: drop them
+ keep = torch.zeros_like(scores, dtype=torch.bool).scatter_(-1, top.indices, top.values > -torch.inf)
+ return keep.repeat_interleave(block_size, dim=-1)[..., :width]
+
+
+class Attention(nn.Module):
+ """Latent attention over two KV sources at once, concatenated into one `sparse_attn` call: a
+ sliding window of raw KV, plus -- when compress_ratio > 0 -- `index_topk` compressed positions
+ reaching further back. Q and the output projection are both low-rank, the latter grouped.
+
+ compress_ratio > 0 does not mean the layer compresses its own KV: only kv_source_layers do,
+ the rest read that same cache.
+ """
+
+ def __init__(self, layer_id: int, args: ModelArgs):
+ super().__init__()
+ self.layer_id = layer_id
+ self.dim = args.dim
+ self.n_heads = args.n_heads
+ self.n_local_heads = args.n_heads // world_size
+ self.q_lora_rank = args.q_lora_rank
+ self.o_lora_rank = args.o_lora_rank
+ self.head_dim = args.head_dim
+ self.rope_head_dim = args.rope_head_dim
+ self.nope_head_dim = args.head_dim - args.rope_head_dim
+ self.n_groups = args.o_groups
+ self.n_local_groups = self.n_groups // world_size
+ self.window_size = args.window_size
+ self.compress_ratio = args.compress_ratios[layer_id]
+ self.eps = args.norm_eps
+
+ self.attn_sink = nn.Parameter(torch.empty(self.n_local_heads, dtype=torch.float32))
+ self.wq_a = Linear(self.dim, self.q_lora_rank)
+ self.q_norm = RMSNorm(self.q_lora_rank, self.eps)
+ self.wq_b = ColumnParallelLinear(self.q_lora_rank, self.n_heads * self.head_dim)
+ self.wkv = Linear(self.dim, self.head_dim)
+ self.kv_norm = RMSNorm(self.head_dim, self.eps)
+ self.wo_a = ColumnParallelLinear(
+ self.n_heads * self.head_dim // self.n_groups,
+ self.n_groups * args.o_lora_rank,
+ dtype=torch.bfloat16,
+ )
+ self.wo_b = RowParallelLinear(self.n_groups * args.o_lora_rank, self.dim)
+ self.softmax_scale = self.head_dim**-0.5
+
+ is_backbone = layer_id < args.n_layers
+ self.is_kv_source = is_backbone and layer_id in args.kv_source_layers
+ self.is_index_source = is_backbone and layer_id in args.index_source_layers
+ self.compressor: Compressor | None = None
+ self.indexer: Indexer | None = None
+ if self.is_kv_source:
+ self.compressor = Compressor(args, layer_id)
+ if self.is_index_source:
+ self.indexer = Indexer(args, layer_id)
+
+ self.window_kv_cache: torch.Tensor
+ self.register_buffer(
+ "window_kv_cache",
+ torch.zeros(args.max_batch_size, args.window_size, self.head_dim),
+ persistent=False,
+ )
+ if self.is_kv_source:
+ self.compress_kv_cache: torch.Tensor
+ self.register_buffer(
+ "compress_kv_cache",
+ torch.zeros(
+ args.max_batch_size,
+ args.max_seq_len // self.compress_ratio,
+ self.head_dim,
+ ),
+ persistent=False,
+ )
+ if self.compress_ratio:
+ original_seq_len, rope_theta = (
+ args.original_seq_len,
+ args.compress_rope_theta,
+ )
+ else:
+ # disable YaRN and use base rope_theta in pure sliding-window attention
+ original_seq_len, rope_theta = 0, args.rope_theta
+ freqs_cis = precompute_freqs_cis(
+ self.rope_head_dim,
+ args.max_seq_len,
+ original_seq_len,
+ rope_theta,
+ args.rope_factor,
+ args.beta_fast,
+ args.beta_slow,
+ )
+ self.freqs_cis: torch.Tensor
+ self.register_buffer("freqs_cis", freqs_cis, persistent=False)
+
+ def _window_kv(self, x, freqs_cis, start_pos):
+ """This layer's sliding-window K and the window positions every query may attend to. The K
+ stays fp8, quantized over the whole post-RoPE vector, RoPE tail included."""
+ bsz, seqlen, _ = x.size()
+ win = self.window_size
+ kv = self.kv_norm(self.wkv(x))
+ apply_rotary_emb(kv[..., -self.rope_head_dim :], freqs_cis)
+ act_quant(kv, fp8_block_size, scale_fmt, scale_dtype, True)
+ if start_pos == 0: # prefill: attend over this chunk, seeding the ring buffer for decode
+ if seqlen <= win:
+ self.window_kv_cache[:bsz, :seqlen] = kv
+ else:
+ cutoff = seqlen % win
+ self.window_kv_cache[:bsz, cutoff:win], self.window_kv_cache[:bsz, :cutoff] = kv[:, -win:].split(
+ [win - cutoff, cutoff], dim=1
+ )
+ window_kv = kv
+ else: # decode: one token into the ring buffer, attend over the whole window
+ self.window_kv_cache[:bsz, start_pos % win] = kv.squeeze(1)
+ window_kv = self.window_kv_cache[:bsz]
+ return window_kv, get_window_topk_idxs(win, bsz, seqlen, start_pos)
+
+ def _compress_topk_idxs(self, x, qr, latent, start_pos, offset, compress_len):
+ """Which compressed positions each query attends to. Index sources run their own indexer;
+ the layers in between reuse the result their source published."""
+ if not self.is_index_source:
+ return shared_attn.topk_idxs
+
+ bsz, seqlen, _ = x.size()
+ if compress_len == 0:
+ idxs = torch.empty(bsz, seqlen, 0, dtype=torch.int32, device=x.device)
+ else:
+ assert self.indexer is not None
+ if self.indexer.freqs_cis is None:
+ self.indexer.freqs_cis = self.freqs_cis
+ idxs = self.indexer(x, qr, latent, start_pos, offset)
+ shared_attn.topk_idxs = idxs
+ return idxs
+
+ def _compress_kv(self, x, qr, start_pos, offset):
+ """The shared compressed KV and the compressed positions every query may attend to. This
+ layer compresses its own KV only when it is a source; otherwise it just reads the cache."""
+ bsz, seqlen, _ = x.size()
+ ratio = self.compress_ratio
+ compress_len = (start_pos + seqlen) // ratio
+ latent = None
+ if self.is_kv_source:
+ latent = self.compressor(x, start_pos)
+ shared_attn.compress_kv = self.compress_kv_cache
+ # the indexer needs the latent before RoPE, so it runs before the cache is written
+ idxs = self._compress_topk_idxs(x, qr, latent, start_pos, offset, compress_len)
+ if latent is not None:
+ # a latent stands for the first token of its group, so group j takes position j * ratio
+ freqs = (
+ self.freqs_cis[: seqlen - seqlen % ratio : ratio]
+ if start_pos == 0
+ else self.freqs_cis[start_pos + 1 - ratio].unsqueeze(0)
+ )
+ apply_rotary_emb(latent[..., -self.rope_head_dim :], freqs)
+ # Compressed KV uses groups of 16 with E4M3 scales; the indexer uses 32 with E8M0.
+ fp4_act_quant(latent, 16, True, scale_dtype=torch.float8_e4m3fn)
+ self.compress_kv_cache[:bsz, start_pos // ratio : start_pos // ratio + latent.size(1)] = latent
+ # read after the write, so this does not depend on the slice aliasing the cache
+ return shared_attn.compress_kv[:bsz, :compress_len], idxs
+
+ def forward(self, x: torch.Tensor, start_pos: int):
+ bsz, seqlen, _ = x.size()
+ freqs_cis = self.freqs_cis[start_pos : start_pos + seqlen]
+ rd = self.rope_head_dim
+
+ qr = self.q_norm(self.wq_a(x))
+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.head_dim))
+ apply_rotary_emb(q[..., -rd:], freqs_cis)
+
+ kv, topk_idxs = self._window_kv(x, freqs_cis, start_pos)
+ if self.compress_ratio:
+ compress_kv, compress_idxs = self._compress_kv(x, qr, start_pos, kv.size(1))
+ kv = torch.cat([kv, compress_kv], dim=1)
+ topk_idxs = torch.cat([topk_idxs, compress_idxs], dim=-1)
+
+ o = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)
+ apply_rotary_emb(o[..., -rd:], freqs_cis, True)
+
+ # wo_a is block-diagonal over groups (each projects only its own heads), hence einsum not
+ # Linear. convert.py dequantizes it to bf16; an fp8 grouped GEMM would halve the memory.
+ o = o.view(bsz, seqlen, self.n_local_groups, -1)
+ wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)
+ o = torch.einsum("bsgd,grd->bsgr", o, wo_a)
+ x = self.wo_b(o.flatten(2))
+ return x
+
+ # FreeToken: Decoder SWA Bounded Replay oracle. Prefill of the prompt TAIL [s, s + n) as its own
+ # window segment (a query sees SWA keys in [max(s, i - win + 1), i]) while the global branch
+ # covers the whole prompt: a kv source compresses ``x_all`` (every token, forward()'s start_pos
+ # == 0 path), the indexer scores the tail queries at their absolute positions.
+ def forward_tail(self, x: torch.Tensor, s: int, x_all: torch.Tensor | None = None):
+ bsz, n, _ = x.size()
+ win, rd, ratio = self.window_size, self.rope_head_dim, self.compress_ratio
+ freqs_cis = self.freqs_cis[s : s + n]
+
+ qr = self.q_norm(self.wq_a(x))
+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.head_dim))
+ apply_rotary_emb(q[..., -rd:], freqs_cis)
+
+ kv = self.kv_norm(self.wkv(x))
+ apply_rotary_emb(kv[..., -rd:], freqs_cis)
+ act_quant(kv, fp8_block_size, scale_fmt, scale_dtype, True)
+ # seed the decode ring with the tail's keys at their absolute slots
+ for i in range(max(0, n - win), n):
+ self.window_kv_cache[:bsz, (s + i) % win] = kv[:, i]
+ topk_idxs = get_window_topk_idxs(win, bsz, n, 0).to(x.device)
+
+ if ratio:
+ end = s + n
+ compress_len = end // ratio
+ latent = None
+ if self.is_kv_source:
+ assert x_all is not None, "the kv source compresses every prompt token"
+ latent = self.compressor(x_all, 0)
+ shared_attn.compress_kv = self.compress_kv_cache
+ if self.indexer is not None:
+ if self.indexer.freqs_cis is None:
+ self.indexer.freqs_cis = self.freqs_cis
+ # publish the index keys for every group (forward()'s start_pos == 0 path)
+ L = x_all.size(1)
+ k = self.indexer.k_norm(self.indexer.wk(latent))
+ apply_rotary_emb(k[..., -rd:], self.freqs_cis[: L - L % ratio : ratio])
+ fp4_act_quant(k, fp4_block_size, True)
+ self.indexer.k_cache[:bsz, : k.size(1)] = k
+ shared_attn.index_k = self.indexer.k_cache
+ if not self.is_index_source:
+ idxs = shared_attn.topk_idxs
+ elif compress_len == 0:
+ idxs = torch.empty(bsz, n, 0, dtype=torch.int32, device=x.device)
+ else:
+ if self.indexer.freqs_cis is None:
+ self.indexer.freqs_cis = self.freqs_cis
+ idxs = self.indexer.forward_tail(x, qr, s, n, kv.size(1))
+ shared_attn.topk_idxs = idxs
+ if latent is not None:
+ L = x_all.size(1)
+ apply_rotary_emb(latent[..., -rd:], self.freqs_cis[: L - L % ratio : ratio])
+ fp4_act_quant(latent, 16, True, scale_dtype=torch.float8_e4m3fn)
+ self.compress_kv_cache[:bsz, : latent.size(1)] = latent
+ kv = torch.cat([kv, shared_attn.compress_kv[:bsz, :compress_len]], dim=1)
+ topk_idxs = torch.cat([topk_idxs, idxs], dim=-1)
+
+ o = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)
+ apply_rotary_emb(o[..., -rd:], freqs_cis, True)
+ o = o.view(bsz, n, self.n_local_groups, -1)
+ wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)
+ o = torch.einsum("bsgd,grd->bsgr", o, wo_a)
+ return self.wo_b(o.flatten(2))
+
+
+class Gate(nn.Module):
+ """MoE gating. The correction bias steers expert selection only; the routing weights come from the
+ unbiased scores. Image-span tokens use a separate bias (training `noaux_tc_for_vl`)."""
+
+ def __init__(self, layer_id: int, args: ModelArgs):
+ super().__init__()
+ n_routed_experts, n_activated_experts = args.get_moe_config(layer_id)
+ self.dim = args.dim
+ self.topk = n_activated_experts
+ self.score_func = args.score_func
+ self.gate_temp = args.gate_temp
+ self.norm_topk_prob = args.norm_topk_prob
+ self.route_scale = args.route_scale
+ self.weight = nn.Parameter(torch.empty(n_routed_experts, args.dim))
+ self.bias = nn.Parameter(torch.empty(n_routed_experts, dtype=torch.float32))
+ self.bias_vl = nn.Parameter(torch.empty(n_routed_experts, dtype=torch.float32)) if args.vision_enabled else None
+
+ def forward(self, x: torch.Tensor, image_mask: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor]:
+ """x: [n, dim]; image_mask: [n] bool, True for tokens inside an image span."""
+ scores = linear(x.float(), self.weight.float()) / self.gate_temp
+ if self.score_func == "softmax":
+ scores = scores.softmax(dim=-1)
+ elif self.score_func == "sigmoid":
+ scores = scores.sigmoid()
+ else:
+ scores = F.softplus(scores).sqrt()
+ bias = self.bias
+ if image_mask is not None and self.bias_vl is not None:
+ bias = torch.where(image_mask.unsqueeze(-1), self.bias_vl, bias)
+ # the bias picks experts but does not scale them: weights come from the raw scores
+ indices = (scores + bias).topk(self.topk, dim=-1)[1]
+ weights = scores.gather(1, indices)
+ if self.norm_topk_prob and self.topk > 1:
+ weights /= weights.sum(dim=-1, keepdim=True) + 1e-20 # not norm_eps, matches training
+ weights *= self.route_scale
+ return weights, indices
+
+
+class Expert(nn.Module):
+ """One SwiGLU FFN. The clamps come straight from training, where they keep fp8/fp4 activations in
+ range: the up branch is clamped on both sides, the gate branch only from above."""
+
+ def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0.0):
+ super().__init__()
+ self.w1 = Linear(dim, inter_dim, dtype=dtype)
+ self.w2 = Linear(inter_dim, dim, dtype=dtype)
+ self.w3 = Linear(dim, inter_dim, dtype=dtype)
+ self.swiglu_limit = swiglu_limit
+
+ def forward(self, x: torch.Tensor, weights: torch.Tensor | None = None) -> torch.Tensor:
+ dtype = x.dtype
+ gate = self.w1(x).float()
+ up = self.w3(x).float()
+ if self.swiglu_limit > 0:
+ up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)
+ gate = torch.clamp(gate, max=self.swiglu_limit)
+ x = F.silu(gate) * up
+ if weights is not None:
+ x = weights * x
+ return self.w2(x.to(dtype))
+
+
+class MoE(nn.Module):
+ """Top-k routed experts plus one shared expert every token goes through. Experts are split
+ across ranks, so `self.experts` is None for those another rank owns."""
+
+ def __init__(self, layer_id: int, args: ModelArgs):
+ super().__init__()
+ n_routed_experts, n_activated_experts = args.get_moe_config(layer_id)
+ self.layer_id = layer_id
+ self.dim = args.dim
+ assert n_routed_experts % world_size == 0, (
+ f"Number of experts must be divisible by world size (world_size={world_size})"
+ )
+ self.n_routed_experts = n_routed_experts
+ self.n_local_experts = n_routed_experts // world_size
+ self.n_activated_experts = n_activated_experts
+ self.experts_start_idx = rank * self.n_local_experts
+ self.experts_end_idx = self.experts_start_idx + self.n_local_experts
+ self.gate = Gate(layer_id, args)
+ expert_dtype = torch.float4_e2m1fn_x2 if args.expert_dtype == "fp4" else None
+ self.experts = nn.ModuleList(
+ [
+ Expert(
+ args.dim,
+ args.moe_inter_dim,
+ dtype=expert_dtype,
+ swiglu_limit=args.swiglu_limit,
+ )
+ if self.experts_start_idx <= i < self.experts_end_idx
+ else None
+ for i in range(self.n_routed_experts)
+ ]
+ )
+ assert args.n_shared_experts == 1
+ self.shared_experts = Expert(args.dim, args.moe_inter_dim, swiglu_limit=args.swiglu_limit)
+
+ def forward(self, x: torch.Tensor, image_mask: torch.Tensor | None = None) -> torch.Tensor:
+ shape = x.size()
+ x = x.view(-1, self.dim)
+ weights, indices = self.gate(x, None if image_mask is None else image_mask.flatten())
+ y = torch.zeros_like(x, dtype=torch.float32)
+ counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
+ for i in range(self.experts_start_idx, self.experts_end_idx):
+ if counts[i] == 0:
+ continue
+ expert = self.experts[i]
+ idx, top = torch.where(indices == i)
+ y[idx] += expert(x[idx], weights[idx, top, None])
+ if world_size > 1:
+ dist.all_reduce(y)
+ y += self.shared_experts(x)
+ return y.type_as(x).view(shape)
+
+
+class Block(nn.Module):
+ """A block whose residual stream is `hc_mult` parallel copies (Hyper-Connections).
+
+ Attention and FFN each sit between `hc_pre` (collapse the copies into one sublayer input) and
+ `hc_post` (expand back out, mixing the residual in through `comb`). `hc_mixes` derives all three
+ coefficient sets from the stream itself, `comb` made doubly stochastic by Sinkhorn.
+
+ The coefficients a sublayer computes are used by the *next* one -- see `forward`.
+ """
+
+ attention_cls = Attention
+
+ def __init__(
+ self,
+ layer_id: int,
+ args: ModelArgs,
+ engram_layout: EngramLayout | None = None,
+ ):
+ super().__init__()
+ self.layer_id = layer_id
+ self.norm_eps = args.norm_eps
+ self.attn = self.attention_cls(layer_id, args)
+ self.ffn = MoE(layer_id, args)
+ self.engram = None
+ if engram_layout is not None and layer_id in engram_layout.layer_ids:
+ self.engram = Engram(args, layer_id, engram_layout)
+ self.attn_norm = RMSNorm(args.dim, self.norm_eps)
+ self.ffn_norm = RMSNorm(args.dim, self.norm_eps)
+ self.hc_mult = hc_mult = args.hc_mult
+ self.hc_sinkhorn_iters = args.hc_sinkhorn_iters
+ self.hc_eps = args.hc_eps
+ mix_hc = (2 + hc_mult) * hc_mult
+ hc_dim = hc_mult * args.dim
+ with set_dtype(torch.float32):
+ self.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
+ self.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
+ self.hc_attn_base = nn.Parameter(torch.empty(mix_hc))
+ self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc))
+ self.hc_attn_scale = nn.Parameter(torch.empty(3))
+ self.hc_ffn_scale = nn.Parameter(torch.empty(3))
+
+ def hc_mixes(self, x: torch.Tensor, hc_fn: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor):
+ """x: [b,s,hc,d], hc_fn: [mix_hc, hc*d], hc_scale: [3], hc_base: [mix_hc]. Returns the
+ pre / post / comb coefficients, split out of one projection of the flattened stream."""
+ # normalized over the whole flattened hc*d stream, one statistic per token
+ x = x.flatten(2).float()
+ rsqrt = torch.rsqrt(x.square().mean(-1, keepdim=True) + self.norm_eps)
+ mixes = F.linear(x, hc_fn) * rsqrt
+ return hc_split_sinkhorn(mixes, hc_scale, hc_base, self.hc_mult, self.hc_sinkhorn_iters, self.hc_eps)
+
+ def hc_pre(self, x: torch.Tensor, pre_mix: torch.Tensor):
+ """Collapse the hc copies into one, weighted by pre_mix. [b,s,hc,d] x [b,s,hc] -> [b,s,d]"""
+ y = torch.sum(pre_mix.unsqueeze(-1) * x.float(), dim=2)
+ return y.to(x.dtype)
+
+ def hc_post(self, x: torch.Tensor, residual: torch.Tensor, post: torch.Tensor, comb: torch.Tensor):
+ """Expand the sublayer output back to hc copies and mix the residual in through `comb`.
+ x: [b,s,d], residual: [b,s,hc,d], post: [b,s,hc], comb: [b,s,hc,hc] -> [b,s,hc,d]"""
+ y = post.unsqueeze(-1) * x.unsqueeze(-2) + torch.sum(comb.unsqueeze(-1) * residual.unsqueeze(-2), dim=2)
+ return y.type_as(x)
+
+ def forward(
+ self,
+ x: torch.Tensor,
+ start_pos: int,
+ pre_mix: torch.Tensor,
+ image_mask: torch.Tensor | None,
+ *attn_args,
+ ) -> tuple[torch.Tensor, torch.Tensor]:
+ """`pre_mix` collapses the hc_mult copies down to one input for this block's attention. Each
+ sub-block's own `hc_mixes` produces the mix for the *next* one, so attention uses what the
+ previous layer's FFN produced and the FFN uses what this attention produced.
+
+ image_mask: [b, s] bool, True inside image spans (selects the VL routing bias)."""
+ residual = x
+ attn_pre, attn_post, attn_comb = self.hc_mixes(x, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
+ x = self.hc_pre(x, pre_mix)
+ x = self.attn_norm(x)
+ x = self.attn(x, start_pos, *attn_args)
+ x = self.hc_post(x, residual, attn_post, attn_comb)
+
+ residual = x
+ ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
+ x = self.hc_pre(x, attn_pre)
+ x = self.ffn_norm(x)
+ x = self.ffn(x, image_mask)
+ x = self.hc_post(x, residual, ffn_post, ffn_comb)
+ return x, ffn_pre
+
+ # FreeToken: Decoder SWA Bounded Replay oracle. ``h`` / ``pre_mix`` are the tail [s, N) streams,
+ # or -- at the decoder's kv source -- ``h_all`` / ``pre_mix_all`` cover every token so the
+ # source compresses them all before the block narrows to the tail.
+ def forward_tail(self, h, s: int, pre_mix, h_all=None, pre_mix_all=None):
+ if h_all is not None:
+ attn_pre, attn_post, attn_comb = self.hc_mixes(h_all, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
+ x_all = self.attn_norm(self.hc_pre(h_all, pre_mix_all))
+ residual = h_all[:, s:]
+ attn_pre, attn_post, attn_comb = attn_pre[:, s:], attn_post[:, s:], attn_comb[:, s:]
+ x = self.attn.forward_tail(x_all[:, s:], s, x_all)
+ else:
+ residual = h
+ attn_pre, attn_post, attn_comb = self.hc_mixes(h, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
+ x = self.attn_norm(self.hc_pre(h, pre_mix))
+ x = self.attn.forward_tail(x, s)
+ x = self.hc_post(x, residual, attn_post, attn_comb)
+
+ residual = x
+ ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
+ x = self.hc_pre(x, attn_pre)
+ x = self.ffn_norm(x)
+ x = self.ffn(x, None)
+ x = self.hc_post(x, residual, ffn_post, ffn_comb)
+ return x, ffn_pre
+
+
+class ParallelHead(nn.Module):
+ def __init__(self, vocab_size: int, dim: int, norm_eps: float = 1e-6, hc_eps: float = 1e-6):
+ super().__init__()
+ self.vocab_size = vocab_size
+ self.dim = dim
+ self.norm_eps = norm_eps
+ self.hc_eps = hc_eps
+ self.part_vocab_size = vocab_size // world_size
+ # bf16 in the checkpoint, kept as fp32 here so the logits come out in fp32 directly
+ self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim, dtype=torch.float32))
+
+ def forward(self, x: torch.Tensor, full_logits=False):
+ """x: [b, s, d]. Generation only needs the last position, so that is the default."""
+ if not full_logits:
+ x = x[:, -1]
+ logits = F.linear(x.float(), self.weight)
+ if world_size > 1:
+ all_logits = [torch.empty_like(logits) for _ in range(world_size)]
+ dist.all_gather(all_logits, logits)
+ logits = torch.cat(all_logits, dim=-1)
+ return logits
+
+
+@lru_cache(1)
+def get_dspark_topk_idxs(window_size: int, bsz: int, block_size: int, start_pos: int):
+ assert start_pos > 0
+ matrix = torch.cat(
+ [
+ torch.arange(min(window_size, start_pos + 1)),
+ window_size + torch.arange(block_size),
+ ]
+ )
+ return matrix.int().view(1, 1, -1).expand(bsz, block_size, -1).contiguous()
+
+
+class DSparkAttention(Attention):
+ def forward(self, x: torch.Tensor, start_pos: int, main_x: torch.Tensor):
+ assert self.compress_ratio == 0
+ bsz, seqlen, _ = main_x.size()
+ win = self.window_size
+ rd = self.rope_head_dim
+
+ main_freqs_cis = self.freqs_cis[start_pos : start_pos + seqlen]
+ main_kv = self.kv_norm(self.wkv(main_x))
+ apply_rotary_emb(main_kv[..., -rd:], main_freqs_cis)
+ act_quant(main_kv, fp8_block_size, scale_fmt, scale_dtype, True)
+
+ if start_pos == 0:
+ if seqlen <= win:
+ self.window_kv_cache[:bsz, :seqlen] = main_kv
+ else:
+ cutoff = seqlen % win
+ self.window_kv_cache[:bsz, cutoff:win], self.window_kv_cache[:bsz, :cutoff] = main_kv[:, -win:].split(
+ [win - cutoff, cutoff], dim=1
+ )
+ return x
+
+ bsz, block_size, _ = x.size()
+ freqs_cis = self.freqs_cis[start_pos + seqlen : start_pos + seqlen + block_size]
+
+ qr = self.q_norm(self.wq_a(x))
+ q = self.wq_b(qr).unflatten(-1, (self.n_local_heads, self.head_dim))
+ apply_rotary_emb(q[..., -rd:], freqs_cis)
+ kv = self.kv_norm(self.wkv(x))
+ apply_rotary_emb(kv[..., -rd:], freqs_cis)
+ act_quant(kv, fp8_block_size, scale_fmt, scale_dtype, True)
+
+ topk_idxs = get_dspark_topk_idxs(win, bsz, block_size, start_pos)
+ self.window_kv_cache[:bsz, start_pos % win] = main_kv.squeeze(1)
+ kv = torch.cat([self.window_kv_cache[:bsz], kv], dim=1)
+ o = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)
+ apply_rotary_emb(o[..., -rd:], freqs_cis, True)
+
+ o = o.view(bsz, block_size, self.n_local_groups, -1)
+ wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)
+ o = torch.einsum("bsgd,grd->bsgr", o, wo_a)
+ x = self.wo_b(o.flatten(2))
+ return x
+
+
+class DSparkMarkovHead(nn.Module):
+ def __init__(self, vocab_size: int, dspark_markov_rank: int):
+ super().__init__()
+ self.embed = ParallelEmbedding(vocab_size, dspark_markov_rank)
+ self.head = ParallelHead(vocab_size, dspark_markov_rank)
+
+ def forward(self, token_ids: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ embed = self.embed(token_ids)
+ logits = self.head(embed, full_logits=True)
+ return logits, embed
+
+
+class DSparkConfidenceHead(nn.Module):
+ def __init__(self, input_dim: int):
+ super().__init__()
+ # proj in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for fp32 confidence score.
+ self.proj = Linear(input_dim, 1, dtype=torch.float32)
+
+ def forward(self, hidden: torch.Tensor, markov_embed: torch.Tensor):
+ hidden = torch.cat([hidden, markov_embed], dim=-1)
+ return self.proj(hidden.float()).squeeze(-1)
+
+
+class DSparkBlock(Block):
+ """DSpark stage stored under the mtp.* checkpoint namespace."""
+
+ attention_cls = DSparkAttention
+
+ def __init__(self, layer_id: int, args: ModelArgs):
+ super().__init__(layer_id, args)
+ stage_id = layer_id - args.n_layers
+ self.block_size = args.dspark_block_size
+ self.noise_token_id = args.dspark_noise_token_id
+ self.temperature = args.temperature
+ if stage_id == 0:
+ assert len(args.dspark_target_layer_ids) > 0, "DSpark needs target layers"
+ self.main_proj = Linear(args.dim * len(args.dspark_target_layer_ids), args.dim)
+ self.main_norm = RMSNorm(args.dim, args.norm_eps)
+ if stage_id == args.n_mtp_layers - 1:
+ self.norm = RMSNorm(args.dim, args.norm_eps)
+ self.markov_head = DSparkMarkovHead(args.vocab_size, args.dspark_markov_rank)
+ self.confidence_head = DSparkConfidenceHead(args.dim + args.dspark_markov_rank)
+ self.embed: ParallelEmbedding | None = None
+ self.head: ParallelHead | None = None
+
+ def forward(self, x: torch.Tensor, start_pos: int, pre_mix: torch.Tensor, main_x: torch.Tensor):
+ if start_pos == 0:
+ self.attn(x, start_pos, main_x) # prefill only seeds the window KV cache
+ return x, pre_mix
+ return super().forward(x, start_pos, pre_mix, None, main_x) # drafts are text: no VL bias
+
+ def forward_embed(self, main_hidden: torch.Tensor, input_ids: torch.Tensor):
+ assert self.embed is not None
+ main_x = self.main_norm(self.main_proj(main_hidden))
+ draft_input_ids = input_ids.new_full([input_ids.size(0), self.block_size], self.noise_token_id)
+ draft_input_ids[:, 0] = input_ids
+ x = self.embed(draft_input_ids)
+ x = x.unsqueeze(2).repeat(1, 1, self.hc_mult, 1)
+ return x, main_x
+
+ def forward_head(
+ self,
+ x: torch.Tensor,
+ pre_mix: torch.Tensor,
+ input_ids: torch.Tensor,
+ ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ assert self.head is not None
+ x = self.hc_pre(x, pre_mix)
+ logits = self.head(self.norm(x), full_logits=True)
+ output_ids = input_ids.new_empty(input_ids.size(0), self.block_size + 1)
+ output_ids[:, 0] = input_ids
+ markov_embeds = []
+ for i in range(self.block_size):
+ logits_bias, markov_embed = self.markov_head(output_ids[:, i])
+ logits[:, i].add_(logits_bias)
+ markov_embeds.append(markov_embed)
+ output_ids[:, i + 1] = sample(logits[:, i], self.temperature)
+ markov_embed = torch.stack(markov_embeds, dim=1)
+ confidence = self.confidence_head(x, markov_embed)
+ return output_ids, logits, confidence
+
+
+def make_identity_pre_mix(x: torch.Tensor, hc_mult: int) -> torch.Tensor:
+ """initial one-hot mix"""
+ pre_mix = x.new_zeros(x.size(0), x.size(1), hc_mult, dtype=torch.float32)
+ pre_mix[:, :, 0] = 1.0
+ return pre_mix
+
+
+class SharedAttentionRuntime:
+ """What attention layers hand down the stack instead of recomputing. Layers run in order and
+ every source writes before its consumers read, so one slot each is enough and nothing needs
+ resetting between forwards. Sources: compress_kv and index_k from kv_source_layers, topk_idxs
+ from index_source_layers, candidates from candidate_source_layer."""
+
+ def __init__(self):
+ self.compress_kv: torch.Tensor | None = None
+ self.index_k: torch.Tensor | None = None
+ self.topk_idxs: torch.Tensor | None = None
+ self.candidates: torch.Tensor | None = None
+
+
+# Only ever one model per process, same as world_size / rank / default_dtype above.
+shared_attn = SharedAttentionRuntime()
+
+
+class Transformer(nn.Module):
+ """DeepSeek-V4.1: embed -> expand to hc_mult copies -> blocks -> collapse -> logits. Building
+ this sets the globals at the top of the file. The tokenizer only feeds the engram token map."""
+
+ def __init__(self, args: ModelArgs, tokenizer=None):
+ global world_size, rank, default_dtype
+ world_size = dist.get_world_size() if dist.is_initialized() else 1
+ rank = dist.get_rank() if dist.is_initialized() else 0
+ default_dtype = torch.float8_e4m3fn if args.dtype == "fp8" else torch.bfloat16
+ super().__init__()
+ self.max_seq_len = args.max_seq_len
+ self.temperature = args.temperature
+ self.norm_eps = args.norm_eps
+ self.hc_eps = args.hc_eps
+ self.engram_layout = EngramLayout.from_args(args)
+ self.engram_hash = (
+ NgramHashState(args, self.engram_layout, tokenizer) if self.engram_layout is not None else None
+ )
+ self.embed = ParallelEmbedding(args.vocab_size, args.dim)
+ self.layers = torch.nn.ModuleList()
+ for layer_id in range(args.n_layers):
+ self.layers.append(Block(layer_id, args, self.engram_layout))
+ self.norm = RMSNorm(args.dim, self.norm_eps)
+ self.head = ParallelHead(args.vocab_size, args.dim, self.norm_eps, self.hc_eps)
+ self.mtp = torch.nn.ModuleList()
+ self.target_layer_ids = args.dspark_target_layer_ids
+ if args.dspark_block_size:
+ for layer_id in range(args.n_mtp_layers):
+ self.mtp.append(DSparkBlock(args.n_layers + layer_id, args))
+ self.mtp[-1].embed = self.embed
+ self.mtp[-1].head = self.head
+ self.hc_mult = args.hc_mult
+ self.vision = None
+ if args.vision_enabled:
+ self.vision = ViT(args)
+ self.aligner = Aligner(args)
+ # learned embeddings for the image span delimiters
+ self.image_start = nn.Parameter(torch.empty(args.dim))
+ self.image_end = nn.Parameter(torch.empty(args.dim))
+ self.image_newline = nn.Parameter(torch.empty(args.dim))
+
+ @torch.inference_mode()
+ def encode_image(self, patches: torch.Tensor, n_vit_h: int, n_vit_w: int) -> torch.Tensor:
+ return self.aligner(self.vision(patches, n_vit_h, n_vit_w), n_vit_h, n_vit_w)
+
+ def merge_image_embeddings(self, images, h: torch.Tensor):
+ """Overwrite each image's token span in h with its ViT/aligner features. The IMAGE slots take
+ the aligner rows in row-major order; the span delimiters take learned embeddings."""
+ for i, sample in enumerate(images):
+ for img in sample or ():
+ types = img.types.to(h.device)
+ span = h[i, img.start : img.start + types.numel()]
+ span[types == IMAGE_START] = self.image_start.to(h.dtype)
+ span[types == IMAGE_END] = self.image_end.to(h.dtype)
+ span[types == IMAGE_NEW_LINE] = self.image_newline.to(h.dtype)
+ embeds = self.encode_image(img.patches.to(h.device), img.n_vit_h, img.n_vit_w)
+ span[types == IMAGE] = embeds.to(h.dtype)
+
+ @torch.inference_mode()
+ def forward(
+ self, input_ids: torch.Tensor, start_pos: int = 0, images=None, token_types: torch.Tensor | None = None
+ ):
+ """input_ids: [b, s], every entry a real token id -- generate.py only ever passes positions it
+ has already filled, so the padding it uses internally never reaches here. token_types /
+ images carry the VL inputs built by image_processor.prepare_vl_inputs; image spans must lie
+ inside the first (start_pos 0) chunk."""
+ image_mask = None if token_types is None else token_types >= 0 # TEXT is -1
+ # image tokens take no part in an n-gram and get no engram contribution; text-only needs no mask
+ engram_mask = None if image_mask is None else ~image_mask
+ engram_hashes = self.engram_hash(input_ids, start_pos, engram_mask) if self.engram_hash is not None else None
+ h = self.embed(input_ids)
+ if images is not None:
+ assert start_pos == 0, "image spans must be prefilled in a single chunk"
+ self.merge_image_embeddings(images, h)
+ # Expand to hc_mult copies for Hyper-Connections
+ h = h.unsqueeze(2).repeat(1, 1, self.hc_mult, 1)
+ main_hiddens = []
+ pre_mix = make_identity_pre_mix(h, self.hc_mult)
+ for i, layer in enumerate(self.layers):
+ if layer.engram is not None:
+ h = layer.engram(h, engram_hashes[:, :, layer.engram.layer_hash_index, :], engram_mask)
+ # the MTP head reads the attention input of its target layers, not their output
+ if i in self.target_layer_ids:
+ main_hiddens.append(h.mean(dim=2))
+ h, pre_mix = layer(h, start_pos, pre_mix, image_mask)
+ h = layer.hc_pre(h, pre_mix)
+ logits = self.head(self.norm(h))
+ output_ids = sample(logits, self.temperature)
+ main_hidden = torch.cat(main_hiddens, dim=-1) if main_hiddens else None
+ return output_ids, logits, main_hidden
+
+ # FreeToken: Decoder SWA Bounded Replay oracle (tech report sec. 3.2.2): the encoder runs on
+ # every prompt token, the decoder on the last ``n_win`` with its sliding window truncated
+ # there; the decoder's kv source (the last kv_source layer) compresses every token. Returns
+ # the last position's logits and leaves the caches ready for decode from position seqlen.
+ @torch.inference_mode()
+ def forward_bounded(self, input_ids: torch.Tensor, n_win: int):
+ engram_hashes = self.engram_hash(input_ids, 0, None) if self.engram_hash is not None else None
+ h = self.embed(input_ids).unsqueeze(2).repeat(1, 1, self.hc_mult, 1)
+ pre_mix = make_identity_pre_mix(h, self.hc_mult)
+ N = input_ids.size(1)
+ s = max(0, N - n_win)
+ dec_start = max(i for i, layer in enumerate(self.layers) if layer.attn.is_kv_source)
+ for i, layer in enumerate(self.layers):
+ if layer.engram is not None:
+ assert i < dec_start, "Engram sits in the encoder"
+ h = layer.engram(h, engram_hashes[:, :, layer.engram.layer_hash_index, :], None)
+ if i < dec_start:
+ h, pre_mix = layer(h, 0, pre_mix, None)
+ elif i == dec_start:
+ h, pre_mix = layer.forward_tail(None, s, None, h_all=h, pre_mix_all=pre_mix)
+ else:
+ h, pre_mix = layer.forward_tail(h, s, pre_mix)
+ h = layer.hc_pre(h, pre_mix)
+ return self.head(self.norm(h))
+
+ @torch.inference_mode()
+ def forward_spec(self, input_ids: torch.Tensor, main_hidden: torch.Tensor, start_pos: int = 0):
+ h, main_x = self.mtp[0].forward_embed(main_hidden, input_ids)
+ pre_mix = make_identity_pre_mix(h, self.hc_mult)
+ for layer in self.mtp:
+ h, pre_mix = layer(h, start_pos, pre_mix, main_x)
+ if start_pos == 0:
+ return None
+ return self.mtp[-1].forward_head(h, pre_mix, input_ids)
+
+
+def sample(logits, temperature: float = 1.0):
+ """Gumbel-max trick: equivalent to multinomial sampling but faster on GPU,
+ since it avoids the GPU-to-CPU sync in torch.multinomial."""
+ if temperature == 0:
+ return logits.argmax(dim=-1)
+ logits = logits / max(temperature, 1e-5)
+ probs = torch.softmax(logits, dim=-1, dtype=torch.float32)
+ return probs.div_(torch.empty_like(probs).exponential_(1)).argmax(dim=-1)
+
+
+if __name__ == "__main__":
+ torch.set_default_dtype(torch.bfloat16)
+ torch.set_default_device("cuda")
+ torch.manual_seed(0)
+ args = ModelArgs(dspark_block_size=6, dspark_target_layer_ids=(3, 4))
+ x = torch.randint(0, args.vocab_size, (2, 150))
+ model = Transformer(args)
+
+ output_ids, logits, main_hidden = model(x[:, :128])
+ model.forward_spec(output_ids, main_hidden)
+ for i in range(128, 150):
+ output_ids, logits, main_hidden = model(x[:, i : i + 1], i)
+ result = model.forward_spec(output_ids, main_hidden, i)
+ assert result is not None
+ output_ids, logits, confidence = result
diff --git a/tests/models/deepseek_v41/reference/vision.py b/tests/models/deepseek_v41/reference/vision.py
new file mode 100644
index 00000000..2fd86160
--- /dev/null
+++ b/tests/models/deepseek_v41/reference/vision.py
@@ -0,0 +1,121 @@
+# Vendored from deepseek-ai/DeepSeek-V4.1-Flash, revision dba1be0a, inference/vision.py.
+# MIT License; see LICENSE in this directory.
+from functools import lru_cache
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+@lru_cache(8)
+def get_vision_cos_sin(n_h: int, n_w: int, dim: int, theta: float):
+ inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
+ hpos = torch.arange(n_h).unsqueeze(1).expand(n_h, n_w)
+ wpos = torch.arange(n_w).unsqueeze(0).expand(n_h, n_w)
+ freqs = torch.stack([hpos, wpos], dim=-1).reshape(-1, 2, 1).float() * inv_freq
+ freqs = freqs.flatten(1)
+ return freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1)
+
+
+def apply_rotary(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+ dtype = x.dtype
+ x1, x2 = x.float().chunk(2, dim=-1)
+ return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1).to(dtype)
+
+
+class RMSNorm(nn.Module):
+ def __init__(self, dim: int, eps: float = 1e-6):
+ super().__init__()
+ self.eps = eps
+ self.weight = nn.Parameter(torch.ones(dim, dtype=torch.float32))
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ dtype = x.dtype
+ x = x.float()
+ x = x * torch.rsqrt(x.square().mean(-1, keepdim=True) + self.eps)
+ return (self.weight * x).to(dtype)
+
+
+class PatchEmbed(nn.Module):
+ def __init__(self, args):
+ super().__init__()
+ self.proj = nn.Linear(3 * args.vision_patch_size**2, args.vision_dim)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ return self.proj(x.flatten(1))
+
+
+class Attention(nn.Module):
+ def __init__(self, args):
+ super().__init__()
+ self.n_heads = args.vision_n_heads
+ self.head_dim = args.vision_dim // args.vision_n_heads
+ self.wqkv = nn.Linear(args.vision_dim, 3 * args.vision_dim)
+ self.wo = nn.Linear(args.vision_dim, args.vision_dim)
+
+ def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+ n = x.size(0)
+ q, k, v = (t.view(n, self.n_heads, self.head_dim) for t in self.wqkv(x).chunk(3, dim=-1))
+ q = apply_rotary(q, cos, sin)
+ k = apply_rotary(k, cos, sin)
+ o = F.scaled_dot_product_attention(q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1))
+ return self.wo(o.transpose(0, 1).reshape(n, -1))
+
+
+class MLP(nn.Module):
+ def __init__(self, args):
+ super().__init__()
+ self.w1 = nn.Linear(args.vision_dim, 2 * args.vision_inter_dim, bias=False)
+ self.w2 = nn.Linear(args.vision_inter_dim, args.vision_dim, bias=False)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ gate, up = self.w1(x).chunk(2, dim=-1)
+ return self.w2(F.silu(gate) * up)
+
+
+class Block(nn.Module):
+ def __init__(self, args):
+ super().__init__()
+ self.norm1 = RMSNorm(args.vision_dim)
+ self.attn = Attention(args)
+ self.norm2 = RMSNorm(args.vision_dim)
+ self.mlp = MLP(args)
+
+ def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+ x = x + self.attn(self.norm1(x), cos, sin)
+ return x + self.mlp(self.norm2(x))
+
+
+class ViT(nn.Module):
+ """DeepSeek ViT: full bidirectional attention over one image with 2D RoPE."""
+
+ def __init__(self, args):
+ super().__init__()
+ self.rope_dim = args.vision_dim // args.vision_n_heads // 2
+ self.rope_theta = args.vision_rope_theta
+ self.patch_embed = PatchEmbed(args)
+ self.blocks = nn.ModuleList([Block(args) for _ in range(args.vision_n_layers)])
+ self.norm = RMSNorm(args.vision_dim)
+
+ def forward(self, patches: torch.Tensor, n_h: int, n_w: int) -> torch.Tensor:
+ x = self.patch_embed(patches)
+ cos, sin = get_vision_cos_sin(n_h, n_w, self.rope_dim, self.rope_theta)
+ for block in self.blocks:
+ x = block(x, cos, sin)
+ return self.norm(x)
+
+
+class Aligner(nn.Module):
+ def __init__(self, args):
+ super().__init__()
+ self.downsample_ratio = args.vision_downsample_ratio
+ in_dim = args.vision_dim * self.downsample_ratio**2
+ self.w1 = nn.Linear(in_dim, args.dim)
+ self.w2 = nn.Linear(args.dim, args.dim)
+
+ def forward(self, x: torch.Tensor, n_h: int, n_w: int) -> torch.Tensor:
+ r = self.downsample_ratio
+ x = x.view(n_h, n_w, -1).permute(2, 0, 1)
+ x = F.pad(x, (0, -n_w % r, 0, -n_h % r))
+ x = F.unfold(x.unsqueeze(0), r, stride=r).squeeze(0).transpose(0, 1)
+ return self.w2(F.gelu(self.w1(x)))
diff --git a/tests/models/deepseek_v41/test_config.py b/tests/models/deepseek_v41/test_config.py
new file mode 100644
index 00000000..fab385fb
--- /dev/null
+++ b/tests/models/deepseek_v41/test_config.py
@@ -0,0 +1,169 @@
+"""Config parsing, the DSV41 layer-role table, registry dispatch and loader / model key parity."""
+
+from __future__ import annotations
+
+from dataclasses import replace
+
+import pytest
+import torch
+
+from freetoken.layers import set_rope_device
+from freetoken.layers.quantization import NoQuantConfig
+from freetoken.distributed.info import set_tp_info, try_get_tp_info
+from freetoken.models import create_model
+from freetoken.models.deepseek_v41.args import DeepseekV41Args, Mode
+from freetoken.models.deepseek_v41.config import parse_config
+from freetoken.models.register import get_model_spec
+from freetoken.utils.torch_utils import torch_dtype
+
+from .common import ENGRAM_LAYERS, INDEX_SOURCES, KV_SOURCES, N_LAYERS, RATIOS, tiny_hf_config, write_tiny_checkpoint
+
+
+def test_roles_follow_the_dsv41_mode_table():
+ args = parse_config(tiny_hf_config()).dsv41_args
+ modes = [r.mode for r in args.roles]
+ assert modes == [Mode.WINDOW, Mode.WINDOW, Mode.FULL, Mode.REUSE, Mode.FULL, Mode.REINDEX]
+ assert [r.kv_source for r in args.roles] == [None, None, 2, 2, 4, 4]
+ assert args.roles[4].is_candidate_source and args.roles[5].uses_candidates and not args.roles[2].uses_candidates
+ assert args.decoder_start_layer == 4 and args.backbone_kv_sources == (2, 4)
+ assert args.freqs_params(0)[1] == 0 and args.freqs_params(2)[1] == 1024 # yarn only on compressing layers
+ assert args.swa_decoder_replay == "bounded"
+
+
+def test_shipping_layout_roles():
+ """The real 40-layer layout: 3 encoder Full layers (ratio 2), the decoder Full at 20 building the
+ candidate pool, Reindex at 24/28/32/36 inside it, Reuse elsewhere."""
+ text = tiny_hf_config().text_config
+ text.update(
+ num_hidden_layers=40, compress_ratios=[0, 0] + [2] * 18 + [1] * 20 + [0, 0, 0],
+ kv_source_layer_ids=[2, 8, 14, 20], index_source_layer_ids=[2, 8, 14, 20, 24, 28, 32, 36],
+ candidate_source_layer_id=20, num_nextn_predict_layers=3, engram_layer_ids=[1, 14], engram_num_embeddings=[10, 10],
+ )
+ args = DeepseekV41Args.from_hf(tiny_hf_config(**text))
+ full = [r.layer_id for r in args.roles if r.mode is Mode.FULL]
+ reindex = [r.layer_id for r in args.roles if r.mode is Mode.REINDEX]
+ assert full == [2, 8, 14, 20] and reindex == [24, 28, 32, 36]
+ assert sum(r.mode is Mode.REUSE for r in args.roles) == 40 - 2 - 4 - 4
+ assert [r.layer_id for r in args.roles if r.uses_candidates] == reindex
+ assert args.decoder_start_layer == 20
+
+
+def test_bad_layouts_are_rejected():
+ with pytest.raises(ValueError, match="no kv source"):
+ DeepseekV41Args.from_hf(tiny_hf_config(kv_source_layer_ids=[4]))
+ with pytest.raises(ValueError, match="reads kv source"):
+ DeepseekV41Args.from_hf(tiny_hf_config(compress_ratios=[0, 0, 2, 1, 1, 1, 0]))
+ with pytest.raises(ValueError, match="different kv source"):
+ DeepseekV41Args.from_hf(tiny_hf_config(compress_ratios=[0, 0, 2, 2, 2, 2, 0], kv_source_layer_ids=[2, 4], candidate_source_layer_id=2, index_source_layer_ids=[2, 4, 5]))
+
+
+def test_registry_and_kv_sources():
+ spec = get_model_spec("DeepseekV41ForCausalLM")
+ assert spec.module == "freetoken.models.deepseek_v41"
+ from freetoken.attention import AttnType
+ from freetoken.models.config import DSV4AttentionGroupConfig
+
+ mc = parse_config(tiny_hf_config())
+ args = mc.dsv41_args
+ assert args.backbone_kv_sources == (2, 4) and tuple(r.ratio for r in args.roles) == tuple(RATIOS[:N_LAYERS])
+ (group,) = mc.attention_groups
+ assert type(group) is DSV4AttentionGroupConfig and group.variant == "v41"
+ assert mc.kv_cache_group_specs()[0].attn_type is AttnType.DSV41 and mc.attn_type_for_layer(0) is AttnType.DSV41
+ assert mc.num_layers == N_LAYERS and mc.dsv41_args is not None
+
+
+def _meta_model(mc):
+ if try_get_tp_info() is None:
+ set_tp_info(0, 1)
+ set_rope_device(torch.device("cpu"))
+ mc = replace(mc, moe_strategy="offload", decode_target="gpu", quant=NoQuantConfig())
+ with torch.device("meta"), torch_dtype(torch.bfloat16):
+ return create_model(mc)
+
+
+def test_loader_names_every_model_parameter(tmp_path):
+ from freetoken.models.deepseek_v41.weight import iter_weights
+
+ write_tiny_checkpoint(str(tmp_path))
+ model = _meta_model(parse_config(tiny_hf_config()))
+ params = model.state_dict()
+ loaded = dict(iter_weights(str(tmp_path), torch.device("cpu"), include_moe_experts=False))
+ assert set(loaded) == set(params), (set(loaded) ^ set(params))
+ for name, tensor in loaded.items():
+ assert tuple(tensor.shape) == tuple(params[name].shape), name
+ # routed experts, the DSpark stage, the vision tower and the image routing bias are not resident weights
+ assert not any(".experts." in k or k.startswith(("mtp.", "vision.")) or k.endswith("bias_vl") for k in loaded)
+ # per-role structure
+ assert model.model.layers.op_list[3].attn.compressor is None and model.model.layers.op_list[3].attn.indexer is None
+ assert model.model.layers.op_list[4].attn.compressor.ratio == 1 and model.model.layers.op_list[2].attn.compressor.ratio == 2
+ assert [b.engram is not None for b in model.model.layers.op_list] == [L in ENGRAM_LAYERS for L in range(N_LAYERS)]
+ assert all(L in INDEX_SOURCES for L in range(N_LAYERS) if model.model.layers.op_list[L].attn.indexer is not None)
+ assert all(L in KV_SOURCES for L in range(N_LAYERS) if model.model.layers.op_list[L].attn.compressor is not None)
+
+
[email protected]("missing_weight", [False, True])
+def test_loader_closes_shards_when_expert_loading_stops(tmp_path, monkeypatch, missing_weight):
+ from freetoken.models.deepseek_v41 import weight
+
+ write_tiny_checkpoint(str(tmp_path))
+ closed = []
+ close, get = weight._ShardReader.close, weight._ShardReader.get
+
+ def close_reader(reader):
+ close(reader)
+ closed.append(True)
+
+ def read_weight(reader, name):
+ if missing_weight and name.endswith(".w3.weight"):
+ raise KeyError(name)
+ return get(reader, name)
+
+ monkeypatch.setattr(weight._ShardReader, "close", close_reader)
+ monkeypatch.setattr(weight._ShardReader, "get", read_weight)
+ tensors = weight.iter_weights(str(tmp_path), torch.device("cpu"))
+ if missing_weight:
+ with pytest.raises(KeyError):
+ next(tensors)
+ else:
+ next(tensors)
+ tensors.close()
+ assert closed == [True]
+
+
+def test_fp8_wo_a_layer_rejects_a_dequantized_ftw_layout_clearly():
+ """An fp8-declared wo_a fed a state dict without ``wo_a_scale`` (an FTW converted before wo_a kept
+ its payload) fails with a reconvert message, not a raw missing-key error; a bf16-declared layer fed
+ the fp8 payload + scale dequantizes it on the way in."""
+ from types import SimpleNamespace
+
+ from freetoken.layers.quantization.scheme import fp8_block_scheme
+ from freetoken.models.deepseek_v41.attention import DSV41Attention
+
+ if try_get_tp_info() is None:
+ set_tp_info(0, 1)
+ args = parse_config(tiny_hf_config()).dsv41_args
+ args.max_seq_len = 512
+ # wo_a is fp8; the LinearReplicated modules stay unquantized (the bf16 quant method)
+ plain = NoQuantConfig()
+ quant = SimpleNamespace(
+ scheme_for=lambda name: fp8_block_scheme("ue8m0", 32) if name.endswith(".wo_a") else None,
+ get_quant_method=lambda layer, prefix: plain.get_quant_method(layer, prefix),
+ )
+ with torch.device("meta"), torch_dtype(torch.bfloat16):
+ fp8_layer = DSV41Attention(args, args.roles[2], quant_config=quant, prefix="attn")
+ bf16_layer = DSV41Attention(args, args.roles[2], quant_config=None, prefix="attn")
+ assert fp8_layer.wo_a.dtype == torch.float8_e4m3fn and bf16_layer.wo_a.dtype == torch.bfloat16
+ rows, cols = fp8_layer.wo_a.shape
+ with torch.device("meta"):
+ state = {k: torch.empty_like(v) for k, v in fp8_layer.state_dict(prefix="attn").items()}
+ del state["attn.wo_a_scale"]
+ state["attn.wo_a"] = torch.empty(rows, cols, dtype=torch.bfloat16, device="meta")
+ with pytest.raises(RuntimeError, match="reconvert"):
+ fp8_layer.load_state_dict(dict(state), prefix="attn")
+ # the bf16 layer given the fp8 payload + scale dequantizes (real tensors: the dequant runs)
+ w = (torch.randn(rows, cols) * 0.05).to(torch.float8_e4m3fn)
+ scale = torch.full((rows // 32, cols // 32), 127, dtype=torch.uint8).view(torch.float8_e8m0fnu)
+ state2 = {k: torch.zeros(v.shape, dtype=v.dtype) for k, v in bf16_layer.state_dict(prefix="attn").items()}
+ state2["attn.wo_a"], state2["attn.wo_a_scale"] = w, scale
+ bf16_layer.load_state_dict(state2, prefix="attn")
+ assert torch.equal(bf16_layer.wo_a, w.to(torch.bfloat16))
diff --git a/tests/models/deepseek_v41/test_engine_config.py b/tests/models/deepseek_v41/test_engine_config.py
new file mode 100644
index 00000000..116209ea
--- /dev/null
+++ b/tests/models/deepseek_v41/test_engine_config.py
@@ -0,0 +1,81 @@
+"""Engine config resolution for DeepSeek-V4.1 on the tiny checkpoint: the DSV41 backend and pool are
+picked, the page size is the window page, the replay knob lands on the model args, and the fp8
+dialect (block 32, fp4 experts) reaches the quant layer with the matching activation block."""
+
+from __future__ import annotations
+
+import json
+import os
+
+import pytest
+import torch
+
+from freetoken.distributed import DistributedInfo
+from freetoken.scheduler.config import SchedulerConfig
+from freetoken.engine.engine import _adjust_config
+from freetoken.kvcache import resolve_pool_class
+from freetoken.kvcache.dsv4.v41_pool import DSV41PagedKVCache
+
+from .common import write_tiny_checkpoint
+
+
+def _engine_config(path, **over):
+ return SchedulerConfig(model_path=path, tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, **over)
+
+
[email protected]("replay", ["bounded", "exact"])
+def test_resolution_picks_dsv41_and_the_replay_knob(tmp_path, monkeypatch, replay):
+ from freetoken.engine import engine
+
+ monkeypatch.setattr(engine, "is_sm100_family", lambda: False)
+ monkeypatch.setattr(engine, "is_sm90_family", lambda: True)
+ write_tiny_checkpoint(str(tmp_path))
+ config = _engine_config(str(tmp_path), attention_backend="auto", moe_strategy="offload", swa_decoder_replay=replay, max_seq_len_override=2048)
+ _adjust_config(config)
+ assert config.attention_backend == "dsv41_sparse"
+ assert config.page_size == 128 and config.cache_type == "swa_radix"
+ assert resolve_pool_class(config.model_config) is DSV41PagedKVCache
+ args = config.model_config.dsv41_args
+ assert args.swa_decoder_replay == replay and args.max_seq_len == 2048 and args.max_batch_size == config.max_running_req + 1
+ assert config.max_extend_tokens == 8192 # the prefill chunk stays bounded (whole window pages)
+ # the cache contract follows the replay mode: a bounded-mode prefix hit must leave the prompt's last
+ # window to the prefill (the pool's prefix_replay_tokens); the history a resume reads stays one window
+ from freetoken.kvcache.dsv4.v41_cost_model import dsv41_pool_sizes
+
+ spec = config.model_config.kv_cache_group_specs()[0]
+ pool = DSV41PagedKVCache(dsv41_pool_sizes(16, args, 1.0, 128), args, torch.device("cpu"))
+ assert pool.prefix_replay_tokens == (128 if replay == "bounded" else 0)
+ assert pool.sliding_window_size == 128 and spec.sliding_window == 128 and not spec.is_swa
+ # bounded replay keeps the decoder's per-request window KV in private rings, off the shared pages
+ assert pool.private_window_layer_ids == (tuple(range(args.decoder_start_layer, args.n_layers)) if replay == "bounded" else ())
+ assert DSV41PagedKVCache.min_kv_tokens(config) // 128 == 8 + 3 * config.max_running_req + 2 * (config.max_running_req + 1) + 1
+
+
+def test_fp8_block32_dialect_reaches_the_quant_layer(tmp_path):
+ """A V4.1-style quantization_config on the tiny checkpoint: dense linears get the 32-block
+ e8m0 scheme, routed experts the MXFP4 scheme with a 32-wide activation block."""
+ from freetoken.distributed.info import set_tp_info, try_get_tp_info
+ from freetoken.layers import OffloadMoELayer
+ from freetoken.layers.quantization import Fp8BlockConfig, QuantKind
+ from freetoken.layers.quantization.scheme import fp8_block_size
+
+ write_tiny_checkpoint(str(tmp_path))
+ cfg_path = os.path.join(str(tmp_path), "config.json")
+ with open(cfg_path) as f:
+ cfg = json.load(f)
+ cfg["quantization_config"] = {"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4"}
+ with open(cfg_path, "w") as f:
+ json.dump(cfg, f)
+ config = _engine_config(str(tmp_path), moe_strategy="offload")
+ quant = config.model_config.quant
+ assert type(quant) is Fp8BlockConfig and quant.block == 32
+ dense = quant.scheme_for("model.layers.3.attn.wq_a")
+ assert dense.kind is QuantKind.FP8_BLOCK and fp8_block_size(dense) == 32
+ if try_get_tp_info() is None:
+ set_tp_info(0, 1)
+ experts = OffloadMoELayer(3, num_experts=4, top_k=2, hidden_size=64, intermediate_size=64, limit=10.0, quant_config=quant, prefix="model.layers.3.ffn.experts")
+ assert experts.quant_method.scheme.kind is QuantKind.MXFP4 and experts.quant_method.cfg.act_block == 32
+ # the family's bf16 modules stay unquantized although the fp8 config lists no exceptions
+ for name in ("model.layers.2.attn.compressor.wkv", "model.layers.2.attn.indexer.wk", "model.layers.5.attn.indexer.weights_proj", "head"):
+ assert quant.scheme_for(name) is None, name
+ assert quant.scheme_for("model.layers.1.engram.wkv").kind is QuantKind.FP8_BLOCK
diff --git a/tests/models/deepseek_v41/test_engram.py b/tests/models/deepseek_v41/test_engram.py
new file mode 100644
index 00000000..dec852ec
--- /dev/null
+++ b/tests/models/deepseek_v41/test_engram.py
@@ -0,0 +1,281 @@
+"""Engram: the hash against a transcription of the reference ``NgramHashState``, the prime buckets,
+the disk table against a torch oracle over the synthetic checkpoint (eager, decode with context,
+CUDA-graph flag sync), and the layer's gate math against the reference ``Engram.forward``."""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+import torch
+
+from freetoken.models.deepseek_v41.args import DeepseekV41Args
+from freetoken.models.deepseek_v41.engram import EngramHash, bucket_primes, hash_multipliers
+
+from .common import VOCAB, requires_cuda, tiny_hf_config, write_tiny_checkpoint
+
+
+def _args(**over) -> DeepseekV41Args:
+ return DeepseekV41Args.from_hf(tiny_hf_config(**over))
+
+
+def _reference_hashes(args: DeepseekV41Args, ids: torch.Tensor, start_pos: int, cache: torch.Tensor) -> torch.Tensor:
+ """``NgramHashState.forward`` for one sequence with an identity token map; ``cache`` is the
+ sequence's compressed-id history buffer (mutated like the reference's)."""
+ layout_primes = bucket_primes(args)
+ flat = [[p for per_ngram in layer for p in per_ngram] for layer in layout_primes]
+ offsets = torch.tensor(np.array([np.cumsum([0, *sizes[:-1]]) for sizes in flat]))
+ primes = torch.tensor(layout_primes)
+ multipliers = hash_multipliers(args.engram_layer_ids, args.engram_max_ngram_size, args.engram_compressed_vocab_size)
+ pad_id = args.engram_pad_id
+ seqlen = ids.numel()
+ cache[start_pos : start_pos + seqlen] = ids
+ positions = torch.arange(start_pos, start_pos + seqlen)
+ tokens, blocked = [], torch.zeros_like(positions, dtype=torch.bool)
+ for shift in range(args.engram_max_ngram_size):
+ source = cache[(positions - shift).clamp_min(0)]
+ blocked = blocked | (positions < shift)
+ tokens.append(torch.where(blocked, pad_id, source))
+ tokens = torch.stack(tokens, dim=-1)
+ products = tokens.unsqueeze(1) * multipliers
+ rolling, hashes = products[..., 0], []
+ for i in range(1, args.engram_max_ngram_size):
+ rolling = torch.bitwise_xor(rolling, products[..., i])
+ hashes.append(rolling.unsqueeze(-1) % primes[:, i - 1])
+ return torch.cat(hashes, dim=-1) + offsets
+
+
+def test_primes_are_distinct_and_ordered():
+ args = _args()
+ primes = bucket_primes(args)
+ flat = [p for layer in primes for per in layer for p in per]
+ assert len(flat) == len(set(flat)) and flat == sorted(flat) and min(flat) > args.engram_vocab_size - 1
+ from sympy import isprime
+
+ assert all(isprime(p) for p in flat)
+
+
+def test_hash_matches_the_reference_across_a_prefill_decode_split():
+ args = _args()
+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)
+ ids = torch.randint(3, VOCAB, (37,), generator=torch.Generator().manual_seed(0))
+ cache = torch.empty(64, dtype=torch.int64)
+ want = _reference_hashes(args, ids[:30], 0, cache)
+ got = hash.row_ids(ids[:30])
+ assert torch.equal(got, want)
+ # decode continues with the three preceding tokens as context
+ for step in range(30, 37):
+ want = _reference_hashes(args, ids[step : step + 1], step, cache)
+ got = hash.row_ids(ids[step : step + 1], context=ids[:step])
+ assert torch.equal(got, want), step
+ # a run resumed mid-sequence (chunked prefill) hashes like the uninterrupted one
+ whole = hash.row_ids(ids)
+ assert torch.equal(torch.cat([hash.row_ids(ids[:20]), hash.row_ids(ids[20:], context=ids[:20])]), whole)
+ assert whole.shape == (37, 1, hash.n_cols) and int(whole.max()) < hash.rows_per_layer[0]
+ with pytest.raises(ValueError, match="compressed vocab"):
+ EngramHash(args, list(range(VOCAB)), VOCAB + 1)
+
+
+@requires_cuda
+def test_disk_table_matches_the_dequantized_shard(tmp_path):
+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, engram_row_source
+
+ tensors = write_tiny_checkpoint(str(tmp_path))
+ args = _args()
+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)
+ src = engram_row_source(str(tmp_path), 1)
+ assert src.num_rows == tensors["layers.1.engram.embed.weight"].shape[0] and src.head_dim == 32
+ table = EngramDiskTable(src, hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=16)
+ fp8 = tensors["layers.1.engram.embed.weight"]
+ scale = torch.exp2(tensors["layers.1.engram.embed.scale"].view(torch.uint8).float() - 127.0)
+
+ def oracle(rows: torch.Tensor) -> torch.Tensor:
+ vals = fp8[rows.flatten()].float().view(-1, 32 // 32, 32) * scale[rows.flatten()].unsqueeze(-1)
+ return vals.view(rows.shape[0], -1).to(torch.bfloat16).cuda()
+
+ ids = torch.randint(3, VOCAB, (40,), generator=torch.Generator().manual_seed(1))
+ rows = hash.row_ids(ids)[:, 0]
+ table.stage(rows, graph=False) # larger than the initial eager staging: grows
+ table.flush(signal=False)
+ assert torch.equal(table.lookup(40), oracle(rows))
+ # the graph staging + flag protocol: capture a lookup, fill after the launch, replay
+ from freetoken.kernel.row_store import probe_wait_sync
+
+ table.wait_sync = probe_wait_sync("auto", torch.device("cuda"))
+ dec_rows = hash.row_ids(ids[40 - 1 :], context=ids[: 40 - 1])[:, 0]
+ out = torch.empty(1, hash.n_cols * 32, dtype=torch.bfloat16, device="cuda")
+ graph = torch.cuda.CUDAGraph()
+ stream = torch.cuda.Stream()
+ with torch.cuda.stream(stream):
+ if not table.wait_sync:
+ table.stage(dec_rows, graph=True)
+ table.flush(signal=False)
+ with torch.cuda.graph(graph, stream=stream):
+ out.copy_(table.lookup(1))
+ torch.cuda.synchronize()
+ graph.replay() # under wait-sync the replay blocks until the host signals
+ table.stage(dec_rows, graph=True)
+ table.flush(signal=table.wait_sync)
+ torch.cuda.synchronize()
+ assert torch.equal(out, oracle(dec_rows))
+
+
+@requires_cuda
+def test_eager_staging_does_not_overwrite_an_in_flight_lookup(tmp_path):
+ """The overlap scheduler stages batch k+1 while batch k's lookup copy is still queued behind
+ the layers before the Engram layer (a sleep kernel here): batch k must read its own rows."""
+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, engram_row_source
+
+ tensors = write_tiny_checkpoint(str(tmp_path))
+ hash = EngramHash(_args(), list(range(VOCAB)), VOCAB)
+ table = EngramDiskTable(engram_row_source(str(tmp_path), 1), hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=64)
+ fp8 = tensors["layers.1.engram.embed.weight"]
+ scale = torch.exp2(tensors["layers.1.engram.embed.scale"].view(torch.uint8).float() - 127.0)
+
+ def oracle(rows):
+ vals = fp8[rows.flatten()].float().view(-1, 1, 32) * scale[rows.flatten()].unsqueeze(-1)
+ return vals.view(rows.shape[0], -1).to(torch.bfloat16).cuda()
+
+ gen = torch.Generator().manual_seed(0)
+ batches = [hash.row_ids(torch.randint(3, VOCAB, (40,), generator=gen))[:, 0] for _ in range(4)]
+ got = []
+ for rows in batches:
+ table.stage(rows, graph=False)
+ table.flush(signal=False)
+ torch.cuda._sleep(50_000_000)
+ got.append(table.lookup(rows.shape[0]))
+ torch.cuda.synchronize()
+ for i, (rows, out) in enumerate(zip(batches, got)):
+ assert torch.equal(out, oracle(rows)), f"batch {i} read another batch's rows"
+
+
+@requires_cuda
+def test_host_fills_a_graph_decode_step_from_a_worker_thread(tmp_path):
+ """The deferred graph-decode fill must not wait for the launch call to return: a 40-layer graph
+ blocks cuGraphLaunch on the host while the GPU sits on the WAIT, so the host coordinator reads the
+ tokens back and signals from a worker thread. The worker hashes against the DISPATCH snapshot of
+ the request state: advancing the request right after dispatch (what the engine does) must not
+ change which rows are fetched."""
+ from types import SimpleNamespace
+
+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, EngramHost, engram_row_source
+
+ tensors = write_tiny_checkpoint(str(tmp_path))
+ args = _args()
+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)
+ table = EngramDiskTable(engram_row_source(str(tmp_path), 1), hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=16)
+ host = EngramHost(hash, [table], torch.device("cuda"))
+ if not host.wait_sync:
+ pytest.skip("stream memops unavailable: launch-gating mode needs no worker")
+ ids = torch.randint(3, VOCAB, (20,), generator=torch.Generator().manual_seed(7))
+ req = SimpleNamespace(input_ids=ids.clone(), device_len=20, cached_len=19)
+ batch = SimpleNamespace(is_decode=True, is_prefill=False, padded_reqs=[req], padded_size=1, input_ids=ids[19:20].to(torch.int32).cuda())
+ out = torch.empty(1, hash.n_cols * 32, dtype=torch.bfloat16, device="cuda")
+ stream = torch.cuda.Stream()
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.stream(stream):
+ with torch.cuda.graph(graph, stream=stream):
+ out.copy_(table.lookup(1))
+ torch.cuda.synchronize()
+ with torch.cuda.stream(stream):
+ # slow the readback so the request advances (complete_one + append) before the worker runs
+ torch.cuda._sleep(200_000_000)
+ with host.forward_host_ctx(batch, use_graph=True):
+ graph.replay()
+ req.device_len += 1
+ req.input_ids = torch.cat([req.input_ids, torch.tensor([5], dtype=ids.dtype)])
+ torch.cuda.synchronize()
+ fp8 = tensors["layers.1.engram.embed.weight"]
+ rows = hash.row_ids(ids[19:20], context=ids[:19])[:, 0].flatten()
+ want = fp8[rows].float().view(1, -1).to(torch.bfloat16).cuda() # scales are all 1.0 in the synthetic table
+ assert torch.equal(out, want)
+ host.close()
+
+
+@requires_cuda
+def test_host_surfaces_a_failed_deferred_fill_at_dispatch_exit(tmp_path):
+ from types import SimpleNamespace
+
+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, EngramHost, engram_row_source
+
+ write_tiny_checkpoint(str(tmp_path))
+ args = _args()
+ hash = EngramHash(args, list(range(VOCAB)), VOCAB)
+ table = EngramDiskTable(engram_row_source(str(tmp_path), 1), hash.n_cols, torch.device("cuda"), max_graph_rows=4, max_extend_tokens=16)
+ host = EngramHost(hash, [table], torch.device("cuda"))
+ if not host.wait_sync:
+ pytest.skip("stream memops unavailable")
+ ids = torch.randint(3, VOCAB, (8,), generator=torch.Generator().manual_seed(8))
+ req = SimpleNamespace(input_ids=ids, device_len=8, cached_len=7)
+ batch = SimpleNamespace(is_decode=True, is_prefill=False, padded_reqs=[req], padded_size=1, input_ids=ids[7:8].to(torch.int32).cuda())
+ stream = torch.cuda.Stream()
+ graph = torch.cuda.CUDAGraph()
+ out = torch.empty(1, hash.n_cols * 32, dtype=torch.bfloat16, device="cuda")
+ with torch.cuda.stream(stream):
+ with torch.cuda.graph(graph, stream=stream):
+ out.copy_(table.lookup(1))
+ torch.cuda.synchronize()
+ table.stage = lambda *a, **k: (_ for _ in ()).throw(RuntimeError("disk gone")) # the fill fails
+ with pytest.raises(RuntimeError, match="disk gone"):
+ with torch.cuda.stream(stream):
+ with host.forward_host_ctx(batch, use_graph=True):
+ graph.replay()
+ torch.cuda.synchronize() # the failing worker released the flag: the stream is not stuck
+ host.close()
+
+
+def test_ftw_side_files_carry_the_tables_and_reopen(tmp_path):
+ """An FTW conversion copies the shards holding the Engram tables next to it (the converter's
+ ``ftw_side_files`` hook, which skips the HF index); the row source resolves the copies through
+ their headers exactly like the HF shards, byte for byte."""
+ from freetoken.kernel.row_store import RowStore
+ from freetoken.models.deepseek_v41.engram_table import engram_row_source
+ from freetoken.models.deepseek_v41.weight import ftw_side_files
+
+ src, out = tmp_path / "hf", tmp_path / "ftw"
+ out.mkdir()
+ tensors = write_tiny_checkpoint(str(src))
+ (out / "config.json").write_bytes((src / "config.json").read_bytes()) # the converter copies the metadata
+ shard = "model-00001-of-00001.safetensors" # the tiny checkpoint's single shard holds the table
+ assert ftw_side_files(str(src), str(out)) == [shard]
+ assert (out / shard).read_bytes() == (src / shard).read_bytes() and not (out / "model.safetensors.index.json").exists()
+ source = engram_row_source(str(out), 1)
+ assert source.path == str(out / shard) and source.num_rows == 4096 and source.head_dim == 32
+ assert torch.equal(source.scales, tensors["layers.1.engram.embed.scale"].view(torch.uint8))
+ store = RowStore(paths=[source.path], extent_file=[0], extent_base=[source.base], rows_per_extent=source.num_rows,
+ row_bytes=source.head_dim, row_stride=source.head_dim, use_io_uring=False)
+ ids = torch.tensor([0, 7, 4095], dtype=torch.int64)
+ buf = torch.zeros(3 * 32, dtype=torch.uint8)
+ store.stage_rows(ids.data_ptr(), 3, buf.data_ptr(), 0)
+ store.flush(0)
+ assert torch.equal(buf.view(3, 32), tensors["layers.1.engram.embed.weight"][ids].view(torch.uint8))
+
+
+def test_layer_gate_matches_the_reference():
+ """``EngramLayer.forward`` against the reference ``Engram.forward`` (bf16 wkv, fp32 gate math)."""
+ from freetoken.distributed.info import set_tp_info, try_get_tp_info
+ from freetoken.models.deepseek_v41.engram import EngramLayer
+
+ if try_get_tp_info() is None:
+ set_tp_info(0, 1)
+ args = _args()
+ torch.manual_seed(0)
+ layer = EngramLayer(args, 1)
+ T, hc, dim = 5, args.hc_mult, args.dim
+ wkv = torch.randn(dim * (hc + 1), layer.width, dtype=torch.bfloat16) * 0.05
+ layer.wkv.weight = wkv
+ layer.q_weight = torch.rand(hc, dim) + 0.5
+ layer.k_weight = torch.rand(hc, dim) + 0.5
+ x = torch.randn(T, hc, dim, dtype=torch.bfloat16)
+ rows = torch.randn(T, layer.width, dtype=torch.bfloat16)
+
+ kv = torch.nn.functional.linear(rows, wkv)
+ key, value = kv.split([hc * dim, dim], dim=-1)
+ key = key.float().view(T, hc, dim)
+ weight = layer.q_weight * layer.k_weight
+ h = x.float()
+ rstd = torch.rsqrt(h.square().mean(-1) + args.norm_eps) * torch.rsqrt(key.square().mean(-1) + args.norm_eps)
+ dot = (h * weight * key).sum(-1) * rstd * dim**-0.5
+ gate = torch.sigmoid(torch.copysign(dot.abs().clamp_min(1e-6).sqrt(), dot))
+ want = (h + gate.unsqueeze(-1) * value.float().unsqueeze(-2)).to(x.dtype)
+ assert torch.equal(layer.forward(x, rows), want)
diff --git a/tests/models/deepseek_v41/test_forward_smoke.py b/tests/models/deepseek_v41/test_forward_smoke.py
new file mode 100644
index 00000000..e0861e88
--- /dev/null
+++ b/tests/models/deepseek_v41/test_forward_smoke.py
@@ -0,0 +1,184 @@
+"""End-to-end forward smoke on the tiny model: cold prefill, batched decode, a second request, chunked
+prefill and bounded replay all produce finite logits of the right shape (numerics: test_reference_parity)."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+from .common import VOCAB, requires_cuda, write_tiny_checkpoint
+
+pytestmark = requires_cuda
+
+
[email protected](scope="module")
+def checkpoint(tmp_path_factory):
+ folder = tmp_path_factory.mktemp("dsv41-tiny")
+ write_tiny_checkpoint(str(folder))
+ return str(folder)
+
+
+def _tokens(n: int, seed: int) -> list[int]:
+ g = torch.Generator().manual_seed(seed)
+ return torch.randint(3, VOCAB, (n,), generator=g).tolist()
+
+
+def test_prefill_and_decode(checkpoint):
+ from .harness import TinyEngine
+
+ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")
+ r0 = eng.new_request(0, _tokens(300, 1))
+ logits = eng.prefill([r0])
+ assert logits.shape == (1, VOCAB) and torch.isfinite(logits).all()
+ eng.finish_prefill([r0])
+ r1 = eng.new_request(1, _tokens(50, 2))
+ assert torch.isfinite(eng.prefill([r1])).all()
+ eng.finish_prefill([r1])
+ for step in range(5): # batched decode over two requests at different positions
+ out = eng.decode([r0, r1], [10 + step, 20 + step])
+ assert out.shape == (2, VOCAB) and torch.isfinite(out).all()
+
+
+def test_chunked_prefill_matches_single_shot(checkpoint):
+ """Two 128-aligned chunks must reproduce the single-shot prefill's last-token logits: the
+ compressor carry, the window ring and the compressed rows are all resumed through the pool. The
+ chunks run their projections at other row counts, so the match is to rounding, not bitwise."""
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare
+
+ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")
+ toks = _tokens(300, 3)
+ whole = eng.new_request(0, toks)
+ ref = eng.prefill([whole])
+ chunked = eng.new_request(1, toks)
+ chunked.device_len = 256 # first chunk [0, 256)
+ eng.prefill([chunked])
+ chunked.cached_len, chunked.device_len = 256, 300
+ got = eng.prefill([chunked])
+ _compare("chunked prefill", got, ref)
+
+
+def test_bounded_replay_is_exact_within_one_window(checkpoint):
+ """A prompt no longer than the window replays every token with no floor: bounded == exact."""
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare
+
+ toks = _tokens(100, 5)
+ exact = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")
+ want = exact.prefill([exact.new_request(0, toks)])
+ bounded = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")
+ got = bounded.prefill([bounded.new_request(0, toks)])
+ _compare("bounded within one window", got, want)
+
+
[email protected]("cut", [128, 256])
+def test_bounded_chunked_prefill_matches_unchunked(checkpoint, cut):
+ """A chunked bounded prefill gives each chunk its part of the prompt's last window ``[172, 300)``:
+ a first chunk that ends before it (cut 128) runs no decoder row and hands the head a placeholder;
+ one that reaches into it (cut 256) leaves rows in the private ring for the last chunk, which is
+ shorter than the window (44 tokens). Both match the unchunked prefill."""
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare
+
+ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")
+ toks = _tokens(300, 6)
+ ref = eng.prefill([eng.new_request(0, toks)])
+ chunked = eng.new_request(1, toks)
+ chunked.device_len = cut
+ first = eng.prefill([chunked])
+ assert first.shape == (1, VOCAB) and torch.isfinite(first).all()
+ chunked.cached_len, chunked.device_len = cut, 300
+ _compare(f"bounded chunks cut at {cut}", eng.prefill([chunked]), ref)
+
+
+def test_bounded_batch_mixes_a_decoderless_chunk_with_a_whole_prompt(checkpoint):
+ """One batch: a chunk that ends before its prompt's last window (no decoder row, a placeholder for
+ the head) next to a whole prompt; the whole prompt's logits are those of its own prefill."""
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare
+
+ eng = TinyEngine(checkpoint, max_seq_len=2048, max_running_req=2, swa_decoder_replay="bounded")
+ whole = _tokens(300, 8)
+ want = eng.prefill([eng.new_request(0, whole)])
+ chunk = eng.new_request(0, _tokens(600, 9))
+ chunk.device_len = 256
+ other = eng.new_request(1, whole)
+ got = eng.prefill([chunk, other])
+ assert got.shape == (2, VOCAB) and torch.isfinite(got).all()
+ _compare("whole prompt next to a decoderless chunk", got[1:], want)
+
+
+def test_bounded_prefix_hit_matches_cold_prefill(checkpoint):
+ """A prefix hit that would leave fewer than a window of new tokens (768 cached + 20 new) is capped
+ one window before the prompt end, so the request prefills ``[640, 788)`` -- the decoder's whole
+ window among it -- and produces the cold prefill's logits."""
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare
+
+ eng = TinyEngine(checkpoint, max_seq_len=2048, max_running_req=2, swa_decoder_replay="bounded")
+ prefix = _tokens(768, 11)
+ donor = eng.new_request(0, prefix + _tokens(20, 12))
+ cold = eng.prefill([donor])
+ eng.finish_prefill([donor])
+ hit = eng.new_request_on_prefix(1, donor, prefix + _tokens(20, 12)) # the same prompt over the shared pages
+ assert hit.cached_len == 640 and hit.extend_len == 148
+ got = eng.prefill([hit])
+ _compare("bounded prefix hit", got, cold)
+
+
+def test_shared_prefix_replays_do_not_disturb_each_other(checkpoint):
+ """Two requests sharing a 768-token prefix with different suffix lengths: B's hit is capped at 640
+ (a window before its prompt end), its decoder writes only B's private ring and it reads the shared
+ encoder history, so A's pending decode sees the logits it would have seen without B, and neither
+ the shared pages nor A's ring change."""
+ from .harness import TinyEngine
+
+ prefix = _tokens(768, 13)
+ a_toks, b_toks = prefix + _tokens(20, 14), prefix + _tokens(40, 15)
+
+ def run(interleave: bool):
+ eng = TinyEngine(checkpoint, max_seq_len=2048, max_running_req=2, swa_decoder_replay="bounded")
+ a = eng.new_request(0, a_toks)
+ eng.prefill([a])
+ eng.finish_prefill([a])
+ dec_layer, enc_layer = eng.args.decoder_start_layer, 2
+ shared_slots = eng.pool.translate_full_to_window(eng.page_table[0, 640:768].long())
+ enc_before = eng.pool.window_pool[enc_layer][shared_slots].clone()
+ P = eng.args.window_size
+ ring_before = eng.pool.window_pool[dec_layer][0 * P : 1 * P].clone() # A's ring row
+ if interleave:
+ b = eng.new_request_on_prefix(1, a, b_toks)
+ eng.prefill([b])
+ eng.finish_prefill([b])
+ assert torch.equal(eng.pool.window_pool[enc_layer][shared_slots], enc_before)
+ assert torch.equal(eng.pool.window_pool[dec_layer][0 * P : 1 * P], ring_before)
+ return eng.decode([a], [7])
+
+ assert torch.equal(run(True), run(False))
+
+
+def test_commit_dedup_onto_a_longer_prompts_pages_keeps_the_decoder_state(checkpoint):
+ """The radix commit may replace a request's freshly written pages with an existing node's (the
+ reviewer's donor case): D prefilled 256 tokens, so under bounded replay its decoder never computed
+ positions [0, 128); A (exactly D's first 128 tokens) is repointed onto D's pages at commit. A's
+ decoder KV lives in A's own ring, so its next decode is unchanged by the repoint (D's shared rows
+ came from a longer prefill, so to rounding)."""
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare
+
+ donor_toks = _tokens(256, 21)
+ a_toks = donor_toks[:128]
+
+ def run(repoint: bool):
+ eng = TinyEngine(checkpoint, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")
+ d = eng.new_request(0, donor_toks)
+ eng.prefill([d])
+ eng.finish_prefill([d])
+ a = eng.new_request(1, a_toks)
+ eng.prefill([a])
+ eng.finish_prefill([a])
+ if repoint: # what CacheManager.cache_req does when the tree already holds the prefix
+ eng.page_table[1, :128] = eng.page_table[0, :128]
+ return eng.decode([a], [3])
+
+ _compare("decode after a commit repoint", run(True), run(False))
diff --git a/tests/models/deepseek_v41/test_reference_parity.py b/tests/models/deepseek_v41/test_reference_parity.py
new file mode 100644
index 00000000..839cdafb
--- /dev/null
+++ b/tests/models/deepseek_v41/test_reference_parity.py
@@ -0,0 +1,219 @@
+"""Logits parity against the vendored reference implementation on the tiny synthetic checkpoint.
+
+The reference runs bf16 or FP8/FP4 weights with its own torch quantizers, sparse attention, engram
+hash and single-pass mHC; FreeToken runs the same checkpoint through its kernels and packed pools.
+Compared: exact-mode prefill, greedy decode steps, a two-request batch, and the bounded-replay path
+against the reference's ``forward_bounded`` oracle.
+"""
+
+from __future__ import annotations
+
+from dataclasses import asdict, fields
+
+import pytest
+import torch
+
+from .common import VOCAB, requires_cuda, tiny_text_config, write_tiny_checkpoint
+
+pytestmark = requires_cuda
+
+# The sparse-attention implementations accumulate in different orders.
+ATOL = RTOL = 1e-2
+# The fp4 experts also round the routed sum to bf16 before the shared-expert add and weight each
+# expert's down output; the reference keeps that sum fp32 and weights the intermediate before its fp8
+# round-trip (10 seeds x 16 checks: max |err| 0.019, against 0.007 in the reference's order).
+QUANT_TOL = 2e-2
+
+
[email protected](scope="module")
+def checkpoint(tmp_path_factory):
+ folder = tmp_path_factory.mktemp("dsv41-parity")
+ tensors = write_tiny_checkpoint(str(folder), seed=11)
+ return str(folder), tensors
+
+
+def _tokens(n: int, seed: int) -> torch.Tensor:
+ return torch.randint(3, VOCAB, (n,), generator=torch.Generator().manual_seed(seed))
+
+
+class Reference:
+ """The vendored ``Transformer`` on CUDA, loaded from the synthetic checkpoint (identity token map)."""
+
+ def __init__(self, tensors: dict, text: dict, *, max_seq_len: int, max_batch_size: int, quantized: bool = False, vision_config: dict | None = None):
+ from freetoken.models.deepseek_v41.args import DeepseekV41Args
+ from types import SimpleNamespace
+
+ from . import reference
+ from .reference import engram as ref_engram
+ from .reference import model as ref_model
+
+ args = DeepseekV41Args.from_hf(SimpleNamespace(text_config=text))
+ names = {f.name for f in fields(ref_model.ModelArgs)}
+ kwargs = {k: v for k, v in asdict(args).items() if k in names}
+ kwargs.update(dtype="fp8" if quantized else "bf16", expert_dtype="fp4" if quantized else None,
+ max_seq_len=max_seq_len, max_batch_size=max_batch_size, vision_n_layers=0, temperature=0.0)
+ if vision_config:
+ kwargs.update(vision_n_layers=vision_config["num_hidden_layers"], vision_dim=vision_config["hidden_size"],
+ vision_n_heads=vision_config["num_attention_heads"], vision_inter_dim=vision_config["intermediate_size"],
+ vision_patch_size=vision_config["patch_size"], vision_downsample_ratio=vision_config["downsample_ratio"])
+ self.margs = ref_model.ModelArgs(**kwargs)
+ # the synthetic checkpoint's tokenizer is the identity over VOCAB ids
+ ref_engram.build_compressed_token_map = lambda tokenizer: (list(range(VOCAB)), VOCAB)
+ torch.set_default_dtype(torch.bfloat16)
+ try:
+ with torch.device("cuda"):
+ self.model = ref_model.Transformer(self.margs, tokenizer=object())
+ finally:
+ torch.set_default_dtype(torch.float32)
+ state = {k: v for k, v in tensors.items() if not k.startswith("mtp.")
+ and (vision_config or (not k.startswith("vision.") and not k.endswith("bias_vl")))}
+ if quantized:
+ state = {k: v.view(torch.float4_e2m1fn_x2) if ".ffn.experts." in k and k.endswith(".weight") else v
+ for k, v in state.items()}
+ # The reference's grouped output projection holds dequantized bf16 weights.
+ for name in list(state):
+ if name.endswith(".wo_a.scale"):
+ scale = state.pop(name).float().repeat_interleave(32, 0).repeat_interleave(32, 1)
+ weight_name = name.removesuffix(".scale") + ".weight"
+ state[weight_name] = (state[weight_name].float() * scale).bfloat16()
+ missing, unexpected = self.model.load_state_dict({k: v.cuda() for k, v in state.items()}, strict=False)
+ assert not unexpected, unexpected
+ assert not [m for m in missing if "freqs_cis" not in m], missing
+ self.reference = reference
+
+ # the reference builds its index tensors on the default device (generate.py sets it to cuda)
+ def prefill(self, ids: torch.Tensor) -> torch.Tensor:
+ with torch.device("cuda"):
+ return self.model(ids.view(1, -1).cuda(), 0)[1].float()
+
+ def prefill_batch(self, ids: torch.Tensor) -> torch.Tensor:
+ with torch.device("cuda"):
+ return self.model(ids.cuda(), 0)[1].float()
+
+ def prefill_bounded(self, ids: torch.Tensor) -> torch.Tensor:
+ with torch.device("cuda"):
+ return self.model.forward_bounded(ids.reshape(-1, ids.shape[-1]).cuda(), self.margs.window_size).float()
+
+ def decode(self, tokens: torch.Tensor, pos: int) -> torch.Tensor:
+ with torch.device("cuda"):
+ return self.model(tokens.view(-1, 1).cuda(), pos)[1].float()
+
+
+def _engram_table_for(engine, tensors, text):
+ """Our engine reads the Engram rows straight from the synthetic shard (the disk table), hashed
+ with the same identity token map the reference oracle uses."""
+ from freetoken.models.deepseek_v41.engram import EngramHash
+ from freetoken.models.deepseek_v41.engram_table import EngramDiskTable, EngramHost, engram_row_source
+
+ hash = EngramHash(engine.args, list(range(VOCAB)), VOCAB)
+ tables = [
+ EngramDiskTable(engram_row_source(engine.checkpoint, layer.layer_id), hash.n_cols, engine.device, max_graph_rows=8, max_extend_tokens=64)
+ for layer in engine.model.engram_layers()
+ ]
+ for layer, table in zip(engine.model.engram_layers(), tables):
+ layer.attach_table(table)
+ host = EngramHost(hash, tables, engine.device)
+ engine.model.forward_host_ctx = host.forward_host_ctx
+ engine.engram_host = host
+
+
+def _compare(name, got, want, *, tol=ATOL):
+ err = (got - want).abs().max().item()
+ assert torch.equal(got.argmax(-1), want.argmax(-1)), f"{name}: argmax differs (max abs err {err:.4f})"
+ torch.testing.assert_close(got, want, atol=tol, rtol=tol, msg=lambda m: f"{name}: {m}")
+
+
[email protected]("mode", ["exact", "bounded"])
[email protected]("batch_size", [1, 2])
+def test_quantized_offload_prefill_and_decode_match_reference(tmp_path, mode, batch_size):
+ from .harness import TinyEngine
+
+ tensors = write_tiny_checkpoint(str(tmp_path), seed=11, quantized=True)
+ text = tiny_text_config(moe_intermediate_size=256)
+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3, quantized=True)
+ eng = TinyEngine(str(tmp_path), swa_decoder_replay=mode, quantized=True)
+ _engram_table_for(eng, tensors, text)
+ ids = torch.stack([_tokens(300, 1 + i) for i in range(batch_size)])
+ want = ref.prefill_batch(ids) if mode == "exact" else ref.prefill_bounded(ids)
+ reqs = [eng.new_request(i, row.tolist()) for i, row in enumerate(ids)]
+ got = eng.prefill(reqs)
+ _compare(f"quantized {mode} prefill", got, want, tol=QUANT_TOL)
+ eng.finish_prefill(reqs)
+ for step in range(3):
+ token = want.argmax(-1)
+ want = ref.decode(token, ids.shape[-1] + step)
+ got = eng.decode(reqs, token.tolist())
+ _compare(f"quantized {mode} decode {step}", got, want, tol=QUANT_TOL)
+
+
+def test_exact_prefill_and_greedy_decode_match_the_reference(checkpoint):
+ from .harness import TinyEngine
+
+ folder, tensors = checkpoint
+ text = tiny_text_config()
+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3)
+ eng = TinyEngine(folder, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")
+ _engram_table_for(eng, tensors, text)
+
+ ids = _tokens(300, 1)
+ want = ref.prefill(ids)
+ req = eng.new_request(0, ids.tolist())
+ got = eng.prefill([req])
+ _compare("prefill", got, want)
+ eng.finish_prefill([req])
+ # greedy decode from the reference's own picks keeps both sides on one trajectory
+ pos = 300
+ for step in range(12):
+ nxt = int(want.argmax(-1).item())
+ want = ref.decode(torch.tensor([nxt]), pos)
+ got = eng.decode([req], [nxt])
+ _compare(f"decode step {step}", got, want)
+ pos += 1
+
+
+def test_batched_prefill_and_decode_match_the_reference(checkpoint):
+ from .harness import TinyEngine
+
+ folder, tensors = checkpoint
+ text = tiny_text_config()
+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3)
+ eng = TinyEngine(folder, max_seq_len=1024, max_running_req=2, swa_decoder_replay="exact")
+ _engram_table_for(eng, tensors, text)
+
+ ids = torch.stack([_tokens(200, 2), _tokens(200, 3)])
+ want = ref.prefill_batch(ids)
+ reqs = [eng.new_request(i, ids[i].tolist()) for i in range(2)]
+ got = eng.prefill(reqs)
+ _compare("batched prefill", got, want)
+ eng.finish_prefill(reqs)
+ pos = 200
+ for step in range(6):
+ nxt = want.argmax(-1).cpu()
+ want = ref.decode(nxt, pos)
+ got = eng.decode(reqs, nxt.tolist())
+ _compare(f"batched decode step {step}", got, want)
+ pos += 1
+
+
+def test_bounded_replay_matches_the_reference_oracle(checkpoint):
+ from .harness import TinyEngine
+
+ folder, tensors = checkpoint
+ text = tiny_text_config()
+ ref = Reference(tensors, text, max_seq_len=1024, max_batch_size=3)
+ eng = TinyEngine(folder, max_seq_len=1024, max_running_req=2, swa_decoder_replay="bounded")
+ _engram_table_for(eng, tensors, text)
+
+ ids = _tokens(300, 4)
+ want = ref.prefill_bounded(ids)
+ req = eng.new_request(0, ids.tolist())
+ got = eng.prefill([req])
+ _compare("bounded prefill", got, want)
+ eng.finish_prefill([req])
+ pos = 300
+ for step in range(6):
+ nxt = int(want.argmax(-1).item())
+ want = ref.decode(torch.tensor([nxt]), pos)
+ got = eng.decode([req], [nxt])
+ _compare(f"decode after bounded prefill, step {step}", got, want)
+ pos += 1
diff --git a/tests/models/deepseek_v41/test_vision.py b/tests/models/deepseek_v41/test_vision.py
new file mode 100644
index 00000000..0e1d4a48
--- /dev/null
+++ b/tests/models/deepseek_v41/test_vision.py
@@ -0,0 +1,214 @@
+"""Image layout, official ViT/logits parity and image replay through the shared encoder cache."""
+
+from __future__ import annotations
+
+import io
+import json
+from dataclasses import asdict
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+import torch
+from PIL import Image
+from safetensors.torch import save_file
+
+from freetoken.core import Batch
+from freetoken.engine.engine import Engine
+from freetoken.mm.config import MultimodalConfig
+from freetoken.mm.encoder_cache import EncoderCache
+from freetoken.mm.processors.deepseek_v41 import DeepseekV41MMProcessor
+from freetoken.models.deepseek_v41.config import VisionConfig
+from freetoken.scheduler.mm import mm_rows_after, plan_mm_batch
+from freetoken.utils.torch_utils import torch_dtype
+
+from .common import DIM, VOCAB, requires_cuda, tiny_text_config, write_tiny_checkpoint
+from .reference.vision import Aligner, ViT
+
+VC = asdict(VisionConfig(num_hidden_layers=2, hidden_size=32, num_attention_heads=4, intermediate_size=48,
+ patch_size=2, min_pixels=0, max_image_tokens=64))
+
+
+def _args():
+ return SimpleNamespace(vision_n_layers=2, vision_dim=32, vision_n_heads=4, vision_inter_dim=48,
+ vision_patch_size=2, vision_downsample_ratio=3, vision_rope_theta=10000.0, dim=DIM)
+
+
+def _checkpoint(path, quantized=False):
+ tensors = write_tiny_checkpoint(str(path), seed=11, quantized=quantized)
+ torch.manual_seed(12)
+ with torch_dtype(torch.bfloat16):
+ vit, aligner = ViT(_args()), Aligner(_args())
+ # Replace the unused dummy vision weight from the text-only fixture.
+ tensors = {k: v for k, v in tensors.items() if not k.startswith("vision.")}
+ for prefix, module in (("vision", vit), ("aligner", aligner)):
+ tensors.update({f"{prefix}.{k}": v.detach().bfloat16().contiguous() for k, v in module.state_dict().items()})
+ tensors.update({k: torch.randn(DIM, dtype=torch.bfloat16) for k in ("image_start", "image_end", "image_newline")})
+ save_file(tensors, path / 'model-00001-of-00001.safetensors')
+ (path / 'model.safetensors.index.json').write_text(json.dumps({'weight_map': {k: 'model-00001-of-00001.safetensors' for k in tensors}}))
+ config = json.loads((path / 'config.json').read_text())
+ config.update(vision_config=VC, image_token_id=VOCAB - 1)
+ (path / 'config.json').write_text(json.dumps(config))
+ return tensors
+
+
+def _processor():
+ cfg = SimpleNamespace(vision_config=SimpleNamespace(**VC), image_token_id=VOCAB - 1)
+ return DeepseekV41MMProcessor(cfg, '.', MultimodalConfig())
+
+
+def _image(color):
+ buf = io.BytesIO()
+ Image.new('RGB', (14, 10), color).save(buf, format='PNG')
+ return buf.getvalue()
+
+
+def _inputs(length=151):
+ ids = [3] * 17 + [VOCAB - 1] + [4] * (length - 19) + [VOCAB - 1]
+ return _processor().apply(torch.tensor(ids, dtype=torch.int32), [_image('red'), _image('blue')])
+
+
+def _prefill(eng, reqs):
+ if not hasattr(eng, 'encoder_cache'):
+ eng.encoder_cache = EncoderCache()
+ eng.dtype = torch.bfloat16
+ eng._mm_registered = set()
+ for req in reqs:
+ if id(req) not in eng._mm_registered:
+ for item in req.mm_items or (): # what the scheduler claims at admission
+ eng.encoder_cache.register(item.hash, req.uid, mm_rows_after(item, req.cached_len))
+ eng._mm_registered.add(id(req))
+ batch = Batch(reqs=reqs, phase='prefill')
+ batch.padded_reqs = reqs
+ batch.input_ids = torch.cat([r.input_ids[r.cached_len:r.device_len] for r in reqs]).cuda()
+ batch.positions = torch.cat([torch.arange(r.cached_len, r.device_len) for r in reqs]).cuda()
+ jobs, plan, rows, _ = plan_mm_batch(reqs, eng.encoder_cache)
+ batch.mm_encoder_jobs, batch.mm_gather_plan = jobs, plan
+ if plan:
+ batch.mm_rows = torch.tensor(rows, dtype=torch.long, device=eng.device)
+ Engine._run_mm_encoder(eng, batch)
+ eng._bind()
+ eng.backend.prepare_metadata(batch)
+ with eng.ctx.forward_batch(batch), eng.model.forward_host_ctx(batch, False):
+ return eng.model.forward().float()
+
+
+def test_processor_patch_order_and_all_span_rows_are_embeddings():
+ pixels = np.arange(10 * 14 * 3, dtype=np.uint8).reshape(10, 14, 3)
+ item = _processor().process([Image.fromarray(pixels)])[0]
+ assert item.grid_thw == [1, 5, 7]
+ expected = ((torch.tensor(pixels[:2, :2]).permute(2, 0, 1).float() / 255 - .5) / .5).bfloat16()
+ assert torch.equal(item.feature[0], expected)
+ assert len(_processor().prompt_replacement(item).full) == 10
+ result = _inputs()
+ assert [i.offsets for i in result.mm_items] == [[[17, 27]], [[159, 169]]]
+ assert result.mrope_positions is None
+ assert result.mm_items[0].hash != result.mm_items[1].hash
+
+
+def test_text_only_skips_every_visual_parameter(tmp_path):
+ from freetoken.models.deepseek_v41.weight import iter_weights
+ from freetoken.mm.processor import get_mm_processor
+ from .test_engine_config import _engine_config
+ _checkpoint(tmp_path)
+ enabled = _engine_config(str(tmp_path), moe_strategy='offload')
+ assert enabled.model_config.is_multimodal
+ disabled = MultimodalConfig(disabled_encoders=frozenset({'vision', 'audio'}))
+ text = _engine_config(str(tmp_path), moe_strategy='offload', mm=disabled)
+ assert not text.model_config.is_multimodal and not text.active_encoders
+ assert get_mm_processor(str(tmp_path), disabled) is None
+ loaded = dict(iter_weights(str(tmp_path), 'cpu', include_moe_experts=False, include_vision=False))
+ assert not any(k.startswith('visual.') for k in loaded)
+ assert get_mm_processor(str(tmp_path)).placeholder == [VOCAB - 1]
+
+
+@requires_cuda
[email protected]('placement', ['gpu', 'host'])
+def test_vision_encoder_matches_official_with_padded_grid(tmp_path, placement):
+ from .harness import TinyEngine
+ tensors = _checkpoint(tmp_path)
+ eng = TinyEngine(str(tmp_path))
+ eng.model.place_encoder_weights(placement)
+ item = _processor().process([Image.new('RGB', (14, 10), 'red')])[0]
+ with torch_dtype(torch.bfloat16), torch.device('cuda'):
+ vit, aligner = ViT(_args()), Aligner(_args())
+ for prefix, module in (('vision', vit), ('aligner', aligner)):
+ module.load_state_dict({k.removeprefix(prefix + '.'): v.cuda() for k, v in tensors.items() if k.startswith(prefix + '.')})
+ with torch.inference_mode(), torch.device('cuda'):
+ x = aligner(vit(item.feature.cuda(), 5, 7), 5, 7)
+ got = eng.model.encode(item)
+ torch.testing.assert_close(got[1:-1].view(2, 4, DIM)[:, :3].reshape(-1, DIM), x, atol=1e-3, rtol=1e-3)
+ assert torch.equal(got[0], tensors['image_start'].cuda())
+ assert torch.equal(got[-1], tensors['image_end'].cuda())
+ assert torch.equal(got[[4, 8]], tensors['image_newline'].cuda().expand(2, -1))
+ eng.model.place_encoder_weights('gpu')
+
+
+@requires_cuda
[email protected]('quantized', [False, True])
+def test_image_logits_and_decode_match_reference(tmp_path, quantized):
+ from .harness import TinyEngine
+ from .test_reference_parity import ATOL, QUANT_TOL, Reference, _compare, _engram_table_for
+ tol = QUANT_TOL if quantized else ATOL
+ tensors = _checkpoint(tmp_path, quantized)
+ text = tiny_text_config(moe_intermediate_size=256) if quantized else tiny_text_config()
+ ref = Reference(tensors, text, max_seq_len=512, max_batch_size=2, quantized=quantized, vision_config=VC)
+ eng = TinyEngine(str(tmp_path), max_seq_len=512, swa_decoder_replay='exact', quantized=quantized)
+ _engram_table_for(eng, tensors, text)
+ result = _inputs()
+ ids = result.input_ids.clone()
+ types = torch.full_like(ids, -1)
+ images = []
+ for item in result.mm_items:
+ lo, hi = item.offsets[0]
+ ids[lo:hi] = VOCAB - 1
+ typ = torch.tensor([0] + ([1] * 3 + [2]) * 2 + [3])
+ types[lo:hi] = typ
+ images.append(SimpleNamespace(start=lo, patches=item.feature, n_vit_h=5, n_vit_w=7, types=typ))
+ with torch.device('cuda'):
+ want = ref.model(ids.cuda()[None], images=[images], token_types=types.cuda()[None])[1]
+ req = eng.new_request(0, result.input_ids.tolist())
+ req.mm_items = result.mm_items
+ got = _prefill(eng, [req])
+ _compare('image prefill', got, want, tol=tol)
+ eng.finish_prefill([req])
+ for step in range(3):
+ token = want.argmax(-1)
+ want = ref.decode(token, len(ids) + step)
+ got = eng.decode([req], token.tolist())
+ _compare(f'image decode {step}', got, want, tol=tol)
+
+
+@requires_cuda
[email protected]('mode', ['exact', 'bounded'])
+def test_images_survive_chunking_and_prefix_replay(tmp_path, mode):
+ from .harness import TinyEngine
+ from .test_reference_parity import _compare, _engram_table_for
+ tensors = _checkpoint(tmp_path)
+ eng = TinyEngine(str(tmp_path), max_seq_len=2048, swa_decoder_replay=mode)
+ _engram_table_for(eng, tensors, tiny_text_config())
+ # bounded mode caps a hit a window before the prompt end: trailing text keeps that cap at 128
+ tail = 150 if mode == 'bounded' else 0
+ def inputs():
+ ids = [3] * 80 + [VOCAB - 1] + [4] * 37 + [VOCAB - 1] + [4] * 31 + [VOCAB - 1] + [4] * tail
+ return _processor().apply(torch.tensor(ids, dtype=torch.int32), [_image(c) for c in ('red', 'blue', 'green')])
+ whole = inputs()
+ assert [i.offsets for i in whole.mm_items] == [[[80, 90]], [[127, 137]], [[168, 178]]]
+ a = eng.new_request(0, whole.input_ids.tolist())
+ a.mm_items = whole.mm_items
+ want = _prefill(eng, [a])
+ chunk = inputs()
+ b = eng.new_request(1, chunk.input_ids.tolist())
+ b.mm_items = chunk.mm_items
+ b.device_len = 128
+ _prefill(eng, [b])
+ b.cached_len, b.device_len = 128, len(chunk.input_ids)
+ got = _prefill(eng, [b])
+ _compare('chunked image prefill', got, want)
+ prefix = inputs()
+ # The second image straddles the chunk / prefix boundary at 128 (in bounded mode the first chunk
+ # also ends before the prompt's last window and runs no decoder row).
+ hit = eng.new_request_on_prefix(1, a, prefix.input_ids.tolist())
+ assert hit.cached_len == 128
+ hit.mm_items = prefix.mm_items
+ _compare('image prefix hit', _prefill(eng, [hit]), want)
diff --git a/tests/models/test_quant_config.py b/tests/models/test_quant_config.py
index 084a2e4a..bfcfb876 100644
--- a/tests/models/test_quant_config.py
+++ b/tests/models/test_quant_config.py
@@ -1,6 +1,7 @@
"""What each checkpoint's quant config resolves to: the dialect, the method behind every layer, and agreement with the stored tensors.
-Rows need the checkpoint's config.json locally (/mnt/nvme/models or the HF cache); absent ones skip.
+Rows need the checkpoint's config.json locally (/mnt/nvme/models or the HF cache); absent ones skip. The
+stored-tensor scan also reads $FREETOKEN_TEST_MODELS_ROOT.
"""
from __future__ import annotations
@@ -47,6 +48,8 @@ from freetoken.utils.hf import sidecar_quantization_config
from freetoken.utils.torch_utils import torch_dtype
MODELS = "/mnt/nvme/models"
+# the stored-tensor scan takes every checkpoint under this root (the case table above matches by name)
+SCAN_ROOT = os.environ.get("FREETOKEN_TEST_MODELS_ROOT", MODELS)
HF_CACHE = os.path.expanduser("~/.cache/huggingface/hub")
BF16, FP8B, FP8T, MXFP8, NVFP4 = UnquantizedLinearMethod, Fp8BlockLinearMethod, Fp8TensorLinearMethod, Mxfp8LinearMethod, Nvfp4LinearMethod
@@ -435,6 +438,25 @@ def test_modelopt_nvfp4_reads_the_activation_quantizer_from_config_groups(extra,
assert scheme.kind is QuantKind.NVFP4 and scheme.has("input_scale") is has_input_scale
+def test_fp8_dialect_reads_the_block_size_and_the_nested_expert_dtype():
+ """DeepSeek-V4.1 keeps the ``fp8`` dialect with 32x32 blocks and puts ``expert_dtype`` inside quantization_config."""
+ from freetoken.layers.quantization.scheme import fp8_block_size
+
+ q = {"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4"}
+ quant = QuantConfig.from_hf(SimpleNamespace(quantization_config=q))
+ assert type(quant) is Fp8BlockConfig and quant.block == 32
+ dense = quant.scheme_for("layers.3.attn.wq_a")
+ assert dense.kind is QuantKind.FP8_BLOCK and fp8_block_size(dense) == 32 and dense.weight.scale == "e8m0"
+ assert quant.storage(dense)["weight_scale_inv"].name == "scale"
+ assert quant.scheme_for("layers.3.ffn.experts.7.w1").kind is QuantKind.MXFP4
+ # the V4 128-block export still resolves to the 128 scheme
+ q128 = {"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [128, 128], "scale_fmt": "ue8m0"}
+ assert fp8_block_size(QuantConfig.from_hf(SimpleNamespace(quantization_config=q128)).scheme_for("layers.3.attn.wq_a")) == 128
+ for bad in ([64, 64], [32, 128], [128]):
+ with pytest.raises(NotImplementedError):
+ QuantConfig.from_hf(SimpleNamespace(quantization_config={**q, "weight_block_size": bad}))
+
+
def test_every_dialect_names_the_tensors_behind_its_schemes():
from freetoken.layers.quantization.registry import dialects
@@ -491,7 +513,7 @@ def _weight_map(p: Path) -> dict[str, str] | None:
def _candidate_dirs() -> list[Path]:
- roots = glob.glob(os.path.join(HF_CACHE, "models--*/snapshots/*/")) + glob.glob(os.path.join(MODELS, "*/"))
+ roots = glob.glob(os.path.join(HF_CACHE, "models--*/snapshots/*/")) + glob.glob(os.path.join(SCAN_ROOT, "*/"))
return [Path(d) for d in roots if (Path(d) / "config.json").exists() and _weight_map(Path(d)) is not None]
diff --git a/tests/moe/test_cpu_moe.py b/tests/moe/test_cpu_moe.py
index f496de31..547c94c6 100644
--- a/tests/moe/test_cpu_moe.py
+++ b/tests/moe/test_cpu_moe.py
@@ -391,6 +391,7 @@ def test_cpu_decode_dsfp4_matches_gpu(bs):
max_tokens=bs,
device=dev,
swiglu_limit=swiglu_limit,
+ act_block=128,
)
hidden = torch.randn(bs, H, device=dev, dtype=torch.bfloat16)
@@ -585,7 +586,7 @@ def test_cpu_moe_decode_cuda_graph_replay_dsfp4():
ex = CpuMoeExecutor(
cache, top_k=top_k, activation="silu",
apply_router_weight_on_input=False, num_threads=8, max_tokens=bs, device=dev,
- swiglu_limit=limit,
+ swiglu_limit=limit, act_block=128,
)
b = cache.bank_sources
layer_banks = [b[n][layer].to(dev) for n in (
diff --git a/tests/moe/test_dsfp4_prequant.py b/tests/moe/test_dsfp4_prequant.py
index 6d6b49d3..11519502 100644
--- a/tests/moe/test_dsfp4_prequant.py
+++ b/tests/moe/test_dsfp4_prequant.py
@@ -46,7 +46,7 @@ def test_dsfp4_gpu_prequant_bit_parity():
def run(cpu_roundtrip: bool) -> torch.Tensor:
ex = CpuMoeExecutor(
cache, top_k=TOPK, activation="silu", apply_router_weight_on_input=False,
- num_threads=4, max_tokens=2, device=dev, swiglu_limit=10.0,
+ num_threads=4, max_tokens=2, device=dev, swiglu_limit=10.0, act_block=128,
)
assert ex._gpu_prequant, "ds_fp4 on CUDA should enable GPU prequant"
if cpu_roundtrip: # control: the original CPU-side scalar round-trip
diff --git a/tests/moe/test_dsfp4_reference_semantics.py b/tests/moe/test_dsfp4_reference_semantics.py
new file mode 100644
index 00000000..2f89cf33
--- /dev/null
+++ b/tests/moe/test_dsfp4_reference_semantics.py
@@ -0,0 +1,170 @@
+"""ds_fp4 (DeepSeek W4A8) experts against an INDEPENDENT torch transcription over
+``fp4_gemm(act_quant(x, block), W)`` at both fp8 activation blocks (128 on V4, 32 on V4.1):
+
+ x_q = fp8_roundtrip(x, block) # act_quant(..., inplace)
+ gate = bf16(x_q @ deq(W1)^T), up = bf16(x_q @ deq(W3)^T) # fp4_gemm outputs bf16
+ h = bf16(silu(min(gate, L)) * clamp(up, -L, L))
+ y_r = bf16(w_r * (fp8_roundtrip(h, block) @ deq(W2)^T)) # routing weight in the down epilogue
+ y = sum_r y_r
+
+The routing weight scales the down output (the placement main serves V4 with); the reference
+``Expert.forward`` scales the intermediate before its fp8 quant instead. The GEMV (decode), the
+grouped GEMM (prefill) and the CPU executor all follow this transcription.
+"""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+E, H, I, TOP_K, LIMIT = 16, 512, 256, 4, 7.0
+E2M1 = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0])
+
+
+def _banks(device, seed=0):
+ g = torch.Generator(device="cpu").manual_seed(seed)
+
+ def u8(*shape, low=0, high=256):
+ return torch.randint(low, high, shape, dtype=torch.uint8, generator=g).to(device)
+
+ return (u8(E, 2 * I, H // 2), u8(E, 2 * I, H // 32, low=120, high=130), u8(E, H, I // 2), u8(E, H, I // 32, low=120, high=130))
+
+
+def _dequant(packed: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:
+ """[E, N, K//2] e2m1 pairs (even channel low nibble) + [E, N, K//32] e8m0 -> [E, N, K] fp32."""
+ lut = E2M1.to(packed.device)
+ lo, hi = lut[(packed & 0xF).long()], lut[(packed >> 4).long()]
+ vals = torch.stack([lo, hi], dim=-1).flatten(-2) # [E, N, K]
+ s = torch.exp2(scale.view(torch.uint8).float() - 127.0).repeat_interleave(32, dim=-1)
+ return vals * s
+
+
+def _fp8_roundtrip(x: torch.Tensor, block: int) -> torch.Tensor:
+ g = x.float().unflatten(-1, (-1, block))
+ s = torch.exp2(torch.ceil(torch.log2(g.abs().amax(-1).clamp_min(1e-4) / 448.0)))
+ q = (g / s.unsqueeze(-1)).clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float()
+ return (q * s.unsqueeze(-1)).flatten(-2).to(torch.bfloat16)
+
+
+def reference(x, slots, weights, banks, block):
+ """The MoE over dequantized banks: per-route bf16 expert outputs accumulated into an fp32 ``y``,
+ returned in fp32 (the kernels round that sum to bf16)."""
+ gu_p, gu_s, dn_p, dn_s = banks
+ W13 = _dequant(gu_p, gu_s) # [E, 2I, H]
+ W2 = _dequant(dn_p, dn_s) # [E, H, I]
+ xq = _fp8_roundtrip(x, block).float()
+ T = x.shape[0]
+ y = torch.zeros(T, H, dtype=torch.float32, device=x.device)
+ for t in range(T):
+ for r in range(TOP_K):
+ e = int(slots[t, r])
+ gu = (xq[t] @ W13[e].T).to(torch.bfloat16).float()
+ gate, up = gu[:I].clamp(max=LIMIT), gu[I:].clamp(-LIMIT, LIMIT)
+ h = (torch.nn.functional.silu(gate) * up).to(torch.bfloat16)
+ hq = _fp8_roundtrip(h.view(1, -1), block).float().view(-1)
+ y[t] += (weights[t, r].float() * (hq @ W2[e].T)).to(torch.bfloat16).float()
+ return y
+
+
+def _inputs(T, device, seed=1):
+ g = torch.Generator(device="cpu").manual_seed(seed)
+ x = (torch.randn(T, H, dtype=torch.bfloat16, generator=g) * 0.5).to(device)
+ slots = torch.stack([torch.randperm(E, generator=g)[:TOP_K] for _ in range(T)]).to(device=device, dtype=torch.int32).contiguous()
+ w = torch.rand(T, TOP_K, generator=g)
+ w = (1.5 * w / w.sum(-1, keepdim=True)).float().to(device).contiguous() # route_scale-style weights > 1 too
+ return x, slots, w
+
+
[email protected]("block", [32, 128])
+def test_gemv_path_matches_the_transcription(block):
+ from freetoken.moe.fused_ds_fp4 import routed_experts_fp4
+
+ banks = _banks("cuda")
+ x, slots, w = _inputs(6, "cuda")
+ got = routed_experts_fp4(x, slots, w, *banks, LIMIT, act_block=block).float()
+ want = reference(x, slots, w, banks, block)
+ torch.testing.assert_close(got, want, atol=2e-2, rtol=2e-2)
+
+
[email protected]("block", [32, 128])
+def test_cpu_executor_matches_the_transcription(block):
+ """The ``ds_fp4`` CPU executor (cpu / hybrid decode) implements the same contract as the GPU path."""
+ from types import SimpleNamespace
+
+ from freetoken.kernel.pinned import alloc_pinned_tensor
+ from freetoken.moe.cpu_executor import CpuMoeExecutor
+
+ gu_p, gu_s, dn_p, dn_s = _banks("cuda")
+ pinned = {}
+ for name, t in (("gate_up_packed", gu_p), ("gate_up_scale", gu_s), ("down_packed", dn_p), ("down_scale", dn_s)):
+ buf = alloc_pinned_tensor(*t.shape, dtype=torch.uint8)
+ buf.copy_(t.cpu())
+ pinned[name] = buf
+ cache = SimpleNamespace(quant_format="ds_fp4", bank_sources={k: [v] for k, v in pinned.items()}, num_layers=1, num_experts=E)
+ dev = torch.device("cuda", 0)
+ x, slots, w = _inputs(2, dev)
+ ex = CpuMoeExecutor(cache, top_k=TOP_K, activation="silu", apply_router_weight_on_input=False, num_threads=4, max_tokens=2, device=dev, swiglu_limit=LIMIT, act_block=block)
+ got = ex.decode(0, x, w, slots.clone()).clone().float()
+ torch.cuda.synchronize()
+ del ex
+ want = reference(x, slots, w, (gu_p, gu_s, dn_p, dn_s), block)
+ torch.testing.assert_close(got, want, atol=2e-2, rtol=2e-2)
+
+
+def test_grouped_prefill_path_matches_the_transcription():
+ from freetoken.moe import fused_ds_fp4
+
+ banks = _banks("cuda")
+ x, slots, w = _inputs(64, "cuda")
+ got = fused_ds_fp4.routed_experts_fp4_prefill(x, slots, w, *banks, LIMIT, E, act_block=32).float()
+ want = reference(x, slots, w, banks, 32)
+ torch.testing.assert_close(got, want, atol=2e-2, rtol=2e-2)
+
+
[email protected]("block", [32, 128])
+def test_inactive_routes_contribute_zero_without_reading_their_slots(block):
+ """Hybrid decode hands the GPU kernel slot -1 (weight 0) for the routes the CPU computes. The kernel
+ must produce exactly zero for them without touching any slot: every slot the active routes do not
+ use is filled with 0xFF (an e8m0 scale of NaN), and the result still equals the reference over the
+ active routes and the CPU executor's output for the same split."""
+ from types import SimpleNamespace
+
+ from freetoken.kernel.pinned import alloc_pinned_tensor
+ from freetoken.moe.cpu_executor import CpuMoeExecutor
+ from freetoken.moe.fused_ds_fp4 import routed_experts_fp4
+
+ gu_p, gu_s, dn_p, dn_s = (t.clone() for t in _banks("cuda"))
+ x, slots, w = _inputs(2, "cuda")
+ # routes: token 0 keeps routes 0..1 on the GPU, token 1 keeps route 3; the rest go to the CPU
+ on_gpu = torch.zeros_like(slots, dtype=torch.bool)
+ on_gpu[0, :2] = True
+ on_gpu[1, 3] = True
+ gpu_slots = torch.where(on_gpu, slots, -1)
+ gpu_w = torch.where(on_gpu, w, 0.0)
+ used = set(slots[on_gpu].tolist())
+ poison = [e for e in range(E) if e not in used]
+ for t in (gu_p, gu_s, dn_p, dn_s):
+ t[poison] = 0xFF
+ got = routed_experts_fp4(x, gpu_slots, gpu_w, gu_p, gu_s, dn_p, dn_s, LIMIT, act_block=block)
+ assert torch.isfinite(got).all()
+ clean = _banks("cuda")
+ # the poisoned-bank result equals the clean-bank result with the same holes (neither reads them),
+ # and each share matches the reference over its own routes (a zero weight drops a route)
+ got_clean = routed_experts_fp4(x, gpu_slots, gpu_w, *clean, LIMIT, act_block=block)
+ assert torch.equal(got, got_clean)
+ torch.testing.assert_close(got.float(), reference(x, slots, gpu_w, clean, block), atol=2e-2, rtol=2e-2)
+ # the CPU executor computes the complementary split from the same raw ids (ids < 0 skipped)
+ pinned = {}
+ for name, t in (("gate_up_packed", clean[0]), ("gate_up_scale", clean[1]), ("down_packed", clean[2]), ("down_scale", clean[3])):
+ buf = alloc_pinned_tensor(*t.shape, dtype=torch.uint8)
+ buf.copy_(t.cpu())
+ pinned[name] = buf
+ cache = SimpleNamespace(quant_format="ds_fp4", bank_sources={k: [v] for k, v in pinned.items()}, num_layers=1, num_experts=E)
+ ex = CpuMoeExecutor(cache, top_k=TOP_K, activation="silu", apply_router_weight_on_input=False, num_threads=4, max_tokens=2, device=torch.device("cuda", 0), swiglu_limit=LIMIT, act_block=block)
+ cpu_ids = torch.where(on_gpu, -1, slots)
+ cpu_out = ex.decode(0, x, w, cpu_ids.clone()).clone()
+ torch.cuda.synchronize()
+ del ex
+ torch.testing.assert_close(cpu_out.float(), reference(x, slots, torch.where(on_gpu, 0.0, w), clean, block), atol=2e-2, rtol=2e-2)
diff --git a/tests/scheduler/test_dsv4_generic_manager.py b/tests/scheduler/test_dsv4_generic_manager.py
index ad69b89e..1bca10bc 100644
--- a/tests/scheduler/test_dsv4_generic_manager.py
+++ b/tests/scheduler/test_dsv4_generic_manager.py
@@ -14,8 +14,8 @@ import pytest
import torch
from freetoken.core import Req, SamplingParams
-from freetoken.kvcache.dsv4_cost_model import dsv4_pool_sizes
-from freetoken.kvcache.dsv4_paged_pool import DSV4PagedKVCache
+from freetoken.kvcache.dsv4.v4_cost_model import dsv4_pool_sizes
+from freetoken.kvcache.dsv4.v4_pool import DSV4PagedKVCache
from freetoken.models.deepseek_v4.args import DeepseekV4Args
from freetoken.scheduler.cache import CacheManager
@@ -191,6 +191,33 @@ def test_chunk_boundaries_stay_page_aligned_under_unaligned_budget():
cm.check_integrity()
[email protected]("replay, new_tokens, cached", [(0, 2, 256), (P, 2, 128), (P, 200, 256)])
+def test_admission_caps_the_hit_by_the_pools_prefix_replay(replay, new_tokens, cached):
+ """A pool whose model recomputes the prompt's tail (DeepSeek-V4.1's bounded decoder replay) holds
+ back ``prefix_replay_tokens`` at admission: the match ends at or before ``L - replay`` (page-aligned
+ by the radix), a hit already that far from the prompt end is untouched, and the request's
+ ``prompt_len`` is the whole prompt."""
+ from freetoken.scheduler.decode import DecodeManager
+ from freetoken.scheduler.prefill import PrefillManager
+ from freetoken.scheduler.table import TableManager
+ from freetoken.scheduler.utils import PendingReq
+
+ cm, pool, pt = _stack(num_pages=48)
+ pool.prefix_replay_tokens = replay
+ prompt = torch.arange(1, 301, dtype=torch.int32)
+ _lifecycle(cm, _req(MRR - 1, prompt, n_decode=310), total_len=310)
+ pm = PrefillManager(cm, TableManager(max_running_reqs=MRR, page_table=pt), DecodeManager(page_size=P))
+ tokens = torch.cat([prompt, torch.arange(1000, 1000 + new_tokens, dtype=torch.int32)])
+ pm.pending_list = [PendingReq(uid=1, input_ids=tokens, sampling_params=SamplingParams(max_tokens=1))]
+ batch = pm.schedule_next_batch(1024)
+ req = batch.reqs[0]
+ assert req.cached_len == cached and req.extend_len == len(tokens) - cached and req.prompt_len == len(tokens)
+ cm.allocate_paged(batch.reqs)
+ req.complete_one()
+ cm.cache_req(req, finished=True)
+ cm.check_integrity()
+
+
def test_abort_anywhere_fuzz_conserves_and_isolates():
"""admit / extend / finish-or-abort at arbitrary points with shared prefixes + eviction
pressure, on a deliberately small window tier. The generic conservation check (free + tree
diff --git a/tests/scheduler/test_scheduler_kv_usage.py b/tests/scheduler/test_scheduler_kv_usage.py
index 6ab94970..bf0e825c 100644
--- a/tests/scheduler/test_scheduler_kv_usage.py
+++ b/tests/scheduler/test_scheduler_kv_usage.py
@@ -2,6 +2,8 @@ from __future__ import annotations
from types import SimpleNamespace
+from freetoken.attention import AttnType
+
def test_kv_usage_pages_excludes_evictable_prefix_cache():
# page_usage now lives on the CacheManagerLike interface (polymorphic vs DSV4); test the
@@ -53,10 +55,15 @@ def _fake_engine(swa=True, moe=True, mamba=True):
config=SimpleNamespace(
page_size=16,
cache_type="swa_radix",
- model_config=SimpleNamespace(dsv4_args=None, has_swa_attention=swa),
+ max_running_req=1,
+ model_config=SimpleNamespace(
+ dsv4_args=None, has_swa_attention=swa,
+ kv_cache_group_specs=lambda: [SimpleNamespace(attn_type=AttnType.SWA, is_swa=True, sliding_window=128)] if swa else [],
+ swa_attention_group=lambda: SimpleNamespace(sliding_window=128),
+ ),
),
kv_cache=SimpleNamespace(
- swa_num_tokens=513, unit_bytes=lambda: (1 << 20, 1 << 21)
+ window_pages=512, swa_num_tokens=513, unit_bytes=lambda: (1 << 20, 1 << 21)
),
moe_offload_cache=SimpleNamespace(
cache_size=24, num_layers=8, num_experts=16,
diff --git a/tests/server/test_effort_dialect.py b/tests/server/test_effort_dialect.py
index 6a68a4ec..a25de8e7 100644
--- a/tests/server/test_effort_dialect.py
+++ b/tests/server/test_effort_dialect.py
@@ -159,6 +159,22 @@ def test_empty_effort_is_treated_as_absent():
assert state.sent.chat_template_kwargs == {}
+def test_numeric_effort_rides_template_kwargs_not_the_wire_field():
+ # The wire field takes named tiers only (as vLLM and SGLang type it); a numeric budget goes
+ # through chat_template_kwargs, where the checkpoint's encoder validates it at render time.
+ from pydantic import ValidationError
+ import pytest
+
+ for effort in (37, True, 25.5):
+ with pytest.raises(ValidationError):
+ chat_request(reasoning_effort=effort)
+ ctk = {"enable_thinking": True, "reasoning_effort": 37}
+ state = FakeState(reasoning_parser="qwen3")
+ response = run(handle_chat_completion(chat_request(chat_template_kwargs=ctk), None, state, {}))
+ assert not isinstance(response, JSONResponse)
+ assert state.sent.chat_template_kwargs == ctk
+
+
def test_foreign_thinking_shapes_stay_ignored():
# extra="allow" swallowed any thinking shape before the field existed;
# a bare string, a bool, or a typeless dict must keep working unchanged.
diff --git a/tests/server/test_function_call_parser.py b/tests/server/test_function_call_parser.py
index 34c1c04d..bc222967 100644
--- a/tests/server/test_function_call_parser.py
+++ b/tests/server/test_function_call_parser.py
@@ -337,6 +337,34 @@ def test_dsv32_streaming_multi_param_args_prefix_stable():
}
+V41_BLOCK = (
+ "<|DSML| calls>\n"
+ '<|DSML| invoke name="glob">\n'
+ '<|DSML| parameter name="pattern" string="true">*.py</|DSML| parameter>\n'
+ '<|DSML| parameter name="path" string="true">/src</|DSML| parameter>\n'
+ "</|DSML| invoke>\n"
+ "</|DSML| calls>"
+)
+
+
+def test_dsv41_dsml_tags_lead_with_a_space():
+ """DeepSeek-V4.1 leads every DSML tag name with a space (its encoder's ``<|DSML| calls>``):
+ the V3.2 detector sees no tool call there, the V4.1 one parses it whole and streamed."""
+ text = "Let me look.\n\n" + V41_BLOCK
+ assert not FunctionCallParser(OPENCODE_TOOLS, tool_call_parser="deepseekv32").has_tool_call(text)
+ parser = FunctionCallParser(OPENCODE_TOOLS, tool_call_parser="deepseekv41")
+ assert parser.has_tool_call(text)
+ result = parser.parse_non_stream(text)
+ assert result.normal_text == "Let me look."
+ assert [(c.name, json.loads(c.parameters)) for c in result.calls] == [("glob", {"pattern": "*.py", "path": "/src"})]
+
+ streamed = FunctionCallParser(OPENCODE_TOOLS, tool_call_parser="deepseekv41")
+ texts, calls = _feed(streamed, [text[i : i + 5] for i in range(0, len(text), 5)])
+ assert "".join(texts).strip() + streamed.finish_stream() == "Let me look."
+ assert [c.name for c in calls if c.name] == ["glob"]
+ assert json.loads("".join(c.parameters for c in calls if c.name is None)) == {"pattern": "*.py", "path": "/src"}
+
+
def test_streaming_support_flags():
# Every registered detector is incremental-safe (buffered fallback remains as
# the escape hatch for future formats, exercised via monkeypatch in
diff --git a/tests/server/test_parser_auto_selection.py b/tests/server/test_parser_auto_selection.py
index 78b1663e..e1eb8798 100644
--- a/tests/server/test_parser_auto_selection.py
+++ b/tests/server/test_parser_auto_selection.py
@@ -87,6 +87,13 @@ def test_qwen3_5_is_not_shadowed_by_the_generic_qwen_branch():
assert _inferred("Qwen3MoeForCausalLM")[0] == "qwen25"
+def test_deepseek_v41_is_not_shadowed_by_the_v4_branch():
+ """V4.1's DSML leads every tag name with a space; the V4 arm (``"v4"`` is a substring of
+ ``"v41"``) would hand it a tool parser that never sees a call."""
+ assert _inferred("DeepseekV41ForCausalLM") == ("deepseekv41", "deepseekv32")
+ assert _inferred("DeepseekV4ForCausalLM")[0] == "deepseekv32"
+
+
def test_an_explicit_choice_beats_inference():
config = _Config({"architectures": ["DeepseekV4ForCausalLM"], "torch_dtype": "bfloat16"})
with patch("freetoken.utils.cached_load_hf_config", lambda _path: config):
diff --git a/tests/server/test_streaming_model_matrix.py b/tests/server/test_streaming_model_matrix.py
index 43922021..eb6e3a2e 100644
--- a/tests/server/test_streaming_model_matrix.py
+++ b/tests/server/test_streaming_model_matrix.py
@@ -67,6 +67,11 @@ CALL_BLOCKS = {
'<|DSML|parameter name="filePath" string="true">/tmp/test_calc.py</|DSML|parameter>'
"</|DSML|invoke></|DSML|function_calls>"
),
+ "deepseekv41": (
+ '<|DSML| calls>\n<|DSML| invoke name="read">\n'
+ '<|DSML| parameter name="filePath" string="true">/tmp/test_calc.py</|DSML| parameter>\n'
+ "</|DSML| invoke>\n</|DSML| calls>"
+ ),
"qwen25": '<tool_call>\n{"name": "read", "arguments": {"filePath": "/tmp/test_calc.py"}}\n</tool_call>',
"qwen3_coder": (
"<tool_call><function=read><parameter=filePath>/tmp/test_calc.py</parameter>"
@@ -104,6 +109,7 @@ CALL_BLOCKS = {
# Substrings that must never leak into user-visible content.
MARKUP_MARKERS = {
"deepseekv32": ["|DSML|"],
+ "deepseekv41": ["|DSML|"],
"qwen25": ["<tool_call>", "</tool_call>"],
"qwen3_coder": ["<tool_call>", "<function="],
"glm47": ["<arg_key>", "<arg_value>", "</tool_call>"],
@@ -121,6 +127,7 @@ MARKUP_MARKERS = {
# opens the block implicitly (model emits only the closing marker).
REASONING_FAMILIES = {
"dsv4": ("deepseekv32", "deepseekv32", "", "</think>"),
+ "dsv4.1": ("deepseekv41", "deepseekv32", "", "</think>"),
"qwen3.5": ("qwen3_coder", "qwen3", "", "</think>"),
"qwen": ("qwen25", "qwen3", "", "</think>"),
"glm4.7": ("glm47", "glm", "", "</think>"),
diff --git a/tests/tokenizer/test_effort.py b/tests/tokenizer/test_effort.py
index e56df5fe..e0cf7a81 100644
--- a/tests/tokenizer/test_effort.py
+++ b/tests/tokenizer/test_effort.py
@@ -1,6 +1,8 @@
"""Unit tests for the reasoning-effort dialect layer (tokenizer/effort.py)."""
from __future__ import annotations
+import pytest
+
from freetoken.tokenizer.effort import (
EFFORT_SCALE,
EffortProfile,
@@ -131,3 +133,37 @@ def test_probe_skips_rounds_whose_baseline_fails():
profile = probe_effort_profile(render)
assert profile.supported == frozenset({"xhigh", "medium", "low"})
assert profile.consumes_effort
+
+
+def _numeric_render(kwargs, tools):
+ if not tools and not kwargs.get("enable_thinking"):
+ return "chat"
+ effort = kwargs.get("reasoning_effort", "high")
+ if isinstance(effort, str):
+ effort = {"low": 50, "high": 75, "max": 100}[effort]
+ if type(effort) is not int or not 1 <= effort <= 100:
+ raise ValueError("invalid budget")
+ return f"Reasoning Effort: {effort}"
+
+
+def test_probe_preserves_validated_numeric_budgets():
+ profile = probe_effort_profile(_numeric_render)
+ assert profile.integer_range == (1, 100)
+ assert profile.default == "high"
+ for budget in range(1, 101):
+ assert quantize_effort(budget, profile) == budget
+ for bad in (0, 101, -1, True, False, 1.0, 1.5):
+ with pytest.raises(ValueError, match="integer"):
+ quantize_effort(bad, profile)
+
+
+def test_numeric_capability_requires_validation_and_the_whole_range():
+ assert probe_effort_profile(_dsv4_render).integer_range is None
+ assert probe_effort_profile(lambda kw, tools: str(kw)).integer_range is None
+
+ def holes(kwargs, tools):
+ if kwargs.get("reasoning_effort") == 42:
+ raise ValueError("unsupported")
+ return _numeric_render(kwargs, tools)
+
+ assert probe_effort_profile(holes).integer_range is None
diff --git a/tests/tokenizer/test_tokenize.py b/tests/tokenizer/test_tokenize.py
index 16ef6d74..fb73fd31 100644
--- a/tests/tokenizer/test_tokenize.py
+++ b/tests/tokenizer/test_tokenize.py
@@ -405,6 +405,36 @@ class FakeDsv4Tokenizer:
return torch.tensor([[4, 5, 6]], dtype=torch.long)
+def test_dsv41_encoder_gets_image_sources_before_and_after_fetch(tmp_path):
+ import copy
+ from freetoken.mm.media import collect_image_refs
+ encoding_dir = tmp_path / "encoding"
+ encoding_dir.mkdir()
+ (encoding_dir / "encoding_dsv4.py").write_text('''
+def encode_messages(messages, thinking_mode, reasoning_effort=None):
+ parts = messages[0]["content"]
+ assert parts[1] == {"type": "text", "text": "compare"}
+ assert parts[0].get("url") == "https://example.com/image.png" or parts[0].get("data") == b"first image bytes"
+ assert parts[2].get("data") in ("aW1hZ2U=", b"second image bytes")
+ return "two images"
+''')
+ messages = [{"role": "user", "content": [
+ {"type": "image", "freetoken_ref": {"kind": "url", "data": "https://example.com/image.png"}},
+ {"type": "text", "text": "compare"},
+ {"type": "image", "freetoken_ref": {"kind": "b64", "data": "aW1hZ2U="}},
+ ]}]
+ manager = TokenizeManager(FakeDsv4Tokenizer(tmp_path))
+ msg = TokenizeMsg(uid=1, text=messages, sampling_params=SamplingParams())
+ original = copy.deepcopy(messages)
+ assert manager.render_prompt(msg) == "two images"
+ assert messages == original
+ assert len(collect_image_refs(messages)) == 2
+ msg.images = [b"first image bytes", b"second image bytes"]
+ stripped = copy.deepcopy(messages)
+ assert manager.render_prompt(msg) == "two images"
+ assert messages == stripped
+
+
def test_tokenize_manager_uses_dsv4_encoder_when_chat_template_is_missing(tmp_path):
encoding_dir = tmp_path / "encoding"
encoding_dir.mkdir()
@@ -761,3 +791,28 @@ def test_tokenize_survives_an_unhashable_effort():
manager.tokenize([msg])
assert "reasoning_effort" not in tokenizer.chat_template_kwargs
+
+
[email protected]("budget", [1, 25, 50, 75, 100])
+def test_numeric_encoder_effort_survives_prompt_rendering(tmp_path, budget):
+ encoding_dir = tmp_path / "encoding"
+ encoding_dir.mkdir()
+ (encoding_dir / "encoding.py").write_text('''
+def encode_messages(messages, thinking_mode, reasoning_effort=None):
+ if thinking_mode != "thinking":
+ return "chat"
+ effort = "high" if reasoning_effort is None else reasoning_effort
+ if isinstance(effort, str):
+ effort = {"low": 50, "high": 75, "max": 100}[effort]
+ if type(effort) is not int or not 1 <= effort <= 100:
+ raise ValueError("invalid budget")
+ return f"Reasoning Effort: {effort}"
+''')
+ manager = TokenizeManager(FakeDsv4Tokenizer(tmp_path))
+ msg = TokenizeMsg(
+ uid=1, text=[{"role": "user", "content": "hello"}],
+ sampling_params=SamplingParams(),
+ chat_template_kwargs={"enable_thinking": True, "reasoning_effort": budget},
+ )
+ assert manager.render_prompt(msg) == f"Reasoning Effort: {budget}"
+ assert msg.chat_template_kwargs["reasoning_effort"] == budget