Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
feat(deepseek-v41): register the family across build, gateway, CI and docs
JustVugg882c102f753e
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 8aa63ba7..44c9a8b7 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -226,7 +226,7 @@ jobs:run: |cd crc=0- ENGINES="colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38"+ ENGINES="colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38 deepseek_v41"if [ "${{ matrix.v4 }}" = "1" ]; then ENGINES="$ENGINES deepseek-v4"; fifor t in $ENGINES; doecho "::group::$t"@@ -562,6 +562,62 @@ jobs:cd cOLMOE_TINY=olmoe_tiny_c python3 -m unittest -v tests.test_olmoe_dashboard_hits+ dsv41-tiny-check:+ name: DeepSeek V4.1 tiny oracle (engram + DSA + fp4 experts + vision)+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: '3.12'+ cache: pip+ cache-dependency-path: c/tools/oracle-requirements.txt+ - name: Install torch (CPU) + sympy+ run: |+ pip install -r c/tools/oracle-requirements.txt+ pip install sympy+ - name: Tiny V4.1-shaped container and its reference+ run: |+ cd c+ # The same tensor names, dtypes and block-scale conventions as the released+ # checkpoint at toy dimensions, so the engine reads the real format: fp8 dense+ # with 32x32 ue8m0 tiles, fp4 experts with per-32 group scales, an fp8 engram+ # table read row by row, and a vision tower.+ python3 tools/make_dsv41_tiny.py --out dsv41_tiny --emit-ref dsv41_tiny/ref.json+ - name: Token-exact against the reference, at several cache capacities+ run: |+ cd c+ make deepseek_v41+ # cap=1 evicts on every routed expert, which is where slot bookkeeping breaks;+ # cap=8 never would. The engine exits non-zero on any token mismatch, and on a+ # vision tower that drifts from its reference rows.+ for cap in 1 2 8; do+ echo "::group::cap=$cap"+ SNAP=dsv41_tiny ./deepseek_v41 "$cap" dsv41_tiny/ref.json+ echo "::endgroup::"+ done+ - name: The published index cache is the vendor's, not the tidy one+ run: |+ cd c+ # model.py republishes shared_attn.index_k only when a layer completes a+ # compression group, so a decode step with an incomplete group scores against+ # whichever cache was published last. That is the released behaviour and every+ # number for this model was produced with it; V41_INDEX_OWNER=1 is the reading+ # the architecture implies, and it is a DIFFERENT model. Assert they differ, so+ # nobody "fixes" the default by accident.+ if SNAP=dsv41_tiny V41_INDEX_OWNER=1 ./deepseek_v41 8 dsv41_tiny/ref.json > /dev/null 2>&1; then+ echo "FAIL: V41_INDEX_OWNER=1 reproduced the vendor's tokens; one of the two"+ echo " paths is not doing what its comment says"+ exit 1+ fi+ echo "the two index-key policies differ, as documented"+ - name: "Brain tab: EMAP after READY and STAT, HITS after every turn"+ run: |+ cd c+ # the oracle fixture ships no tokenizer; serve mode needs one to encode+ python3 tools/make_edge_tiny_tokenizer.py --vocab-size 256 dsv41_tiny+ COLI_DSV41_FIXTURE=dsv41_tiny python3 -m unittest -v tests.test_dsv41_dashboard+qwen38-tiny-check:name: Qwen3.8 tiny oracle (GDN + QSA + PLE + MoE)runs-on: ubuntu-latestdiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex faaa6b9e..1ef3d3af 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -60,10 +60,10 @@ jobs:- name: Build enginesrun: |cd c- for t in colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38; do+ for t in colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38 deepseek_v41; domake $t ${{ matrix.make_args }}done- ls -lh colibri${{ matrix.ext }} glm53${{ matrix.ext }} inkling${{ matrix.ext }} kimi_k3${{ matrix.ext }} olmoe${{ matrix.ext }} qwen36${{ matrix.ext }} qwen38${{ matrix.ext }}+ ls -lh colibri${{ matrix.ext }} glm53${{ matrix.ext }} inkling${{ matrix.ext }} kimi_k3${{ matrix.ext }} olmoe${{ matrix.ext }} qwen36${{ matrix.ext }} qwen38${{ matrix.ext }} deepseek_v41${{ matrix.ext }}# DeepSeek V4 has its own Makefile and its own target name (`deepseek-v4`,# producing `deepseek_v4`), so it never joined the loop above -- and #858 is@@ -124,6 +124,7 @@ jobs:# with no platform gate, so every archive can carry it.cp c/qwen36${{ matrix.ext }} dist/qwen36${{ matrix.ext }}cp c/qwen38${{ matrix.ext }} dist/qwen38${{ matrix.ext }}+ cp c/deepseek_v41${{ matrix.ext }} dist/deepseek_v41${{ matrix.ext }}# deepseek_v4 -- underscore, not the hyphen of the make target: engine_for()# looks for "deepseek_v4" next to coli. Guarded, not unconditional, because# the macos-arm64 archive has no such engine to ship (#858).diff --git a/.gitignore b/.gitignoreindex 8837be44..55239670 100644--- a/.gitignore+++ b/.gitignore@@ -27,6 +27,8 @@ c/qwen38.exe# `git status`.c/deepseek_v4c/deepseek_v4.exe+c/deepseek_v41+c/deepseek_v41.exec/COLI_V4_UNIT_*.o# ...and the ownership-test objects, which the same Makefile puts in a build/# subdirectory (V4_OWN_DIR) rather than next to the sources. #868 caught the@@ -122,6 +124,7 @@ c/olmoe_i4/.ideac/tiny_inkling/c/qwen38_tiny/+c/dsv41_tiny/# Claude Code working directory (agent worktrees, session scratch) — never a repo artifact.claude/diff --git a/README.it.md b/README.it.mdindex a37d25e4..57fe4541 100644--- a/README.it.md+++ b/README.it.md@@ -13,7 +13,7 @@ in C puro e senza dipendenze del motore, trattando storage, RAM e VRAM comeun'unica gerarchia di inferenza.Oggi girano otto famiglie: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, con-vision), **Inkling** (975B), **Kimi K3** (2,8T), **DeepSeek V4 Flash** (284B),+vision), **Inkling** (975B), **Kimi K3** (2,8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B),**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) e**OLMoE** (7B) — unfile C ciascuna, la stessa interfaccia `coli chat` / `coli serve` / `coli web`. [Elenco completo](README.md#other-supported-models)diff --git a/README.md b/README.mdindex a875d04d..896e70e4 100644--- a/README.md+++ b/README.md@@ -19,7 +19,7 @@ engine dependencies, by treating storage, RAM, and VRAM as a single inferencehierarchy (AI memory multitiering).Eight families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, with-vision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B),+vision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B),**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) and**OLMoE** (7B) —one C file each, the same `coli chat` / `coli serve` / `coli web` front end.@@ -424,6 +424,7 @@ the model's `config.json`):| **GLM-5.3-Flash** (Z.ai) | 321B / 40B | [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) — converted to **int4-gs64** routed experts, dense stays BF16 and the precision is a load-time choice; vision included | `make -C c glm53` | [glm53-flash.md](docs/glm53-flash.md) || **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) — original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) || **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint — routed experts stay **native fp4**, dense stays fp8-e4m3; the **REAP-pruned 150B** ([`puwaer/DeepSeek-V4-Flash-0731-reap-150b`](https://huggingface.co/puwaer/DeepSeek-V4-Flash-0731-reap-150b), 85 GB, 132 of 256 experts) loads with the same engine and no conversion | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |+| **DeepSeek V4.1 Flash** | 552B / 16B | official checkpoint, **no conversion**: experts are already fp4, dense is fp8-e4m3. 203 GB of it is an n-gram memory read from disk a few hundred bytes at a time, and the routed experts cost **4.5 GB per token** against GLM-5.2's 12.7 | `make -C c deepseek_v41` | [deepseek-v41.md](docs/deepseek-v41.md) || **Qwen3.8-Flash-Next** (Alibaba) | 125B + 51B n-gram / 6B | [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) — original checkpoint; PLE stays pageable and experts stay **native block-FP8** | `make -C c qwen38` (CPU only) | [qwen38.md](docs/qwen38.md) || **Qwen3.6** (Alibaba) | 35B / 3B | [`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64) (~20 GB, **recommended**) — hybrid Gated Attention + Gated DeltaNet | `make -C c qwen36` (`CUDA=1` for the VRAM expert tier) | [qwen36.md](docs/qwen36.md) || **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` — **int8** container, ~7 GB | `make -C c olmoe` | — |diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 39cda7ab..77a981ba 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -11,7 +11,7 @@2.8T 参数**——以引擎零依赖的纯 C 实现,将存储、RAM 与 VRAM 视为统一的推理层级。目前可运行八个模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含视觉)、-**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、+**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、, **DeepSeek V4.1 Flash** (552B)**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)与 **OLMoE**(7B)——各自一个 C 文件,共用同一套 `coli chat` / `coli serve` / `coli web` 前端。[完整列表](README.md#other-supported-models)diff --git a/README.zh-TW.md b/README.zh-TW.mdindex f377d33e..f2a5272b 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -11,7 +11,7 @@2.8T 參數**——以引擎零相依套件的純 C 實作,將儲存、RAM 與 VRAM 視為統一的推論階層。目前可執行八個模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含視覺)、-**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、+**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、, **DeepSeek V4.1 Flash** (552B)**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)與 **OLMoE**(7B)——各自一個 C 檔案,共用同一套 `coli chat` / `coli serve` / `coli web` 前端。[完整清單](README.md#other-supported-models)diff --git a/c/Makefile b/c/Makefileindex bf2e43d6..86193f9d 100644--- a/c/Makefile+++ b/c/Makefile@@ -1045,6 +1045,13 @@ qwen36$(EXE): qwen36.c serve_poll.h cli_args.h qwen36_tier.h st.h json.h compat.# Qwen3.8-Flash-Next text-only sibling. This target is intentionally CPU-only;# qwen38.c owns the direct checkpoint loader and SERVE=1 protocol, with no# CUDA/Metal tier advertised by the control plane.+# DeepSeek V4.1 Flash: one file, like every other portable engine. The fp4 experts+# stream from the official checkpoint and quant.h's mxfp4 kernel reads them as they+# are, so there is no conversion target here to go with it.+deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h compat.h quant.h \+ hyper_connections.h serve_codec.h serve_poll.h .build-config+ $(CC) $(CFLAGS) deepseek_v41.c -o deepseek_v41$(EXE) $(LDFLAGS)+qwen38$(EXE): qwen38.c cli_args.h qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h tok.h tok_unicode.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h$(CC) $(NOCUDA_CFLAGS) qwen38.c -o qwen38$(EXE) $(NOCUDA_LDFLAGS)@@ -1896,7 +1903,7 @@ check:$(MAKE) portable$(MAKE) test-install: colibri$(EXE) glm53$(EXE) inkling$(EXE) kimi_k3$(EXE) olmoe$(EXE) qwen36$(EXE) qwen38$(EXE) \+install: colibri$(EXE) glm53$(EXE) inkling$(EXE) kimi_k3$(EXE) olmoe$(EXE) qwen36$(EXE) qwen38$(EXE) deepseek_v41$(EXE) \$(if $(filter 1,$(COLI_V4_SUPPORTED)),deepseek-v4,)$(INSTALL) -d $(DESTDIR)$(BINDIR)$(INSTALL) -d $(DESTDIR)$(LIBEXECDIR)@@ -1909,6 +1916,7 @@ install: colibri$(EXE) glm53$(EXE) inkling$(EXE) kimi_k3$(EXE) olmoe$(EXE) qwen3$(INSTALL) -m 755 olmoe$(EXE) $(DESTDIR)$(LIBEXECDIR)/olmoe$(EXE)$(INSTALL) -m 755 qwen36$(EXE) $(DESTDIR)$(LIBEXECDIR)/qwen36$(EXE)$(INSTALL) -m 755 qwen38$(EXE) $(DESTDIR)$(LIBEXECDIR)/qwen38$(EXE)+ $(INSTALL) -m 755 deepseek_v41$(EXE) $(DESTDIR)$(LIBEXECDIR)/deepseek_v41$(EXE)@if [ -f deepseek_v4$(EXE) ]; then \$(INSTALL) -m 755 deepseek_v4$(EXE) $(DESTDIR)$(LIBEXECDIR)/deepseek_v4$(EXE); \fidiff --git a/c/family_registry.py b/c/family_registry.pyindex b4c981e6..9042d73a 100644--- a/c/family_registry.py+++ b/c/family_registry.py@@ -689,6 +689,58 @@ def _dsv4_geometry(config, context, _model_dir):return PlannerGeometry(state, fixed, workspace, experts)+def _dsv41_geometry(config, context, _model_dir):+ """DeepSeek V4.1 Flash: window ring, compressed KV and index keys, engram rows.++ What the engine keeps resident per layer (deepseek_v41.c model_load):++ window ring n_layers * window_size * head_dim * 4 (fixed)+ compressed KV ceil(context / ratio) * head_dim * 4 (per kv source)+ index keys ceil(context / ratio) * index_head_dim * 4 (per kv source)++ The engram tables are NOT in here on purpose: 203 GB of the released+ checkpoint is n-gram memory read row by row from disk (264 bytes per row,+ at most (max_ngram - 1) * n_heads rows per layer per token), with a small+ LRU whose size is a runtime knob rather than a function of the context.+ Counting them as resident would tell a user with 128 GB that this model+ cannot be served, which is the opposite of true.++ Workspace mirrors forward()'s per-chunk buffers: hc_mult residual copies+ plus the sublayer scratch, at batch == context.+ """+ section = config.get("text_config", config)+ layers = _required_int(section, "num_hidden_layers", "deepseek_v41")+ experts = _required_int(section, "n_routed_experts", "deepseek_v41")+ hidden = _required_int(section, "hidden_size", "deepseek_v41")+ head_dim = _required_int(section, "head_dim", "deepseek_v41")+ window = _required_int(section, "sliding_window", "deepseek_v41") \+ if "sliding_window" in section else _required_int(section, "window_size", "deepseek_v41")+ index_hd = _required_int(section, "index_head_dim", "deepseek_v41")+ hc_mult = _required_int(section, "hc_mult", "deepseek_v41")++ ratios = section.get("compress_ratios")+ if not isinstance(ratios, list) or len(ratios) < layers:+ raise ValueError("deepseek_v41: compress_ratios must be a list of at least "+ "num_hidden_layers entries")+ sources = section.get("kv_source_layer_ids") or section.get("kv_source_layers") or []+ if not isinstance(sources, list):+ raise ValueError("deepseek_v41: kv_source_layer_ids must be a list")++ fixed = layers * window * head_dim * 4+ state = 0+ for layer in sources:+ if not isinstance(layer, int) or isinstance(layer, bool) or not 0 <= layer < layers:+ raise ValueError("deepseek_v41: kv_source_layer_ids entries must index a layer")+ ratio = ratios[layer]+ if not isinstance(ratio, int) or isinstance(ratio, bool) or ratio < 1:+ raise ValueError("deepseek_v41: a kv source layer must compress")+ compressed = (context + ratio - 1) // ratio+ state += compressed * (head_dim + index_hd) * 4++ workspace = (context * hc_mult * hidden * 2 + context * hidden * 2 + hidden) * 4+ return PlannerGeometry(state, fixed, workspace, experts)++_GLM_EXPERT = re.compile(r"(?:^|\.)model\.layers\.(\d+)\.mlp\.experts\.(\d+)\.")@@ -1197,6 +1249,33 @@ FAMILIES = (has_gateway_adapter=True,has_cli_adapter=True,),+ FamilyDescriptor(+ id="deepseek_v41",+ model_types=("deepseek_v41", "deepseek_v41_text"),+ display_name="DeepSeek V4.1 Flash",+ display_scale="552B",+ # deepseek-ai/DeepSeek-V4.1-Flash: 40 layers, 384 experts, top-6, plus+ # two 384M-row engram tables that never enter RAM.+ reference_experts=384,+ engine_artifact="deepseek_v41",+ engine_aliases=(),+ engine_group="deepseek_v41",+ internal_arch="deepseek_v41",+ build_target="deepseek_v41",+ process_names=("deepseek_v41",),+ default_model_id="deepseek-v4.1-flash-colibri",+ cli_adapter="deepseek_v41",+ gateway_adapter="deepseek_v41",+ planner_id="deepseek_v41",+ planner_geometry=_dsv41_geometry,+ planner_unsupported_reason="",+ expert_inventory=_individual_expert_inventory(_V4_EXPERT),+ config_section="text_config",+ limits=FamilyLimits(4096, 1048576, 1024, 16384, 1, 8, "CTX"),+ capabilities=FamilyCapabilities(True, True, False, True),+ has_gateway_adapter=True,+ has_cli_adapter=True,+ ),)diff --git a/c/openai_server.py b/c/openai_server.pyindex f030eff8..24c95644 100644--- a/c/openai_server.py+++ b/c/openai_server.py@@ -1742,6 +1742,57 @@ def expand_glm53_images(messages, model_dir):return rewritten, images+# DeepSeek V4.1's image placeholder: every position of an image span carries the same+# id, and what each one MEANS follows from the aligner grid (image_processor.py+# image_token_types): start, then one newline per row of image tokens, then end. The+# engine rebuilds that layout from the grid it gets in the IMAGE frame, so the gateway+# only has to insert the right NUMBER of placeholders -- and getting that number wrong+# is the one failure the engine cannot paper over, which is why it refuses instead.+DSV41_IMAGE_PLACEHOLDER = "<|deepseek_image|>"+++def expand_dsv41_images(messages, model_dir, max_tokens=None):+ """Replace image parts with their placeholder span and pull out the patches."""+ images, rewritten = [], []+ for message in messages:+ content = message.get("content") if isinstance(message, dict) else None+ if not isinstance(content, list):+ rewritten.append(message)+ continue+ pieces = []+ for part in content:+ if not isinstance(part, dict):+ continue+ kind = part.get("type")+ if kind == "text":+ pieces.append(part.get("text", ""))+ elif kind in ("image_url", "input_image"):+ url = (part.get("image_url") or {}).get("url") if kind == "image_url" \+ else part.get("image_url") or part.get("url")+ data = _image_bytes_from_url(url)+ patches, grid_h, grid_w, llm_h, llm_w = _preprocess_dsv41_image(+ data, model_dir, max_tokens)+ span = 1 + (llm_w + 1) * llm_h + 1+ images.append((patches, grid_h, grid_w))+ pieces.append(DSV41_IMAGE_PLACEHOLDER * span)+ else:+ raise APIError(400, f"unsupported content part {kind!r}.", "messages")+ rewritten.append({**message, "content": "".join(pieces)})+ return rewritten, images+++def _preprocess_dsv41_image(data, model_dir, max_tokens=None):+ try:+ import sys as _sys+ from pathlib import Path as _Path+ _sys.path.insert(0, str(_Path(__file__).resolve().parent / "tools"))+ from dsv41_image import preprocess+ except ImportError as problem:+ raise APIError(400, f"image support needs Pillow and numpy ({problem}).",+ "messages")+ return preprocess(data, model_dir, max_tokens)++def _preprocess_image(data, model_dir):"""L'immagine nelle patch che la torre vuole. Il lavoro sta intools/glm53_image.py, verificato contro il processore ufficiale."""@@ -1919,6 +1970,79 @@ def render_chat_glm53(messages, enable_thinking=False, reasoning_effort=None, toreturn "".join(prompt)+# ---- DeepSeek V4.1 Flash -----------------------------------------------------------------+# The checkpoint ships its chat format as a Python module (encoding/encoding.py), not a+# jinja template, so this is that module's render_message transcribed for the shapes the+# gateway sends. Its pieces, with the vendor's own names:+#+# <|begin▁of▁sentence|> once, at the head of a fresh conversation+# <|System|> leads the conversation when there is a system message or a+# reasoning-effort line; also precedes a mid-conversation one+# Reasoning Effort: N index 0 only, thinking mode only, N in 1..100+# <|User|> / <|Assistant|>+# <think> or </think> the generation cue: open in thinking mode, closed otherwise+# assistant turns end with <|end▁of▁sentence|>+DSV41_BOS = "<|begin▁of▁sentence|>"+DSV41_EOS = "<|end▁of▁sentence|>"+DSV41_SYSTEM = "<|System|>"+DSV41_USER = "<|User|>"+DSV41_ASSISTANT = "<|Assistant|>"+# encoding.py REASONING_EFFORT_MAPPINGS; the default there is "high"+DSV41_EFFORT = {"minimal": 50, "low": 50, "medium": 75, "high": 75, "xhigh": 100}+++def render_chat_dsv41(messages, enable_thinking=False, reasoning_effort=None, tools=None,+ tool_choice=None):+ """encoding.py _encode_messages_text for one turn.++ `tools` and `tool_choice` are accepted and refused rather than ignored: V4.1 declares+ tools inside the system message as a DSML block, and half-rendering that would produce+ a model that announces tools it cannot be told the results of.+ """+ if tools or (isinstance(tool_choice, dict) and tool_choice):+ raise APIError(400, "DeepSeek V4.1 tool calling is not wired in this build; send "+ "the request without `tools`.", "tools")+ if not isinstance(messages, list) or not messages:+ raise APIError(400, "`messages` must be a non-empty array.", "messages")+ budget = DSV41_EFFORT.get(reasoning_effort or "high", 75)+ prompt = [DSV41_BOS]+ last_user = -1+ for index, message in enumerate(messages):+ if isinstance(message, dict) and message.get("role") == "user":+ last_user = index+ for index, message in enumerate(messages):+ if not isinstance(message, dict):+ raise APIError(400, "Each message must be an object.", f"messages.{index}")+ role = message.get("role")+ if role == "developer":+ role = "system"+ if role not in ("system", "user", "assistant"):+ raise APIError(400, f"Unsupported role {role!r}.", f"messages.{index}.role")+ content = content_text(message.get("content"), f"messages.{index}.content")+ if role == "system":+ prompt.append(DSV41_SYSTEM)+ if index == 0 and enable_thinking:+ prompt.append(f"Reasoning Effort: {budget} (range 1-100, the higher the "+ f"value, the more thorough the reasoning)\n\n")+ prompt.append(content)+ elif role == "user":+ if index == 0 and enable_thinking:+ prompt.append(DSV41_SYSTEM)+ prompt.append(f"Reasoning Effort: {budget} (range 1-100, the higher the "+ f"value, the more thorough the reasoning)\n\n")+ prompt.append(DSV41_USER + content)+ else:+ reasoning = message.get("reasoning_content")+ opened = ""+ if isinstance(reasoning, str) and reasoning:+ opened = f"<think>{reasoning}</think>"+ prompt.append(DSV41_ASSISTANT + opened + content + DSV41_EOS)+ # the generation cue, exactly as render_message appends it after a user turn+ prompt.append(DSV41_ASSISTANT)+ prompt.append("<think>" if enable_thinking and len(messages) - 1 >= last_user else "</think>")+ return "".join(prompt)++def render_chat_for_arch(messages, enable_thinking=False, reasoning_effort=None, tools=None,tool_choice=None, audio_out=None):"""Render a chat request with the active engine's native prompt contract."""@@ -1930,6 +2054,7 @@ def render_chat_for_arch(messages, enable_thinking=False, reasoning_effort=None,render_chat_qwen if ARCH == "qwen36" elserender_chat_qwen38 if ARCH == "qwen38" elserender_chat_v4 if ARCH == "deepseek_v4" else+ render_chat_dsv41 if ARCH == "deepseek_v41" elserender_chat_olmoe if ARCH == "olmoe" else render_chat)return renderer(messages, enable_thinking, reasoning_effort, tools, tool_choice)@@ -3935,6 +4060,15 @@ class APIHandler(BaseHTTPRequestHandler):raise APIError(400, "one image per request for now; the engine ""holds a single pending image.", "messages")image = images[0] if images else None+ elif ARCH == "deepseek_v41":+ ceiling = os.environ.get("V41_MAX_IMAGE_TOKENS")+ messages, images = expand_dsv41_images(+ messages, getattr(self.server.engine, "model_dir", None),+ int(ceiling) if ceiling else None)+ if len(images) > 1:+ raise APIError(400, "one image per request for now; the engine "+ "holds a single pending image.", "messages")+ image = images[0] if images else Noneelif ARCH == "qwen38":ceiling = os.environ.get("Q38_MAX_IMAGE_TOKENS")messages, images = expand_qwen38_images(diff --git a/c/tests/segment_conformance_fixtures.c b/c/tests/segment_conformance_fixtures.cindex 0cbf945c..9c9ff910 100644--- a/c/tests/segment_conformance_fixtures.c+++ b/c/tests/segment_conformance_fixtures.c@@ -115,6 +115,18 @@ static const ColiSegmentConformanceFixture g_fixtures[] = {COLI_SEGMENT_FIXTURE_DEVICE_CACHE,5, 10, 4, 64, UINT32_C(0x44535634),},+ {+ "deepseek_v41", "DeepSeek V4.1", "fixture/dsv41-engram-compressed-v1",+ "mHC + window/compressed attention + compressor + indexer + engram + vision",+ "tools/make_dsv41_tiny.py",+ COLI_SEGMENT_FIXTURE_MHC |+ COLI_SEGMENT_FIXTURE_SLIDING_RING |+ COLI_SEGMENT_FIXTURE_COMPRESSOR |+ COLI_SEGMENT_FIXTURE_DSA_INDEXER |+ COLI_SEGMENT_FIXTURE_ENGRAM |+ COLI_SEGMENT_FIXTURE_VISION_TOWER,+ 6, 8, 4, 64, UINT32_C(0x44533431),+ },};static int fail(char *error, size_t error_size, const char *message) {@@ -365,6 +377,7 @@ DECLARE_OPEN_WRAPPER(olmoe, 4)DECLARE_OPEN_WRAPPER(qwen36, 5)DECLARE_OPEN_WRAPPER(qwen38, 6)DECLARE_OPEN_WRAPPER(deepseek_v4, 7)+DECLARE_OPEN_WRAPPER(deepseek_v41, 8)#define FIXTURE_ADAPTER(name) \{ \@@ -390,6 +403,7 @@ static const ColiSegmentAdapter g_adapters[] = {FIXTURE_ADAPTER(qwen36),FIXTURE_ADAPTER(qwen38),FIXTURE_ADAPTER(deepseek_v4),+ FIXTURE_ADAPTER(deepseek_v41),};int coli_segment_conformance_register_fixtures(void) {diff --git a/c/tests/segment_conformance_fixtures.h b/c/tests/segment_conformance_fixtures.hindex dfada274..55b28c86 100644--- a/c/tests/segment_conformance_fixtures.h+++ b/c/tests/segment_conformance_fixtures.h@@ -29,6 +29,9 @@ typedef enum {COLI_SEGMENT_FIXTURE_HYPER_RESIDUAL = UINT32_C(1) << 10,COLI_SEGMENT_FIXTURE_SPARSE_ATTN = UINT32_C(1) << 11,COLI_SEGMENT_FIXTURE_PLE = UINT32_C(1) << 12,+ /* DeepSeek V4.1: the n-gram memory read from disk, and the ViT tower */+ COLI_SEGMENT_FIXTURE_ENGRAM = UINT32_C(1) << 13,+ COLI_SEGMENT_FIXTURE_VISION_TOWER = UINT32_C(1) << 14,} ColiSegmentFixtureState;typedef struct {diff --git a/c/tests/segment_conformance_manifest.json b/c/tests/segment_conformance_manifest.jsonindex 08f95df0..2d2bdb49 100644--- a/c/tests/segment_conformance_manifest.json+++ b/c/tests/segment_conformance_manifest.json@@ -91,8 +91,18 @@{"family_id": "qwen38","state_schema": "fixture/qwen38-hyper-qsa-ple-v1",- "state_components": ["hyper_residual", "deltanet_recurrent", "convolution_ring", "attention_kv", "sparse_indexer", "ple_hash_history"],- "oracle": {"kind": "generated_tiny", "path": "tools/make_qwen38_tiny.py"},+ "state_components": [+ "hyper_residual",+ "deltanet_recurrent",+ "convolution_ring",+ "attention_kv",+ "sparse_indexer",+ "ple_hash_history"+ ],+ "oracle": {+ "kind": "generated_tiny",+ "path": "tools/make_qwen38_tiny.py"+ },"real_adapter_required": true},{@@ -110,6 +120,24 @@"path": "tools/make_deepseek_v4_tiny.py"},"real_adapter_required": true+ },+ {+ "family_id": "deepseek_v41",+ "state_schema": "fixture/dsv41-engram-compressed-v1",+ "state_components": [+ "mhc",+ "window_attention",+ "compressed_attention",+ "compressor",+ "indexer",+ "engram",+ "vision"+ ],+ "oracle": {+ "kind": "generated_tiny",+ "path": "tools/make_dsv41_tiny.py"+ },+ "real_adapter_required": true}]}diff --git a/c/tests/test_dsv41_dashboard.py b/c/tests/test_dsv41_dashboard.pynew file mode 100644index 00000000..bd93e904--- /dev/null+++ b/c/tests/test_dsv41_dashboard.py@@ -0,0 +1,112 @@+"""The dashboard's Brain and Profile tabs on DeepSeek V4.1 Flash.++Same contract as every other family: EMAP after READY and STAT, HITS and PROF after+DONE, over the rows and columns the grid declares. This drives the engine the way the+gateway does -- through `coli serve` -- so a family that stops emitting one of the four+lines fails here rather than as a blank tab.++Needs a V4.1-shaped container with a tokenizer and a built `deepseek_v41`. Set+COLI_DSV41_FIXTURE to the container; without it the test skips with the reason on+the record. CI builds the fixture with tools/make_dsv41_tiny.py.+"""+import json+import os+import socket+import subprocess+import sys+import time+import unittest+import urllib.request+from pathlib import Path++HERE = Path(__file__).resolve().parent.parent+FIXTURE = Path(os.environ.get("COLI_DSV41_FIXTURE", ""))+++def free_port():+ with socket.socket() as s:+ s.bind(("127.0.0.1", 0))+ return s.getsockname()[1]+++def get_json(url, timeout=10):+ with urllib.request.urlopen(url, timeout=timeout) as response:+ return json.load(response)++[email protected](FIXTURE.name and (FIXTURE / "config.json").is_file()+ and (FIXTURE / "tokenizer.json").is_file(),+ "COLI_DSV41_FIXTURE not set to a V4.1 container with a tokenizer")[email protected]((HERE / "deepseek_v41").exists() or (HERE / "deepseek_v41.exe").exists(),+ "deepseek_v41 engine is not built")+class Dsv41DashboardTest(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ cls.port = free_port()+ cls.proc = subprocess.Popen(+ [sys.executable, str(HERE / "coli"), "serve", "--model", str(FIXTURE),+ "--port", str(cls.port), "--cap", "4"],+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)+ deadline = time.time() + 180+ while time.time() < deadline:+ try:+ get_json(f"http://127.0.0.1:{cls.port}/health", timeout=2)+ break+ except Exception:+ if cls.proc.poll() is not None:+ raise RuntimeError("coli serve exited: " + cls.proc.stdout.read()[-2000:])+ time.sleep(1)+ else:+ raise RuntimeError("coli serve did not come up in 180 s")+ model = get_json(f"http://127.0.0.1:{cls.port}/v1/models")["data"][0]["id"]+ body = json.dumps({"model": model, "prompt": "abcabcabc", "max_tokens": 3}).encode()+ request = urllib.request.Request(f"http://127.0.0.1:{cls.port}/v1/completions",+ data=body, headers={"Content-Type": "application/json"})+ with urllib.request.urlopen(request, timeout=300) as response:+ cls.completion = json.load(response)+ cls.experts = get_json(f"http://127.0.0.1:{cls.port}/experts")+ cls.profile = get_json(f"http://127.0.0.1:{cls.port}/profile")+ cls.config = json.loads((FIXTURE / "config.json").read_text())++ @classmethod+ def tearDownClass(cls):+ cls.proc.terminate()+ try:+ cls.proc.wait(10)+ except subprocess.TimeoutExpired:+ cls.proc.kill()++ def test_brain_grid_matches_the_model(self):+ text = self.config.get("text_config", self.config)+ experts = self.experts+ self.assertEqual(experts["rows"], text["num_hidden_layers"])+ self.assertEqual(experts["cols"], text["n_routed_experts"])+ self.assertEqual(len(experts["map"]), experts["rows"] * experts["cols"] * 2)++ def test_brain_lights_up_and_shows_residents(self):+ experts = self.experts+ bits = experts["rows"] * experts["cols"]+ self.assertEqual(len(experts["hits"]), ((bits + 7) // 8) * 2)+ self.assertNotEqual(int(experts["hits"], 16), 0,+ "a 9-token prompt routed no expert at all")+ self.assertNotEqual(int(experts["map"], 16), 0,+ "after a turn the cache holds experts: the grid must show them")++ def test_profile_records_the_turn(self):+ turns = self.profile.get("turns", [])+ self.assertEqual(len(turns), 1)+ turn = turns[0]+ for key in ("wall_s", "prompt_tokens", "completion_tokens", "expert_disk_s",+ "expert_wait_s", "expert_matmul_s", "attention_s", "lm_head_s", "forwards"):+ self.assertIn(key, turn)+ self.assertEqual(turn["completion_tokens"], 3)+ self.assertEqual(turn["forwards"], 3, "prefill plus one forward per token after the first")+ phases = (turn["expert_disk_s"] + turn["expert_matmul_s"] ++ turn["attention_s"] + turn["lm_head_s"])+ self.assertGreaterEqual(phases, 0.0)+ self.assertLessEqual(phases, turn["wall_s"] * 1.05 + 0.01,+ "phase timings exceed the wall clock: a timer double-counts")+++if __name__ == "__main__":+ unittest.main()diff --git a/c/tests/test_family_registry.py b/c/tests/test_family_registry.pyindex 1849bf14..52501f2b 100644--- a/c/tests/test_family_registry.py+++ b/c/tests/test_family_registry.py@@ -1027,6 +1027,9 @@ class FamilyRegistryTest(unittest.TestCase):"<|im_end|>\n<|im_start|>user\nhello {world}<|im_end|>\n""<|im_start|>assistant\n<think>\n","deepseek_v4": "hello {world}",+ # V4.1 ships its chat encoding as a Python module (encoding/encoding.py),+ # not a jinja template, so the replay prompt stays the bare text like V4.+ "deepseek_v41": "hello {world}",}self.assertEqual({family.id: tuning_replay_prompt(family, prompt) for family in FAMILIES},diff --git a/c/tools/clean.py b/c/tools/clean.pyindex 8105be33..c309212a 100644--- a/c/tools/clean.py+++ b/c/tools/clean.py@@ -33,6 +33,7 @@ FILES = [# hipcc emits an import library, export file and PDB alongside the DLL."coli_hip.dll", "coli_hip.lib", "coli_hip.exp", "coli_hip.pdb","deepseek_v4", "deepseek_v4.exe",+ "deepseek_v41", "deepseek_v41.exe","native_quant.o", "native_quant_parallel.o", "native_quant_dual.o","native_quant_batch_avx512.o", "native_quant_fp4_rows16.o",]diff --git a/c/tools/convert_fp8_to_int4.py b/c/tools/convert_fp8_to_int4.pyindex d1cf02d2..6f87f9ad 100644--- a/c/tools/convert_fp8_to_int4.py+++ b/c/tools/convert_fp8_to_int4.py@@ -62,6 +62,8 @@ OTHER_FAMILY_PATHS = {"kimi_k3": "Kimi K3: see docs/kimi_k3.md for its container pipeline","kimi_linear": "Kimi K3: see docs/kimi_k3.md for its container pipeline","deepseek_v4": "DeepSeek V4: see docs/deepseek-v4.md, section Download",+ "deepseek_v41": "DeepSeek V4.1: no conversion needed -- its experts already ship fp4 and its dense fp8; run tools/prepare_dsv41.py once, see docs/deepseek-v41.md",+ "deepseek_v41_text": "DeepSeek V4.1: no conversion needed -- see docs/deepseek-v41.md",}diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 9e46a569..3ce40f71 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -493,3 +493,7 @@ COLI_METAL=1 DIRECT=1 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=6 MTP=0 \COLI_TEMP=0 COLI_METAL=1 DIRECT=1 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=6 MTP=0 \./coli run --model /path/to/model --ram 113 "your prompt"```+| `V41_ENGRAM_ROWS` | 65536 | DeepSeek V4.1: rows of engram cache per table. The n-gram traffic is Zipfian, so a small cache absorbs most of it; 65536 rows is 64 MB per table on the released head_dim. |+| `V41_INDEX_OWNER` | unset | DeepSeek V4.1: score each layer against its OWN index keys instead of the last published cache. The default reproduces the released inference code; this changes the model's behaviour, see docs/deepseek-v41.md. |+| `V41_MAX_IMAGE_TOKENS` | the checkpoint's `max_image_tokens` | DeepSeek V4.1: ceiling on what one image costs in prompt tokens. |+| `V41_TRACE` | unset | DeepSeek V4.1: print per-sublayer checksums, matching tools/dsv41_ref.py's, to locate a divergence by diffing two columns. |diff --git a/docs/deepseek-v41.md b/docs/deepseek-v41.mdnew file mode 100644index 00000000..43217f17--- /dev/null+++ b/docs/deepseek-v41.md@@ -0,0 +1,105 @@+# DeepSeek V4.1 Flash++552B parameters, 510 GB on disk, and the shape of it is why this engine exists.++| part | on disk | how it is read |+|---|---|---|+| **engram** (2 tables, layers 1 and 14) | **203 GB** | `[384,006,168 x 256]` fp8 plus ue8m0 scales: **264 bytes per row**, looked up by n-gram hash |+| **routed experts** | **289 GB** | 15,360 experts of 18.8 MB, already fp4; 6 of 384 per layer |+| dense, embeddings, vision | ~18 GB | fp8 with 32x32 block scales, bf16 elsewhere; resident |++Two numbers decide whether this model is servable on a machine you own:++- **4.5 GB of experts per token** (6 routed x 40 layers x 18.8 MB). GLM-5.2 reads+ 12.7 GB per token, so V4.1 Flash is lighter per token than a model less than half+ its size.+- **~13 KB of engram per token**. The n-gram memory is 40% of the checkpoint and+ costs a few dozen random reads of 264 bytes: at most `(max_ngram_size - 1) x+ n_heads` rows per table per token, 48 rows for the released layout. Streaming it+ is not a compromise, it is the only sane way to hold it -- 203 GB does not fit in+ anyone's VRAM.++## Running it++No conversion. The released checkpoint is already fp8 (dense) and fp4 (experts), and+colibri reads both formats natively -- the fp4 expert layout is byte-identical to the+mxfp4 it already reads for Kimi K3.++```sh+hf download deepseek-ai/DeepSeek-V4.1-Flash --local-dir ~/Models/DeepSeek-V4.1-Flash+python3 c/tools/prepare_dsv41.py --model ~/Models/DeepSeek-V4.1-Flash # ~1 MB sidecar, once+make -C c deepseek_v41+coli chat --model ~/Models/DeepSeek-V4.1-Flash+```++`prepare_dsv41.py` writes the three tables the engine cannot derive at load time and+that never change for a checkpoint: the compressed-token map (the tokenizer's+normalizer collapses ids that hash alike), the bucket primes (one prime-sized range+per n-gram size and head), and the hash multipliers (numpy's PCG64, seeded per layer).+They ship as `dsv41_engram.json` beside `config.json`. The multipliers are stored as+decimal **strings**: they run past a double's 53-bit mantissa, and a rounded+multiplier would silently hash every n-gram into a different -- and perfectly+valid-looking -- row.++## What the engine implements++Each mechanism mirrors a named piece of the vendor's `inference/model.py`:++- **hyper-connections**: the residual stream is `hc_mult` parallel copies, mixed+ through a doubly stochastic matrix (Sinkhorn). Shared with DeepSeek V4 and+ GLM-5.3-Flash through `hyper_connections.h`.+- **sliding window**: every layer attends a ring of `window_size` raw KV, MQA --+ one KV vector per position for all 64 heads, which is why the KV cache is small.+- **compressed KV**: `kv_source_layers` pool `compress_ratio` tokens into one latent+ with a softmax gate and publish it; the layers between them read that cache.+- **DSA indexer**: `index_source_layers` score compressed positions and keep+ `index_topk` of them, two-level when a candidate source is configured.+- **engram**: n-gram lookups gated into the residual stream by how well the looked-up+ key agrees with it.+- **MoE**: `sqrtsoftplus` scores with the noaux_tc bias, which picks experts but does+ not scale them, plus one shared expert every token pays for.+- **vision**: a 32-layer ViT with 2D split-half RoPE and a 3x3 aligner, resident.++## The index-key slot, and why the default is the odd one++`model.py` keeps the published index-key cache in one module-level slot and+republishes it only when a layer **completes** a compression group:++```python+if self.owns_k and latent is not None:+ ...+ shared_attn.index_k = self.k_cache+```++On a decode step where a ratio-2 layer's group is still filling, that layer therefore+scores its queries against whichever cache was published last -- in the released+config, layer 20's, whose rows are ratio-1 latents and mean something else. In the+released layout this happens to layers 2, 8 and 14 on every other decode step.++This engine reproduces that, because it is what DeepSeek ships and what every+published number for this model was measured with. `V41_INDEX_OWNER=1` selects the+reading the architecture implies instead -- each layer against its own owner's keys.+That is a different model, not a bug fix, and the CI asserts the two disagree so the+default cannot be "tidied" by accident.++## Environment++| variable | default | what it does |+|---|---|---|+| `V41_ENGRAM_ROWS` | 65536 | rows of engram cache per table. The traffic is Zipfian: common 2-grams repeat constantly, so a small cache absorbs most of it. 65536 rows is 64 MB per table on the released head_dim. |+| `V41_INDEX_OWNER` | unset | each layer scores against its own owner's index keys (see above). Changes the model's behaviour. |+| `V41_MAX_IMAGE_TOKENS` | the checkpoint's `max_image_tokens` | a ceiling on what one image costs in prompt tokens. |+| `V41_TRACE` | unset | print a checksum of the tensors the reference prints too, for locating a divergence by diffing two columns. |++## How it is tested++`tools/dsv41_ref.py` is a torch reimplementation of the vendor's kernels on the CPU:+the vendor's own forward runs through tilelang GPU kernels and cannot be an oracle+here. `tools/make_dsv41_tiny.py` builds a container with the released structure at toy+dimensions and emits a reference; CI runs the engine against it at three cache+capacities and holds the vision tower to its own reference rows.++Writing both sides found three defects that a single implementation would have kept:+a hyper-connection mix summed over the wrong axis, hash multipliers rounded by a JSON+reader that stores numbers as doubles, and an index list that a non-source layer+inherited from uninitialized memory instead of from its source.diff --git a/site/index.html b/site/index.htmlindex 3d4a50bd..10f85750 100644--- a/site/index.html+++ b/site/index.html@@ -285,6 +285,8 @@ footer li a:hover{color:var(--on-dark)}<a class="btn ghost" href="https://github.com/JustVugg/colibri/blob/main/docs/inkling.md">Run it →</a></div><div class="card"><h3>DeepSeek V4 Flash</h3><p class="desc">Sparse attention, multi-token prediction and a trained drafter: the most instrumented engine here.</p><div class="tags"><span class="tag">reasoning</span><span class="tag">tools</span></div>+ <div class="card"><h3>DeepSeek V4.1 Flash</h3><p class="desc">552B, and 40% of it is an n-gram memory read from disk a few hundred bytes at a time: the model this engine was built for.</p>+ </div><div class="meta"><div class="row"><span class="k">Maker</span><span class="v">DeepSeek AI</span></div><div class="row"><span class="k">Size · RAM</span><span class="v">284B MoE · from 16 GB</span></div></div><a class="btn ghost" href="https://github.com/JustVugg/colibri/blob/main/docs/deepseek-v4.md">Run it →</a></div><div class="card"><h3>Qwen3.6</h3><p class="desc">Gated attention plus Gated DeltaNet. Its CUDA tier keeps hot experts in VRAM: 7x faster than CPU on two 8 GB cards, output bit-identical.</p>@@ -366,7 +368,7 @@ footer li a:hover{color:var(--on-dark)}<div><h4>Models</h4><ul><li><a href="#models">GLM-5.2/5.3</a></li><li><a href="#models">GLM-5.3-Flash</a></li><li><a href="#models">Qwen3.8-Flash-Next</a></li><li><a href="#models">Kimi K3</a></li>- <li><a href="#models">Inkling</a></li><li><a href="#models">DeepSeek V4 Flash</a></li>+ <li><a href="#models">Inkling</a></li><li><a href="#models">DeepSeek V4 Flash</a></li><li><a href="#models">DeepSeek V4.1 Flash</a></li><li><a href="#models">Qwen3.6</a></li><li><a href="#models">OLMoE</a></li></ul></div><div><h4>Project</h4><ul>