Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: MiMo-V2.6 (docs/mimo.md, ENVIRONMENT, READMEs, site)
JustVugg94691e2f74ed
diff --git a/README.it.md b/README.it.mdindex 8c2f3de6..b9cc09e8 100644--- a/README.it.md+++ b/README.it.md@@ -12,8 +12,9 @@ miliardi a 2,8 mila miliardi di parametri** — su hardware consumer ed eterogenin C puro e senza dipendenze del motore, trattando storage, RAM e VRAM comeun'unica gerarchia di inferenza.-Oggi girano nove famiglie: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, con+Oggi girano dieci famiglie: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, convision), **Inkling** (975B), **Kimi K3** (2,8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, con visione),+**MiMo-V2.6 Flash** (309B, con visione),**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) e**OLMoE** (7B) — unfile C ciascuna, la stessa interfaccia `coli chat` / `coli serve` / `coli web`. [Elenco completo](README.md#other-supported-models)@@ -395,8 +396,8 @@ e per il gateway API opzionale.end-to-end, revisionati e sviluppati apertamente.- **Più modelli aperti.** L'algoritmo di tiering è indipendente dal modello:qualsiasi MoE con expert instradati può essere organizzato allo stesso modo.- Nove famiglie funzionano già (GLM-5.2, GLM-5.3-Flash con la vision,- Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next, Qwen3.6, OLMoE);+ Dieci famiglie funzionano già (GLM-5.2, GLM-5.3-Flash con la vision,+ Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash, Qwen3.8-Flash-Next, Qwen3.6, OLMoE);altre famiglie open-weight, **MiniMax** tra le candidate, si guadagnano unengine come le prime otto: quando qualcuno le misura end-to-end.diff --git a/README.ja.md b/README.ja.mdindex f271ce28..24a30df1 100644--- a/README.ja.md+++ b/README.ja.md@@ -17,8 +17,8 @@(AI メモリのマルチティア化)ことで、**744B から 2.8T パラメータのフロンティア MoE モデル**を、コンシューマー向けや異種混在のハードウェア上で、エンジン依存ゼロの純粋な C で実行します。-現在動作するのは 9 つのファミリーです: **GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B、-ビジョン対応)、**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B、ビジョン対応)、+現在動作するのは 10 のファミリーです: **GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B、+ビジョン対応)、**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B、ビジョン対応)、**MiMo-V2.6 Flash**(309B、ビジョン対応)、**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)、そして**OLMoE**(7B)——それぞれが C ファイル 1 つで、同じ `coli chat` / `coli serve` / `coli web` フロントエンドを共有します。@@ -425,6 +425,7 @@ GLM-5.2 がリファレンスモデルですが、同じストリーミング手| **GLM-5.3-Flash**(Z.ai) | 321B / 40B | [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) — ルーティングエキスパートを **int4-gs64** に変換、密部分は BF16 のままで精度はロード時に選択。ビジョン対応 | `make -C c glm53` | [glm53-flash.md](docs/glm53-flash.md) || **Kimi K3**(Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) — オリジナルのチェックポイント、ルーティングエキスパートは **ネイティブ MXFP4** のまま | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) || **DeepSeek V4 Flash** | 284B / 13B | 公式のシャード化チェックポイント — ルーティングエキスパートは **ネイティブ fp4**、密部分は fp8-e4m3 のまま。**REAP で枝刈りした 150B**([`puwaer/DeepSeek-V4-Flash-0731-reap-150b`](https://huggingface.co/puwaer/DeepSeek-V4-Flash-0731-reap-150b)、85 GB、256 個中 132 個のエキスパート)も同じエンジンで変換なしにロード可能 | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |+| **MiMo-V2.6 Flash**(Xiaomi) | 309B / 15B | [`XiaomiMiMo/MiMo-V2.6-Flash-MOPD`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD)(178 GB)、公式チェックポイント、**変換不要**: ルーティングエキスパートは **ネイティブ MXFP4** のまま、密部分は FP8/BF16。48 層のうち 39 層は 128 トークンのウィンドウを参照するため、長いコンテキストでも KV は 9 層分だけ。ビジョンとツール呼び出しに対応 | `make -C c mimo` | [mimo.md](docs/mimo.md) || **DeepSeek V4.1 Flash** | 552B / 16B | 公式チェックポイント、**変換不要**: エキスパートはすでに fp4、密部分は fp8-e4m3。そのうち 203 GB は一度に数百バイトずつディスクから読まれる n-gram メモリで、ルーティングエキスパートのコストは GLM-5.2 の 12.7 GB に対して **1 トークンあたり 4.5 GB**。ビジョン、ツール呼び出し、DSpark ドラフトヘッドはすべて有効 | `make -C c deepseek_v41` | [deepseek-v41.md](docs/deepseek-v41.md) || **Qwen3.8-Flash-Next**(Alibaba) | 125B + 51B n-gram / 6B | [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) — オリジナルのチェックポイント。PLE はページング可能なまま、エキスパートは **ネイティブのブロック FP8** のまま | `make -C c qwen38`(CPU のみ) | [qwen38.md](docs/qwen38.md) || **Qwen3.6**(Alibaba) | 35B / 3B | [`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64)(約 20 GB、**推奨**)— Gated Attention + Gated DeltaNet のハイブリッド | `make -C c qwen36`(VRAM エキスパートティアには `CUDA=1`) | [qwen36.md](docs/qwen36.md) |@@ -581,8 +582,8 @@ CUDA ティア(ビルド、DLL の選択、GPU の対応範囲)、環境変目的はハードウェア要件と有用トークンあたりのコストを下げることです。すべてはこのプロジェクトのやり方で取り込まれます: エンドツーエンドで計測され、レビューされ、オープンに開発されます。- **より多くのオープンモデル。** ティアリングアルゴリズムはモデルに依存しません。ルーティング- エキスパートを持つ MoE であれば、どれも同じ方法でステージングできます。現在 9 つのファミリーが- 動作しています(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、+ エキスパートを持つ MoE であれば、どれも同じ方法でステージングできます。現在 10 のファミリーが+ 動作しています(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、MiMo-V2.6 Flash、Qwen3.8-Flash-Next、Qwen3.6、OLMoE)。さらなるオープンウェイトのファミリー — 候補には**MiniMax** も含まれます — は、最初の 8 つと同じ方法でエンジンを獲得します:誰かがエンドツーエンドで計測したときにです。diff --git a/README.md b/README.mdindex df75bf78..92c4f8fc 100644--- a/README.md+++ b/README.md@@ -18,8 +18,9 @@ parameters** — on consumer and heterogeneous hardware, in pure C with zeroengine dependencies, by treating storage, RAM, and VRAM as a single inferencehierarchy (AI memory multitiering).-Nine families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, with+Ten families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, withvision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, with vision),+**MiMo-V2.6 Flash** (309B, with vision),**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) and**OLMoE** (7B) —one C file each, the same `coli chat` / `coli serve` / `coli web` front end.@@ -471,6 +472,7 @@ the model's `config.json`):| **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) — original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) || **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint — routed experts stay **native fp4**, dense stays fp8-e4m3; the **REAP-pruned 150B** ([`puwaer/DeepSeek-V4-Flash-0731-reap-150b`](https://huggingface.co/puwaer/DeepSeek-V4-Flash-0731-reap-150b), 85 GB, 132 of 256 experts) loads with the same engine and no conversion | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) || **DeepSeek V4.1 Flash** | 552B / 16B | official checkpoint, **no conversion**: experts are already fp4, dense is fp8-e4m3. 203 GB of it is an n-gram memory read from disk a few hundred bytes at a time, and the routed experts cost **4.5 GB per token** against GLM-5.2's 12.7. Vision, tool calling and the DSpark draft head are all on | `make -C c deepseek_v41` | [deepseek-v41.md](docs/deepseek-v41.md) |+| **MiMo-V2.6 Flash** (Xiaomi) | 309B / 15B | [`XiaomiMiMo/MiMo-V2.6-Flash-MOPD`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD) (178 GB), official checkpoint, **no conversion**: routed experts stay **native MXFP4**, dense stays FP8/BF16. 39 of its 48 layers attend a 128-token window, so a long context costs the KV of 9. Vision and tool calling on | `make -C c mimo` | [mimo.md](docs/mimo.md) || **Qwen3.8-Flash-Next** (Alibaba) | 125B + 51B n-gram / 6B | [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) — original checkpoint; PLE stays pageable and experts stay **native block-FP8** | `make -C c qwen38` (`CUDA=1` for the VRAM expert tier) | [qwen38.md](docs/qwen38.md) || **Qwen3.6** (Alibaba) | 35B / 3B | [`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64) (~20 GB, **recommended**) — hybrid Gated Attention + Gated DeltaNet | `make -C c qwen36` (`CUDA=1` for the VRAM expert tier) | [qwen36.md](docs/qwen36.md) || **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` — **int8** container, ~7 GB | `make -C c olmoe` | — |@@ -665,9 +667,9 @@ checkpoint validation, and the generated tiny independent oracle.lower cost per useful token. Everything lands the way this project works:measured end to end, reviewed, and developed in the open.- **More open models.** The tiering algorithm is model-agnostic: any MoE with- routed experts can be staged the same way. Nine families run today (GLM-5.2,- GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next,- Qwen3.6, OLMoE); further open-weight families — **MiniMax** among the+ routed experts can be staged the same way. Ten families run today (GLM-5.2,+ GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash,+ Qwen3.8-Flash-Next, Qwen3.6, OLMoE); further open-weight families — **MiniMax** among thecandidates — earn an engine the way the first eight did: when someonemeasures one end to end.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 188d97cf..3114d74c 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -10,8 +10,8 @@**小巧引擎,庞大模型。**在消费级与异构硬件上运行**前沿 MoE 模型——从 744B 到2.8T 参数**——以引擎零依赖的纯 C 实现,将存储、RAM 与 VRAM 视为统一的推理层级。-目前可运行九个模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含视觉)、-**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B,含视觉)、+目前可运行十个模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含视觉)、+**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B,含视觉)、**MiMo-V2.6 Flash**(309B,含视觉)、**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)与 **OLMoE**(7B)——各自一个 C 文件,共用同一套 `coli chat` / `coli serve` / `coli web` 前端。[完整列表](README.md#other-supported-models)也能生成图像:**Qwen-Image-2.1** 根据文字生成图片,`coli chat` 直接在终端中显示,@@ -357,7 +357,7 @@ oracle 说明,请参阅[中文版 DeepSeek V4 文档](docs/deepseek-v4.zh-CN.m放置、调度、I/O、CPU/GPU 内核、异构重叠、KV 状态与路由感知推测。目标是降低硬件要求和每个有效 token 的成本,所有成果都以端到端测量为准、经审查并公开开发。- **支持更多开放模型**。层级算法与模型无关,任何带路由专家的 MoE 都能用相同方式分层。- 目前已有九个模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、+ 目前已有十个模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、MiMo-V2.6 Flash、Qwen3.8-Flash-Next、Qwen3.6、OLMoE);更多开放权重家族(候选包括 **MiniMax**)将沿用同样的规则获得引擎支持:有人完成端到端实测之后。diff --git a/README.zh-TW.md b/README.zh-TW.mdindex db1e978b..cf94d02f 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -10,8 +10,8 @@**小巧引擎,龐大模型。**在消費級與異質硬體上執行**前沿 MoE 模型——從 744B 到2.8T 參數**——以引擎零相依套件的純 C 實作,將儲存、RAM 與 VRAM 視為統一的推論階層。-目前可執行九個模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含視覺)、-**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B,含視覺)、+目前可執行十個模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含視覺)、+**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B,含視覺)、**MiMo-V2.6 Flash**(309B,含視覺)、**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)與 **OLMoE**(7B)——各自一個 C 檔案,共用同一套 `coli chat` / `coli serve` / `coli web` 前端。[完整清單](README.md#other-supported-models)也能生成圖像:**Qwen-Image-2.1** 依據文字生成圖片,`coli chat` 直接在終端機中顯示,@@ -337,7 +337,7 @@ curl -s http://127.0.0.1:8000/v1/brio -H 'Content-Type: application/json' -d '{配置、排程、I/O、CPU/GPU 核心、異質重疊、KV 狀態與路由感知推測。目標是降低硬體要求和每個有效 token 的成本,所有成果都以端到端測量為準、經審查並公開開發。- **支援更多開放模型。**階層演算法與模型無關,任何帶路由專家的 MoE 都能用相同方式分層。- 目前已有九個模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、+ 目前已有十個模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、MiMo-V2.6 Flash、Qwen3.8-Flash-Next、Qwen3.6、OLMoE);更多開放權重家族(候選包括 **MiniMax**)將沿用同樣的規則獲得引擎支援:有人完成端到端實測之後。diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 044a4498..8538c9de 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -6,7 +6,7 @@ Reference for the environment variables read by the colibrì engine.## Which program reads these?-**There are nine engine binaries, and they do not share a knob set.** The main+**There are ten engine binaries, and they do not share a knob set.** The mainengine `c/colibri` (built from `c/colibri.c`, formerly `glm.c`) reads most ofwhat follows, but the sister engines read their own:@@ -21,6 +21,7 @@ what follows, but the sister engines read their own:| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` — see [OLMoE engine](#olmoe-engine-olmoe) || `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates — see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read — its GPU switch is `DSV4_CUDA` || `deepseek_v41` | `c/deepseek_v41.c` | the `V41_*` family: see [DeepSeek V4.1 engine](#deepseek-v41-engine-deepseek_v41) |+| `mimo` | `c/mimo.c` | the `MIMO_*` family: see [MiMo-V2.6 engine](#mimo-v26-engine-mimo) |Setting an `INK_*` variable while running `colibri` does nothing, and viceversa; nothing warns you about it. A few variables are genuinely shared because@@ -492,6 +493,25 @@ Read **only** by `c/deepseek_v41.c`. See [deepseek-v41.md](deepseek-v41.md).| `V41_DSPARK_MINACC` | 60 | DeepSeek V4.1: percent of drafts that must be accepted over a window of ten before drafting pauses for 64 tokens. 60 is the measured break-even. || `V41_SPEC_FORCE` | unset | DeepSeek V4.1, oracle mode only: draft the reference's own tokens (`1`), corrupt the last one (`2`), keep the head's (`3`), corrupt the first one (`4`) or a different one each round (`5`), so the verification path runs on a fixture whose draft head is random noise. |+## MiMo-V2.6 engine (`mimo`)++Read **only** by `c/mimo.c` (and `MIMO_MAX_IMAGE_TOKENS` by the gateway). See [mimo.md](mimo.md).++| Variable | Default | Effect |+|---|---|---|+| `MIMO_DENSE_BITS` | 0 | MiMo: 0 keeps the dense weights as released (FP8, BF16), exact; 8 is int8 per row, less RAM, not exact; 32 is f32, the oracle's configuration. |+| `MIMO_IDOT` | 0 | MiMo: 1 quantizes the expert matmuls' activations to int8 per 32 (faster, not exact). |+| `MIMO_DIRECT` | 1 | MiMo: 0 reads experts through the page cache instead of `O_DIRECT`. |+| `MIMO_READ_THREADS` | 8 | MiMo: parallel expert reads per layer. |+| `MIMO_CHUNK` | 64 | MiMo: prompt tokens per prefill block. |+| `MIMO_CTX` | `CTX`, else 8192 | MiMo: context the KV cache is sized for, capped by the checkpoint. |+| `MIMO_CAP` | 64 | MiMo: expert cache slots per layer when no argument gives one; never below one routing step. |+| `MIMO_MAX_IMAGE_TOKENS` | unset | MiMo, gateway: ceiling on what one picture costs in prompt tokens. |+| `MIMO_LOGITS` | unset | MiMo, oracle: dump every prompt position's logits (f32) to this file. |+| `MIMO_TRACE` | unset | MiMo, oracle: dump the residual after every sublayer of the first block. |+| `MIMO_DIRS` | unset | MiMo: extra directories holding shards. |+| `MIMO_STATS` | unset | MiMo: report the vision tower's time per picture. |+## OLMoE engine (`olmoe`)Read **only** by `c/olmoe.c`. This is the sister engine used for streaming-cache research, so most of these are experiment knobs.diff --git a/docs/docs.json b/docs/docs.jsonindex fcf76af3..fabfa78c 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -23,7 +23,7 @@},{"group": "Model engines",- "pages": ["deepseek-v4", "kimi_k3", "qwen38", "qwen36", "inkling", "qwen-image"]+ "pages": ["deepseek-v4", "kimi_k3", "qwen38", "qwen36", "inkling", "mimo", "qwen-image"]},{"group": "Configure and tune",diff --git a/docs/mimo.md b/docs/mimo.mdnew file mode 100644index 00000000..bc891254--- /dev/null+++ b/docs/mimo.md@@ -0,0 +1,140 @@+# MiMo-V2.6 (Xiaomi)++Xiaomi's MiMo-V2.6, MIT licensed, in two sizes with one architecture:++| | Flash | Pro |+|---|---|---|+| parameters | 309B, 15B active | 1.02T, 42B active |+| layers | 48: 39 sliding window + 9 full attention | 70: 60 + 10 |+| routed experts | 256, top-8, no shared expert | 384, top-8 |+| on disk | 177.7 GB | 573.5 GB |+| experts per token (cold) | 8 x 47 layers x 12.75 MiB = about 5 GB | about 11 GB |++The checkpoint is read as it is released, **no conversion**:++- **routed experts** are MXFP4 (e2m1 values two per byte, one e8m0 scale per 32+ columns), the same layout colibri already reads for Kimi K3 and DeepSeek V4.1.+ The six tensors of an expert sit back to back in the shard, so an expert is one+ read (`O_DIRECT` by default).+- **dense weights** stay as released: FP8 e4m3 with a scale per 128x128 block for+ qkv and the layer-0 MLP, BF16 for o_proj, router, norms, embeddings and head.+ Re-quantizing the experts to colibri's int4 would cost more bytes (4.5 bits per+ weight against MXFP4's 4.25) and a second rounding.++## Running it++```sh+hf download XiaomiMiMo/MiMo-V2.6-Flash-MOPD --local-dir ~/Models/MiMo-V2.6-Flash+make -C c mimo+coli chat --model ~/Models/MiMo-V2.6-Flash+```++`coli chat`, `coli serve` and `coli web` size the expert cache from the resource plan+when `--cap` is omitted; `--ram N` gives it a budget. The standalone engine takes the+cache (experts per layer) as its first argument, and never goes below one routing+step (8).++```sh+SNAP=~/Models/MiMo-V2.6-Flash SERVE=1 ./c/mimo 64 # the SERVE protocol+./c/mimo ~/Models/MiMo-V2.6-Flash --prompt "Ciao" --ngen 64 # one prompt, by hand+```++## What the engine implements++Each piece mirrors `modeling_mimo_v2.py` from the release:++- **hybrid attention**: `hybrid_layer_pattern` marks each layer sliding window (1) or+ full attention (0). The two kinds have their own KV head counts (8 and 4 on Flash)+ and their own RoPE theta (1e4 and 1e7). A windowed layer keeps a ring of 128+ positions, so on Flash a long context costs the KV of 9 layers.+- **partial RoPE** on the first `int(192 * 0.334) = 64` dims of each q/k head,+ rotate-half style; queries and keys are 192 wide, values 128.+- **value scale**: values are multiplied by `attention_value_scale` (0.707 Flash,+ 0.612 Pro) before attention.+- **attention sink** on the windowed layers: one logit per head that joins the+ softmax denominator and nothing else.+- **router**: sigmoid scores; `e_score_correction_bias` chooses the experts but does+ not weigh them; the chosen scores are renormalized.+- **vision**: the MiMo ViT (681M on the release). 2D RoPE, grouped-query attention,+ full attention in 4 of its 28 blocks and a 1D window of 64 tokens in the others,+ alternating row and column order of the 2x2 merge units, a sink on the first key+ of the window blocks, and a LayerNorm + GELU merger. Pictures are preprocessed by+ the gateway exactly as the Qwen2-VL processor does (`tools/qwen38_image.py`).++One storage detail the modeling code does not describe: the fused `qkv_proj` is+**pre-sharded** in `num_key_value_heads` chunks, `[Q_0|K_0|V_0|Q_1|K_1|V_1|...]`, with+the FP8 scale grid tiled per chunk (a Flash full-attention layer has 108 scale rows,+not the 106 a contiguous grid would have). The layout is the one vLLM loads+(`vllm/model_executor/models/mimo_v2.py`); the engine undoes it at load time.++Not implemented yet: the three MTP layers and the separate DFlash drafter (no+speculative decoding), audio and video input.++## Chat template++The gateway renders MiMo's `chat_template.jinja` by hand, byte for byte (held to the+real template by `tests/test_mimo_chat_template.py`): ChatML with no newline after+`<|im_end|>`, every assistant turn carrying its `<think>` block, tools in their own+system turn, calls written inline as+`<tool_call><function=NAME><parameter=KEY>VALUE</parameter></function></tool_call>`,+tool results as a `tool` turn. Thinking is on unless the request turns it off+(`enable_thinking: false`, or `reasoning_effort: "none"`), as in the template; with+thinking on the cue ends in `<think>`, the token the model writes first.++## Correctness++`make -C c mimo-tiny-check` generates a tiny checkpoint (numpy, deterministic) stored+exactly as the release stores its tensors, and compares the engine with tokens from+Xiaomi's own `modeling_mimo_v2.py` (vendor files pinned by SHA-256,+`tools/make_mimo_ref.py`): greedy and teacher-forced, with and without a picture,+every exact configuration of the engine (native dense, an expert cache that evicts+at every layer, prefill in blocks of 1 and 3), and checks that corrupting an expert+scale, the qkv scale grid or the picture changes the answer. The gateway's patches+match the official Qwen2-VL processor bit for bit on the fixture's picture.++On the real checkpoint, `tools/mimo_real_check.py` builds Xiaomi's model from the+release's own `modeling_mimo_v2.py` (experts made one at a time as the vendor's+`MiMoV2MLP`, so 48 layers fit in RAM) and compares it with the engine on the same bytes.+Measured on MiMo-V2.6-Flash-MOPD:++| | argmax | max logit difference | max KL |+|---|---|---|---|+| text, 2 layers, 25 positions | 25/25 | 6.3e-7 of the logit range | 1.4e-11 |+| text, 48 layers, 25 positions | 25/25 | 2.5e-6 of the logit range | 1.9e-10 |+| text with a picture, 48 layers, 271 positions | 271/271 | 1.7e-4 of the logit range | 1.0e-6 |++The ViT's 252 output rows for that picture differ from Xiaomi's by at most 3.7e-5+(relative 1.6e-5), and the gateway's patches equal the official processor's exactly.++## Measured++A first measurement, not a benchmark: the machine (Ryzen 7 PRO 8700GE, 8 cores, 64 GB,+NVMe RAID) was running two other jobs on about 13 of its 16 threads, and the numbers+move with that.++| Flash, 32 cached experts per layer, dense as released | |+|---|---|+| resident | 30 GB (dense 10 GB, expert cache up to 19 GB) |+| 24-token prompt from a cold cache | 21.8 s |+| decode, 4 threads | 0.89 tok/s |+| expert reads | 5.8 GB/s, about 1.7 GB per token while the cache warms |++Through the gateway: a tool call comes back as `get_weather({"city": "Roma", "days": 3})`+with the integer typed as declared, and asked what is written in a picture made by+colibri's Qwen-Image the model answers with its text, word for word.++## Environment++| Variable | Default | Effect |+|---|---|---|+| `MIMO_DENSE_BITS` | 0 | 0: dense weights as released (FP8, BF16), exact. 8: int8 per row, less RAM, not exact. 32: f32, the oracle's configuration. |+| `MIMO_IDOT` | 0 | 1: int8 activations for the expert matmuls (faster, not exact). |+| `MIMO_DIRECT` | 1 | 0: buffered expert reads instead of `O_DIRECT`. |+| `MIMO_READ_THREADS` | 8 | Parallel expert reads per layer. |+| `MIMO_CHUNK` | 64 | Prompt tokens per prefill block. |+| `MIMO_CTX` / `CTX` | 8192 | Context the KV cache is sized for (capped by the checkpoint). |+| `MIMO_CAP` | 64 | Expert cache slots per layer when no argument gives one. |+| `MIMO_MAX_IMAGE_TOKENS` | unset | Gateway: ceiling on what one picture costs in prompt tokens. |+| `MIMO_LOGITS` | unset | Oracle: dump every prompt position's logits (f32) to this file. |+| `MIMO_TRACE` | unset | Oracle: dump the residual after every sublayer of the first block. |+| `MIMO_DIRS` | unset | Extra directories holding shards (multi-disk). |diff --git a/site/index.html b/site/index.htmlindex ca0b9d10..ffbf4966 100644--- a/site/index.html+++ b/site/index.html@@ -381,10 +381,10 @@ footer{border-top:1px solid var(--line)}<h2>Frontier open models</h2><p class="sub">Nine families run today, one hand-written C file each. Not a wrapper aroundsomeone else's runtime.</p>- <div class="chart"><div class="crow"><span class="cl">kimi-k3<s>104B active</s></span><span class="ct"><i style="width:100.0%"></i></span><span class="cv">2.8T</span></div><div class="crow"><span class="cl">inkling<s>41B active</s></span><span class="ct"><i style="width:34.8%"></i></span><span class="cv">975B</span></div><div class="crow"><span class="cl">glm-5.2<s>40B active</s></span><span class="ct"><i style="width:26.6%"></i></span><span class="cv">744B</span></div><div class="crow"><span class="cl">deepseek-v4.1-flash<s>16B active</s></span><span class="ct"><i style="width:19.7%"></i></span><span class="cv">552B</span></div><div class="crow"><span class="cl">glm-5.3-flash<s>18B active</s></span><span class="ct"><i style="width:11.4%"></i></span><span class="cv">320B</span></div><div class="crow"><span class="cl">deepseek-v4-flash<s>13B active</s></span><span class="ct"><i style="width:10.1%"></i></span><span class="cv">284B</span></div><div class="crow"><span class="cl">qwen3.8-flash-next<s>6B active</s></span><span class="ct"><i style="width:6.4%"></i></span><span class="cv">180B</span></div><div class="crow"><span class="cl">qwen3.6-35b-a3b<s>3B active</s></span><span class="ct"><i style="width:1.2%"></i></span><span class="cv">35B</span></div><div class="crow"><span class="cl">olmoe<s>1B active</s></span><span class="ct"><i style="width:0.8%"></i></span><span class="cv">7B</span></div></div>+ <div class="chart"><div class="crow"><span class="cl">kimi-k3<s>104B active</s></span><span class="ct"><i style="width:100.0%"></i></span><span class="cv">2.8T</span></div><div class="crow"><span class="cl">inkling<s>41B active</s></span><span class="ct"><i style="width:34.8%"></i></span><span class="cv">975B</span></div><div class="crow"><span class="cl">glm-5.2<s>40B active</s></span><span class="ct"><i style="width:26.6%"></i></span><span class="cv">744B</span></div><div class="crow"><span class="cl">deepseek-v4.1-flash<s>16B active</s></span><span class="ct"><i style="width:19.7%"></i></span><span class="cv">552B</span></div><div class="crow"><span class="cl">mimo-v2.6-flash<s>15B active</s></span><span class="ct"><i style="width:11.0%"></i></span><span class="cv">309B</span></div><div class="crow"><span class="cl">glm-5.3-flash<s>18B active</s></span><span class="ct"><i style="width:11.4%"></i></span><span class="cv">320B</span></div><div class="crow"><span class="cl">deepseek-v4-flash<s>13B active</s></span><span class="ct"><i style="width:10.1%"></i></span><span class="cv">284B</span></div><div class="crow"><span class="cl">qwen3.8-flash-next<s>6B active</s></span><span class="ct"><i style="width:6.4%"></i></span><span class="cv">180B</span></div><div class="crow"><span class="cl">qwen3.6-35b-a3b<s>3B active</s></span><span class="ct"><i style="width:1.2%"></i></span><span class="cv">35B</span></div><div class="crow"><span class="cl">olmoe<s>1B active</s></span><span class="ct"><i style="width:0.8%"></i></span><span class="cv">7B</span></div></div><p class="src">Total parameters. Only the active ones are computed per token, and only therouted experts move. The engines, by the names the docs use:- GLM-5.3-Flash, GLM-5.2/5.3, Inkling, Kimi K3, OLMoE, Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, DeepSeek V4 Flash, DeepSeek V4.1 Flash.+ GLM-5.3-Flash, GLM-5.2/5.3, Inkling, Kimi K3, OLMoE, Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, DeepSeek V4 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash.And for images, Qwen-Image-2.1: text to picture, drawn inside the terminal.</p><a class="more" href="#models">Browse all models →</a></div>