Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs(readme): catch the README up with what already runs
JustVugg6211ccd26f7e
diff --git a/README.it.md b/README.it.mdindex 5d61892c..107d24b1 100644--- a/README.it.md+++ b/README.it.md@@ -32,7 +32,7 @@ ma non ridefinire il modello di nascosto.```$ ./coli chat- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -254,7 +254,7 @@ compilatore necessario. Prendi l'archivio della tua piattaforma dalla pagina[Releases](https://github.com/JustVugg/colibri/releases) e scompattalo:```bash-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibripython3 coli info # engine ready ✓```@@ -336,8 +336,10 @@ e per il gateway API opzionale.end-to-end, revisionati e sviluppati apertamente.- **Più modelli aperti.** L'algoritmo di tiering è indipendente dal modello:qualsiasi MoE con expert instradati può essere organizzato allo stesso modo.- GLM-5.2 e OLMoE funzionano già; **Kimi K2**, **Qwen3 MoE** e **MiniMax** sono- nella roadmap.+ Sei famiglie funzionano già (GLM-5.2, Inkling, Kimi K3, DeepSeek V4 Flash,+ Qwen3.6, OLMoE); altre famiglie open-weight, **MiniMax** tra le candidate,+ si guadagnano un engine come le prime sei: quando qualcuno le misura+ end-to-end.## Sostenere il progettodiff --git a/README.md b/README.mdindex 164a2aae..b4effc89 100644--- a/README.md+++ b/README.md@@ -38,7 +38,7 @@ may reduce speed; it must not quietly redefine the model.```$ ./coli chat- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -343,7 +343,7 @@ Take the archive for your platform from[Releases](https://github.com/JustVugg/colibri/releases) and unpack it:```bash-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibripython3 coli info # engine ready ✓```@@ -406,7 +406,7 @@ the model's `config.json`):> | **GLM-5.2** | ~372 GB | 16 GB min, 24 GB comfortable | not needed |> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |-> | **DeepSeek V4 Flash** | ~167 GB | 16 GB min, 32 GB comfortable | optional; an NVIDIA card (any sm_80+, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |+> | **DeepSeek V4 Flash** | ~167 GB | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |> | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU |>> A GPU only ever makes it faster. Speed is set by your disk, because the experts@@ -432,6 +432,14 @@ without a code path of its own. With `CUDA=1` the VRAM expert tier measuredKimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight fromthe original Hugging Face shards, and the bf16 dense set is quantized at load time.+Long agent sessions can opt into recurrent-state checkpoints (`COLI_K3_CKPT=N`+slots in RAM, or parked on disk with `COLI_K3_CKPT_DIR`): an edited or follow-up+prompt restores the deepest surviving checkpoint and re-prefills only the tail,+instead of replaying the whole conversation through the SSM layers. On Vulkan+hosts `K3_VK_UP=auto` sizes the expert tier upload from measured bandwidth. The+engine's KDA and MLA paths are validated token-exact in CI against the vendor+implementation.+Inkling ships int4 experts but **bf16 dense weights** (49.4 GB resident); on a hostthat cannot hold those, [inkling.md](docs/inkling.md) has a one-pass tool that bringsthe dense set to 15.3 GB and lets the 975B run on a 25 GB box — with the honest@@ -519,6 +527,15 @@ python ./coli chat --model /path/to/DeepSeek-V4-Flash --ram 32# Windows CUDA tier: make cuda-dsv4-dll CUDA_ARCH=portable (+ make cuda-dsv4-dg-dll on RTX 50)```+Two opt-in GPU levers are new and looking for community numbers, both default+off and byte-identical when unset: `DSV4_HYBRID=1` splits VRAM-tier misses+between the GPU fill branch and the CPU branch using bandwidths measured at+runtime, and `COLI_CUDA_MOE_DOUBLE=1` (on top of `COLI_CUDA_MOE_BATCH=1`)+prefetches the next layer's full expert set into a second VRAM bank while the+current layer computes, falling back to the single bank when VRAM is short.+The CUDA tier also runs on Pascal and Turing cards now (GTX 10 / RTX 20+series): build with `CUDA_ARCH=portable-pre-ampere NO_TC=1`.+Greedy decode and one KV slot. Tool calling is wired through the HTTP gatewaywith V4's native prompt and DSML call blocks; grammar is not supported. See the[per-engine API matrix](docs/api.md#tool-calling-support). Prefix checkpoints@@ -554,9 +571,10 @@ checkpoint validation, and the generated tiny independent oracle.lower cost per useful token. Everything lands the way this project works:measured end to end, reviewed, and developed in the open.- **More open models.** The tiering algorithm is model-agnostic: any MoE with- routed experts can be staged the same way. GLM-5.2 and OLMoE run today;- support for more open-weight families — **Kimi K2** (Moonshot AI),- **Qwen3 MoE** (Alibaba), **MiniMax** — is on the roadmap.+ routed experts can be staged the same way. Six families run today (GLM-5.2,+ Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.6, OLMoE); further open-weight+ families — **MiniMax** among the candidates — earn an engine the way the+ first six did: when someone measures one end to end.## Supporting the projectdiff --git a/README.zh-CN.md b/README.zh-CN.mdindex c8341105..a887d49a 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -24,7 +24,7 @@ Colibrì 刻意用于验证激进的系统思路——因此**对速度不作 SL```$ ./coli chat- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -216,7 +216,7 @@ MTP head 必须是 **int8**(int4 head 的接受率会崩塌到 0–4%,见[Releases](https://github.com/JustVugg/colibri/releases) 下载对应平台的压缩包并解压:```bash-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibripython3 coli info # engine ready ✓```@@ -285,10 +285,12 @@ COLI_MODEL=/nvme/glm52_i4 ./coli doctor # 只读就绪检查## DeepSeek V4-实验性的 **DeepSeek V4 Flash** CPU 路径使用原生 FP4 专家、自动 RAM-规划、共享的 `st.h`/`quant.h` 基础设施,以及常驻 target engine。-DSpark 有意保留给紧随其后的独立 stacked PR。本 target engine 支持-x86-64/aarch64 Linux 与 Windows/MSYS2。+**DeepSeek V4 Flash** 直接流式读取官方 checkpoint,无需转换:路由专家+保持原生 FP4,稠密部分保持 fp8-e4m3。支持 x86-64/aarch64 Linux 与+Windows/MSYS2(CPU),并提供可选的 CUDA 层(Windows 运行时 DLL;Linux+`CUDA=1` 直接链接)——GTX 10 系及更新的 NVIDIA 显卡均可使用(Pascal/Turing+需 `CUDA_ARCH=portable-pre-ampere NO_TC=1` 构建),每个阶段都保持 CPU 语义一致并可+逐阶段回退。```bashcd c@@ -308,8 +310,9 @@ oracle 说明,请参阅[中文版 DeepSeek V4 文档](docs/deepseek-v4.zh-CN.m放置、调度、I/O、CPU/GPU 内核、异构重叠、KV 状态与路由感知推测。目标是降低硬件要求和每个有效 token 的成本,所有成果都以端到端测量为准、经审查并公开开发。- **支持更多开放模型。**层级算法与模型无关,任何带路由专家的 MoE 都能用相同方式分层。- GLM-5.2 与 OLMoE 目前可用;**Kimi K2**、**Qwen3 MoE**、**MiniMax** 等开放权重模型- 已列入路线图。+ 目前已有六个模型家族可用(GLM-5.2、Inkling、Kimi K3、DeepSeek V4 Flash、+ Qwen3.6、OLMoE);更多开放权重家族(候选包括 **MiniMax**)将沿用同样的+ 规则获得引擎支持:有人完成端到端实测之后。## 支持项目diff --git a/README.zh-TW.md b/README.zh-TW.mdindex 2cb90082..aff2c255 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -24,7 +24,7 @@ Colibrì 刻意用於驗證激進的系統構想——因此**對速度不作 SL```$ ./coli chat- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -217,7 +217,7 @@ MTP head 必須是 **int8**(int4 head 的接受率會崩落到 0–4%,見[Releases](https://github.com/JustVugg/colibri/releases) 下載對應平台的壓縮檔並解壓:```bash-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibripython3 coli info # engine ready ✓```@@ -290,8 +290,9 @@ COLI_MODEL=/nvme/glm52_i4 ./coli doctor # 唯讀就緒檢查配置、排程、I/O、CPU/GPU 核心、異質重疊、KV 狀態與路由感知推測。目標是降低硬體要求和每個有效 token 的成本,所有成果都以端到端測量為準、經審查並公開開發。- **支援更多開放模型。**階層演算法與模型無關,任何帶路由專家的 MoE 都能用相同方式分層。- GLM-5.2 與 OLMoE 目前可用;**Kimi K2**、**Qwen3 MoE**、**MiniMax** 等開放權重模型- 已列入路線圖。+ 目前已有六個模型家族可用(GLM-5.2、Inkling、Kimi K3、DeepSeek V4 Flash、+ Qwen3.6、OLMoE);更多開放權重家族(候選包括 **MiniMax**)將沿用同樣的+ 規則獲得引擎支援:有人完成端到端實測之後。## 支持專案