Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs(readme): catch the README up with what already runs

JustVugg6211ccd26f7e
diff --git a/README.it.md b/README.it.md
index 5d61892c..107d24b1 100644
--- a/README.it.md
+++ b/README.it.md
@@ -32,7 +32,7 @@ ma non ridefinire il modello di nascosto.
```
$ ./coli chat
- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
@@ -254,7 +254,7 @@ compilatore necessario. Prendi l'archivio della tua piattaforma dalla pagina
[Releases](https://github.com/JustVugg/colibri/releases) e scompattalo:
```bash
-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri
+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info # engine ready ✓
```
@@ -336,8 +336,10 @@ e per il gateway API opzionale.
end-to-end, revisionati e sviluppati apertamente.
- **Più modelli aperti.** L'algoritmo di tiering è indipendente dal modello:
qualsiasi MoE con expert instradati può essere organizzato allo stesso modo.
- GLM-5.2 e OLMoE funzionano già; **Kimi K2**, **Qwen3 MoE** e **MiniMax** sono
- nella roadmap.
+ Sei famiglie funzionano già (GLM-5.2, Inkling, Kimi K3, DeepSeek V4 Flash,
+ Qwen3.6, OLMoE); altre famiglie open-weight, **MiniMax** tra le candidate,
+ si guadagnano un engine come le prime sei: quando qualcuno le misura
+ end-to-end.
## Sostenere il progetto
diff --git a/README.md b/README.md
index 164a2aae..b4effc89 100644
--- a/README.md
+++ b/README.md
@@ -38,7 +38,7 @@ may reduce speed; it must not quietly redefine the model.
```
$ ./coli chat
- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
@@ -343,7 +343,7 @@ Take the archive for your platform from
[Releases](https://github.com/JustVugg/colibri/releases) and unpack it:
```bash
-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri
+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info # engine ready ✓
```
@@ -406,7 +406,7 @@ the model's `config.json`):
> | **GLM-5.2** | ~372 GB | 16 GB min, 24 GB comfortable | not needed |
> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |
> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |
-> | **DeepSeek V4 Flash** | ~167 GB | 16 GB min, 32 GB comfortable | optional; an NVIDIA card (any sm_80+, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |
+> | **DeepSeek V4 Flash** | ~167 GB | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |
> | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU |
>
> A GPU only ever makes it faster. Speed is set by your disk, because the experts
@@ -432,6 +432,14 @@ without a code path of its own. With `CUDA=1` the VRAM expert tier measured
Kimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight from
the original Hugging Face shards, and the bf16 dense set is quantized at load time.
+Long agent sessions can opt into recurrent-state checkpoints (`COLI_K3_CKPT=N`
+slots in RAM, or parked on disk with `COLI_K3_CKPT_DIR`): an edited or follow-up
+prompt restores the deepest surviving checkpoint and re-prefills only the tail,
+instead of replaying the whole conversation through the SSM layers. On Vulkan
+hosts `K3_VK_UP=auto` sizes the expert tier upload from measured bandwidth. The
+engine's KDA and MLA paths are validated token-exact in CI against the vendor
+implementation.
+
Inkling ships int4 experts but **bf16 dense weights** (49.4 GB resident); on a host
that cannot hold those, [inkling.md](docs/inkling.md) has a one-pass tool that brings
the dense set to 15.3 GB and lets the 975B run on a 25 GB box — with the honest
@@ -519,6 +527,15 @@ python ./coli chat --model /path/to/DeepSeek-V4-Flash --ram 32
# Windows CUDA tier: make cuda-dsv4-dll CUDA_ARCH=portable (+ make cuda-dsv4-dg-dll on RTX 50)
```
+Two opt-in GPU levers are new and looking for community numbers, both default
+off and byte-identical when unset: `DSV4_HYBRID=1` splits VRAM-tier misses
+between the GPU fill branch and the CPU branch using bandwidths measured at
+runtime, and `COLI_CUDA_MOE_DOUBLE=1` (on top of `COLI_CUDA_MOE_BATCH=1`)
+prefetches the next layer's full expert set into a second VRAM bank while the
+current layer computes, falling back to the single bank when VRAM is short.
+The CUDA tier also runs on Pascal and Turing cards now (GTX 10 / RTX 20
+series): build with `CUDA_ARCH=portable-pre-ampere NO_TC=1`.
+
Greedy decode and one KV slot. Tool calling is wired through the HTTP gateway
with V4's native prompt and DSML call blocks; grammar is not supported. See the
[per-engine API matrix](docs/api.md#tool-calling-support). Prefix checkpoints
@@ -554,9 +571,10 @@ checkpoint validation, and the generated tiny independent oracle.
lower cost per useful token. Everything lands the way this project works:
measured end to end, reviewed, and developed in the open.
- **More open models.** The tiering algorithm is model-agnostic: any MoE with
- routed experts can be staged the same way. GLM-5.2 and OLMoE run today;
- support for more open-weight families — **Kimi K2** (Moonshot AI),
- **Qwen3 MoE** (Alibaba), **MiniMax** — is on the roadmap.
+ routed experts can be staged the same way. Six families run today (GLM-5.2,
+ Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.6, OLMoE); further open-weight
+ families — **MiniMax** among the candidates — earn an engine the way the
+ first six did: when someone measures one end to end.
## Supporting the project
diff --git a/README.zh-CN.md b/README.zh-CN.md
index c8341105..a887d49a 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -24,7 +24,7 @@ Colibrì 刻意用于验证激进的系统思路——因此**对速度不作 SL
```
$ ./coli chat
- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
@@ -216,7 +216,7 @@ MTP head 必须是 **int8**(int4 head 的接受率会崩塌到 0–4%,见
[Releases](https://github.com/JustVugg/colibri/releases) 下载对应平台的压缩包并解压:
```bash
-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri
+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info # engine ready ✓
```
@@ -285,10 +285,12 @@ COLI_MODEL=/nvme/glm52_i4 ./coli doctor # 只读就绪检查
## DeepSeek V4
-实验性的 **DeepSeek V4 Flash** CPU 路径使用原生 FP4 专家、自动 RAM
-规划、共享的 `st.h`/`quant.h` 基础设施,以及常驻 target engine。
-DSpark 有意保留给紧随其后的独立 stacked PR。本 target engine 支持
-x86-64/aarch64 Linux 与 Windows/MSYS2。
+**DeepSeek V4 Flash** 直接流式读取官方 checkpoint,无需转换:路由专家
+保持原生 FP4,稠密部分保持 fp8-e4m3。支持 x86-64/aarch64 Linux 与
+Windows/MSYS2(CPU),并提供可选的 CUDA 层(Windows 运行时 DLL;Linux
+`CUDA=1` 直接链接)——GTX 10 系及更新的 NVIDIA 显卡均可使用(Pascal/Turing
+需 `CUDA_ARCH=portable-pre-ampere NO_TC=1` 构建),每个阶段都保持 CPU 语义一致并可
+逐阶段回退。
```bash
cd c
@@ -308,8 +310,9 @@ oracle 说明,请参阅[中文版 DeepSeek V4 文档](docs/deepseek-v4.zh-CN.m
放置、调度、I/O、CPU/GPU 内核、异构重叠、KV 状态与路由感知推测。目标是降低硬件要求
和每个有效 token 的成本,所有成果都以端到端测量为准、经审查并公开开发。
- **支持更多开放模型。**层级算法与模型无关,任何带路由专家的 MoE 都能用相同方式分层。
- GLM-5.2 与 OLMoE 目前可用;**Kimi K2**、**Qwen3 MoE**、**MiniMax** 等开放权重模型
- 已列入路线图。
+ 目前已有六个模型家族可用(GLM-5.2、Inkling、Kimi K3、DeepSeek V4 Flash、
+ Qwen3.6、OLMoE);更多开放权重家族(候选包括 **MiniMax**)将沿用同样的
+ 规则获得引擎支持:有人完成端到端实测之后。
## 支持项目
diff --git a/README.zh-TW.md b/README.zh-TW.md
index 2cb90082..aff2c255 100644
--- a/README.zh-TW.md
+++ b/README.zh-TW.md
@@ -24,7 +24,7 @@ Colibrì 刻意用於驗證激進的系統構想——因此**對速度不作 SL
```
$ ./coli chat
- 🐦 colibri v1.4.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
+ 🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
@@ -217,7 +217,7 @@ MTP head 必須是 **int8**(int4 head 的接受率會崩落到 0–4%,見
[Releases](https://github.com/JustVugg/colibri/releases) 下載對應平台的壓縮檔並解壓:
```bash
-mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri
+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info # engine ready ✓
```
@@ -290,8 +290,9 @@ COLI_MODEL=/nvme/glm52_i4 ./coli doctor # 唯讀就緒檢查
配置、排程、I/O、CPU/GPU 核心、異質重疊、KV 狀態與路由感知推測。目標是降低硬體要求
和每個有效 token 的成本,所有成果都以端到端測量為準、經審查並公開開發。
- **支援更多開放模型。**階層演算法與模型無關,任何帶路由專家的 MoE 都能用相同方式分層。
- GLM-5.2 與 OLMoE 目前可用;**Kimi K2**、**Qwen3 MoE**、**MiniMax** 等開放權重模型
- 已列入路線圖。
+ 目前已有六個模型家族可用(GLM-5.2、Inkling、Kimi K3、DeepSeek V4 Flash、
+ Qwen3.6、OLMoE);更多開放權重家族(候選包括 **MiniMax**)將沿用同樣的
+ 規則獲得引擎支援:有人完成端到端實測之後。
## 支持專案