Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
release: 1.11.0
JustVuggc6ae5d4788ce
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 675dd189..249343e9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3,6 +3,150 @@All notable changes to colibrì are documented here.Format follows [Keep a Changelog](https://keepachangelog.com/).+## [1.11.0] — 2026-09-13++54 pull requests since v1.10.2. A ninth model family, five real bugs closed+across four engines, and the two platforms the C tests never built on now+building them in CI.++### A ninth engine: DeepSeek V4.1 Flash++- **#1453**: DeepSeek V4.1 Flash (552B, 510 GB on disk) runs on a CPU box+ streaming experts from an SSD, with no conversion: the released checkpoint+ is read natively, fp8 dense with 32x32 ue8m0 tiles and fp4 experts whose+ layout is byte-identical to the mxfp4 the Kimi K3 engine already reads.+ Everything in the architecture that is not V4 is in: Engram (two n-gram+ memories of 384M rows, 203 GB, that never enter RAM), the DSA indexer with+ its two-level candidate source, hyper-connections, the compressor, the+ 32-layer vision tower with its aligner, DSpark speculative decoding (3+ stages, blocks of 5, verified in one batched forward with a rollback for+ the rejected rows) and tool calling in the checkpoint's own DSML format.+- Held token-exact in CI against a torch-only CPU reference+ (`c/tools/dsv41_ref.py`, written because the vendor's own forward needs+ tilelang GPU kernels), at three cache capacities, on a short and a+ 40-token prompt, under all three speculative modes; the vision tower is+ matched to 5e-06. The CI also asserts that the vendor's index-key+ republication policy and the tidy reading disagree, so the shipped default+ cannot be "corrected" by accident (`V41_INDEX_OWNER`).+- Measured on the released checkpoint, cold, caches dropped before every run,+ same prompt and seed: a turn went from 78.7 s to 25.1 s during the work+ (0.305 to 0.957 tok/s) through batched expert reads (`V41_READ_DEPTH`,+ default 8 is the measured knee), attention matrices read once per block of+ positions instead of once per token, and an expert-major MoE. Every step+ is bit-exact against what it replaced. A five-turn chat session runs at+ 1.14 to 1.58 tok/s; an `image_url` part goes through the gateway into the+ vision tower on the same engine.+- Two ways of hiding the expert reads behind the matmuls were built,+ measured worse, and removed; both are written up in `docs/deepseek-v41.md`+ with the numbers, so the next attempt starts from them.+- `coli chat`, `coli serve` and `coli web` work; `coli run` is deliberately+ unwired for this family, as for qwen36 and qwen38. Per-turn accounting is+ behind `V41_STATS`, off by default, so the chat stays clean.++### Fixed++- **#1390** (@crichalchemist): the qwen36 GPU tier's `qt_fill_wait` returned+ when the upload queue was empty, but the uploader frees the ring slot at+ dequeue, before the backend copies anything, so the warmstart freed the RAM+ int8 copy of experts still in flight. An in-flight count that the uploader+ decrements only once the slot is resident (#1360).+- **#1434** (@njloof): `fmt=0` is raw f32 and has no scale array, but the+ Metal sizing helper returned a per-row scale size for it, so+ `coli_metal_matmul` wrapped a NULL pointer and segfaulted inside+ `newBufferWithBytes`, or silently produced all-zero output when the size+ happened to be a page multiple. The kernel no longer applies a scale for+ `fmt=0`, and a fail-closed guard sized off the same helper falls back to+ the CPU path for any format that needs scales and got none.+- **#1389** (@bherald): GLM-5.3 teardown leaked the lazily allocated dashboard+ HITS table.+- **#1461**: `coli cluster worker` could never start without an explicit+ `--cap`: the default is `None`, and `str(None)` reached the engine's+ argument check as the literal `"None"` (#1452). The worker now resolves the+ cap the way every other launcher does.+- **#1460**: olmoe's serve-mode `PROF` line published five literal zeros, so+ `/profile` reported `expert_disk_s = 0.0` on turns where the expert reads+ were the workload. The disk phase is measured now; the other four fields+ stay zero because they are unmeasured, not because a guess would look+ better (#1449, half of it).+- **#1445**: olmoe's `--ram` was inert and "no `--cap`" meant a constant eight+ slots per layer regardless of memory; the cap is derived from the RAM budget+ (#1443). On the reporter's box a 2,291-token prompt went from 675 s to 353 s+ with no flags (#1442).+- **#1377**: available RAM is measured on every platform (glm53 read+ `/proc/meminfo` unconditionally, so on Windows and macOS it read 0 and the+ expert budget collapsed to 1 GB, which is how `coli tune` went OOM in+ #1375), and the Windows commit limit is respected.+- **#1381**: the context ceiling is announced, a request that cannot be+ honoured is refused up front, and the number reported is the right one+ (#1376).+- **#1393**: GLM-5.3 replies always start inside the think block (#1278).+- **#1414** (@dajiaohuang): the GLM-5.3 serve loop honours `CANCEL` while the+ turn is still running (#1332).+- **#1423** (@kreuzzelg): the lazy HITS table is built privately and published+ under a lock in qwen36, kimi_k3, glm53 and qwen38; the parallel warmstart+ segfaulted on first touch about one run in twelve (#1422).+- **#1404** (@Petsku01): the qwen36 tier offers int8 experts on the decode+ path (#1391). **#1388** (@crichalchemist): `QT_MAX_ROWS` replaces seven+ literal 32s that had to agree, and the int8 copy is freed by ownership+ rather than by format.+- **#1421** (@bherald): `coli` no longer overwrites an explicit+ `CUDA_EXPERT_GB` when it decides to place the dense trunk on the card.+- **#1244** (@Unknown-Findout): the CUDA expert tier is charged real VRAM, not+ logical bytes (#687). **#1394**: the VRAM prefix is priced per row, not at+ the container's widest (#1351). **#1410**, **#1411**: on a single GPU the+ VRAM prefix is sized from the VRAM budget and the measured headroom, not+ from the RAM pin plan capped by the LRU reserve (#1409, #1405).+- **#1447** (@trigger2k20): `coli mirror verify` accepts a complete mirror+ copied by other means, with no receipt, and fails closed on a short or+ corrupt shard.+- **#1396** (@dmoraesrs): the web reasoning selector drops "Medium" for+ GLM 5.3, which the engine renders identically to "High".+- **#1428**, **#1435** (@iiEliJas), **#1433** and **#1432** (@texasich),+ **#1440**, **#1459**, **#1458** (@trigger2k20), **#1463**: `setenv` visible+ to `getenv` in-process on Windows, `malloc_trim` guarded to glibc so the+ engines build on musl, a missing bench recipe restored, the per-test+ `_putenv_s` helpers dropped, tests including `compat.h` directly, and+ Clang warnings cleaned up.++### Added++- **#1462** (@trigger2k20): an opt-in Metal path for GLM-5.3-Flash's routed+ experts on Apple Silicon (`COLI_METAL=1`): gate, up, clamped SwiGLU, down+ and the route-weighted scatter on the GPU, CPU path unchanged when Metal+ is off. Validated against the CPU implementation to 2.8e-09 and measured+ on an M4 Max: 0.469 to 0.621 tok/s at 32 tokens, up to 2.15 tok/s with+ the expert cache tuned. Also fixes `coli run --cap N` being silently+ ignored on the glm53 one-shot path.+- **#1454** (@yuripourre): the qwen36 GPU tier compiles with `HIP=1`.+- **#1399** (@SebaWag): `TRUNK_RESIDENT_LAYERS=N` streams the dense trunk+ through mmap (#826).+- **#1361**, **#1374** (@kreuzzelg): the qwen36 dense trunk places itself+ (`COLI_PLACE=auto`, priced in bytes saved per byte of VRAM), fp8 streaming+ mode, VRAM accounting at allocator granularity, thread affinity, `COLI_GPU`.+- **#1383**, **#1385**, **#1386**, **#1387**: Brain and Profile tabs (EMAP,+ HITS, PROF) on every engine. **#1382** (@dmoraesrs): a reasoning depth+ selector in the web UI (#1311).+- **#1407**, **#1408**, **#1406**: CI builds the portable Windows CUDA DLL and+ publishes it as an artifact; the Makefile refuses the 32-bit `cl.exe` before+ nvcc runs (#1405).+- **#1436**: CI builds against musl (Alpine) and compiles the environment+ tests on Windows, the two holes that let #1430 and #1420 ship.+- **#1360** (@kreuzzelg): tier invariants on the fake backend, and the+ reservation leak they found. **#1419**, **#1415**, **#1418**+ (@dajiaohuang): the V4 and fp8 test harnesses build on Windows.+- **#1439** (@bherald): the planner tests no longer materialise four 3 GB+ zero-filled payloads. **#1358** (@monotophic): a raw-evidence adapter for+ the scoring corpus. **#1238** (@ZacharyZcR): reproducible performance+ records are validated.++### Docs++- **#1446**: the qwen36 `--ram` section said the engine does not stream at+ all. **#1392**: Azure GLM-5.2 benchmark rows (#1379, #1380, #1384), the rANS+ cross-reference (#1273), and the int8 probe pinned in CI (#1331).+- **#1427** (@ZH1995): README.zh-CN formatting. **#1438** (@iiEliJas): the+ `malloc_trim` comment.+## [1.10.2] — 2026-09-06Patch release. Three of these fixes answer reports made against 1.10.1 in thediff --git a/README.it.md b/README.it.mdindex c201e6cc..fbfb7f79 100644--- a/README.it.md+++ b/README.it.md@@ -12,7 +12,7 @@ miliardi a 2,8 mila miliardi di parametri** — su hardware consumer ed eterogenin C puro e senza dipendenze del motore, trattando storage, RAM e VRAM comeun'unica gerarchia di inferenza.-Oggi girano otto famiglie: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, con+Oggi girano nove famiglie: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, convision), **Inkling** (975B), **Kimi K3** (2,8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, con visione),**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) e**OLMoE** (7B) — un@@ -34,7 +34,7 @@ ma non ridefinire il modello di nascosto.```$ ./coli chat- 🐦 colibri v1.10.2 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -338,8 +338,8 @@ e per il gateway API opzionale.end-to-end, revisionati e sviluppati apertamente.- **Più modelli aperti.** L'algoritmo di tiering è indipendente dal modello:qualsiasi MoE con expert instradati può essere organizzato allo stesso modo.- Otto famiglie funzionano già (GLM-5.2, GLM-5.3-Flash con la vision,- Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6, OLMoE);+ Nove famiglie funzionano già (GLM-5.2, GLM-5.3-Flash con la vision,+ Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next, Qwen3.6, OLMoE);altre famiglie open-weight, **MiniMax** tra le candidate, si guadagnano unengine come le prime otto: quando qualcuno le misura end-to-end.diff --git a/README.md b/README.mdindex 4028cecd..7cb05c9c 100644--- a/README.md+++ b/README.md@@ -18,7 +18,7 @@ parameters** — on consumer and heterogeneous hardware, in pure C with zeroengine dependencies, by treating storage, RAM, and VRAM as a single inferencehierarchy (AI memory multitiering).-Eight families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, with+Nine families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, withvision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, with vision),**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) and**OLMoE** (7B) —@@ -40,7 +40,7 @@ may reduce speed; it must not quietly redefine the model.```$ ./coli chat- 🐦 colibri v1.10.2 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -584,8 +584,8 @@ checkpoint validation, and the generated tiny independent oracle.lower cost per useful token. Everything lands the way this project works:measured end to end, reviewed, and developed in the open.- **More open models.** The tiering algorithm is model-agnostic: any MoE with- routed experts can be staged the same way. Eight families run today (GLM-5.2,- GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next,+ routed experts can be staged the same way. Nine families run today (GLM-5.2,+ GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next,Qwen3.6, OLMoE); further open-weight families — **MiniMax** among thecandidates — earn an engine the way the first eight did: when someonemeasures one end to end.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex c15773bd..ef32c684 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -10,7 +10,7 @@**小巧引擎,庞大模型。**在消费级与异构硬件上运行**前沿 MoE 模型——从 744B 到2.8T 参数**——以引擎零依赖的纯 C 实现,将存储、RAM 与 VRAM 视为统一的推理层级。-目前可运行八个模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含视觉)、+目前可运行九个模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含视觉)、**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B,含视觉)、**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)与 **OLMoE**(7B)——各自一个 C 文件,共用同一套 `coli chat` / `coli serve` / `coli web` 前端。[完整列表](README.md#other-supported-models)@@ -25,7 +25,7 @@ Colibrì 刻意用于验证激进的系统思路——因此**对速度不作 SL```$ ./coli chat- 🐦 colibri v1.10.2 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -311,7 +311,7 @@ oracle 说明,请参阅[中文版 DeepSeek V4 文档](docs/deepseek-v4.zh-CN.m放置、调度、I/O、CPU/GPU 内核、异构重叠、KV 状态与路由感知推测。目标是降低硬件要求和每个有效 token 的成本,所有成果都以端到端测量为准、经审查并公开开发。- **支持更多开放模型**。层级算法与模型无关,任何带路由专家的 MoE 都能用相同方式分层。- 目前已有八个模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、+ 目前已有九个模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、Qwen3.8-Flash-Next、Qwen3.6、OLMoE);更多开放权重家族(候选包括 **MiniMax**)将沿用同样的规则获得引擎支持:有人完成端到端实测之后。diff --git a/README.zh-TW.md b/README.zh-TW.mdindex f397e739..8211a0b1 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -10,7 +10,7 @@**小巧引擎,龐大模型。**在消費級與異質硬體上執行**前沿 MoE 模型——從 744B 到2.8T 參數**——以引擎零相依套件的純 C 實作,將儲存、RAM 與 VRAM 視為統一的推論階層。-目前可執行八個模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含視覺)、+目前可執行九個模型家族:**GLM-5.2/5.3**(744B)、**GLM-5.3-Flash**(321B,含視覺)、**Inkling**(975B)、**Kimi K3**(2.8T)、**DeepSeek V4 Flash**(284B)、**DeepSeek V4.1 Flash**(552B,含視覺)、**Qwen3.8-Flash-Next**(125B + 51B n-gram)、**Qwen3.6**(35B-A3B)與 **OLMoE**(7B)——各自一個 C 檔案,共用同一套 `coli chat` / `coli serve` / `coli web` 前端。[完整清單](README.md#other-supported-models)@@ -25,7 +25,7 @@ Colibrì 刻意用於驗證激進的系統構想——因此**對速度不作 SL```$ ./coli chat- 🐦 colibri v1.10.2 — GLM-5.2 · 744B MoE · int4 · streaming CPU+ 🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! 😊 Come posso aiutarti oggi?@@ -291,7 +291,7 @@ COLI_MODEL=/nvme/glm52_i4 ./coli doctor # 唯讀就緒檢查配置、排程、I/O、CPU/GPU 核心、異質重疊、KV 狀態與路由感知推測。目標是降低硬體要求和每個有效 token 的成本,所有成果都以端到端測量為準、經審查並公開開發。- **支援更多開放模型。**階層演算法與模型無關,任何帶路由專家的 MoE 都能用相同方式分層。- 目前已有八個模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、+ 目前已有九個模型家族可用(GLM-5.2、GLM-5.3-Flash、Inkling、Kimi K3、DeepSeek V4 Flash、DeepSeek V4.1 Flash、Qwen3.8-Flash-Next、Qwen3.6、OLMoE);更多開放權重家族(候選包括 **MiniMax**)將沿用同樣的規則獲得引擎支援:有人完成端到端實測之後。diff --git a/c/version.py b/c/version.pyindex 06573da8..fe23c815 100644--- a/c/version.py+++ b/c/version.py@@ -1,3 +1,3 @@"""Single source of truth for the colibri version number."""-__version__ = "1.10.2"+__version__ = "1.11.0"diff --git a/site/index.html b/site/index.htmlindex 10f85750..e8e5b55a 100644--- a/site/index.html+++ b/site/index.html@@ -4,9 +4,9 @@<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>colibrì — run models bigger than your machine</title>-<meta name="description" content="An inference engine in pure C that streams Mixture-of-Experts weights from disk. A 2.8-trillion-parameter model runs on a 32 GB box. Six model families, zero dependencies, CPU or GPU.">+<meta name="description" content="An inference engine in pure C that streams Mixture-of-Experts weights from disk. A 2.8-trillion-parameter model runs on a 32 GB box. Nine model families, zero dependencies, CPU or GPU."><meta property="og:title" content="colibrì — run models bigger than your machine">-<meta property="og:description" content="Six frontier MoE families, up to 2.8T parameters, on consumer hardware: pure C, zero dependencies, experts tiered across VRAM, RAM and disk. Run it, study it, improve it.">+<meta property="og:description" content="Nine frontier MoE families, up to 2.8T parameters, on consumer hardware: pure C, zero dependencies, experts tiered across VRAM, RAM and disk. Run it, study it, improve it."><meta property="og:type" content="website"><meta property="og:image" content="https://raw.githubusercontent.com/JustVugg/colibri/main/docs/media/colibri-atlas.png"><link rel="icon" type="image/svg+xml" href="colibri-icon.svg">@@ -229,7 +229,7 @@ footer li a:hover{color:var(--on-dark)}<div><div class="n">510</div><div class="l">pull requests merged</div></div><div><div class="n">6</div><div class="l">families · 4 backends · 3 platforms</div></div><div class="since">All of it in <b data-gh-age>six weeks</b>, in the open: issues filed from strangers' machines,- reproduced and fixed the same day. Currently shipping <b>v1.10.2</b>.</div>+ reproduced and fixed the same day. Currently shipping <b>v1.11.0</b>.</div></div></section>