Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

release 2.0.0

JustVugg03099c6e31e8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2b8ce23b..b86722ea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,272 @@
All notable changes to colibrì are documented here.
Format follows [Keep a Changelog](https://keepachangelog.com/).
+## [2.0.0] - 2026-10-06
+
+166 pull requests since v1.12.1, 87 of them from contributors. Every MoE
+engine now runs on any GPU a Vulkan driver can see: the routed experts on a
+shared device tier, the dense layers on a device chain (all of them, or the
+first N that fit), and a second GPU for more experts. Three model families
+arrive (MiMo-V2.6, Qwen3-Coder, Qwen-Image-2.1) and one dense checkpoint
+(Qwen3.8-27B). Qwen3.8 Flash Next gets int4 experts and speculative decoding
+that is on by default. Every text engine serves up to 16 conversations at once.
+The release archives carry the GPU backends. Setup is one step, and there is one
+decision API (System One).
+
+### Several conversations at once
+
+- **#1955**: every text engine serves several conversations at once (`coli serve
+ --kv-slots N`, up to 16), as GLM-5.2 and GLM-5.3 Flash did: qwen36 (Qwen3.6,
+ Qwen3-Coder, Qwen3.8-27B), qwen38, OLMoE, Inkling, Kimi K3, MiMo-V2.6,
+ DeepSeek V4 and V4.1. Each slot keeps its own conversation's state, and every
+ decode step takes the next token of each active request as one batch, so the
+ weights and experts a step reads serve all of them. A request gets the tokens
+ it gets alone, checked frame for frame on every engine, on the CPU and with
+ the expert tier. With more than one slot nothing drafts and the dense chain
+ stays off.
+
+### Vulkan on every engine
+
+- **#1830**: every engine's resident matrices run on any Vulkan GPU (fmt 10, 11
+ and 12 added), each configuration checked against the engine's own CPU run on
+ Lavapipe (`tests/vulkan_engines.sh`).
+- **#1834**: tiled GEMMs for prefill in every weight format, and a
+ cooperative-matrix GEMM where the device has one.
+- **#1840**, **#1843**: a shared routed-expert tier (`vk_tier.c`). It holds a
+ cache of experts in device memory, filled from the expert history and adapting
+ while you chat, and computes the resident experts of each step as one async
+ batch while the CPU computes the rest. First on qwen38 and qwen36, then on
+ every MoE engine: GLM-5.2, GLM-5.3 Flash, Inkling, OLMoE, Kimi K3, MiMo-V2.6,
+ DeepSeek V4 and V4.1.
+- **#1844**, **#1848**, **#1849**, **#1851**, **#1857**: the dense chain
+ (`vk_chain.c`), a layer's dense part in one submission, on qwen36, qwen38,
+ MiMo-V2.6 (sliding window and sinks on the device), OLMoE, Inkling (and
+ OLMoE's PILOT race fixed), GLM-5.2 and GLM-5.3 Flash (shared MLA, DSA, KDA and
+ mHC ops), Kimi K3, DeepSeek V4.1 and DeepSeek V4.
+- **#1871**: staged device-local uploads for cards without Resizable BAR, where
+ the mapped path failed or spilled to system RAM.
+- **#1901**: expert streaming for big prompt steps, memory budgets and
+ speculative verification on the device.
+- **#1909**, **#1913**, **#1916**, **#1918**: a partial dense chain. The first N
+ layers that fit go to the device, the CPU runs the rest, and nothing is
+ uploaded that the chain will not use, so an 8 GB card with a larger trunk keeps
+ the chain and its expert tier instead of losing both. `COLI_VK_CHAIN_LAYERS`
+ forces N, `coli plan` predicts it. DeepSeek V4, qwen36 and olmoe, qwen38,
+ GLM-5.2 and GLM-5.3 Flash.
+- **#1920**: a second GPU for the expert tier (`COLI_VK_DEV2=auto|<index>`) on
+ every MoE engine: the experts after the primary device's, each step one batch
+ per device, both in flight at once. A failing second device gives its experts
+ back to the CPU and the tier goes on with the first.
+- **#1942**: the dense chain's layers on two GPUs. With `COLI_VK_DEV2`, the
+ layers a partial chain leaves go to the second device instead of the CPU, on
+ every chain engine (qwen36, qwen38, OLMoE, MiMo-V2.6, Inkling, GLM-5.2,
+ GLM-5.3 Flash, Kimi K3, DeepSeek V4.1 and V4). `COLI_VK_CHAIN_LAYERS2` forces
+ how many, `COLI_VK_CHAIN_DEV2=0` keeps them off. A lost second device leaves
+ the CPU to run its layers, the recurrent state rebuilt where it was held.
+ Checked on Lavapipe opened twice; two real GPUs not measured yet.
+- **#1933**: `COLI_VK_KV_COLD=device` (opt-in): with the KV cache split past the
+ device's budget, the host's part of the attention runs on the device too, from
+ host memory it reads in place, so a step runs in one frame. The prompt gains
+ 4-6% on a Radeon 780M, decode is slower there (hence off by default); a
+ dedicated GPU has not been measured.
+- **#1948**: several conversations at once on the dense chain. colibri's
+ multiplexed decode (`KV_SLOTS`, one row from each active conversation) runs on
+ the device: the batch's matrices as one, each row's attention over its own
+ conversation's KV mirror (up to `COLI_VK_CHAIN_MUX`, default 16, beside the
+ chain's own) with its own DSA list; two devices and the partial chain too.
+ Every frame checked against the CPU's on Lavapipe; not measured with a real
+ checkpoint yet.
+- **#1934**, **#1935**, **#1936**, **#1937**, **#1938**, **#1939** (@huppiflupp):
+ the matrix units for prompt attention and int8/int4 prompt GEMMs on the
+ chain, a step's experts as one grouped GEMM (decode: one grouped GEMV per
+ phase), qwen36's routing in parallel, and int8 decode GEMVs sized by the
+ matrix. On a Radeon 780M with Qwen3.6-35B-A3B the first token of a
+ 1000-token prompt comes in 9.3 s instead of 12.6, decode 6.1 tok/s instead
+ of 5.7, perplexity unchanged.
+- **#1954**: on an Intel GPU the dense chain decoded garbage (Qwen3.6 on an Iris
+ Xe, the default there). Its pipelines now run at the subgroup size the device
+ reports; the device compiles at 8 to 32 lanes, and the chain's GEMVs had run
+ at another width than the one they read. On Windows the chain also finds its
+ optional shaders (MLA, KDA, mHC, DeepSeek V4, the split KV cache), and the
+ split cache frees its host memory with the matching call.
+- **#1911**: `COLI_VK_DEV=<index>` picks the device on a machine with two GPUs of
+ the same kind; the cooperative GEMM's epilogue stride is aligned to 16 bytes, a
+ validation error on some drivers (#1908).
+- **#1846** (@crichalchemist): correct expert-batch results under MoltenVK's
+ default mode.
+- **#1762**, **#1763** (@Kenneth-Javier): the Vulkan loader linked on Windows
+ under `-static`, and the Vulkan objects linked into every engine-including
+ test rule.
+
+### Models
+
+- **#1813**, **#1819**, **#1820**, **#1827**, **#1835**: MiMo-V2.6 (Flash and
+ Pro): the engine, the vision tower, tool calling and Brio mode with logprobs,
+ read as released; Pro verified against Xiaomi's reference on the real
+ checkpoint, and `coli serve` names the loaded variant.
+- **#1828**, **#1829**: Qwen3-Coder-30B-A3B (`qwen3_moe`) on the qwen36 engine.
+- **#1777**, **#1779**: dense checkpoints of the Qwen3.6 family on the qwen36
+ engine (Qwen3.8-27B), text and images.
+- **#1780**: Qwen-Image-2.1, text to image: a C engine, pictures in the
+ terminal, the web app and the OpenAI images API.
+- **#1951**: Qwen-Image on the GPU and a faster start. The weights load in
+ 7.5 s instead of 34 (four layers at once, quantized straight from bf16, the
+ same int8 bytes). With Vulkan every block of a denoising step runs on the
+ device, its attention included, and the VAE decodes there too; a card that
+ does not hold the 7.1 GB transformer keeps the blocks that fit and streams
+ the others each step. On a Radeon 780M a 512x512 image takes 61 s instead of
+ 112 and a 1024x1024 step 38 s instead of 67; every stage checked against the
+ diffusers reference.
+- **#1831**: Qwen3.8 Flash Next's routed experts as int4-g64: 1.4 to 1.56x
+ decode.
+- **#1832**, **#1917**: speculative decoding with the checkpoint's MTP head,
+ lossless, +12-14% on the release; with #1917 it is on by default with prompt
+ lookup on every engine that has it (`Q38_MTP=0`, `COLI_LOOKUP=0` turn them
+ off), and `coli plan` counts the MTP head's memory.
+- **#1814**: Inkling reads the embedding norm under the name transformers 5.18
+ writes.
+
+### System One: one decision API
+
+- **#1856**: `POST /v1/systemone`, one decision API, and the Laya decision
+ engine; `/v1/brio` is removed.
+- **#1868**: GLiNER2.5-Decide as a native decision engine (DeBERTa-v3 and
+ GLiNER2's head).
+- **#1872**: Clef as a native decision model on qwen36, a joint head over
+ Qwen3.8-27B, with a 16-bit dense mode.
+- **#1743**: the brio serve gates run in CI against the converted fixture.
+- **#1753** (@tarazum): brio normalizes to sum by default and warns when a mean
+ mixes option token counts.
+
+### Setup and the CLI
+
+- **#1953**: the release archives carry the GPU backends: every Linux and
+ Windows engine built with Vulkan and its shaders, the macOS engines with
+ Metal, and a Windows CUDA package (`coli_cuda.dll` for compute 8.0 and newer,
+ with colibri, qwen36 and kimi_k3 built to load it). The Vulkan loader is
+ opened at run time, so these engines still start on a machine without one,
+ on the CPU. `coli setup` uses them without a Vulkan toolchain.
+
+- **#1842**: one-step install and start (`start-here`, `coli setup`), the
+ `AI_SETUP.md` guide and an MCP server.
+- **#1850**, **#1896**: setup picks the backend that works on the machine: on an
+ integrated GPU, Vulkan only where it was measured faster; CUDA only when the
+ toolkit builds for the card, Vulkan otherwise, and a failed build falls back.
+- **#1903**: an engine already built is brought up to date with `make` before it
+ is used (a `git pull` used to keep running the old binary), and `coli logs`
+ says where a foreground server prints (#1852).
+- **#1919**: a fresh install of Qwen3.6-35B-A3B or Qwen3.8 Flash Next gets a
+ starting expert history, so its first run fills the GPU tier from the start:
+ on a Radeon 780M Qwen3.6's first run went from about 6 to 11 tok/s.
+- **#1817**: in `coli chat`, a line that starts with a picture's path is a
+ message, not a command.
+- **#1315** (@Avicennasis): resumable, hash-verified qpack installers for
+ Hugging Face and static mirrors.
+- **#1316** (@Avicennasis): the planner honours cgroup memory limits.
+- **#1949**: on Windows every engine has its bare `make <engine>` target (six
+ fell to make's built-in rule and linked without CUDA or Vulkan, #1945,
+ #1900); `coli setup` prefers an MSYS2 that can build Vulkan over a PATH
+ compiler that cannot, and installs libgomp; `COLI_VK_DENSE=0` keeps the dense
+ chain off too (on an 8 GB laptop it had taken the expert tier's budget,
+ #1900); DeepSeek V4 uses the physical cores on an SMT machine (+10% decode on
+ a Threadripper, #1906); `coli plan` leaves a card below an engine's CUDA
+ floor out; the server says why its engine stopped (#1941); clearer messages
+ for a missing model and a CUDA DLL that does not load.
+
+### The serve contract
+
+- **#1353** (@monotophic): `logprobs`, `top_logprobs` and `echo` on the OpenAI
+ endpoints, aligned by raw-stream span.
+- **#1794**, **#1767**: Qwen3.6 tool calling, and the template's
+ `preserve_thinking` rendered so standard clients get KV prefix reuse.
+- **#1802** (@bherald): opt-in strict GLM tool-call parsing.
+- **#1765** (@SulimanAbdulrazzaq): GLM-5.3's reasoning stays out of the
+ `/v1/messages` answer.
+- **#1791**, **#1793** (@Avicennasis): Anthropic image blocks are served and a
+ tool result's picture is not dropped; the modalities a model card claims are
+ the ones the engine loaded.
+- **#1787**, **#1804** (@benmaster82): unsupported OpenAI result shaping is
+ refused; the completions keepalive streams as a text chunk.
+- **#1782**, **#1783**, **#1785**, **#1786** (@kevin9327): malformed tool calls,
+ text parts, tools, messages and image URLs answer 400 instead of 500.
+- **#1809** (@Stamina9): non-string brio message text is refused.
+- **#1749**, **#1751**, **#1789**, **#1803** (@enitimeago): glm53 reports REUSE
+ in detail and logs CANCEL, does not restore a pin over rows another branch
+ rewrote, handles cancellation during prompt processing, and continues from the
+ cache instead of prefilling again.
+
+### Correctness and security
+
+- **#1930**: hardening of three input paths. The files the engines keep beside a
+ model (`.coli_kv`, `.coli_usage.tmp`, `hot_pinned.bin`, `.coli_ckpt/`) are
+ opened without following a symlink and only as regular files, so a link
+ planted in a downloaded model directory no longer redirects the engine's
+ writes. Request grammars are limited to 32767 alternates and symbols per rule
+ (past that the walker's index wrapped) and are refused with a message.
+ colibri, kimi_k3 and deepseek_v41 check the indexer's and RoPE's head
+ dimensions in the config.
+- **#1739**, **#1740**, **#1754**, **#1755**, **#1854**, **#1855**
+ (@namespaceMarcello): the published advisory fixes carried to glm53, qwen36 and
+ deepseek_v41; glm53 refuses f32 tensors shorter than the config reads and
+ matrices whose shape differs from it; deepseek_v41 keeps engram lookups inside
+ their table; qwen36 gives the same logits at every expert-cache capacity on FMA
+ builds and never evicts an expert a MoE run is still reading.
+- **#1775**: the DeepSeek V4/V4.1 pre-tokenizer in `tok.h`, so Turkish
+ tokenizes as HF does.
+- **#1799** (@RayanR000): the DeepSeek V4.1 reference sends top-k ties to the
+ lowest index and its indexer never keeps -inf.
+- **#1812** (@ZacharyZcR): the shared-expert bindings of DeepSeek V4's fused MoE.
+- **#1611** (@Cometbuster4969): OLMoE uses the shared RAM probe, with a
+ plausibility floor.
+- **#1744**, **#1745** (@kevin9327): a numeric `node_id` no longer kills the
+ cluster's topology and health requests, and a manifest that is not a JSON
+ object no longer crashes the experiments validator.
+- **#1356** (@monotophic): the ablation scoring mode replaced by a checked,
+ digest-bound one.
+- **#1904**: DeepSeek V4's `PROF` measures the expert wait instead of printing 0
+ and folding it into `attention_s` (#1852).
+- **#1902**: Windows expert prefetch is asynchronous and its queued work bounded.
+
+### Speed on the CPU
+
+- **#1837**: every engine's per-token projections run per block of rows in
+ prefill.
+- **#1761** (@mfethe1): a NEON arm for batched `matmul_fp8`, bit-exact, 6 to 10x
+ at prefill shapes.
+- **#1790**, **#1798** (@RayanR000): the f32 matmul runs eight `fmaf` chains at
+ once with the same bits on every FMA build, and one vectorized f32 matmul
+ serves quant.h, olmoe, inkling and qwen36 (#442).
+- **#1773** (@JackKnifeAI): glm53's dense prefill batched, bit-identical per
+ token.
+- **#1784** (@kreuzzelg): qwen36's DeltaNet layer on the card
+ (`Q36_DN_GPU=1`), expert homes per layer range (`QT_HOME=layer`), the shared
+ expert offered on request.
+- **#1323**, **#1343** (@Avicennasis): qwen36 reads routed experts from a qpack
+ container through bounded Metal slots, and MLX-affine dense weights and norms.
+- **#1261** (@Kenneth-Javier): an optional explicit AMD XDNA2 lane for the GLM
+ shared expert.
+
+### Build, tests and CI
+
+- **#1758** (@Kenneth-Javier): header prerequisites generated with `-MMD -MP`
+ (#1741).
+- **#1845**, **#1912**: the sanitizer runs and the bf16 gates hold on AVX-512
+ runners, whose CPUs keep bf16 on the CPU.
+- **#1914**: the Windows UCRT64 job's limit from 35 to 60 minutes.
+- **#1795**, **#1821**, **#1836**, **#1839**: four flaky tests made
+ deterministic.
+
+### Docs
+
+- **#1897**, **#1833**: the README rewritten for the release (one-step setup,
+ models per machine, GPUs, System One) in five languages, with the site.
+- **#1768**, **#1776**: every engine in the environment reference, DeepSeek V4.1
+ in its own section, and the weights-from-disk options.
+- **#1788** (@jamshaid120): duplicate entries removed from the DeepSeek V4 docs.
+- **#1801** (@benmaster82): the web app's image alt count and German image
+ labels.
+- **#1738**: the 1.12.1 entry's pull request count corrected.
+
## [1.12.1] — 2026-09-24
96 pull requests since v1.12.0, 80 of them from contributors. Two tokenizers
diff --git a/README.id.md b/README.id.md
index 57ecfeb5..856f6b67 100644
--- a/README.id.md
+++ b/README.id.md
@@ -31,7 +31,7 @@ engine Qwen3.6. [Mana yang cocok untuk mesin saya](#which-model-for-my-machine)
```
$ ./coli chat
- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU
+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?
diff --git a/README.it.md b/README.it.md
index 4934f5d6..2d8cbe6e 100644
--- a/README.it.md
+++ b/README.it.md
@@ -30,7 +30,7 @@ motore Qwen3.6. [Quale scegliere per la mia macchina](#which-model-for-my-machin
```
$ ./coli chat
- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU
+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?
diff --git a/README.ja.md b/README.ja.md
index 5a258df5..a07a3ffe 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -19,7 +19,7 @@
```
$ ./coli chat
- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU
+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?
diff --git a/README.md b/README.md
index ef40578a..b1bb1cf7 100644
--- a/README.md
+++ b/README.md
@@ -29,7 +29,7 @@ Qwen3.6 engine. [Which one for my machine](#which-model-for-my-machine)
```
$ ./coli chat
- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU
+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?
diff --git a/README.zh-CN.md b/README.zh-CN.md
index e381a4b5..ed373044 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -19,7 +19,7 @@
```
$ ./coli chat
- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU
+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?
diff --git a/README.zh-TW.md b/README.zh-TW.md
index 0ce37220..c818e15a 100644
--- a/README.zh-TW.md
+++ b/README.zh-TW.md
@@ -28,7 +28,7 @@ Qwen3.6 引擎上。[我的機器適合哪一個](#which-model-for-my-machine)
```
$ ./coli chat
- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU
+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?
diff --git a/c/setup_catalog.py b/c/setup_catalog.py
index 34c44d3a..7cfd8fe9 100644
--- a/c/setup_catalog.py
+++ b/c/setup_catalog.py
@@ -76,7 +76,7 @@ CATALOG = (
# docs/qwen36.md: 6.5 GB resident at 32 experts/layer, 15.2 GB with all 128.
ram_min_gb=8, ram_good_gb=18, dense_gb=3.6, rank=2, size_class="small",
summary="coding model with tool calls, no thinking; int4-gs64 container",
- prebuilt_since="1.12.2", doc="docs/qwen36.md#qwen3-coder-30b-a3b"),
+ prebuilt_since="2.0.0", doc="docs/qwen36.md#qwen3-coder-30b-a3b"),
CatalogModel(
id="deepseek-v4-flash-reap", family="deepseek_v4", name="DeepSeek V4 Flash REAP 150B",
repo="puwaer/DeepSeek-V4-Flash-0731-reap-150b", disk_gb=84.7,
@@ -104,7 +104,7 @@ CATALOG = (
# The engine never loads these three (docs/mimo.md, measured on Pro,
# the same engine and layout).
exclude=("dflash/*", "audio_tokenizer/*", "model_mtp.safetensors"),
- prebuilt_since="1.12.2", doc="docs/mimo.md"),
+ prebuilt_since="2.0.0", doc="docs/mimo.md"),
CatalogModel(
id="glm-5.2", family="glm", name="GLM-5.2",
repo="mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp", disk_gb=429.3,
@@ -123,7 +123,7 @@ CATALOG = (
ram_min_gb=54, ram_good_gb=64, dense_gb=32.4, rank=9, size_class="large",
summary="1.02T with vision and tools, official checkpoint",
exclude=("dflash/*", "audio_tokenizer/*", "model_mtp.safetensors"),
- prebuilt_since="1.12.2", doc="docs/mimo.md#pro"),
+ prebuilt_since="2.0.0", doc="docs/mimo.md#pro"),
CatalogModel(
id="inkling", family="inkling", name="Inkling",
repo="nbeerbower/Inkling-colibri-int4", disk_gb=514.1,
@@ -148,7 +148,7 @@ CATALOG = (
ram_min_gb=12, ram_good_gb=18, dense_gb=8.5, rank=1, size_class="small",
modality="image", summary="text to image (pictures, not chat)",
license_note="Qwen Research License: non-commercial use only",
- prebuilt_since="1.12.2", doc="docs/qwen-image.md"),
+ prebuilt_since="2.0.0", doc="docs/qwen-image.md"),
)
diff --git a/c/version.py b/c/version.py
index db24fa2d..d42baa29 100644
--- a/c/version.py
+++ b/c/version.py
@@ -1,3 +1,3 @@
"""Single source of truth for the colibri version number."""
-__version__ = "1.12.1"
+__version__ = "2.0.0"
diff --git a/site/index.html b/site/index.html
index 08001013..bd89e979 100644
--- a/site/index.html
+++ b/site/index.html
@@ -630,7 +630,7 @@ cd colibri
<footer><div class="fin">
<span>&copy; 2026 colibri</span>
- <span>Currently shipping <b>v1.12.1</b></span>
+ <span>Currently shipping <b>v2.0.0</b></span>
<span class="sp">
<a href="https://github.com/JustVugg/colibri/">GitHub</a><a href="https://github.com/JustVugg/colibri/blob/main/docs/quickstart.md">Docs</a>
<a href="https://github.com/JustVugg/colibri/blob/main/docs/systemone.md">System One</a><a href="https://github.com/JustVugg/colibri/blob/main/docs/api.md">API</a>