Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
release 2.0.0
JustVugg03099c6e31e8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2b8ce23b..b86722ea 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3,6 +3,272 @@All notable changes to colibrì are documented here.Format follows [Keep a Changelog](https://keepachangelog.com/).+## [2.0.0] - 2026-10-06++166 pull requests since v1.12.1, 87 of them from contributors. Every MoE+engine now runs on any GPU a Vulkan driver can see: the routed experts on a+shared device tier, the dense layers on a device chain (all of them, or the+first N that fit), and a second GPU for more experts. Three model families+arrive (MiMo-V2.6, Qwen3-Coder, Qwen-Image-2.1) and one dense checkpoint+(Qwen3.8-27B). Qwen3.8 Flash Next gets int4 experts and speculative decoding+that is on by default. Every text engine serves up to 16 conversations at once.+The release archives carry the GPU backends. Setup is one step, and there is one+decision API (System One).++### Several conversations at once++- **#1955**: every text engine serves several conversations at once (`coli serve+ --kv-slots N`, up to 16), as GLM-5.2 and GLM-5.3 Flash did: qwen36 (Qwen3.6,+ Qwen3-Coder, Qwen3.8-27B), qwen38, OLMoE, Inkling, Kimi K3, MiMo-V2.6,+ DeepSeek V4 and V4.1. Each slot keeps its own conversation's state, and every+ decode step takes the next token of each active request as one batch, so the+ weights and experts a step reads serve all of them. A request gets the tokens+ it gets alone, checked frame for frame on every engine, on the CPU and with+ the expert tier. With more than one slot nothing drafts and the dense chain+ stays off.++### Vulkan on every engine++- **#1830**: every engine's resident matrices run on any Vulkan GPU (fmt 10, 11+ and 12 added), each configuration checked against the engine's own CPU run on+ Lavapipe (`tests/vulkan_engines.sh`).+- **#1834**: tiled GEMMs for prefill in every weight format, and a+ cooperative-matrix GEMM where the device has one.+- **#1840**, **#1843**: a shared routed-expert tier (`vk_tier.c`). It holds a+ cache of experts in device memory, filled from the expert history and adapting+ while you chat, and computes the resident experts of each step as one async+ batch while the CPU computes the rest. First on qwen38 and qwen36, then on+ every MoE engine: GLM-5.2, GLM-5.3 Flash, Inkling, OLMoE, Kimi K3, MiMo-V2.6,+ DeepSeek V4 and V4.1.+- **#1844**, **#1848**, **#1849**, **#1851**, **#1857**: the dense chain+ (`vk_chain.c`), a layer's dense part in one submission, on qwen36, qwen38,+ MiMo-V2.6 (sliding window and sinks on the device), OLMoE, Inkling (and+ OLMoE's PILOT race fixed), GLM-5.2 and GLM-5.3 Flash (shared MLA, DSA, KDA and+ mHC ops), Kimi K3, DeepSeek V4.1 and DeepSeek V4.+- **#1871**: staged device-local uploads for cards without Resizable BAR, where+ the mapped path failed or spilled to system RAM.+- **#1901**: expert streaming for big prompt steps, memory budgets and+ speculative verification on the device.+- **#1909**, **#1913**, **#1916**, **#1918**: a partial dense chain. The first N+ layers that fit go to the device, the CPU runs the rest, and nothing is+ uploaded that the chain will not use, so an 8 GB card with a larger trunk keeps+ the chain and its expert tier instead of losing both. `COLI_VK_CHAIN_LAYERS`+ forces N, `coli plan` predicts it. DeepSeek V4, qwen36 and olmoe, qwen38,+ GLM-5.2 and GLM-5.3 Flash.+- **#1920**: a second GPU for the expert tier (`COLI_VK_DEV2=auto|<index>`) on+ every MoE engine: the experts after the primary device's, each step one batch+ per device, both in flight at once. A failing second device gives its experts+ back to the CPU and the tier goes on with the first.+- **#1942**: the dense chain's layers on two GPUs. With `COLI_VK_DEV2`, the+ layers a partial chain leaves go to the second device instead of the CPU, on+ every chain engine (qwen36, qwen38, OLMoE, MiMo-V2.6, Inkling, GLM-5.2,+ GLM-5.3 Flash, Kimi K3, DeepSeek V4.1 and V4). `COLI_VK_CHAIN_LAYERS2` forces+ how many, `COLI_VK_CHAIN_DEV2=0` keeps them off. A lost second device leaves+ the CPU to run its layers, the recurrent state rebuilt where it was held.+ Checked on Lavapipe opened twice; two real GPUs not measured yet.+- **#1933**: `COLI_VK_KV_COLD=device` (opt-in): with the KV cache split past the+ device's budget, the host's part of the attention runs on the device too, from+ host memory it reads in place, so a step runs in one frame. The prompt gains+ 4-6% on a Radeon 780M, decode is slower there (hence off by default); a+ dedicated GPU has not been measured.+- **#1948**: several conversations at once on the dense chain. colibri's+ multiplexed decode (`KV_SLOTS`, one row from each active conversation) runs on+ the device: the batch's matrices as one, each row's attention over its own+ conversation's KV mirror (up to `COLI_VK_CHAIN_MUX`, default 16, beside the+ chain's own) with its own DSA list; two devices and the partial chain too.+ Every frame checked against the CPU's on Lavapipe; not measured with a real+ checkpoint yet.+- **#1934**, **#1935**, **#1936**, **#1937**, **#1938**, **#1939** (@huppiflupp):+ the matrix units for prompt attention and int8/int4 prompt GEMMs on the+ chain, a step's experts as one grouped GEMM (decode: one grouped GEMV per+ phase), qwen36's routing in parallel, and int8 decode GEMVs sized by the+ matrix. On a Radeon 780M with Qwen3.6-35B-A3B the first token of a+ 1000-token prompt comes in 9.3 s instead of 12.6, decode 6.1 tok/s instead+ of 5.7, perplexity unchanged.+- **#1954**: on an Intel GPU the dense chain decoded garbage (Qwen3.6 on an Iris+ Xe, the default there). Its pipelines now run at the subgroup size the device+ reports; the device compiles at 8 to 32 lanes, and the chain's GEMVs had run+ at another width than the one they read. On Windows the chain also finds its+ optional shaders (MLA, KDA, mHC, DeepSeek V4, the split KV cache), and the+ split cache frees its host memory with the matching call.+- **#1911**: `COLI_VK_DEV=<index>` picks the device on a machine with two GPUs of+ the same kind; the cooperative GEMM's epilogue stride is aligned to 16 bytes, a+ validation error on some drivers (#1908).+- **#1846** (@crichalchemist): correct expert-batch results under MoltenVK's+ default mode.+- **#1762**, **#1763** (@Kenneth-Javier): the Vulkan loader linked on Windows+ under `-static`, and the Vulkan objects linked into every engine-including+ test rule.++### Models++- **#1813**, **#1819**, **#1820**, **#1827**, **#1835**: MiMo-V2.6 (Flash and+ Pro): the engine, the vision tower, tool calling and Brio mode with logprobs,+ read as released; Pro verified against Xiaomi's reference on the real+ checkpoint, and `coli serve` names the loaded variant.+- **#1828**, **#1829**: Qwen3-Coder-30B-A3B (`qwen3_moe`) on the qwen36 engine.+- **#1777**, **#1779**: dense checkpoints of the Qwen3.6 family on the qwen36+ engine (Qwen3.8-27B), text and images.+- **#1780**: Qwen-Image-2.1, text to image: a C engine, pictures in the+ terminal, the web app and the OpenAI images API.+- **#1951**: Qwen-Image on the GPU and a faster start. The weights load in+ 7.5 s instead of 34 (four layers at once, quantized straight from bf16, the+ same int8 bytes). With Vulkan every block of a denoising step runs on the+ device, its attention included, and the VAE decodes there too; a card that+ does not hold the 7.1 GB transformer keeps the blocks that fit and streams+ the others each step. On a Radeon 780M a 512x512 image takes 61 s instead of+ 112 and a 1024x1024 step 38 s instead of 67; every stage checked against the+ diffusers reference.+- **#1831**: Qwen3.8 Flash Next's routed experts as int4-g64: 1.4 to 1.56x+ decode.+- **#1832**, **#1917**: speculative decoding with the checkpoint's MTP head,+ lossless, +12-14% on the release; with #1917 it is on by default with prompt+ lookup on every engine that has it (`Q38_MTP=0`, `COLI_LOOKUP=0` turn them+ off), and `coli plan` counts the MTP head's memory.+- **#1814**: Inkling reads the embedding norm under the name transformers 5.18+ writes.++### System One: one decision API++- **#1856**: `POST /v1/systemone`, one decision API, and the Laya decision+ engine; `/v1/brio` is removed.+- **#1868**: GLiNER2.5-Decide as a native decision engine (DeBERTa-v3 and+ GLiNER2's head).+- **#1872**: Clef as a native decision model on qwen36, a joint head over+ Qwen3.8-27B, with a 16-bit dense mode.+- **#1743**: the brio serve gates run in CI against the converted fixture.+- **#1753** (@tarazum): brio normalizes to sum by default and warns when a mean+ mixes option token counts.++### Setup and the CLI++- **#1953**: the release archives carry the GPU backends: every Linux and+ Windows engine built with Vulkan and its shaders, the macOS engines with+ Metal, and a Windows CUDA package (`coli_cuda.dll` for compute 8.0 and newer,+ with colibri, qwen36 and kimi_k3 built to load it). The Vulkan loader is+ opened at run time, so these engines still start on a machine without one,+ on the CPU. `coli setup` uses them without a Vulkan toolchain.++- **#1842**: one-step install and start (`start-here`, `coli setup`), the+ `AI_SETUP.md` guide and an MCP server.+- **#1850**, **#1896**: setup picks the backend that works on the machine: on an+ integrated GPU, Vulkan only where it was measured faster; CUDA only when the+ toolkit builds for the card, Vulkan otherwise, and a failed build falls back.+- **#1903**: an engine already built is brought up to date with `make` before it+ is used (a `git pull` used to keep running the old binary), and `coli logs`+ says where a foreground server prints (#1852).+- **#1919**: a fresh install of Qwen3.6-35B-A3B or Qwen3.8 Flash Next gets a+ starting expert history, so its first run fills the GPU tier from the start:+ on a Radeon 780M Qwen3.6's first run went from about 6 to 11 tok/s.+- **#1817**: in `coli chat`, a line that starts with a picture's path is a+ message, not a command.+- **#1315** (@Avicennasis): resumable, hash-verified qpack installers for+ Hugging Face and static mirrors.+- **#1316** (@Avicennasis): the planner honours cgroup memory limits.+- **#1949**: on Windows every engine has its bare `make <engine>` target (six+ fell to make's built-in rule and linked without CUDA or Vulkan, #1945,+ #1900); `coli setup` prefers an MSYS2 that can build Vulkan over a PATH+ compiler that cannot, and installs libgomp; `COLI_VK_DENSE=0` keeps the dense+ chain off too (on an 8 GB laptop it had taken the expert tier's budget,+ #1900); DeepSeek V4 uses the physical cores on an SMT machine (+10% decode on+ a Threadripper, #1906); `coli plan` leaves a card below an engine's CUDA+ floor out; the server says why its engine stopped (#1941); clearer messages+ for a missing model and a CUDA DLL that does not load.++### The serve contract++- **#1353** (@monotophic): `logprobs`, `top_logprobs` and `echo` on the OpenAI+ endpoints, aligned by raw-stream span.+- **#1794**, **#1767**: Qwen3.6 tool calling, and the template's+ `preserve_thinking` rendered so standard clients get KV prefix reuse.+- **#1802** (@bherald): opt-in strict GLM tool-call parsing.+- **#1765** (@SulimanAbdulrazzaq): GLM-5.3's reasoning stays out of the+ `/v1/messages` answer.+- **#1791**, **#1793** (@Avicennasis): Anthropic image blocks are served and a+ tool result's picture is not dropped; the modalities a model card claims are+ the ones the engine loaded.+- **#1787**, **#1804** (@benmaster82): unsupported OpenAI result shaping is+ refused; the completions keepalive streams as a text chunk.+- **#1782**, **#1783**, **#1785**, **#1786** (@kevin9327): malformed tool calls,+ text parts, tools, messages and image URLs answer 400 instead of 500.+- **#1809** (@Stamina9): non-string brio message text is refused.+- **#1749**, **#1751**, **#1789**, **#1803** (@enitimeago): glm53 reports REUSE+ in detail and logs CANCEL, does not restore a pin over rows another branch+ rewrote, handles cancellation during prompt processing, and continues from the+ cache instead of prefilling again.++### Correctness and security++- **#1930**: hardening of three input paths. The files the engines keep beside a+ model (`.coli_kv`, `.coli_usage.tmp`, `hot_pinned.bin`, `.coli_ckpt/`) are+ opened without following a symlink and only as regular files, so a link+ planted in a downloaded model directory no longer redirects the engine's+ writes. Request grammars are limited to 32767 alternates and symbols per rule+ (past that the walker's index wrapped) and are refused with a message.+ colibri, kimi_k3 and deepseek_v41 check the indexer's and RoPE's head+ dimensions in the config.+- **#1739**, **#1740**, **#1754**, **#1755**, **#1854**, **#1855**+ (@namespaceMarcello): the published advisory fixes carried to glm53, qwen36 and+ deepseek_v41; glm53 refuses f32 tensors shorter than the config reads and+ matrices whose shape differs from it; deepseek_v41 keeps engram lookups inside+ their table; qwen36 gives the same logits at every expert-cache capacity on FMA+ builds and never evicts an expert a MoE run is still reading.+- **#1775**: the DeepSeek V4/V4.1 pre-tokenizer in `tok.h`, so Turkish+ tokenizes as HF does.+- **#1799** (@RayanR000): the DeepSeek V4.1 reference sends top-k ties to the+ lowest index and its indexer never keeps -inf.+- **#1812** (@ZacharyZcR): the shared-expert bindings of DeepSeek V4's fused MoE.+- **#1611** (@Cometbuster4969): OLMoE uses the shared RAM probe, with a+ plausibility floor.+- **#1744**, **#1745** (@kevin9327): a numeric `node_id` no longer kills the+ cluster's topology and health requests, and a manifest that is not a JSON+ object no longer crashes the experiments validator.+- **#1356** (@monotophic): the ablation scoring mode replaced by a checked,+ digest-bound one.+- **#1904**: DeepSeek V4's `PROF` measures the expert wait instead of printing 0+ and folding it into `attention_s` (#1852).+- **#1902**: Windows expert prefetch is asynchronous and its queued work bounded.++### Speed on the CPU++- **#1837**: every engine's per-token projections run per block of rows in+ prefill.+- **#1761** (@mfethe1): a NEON arm for batched `matmul_fp8`, bit-exact, 6 to 10x+ at prefill shapes.+- **#1790**, **#1798** (@RayanR000): the f32 matmul runs eight `fmaf` chains at+ once with the same bits on every FMA build, and one vectorized f32 matmul+ serves quant.h, olmoe, inkling and qwen36 (#442).+- **#1773** (@JackKnifeAI): glm53's dense prefill batched, bit-identical per+ token.+- **#1784** (@kreuzzelg): qwen36's DeltaNet layer on the card+ (`Q36_DN_GPU=1`), expert homes per layer range (`QT_HOME=layer`), the shared+ expert offered on request.+- **#1323**, **#1343** (@Avicennasis): qwen36 reads routed experts from a qpack+ container through bounded Metal slots, and MLX-affine dense weights and norms.+- **#1261** (@Kenneth-Javier): an optional explicit AMD XDNA2 lane for the GLM+ shared expert.++### Build, tests and CI++- **#1758** (@Kenneth-Javier): header prerequisites generated with `-MMD -MP`+ (#1741).+- **#1845**, **#1912**: the sanitizer runs and the bf16 gates hold on AVX-512+ runners, whose CPUs keep bf16 on the CPU.+- **#1914**: the Windows UCRT64 job's limit from 35 to 60 minutes.+- **#1795**, **#1821**, **#1836**, **#1839**: four flaky tests made+ deterministic.++### Docs++- **#1897**, **#1833**: the README rewritten for the release (one-step setup,+ models per machine, GPUs, System One) in five languages, with the site.+- **#1768**, **#1776**: every engine in the environment reference, DeepSeek V4.1+ in its own section, and the weights-from-disk options.+- **#1788** (@jamshaid120): duplicate entries removed from the DeepSeek V4 docs.+- **#1801** (@benmaster82): the web app's image alt count and German image+ labels.+- **#1738**: the 1.12.1 entry's pull request count corrected.+## [1.12.1] — 2026-09-2496 pull requests since v1.12.0, 80 of them from contributors. Two tokenizersdiff --git a/README.id.md b/README.id.mdindex 57ecfeb5..856f6b67 100644--- a/README.id.md+++ b/README.id.md@@ -31,7 +31,7 @@ engine Qwen3.6. [Mana yang cocok untuk mesin saya](#which-model-for-my-machine)```$ ./coli chat- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! Come posso aiutarti oggi?diff --git a/README.it.md b/README.it.mdindex 4934f5d6..2d8cbe6e 100644--- a/README.it.md+++ b/README.it.md@@ -30,7 +30,7 @@ motore Qwen3.6. [Quale scegliere per la mia macchina](#which-model-for-my-machin```$ ./coli chat- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! Come posso aiutarti oggi?diff --git a/README.ja.md b/README.ja.mdindex 5a258df5..a07a3ffe 100644--- a/README.ja.md+++ b/README.ja.md@@ -19,7 +19,7 @@```$ ./coli chat- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! Come posso aiutarti oggi?diff --git a/README.md b/README.mdindex ef40578a..b1bb1cf7 100644--- a/README.md+++ b/README.md@@ -29,7 +29,7 @@ Qwen3.6 engine. [Which one for my machine](#which-model-for-my-machine)```$ ./coli chat- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! Come posso aiutarti oggi?diff --git a/README.zh-CN.md b/README.zh-CN.mdindex e381a4b5..ed373044 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -19,7 +19,7 @@```$ ./coli chat- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! Come posso aiutarti oggi?diff --git a/README.zh-TW.md b/README.zh-TW.mdindex 0ce37220..c818e15a 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -28,7 +28,7 @@ Qwen3.6 引擎上。[我的機器適合哪一個](#which-model-for-my-machine)```$ ./coli chat- colibri v1.12.1 · GLM-5.2 · 744B MoE · int4 · streaming CPU+ colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU✓ ready in 32s · resident 9.9 GB› ciao!◆ Ciao! Come posso aiutarti oggi?diff --git a/c/setup_catalog.py b/c/setup_catalog.pyindex 34c44d3a..7cfd8fe9 100644--- a/c/setup_catalog.py+++ b/c/setup_catalog.py@@ -76,7 +76,7 @@ CATALOG = (# docs/qwen36.md: 6.5 GB resident at 32 experts/layer, 15.2 GB with all 128.ram_min_gb=8, ram_good_gb=18, dense_gb=3.6, rank=2, size_class="small",summary="coding model with tool calls, no thinking; int4-gs64 container",- prebuilt_since="1.12.2", doc="docs/qwen36.md#qwen3-coder-30b-a3b"),+ prebuilt_since="2.0.0", doc="docs/qwen36.md#qwen3-coder-30b-a3b"),CatalogModel(id="deepseek-v4-flash-reap", family="deepseek_v4", name="DeepSeek V4 Flash REAP 150B",repo="puwaer/DeepSeek-V4-Flash-0731-reap-150b", disk_gb=84.7,@@ -104,7 +104,7 @@ CATALOG = (# The engine never loads these three (docs/mimo.md, measured on Pro,# the same engine and layout).exclude=("dflash/*", "audio_tokenizer/*", "model_mtp.safetensors"),- prebuilt_since="1.12.2", doc="docs/mimo.md"),+ prebuilt_since="2.0.0", doc="docs/mimo.md"),CatalogModel(id="glm-5.2", family="glm", name="GLM-5.2",repo="mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp", disk_gb=429.3,@@ -123,7 +123,7 @@ CATALOG = (ram_min_gb=54, ram_good_gb=64, dense_gb=32.4, rank=9, size_class="large",summary="1.02T with vision and tools, official checkpoint",exclude=("dflash/*", "audio_tokenizer/*", "model_mtp.safetensors"),- prebuilt_since="1.12.2", doc="docs/mimo.md#pro"),+ prebuilt_since="2.0.0", doc="docs/mimo.md#pro"),CatalogModel(id="inkling", family="inkling", name="Inkling",repo="nbeerbower/Inkling-colibri-int4", disk_gb=514.1,@@ -148,7 +148,7 @@ CATALOG = (ram_min_gb=12, ram_good_gb=18, dense_gb=8.5, rank=1, size_class="small",modality="image", summary="text to image (pictures, not chat)",license_note="Qwen Research License: non-commercial use only",- prebuilt_since="1.12.2", doc="docs/qwen-image.md"),+ prebuilt_since="2.0.0", doc="docs/qwen-image.md"),)diff --git a/c/version.py b/c/version.pyindex db24fa2d..d42baa29 100644--- a/c/version.py+++ b/c/version.py@@ -1,3 +1,3 @@"""Single source of truth for the colibri version number."""-__version__ = "1.12.1"+__version__ = "2.0.0"diff --git a/site/index.html b/site/index.htmlindex 08001013..bd89e979 100644--- a/site/index.html+++ b/site/index.html@@ -630,7 +630,7 @@ cd colibri<footer><div class="fin"><span>© 2026 colibri</span>- <span>Currently shipping <b>v1.12.1</b></span>+ <span>Currently shipping <b>v2.0.0</b></span><span class="sp"><a href="https://github.com/JustVugg/colibri/">GitHub</a><a href="https://github.com/JustVugg/colibri/blob/main/docs/quickstart.md">Docs</a><a href="https://github.com/JustVugg/colibri/blob/main/docs/systemone.md">System One</a><a href="https://github.com/JustVugg/colibri/blob/main/docs/api.md">API</a>