Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
Changelog
All notable changes to colibrì are documented here. Format follows Keep a Changelog.
[2.0.0] - 2026-10-06
169 pull requests since v1.12.1, 88 of them from contributors. Every MoE engine now runs on any GPU a Vulkan driver can see: the routed experts on a shared device tier, the dense layers on a device chain (all of them, or the first N that fit), and a second GPU for more experts. Three model families arrive (MiMo-V2.6, Qwen3-Coder, Qwen-Image-2.1) and one dense checkpoint (Qwen3.8-27B). Qwen3.8 Flash Next gets int4 experts and speculative decoding that is on by default. Every text engine serves up to 16 conversations at once. The release archives carry the GPU backends. Setup is one step, and there is one decision API (System One).
Several conversations at once
- #1955: every text engine serves several conversations at once (
coli serve --kv-slots N, up to 16), as GLM-5.2 and GLM-5.3 Flash did: qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B), qwen38, OLMoE, Inkling, Kimi K3, MiMo-V2.6, DeepSeek V4 and V4.1. Each slot keeps its own conversation's state, and every decode step takes the next token of each active request as one batch, so the weights and experts a step reads serve all of them. A request gets the tokens it gets alone, checked frame for frame on every engine, on the CPU and with the expert tier. With more than one slot nothing drafts and the dense chain stays off.
Vulkan on every engine
- #1830: every engine's resident matrices run on any Vulkan GPU (fmt 10, 11
and 12 added), each configuration checked against the engine's own CPU run on
Lavapipe (
tests/vulkan_engines.sh). - #1834: tiled GEMMs for prefill in every weight format, and a cooperative-matrix GEMM where the device has one.
- #1840, #1843: a shared routed-expert tier (
vk_tier.c). It holds a cache of experts in device memory, filled from the expert history and adapting while you chat, and computes the resident experts of each step as one async batch while the CPU computes the rest. First on qwen38 and qwen36, then on every MoE engine: GLM-5.2, GLM-5.3 Flash, Inkling, OLMoE, Kimi K3, MiMo-V2.6, DeepSeek V4 and V4.1. - #1844, #1848, #1849, #1851, #1857: the dense chain
(
vk_chain.c), a layer's dense part in one submission, on qwen36, qwen38, MiMo-V2.6 (sliding window and sinks on the device), OLMoE, Inkling (and OLMoE's PILOT race fixed), GLM-5.2 and GLM-5.3 Flash (shared MLA, DSA, KDA and mHC ops), Kimi K3, DeepSeek V4.1 and DeepSeek V4. - #1871: staged device-local uploads for cards without Resizable BAR, where the mapped path failed or spilled to system RAM.
- #1901: expert streaming for big prompt steps, memory budgets and speculative verification on the device.
- #1909, #1913, #1916, #1918: a partial dense chain. The first N
layers that fit go to the device, the CPU runs the rest, and nothing is
uploaded that the chain will not use, so an 8 GB card with a larger trunk keeps
the chain and its expert tier instead of losing both.
COLI_VK_CHAIN_LAYERSforces N,coli planpredicts it. DeepSeek V4, qwen36 and olmoe, qwen38, GLM-5.2 and GLM-5.3 Flash. - #1920: a second GPU for the expert tier (
COLI_VK_DEV2=auto|<index>) on every MoE engine: the experts after the primary device's, each step one batch per device, both in flight at once. A failing second device gives its experts back to the CPU and the tier goes on with the first. - #1942: the dense chain's layers on two GPUs. With
COLI_VK_DEV2, the layers a partial chain leaves go to the second device instead of the CPU, on every chain engine (qwen36, qwen38, OLMoE, MiMo-V2.6, Inkling, GLM-5.2, GLM-5.3 Flash, Kimi K3, DeepSeek V4.1 and V4).COLI_VK_CHAIN_LAYERS2forces how many,COLI_VK_CHAIN_DEV2=0keeps them off. A lost second device leaves the CPU to run its layers, the recurrent state rebuilt where it was held. Checked on Lavapipe opened twice; two real GPUs not measured yet. - #1933:
COLI_VK_KV_COLD=device(opt-in): with the KV cache split past the device's budget, the host's part of the attention runs on the device too, from host memory it reads in place, so a step runs in one frame. The prompt gains 4-6% on a Radeon 780M, decode is slower there (hence off by default); a dedicated GPU has not been measured. - #1948: several conversations at once on the dense chain. colibri's
multiplexed decode (
KV_SLOTS, one row from each active conversation) runs on the device: the batch's matrices as one, each row's attention over its own conversation's KV mirror (up toCOLI_VK_CHAIN_MUX, default 16, beside the chain's own) with its own DSA list; two devices and the partial chain too. Every frame checked against the CPU's on Lavapipe; not measured with a real checkpoint yet. - #1934, #1935, #1936, #1937, #1938, #1939 (@huppiflupp): the matrix units for prompt attention and int8/int4 prompt GEMMs on the chain, a step's experts as one grouped GEMM (decode: one grouped GEMV per phase), qwen36's routing in parallel, and int8 decode GEMVs sized by the matrix. On a Radeon 780M with Qwen3.6-35B-A3B the first token of a 1000-token prompt comes in 9.3 s instead of 12.6, decode 6.1 tok/s instead of 5.7, perplexity unchanged.
- #1954: on an Intel GPU the dense chain decoded garbage (Qwen3.6 on an Iris Xe, the default there). Its pipelines now run at the subgroup size the device reports; the device compiles at 8 to 32 lanes, and the chain's GEMVs had run at another width than the one they read. On Windows the chain also finds its optional shaders (MLA, KDA, mHC, DeepSeek V4, the split KV cache), and the split cache frees its host memory with the matching call.
- #1911:
COLI_VK_DEV=<index>picks the device on a machine with two GPUs of the same kind; the cooperative GEMM's epilogue stride is aligned to 16 bytes, a validation error on some drivers (#1908). - #1846 (@crichalchemist): correct expert-batch results under MoltenVK's default mode.
- #1762, #1763 (@Kenneth-Javier): the Vulkan loader linked on Windows
under
-static, and the Vulkan objects linked into every engine-including test rule.
Models
- #1813, #1819, #1820, #1827, #1835: MiMo-V2.6 (Flash and
Pro): the engine, the vision tower, tool calling and Brio mode with logprobs,
read as released; Pro verified against Xiaomi's reference on the real
checkpoint, and
coli servenames the loaded variant. - #1828, #1829: Qwen3-Coder-30B-A3B (
qwen3_moe) on the qwen36 engine. - #1777, #1779: dense checkpoints of the Qwen3.6 family on the qwen36 engine (Qwen3.8-27B), text and images.
- #1780: Qwen-Image-2.1, text to image: a C engine, pictures in the terminal, the web app and the OpenAI images API.
- #1951: Qwen-Image on the GPU and a faster start. The weights load in 7.5 s instead of 34 (four layers at once, quantized straight from bf16, the same int8 bytes). With Vulkan every block of a denoising step runs on the device, its attention included, and the VAE decodes there too; a card that does not hold the 7.1 GB transformer keeps the blocks that fit and streams the others each step. On a Radeon 780M a 512x512 image takes 61 s instead of 112 and a 1024x1024 step 38 s instead of 67; every stage checked against the diffusers reference.
- #1831: Qwen3.8 Flash Next's routed experts as int4-g64: 1.4 to 1.56x decode.
- #1832, #1917: speculative decoding with the checkpoint's MTP head,
lossless, +12-14% on the release; with #1917 it is on by default with prompt
lookup on every engine that has it (
Q38_MTP=0,COLI_LOOKUP=0turn them off), andcoli plancounts the MTP head's memory. - #1814: Inkling reads the embedding norm under the name transformers 5.18 writes.
System One: one decision API
- #1856:
POST /v1/systemone, one decision API, and the Laya decision engine;/v1/briois removed. - #1868: GLiNER2.5-Decide as a native decision engine (DeBERTa-v3 and GLiNER2's head).
- #1872: Clef as a native decision model on qwen36, a joint head over Qwen3.8-27B, with a 16-bit dense mode.
- #1743: the brio serve gates run in CI against the converted fixture.
- #1753 (@tarazum): brio normalizes to sum by default and warns when a mean mixes option token counts.
Setup and the CLI
#1953: the release archives carry the GPU backends: every Linux and Windows engine built with Vulkan and its shaders, the macOS engines with Metal, and a Windows CUDA package (
coli_cuda.dllfor compute 8.0 and newer, with colibri, qwen36 and kimi_k3 built to load it). The Vulkan loader is opened at run time, so these engines still start on a machine without one, on the CPU.coli setupuses them without a Vulkan toolchain.#1842: one-step install and start (
start-here,coli setup), theAI_SETUP.mdguide and an MCP server.#1850, #1896: setup picks the backend that works on the machine: on an integrated GPU, Vulkan only where it was measured faster; CUDA only when the toolkit builds for the card, Vulkan otherwise, and a failed build falls back.
#1903: an engine already built is brought up to date with
makebefore it is used (agit pullused to keep running the old binary), andcoli logssays where a foreground server prints (#1852).#1919: a fresh install of Qwen3.6-35B-A3B or Qwen3.8 Flash Next gets a starting expert history, so its first run fills the GPU tier from the start: on a Radeon 780M Qwen3.6's first run went from about 6 to 11 tok/s.
#1817: in
coli chat, a line that starts with a picture's path is a message, not a command.#1959 (@rudycelekli):
coli setupcounts a server on its port as a running colibri only when/healthanswers as colibri does; another program's HTTP 200 no longer passes for one, and the setup starts its own on the next free port.#1315 (@Avicennasis): resumable, hash-verified qpack installers for Hugging Face and static mirrors.
#1316 (@Avicennasis): the planner honours cgroup memory limits.
#1949: on Windows every engine has its bare
make <engine>target (six fell to make's built-in rule and linked without CUDA or Vulkan, #1945, #1900);coli setupprefers an MSYS2 that can build Vulkan over a PATH compiler that cannot, and installs libgomp;COLI_VK_DENSE=0keeps the dense chain off too (on an 8 GB laptop it had taken the expert tier's budget, #1900); DeepSeek V4 uses the physical cores on an SMT machine (+10% decode on a Threadripper, #1906);coli planleaves a card below an engine's CUDA floor out; the server says why its engine stopped (#1941); clearer messages for a missing model and a CUDA DLL that does not load.
The serve contract
- #1353 (@monotophic):
logprobs,top_logprobsandechoon the OpenAI endpoints, aligned by raw-stream span. - #1794, #1767: Qwen3.6 tool calling, and the template's
preserve_thinkingrendered so standard clients get KV prefix reuse. - #1802 (@bherald): opt-in strict GLM tool-call parsing.
- #1765 (@SulimanAbdulrazzaq): GLM-5.3's reasoning stays out of the
/v1/messagesanswer. - #1791, #1793 (@Avicennasis): Anthropic image blocks are served and a tool result's picture is not dropped; the modalities a model card claims are the ones the engine loaded.
- #1787, #1804 (@benmaster82): unsupported OpenAI result shaping is refused; the completions keepalive streams as a text chunk.
- #1782, #1783, #1785, #1786 (@kevin9327): malformed tool calls, text parts, tools, messages and image URLs answer 400 instead of 500.
- #1809 (@Stamina9): non-string brio message text is refused.
- #1749, #1751, #1789, #1803 (@enitimeago): glm53 reports REUSE in detail and logs CANCEL, does not restore a pin over rows another branch rewrote, handles cancellation during prompt processing, and continues from the cache instead of prefilling again.
Correctness and security
- #1930: hardening of three input paths. The files the engines keep beside a
model (
.coli_kv,.coli_usage.tmp,hot_pinned.bin,.coli_ckpt/) are opened without following a symlink and only as regular files, so a link planted in a downloaded model directory no longer redirects the engine's writes. Request grammars are limited to 32767 alternates and symbols per rule (past that the walker's index wrapped) and are refused with a message. colibri, kimi_k3 and deepseek_v41 check the indexer's and RoPE's head dimensions in the config. - #1739, #1740, #1754, #1755, #1854, #1855 (@namespaceMarcello): the published advisory fixes carried to glm53, qwen36 and deepseek_v41; glm53 refuses f32 tensors shorter than the config reads and matrices whose shape differs from it; deepseek_v41 keeps engram lookups inside their table; qwen36 gives the same logits at every expert-cache capacity on FMA builds and never evicts an expert a MoE run is still reading.
- #1775: the DeepSeek V4/V4.1 pre-tokenizer in
tok.h, so Turkish tokenizes as HF does. - #1799 (@RayanR000): the DeepSeek V4.1 reference sends top-k ties to the lowest index and its indexer never keeps -inf.
- #1812 (@ZacharyZcR): the shared-expert bindings of DeepSeek V4's fused MoE.
- #1611 (@Cometbuster4969): OLMoE uses the shared RAM probe, with a plausibility floor.
- #1744, #1745 (@kevin9327): a numeric
node_idno longer kills the cluster's topology and health requests, and a manifest that is not a JSON object no longer crashes the experiments validator. - #1356 (@monotophic): the ablation scoring mode replaced by a checked, digest-bound one.
- #1904: DeepSeek V4's
PROFmeasures the expert wait instead of printing 0 and folding it intoattention_s(#1852). - #1902: Windows expert prefetch is asynchronous and its queued work bounded.
Speed on the CPU
- #1837: every engine's per-token projections run per block of rows in prefill.
- #1761 (@mfethe1): a NEON arm for batched
matmul_fp8, bit-exact, 6 to 10x at prefill shapes. - #1790, #1798 (@RayanR000): the f32 matmul runs eight
fmafchains at once with the same bits on every FMA build, and one vectorized f32 matmul serves quant.h, olmoe, inkling and qwen36 (#442). - #1773 (@JackKnifeAI): glm53's dense prefill batched, bit-identical per token.
- #1784 (@kreuzzelg): qwen36's DeltaNet layer on the card
(
Q36_DN_GPU=1), expert homes per layer range (QT_HOME=layer), the shared expert offered on request. - #1323, #1343 (@Avicennasis): qwen36 reads routed experts from a qpack container through bounded Metal slots, and MLX-affine dense weights and norms.
- #1261 (@Kenneth-Javier): an optional explicit AMD XDNA2 lane for the GLM shared expert.
Build, tests and CI
- #1758 (@Kenneth-Javier): header prerequisites generated with
-MMD -MP(#1741). - #1845, #1912: the sanitizer runs and the bf16 gates hold on AVX-512 runners, whose CPUs keep bf16 on the CPU.
- #1914: the Windows UCRT64 job's limit from 35 to 60 minutes.
- #1795, #1821, #1836, #1839: four flaky tests made deterministic.
Docs
- #1897, #1833: the README rewritten for the release (one-step setup, models per machine, GPUs, System One) in five languages, with the site.
- #1768, #1776: every engine in the environment reference, DeepSeek V4.1 in its own section, and the weights-from-disk options.
- #1788 (@jamshaid120): duplicate entries removed from the DeepSeek V4 docs.
- #1801 (@benmaster82): the web app's image alt count and German image labels.
- #1738: the 1.12.1 entry's pull request count corrected.
[1.12.1] β 2026-09-24
96 pull requests since v1.12.0, 80 of them from contributors. Two tokenizers
brought back to the reference, brio on the ninth engine, coli chat working
again at the default context on two families, and a placement decision that
is now measured on the card in front of it instead of predicted.
Tokenizers, measured against the reference
- #1654: qwen36 tokenized differently from HF
tokenizersin two ways. An added token right after punctuation was encoded as text (X.<|im_end|>was 7 tokens instead of 3, every chat turn ending in punctuation paid +4, #1653), and a whitespace run followed by a non-space was one piece where the regex's\s+(?!\S)leaves the last char to the next one, so every indented line of code tokenized differently. Measured on the real vocabulary: 2,803 lines and blocks of code, Markdown, Chinese and Japanese went from 757 identical to 2,803, with 5.4% fewer tokens. - #1656: OLMoE's
tokenizer.jsonhas no Split, a bare ByteLevel withuse_regex, for which HF runs the original GPT-2 pattern;tok.happlied cl100k. A GPT-2 family intok.h: 1,560/1,708 identical before, 1,708/1,708 after. The same measurement on GLM-5.2/5.3/5.3-Flash, DeepSeek V4 and V4.1, Inkling and Qwen3.8 came back identical on every case.
Brio and the serve contract
- #1662:
POST /v1/systemone, the request and the reply of TypeSafe's Jev API, served by the brio channel: a client written for it points at colibri and changes the base URL.noulis a yes/no question,choicescores the labels with their descriptions in the text,scorethe level numbers with the expected value and the legend;confidenceby their documented formula. Anymodelname is accepted on that route. Measured on the real Qwen3.6: the three-question example of the docs in 1m46 with the state read once. - #1655: the DeepSeek V4 engine speaks the numeric channel (
logprobs=k,pin=1,max_tokens=0), so/v1/brioworks on the ninth engine instead of answering 500 (#1648). The head that used to keep only its argmax now returns the whole row;ECHOper prompt position during prefill, the prompt-end scores kept with a state snapshot onpin, and a logprob tail on everyDATAframe during generation. The tiny fixture pins that the bestECHOtoken equals the greedy token from the same prefix, and that the pinned predictor equals a cold prefill's. - #1659: qwen36 and qwen38 refused a request when
prompt + max_tokensexceeded the context, and the gateway's default budget for these two families is 8192, the whole default context: every request withoutmax_tokensand everycoli chatmessage answered 400 on a two-token prompt (#1641).max_tokensis now a ceiling, clamped to the room the prompt leaves, as GLM and DeepSeek V4 already did; only a prompt that does not fit is refused.docs/api.mdstates the rule,docs/qwen38.mdnamesQ38_MAXTas the variable--ctxbecomes.
The dense trunk in VRAM, measured before it is placed
- #1657: qwen36 offers the rest of its dense trunk to the VRAM placer:
the DeltaNet out_proj (
dnout), the attention q/k/v/o (attnproj) and the shared expert (shexp), about 650 MB more of int8 on the 35B besidelmheadanddnproj. Measured on four Tesla M10 by the reporter of #1652, every placed component ran slower than the CPU (lm_head 68.8 ms against 41.7), so the engine now times one GEMV both ways at startup and withdraws the whole automatic placement when the GPU loses, giving the VRAM back to the experts:autoequalsoffon that box, byte-identical output. A hand-writtenCOLI_PLACEstands;COLI_TRUNK_PROBE=0trusts the placer.
Performance
- #1664: qwen36's dense trunk and routed experts multiply with integer
dot products. The activation is quantized to int8 once per call and the
weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd
instead of a float conversion per weight; the integer kernels move from
quant.hintoidot.h, shared by every engine. Measured on the 35B, 8 threads, every expert resident: decode 6.71 to 8.23 tok/s (+22.6%), lm_head 12.6 to 10.1 ms/token, the expert compute 22.7 to 15.6, for +1.3% perplexity on 4 x 512 tokens. Both are the default (COLI_DENSE_IDOT=0,QWEN_EXPERT_ACT=f32restore the f32 kernels).COLI_DENSE_BITS=4withCOLI_DENSE_INT4=<components>stores part of the trunk as int4 in blocks of 64: opt-in, with the perplexity it costs per component in the docs (lm_head alone +2.4%, everything +10%). - #1668: qwen38's dense trunk (553 matrices, 3.6 G weights, 8 GiB of
BF16 read on every token, more than the ten routed experts) is kept on the
CPU as int8 rows with the BF16 copy released, and multiplied with the same
integer kernels; the routed experts' e4m3 blocks are decoded eight at a
time in registers and multiplied with FMA instead of one table lookup per
weight. Measured on the released Qwen3.8-Flash-Next-FP8, 8 threads, RAM
LRU 96 per layer: decode 0.61 to 1.42 tok/s, the trunk 434 to 85 ms/token,
lm_head 76 to 13, the expert GEMVs 388 to 140, peak RSS 32.2 to 28.5 GB;
prefill of 512 tokens 495 to 149 s. Perplexity on 4 x 512 tokens +0.5%
(two chunks lower, two higher); the vector FP8 kernel alone reproduces the
BF16 run to four decimals. Both are the default (
Q38_TRUNK_CPU_INT8=0keeps the BF16 trunk,Q38_FP8_KERNEL=scalarthe table kernel).
Performance, from contributors
- #1606: the K1b grouped int4 family gets a multi-row tile and AVX-512 and AMX arms, and is no longer switched off on AVX-512 builds; exact on all eight engines on a 16-core AVX-512 host.
- #1239: an SSE4.1 tier for the olmoe and qwen36 int8 GEMV, for hosts with SSE4.1 but no AVX2; on the Sandy Bridge of #1652 decode went from 2.45 to 3.54 tok/s.
- #1313:
matmul_fp8computes four output rows per pass under clang, where the contraction makes it bit-exact; GCC keeps the one-row kernel. - #1612: qwen36 gains the GLM engine's
CACHE_ROUTElever, with the VRAM tier as the first residency level, opt-in. - #906:
DEGRADE_ZERO, an opt-in policy that zero-fills a missed expert slot below a gate-weight threshold instead of blocking on the load (#865). - #1677: qwen36 projects a prompt's DeltaNet inputs (qkv and z) on the card in blocks of up to 256 rows instead of one row at a time; a 259-row prefill makes 2 projection calls instead of 259, with the convolution history and the recurrent state checked against the CPU run. The paired microbenchmark on an RTX 4070 read 4 to 10x per projection.
- #1674: qwen36's attention projections the tier placed in VRAM answer a whole prompt batch with one call per matrix; a failed call turns only that handle off and the prompt continues on the CPU.
- #1676: Kimi K3's streaming CUDA expert keeps the gate, up and SiTU intermediates on the device and applies down there (one fused entry point, optional in the DLL: an older backend keeps the three-call path).
- #1673: the streaming MXFP4 matmul reuses one grow-only device scratch per card instead of allocating and freeing weights and scales on every call.
- #1559 (kreuzzelg):
convert_qwen36.py --down-bits 8writes the mixed expert layout, int4 gs64 gate/up and int8 down in one slab (5.7 bits per weight against gs64's 4.5); the engine tells it apart by size and reads each matrix in its own format on the CPU path, and refuses the VRAM tier with a line. It is the knob behind the #1370 numbers: on wikitext-2 the int8 down alone recovers a quarter of the gap between gs64 and all-int8, the rest sits in gate/up. A measurement tool and a middle step, not the answer to the gap. - #1730 (mfethe1): DeepSeek V4's FP4 expert kernels, the prefill batch and the decode matvec, get a NEON arm; arm64 used to take the scalar arm, which is why Apple Silicon prefilled at decode speed (#1696). Bit identical to the scalar arm, and the ARM CI job now checks that on every change; 17 to 22x on the kernel at the V4 expert shapes on an M-series Mac, as measured by the author.
- #1716 (jtinbergen): qwen36 quantizes its dense weights to int8 while loading instead of keeping an f32 copy first, and converts f16/bf16 with SIMD. On the 35B the resident set after load goes from 9.2 to 4.8 GB; the generated text and the perplexity are identical to before.
- #1286 (cameron): the grouped int4 GEMV and the fused gate/up GEMV get an SSE4.1 arm for CPUs without AVX2 (Ivy Bridge and older). It vectorizes across output rows, so each lane runs the scalar row's exact sequence and the result is bit-identical; forced-SSE4.1 tests at -O1, -O3 and without FP contraction pin it. 2.2 to 2.7x on the isolated kernel on a dual E5-2680 v2.
- #1686 (DebugSultan): qwen38's prefill chunk (
Q38_PREFILL_BATCH_ROWS) and workspace (Q38_PREFILL_WORKSPACE_MIB) are runtime knobs, the expert load batch is no longer capped at top-k, and the QSA ranking and attention run per position in parallel at prefill. Measured on the released checkpoint on top of the int8 trunk: output byte-identical at every chunk width, no speed change on our 16-core server; the knobs are there for hardware where the chunk binds.
Fixed
- #1650 (bokiko): a Qwen3.8 pin snapshots the recurrent and PLE state
but reuses the live attention and indexer rows; after an unrelated prompt
overwrote those rows, returning to the pin could change brio logprobs
without a warning. The engine now records the token identity of the live
rows (
kv_prefix.h) and refuses a stale pin or prefix restore; image rows are tainted. Wire regressions run on the BF16 and FP8 fixtures. - #1626:
SNAPis the model directory for every non-GLM engine, socoli runstops handing them a leftover environment (#1600). - #1604: glm53 honours
Mat.residentin the Vulkan gate and frees the Vulkan handle inmat_release. - #1321: glm53 sizes its expert cache around the model rather than
around
MemAvailable, which the page cache had been inflating. - #1588: qwen36 refuses loudly on a failed encode-buffer realloc instead of writing through NULL.
- #1546:
coli convertroutes OLMoE toconvert_olmoe_merged.py. - #1630: olmoe emits the
ROUTE_TRACErecords it announced; the stream used to be a zero-byte file. - #1610:
v41_dsml.pyis staged during installation (and the nix flake bumped). - #1511: the GPU test suite builds under HIP on gfx1151.
- #1658:
test_mem_availablecompared two reads of available memory with==and failed on a busy Windows runner; a quarter of a GB of tolerance. - #1670: DeepSeek V4.1 read only its argv cache cap, so
RAM_GB=120on a 128 GB box left the engine at eight expert slots per layer and 23.8 GB of RSS (#1666). With--capomitted,coli chat,coli serveandcoli webnow size the cache from the resource plan, withRAM_GBor--ramas the budget; an explicit--cap, a measured profile and an auto-tier plan keep precedence. - #1671: DeepSeek V4.1 treats
max_tokensas a ceiling like the other engines (#1641): a fitting prompt with a large request generates what the context leaves, a score-only prompt may fill the context, and a prompt one token over it is refused instead of silently truncated. - #1675: resident MXFP4 tensors on CUDA carried O float scales where the kernel reads O x ceil(I/32) exponent bytes: short buffers were over-read and long ones truncated. One format-aware size for upload, refresh, accounting and release.
- #1678, #1679, #1680, #1682, #1683, #1684: the Qwen CUDA tier's lifecycle, end to end. Shutdown releases every resident expert, projection handle and host table after parked callers resume; a failed gate, up or down upload frees what it had already allocated; the expert budget charges the three scale buffers at their own sizes (two experts used to be admitted where one fit); a failed result collection stops inference instead of publishing a partial MoE sum; a failed or explicitly disabled tier start unwinds its storage and synchronization objects, and a second init cannot overwrite a running tier; the CUDA backend validates the whole device list before touching state and keeps live contexts on a repeated init. Fault-injected on the fake backend, then run together on an RTX 4070 under compute-sanitizer with zero errors and zero bytes leaked.
- #1669:
test_systemone_apiimports its scoring engine relative to its package, so an installedtestspackage no longer breaks discovery. - #1697 (kevin9327): the dashboard redesign had dropped the reasoning
stream: thinking tokens arrived on
delta.reasoning_contentand vanished, the bubble stayed empty until the answer and a stop during thinking lost the turn. The stream is read again, rendered as its own folding block, counted in the rate and the time to first token, and a unit test pins the split. - #1693 (namespaceMarcello): with
PILOTon, OLMoE could read the same expert twice, once from the prefetcher and once from the forward pass, into two slots; a slot being read now keeps a reservation in the index (thecolibri.cpattern) and the second caller waits for the first read to publish. Three model-free scenarios pin it. - #1695 (namespaceMarcello): the prefill echo state and
serve_echosit under the sameQWEN36_NO_MAINguard, so the segment build no longer warns about a function it never gets; the full build is byte-identical. - #1724 (GenericRikka): Qwen3.6 decoded
<think>,</think>and the tool tags to nothing, because they live only in the tokenizer'sadded_tokens; with thinking on, the closing tag never reached the gateway and the whole answer came back asreasoning_content. The non-special added tokens are decoded now; special ones such as<|im_start|>still decode to nothing. - #1734 (tarazum): stopping
coli servecloses the engine's stdin and waits for it to exit on its own before the hard-stop ladder, so the engine's teardown runs; qwen36 never saved itsHEAT_FILEundercoli serve(#1733). On Windows the gateway handles SIGBREAK and the engine runs in its own process group. - #1726 (kevin9327): Inkling measured no RAM on Windows and sized its expert cache to 16 per layer; it uses the shared probe now, which on Linux and macOS reads the same numbers as before.
- #1735: on GNU Make 3.81, the system make on macOS,
.build-configwas never written and every build relinked (#1732). - #1731 (bokiko): the DeepSeek V4 CUDA object rebuilds when the nvcc
command changes, so a new
CUDA_ARCHno longer links the old object. - #1728 (crichalchemist):
make test-c VK=1built 43 test binaries without the Vulkan object; they link it now. - #1712 (kevin9327): Kimi K3, Inkling and OLMoE now treat
max_tokensas a ceiling like the other engines;coli chat's default of 16384 answered 400 on every Kimi and Inkling message against their 8192-token window. - #1713 (kevin9327):
coli plan,doctorand--auto-tierexport the variable that actually sizes the expert cache on Kimi K3 (K3_EXPERT_GB) and GLM-5.3 (GLM53_EXPERT_GB); onlyRAM_GBwas exported, which neither engine reads as the cache size. - #1711 (kevin9327): on Windows, a Kimi K3
CUDA_DLLbuild and a HIP host were refused by--gpuas CPU-only; the probe reads the backend DLL name the host was built with, and the launcher maps--gpuonto Kimi'sK3_CUDA. - #1710 (kevin9327): a
tools[]entry whosefunctionis not an object answered HTTP 500 from the GLM and DeepSeek renderers; it is the 400 thatgeneration_optionsalready had. - #1721 (monotophic): every frame the gateway writes to the engine is
checked, short writes are completed, and a failed
CANCELorSTOPdrops the request's pending entry and answers a named 500 instead of a silent close. - #1714, #1719 (benmaster82): the brio options form pins the shared state, so per-question requests on one document read it once; the web page can stop a scoring run, and duplicate options are removed on both clients.
- #1709 (kevin9327): regenerating a turn with pictures sends them again and leaves the composer alone.
- #1646 (Stamina9): qwen38 says once, on stderr, why the parallel expert read path is not taken (disabled, cache smaller than the route, no FP8 scale bank, repeated expert, converted layout).
- #1708 (wittchen): every
VK=1build of glm53 failed to compile on a misplaced parenthesis. - #1707 (namespaceMarcello): the DeepSeek V4 unit objects rebuild when
the build flags change, so a CUDA engine build followed by
make test-cno longer links the wrong objects (#1702). - #1715 (namespaceMarcello): seven GLM-5.3 harnesses matched the unittest glob and counted as zero tests; they are renamed, a skip exits 2, the two tiny oracles run in CI, and a discovery test catches the next empty module (#1700).
- #1622: DeepSeek V4's REAP checkpoints store each expert as six per-matrix records; the engine read them through buffered pread and counted every one as a direct-I/O fallback (36% of expert reads on the 150B, #1615). Each segment now goes through the aligned direct window, with a regression on a generated per-matrix fixture.
- #1597: a replayed tool call whose
argumentsparsed as JSON but was not an object ("[1, 2]","5") answered HTTP 500 from the GLM renderers before the engine was asked anything; it renders the call without arguments, as every other renderer already did. - #1624: the five gcc 13 warnings left in
make checkare gone, andst_index_loadrefuses an index path that would not fit its buffer instead of opening a truncated one, with a long-path case in the tests. - #1651: a
pyflakespass over the launcher, autotune, the family registry and the qwen36 converter: ameasure()defined twice, areadlineimport without a fallback, a stray f-string, dead variables. - #1580:
make qwen36 CUDA_DLL=1on Windows reached GNU make's implicit rule and built a CPU-only binary; a bareqwen36alias, a.build-configprerequisite so a CUDA_DLL change rebuilds, a loader-against-header parity test, and the Windows CUDA tier documented. - #1556:
coli planon macOS said "no supported GPU detected" on every Mac; it now lists the Metal device by name, as identity only, without pretending unified memory is a VRAM budget. - #1691: the installed launcher invoked as
/bin/colior/sbin/colion a merged-/usr system derived/libexec/colibriinstead of/usr/libexec/colibri, becauseabspathkept the alias (#1689, florin65's patch):realpathfirst. A test runs the launcher through such an alias, and another checks that every root module the launcher reaches is in themake installlist, the gap #1610 closed by hand.
Tools and the gateway
- #1425: a general GGUF reader, pure stdlib, and a converter from GGUF OLMoE checkpoints to a colibri container, with the numerical evidence in its own CI job.
- #1497: opt-in prompt-injected tool calling for the families without
native tool tokens (OLMoE, Qwen3.6), behind
COLI_TOOL_FALLBACK=1, with a two-turn end-to-end test. - #1355, #1357: durable per-request results and strict stdout classification in the eval harness, and the logprob-gap check gated on the engine preamble.
- #1687:
GET /metricsin Prometheus text format, behind the API key: four gauges, six outcome counters and four histograms (queue wait, slot occupancy, first output, engine call), no request labels, no new dependency. The admission scheduler distinguishes completion, failure and cancellation, lets a request use a free slot that no earlier waiter reserved, and joins the keepalive pump before the slot is released. - #1717 (enitimeago): the web chat offers Continue on the last
assistant message when it stopped at the token limit, by hand or on an
error, and only when
/healthsays the server continues assistant turns (#1699). - #1402 (enitimeago): a request whose last message is a non-empty
assistantturn continues that turn instead of answering in a new one, on/v1/chat/completionsand/v1/messages, for all nine families (Kimi K3 frames the open turn engine-side); the prompt ends inside the turn as the official template renders it without a generation cue. On by default,COLI_CONTINUE_ASSISTANT=0restores the old behaviour; refused together with tools or a turn ending in whitespace, with a 400 that says why. Each renderer is pinned against the vendored template (#1401). - #1102 (monotophic): checkpoint-faithful FP8 containers that store
kv_b_projas fmt=8 could load but not decode attention; the absorb path now decodes fmt=8 on CPU (bit-exact against the reference) and CUDA (within the documented tolerance), and the kv_b sharding refuses by name the formats it cannot serve, which also closes two silent misreads of fmt=5 and fmt=6. - #1395 (rybruscoe):
COLI_EXACT_VERIFY=1makes the speculative verify batch token-exact against sequential decode, at a measured cost on the dot itself; off by default, the default path is unchanged. - #1720 (monotophic): a request carrying
seedis accepted and the seed ignored, asdocs/api.mdnow says, instead of a 400; no determinism is implied. - #1605:
ORACLE_STRICT=1makes a GLM oracle comparison exit non-zero when it fails, token-exact by default withORACLE_TF_MAX_MISMATCHESfor the documented teacher-forcing allowance; references are validated before the comparison and non-finite logits cannot pass. Both oracle CI jobs run real-process regressions against it. - #1705:
tools/benchmark_baseline.py, a collection protocol on top of the HTTP harness for a repeated three-engine serving baseline: one frozen manifest (hardware, model and template identity, per-engine launch settings, cache and speculation policy), a rotating plan over a concurrency matrix, one collector per engine and round that manages no server, and a comparison that keeps failed and missing cells visible and distinguishes matched artifacts from deployment comparisons. No results are bundled and no ranking is emitted. - #1688:
tools/benchmark_http_serving.py, a stdlib HTTP streaming benchmark over fixed JSONL conversations: closed-loop or paced arrivals (periodic or Poisson, seeded), warmup separated from measurement, first-output and duration SLOs, latency percentiles and usage-based token throughput; a truncated or malformed stream is a failure, not a sample.
Docs
- #1639, #1644: the README shows brio mode and the dashboard as it
is: the workspace, the Brain page (the measured expert atlas as a cortex,
and a region inside it) and the Profiling page, in four languages;
docs/api.mddescribes the four pages instead of the old console. - #1492, #1643: a Japanese README, and its banner at the shipping version, which the banner test now checks in every language.
- #1634: the multi-disk guide states measured gains and limits instead of "twice the bandwidth", with Bash and PowerShell examples.
- #1617: connecting the pi coding agent to
coli serve. - #1619:
expected_bytesidentity versus physical extent for int4-rans256-g0 (#1273). - #1618 (bherald):
docs/qwen38.mdno longer calls the engine text-only; the vision tower and the gateway image path shipped in 1.12.0. - #1649, #1647 (Suraj2105-1): the musl CI job runs on Alpine 3.24 and the release pipeline on Node.js 22, ahead of the 3.21 and Node 20 end of life.
- #1568 (Yoruxyv): an Indonesian translation of the dashboard.
- #1681 (XBold): the README and
docs/qwen38.mdno longer say Qwen3.8 has no GPU backend; the CUDA VRAM expert tier and the int8 trunk in VRAM shipped in 1.12.0.
[1.12.0] β 2026-09-20
81 pull requests since v1.11.0. A new way to ask a model a closed question, a redesigned dashboard and landing page, and a long run of small failures that used to answer 500 or die on a locale.
Brio mode: score a closed set instead of generating
- #1632: brio mode, on all nine engines. Hand the engine the options an
answer is allowed to take and it reports the probability of each one
instead of writing the answer: the option tokens are read, not sampled,
so
completion_tokensis 0 and no reply can fall outside the list. It is opt-in per request through twoSUBMITkeys,logprobs=kandpin=1; without them every frame is byte-identical to before, which is asserted per engine rather than claimed.max_tokens=0became legal, but only together withlogprobs>0. - The shared contract is three headers:
decode_batch.hfor the logprob tail,serve_codec.hfor the wire, andpin_pool.hfor nested state snapshots, so a request can keep one photograph of the shared instructions and a deeper one of the instructions plus the question. With one level the question is re-read once per option; with two, four items cost 176 tokens instead of 496. kv_prefix_holds()is the guard that makes the snapshots safe: a snapshot is reused only when the prefix record still holds the same ids, because a K/V bank can be dropped between two options.- Exactness is measured, not asserted: the logprobs read from a snapshot match a cold recomputation to 0.00e+00 on all nine tiny fixtures, and on the two real checkpoints available (qwen36 at 22 GB and DeepSeek V4.1 Flash at 476 GB).
- Three clients:
POST /v1/brioon the gateway,/brioincoli chat, and a page in the dashboard.docs/brio.mdhas the request, the reply, the normalisation choice and the cases where the mode does not help. - The endpoint takes three forms.
optionsis one closed question.questionsis many questions on one state, each with its own options, with the state photographed once and the snapshots ordered by the server.schemais an object of field to allowed values: the server writes the JSON skeleton and fills it one field at a time, so the object is valid by construction and every value is one the caller allowed, each with its own probability and entropy. Both were measured before they were exposed: the 5.7x and the 2.4x above are these two forms.
The dashboard and the landing page
- #1633: the web dashboard is redesigned around a workspace and a navigation dock, with a brio page that reads a document once and asks it several questions, each with its own option set, and a light and dark theme.
- #1635: the landing page lists the models as a searchable list instead
of a card grid, gains a brio section, and gets a light and dark theme
built on the mechanism proposed in #1551. The shipping version and the
family names are generated from
c/version.pyandc/family_registry.py, so the page cannot claim eight families again while nine are running. The hero plays a recorded session rather than a mock-up.
Performance
- #1477: one routed-expert kernel,
expert_ffn.h, shared by the MoE engines and used by qwen36 first: int4 kept planar in RAM instead of unpacked to int8, and two OpenMP regions per layer instead of three per expert. 12.8 to 15.7 tok/s with the resident set down from 29 to 17 GB, output identical. - #1424: a CUDA VRAM tier for Qwen3.8-Flash-Next, 43 % on one 8 GB card and 60 % on two.
- #1553 and #1558: the previous turn's KV is reused instead of re-prefilling the transcript, on qwen36, olmoe and DeepSeek V4.1, with a resumed prefill that is exact.
- #1521 and #1524: the mHC mix and the KDA convolution and head loops run in parallel, bit-identical, for 3.9 % and 3.0 %.
- #1517: the four engines that never sized their OpenMP team now do.
- #1522: the DeepSeek V4.1 expert stream can be read from more than one
drive; #1455 weights glm53 expert reads by disk and honours
COLI_MODEL_MIRROR; #1543 fixes the V4 mirror cache on unified memory. - #1495: less overhead in the sparse router's top-K selection.
Fixed
- Requests that answered
500 engine failedand now answer properly: a non-objectjson_schema(#1587), an unpaired surrogate escape (#1589), a non-objecttool_choice.function(#1598), and a tool call whose arguments are not an object. - #1595: the grammar draft walker switched itself off after about sixty repetitions. #1596: a nested array schema doubled the compiled grammar at every level.
- #1483 and #1514: olmoe, inkling and glm53 refuse an over-long
prompt with
CONTEXT_EXCEEDEDinstead of failing late. - #1549: the lazy shard-mapping table in
st.hwas published without synchronisation and glm53 reached it from an OpenMP region. - #1557: a truncated multibyte tail made the qwen36 tokenizer read past
the prompt. #1530: an integer overflow check in
kimi_k3.c. - #1631:
coli servereaches the expert-history save when it is stopped with SIGTERM, not only on Ctrl-C. #1628: the expert grid is resent after every turn, so the dashboard stops showing the cold snapshot forever. - #1608:
coli doctormatches core tensor roles by component instead of by one family's spelling. - #1526, #1584, #1585, #1536: the planner, the auto-tuner and the registry agree about which engine is running, and stop advising GLM-only knobs on every family.
- #1489:
coli benchran the GLM engine on any model. #1503: every engine is handed its model directory throughSNAP. #1486 and #1488: a directory withoutconfig.jsonnames no engine, andcoli convertrefuses an output directory that already holds one.
Windows, macOS and locales
- #1515, #1560, #1561, #1562, #1516: the chat,
coli run,coli tuneand the datapoint harness no longer die when an engine line, a prompt or a dash does not fit the console code page. - #1537, #1579, #1535: the qwen36 CUDA_DLL build on Windows gets its tier, is recognised, and the device probe loads the backend before counting devices. #1563: the Inkling dense int4 converter finishes.
- #1599: MacPorts libomp built every macOS engine single-threaded.
#1548: Metal falls back to
MTLCopyAllDeviceswhen there is no system default device. #1504: olmoe measures available RAM on macOS and Windows too.
Diagnostics
- #1494, #1506, #1512:
PROFreports block time, head time, matmul, attention and lm_head, at microsecond resolution, on deepseek_v4, olmoe and qwen36. #1555:CONSIST=1checks prefill against decode. - #1482: configurable generation statistics in
coli chat, with exact token throughput when the server reports it. - #1468, #1487, #1480: a failure in the MoE step, a missing core tensor and duplicate tensor names are diagnosed by name instead of by a generic message.
Docs and licence
- #1540 and #1541: the copyright holder is named and a
NOTICEfile is added. - #1294: a reproducible benchmarking protocol; #1508: a GPU-backend datapoint needs a correctness line to be accepted.
- #1471 and #1473: the expired Discord invite is replaced in all four READMEs and on the site.
[1.11.0] β 2026-09-13
56 pull requests since v1.10.2. A ninth model family, five real bugs closed across four engines, and the two platforms the C tests never built on now building them in CI.
A ninth engine: DeepSeek V4.1 Flash
- #1453: DeepSeek V4.1 Flash (552B, 510 GB on disk) runs on a CPU box streaming experts from an SSD, with no conversion: the released checkpoint is read natively, fp8 dense with 32x32 ue8m0 tiles and fp4 experts whose layout is byte-identical to the mxfp4 the Kimi K3 engine already reads. Everything in the architecture that is not V4 is in: Engram (two n-gram memories of 384M rows, 203 GB, that never enter RAM), the DSA indexer with its two-level candidate source, hyper-connections, the compressor, the 32-layer vision tower with its aligner, DSpark speculative decoding (3 stages, blocks of 5, verified in one batched forward with a rollback for the rejected rows) and tool calling in the checkpoint's own DSML format.
- Held token-exact in CI against a torch-only CPU reference
(
c/tools/dsv41_ref.py, written because the vendor's own forward needs tilelang GPU kernels), at three cache capacities, on a short and a 40-token prompt, under all three speculative modes; the vision tower is matched to 5e-06. The CI also asserts that the vendor's index-key republication policy and the tidy reading disagree, so the shipped default cannot be "corrected" by accident (V41_INDEX_OWNER). - Measured on the released checkpoint, cold, caches dropped before every run,
same prompt and seed: a turn went from 78.7 s to 25.1 s during the work
(0.305 to 0.957 tok/s) through batched expert reads (
V41_READ_DEPTH, default 8 is the measured knee), attention matrices read once per block of positions instead of once per token, and an expert-major MoE. Every step is bit-exact against what it replaced. A five-turn chat session runs at 1.14 to 1.58 tok/s; animage_urlpart goes through the gateway into the vision tower on the same engine. - Two ways of hiding the expert reads behind the matmuls were built,
measured worse, and removed; both are written up in
docs/deepseek-v41.mdwith the numbers, so the next attempt starts from them. coli chat,coli serveandcoli webwork;coli runis deliberately unwired for this family, as for qwen36 and qwen38. Per-turn accounting is behindV41_STATS, off by default, so the chat stays clean.
Fixed
- #1390 (@crichalchemist): the qwen36 GPU tier's
qt_fill_waitreturned when the upload queue was empty, but the uploader frees the ring slot at dequeue, before the backend copies anything, so the warmstart freed the RAM int8 copy of experts still in flight. An in-flight count that the uploader decrements only once the slot is resident (#1360). - #1434 (@njloof):
fmt=0is raw f32 and has no scale array, but the Metal sizing helper returned a per-row scale size for it, socoli_metal_matmulwrapped a NULL pointer and segfaulted insidenewBufferWithBytes, or silently produced all-zero output when the size happened to be a page multiple. The kernel no longer applies a scale forfmt=0, and a fail-closed guard sized off the same helper falls back to the CPU path for any format that needs scales and got none. - #1389 (@bherald): GLM-5.3 teardown leaked the lazily allocated dashboard HITS table.
- #1461:
coli cluster workercould never start without an explicit--cap: the default isNone, andstr(None)reached the engine's argument check as the literal"None"(#1452). The worker now resolves the cap the way every other launcher does. - #1460: olmoe's serve-mode
PROFline published five literal zeros, so/profilereportedexpert_disk_s = 0.0on turns where the expert reads were the workload. The disk phase is measured now; the other four fields stay zero because they are unmeasured, not because a guess would look better (#1449, half of it). - #1445: olmoe's
--ramwas inert and "no--cap" meant a constant eight slots per layer regardless of memory; the cap is derived from the RAM budget (#1443). On the reporter's box a 2,291-token prompt went from 675 s to 353 s with no flags (#1442). - #1377: available RAM is measured on every platform (glm53 read
/proc/meminfounconditionally, so on Windows and macOS it read 0 and the expert budget collapsed to 1 GB, which is howcoli tunewent OOM in #1375), and the Windows commit limit is respected. - #1381: the context ceiling is announced, a request that cannot be honoured is refused up front, and the number reported is the right one (#1376).
- #1393: GLM-5.3 replies always start inside the think block (#1278).
- #1414 (@dajiaohuang): the GLM-5.3 serve loop honours
CANCELwhile the turn is still running (#1332). - #1423 (@kreuzzelg): the lazy HITS table is built privately and published under a lock in qwen36, kimi_k3, glm53 and qwen38; the parallel warmstart segfaulted on first touch about one run in twelve (#1422).
- #1404 (@Petsku01): the qwen36 tier offers int8 experts on the decode
path (#1391). #1388 (@crichalchemist):
QT_MAX_ROWSreplaces seven literal 32s that had to agree, and the int8 copy is freed by ownership rather than by format. - #1421 (@bherald):
colino longer overwrites an explicitCUDA_EXPERT_GBwhen it decides to place the dense trunk on the card. - #1244 (@Unknown-Findout): the CUDA expert tier is charged real VRAM, not logical bytes (#687). #1394: the VRAM prefix is priced per row, not at the container's widest (#1351). #1410, #1411: on a single GPU the VRAM prefix is sized from the VRAM budget and the measured headroom, not from the RAM pin plan capped by the LRU reserve (#1409, #1405).
- #1447 (@trigger2k20):
coli mirror verifyaccepts a complete mirror copied by other means, with no receipt, and fails closed on a short or corrupt shard. - #1456 (@trigger2k20): the planner reports
memory.unifiedfrom the host, not from whether the selected engine has a placement-capable GPU, so a CPU-only engine on Apple Silicon no longer reads as non-unified; andcoli tunestops suggestingDRAFT/PIPE/PIN/NUMAto glm53, which ignores them. - #1396 (@dmoraesrs): the web reasoning selector drops "Medium" for GLM 5.3, which the engine renders identically to "High".
- #1428, #1435 (@iiEliJas), #1433 and #1432 (@texasich),
#1440, #1459, #1458 (@trigger2k20), #1463:
setenvvisible togetenvin-process on Windows,malloc_trimguarded to glibc so the engines build on musl, a missing bench recipe restored, the per-test_putenv_shelpers dropped, tests includingcompat.hdirectly, and Clang warnings cleaned up.
Added
- #1462 (@trigger2k20): an opt-in Metal path for GLM-5.3-Flash's routed
experts on Apple Silicon (
COLI_METAL=1): gate, up, clamped SwiGLU, down and the route-weighted scatter on the GPU, CPU path unchanged when Metal is off. Validated against the CPU implementation to 2.8e-09 and measured on an M4 Max: 0.469 to 0.621 tok/s at 32 tokens, up to 2.15 tok/s with the expert cache tuned. Also fixescoli run --cap Nbeing silently ignored on the glm53 one-shot path. - #1457 (@trigger2k20): GLM-5.3-Flash feeds the shared routing telemetry
and writes
.coli_usage, so usage-driven placement and partial-mirror planning have data for it. - #1454 (@yuripourre): the qwen36 GPU tier compiles with
HIP=1. - #1399 (@SebaWag):
TRUNK_RESIDENT_LAYERS=Nstreams the dense trunk through mmap (#826). - #1361, #1374 (@kreuzzelg): the qwen36 dense trunk places itself
(
COLI_PLACE=auto, priced in bytes saved per byte of VRAM), fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity,COLI_GPU. - #1383, #1385, #1386, #1387: Brain and Profile tabs (EMAP, HITS, PROF) on every engine. #1382 (@dmoraesrs): a reasoning depth selector in the web UI (#1311).
- #1407, #1408, #1406: CI builds the portable Windows CUDA DLL and
publishes it as an artifact; the Makefile refuses the 32-bit
cl.exebefore nvcc runs (#1405). - #1436: CI builds against musl (Alpine) and compiles the environment tests on Windows, the two holes that let #1430 and #1420 ship.
- #1360 (@kreuzzelg): tier invariants on the fake backend, and the reservation leak they found. #1419, #1415, #1418 (@dajiaohuang): the V4 and fp8 test harnesses build on Windows.
- #1439 (@bherald): the planner tests no longer materialise four 3 GB zero-filled payloads. #1358 (@monotophic): a raw-evidence adapter for the scoring corpus. #1238 (@ZacharyZcR): reproducible performance records are validated.
Docs
- #1446: the qwen36
--ramsection said the engine does not stream at all. #1392: Azure GLM-5.2 benchmark rows (#1379, #1380, #1384), the rANS cross-reference (#1273), and the int8 probe pinned in CI (#1331). - #1427 (@ZH1995): README.zh-CN formatting. #1438 (@iiEliJas): the
malloc_trimcomment.
[1.10.2] β 2026-09-06
Patch release. Three of these fixes answer reports made against 1.10.1 in the days after it shipped.
Fixed
coli convertpicks the converter from the checkpoint'sconfig.json. It ran GLM-5.2's converter on everything; on GLM-5.3-Flash that quantized the nested embedding and the engine refused the result hours later insidecoli web. An option the target converter does not take is refused, not dropped (#1368, #1369). The converter also refuses a checkpoint it cannot serve, with a per-family pointer (#1305).coli doctorno longer reports two missing core tensors on a healthy GLM-5.3-Flash: it matches roles, prefix-agnostically, instead of GLM-5.2's literal names (#1365, #1366).- The release archive ships every file
colireaches:iq3_pack.py, its grid data file, andtools/convert_glm53.py, which had never been packaged (#1359, #1364). - The RSS guard counts anonymous memory, not reclaimable page cache, so a mapped container no longer evicts experts to free memory it was not using (#1350).
servehonors CANCEL while a turn is still running (#1336), and the disconnect scenario is built rather than hoped for (#1329).- Metal: bit-exact fp8-e4m3 decode (#1346). Qwen3.6: tokenizer merges in both
spellings (#1319). macOS: Homebrew prefixes found when
brewis off the PATH (#1320). DeepSeek V4 on macOS: real CPU and memory in HWINFO (#1308). coli doctoromits the GPU plan for a CPU-only engine (#1322); a missing core tensor explains itself on every fatal path (#1318); GLM-5.3-Flash--no-thinkis the template's lowest effort level, not a shape of ours (#1327).- GLM-5.3-Flash
servecan use the 16 KV slots the engine has; the registry declared 1 (#1283). image_urllocal reads:..is refused, and withCOLI_IMAGE_ROOTset a path must resolve inside it, so an authenticated client of a non-loopback server cannot read arbitrary files through the image API. Error messages no longer confirm a path or its permissions (#1354).
Changed
- The GLM family is named
GLM-5.2/5.3: the two checkpoints share the base model and cannot be told apart from their configuration (#1367). - DeepSeek V4 Flash REAP-150B (85 GB, 132 of 256 experts) loads with the same engine (#1310), is documented, and is announced by its measured geometry rather than the official checkpoint's 284B.
- The qwen36 VRAM tier promotes int8 experts instead of reserving for nothing
(#1334), and the three tier bugs that surfaced with it are fixed: an
is_xoverrun with two or more GPUs, a shutdown that could hang, and a use-after-free on int8 containers (#1339, #1340, #1341, #1344). Kimi K3 stops paying for a DSA indexer nothing reads (#1335). - Opt-in:
COLI_MAP_EXPERTS=1serves experts through a per-shard file mapping (#1325). Off by default; output is byte-identical either way.
Docs
- Windows DeepSeek V4 users are led to the release launcher (
coli.cmd) instead of a source build (#1291).
Build and CI
- CI workflows run with
contents: read, and third-party actions are pinned by commit SHA (#1354). - Makefile lists the headers each engine includes as prerequisites (#1284, #1349). Site and READMEs carry the eight families with real RAM figures under a contract test (#1302).
[1.10.1] β 2026-08-31
Packaging repair for the prebuilt archives; no engine changes.
[1.10.0] β 2026-08-31
A seventh engine: Qwen3.8-Flash-Next
- Added complete text-only inference for the official Qwen3.8-Flash-Next FP8 checkpoint: four-stream Gated Residual, Gated DeltaNet, Qwen Sparse Attention, pageable hashed n-gram embeddings, top-10 routed MoE, and the shared expert.
- The original 131 safetensors shards run directly. Vision and MTP are not loaded or advertised; tools and non-text gateway content are refused.
- Added family/planner/doctor/build/release integration and a pinned upstream tiny oracle covering sparse selection, cached decode, LRU eviction, and sanitizer runs.
- Kept native FP8 expert payloads and normalized scale banks bounded, added cache-sized parallel demand loading, and made prompt MoE execution expert-major with bounded shared-expert and causal DeltaNet batching.
- Added exact single-slot hybrid prompt-prefix reuse across QSA, DeltaNet and PLE state, with persistent state and bounded workspace reflected explicitly in planner RAM accounting.
- Added the boundary-only Qwen3.8 Edge adapter and the seventh-family real Edge -> Segment -> Edge oracle gate without loading transformer, vision or MTP tensors into the Edge process.
[1.7.0] β 2026-08-19
71 pull requests since v1.6.2. A sixth model family with its GPU tier, a rebuilt expert-matmul path, and the CI that would have caught the class of bug we shipped twice.
A sixth engine: Qwen3.6-35B-A3B β CPU and GPU
- #712 (@kreuzzelg) β hybrid Gated Attention + Gated DeltaNet + streaming
MoE, in
c/qwen36.c. Pre-converted containers published (int4-gs64 recommended: cosine to the int8 anchor 0.98777 β 0.99313, KL 0.109 β 0.080 vs per-row). The engine takes any architecture-identical checkpoint unchanged β KAT-Coder v2.5 runs on it with no code path of its own. - #713 (@kreuzzelg) β CUDA VRAM expert tier with heat-based placement
across GPUs: 1.44 β 10.05 tok/s (7.0Γ) on 2Γ 8 GB cards, output
bit-identical to CPU (
cmpover the full 200-token generation), measured cold with no heat table. Aqt_ready()gate keeps CPU-only builds from allocating the packed int4 buffers they never read: 7.33 GB saved.
The expert matmul path, rebuilt (all bit-identical)
- #1071 / #1075 / #1076 / #1077 β activation quantization hoisted to layer
level across GLM, Kimi K3 and DeepSeek V4: the same vector was being
re-quantized ~16Γ per layer, serially. Removes ~5.2 ms/token of serial time
and every per-call
mallocfrom the hot path. - #1079 / #1086 β K1: plane-nibble int4 layout + unsigned-VNNI dot.
Storing element k and k+32 in one byte deletes the unpack, and since
nibbles are stored unsigned,
dot(v,x) = dot(u,x) β 8Β·Ξ£xfeedsvpdpbusdnatively. 8 uops per 64 MACs instead of 32: 1.45β2.65Γ on the IDOT kernels, zero bytes added. - #1088 β K2: 1Γ4 union tile. The prefill union hands each expert 2β16 rows; the weight block's load+mask is now paid once per four rows instead of once per row: 2.67β3.10Γ at S=4 (peak 253 GMAC/s).
- #1093 β parallel silu and the down-side activation hoist.
- #1094 β K1b: grouped planar IDOT for gs64 containers (
IDOT_GS=1, opt-in): the recommended container format could not reach the integer kernels at any batch size before this. - #1082 (@outtodata) β
FUSED3=1opt-in fused AVX2 expert matmul.
Streaming and I/O
- #1097 β DeepSeek V4 expert-loader pool default 3 β 9 lanes:
1.41Γ decode on the real V4-Flash checkpoint (8 interleaved runs on a
quiet 25 GB box).
V4_LOADER_LANESstill overrides. - #1056 (@dcutugno) β DeepGEMM sm120 headers fetched at a pinned commit on first build: 2.5Γ prefill on sm120 with nothing vendored in-tree.
- #988 / #1054 / #1055 β DeepSeek V4 CUDA tier and dual-SSD mirror.
Correctness and CI
- #1083 β ARM CI job (
ubuntu-24.04-arm) plus an integer-kernel bit-exactness gate that runs on both ISAs. Every tiny-oracle job ran on x86 before this, so NEON-divergent paths were invisible β which is how the IDOT defaults below shipped. Closes #1081. - #1044 / #1080 β IDOT made opt-in in olmoe and inkling: the fast path is x86-only and quantizes activations, so the same model produced different tokens on x86 and ARM by default.
- #1109 (@SebaWag) β ARM64 dotprod probed by compiling the intrinsic:
GCC 11 defines
__ARM_FEATURE_DOTPRODfor a base it cannot emitvdotq_s32for. Fixes #1104. - #1111 β
__syncwarp()aftergrouped_s4_wmma's store (reported by @monotophic withcompute-sanitizerevidence). Fixes #1099. - #1073 / #1074 (@bherald) β Kimi K3 cancels prefill between layers instead of holding the engine for a minutes-long prompt; cancelled requests no longer count as completed.
Apple Silicon
- #790 β #1113 (@RDouglasSharp) β Metal backend for Kimi K3: KDA state and window buffers aligned, wrap-once buffer cache, CPU-side MLA KV cache. 1.7Γ/2.4Γ on the compute-bound phases (KDA attention + projections dispatched to the GPU); MoE experts stay on the CPU. Kimi K3's first GPU backend.
More correctness fixes
- #1098 (@monotophic) β
__syncthreads()missing from the absorb softmax reduction, with a determinism test that reproduces the hazard. - #1101 (@monotophic) β allocation and
snprintfresults checked on the checkpoint-load path (#798). - #1100 / #1108 (@monotophic) β fmt=8/fmt=6 scale-byte accounting in
tensor_bytes/tensor_free, andweights_ownedset before the host-to-device copy so a failed upload frees its buffer. Each ships with its own regression test; all four of this contributor's CUDA fixes landed in this release. - #1122 (@ZacharyZcR) β
USAGE_SAVE=0honoured in every engine (#1039): the history was loaded but written back anyway, which quietly contaminated any A/B that shared a usage file between arms. - #1121 (@ZacharyZcR) β LRU victim selection now respects a lowered
ecap(#1034): after an RSS-guard reduction the cache kept evicting against the old capacity. - #1123 (@ZacharyZcR) β the v1.6.2 warning-cleanup patches landed (#1032).
- #1106 (@monotophic) β duplicate tensor names across indexed shards are now refused rather than silently resolved to one of them (untrusted containers).
Interfaces
- #829 (@aaristov) β GPU-vs-fallback counters and chat status made visible in
coli serve: the tier's behaviour is now observable instead of inferred. - #1095 (@benmaster82) β OLMoE planner geometry adapter, and #1103
(@SebaWag) β Kimi K3, Inkling and DeepSeek V4 adapters with 23 tests: every
family now has real planner geometry, so
coli planstops guessing (#1066). - #1096 (@terrizoaguimor) β DeepSeek V4 serve framing on the shared codec, completing the codec migration across OLMoE, Kimi K3 and V4.
- #1063 / #1068 (@terrizoaguimor) β model families are registry-owned:
coli, the gateway,doctorand the planner read one descriptor table. - #1087 / #1090 / #1096 / #1116 (@terrizoaguimor) β a shared serve framing codec, now adopted by every engine: OLMoE, Kimi K3, DeepSeek V4 and Inkling (whose audio payload rides as an opaque extension). Each migration landed behind a byte-exact wire-transcript freeze, so the gateway contract is provably unchanged. Byte framing had been duplicated five times, which is how Windows binary mode silently disappeared from sibling engines (#748).
- #1036 (@lineape) β distributed expert workers (LAN, opt-in via
CLUSTER_WORKERS). - Planner: DeepSeek V4 expert naming now recognized, so
coli planandcoli doctorstop counting every routed expert as dense (fixes #1110).
[1.6.2] β 2026-08-14
Security release: six privately-reported memory-safety issues fixed, all reachable
from attacker-controlled input (malicious model file / config.json, or the kimi_k3
SERVE stdin). Every fix validates at the trust boundary β no behavioural change on
well-formed models or requests. Advisories: GHSA-gf38-c8fx-ppvv (kimi_k3 SERVE OOB
write), GHSA-2qrj-xjmh-mv74 (json.h OOB read), GHSA-w696-h9p7-6rgc (inkling audio
OOB), GHSA-7654-r78q-vc3r (deepseek_v4 indexer OOB R/W), and two more in the same
class. See the GitHub Security Advisories for details.
[1.6.1] β 2026-08-13
--allowed-host '*'lets an operator reach a public bind deliberately (#990, #993)- OLMoE streaming no longer drops the answer into the reasoning channel (#984, #985)
- chat reasoning-channel fixes and pty test hardening (#980)
[1.6.0] β 2026-08-12
If you are on v1.5.0, update: it shipped a performance regression on GLM-5.2 (#856), left up to ~60 GB of RAM unused with a 13-point expert hit-rate loss (#885), and broke Kimi K3 outright on some machines (#888).
- #869 β the planner priced every expert row at the container's widest width; mixed-width containers had their cache silently halved
- #914 (bherald) β pin budgets accounted correctly
- prefill batch-union: each distinct expert is read once per chunk
[1.5.0] β 2026-08-05
51 pull requests from 15 contributors.
- Fifth engine: DeepSeek V4 Flash (@DrewZt, #165) β MLA + DSA sparse attention, 43 layers, 256 routed experts + 1 shared, top-6; official checkpoint streams with no conversion (fp4 experts, fp8-e4m3 dense with UE8M0 block scales)
- #839 β the rows16 fp4 fast path no longer requires AVX-512: consumer Intel/AMD CPUs since Alder Lake get the fast path
[1.4.0] β 2026-08-01
162 commits, 29 pull requests (25 from contributors).
- Release archives now contain every engine β v1.3.0 archives shipped
c/colibrialone while the README promised four families (#720);inklingandkimi_k3are built, packaged and smoke-tested per platform - Third GPU backend lands
[1.3.0] β 2026-07-29
Three MoE families on one engine, 744B β 2.8T.
- Kimi K3 (#676) β 2.8T/104B active: KDA + gated-NoPE-MLA + AttnRes + LatentMoE, streams Moonshot's QAT MXFP4 experts straight from the original HF shards
- Inkling β a 975B model answers on a 25 GB machine
[1.2.0] β 2026-07-28
- GB10 / DGX Spark unified-memory OOM fix (#653); AMD/ROCm recognized by
doctorandresource_plan(#662, #663) - AVX2
matmul_e8β fmt=6 was 92% of decode on the scalar kernel (#654); native SIMD fmt=6 encoder, 15Γ over numpy - chat stop-set fix (#633/#381), async packed-int4 parity (#632), stable KV slot per conversation (#634, #639)
[1.1.1] β 2026-07-23
A same-day patch release. Windows users on v1.1.0 should upgrade: Microsoft Defender flags the v1.1.0 Windows binary, and the cause was ours.
Fixed
- 107 KB of zeros were shipped inside every binary (#527, #532) β and that is
what antivirus ML heuristics were reacting to.
static GrDraft g_grd={.max=24};looks harmless, butGrDraftis ~107 KB (the grammar's 1024 static rules plus the PDA walker) and any initializer moves the whole struct out of.bssand into.data, writing 106,848 bytes of near-zero-entropy data into the file β in a writable section, which is the classic shape of an unpacking buffer for a packed payload. Section forensics against v1.0.0 (clean on the same Defender definitions) isolated it: identical toolchain, identical PE layout,.data1,840 β 108,752 bytes. A Windows build with the fix scans clean where v1.1.0 does not. Every platform's binary also gets smaller: the Linux engine drops 474,904 β 368,016 bytes, -22.5%. python3 openai_server.pywas broken on a clean checkout (#526) β the gateway still looked for an engine namedglmafter the #391 rename. It resolvescolibri/colibri.exefirst now, falling back toglmfor older trees.coli servewas unaffected. Spotted by @RDouglasSharp while debugging #488.
Added
- Anthropic Messages API on
/v1/messages(#343, #525) β clients that only speak to Anthropic endpoints, Claude Code above all, now work against colibri with no shim: same port, nothing to enable. It is a translation layer over the same generation path, so tools, streaming and the KV cache behave exactly as on/v1/chat/completions. Covers system prompts,text/tool_use/tool_resultblocks,input_schematools, everytool_choicemode, the full named-event SSE sequence with protocolpingkeepalives,stop_reasonmapping, extended thinking, andx-api-keyauth (Bearerstill works).stop_sequences,top_kand non-text blocks are refused explicitly rather than silently ignored. SHA256SUMS.txtpublished with every release (#530) β verify a download is exactly what CI built from the tagged source.- The Windows engine is uploaded as a CI artifact (#532) β an antivirus report can now be verified on a pull request instead of only after a release is published.
Changed
- Docs: "Get started" now starts by getting the program (#521). The README told
newcomers to download the 372 GB model before it told them how to obtain colibri β
and for Linux/macOS it never told them at all. New order: get colibri (prebuilt
archive or build from source) β get the model (372 GB stated up front) β run it.
The obsolete "rename the engine to
glm.exe" step is gone; archives have shipped a plainly-namedcolibri.exesince #508. Applied in all four languages.
[1.1.0] β 2026-07-22
A community release. 27 pull requests from more than 20 contributors, 216 commits since v1.0.0. Most of what follows was found, measured, or fixed by people who do not work on this project and had nothing to gain from it.
Added
- AMD GPU support (HIP/ROCm) (#339) β single-source
backend_gpu_compat.hwith a WMMA dispatch gate, so one codebase builds for CUDA and HIP. Validated on an RX 9070 XT (RDNA4, ROCm 7.2): token-exact against CPU on a real fmt=4 gs64 container, with resident dense and with routed experts in VRAM, plus a fail-injection control proving the GPU actually executed the work. - Dual-SSD streaming (
COLI_MODEL_MIRROR, #421) β read the model from two drives at once, roughly doubling streaming bandwidth on a disk-bound host. - N-drive shard split (
COLI_MODEL_DIRS, #469) β capacity aggregation: run a container no single drive can hold, spread across several with no duplication. - fmt=5 (int3-g64) (#168) β 3-bit weights with per-64 group scales: measured 3.3x lower outlier-row error than per-row int4 at 25% fewer bytes.
- fmt=6 (E8/IQ3 lattice) (#465) β CPU decode kernel and dispatch; index codec tooling (#458).
tools/try_tool_calling.pyβ dependency-free two-turn tool-calling probe that doubles as a smoke test.
Fixed
- Tool calling in coding clients (#401), root cause found, in two parts:
- #506 β the engine capped prompt encoding at
CTX-2, and the tokenizer stops dead at its limit without reporting anything. A prompt longer than the context was therefore silently truncated to its firstCTX-2tokens and answered anyway. With the 4096 default that is 4094 β exactly theprefill 4094in the field report. The dropped tail was the tool instructions and the user's actual turn, so the model emitted a bare<and stopped; and because clients append to the end while truncation keeps the head, every retry re-sent a byte-identical prompt. Now refused with a 400context_length_exceeded. - #505 β a tool call whose closing
</tool_call>never arrived was dropped whole, because the parser required both tags. Now recovered when unambiguous, on both the streamed and non-streamed paths. - #437 β non-EOS role markers were armed as hard stops in serve mode and cut generation the instant a tool block started.
- #506 β the engine capped prompt encoding at
- Grouped-int4 (fmt=4) produced garbage output on CUDA with
CUDA_DENSE=1(#298) β the dense and attention kernels applied per-group scales as if they were per-row. Hardware-verified. - OpenMP tuning re-exec preserved the CPU affinity mask (#476), jailing every thread onto
one core when
OMP_PROC_BIND/OMP_PLACESwere set: roughly a 20x slowdown. - Pilot eviction guard dropped ~100% of speculations once the cache filled (#497),
collapsing
PILOT_REALto a hint-only path. - Silent budget clamp capped the CUDA expert tier at ~109 experts regardless of
CUDA_EXPERT_GB(#495). - fmt=4 guard at the per-row-only CUDA entry points (#464/#470);
COLI_CUDA_MTP=1andCOLI_CUDA=0are now honoured over implicit defaults (#468).
Security
Threat model: model files come from mirrors that are not trusted.
- #368 β server hardening, JSON and tokenizer parser hardening, build flags, downloader and dependency pinning.
- #413 β the quant layout is resolved and validated against the on-disk byte counts
(unknown layouts are refused rather than falling through to int2), shape-product overflow
is rejected, and the olmoe dtype-3 path no longer trusts a crafted
nbytes(heap overflow).
Performance β all byte-identical
- #481 +4.7x on the MLA-absorb score and value-mix reductions
- #477 +13% decode on AVX-512 (
qt_addrow/qt_matvec_rows) - #475 +11.6% with opt-in
XEXP=1(one OpenMP region per expert block at S=1 full residency) - #473 +5.5% int4 IDOT at S=1 on AVX-512 VNNI
Changed
glm.cis nowcolibri.cplus header modules (#391);make glmremains as an alias.- Serve stage 2 (#192):
response_format, per-request grammars, grammar-forced drafts.
Upgrade notes
CTXstill defaults to 4096. Coding clients send far more than that in a single system prompt. UseCTX=32768. Before this release an over-long prompt was silently truncated; now you get a clear 400 instead.
[1.0.0] β 2026-07-19
First tagged release. The engine has been running in production since late June 2026; this tag marks the baseline for semantic versioning going forward.
Highlights
- GLM-5.2 (744B MoE) runs on ~25 GB RAM in pure C, streaming experts from disk
- Three-tier placement: VRAM (hot) / RAM (warm) / NVMe (cold), with a learning cache that pins your workload's hottest experts automatically
- CUDA backend: multi-GPU expert tier, dense tensor distribution, batched
ragged attention, resident pipeline (
COLI_CUDA_PIPE=2) - Metal backend (Apple Silicon): batched expert SwiGLU + fused decode attention on unified memory GPU
- MTP speculation: native GLM-5.2 draft heads, grammar-forced drafts, kernel-
pinned verification (
SPEC_PIN=1) - OpenAI-compatible API:
coli servewith SSE streaming, KV slots, bounded queue, web dashboard (coli web) - Web UI: chat with live metrics, expert cortex brain page, profiling breakdown, expert atlas 3-D galaxy
- Cross-platform: Linux, macOS, Windows 11 (native MinGW), PowerPC; CI on all three
- Auto-tune:
coli plan --auto-tierclassifies the bottleneck and derives MTP/PIPE/NUMA/PIN settings with explanations
Engine
- Token-exact validation against
transformersoracle (teacher-forcing 32/32) - Compressed MLA KV cache (576 floats/token, 57Γ smaller), persisted across
restarts (
.coli_kv, zero re-prefill) - DSA sparse attention (lightning indexer), faithfully implemented
- Router-lookahead prefetch (
PILOT=1, 71.6% predictive) - Async expert I/O pool (
PIPE=1), io_uring batching (URING=1) - NUMA-aware expert placement (
COLI_NUMA=1, +13β40% on multi-socket) - AVX2 / AVX-512 / AVX-VNNI / ARM NEON / NEON-i8mm / POWER VSX kernels
- int4 / int8 / int2 / grouped-int4 (fmt=4) quantization formats
Tools
coli convertβ FP8βint4 one-shard-at-a-time convertercoli doctorβ read-only setup diagnosticscoli planβ resource planner with auto-tune prescriptioncoli benchβ MMLU / HellaSwag / ARC quality benchmarks- Expert atlas (
tools/analyze.py --web) β measured topic affinity for 19,456 experts
Community
- 30+ hardware datapoints in the benchmark tracker
- Contributions from 20+ authors across engine, docs, tooling, and ports