Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
CHANGELOG 1.12.1: the eighteen pull requests merged today
JustVugg31b090586299
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3b140188..f99ea0b7 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,7 +5,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).## [1.12.1] — 2026-09-22-34 pull requests since v1.12.0, 23 of them from contributors. Two tokenizers+52 pull requests since v1.12.0, 41 of them from contributors. Two tokenizersbrought back to the reference, brio on the ninth engine, `coli chat` workingagain at the default context on two families, and a placement decision thatis now measured on the card in front of it instead of predicted.@@ -108,6 +108,27 @@ is now measured on the card in front of it instead of predicted.- **#906**: `DEGRADE_ZERO`, an opt-in policy that zero-fills a missedexpert slot below a gate-weight threshold instead of blocking on theload (#865).+- **#1677**: qwen36 projects a prompt's DeltaNet inputs (qkv and z) on the+ card in blocks of up to 256 rows instead of one row at a time; a 259-row+ prefill makes 2 projection calls instead of 259, with the convolution+ history and the recurrent state checked against the CPU run. The paired+ microbenchmark on an RTX 4070 read 4 to 10x per projection.+- **#1674**: qwen36's attention projections the tier placed in VRAM answer+ a whole prompt batch with one call per matrix; a failed call turns only+ that handle off and the prompt continues on the CPU.+- **#1676**: Kimi K3's streaming CUDA expert keeps the gate, up and SiTU+ intermediates on the device and applies down there (one fused entry+ point, optional in the DLL: an older backend keeps the three-call path).+- **#1673**: the streaming MXFP4 matmul reuses one grow-only device scratch+ per card instead of allocating and freeing weights and scales on every+ call.+- **#1686** (DebugSultan): qwen38's prefill chunk (`Q38_PREFILL_BATCH_ROWS`)+ and workspace (`Q38_PREFILL_WORKSPACE_MIB`) are runtime knobs, the expert+ load batch is no longer capped at top-k, and the QSA ranking and+ attention run per position in parallel at prefill. Measured on the+ released checkpoint on top of the int8 trunk: output byte-identical at+ every chunk width, no speed change on our 16-core server; the knobs are+ there for hardware where the chunk binds.### Fixed@@ -134,6 +155,35 @@ is now measured on the card in front of it instead of predicted.- **#1658**: `test_mem_available` compared two reads of available memorywith `==` and failed on a busy Windows runner; a quarter of a GB oftolerance.+- **#1670**: DeepSeek V4.1 read only its argv cache cap, so `RAM_GB=120`+ on a 128 GB box left the engine at eight expert slots per layer and+ 23.8 GB of RSS (#1666). With `--cap` omitted, `coli chat`, `coli serve`+ and `coli web` now size the cache from the resource plan, with `RAM_GB`+ or `--ram` as the budget; an explicit `--cap`, a measured profile and an+ auto-tier plan keep precedence.+- **#1671**: DeepSeek V4.1 treats `max_tokens` as a ceiling like the other+ engines (#1641): a fitting prompt with a large request generates what+ the context leaves, a score-only prompt may fill the context, and a+ prompt one token over it is refused instead of silently truncated.+- **#1675**: resident MXFP4 tensors on CUDA carried O float scales where+ the kernel reads O x ceil(I/32) exponent bytes: short buffers were+ over-read and long ones truncated. One format-aware size for upload,+ refresh, accounting and release.+- **#1678**, **#1679**, **#1680**, **#1682**, **#1683**, **#1684**: the+ Qwen CUDA tier's lifecycle, end to end. Shutdown releases every resident+ expert, projection handle and host table after parked callers resume;+ a failed gate, up or down upload frees what it had already allocated;+ the expert budget charges the three scale buffers at their own sizes+ (two experts used to be admitted where one fit); a failed result+ collection stops inference instead of publishing a partial MoE sum; a+ failed or explicitly disabled tier start unwinds its storage and+ synchronization objects, and a second init cannot overwrite a running+ tier; the CUDA backend validates the whole device list before touching+ state and keeps live contexts on a repeated init. Fault-injected on the+ fake backend, then run together on an RTX 4070 under compute-sanitizer+ with zero errors and zero bytes leaked.+- **#1669**: `test_systemone_api` imports its scoring engine relative to+ its package, so an installed `tests` package no longer breaks discovery.### Tools and the gateway@@ -146,6 +196,18 @@ is now measured on the card in front of it instead of predicted.- **#1355**, **#1357**: durable per-request results and strict stdoutclassification in the eval harness, and the logprob-gap check gated onthe engine preamble.+- **#1687**: `GET /metrics` in Prometheus text format, behind the API key:+ four gauges, six outcome counters and four histograms (queue wait, slot+ occupancy, first output, engine call), no request labels, no new+ dependency. The admission scheduler distinguishes completion, failure+ and cancellation, lets a request use a free slot that no earlier waiter+ reserved, and joins the keepalive pump before the slot is released.+- **#1688**: `tools/benchmark_http_serving.py`, a stdlib HTTP streaming+ benchmark over fixed JSONL conversations: closed-loop or paced arrivals+ (periodic or Poisson, seeded), warmup separated from measurement,+ first-output and duration SLOs, latency percentiles and usage-based+ token throughput; a truncated or malformed stream is a failure, not a+ sample.### Docs@@ -160,6 +222,9 @@ is now measured on the card in front of it instead of predicted.- **#1617**: connecting the pi coding agent to `coli serve`.- **#1619**: `expected_bytes` identity versus physical extent forint4-rans256-g0 (#1273).+- **#1681** (XBold): the README and `docs/qwen38.md` no longer say Qwen3.8+ has no GPU backend; the CUDA VRAM expert tier and the int8 trunk in VRAM+ shipped in 1.12.0.## [1.12.0] — 2026-09-20