Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

CHANGELOG 1.12.1: the eighteen pull requests merged today

JustVugg31b090586299
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3b140188..f99ea0b7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,7 +5,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).
## [1.12.1] — 2026-09-22
-34 pull requests since v1.12.0, 23 of them from contributors. Two tokenizers
+52 pull requests since v1.12.0, 41 of them from contributors. Two tokenizers
brought back to the reference, brio on the ninth engine, `coli chat` working
again at the default context on two families, and a placement decision that
is now measured on the card in front of it instead of predicted.
@@ -108,6 +108,27 @@ is now measured on the card in front of it instead of predicted.
- **#906**: `DEGRADE_ZERO`, an opt-in policy that zero-fills a missed
expert slot below a gate-weight threshold instead of blocking on the
load (#865).
+- **#1677**: qwen36 projects a prompt's DeltaNet inputs (qkv and z) on the
+ card in blocks of up to 256 rows instead of one row at a time; a 259-row
+ prefill makes 2 projection calls instead of 259, with the convolution
+ history and the recurrent state checked against the CPU run. The paired
+ microbenchmark on an RTX 4070 read 4 to 10x per projection.
+- **#1674**: qwen36's attention projections the tier placed in VRAM answer
+ a whole prompt batch with one call per matrix; a failed call turns only
+ that handle off and the prompt continues on the CPU.
+- **#1676**: Kimi K3's streaming CUDA expert keeps the gate, up and SiTU
+ intermediates on the device and applies down there (one fused entry
+ point, optional in the DLL: an older backend keeps the three-call path).
+- **#1673**: the streaming MXFP4 matmul reuses one grow-only device scratch
+ per card instead of allocating and freeing weights and scales on every
+ call.
+- **#1686** (DebugSultan): qwen38's prefill chunk (`Q38_PREFILL_BATCH_ROWS`)
+ and workspace (`Q38_PREFILL_WORKSPACE_MIB`) are runtime knobs, the expert
+ load batch is no longer capped at top-k, and the QSA ranking and
+ attention run per position in parallel at prefill. Measured on the
+ released checkpoint on top of the int8 trunk: output byte-identical at
+ every chunk width, no speed change on our 16-core server; the knobs are
+ there for hardware where the chunk binds.
### Fixed
@@ -134,6 +155,35 @@ is now measured on the card in front of it instead of predicted.
- **#1658**: `test_mem_available` compared two reads of available memory
with `==` and failed on a busy Windows runner; a quarter of a GB of
tolerance.
+- **#1670**: DeepSeek V4.1 read only its argv cache cap, so `RAM_GB=120`
+ on a 128 GB box left the engine at eight expert slots per layer and
+ 23.8 GB of RSS (#1666). With `--cap` omitted, `coli chat`, `coli serve`
+ and `coli web` now size the cache from the resource plan, with `RAM_GB`
+ or `--ram` as the budget; an explicit `--cap`, a measured profile and an
+ auto-tier plan keep precedence.
+- **#1671**: DeepSeek V4.1 treats `max_tokens` as a ceiling like the other
+ engines (#1641): a fitting prompt with a large request generates what
+ the context leaves, a score-only prompt may fill the context, and a
+ prompt one token over it is refused instead of silently truncated.
+- **#1675**: resident MXFP4 tensors on CUDA carried O float scales where
+ the kernel reads O x ceil(I/32) exponent bytes: short buffers were
+ over-read and long ones truncated. One format-aware size for upload,
+ refresh, accounting and release.
+- **#1678**, **#1679**, **#1680**, **#1682**, **#1683**, **#1684**: the
+ Qwen CUDA tier's lifecycle, end to end. Shutdown releases every resident
+ expert, projection handle and host table after parked callers resume;
+ a failed gate, up or down upload frees what it had already allocated;
+ the expert budget charges the three scale buffers at their own sizes
+ (two experts used to be admitted where one fit); a failed result
+ collection stops inference instead of publishing a partial MoE sum; a
+ failed or explicitly disabled tier start unwinds its storage and
+ synchronization objects, and a second init cannot overwrite a running
+ tier; the CUDA backend validates the whole device list before touching
+ state and keeps live contexts on a repeated init. Fault-injected on the
+ fake backend, then run together on an RTX 4070 under compute-sanitizer
+ with zero errors and zero bytes leaked.
+- **#1669**: `test_systemone_api` imports its scoring engine relative to
+ its package, so an installed `tests` package no longer breaks discovery.
### Tools and the gateway
@@ -146,6 +196,18 @@ is now measured on the card in front of it instead of predicted.
- **#1355**, **#1357**: durable per-request results and strict stdout
classification in the eval harness, and the logprob-gap check gated on
the engine preamble.
+- **#1687**: `GET /metrics` in Prometheus text format, behind the API key:
+ four gauges, six outcome counters and four histograms (queue wait, slot
+ occupancy, first output, engine call), no request labels, no new
+ dependency. The admission scheduler distinguishes completion, failure
+ and cancellation, lets a request use a free slot that no earlier waiter
+ reserved, and joins the keepalive pump before the slot is released.
+- **#1688**: `tools/benchmark_http_serving.py`, a stdlib HTTP streaming
+ benchmark over fixed JSONL conversations: closed-loop or paced arrivals
+ (periodic or Poisson, seeded), warmup separated from measurement,
+ first-output and duration SLOs, latency percentiles and usage-based
+ token throughput; a truncated or malformed stream is a failure, not a
+ sample.
### Docs
@@ -160,6 +222,9 @@ is now measured on the card in front of it instead of predicted.
- **#1617**: connecting the pi coding agent to `coli serve`.
- **#1619**: `expected_bytes` identity versus physical extent for
int4-rans256-g0 (#1273).
+- **#1681** (XBold): the README and `docs/qwen38.md` no longer say Qwen3.8
+ has no GPU backend; the CUDA VRAM expert tier and the int8 trunk in VRAM
+ shipped in 1.12.0.
## [1.12.0] — 2026-09-20