Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: qwen38 int8 trunk and vector FP8 kernel, measured; 1.12.1 notes for #1650 and #1668
JustVuggcde709c4a634
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 62ddb50d..3b140188 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,7 +5,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).## [1.12.1] β 2026-09-22-32 pull requests since v1.12.0, 22 of them from contributors. Two tokenizers+34 pull requests since v1.12.0, 23 of them from contributors. Two tokenizersbrought back to the reference, brio on the ninth engine, `coli chat` workingagain at the default context on two families, and a placement decision thatis now measured on the card in front of it instead of predicted.@@ -68,7 +68,7 @@ is now measured on the card in front of it instead of predicted.### Performance-- **#PR**: qwen36's dense trunk and routed experts multiply with integer+- **#1664**: qwen36's dense trunk and routed experts multiply with integerdot products. The activation is quantized to int8 once per call and theweights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusdinstead of a float conversion per weight; the integer kernels move from@@ -80,6 +80,18 @@ is now measured on the card in front of it instead of predicted.`COLI_DENSE_INT4=<components>` stores part of the trunk as int4 in blocksof 64: opt-in, with the perplexity it costs per component in the docs(lm_head alone +2.4%, everything +10%).+- **#1668**: qwen38's dense trunk (553 matrices, 3.6 G weights, 8 GiB of+ BF16 read on every token, more than the ten routed experts) is kept on the+ CPU as int8 rows with the BF16 copy released, and multiplied with the same+ integer kernels; the routed experts' e4m3 blocks are decoded eight at a+ time in registers and multiplied with FMA instead of one table lookup per+ weight. Measured on the released Qwen3.8-Flash-Next-FP8, 8 threads, RAM+ LRU 96 per layer: decode 0.61 to 1.42 tok/s, the trunk 434 to 85 ms/token,+ lm_head 76 to 13, the expert GEMVs 388 to 140, peak RSS 32.2 to 28.5 GB;+ prefill of 512 tokens 495 to 149 s. Perplexity on 4 x 512 tokens +0.5%+ (two chunks lower, two higher); the vector FP8 kernel alone reproduces the+ BF16 run to four decimals. Both are the default (`Q38_TRUNK_CPU_INT8=0`+ keeps the BF16 trunk, `Q38_FP8_KERNEL=scalar` the table kernel).### Performance, from contributors@@ -99,6 +111,12 @@ is now measured on the card in front of it instead of predicted.### Fixed+- **#1650** (bokiko): a Qwen3.8 pin snapshots the recurrent and PLE state+ but reuses the live attention and indexer rows; after an unrelated prompt+ overwrote those rows, returning to the pin could change brio logprobs+ without a warning. The engine now records the token identity of the live+ rows (`kv_prefix.h`) and refuses a stale pin or prefix restore; image rows+ are tainted. Wire regressions run on the BF16 and FP8 fixtures.- **#1626**: `SNAP` is the model directory for every non-GLM engine, so`coli run` stops handing them a leftover environment (#1600).- **#1604**: glm53 honours `Mat.resident` in the Vulkan gate and frees thediff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 91969d00..04063bd7 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -16,7 +16,7 @@ what follows, but the sister engines read their own:| `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family β see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) || `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU` β see [Inkling engine](#inkling-engine-inkling) || `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, and the `CACHE_ROUTE` family (VRAM tier over RAM cache) β see [Qwen3.6 engine](#qwen36-engine-qwen36) |-| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_PREFILL_BATCH`, `COLI_TIMERS` β see [Qwen3.8 engine](#qwen38-engine-qwen38) |+| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS` β see [Qwen3.8 engine](#qwen38-engine-qwen38) || `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` β see [OLMoE engine](#olmoe-engine-olmoe) || `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates β see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read β its GPU switch is `DSV4_CUDA` |@@ -412,6 +412,8 @@ checkpoint layout and the text-only capability boundary.| `Q38_NATIVE_FP8` | `1` (on) | Keep routed E4M3 expert bytes and their F32 128Γ128 block scales native in the LRU. `=0` restores expanded-FP32 slots for A/B validation. || `Q38_NATIVE_BF16` | `1` (on) | Keep resident and routed BF16 matrices in two-byte storage while retaining FP32 activations/accumulation. `=0` restores the expanded-FP32 reference. || `Q38_PREFILL_BATCH` | `1` (on) | Route prompt rows in bounded expert-major chunks and batch resident shared-expert/DeltaNet projections. `=0` restores row-at-a-time prompt execution for A/B diagnosis; decode is unchanged. |+| `Q38_TRUNK_CPU_INT8` | `1` (on) | The dense trunk (DeltaNet and attention projections, hyper-connection mixers, shared expert, router, lm_head; every matrix of at least `Q38_TRUNK_MIN_KB`) is kept on the CPU as int8 rows with one scale per row and the BF16 copy is released; `q38_weight_matmul` quantizes the activation to int8 and uses the integer kernels of `idot.h` for decode and prefill. `=0` keeps the BF16 rows and the f32 kernel (the numeric reference). See [qwen38.md](qwen38.md#the-trunk-on-the-cpu-int8-rows). |+| `Q38_FP8_KERNEL` | vector | The routed experts' e4m3 blocks are decoded eight at a time in registers and multiplied with FMA (AVX2 builds); `scalar` restores `quant.h`'s table kernel, which differs only by float summation order inside a block. || `COLI_TIMERS` | `0` (off) | Set to `1` for the detailed Qwen3.8 phase breakdown on stderr. The shared per-request `PROF` frame is emitted regardless. |## DeepSeek V4 engine (`deepseek_v4`)diff --git a/docs/qwen38.md b/docs/qwen38.mdindex fd4dccc7..79b90341 100644--- a/docs/qwen38.md+++ b/docs/qwen38.md@@ -249,8 +249,8 @@ whatever the placer accepts is quantized to **int8 per row** (scale = max|w|/ 127, the qwen36 dnproj/lmhead format) when the tier starts -- about 2 sfor the 553 matrices, 3.96 GiB on one card -- and answers decode GEMVs fromthere, one round trip per matmul (`x` up, `y` down; activations stay on the-CPU). Prefill rows (S > 1) and any backend failure take the BF16 CPU path,-which stays in RAM. The placer takes the trunk before the experts (it is+CPU). Prefill rows (S > 1) and any backend failure take the CPU path, which+holds the same int8 rows (see [the trunk on the CPU](#the-trunk-on-the-cpu-int8-rows)). The placer takes the trunk before the experts (it isread on every token), so on an 8 GB card about 2 GB remain for hot experts;`coli plan` prices it the same way (`4.0 GB int8 trunk + ... hot tier`).@@ -260,7 +260,7 @@ read on every token), so on an 8 GB card about 2 GB remain for hot experts;| `Q38_TRUNK_MIN_KB=<n>` | offer matrices of at least n KiB (default 1024; a round trip costs more than a tiny GEMV saves) || `Q38_TRUNK_SKIP=name,name` | leave the named components on the CPU (bisecting, or a component that does not pay) || `QT_UPLOAD_SYNC=1` | the tier's `qt_issue` waits for every in-flight upload first (tests and diagnostics: deterministic residency, no upload/compute overlap) |-| `Q38_TRUNK_CPU_INT8=1` | the same int8 rows on the CPU instead -- what the quantization alone does to the output, no GPU needed (perplexity, token parity); a matrix the GPU holds is still answered from VRAM |+| `Q38_TRUNK_CPU_INT8=0` | keep the trunk BF16 on the CPU with the f32 kernel (the numeric reference); the default is int8 rows with integer dot products, GPU or not || `Q38_TRUNK_SELFTEST=1` | at start, every placed matrix is checked once: GPU GEMV against the same int8 rows on the CPU (relative error printed per matrix) |**Numerics.** GPU int8 against CPU int8 on the same rows: relative error@@ -312,6 +312,75 @@ error, only wrong tokens, worse the more experts were resident. The densepath has its own buffers now ([qwen36-cuda-tier.md](qwen36-cuda-tier.md));qwen36 never called the dense path inside that window.+### The trunk on the CPU: int8 rows++Without a GPU the same trunk is the decode's floor: 8 GiB of BF16 read on+every token, multiplied by a scalar loop. Since 1.12.1 the engine keeps the+trunk on the CPU as **int8 rows with one scale per row** (the format the+tier uploads, so GPU or not the rows are the same bytes), releases the BF16+copy once the rows exist, and multiplies with the integer kernels of+`idot.h`: the activation is quantized to int8 once per row and meets the+weights with maddubs / vpdpbusd (AVX2 / AVX-512 VNNI) or NEON dot products.+Decode and prefill take this path. Every matrix of at least+`Q38_TRUNK_MIN_KB` (default 1 MiB) is in; the small ones stay BF16 because+they cost nothing either way. `Q38_TRUNK_CPU_INT8=0` keeps the BF16 rows and+the f32 kernel, the numeric reference.++The routed experts stay e4m3 with their 128 x 128 block scales, but the+kernel decodes eight bytes at a time in registers and multiplies with FMA+(`q38_matmul_fp8_vec`); the block scale still applies once per block and the+blocks still add in double, so it differs from the table kernel only by the+float summation order inside a block. For a batch of prefill rows the block+is decoded once and every row runs through it. `Q38_FP8_KERNEL=scalar`+restores the table kernel.++**Measured** on the released Qwen3.8-Flash-Next-FP8, a 16-core server shared+with a training job, `OMP_NUM_THREADS=8`, RAM LRU cap 96 per layer, 100+generated tokens, `COLI_TIMERS=1` decode bank:++| | BF16 trunk, table FP8 (1.12.0) | BF16 trunk, vector FP8 | int8 trunk, vector FP8 (default) |+|---|---:|---:|---:|+| decode, tok/s | 0.61 | 0.77 | **1.42** |+| resident-mm (dense trunk), ms/token | 434 | 437 | 85 |+| lm-head, ms/token | 76 | 79 | 13 |+| routed-expert GEMV, ms/token | 388 | 154 | 140 |+| shared-expert, ms/token | 43 | 44 | 15 |+| expert-read (page cache), ms/token | 156 | 155 | 140 |+| peak RSS, GB | 32.2 | 32.2 | 28.5 |++The middle column reproduces the first's text byte for byte over the 100+tokens: the vector kernel is a faster way to compute the same thing. The+int8 trunk is a quantization, and its cost is measured as perplexity, not+assumed. Teacher-forced NLL on four 512-token chunks of the repository+docs (8 prompt tokens, 504 scored), same machine:++| chunk | BF16 trunk | int8 trunk | BF16 trunk, vector FP8 |+|---|---:|---:|---:|+| 0 | 1.4527 | 1.4416 | 1.4527 |+| 1 | 2.4594 | 2.4544 | |+| 2 | 2.3845 | 2.3929 | |+| 3 | 2.7463 | 2.7753 | |+| mean nats/token | 2.2607 | 2.2661 | |+++0.24% nats, about +0.5% perplexity, two chunks lower and two higher: the+per-row int8 error is noise at this size. The vector FP8 kernel alone+reproduces the BF16 run to four decimals.++Prefill takes the same paths, and it is where the scalar BF16 loop hurt+most. The 512-token chunk 0 as a prompt, one generated token:++| | BF16 trunk, table FP8 | int8 trunk, vector FP8 |+|---|---:|---:|+| wall, 512 prompt tokens | 495 s | 149 s |+| resident-mm | 205 s | 12 s |+| routed-expert GEMV | 172 s | 34 s |+| shared-expert | 15.5 s | 1.8 s |+| expert-read | 28 s | 26 s |++The trunk quantization takes 3.3 s at load for the 553 matrices and the+RSS after load grows by 0.6 GB while the int8 rows and the BF16 copy+coexist; once the BF16 is released the peak RSS of a run is 3.7 GB lower.+## Performance telemetryEvery served request reports its own routed-expert cache hit rate; persistent