Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: qwen38 int8 trunk and vector FP8 kernel, measured; 1.12.1 notes for #1650 and #1668

JustVuggcde709c4a634
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 62ddb50d..3b140188 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,7 +5,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).
## [1.12.1] β€” 2026-09-22
-32 pull requests since v1.12.0, 22 of them from contributors. Two tokenizers
+34 pull requests since v1.12.0, 23 of them from contributors. Two tokenizers
brought back to the reference, brio on the ninth engine, `coli chat` working
again at the default context on two families, and a placement decision that
is now measured on the card in front of it instead of predicted.
@@ -68,7 +68,7 @@ is now measured on the card in front of it instead of predicted.
### Performance
-- **#PR**: qwen36's dense trunk and routed experts multiply with integer
+- **#1664**: qwen36's dense trunk and routed experts multiply with integer
dot products. The activation is quantized to int8 once per call and the
weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd
instead of a float conversion per weight; the integer kernels move from
@@ -80,6 +80,18 @@ is now measured on the card in front of it instead of predicted.
`COLI_DENSE_INT4=<components>` stores part of the trunk as int4 in blocks
of 64: opt-in, with the perplexity it costs per component in the docs
(lm_head alone +2.4%, everything +10%).
+- **#1668**: qwen38's dense trunk (553 matrices, 3.6 G weights, 8 GiB of
+ BF16 read on every token, more than the ten routed experts) is kept on the
+ CPU as int8 rows with the BF16 copy released, and multiplied with the same
+ integer kernels; the routed experts' e4m3 blocks are decoded eight at a
+ time in registers and multiplied with FMA instead of one table lookup per
+ weight. Measured on the released Qwen3.8-Flash-Next-FP8, 8 threads, RAM
+ LRU 96 per layer: decode 0.61 to 1.42 tok/s, the trunk 434 to 85 ms/token,
+ lm_head 76 to 13, the expert GEMVs 388 to 140, peak RSS 32.2 to 28.5 GB;
+ prefill of 512 tokens 495 to 149 s. Perplexity on 4 x 512 tokens +0.5%
+ (two chunks lower, two higher); the vector FP8 kernel alone reproduces the
+ BF16 run to four decimals. Both are the default (`Q38_TRUNK_CPU_INT8=0`
+ keeps the BF16 trunk, `Q38_FP8_KERNEL=scalar` the table kernel).
### Performance, from contributors
@@ -99,6 +111,12 @@ is now measured on the card in front of it instead of predicted.
### Fixed
+- **#1650** (bokiko): a Qwen3.8 pin snapshots the recurrent and PLE state
+ but reuses the live attention and indexer rows; after an unrelated prompt
+ overwrote those rows, returning to the pin could change brio logprobs
+ without a warning. The engine now records the token identity of the live
+ rows (`kv_prefix.h`) and refuses a stale pin or prefix restore; image rows
+ are tainted. Wire regressions run on the BF16 and FP8 fixtures.
- **#1626**: `SNAP` is the model directory for every non-GLM engine, so
`coli run` stops handing them a leftover environment (#1600).
- **#1604**: glm53 honours `Mat.resident` in the Vulkan gate and frees the
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 91969d00..04063bd7 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -16,7 +16,7 @@ what follows, but the sister engines read their own:
| `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family β€” see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) |
| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU` β€” see [Inkling engine](#inkling-engine-inkling) |
| `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, and the `CACHE_ROUTE` family (VRAM tier over RAM cache) β€” see [Qwen3.6 engine](#qwen36-engine-qwen36) |
-| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_PREFILL_BATCH`, `COLI_TIMERS` β€” see [Qwen3.8 engine](#qwen38-engine-qwen38) |
+| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS` β€” see [Qwen3.8 engine](#qwen38-engine-qwen38) |
| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` β€” see [OLMoE engine](#olmoe-engine-olmoe) |
| `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates β€” see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read β€” its GPU switch is `DSV4_CUDA` |
@@ -412,6 +412,8 @@ checkpoint layout and the text-only capability boundary.
| `Q38_NATIVE_FP8` | `1` (on) | Keep routed E4M3 expert bytes and their F32 128Γ—128 block scales native in the LRU. `=0` restores expanded-FP32 slots for A/B validation. |
| `Q38_NATIVE_BF16` | `1` (on) | Keep resident and routed BF16 matrices in two-byte storage while retaining FP32 activations/accumulation. `=0` restores the expanded-FP32 reference. |
| `Q38_PREFILL_BATCH` | `1` (on) | Route prompt rows in bounded expert-major chunks and batch resident shared-expert/DeltaNet projections. `=0` restores row-at-a-time prompt execution for A/B diagnosis; decode is unchanged. |
+| `Q38_TRUNK_CPU_INT8` | `1` (on) | The dense trunk (DeltaNet and attention projections, hyper-connection mixers, shared expert, router, lm_head; every matrix of at least `Q38_TRUNK_MIN_KB`) is kept on the CPU as int8 rows with one scale per row and the BF16 copy is released; `q38_weight_matmul` quantizes the activation to int8 and uses the integer kernels of `idot.h` for decode and prefill. `=0` keeps the BF16 rows and the f32 kernel (the numeric reference). See [qwen38.md](qwen38.md#the-trunk-on-the-cpu-int8-rows). |
+| `Q38_FP8_KERNEL` | vector | The routed experts' e4m3 blocks are decoded eight at a time in registers and multiplied with FMA (AVX2 builds); `scalar` restores `quant.h`'s table kernel, which differs only by float summation order inside a block. |
| `COLI_TIMERS` | `0` (off) | Set to `1` for the detailed Qwen3.8 phase breakdown on stderr. The shared per-request `PROF` frame is emitted regardless. |
## DeepSeek V4 engine (`deepseek_v4`)
diff --git a/docs/qwen38.md b/docs/qwen38.md
index fd4dccc7..79b90341 100644
--- a/docs/qwen38.md
+++ b/docs/qwen38.md
@@ -249,8 +249,8 @@ whatever the placer accepts is quantized to **int8 per row** (scale = max|w|
/ 127, the qwen36 dnproj/lmhead format) when the tier starts -- about 2 s
for the 553 matrices, 3.96 GiB on one card -- and answers decode GEMVs from
there, one round trip per matmul (`x` up, `y` down; activations stay on the
-CPU). Prefill rows (S > 1) and any backend failure take the BF16 CPU path,
-which stays in RAM. The placer takes the trunk before the experts (it is
+CPU). Prefill rows (S > 1) and any backend failure take the CPU path, which
+holds the same int8 rows (see [the trunk on the CPU](#the-trunk-on-the-cpu-int8-rows)). The placer takes the trunk before the experts (it is
read on every token), so on an 8 GB card about 2 GB remain for hot experts;
`coli plan` prices it the same way (`4.0 GB int8 trunk + ... hot tier`).
@@ -260,7 +260,7 @@ read on every token), so on an 8 GB card about 2 GB remain for hot experts;
| `Q38_TRUNK_MIN_KB=<n>` | offer matrices of at least n KiB (default 1024; a round trip costs more than a tiny GEMV saves) |
| `Q38_TRUNK_SKIP=name,name` | leave the named components on the CPU (bisecting, or a component that does not pay) |
| `QT_UPLOAD_SYNC=1` | the tier's `qt_issue` waits for every in-flight upload first (tests and diagnostics: deterministic residency, no upload/compute overlap) |
-| `Q38_TRUNK_CPU_INT8=1` | the same int8 rows on the CPU instead -- what the quantization alone does to the output, no GPU needed (perplexity, token parity); a matrix the GPU holds is still answered from VRAM |
+| `Q38_TRUNK_CPU_INT8=0` | keep the trunk BF16 on the CPU with the f32 kernel (the numeric reference); the default is int8 rows with integer dot products, GPU or not |
| `Q38_TRUNK_SELFTEST=1` | at start, every placed matrix is checked once: GPU GEMV against the same int8 rows on the CPU (relative error printed per matrix) |
**Numerics.** GPU int8 against CPU int8 on the same rows: relative error
@@ -312,6 +312,75 @@ error, only wrong tokens, worse the more experts were resident. The dense
path has its own buffers now ([qwen36-cuda-tier.md](qwen36-cuda-tier.md));
qwen36 never called the dense path inside that window.
+### The trunk on the CPU: int8 rows
+
+Without a GPU the same trunk is the decode's floor: 8 GiB of BF16 read on
+every token, multiplied by a scalar loop. Since 1.12.1 the engine keeps the
+trunk on the CPU as **int8 rows with one scale per row** (the format the
+tier uploads, so GPU or not the rows are the same bytes), releases the BF16
+copy once the rows exist, and multiplies with the integer kernels of
+`idot.h`: the activation is quantized to int8 once per row and meets the
+weights with maddubs / vpdpbusd (AVX2 / AVX-512 VNNI) or NEON dot products.
+Decode and prefill take this path. Every matrix of at least
+`Q38_TRUNK_MIN_KB` (default 1 MiB) is in; the small ones stay BF16 because
+they cost nothing either way. `Q38_TRUNK_CPU_INT8=0` keeps the BF16 rows and
+the f32 kernel, the numeric reference.
+
+The routed experts stay e4m3 with their 128 x 128 block scales, but the
+kernel decodes eight bytes at a time in registers and multiplies with FMA
+(`q38_matmul_fp8_vec`); the block scale still applies once per block and the
+blocks still add in double, so it differs from the table kernel only by the
+float summation order inside a block. For a batch of prefill rows the block
+is decoded once and every row runs through it. `Q38_FP8_KERNEL=scalar`
+restores the table kernel.
+
+**Measured** on the released Qwen3.8-Flash-Next-FP8, a 16-core server shared
+with a training job, `OMP_NUM_THREADS=8`, RAM LRU cap 96 per layer, 100
+generated tokens, `COLI_TIMERS=1` decode bank:
+
+| | BF16 trunk, table FP8 (1.12.0) | BF16 trunk, vector FP8 | int8 trunk, vector FP8 (default) |
+|---|---:|---:|---:|
+| decode, tok/s | 0.61 | 0.77 | **1.42** |
+| resident-mm (dense trunk), ms/token | 434 | 437 | 85 |
+| lm-head, ms/token | 76 | 79 | 13 |
+| routed-expert GEMV, ms/token | 388 | 154 | 140 |
+| shared-expert, ms/token | 43 | 44 | 15 |
+| expert-read (page cache), ms/token | 156 | 155 | 140 |
+| peak RSS, GB | 32.2 | 32.2 | 28.5 |
+
+The middle column reproduces the first's text byte for byte over the 100
+tokens: the vector kernel is a faster way to compute the same thing. The
+int8 trunk is a quantization, and its cost is measured as perplexity, not
+assumed. Teacher-forced NLL on four 512-token chunks of the repository
+docs (8 prompt tokens, 504 scored), same machine:
+
+| chunk | BF16 trunk | int8 trunk | BF16 trunk, vector FP8 |
+|---|---:|---:|---:|
+| 0 | 1.4527 | 1.4416 | 1.4527 |
+| 1 | 2.4594 | 2.4544 | |
+| 2 | 2.3845 | 2.3929 | |
+| 3 | 2.7463 | 2.7753 | |
+| mean nats/token | 2.2607 | 2.2661 | |
+
++0.24% nats, about +0.5% perplexity, two chunks lower and two higher: the
+per-row int8 error is noise at this size. The vector FP8 kernel alone
+reproduces the BF16 run to four decimals.
+
+Prefill takes the same paths, and it is where the scalar BF16 loop hurt
+most. The 512-token chunk 0 as a prompt, one generated token:
+
+| | BF16 trunk, table FP8 | int8 trunk, vector FP8 |
+|---|---:|---:|
+| wall, 512 prompt tokens | 495 s | 149 s |
+| resident-mm | 205 s | 12 s |
+| routed-expert GEMV | 172 s | 34 s |
+| shared-expert | 15.5 s | 1.8 s |
+| expert-read | 28 s | 26 s |
+
+The trunk quantization takes 3.3 s at load for the 553 matrices and the
+RSS after load grows by 0.6 GB while the int8 rows and the BF16 copy
+coexist; once the BF16 is released the peak RSS of a run is 3.7 GB lower.
+
## Performance telemetry
Every served request reports its own routed-expert cache hit rate; persistent