Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: Qwen3.8 routed experts as int4-g64 (Q38_EXPERT_INT4)

JustVugg62090444264d
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 0af9ed84..10ddc986 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -17,7 +17,7 @@ what follows, but the sister engines read their own:
| `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family β€” see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) |
| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU` β€” see [Inkling engine](#inkling-engine-inkling) |
| `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, and the `CACHE_ROUTE` family (VRAM tier over RAM cache) β€” see [Qwen3.6 engine](#qwen36-engine-qwen36) |
-| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS` β€” see [Qwen3.8 engine](#qwen38-engine-qwen38) |
+| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_EXPERT_INT4`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS` β€” see [Qwen3.8 engine](#qwen38-engine-qwen38) |
| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` β€” see [OLMoE engine](#olmoe-engine-olmoe) |
| `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates β€” see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read β€” its GPU switch is `DSV4_CUDA` |
| `deepseek_v41` | `c/deepseek_v41.c` | the `V41_*` family: see [DeepSeek V4.1 engine](#deepseek-v41-engine-deepseek_v41) |
@@ -140,7 +140,7 @@ What each engine can do:
| `kimi_k3` | `K3_MMAP=1`: every prepared matrix mapped, LM head included. The embedding is read one row per token in any case. CPU only. | Cache of at least one slot per layer (`K3_EXPERT_GB`). |
| `deepseek_v4` | Automatic: when the dense trunk or the BF16 head does not fit in `RAM_GB`, it is read from disk again on every use. The `ram_tiers` line on stderr says `dense=streamed`. | Cache of at least the top-k slots per layer. |
| `glm53` | Resident (`GLM53_BITS` picks 4, 8 or 32 bits). | `COLI_MAP_EXPERTS=1`: views of a per-shard mapping, CPU runs only. |
-| `qwen38` | Resident (`Q38_TRUNK_CPU_INT8` keeps it as int8). | `COLI_MAP_EXPERTS=1`, native FP8 experts. |
+| `qwen38` | Resident (`Q38_TRUNK_CPU_INT8` keeps it as int8). | `COLI_MAP_EXPERTS=1`, native FP8 experts or the int4-g64 sidecar's records. |
| `qwen36`, `inkling`, `deepseek_v41`, `olmoe` | Resident. | Cache of at least one slot per layer. |
A mapped weight costs a disk read whenever the OS has evicted it, so in the
@@ -444,9 +444,10 @@ checkpoint layout and the text-only capability boundary.
| `Q38_EOS` | tokenizer/config stop IDs | Override the served end-of-sequence token ID for controlled experiments. Normally the engine stops on the tokenizer's `<|im_end|>` / `<|endoftext|>` IDs, falling back to `eos_token_id`. |
| `Q38_NATIVE_FP8` | `1` (on) | Keep routed E4M3 expert bytes and their F32 128Γ—128 block scales native in the LRU. `=0` restores expanded-FP32 slots for A/B validation. |
| `Q38_NATIVE_BF16` | `1` (on) | Keep resident and routed BF16 matrices in two-byte storage while retaining FP32 activations/accumulation. `=0` restores the expanded-FP32 reference. |
+| `Q38_EXPERT_INT4` | auto | Routed experts from `<snap>/experts-int4g64/`, the int4-g64 sidecar `tools/convert_qwen38_experts_int4.py` writes next to the FP8 shards: 2.76 MB per expert instead of 4.92 MB, so the same RAM holds 1.78x the experts and a miss reads 56% of the bytes. Unset: used when its `index.json` says the conversion is complete (an incomplete one is skipped with a line on stderr; one that disagrees with the model is refused). `=0` keeps the snapshot's FP8 experts; `=1` refuses to start without a complete sidecar. The CUDA expert tier streams FP8 only and stays off with int4 experts; `coli plan` / `--auto-tier` size the cap with the sidecar's records. See [qwen38.md](qwen38.md#routed-experts-as-int4-g64). |
| `Q38_PREFILL_BATCH` | `1` (on) | Route prompt rows in bounded expert-major chunks and batch resident shared-expert/DeltaNet projections. `=0` restores row-at-a-time prompt execution for A/B diagnosis; decode is unchanged. |
| `Q38_TRUNK_CPU_INT8` | `1` (on) | The dense trunk (DeltaNet and attention projections, hyper-connection mixers, shared expert, router, lm_head; every matrix of at least `Q38_TRUNK_MIN_KB`) is kept on the CPU as int8 rows with one scale per row and the BF16 copy is released; `q38_weight_matmul` quantizes the activation to int8 and uses the integer kernels of `idot.h` for decode and prefill. `=0` keeps the BF16 rows and the f32 kernel (the numeric reference). See [qwen38.md](qwen38.md#the-trunk-on-the-cpu-int8-rows). |
-| `COLI_MAP_EXPERTS` | `0` | Point the native-FP8 routed-expert slots at a read-only mapping of their shard instead of copying 14 MB per miss into a slab. Same variable as in `glm53`. |
+| `COLI_MAP_EXPERTS` | `0` | Point the native-FP8 routed-expert slots (or the int4-g64 sidecar's records) at a read-only mapping of their shard instead of copying 14 MB per miss into a slab. Same variable as in `glm53`. |
| `Q38_FP8_KERNEL` | vector | The routed experts' e4m3 blocks are decoded eight at a time in registers and multiplied with FMA (AVX2 builds); `scalar` restores `quant.h`'s table kernel, which differs only by float summation order inside a block. |
| `COLI_TIMERS` | `0` (off) | Set to `1` for the detailed Qwen3.8 phase breakdown on stderr. The shared per-request `PROF` frame is emitted regardless. |
diff --git a/docs/qwen38.md b/docs/qwen38.md
index d67c8ccb..2be62741 100644
--- a/docs/qwen38.md
+++ b/docs/qwen38.md
@@ -118,7 +118,7 @@ prompt:
| | |
|---|---|
| resident weights (native BF16) | 9.2 GiB, fixed |
-| routed-expert cache | 4.7 MiB per slot per layer over 48 layers: cap 16 is 3.5 GiB, cap 32 is 7.0 GiB, cap 64 is 14.1 GiB |
+| routed-expert cache | 4.7 MiB per slot per layer over 48 layers: cap 16 is 3.5 GiB, cap 32 is 7.0 GiB, cap 64 is 14.1 GiB (2.6 MiB with the [int4-g64 sidecar](#routed-experts-as-int4-g64): cap 64 is 7.9 GiB) |
| FP8 scale bank | 28 MiB, fixed; every expert's block scales stay resident so a miss is one FP8 read |
| context state | 54 KiB per token, allocated for the whole `Q38_MAXT` ceiling before `READY`: 432 MiB at the 8,192 default |
| recurrent and PLE state, prefix snapshot, cached logits | 226 MiB, fixed |
@@ -157,6 +157,81 @@ when nothing is cached and roughly half that at the cap-32 hit rate, so at this
cache size the engine spends about two thirds of every request waiting on the
disk, and the planner labels cold expert reads as the expected bottleneck.
+## Routed experts as int4-g64
+
+Optional. Decode waits on routed experts: a miss is a disk read, and how many
+experts fit in the cache decides how often that happens.
+`tools/convert_qwen38_experts_int4.py` rewrites them as int4 with one f32 scale
+per 64 inputs into `<model>/experts-int4g64/`, next to the FP8 shards, which it
+only reads; qwen38 picks the directory up by itself.
+
+```sh
+python3 c/tools/convert_qwen38_experts_int4.py --model ~/Models/Qwen3.8-Flash-Next-FP8 --plan
+python3 c/tools/convert_qwen38_experts_int4.py --model ~/Models/Qwen3.8-Flash-Next-FP8
+```
+
+| | FP8 (release) | int4-g64 sidecar |
+|---|---|---|
+| bytes per expert | 4,915,200 (scales in the 28 MiB resident bank) | 2,764,800: codes 2,457,600, scales 307,200 |
+| experts per GiB of cache | 218 | 388 |
+| cap per GiB of cache, 48 layers | 4.5 | 8.1 |
+| all routed experts on disk | 120.8 GB | 68.0 GB, in addition to the FP8 shards |
+
+The values are the ones the engine computes from the release (e4m3 byte times
+its block's `weight_scale_inv`), quantized as `tools/convert_qwen36.py --ebits 4
+--gs 64` does: per 64 inputs, scale = absmax / 7, code = round half to even of
+w / scale, clamped to [-8, 7]. The codes are stored as v+8 in `expert_ffn.h`'s
+planar layout and multiplied by its f32 kernel, activations staying f32 as on
+the FP8 path. On synthetic Gaussian weights at the release's geometry the
+relative L2 error is 10.9% per matrix; the converter prints the mean and the
+worst per layer on the real weights. What that costs the model in perplexity
+or answers has not been measured yet.
+
+The converter streams: one expert per worker in flight, so peak RAM is a few
+hundred MB for the writer (records queue there while the disk catches up; at
+most one layer, 1.4 GB) plus 0.11 GB per worker, whatever the model size. On a 12-thread laptop
+CPU (i7-1355U) and a synthetic snapshot of the release's expert geometry a
+layer is 22 s of compute with 12 workers; the disk sets the rest. It is
+resumable: a layer is written to a `.part` file and renamed when whole, and
+`index.json` lists the layers done, so a second run continues where the first
+stopped. A run against a snapshot whose `config.json` or expert shards changed
+is refused.
+
+Layout: one safetensors file per layer (`layer-NNN.safetensors`) and
+`index.json`. Every expert is one record, so a miss is one read:
+
+```
+gate codes U8 [I, H/2] | up codes U8 [I, H/2] | down codes U8 [H, I/2] |
+gate scales F32 [I, H/64] | up scales F32 [I, H/64] | down scales F32 [H, I/64]
+```
+
+named like the release's tensors (`...experts.E.gate_proj.weight`, and the same
+name plus `.qs` for its scales). The data region of each file starts on a
+4096-byte boundary; at the release's geometry a record is 675 pages, so every
+record is page aligned. `index.json` carries the format name and version, the
+geometry, the group size, the record size, the layers done and `complete`;
+the engine refuses a sidecar whose index or files disagree with `config.json`
+and skips (with a line on stderr) one still being written.
+
+`Q38_EXPERT_INT4=0` keeps the snapshot's FP8 experts; `=1` refuses to start
+without a complete sidecar. Every expert path reads the sidecar: the single LRU
+load, the parallel batch reads (`Q38_EXPERT_PARALLEL_READS`), the prefetch, the
+prefill batch and `COLI_MAP_EXPERTS=1`. The startup line names the expert
+representation, the bytes of one expert, what the cache costs full at the
+given cap, and how many experts per layer the RAM available at that moment
+would hold. `coli plan` and `--auto-tier` size the cache cap with the
+sidecar's records and plan no VRAM for them: the CUDA expert tier streams FP8
+experts only, and with int4 experts it stays off, trunk included. The Segment and Edge adapters keep the snapshot's
+experts.
+
+`make -C c qwen38-tiny-int4-check` converts the FP8 tiny fixture (experts
+scaled by 3, so that they move the logits beyond the oracle's tolerance) and
+gates on the int4 matmul against a dequantized f32 reference, the converter's
+round trip against the FP8 source, the engine reproducing the float32
+reference of the dequantized sidecar through every expert path with the last
+logits identical to the bit across them, and the CUDA tier declining the int4
+experts on the fake backend.
+
## GPU: CUDA VRAM expert tier
With `CUDA=1` the engine links the same expert tier as Qwen3.6
@@ -414,6 +489,8 @@ eviction. The gate checks both greedy token IDs and the final upstream logit
vector, and runs both native-BF16 and expanded-FP32 resident modes at cache
capacities one and four. CI repeats the capacity-one path under ASan and UBSan
and verifies that a config/tensor shape disagreement is refused.
+`make -C c qwen38-tiny-int4-check` gates the optional int4-g64 experts (see
+[Routed experts as int4-g64](#routed-experts-as-int4-g64)).
## Supported checkpoint layouts