Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: Qwen3.8 routed experts as int4-g64 (Q38_EXPERT_INT4)
JustVugg62090444264d
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 0af9ed84..10ddc986 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -17,7 +17,7 @@ what follows, but the sister engines read their own:| `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family β see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) || `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU` β see [Inkling engine](#inkling-engine-inkling) || `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, and the `CACHE_ROUTE` family (VRAM tier over RAM cache) β see [Qwen3.6 engine](#qwen36-engine-qwen36) |-| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS` β see [Qwen3.8 engine](#qwen38-engine-qwen38) |+| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_EXPERT_INT4`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS` β see [Qwen3.8 engine](#qwen38-engine-qwen38) || `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` β see [OLMoE engine](#olmoe-engine-olmoe) || `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates β see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read β its GPU switch is `DSV4_CUDA` || `deepseek_v41` | `c/deepseek_v41.c` | the `V41_*` family: see [DeepSeek V4.1 engine](#deepseek-v41-engine-deepseek_v41) |@@ -140,7 +140,7 @@ What each engine can do:| `kimi_k3` | `K3_MMAP=1`: every prepared matrix mapped, LM head included. The embedding is read one row per token in any case. CPU only. | Cache of at least one slot per layer (`K3_EXPERT_GB`). || `deepseek_v4` | Automatic: when the dense trunk or the BF16 head does not fit in `RAM_GB`, it is read from disk again on every use. The `ram_tiers` line on stderr says `dense=streamed`. | Cache of at least the top-k slots per layer. || `glm53` | Resident (`GLM53_BITS` picks 4, 8 or 32 bits). | `COLI_MAP_EXPERTS=1`: views of a per-shard mapping, CPU runs only. |-| `qwen38` | Resident (`Q38_TRUNK_CPU_INT8` keeps it as int8). | `COLI_MAP_EXPERTS=1`, native FP8 experts. |+| `qwen38` | Resident (`Q38_TRUNK_CPU_INT8` keeps it as int8). | `COLI_MAP_EXPERTS=1`, native FP8 experts or the int4-g64 sidecar's records. || `qwen36`, `inkling`, `deepseek_v41`, `olmoe` | Resident. | Cache of at least one slot per layer. |A mapped weight costs a disk read whenever the OS has evicted it, so in the@@ -444,9 +444,10 @@ checkpoint layout and the text-only capability boundary.| `Q38_EOS` | tokenizer/config stop IDs | Override the served end-of-sequence token ID for controlled experiments. Normally the engine stops on the tokenizer's `<|im_end|>` / `<|endoftext|>` IDs, falling back to `eos_token_id`. || `Q38_NATIVE_FP8` | `1` (on) | Keep routed E4M3 expert bytes and their F32 128Γ128 block scales native in the LRU. `=0` restores expanded-FP32 slots for A/B validation. || `Q38_NATIVE_BF16` | `1` (on) | Keep resident and routed BF16 matrices in two-byte storage while retaining FP32 activations/accumulation. `=0` restores the expanded-FP32 reference. |+| `Q38_EXPERT_INT4` | auto | Routed experts from `<snap>/experts-int4g64/`, the int4-g64 sidecar `tools/convert_qwen38_experts_int4.py` writes next to the FP8 shards: 2.76 MB per expert instead of 4.92 MB, so the same RAM holds 1.78x the experts and a miss reads 56% of the bytes. Unset: used when its `index.json` says the conversion is complete (an incomplete one is skipped with a line on stderr; one that disagrees with the model is refused). `=0` keeps the snapshot's FP8 experts; `=1` refuses to start without a complete sidecar. The CUDA expert tier streams FP8 only and stays off with int4 experts; `coli plan` / `--auto-tier` size the cap with the sidecar's records. See [qwen38.md](qwen38.md#routed-experts-as-int4-g64). || `Q38_PREFILL_BATCH` | `1` (on) | Route prompt rows in bounded expert-major chunks and batch resident shared-expert/DeltaNet projections. `=0` restores row-at-a-time prompt execution for A/B diagnosis; decode is unchanged. || `Q38_TRUNK_CPU_INT8` | `1` (on) | The dense trunk (DeltaNet and attention projections, hyper-connection mixers, shared expert, router, lm_head; every matrix of at least `Q38_TRUNK_MIN_KB`) is kept on the CPU as int8 rows with one scale per row and the BF16 copy is released; `q38_weight_matmul` quantizes the activation to int8 and uses the integer kernels of `idot.h` for decode and prefill. `=0` keeps the BF16 rows and the f32 kernel (the numeric reference). See [qwen38.md](qwen38.md#the-trunk-on-the-cpu-int8-rows). |-| `COLI_MAP_EXPERTS` | `0` | Point the native-FP8 routed-expert slots at a read-only mapping of their shard instead of copying 14 MB per miss into a slab. Same variable as in `glm53`. |+| `COLI_MAP_EXPERTS` | `0` | Point the native-FP8 routed-expert slots (or the int4-g64 sidecar's records) at a read-only mapping of their shard instead of copying 14 MB per miss into a slab. Same variable as in `glm53`. || `Q38_FP8_KERNEL` | vector | The routed experts' e4m3 blocks are decoded eight at a time in registers and multiplied with FMA (AVX2 builds); `scalar` restores `quant.h`'s table kernel, which differs only by float summation order inside a block. || `COLI_TIMERS` | `0` (off) | Set to `1` for the detailed Qwen3.8 phase breakdown on stderr. The shared per-request `PROF` frame is emitted regardless. |diff --git a/docs/qwen38.md b/docs/qwen38.mdindex d67c8ccb..2be62741 100644--- a/docs/qwen38.md+++ b/docs/qwen38.md@@ -118,7 +118,7 @@ prompt:| | ||---|---|| resident weights (native BF16) | 9.2 GiB, fixed |-| routed-expert cache | 4.7 MiB per slot per layer over 48 layers: cap 16 is 3.5 GiB, cap 32 is 7.0 GiB, cap 64 is 14.1 GiB |+| routed-expert cache | 4.7 MiB per slot per layer over 48 layers: cap 16 is 3.5 GiB, cap 32 is 7.0 GiB, cap 64 is 14.1 GiB (2.6 MiB with the [int4-g64 sidecar](#routed-experts-as-int4-g64): cap 64 is 7.9 GiB) || FP8 scale bank | 28 MiB, fixed; every expert's block scales stay resident so a miss is one FP8 read || context state | 54 KiB per token, allocated for the whole `Q38_MAXT` ceiling before `READY`: 432 MiB at the 8,192 default || recurrent and PLE state, prefix snapshot, cached logits | 226 MiB, fixed |@@ -157,6 +157,81 @@ when nothing is cached and roughly half that at the cap-32 hit rate, so at thiscache size the engine spends about two thirds of every request waiting on thedisk, and the planner labels cold expert reads as the expected bottleneck.+## Routed experts as int4-g64++Optional. Decode waits on routed experts: a miss is a disk read, and how many+experts fit in the cache decides how often that happens.+`tools/convert_qwen38_experts_int4.py` rewrites them as int4 with one f32 scale+per 64 inputs into `<model>/experts-int4g64/`, next to the FP8 shards, which it+only reads; qwen38 picks the directory up by itself.++```sh+python3 c/tools/convert_qwen38_experts_int4.py --model ~/Models/Qwen3.8-Flash-Next-FP8 --plan+python3 c/tools/convert_qwen38_experts_int4.py --model ~/Models/Qwen3.8-Flash-Next-FP8+```++| | FP8 (release) | int4-g64 sidecar |+|---|---|---|+| bytes per expert | 4,915,200 (scales in the 28 MiB resident bank) | 2,764,800: codes 2,457,600, scales 307,200 |+| experts per GiB of cache | 218 | 388 |+| cap per GiB of cache, 48 layers | 4.5 | 8.1 |+| all routed experts on disk | 120.8 GB | 68.0 GB, in addition to the FP8 shards |++The values are the ones the engine computes from the release (e4m3 byte times+its block's `weight_scale_inv`), quantized as `tools/convert_qwen36.py --ebits 4+--gs 64` does: per 64 inputs, scale = absmax / 7, code = round half to even of+w / scale, clamped to [-8, 7]. The codes are stored as v+8 in `expert_ffn.h`'s+planar layout and multiplied by its f32 kernel, activations staying f32 as on+the FP8 path. On synthetic Gaussian weights at the release's geometry the+relative L2 error is 10.9% per matrix; the converter prints the mean and the+worst per layer on the real weights. What that costs the model in perplexity+or answers has not been measured yet.++The converter streams: one expert per worker in flight, so peak RAM is a few+hundred MB for the writer (records queue there while the disk catches up; at+most one layer, 1.4 GB) plus 0.11 GB per worker, whatever the model size. On a 12-thread laptop+CPU (i7-1355U) and a synthetic snapshot of the release's expert geometry a+layer is 22 s of compute with 12 workers; the disk sets the rest. It is+resumable: a layer is written to a `.part` file and renamed when whole, and+`index.json` lists the layers done, so a second run continues where the first+stopped. A run against a snapshot whose `config.json` or expert shards changed+is refused.++Layout: one safetensors file per layer (`layer-NNN.safetensors`) and+`index.json`. Every expert is one record, so a miss is one read:++```+gate codes U8 [I, H/2] | up codes U8 [I, H/2] | down codes U8 [H, I/2] |+gate scales F32 [I, H/64] | up scales F32 [I, H/64] | down scales F32 [H, I/64]+```++named like the release's tensors (`...experts.E.gate_proj.weight`, and the same+name plus `.qs` for its scales). The data region of each file starts on a+4096-byte boundary; at the release's geometry a record is 675 pages, so every+record is page aligned. `index.json` carries the format name and version, the+geometry, the group size, the record size, the layers done and `complete`;+the engine refuses a sidecar whose index or files disagree with `config.json`+and skips (with a line on stderr) one still being written.++`Q38_EXPERT_INT4=0` keeps the snapshot's FP8 experts; `=1` refuses to start+without a complete sidecar. Every expert path reads the sidecar: the single LRU+load, the parallel batch reads (`Q38_EXPERT_PARALLEL_READS`), the prefetch, the+prefill batch and `COLI_MAP_EXPERTS=1`. The startup line names the expert+representation, the bytes of one expert, what the cache costs full at the+given cap, and how many experts per layer the RAM available at that moment+would hold. `coli plan` and `--auto-tier` size the cache cap with the+sidecar's records and plan no VRAM for them: the CUDA expert tier streams FP8+experts only, and with int4 experts it stays off, trunk included. The Segment and Edge adapters keep the snapshot's+experts.++`make -C c qwen38-tiny-int4-check` converts the FP8 tiny fixture (experts+scaled by 3, so that they move the logits beyond the oracle's tolerance) and+gates on the int4 matmul against a dequantized f32 reference, the converter's+round trip against the FP8 source, the engine reproducing the float32+reference of the dequantized sidecar through every expert path with the last+logits identical to the bit across them, and the CUDA tier declining the int4+experts on the fake backend.+## GPU: CUDA VRAM expert tierWith `CUDA=1` the engine links the same expert tier as Qwen3.6@@ -414,6 +489,8 @@ eviction. The gate checks both greedy token IDs and the final upstream logitvector, and runs both native-BF16 and expanded-FP32 resident modes at cachecapacities one and four. CI repeats the capacity-one path under ASan and UBSanand verifies that a config/tensor shape disagreement is refused.+`make -C c qwen38-tiny-int4-check` gates the optional int4-g64 experts (see+[Routed experts as int4-g64](#routed-experts-as-int4-g64)).## Supported checkpoint layouts