Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: GLM-5.2 6x RTX 5090 full-residency experiment β€” 6.84 tok/s, honest dead-ends log (#94)

ZacharyZcR3fb5a001060c
diff --git a/README.md b/README.md
index b8d3dc6d..66bf1efe 100644
--- a/README.md
+++ b/README.md
@@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
+Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) β€” full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
+
| metric | value |
|---|---|
| model on disk (int4 container) | ~370 GB |
diff --git a/docs/experiments/glm52-6x5090-2026-07-12.md b/docs/experiments/glm52-6x5090-2026-07-12.md
new file mode 100644
index 00000000..2c86c7cc
--- /dev/null
+++ b/docs/experiments/glm52-6x5090-2026-07-12.md
@@ -0,0 +1,302 @@
+# GLM-5.2 on 6x RTX 5090
+
+This note records the July 12, 2026 experiments that compared colibri with
+vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon
+Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe
+storage. It exists to preserve the measured facts and avoid repeating the same
+dead ends.
+
+The day had two halves. The morning runs (below) measured colibri at 0.12
+tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on
+colibri (see "colibri full-resident experiments") removed disk from the decode
+path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark
+and 6.84 tok/s on a 256-token run β€” the fastest single-request result measured
+on this machine so far.
+
+## Models and artifacts
+
+- Official NVFP4 model: `/data/models/GLM-5.2-NVFP4` (434 GB, 47 shards)
+- colibri INT4 model: `/data/models/GLM-5.2-colibri-int4` (144 shards)
+- vLLM-Moet TP4 pack: `/data/models/GLM-5.2-NVFP4-packs` (190 GB)
+- vLLM-Moet TP2/PP3 pack: `/data/models/GLM-5.2-NVFP4-packs-tp2pp3` (190 GB)
+- vLLM-Moet checkout: `/data/test/vLLM-Moet`
+- Built image: `vllm-moet-sm120:v024`
+
+The official model was downloaded through `hf-mirror.com` with
+`HF_HUB_DISABLE_XET=1`. Direct Hugging Face access failed and Xet/CAS stalled;
+ordinary HTTP from the mirror sustained roughly 22-46 MB/s.
+
+## Kernel validation
+
+All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke
+tests. Representative correctness results were:
+
+| Path | Shape | Max relative error | Cosine similarity |
+| --- | --- | ---: | ---: |
+| Decode | T=2 | 2.086e-2 | 0.999878 |
+| NVFP4 delta | T=2 | 1.818e-2 | 0.999884 |
+| AFRAG prefill | T=128 | 2.591e-2 | 0.999883 |
+
+The kernels work on SM120. The performance limitation below is therefore not
+an unsupported-GPU fallback.
+
+## Real generation results
+
+All decode rates below are for a single request, not aggregate throughput.
+
+| Runtime and layout | GPU use | TTFT | Decode rate | Important observation |
+| --- | --- | ---: | ---: | --- |
+| colibri INT4 (morning, partial residency) | 6 GPUs | about 42 s | 0.12 tok/s | GPU hot tier was active, but dense compute remained on CPU |
+| colibri INT4 (afternoon, full residency) | 6 GPUs + 24 cores | β€” | 6.28-6.84 tok/s | all 19,456 experts resident in VRAM+RAM, zero disk waits |
+| vLLM-Moet TP4, 12 GiB expert cache/card | GPUs 0-3 | 2.03 s | median 2.5 tok/s | 25.3% coverage; 100% whole-step replay |
+| vLLM-Moet TP4, 14 GiB expert cache/card | GPUs 0-3 | not remeasured | median 2.6 tok/s | 29.5% coverage; replay still 100% |
+| vLLM-Moet TP2 x PP3 | all 6 GPUs | 5.39 s | 1.78 tok/s | more local coverage, but serial PP and communication made latency worse |
+
+The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only
+about 0.1 tok/s because every decode step still missed experts and replayed the
+whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not
+reduce replay enough to offset pipeline and PYNCCL communication costs.
+
+The morning colibri run generated 16 tokens in 132.46 seconds. Its measured
+hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the
+expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk
+reads (29.31 s), and attention (14.28 s). The afternoon experiments below
+showed the dominant problem was expert loading from disk, not the CUDA path
+itself.
+
+## colibri full-resident experiments (afternoon)
+
+Test directory: `/data/test/colibri-full-resident`. Related PR:
+[#80](https://github.com/JustVugg/colibri/pull/80) (on top of PR #68). All A/B
+runs used the same binary flags, environment, prompt, `TEMP=0`, and token
+count; only decode throughput is compared, never load or prefill time.
+
+### The optimization ladder (each step cumulative)
+
+| Change | Decode rate | Evidence |
+| --- | ---: | --- |
+| Baseline: 150 GB VRAM + 150 GB RAM fixed placement | 2.30 tok/s | 4.15 s of disk waits per 20 tokens |
+| Full residency: all 19,456 experts in VRAM+RAM | 5.77 tok/s | 100% expert hit, zero disk waits |
+| In-session dynamic GPU expert repinning (`REPIN=16`) | 6.00 tok/s | expert compute 6.76 s β†’ 5.96 s |
+| 24 physical cores, `OMP_PROC_BIND=spread OMP_PLACES=cores` | +39.6% vs unpinned | unpinned 3.64 β†’ pinned 5.08 tok/s on the same prompt |
+| Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT) | 6.05-6.08 tok/s | GPU expert calls 36,865 β†’ 37,285 |
+| Decode swap cap 32 β†’ 16 per repin round | 6.10-6.28 tok/s | swap cost 0.18 β†’ 0.09 s/round |
+
+The final configuration measured 6.28 tok/s on the fixed 96-token greedy
+benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed
+costs; always state the token count when quoting a rate).
+
+Winning layout, found automatically:
+
+```text
+GPU experts: 9,343 / 19,456 (176.73 GB across six cards)
+RAM experts: 10,113 / 19,456 (about 191.3 GB)
+Disk service/wait during decode: 0 s
+```
+
+Run configuration: `COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto
+CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto`, with
+`OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores` (now the default
+generated by `resource_plan.py`).
+
+### Directions tested and rejected (all reverted)
+
+- Faster (`REPIN=8`) or slower (`REPIN=32`) repinning, and 64 swaps per
+ round: swap/sync overhead or stale GPU expert sets lose more than they gain.
+ 16 tokens between repins and 16 swaps per round is the current local optimum
+ (8 swaps also regresses: fewer GPU calls outweigh the sync savings).
+- A second prefill correction pass: overfits the short prompt's routing and
+ disturbs the history-based layout (5.86 tok/s).
+- Lazy demotion of evicted GPU experts (4.77 tok/s) and GPU→RAM D2H recovery
+ (6.15 tok/s): both cost as much as re-reading from page cache.
+- CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30
+ tok/s); GPU expert groups are too small a fraction of step time, and the
+ extra threads compete with the 24 CPU workers.
+- OpenMP restructuring of RAM experts (row batching across experts,
+ persistent teams): neutral or negative; libgomp already reuses hot teams,
+ and interleaving experts in one static schedule destroys per-expert
+ NUMA/cache locality.
+- `numactl --interleave=all`, 2 MB transparent huge pages, 12-core runs,
+ smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI
+ int4Γ—int8 decode: all neutral or negative. Capacity and TLB are not the
+ limit; raw history counts predict routing better than normalized ones.
+- Next-layer expert prediction with GPU staging: the routing signal is real
+ (70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the
+ expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth
+ revisiting only with dedicated streams that do not stall attention.
+
+### Candidate not yet merged: AVX-512 int4 kernel
+
+The lossless int4β†’float RAM expert kernel still uses 256-bit AVX2. A 512-bit
+unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 β†’ 5.89
+tok/s, expert time βˆ’19.7%), and a dual-accumulator variant added a little more
+(5.89 β†’ 5.94 tok/s at 96 tokens). However, the changed accumulation order
+produces small float differences and the 64-token outputs eventually diverge,
+so it cannot be claimed bit-exact. It stays out of the tree until it has
+numeric-error unit tests, a perplexity/token-consistency evaluation, and an
+explicit decision on whether float-accumulation differences are acceptable.
+Row blocking (2/4 output rows) looked good in short microbenchmarks but the
+gain disappeared over 10,000 sustained iterations and in a 256-token run
+(6.84 vs 6.83 tok/s) β€” a turbo/cache artifact, reverted.
+
+### Where the time goes now
+
+At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other
+2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of
+CPU time in `matmul_qt` and 8.8% in `rope_interleave`; a per-thread RoPE
+sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention
+βˆ’5.0%) with no math change. Scheduling-level CPU optimizations are exhausted β€”
+the remaining expert time is the int4 matmul kernel itself.
+
+## MTP speculative decoding at full residency (July 13)
+
+Motivation: with all experts resident and disk waits at zero, the old
+objection to speculation (a rejected draft wastes a disk read) no longer
+applies, so the July 3 negative result deserved a retest. All runs used the
+same binary and environment as the 16-swap configuration above, `TEMP=0`,
+96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy
+output degenerates into repetition, and a non-degenerate "water cycle"
+prompt).
+
+Phase 1 found the installed MTP head unusable: acceptance 0-4% at every
+depth, exactly the known int4-head defect from issue #8 (the three
+`out-mtp-*.safetensors` byte sizes matched the "unusable" fingerprint).
+Speculation with the broken head costs 15-18% throughput β€” pure verification
+tax.
+
+Phase 2 replaced the head with the community int8 MTP shards
+(`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`, ~10 GB; originals kept as
+`.int4.bak`). The head then worked as designed β€” and speculation still lost
+at every depth:
+
+| Prompt | DRAFT | tok/s | tok/forward | Acceptance |
+| --- | ---: | ---: | ---: | ---: |
+| water cycle | 0 | 6.79 | 1.01 | β€” |
+| water cycle | 1 | 6.45 | 1.81 | 79% |
+| water cycle | 2 | 5.65 | 2.29 | 64% |
+| water cycle | 3 | 6.13 | 3.10 | 69% |
+| sky (degenerate) | 0 | 6.12 | 1.01 | β€” |
+| sky (degenerate) | 1 | 5.51 | 1.75 | 73% |
+| sky (degenerate) | 2 | 4.42 | 1.88 | 44% |
+| sky (degenerate) | 3 | 3.86 / 3.81 | 2.13 | 38% |
+
+The failure is structural, not a tuning problem. In a dense model a verify
+batch of S tokens rereads the same weights once, so speculation amortizes
+the dominant cost. In this MoE the dominant cost is per-unique-expert weight
+processing, and the S positions of a verify batch route to mostly different
+experts: measured per-forward expert time scaled almost linearly with S
+(80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt).
+There is no amortization to collect, and the MTP head's own forward (int8
+experts, one extra layer per draft token) adds cost on top. Even 79%
+acceptance at DRAFT=1 nets -5%.
+
+Two useful facts survive the negative result. First, the int8 head is
+genuinely good: 69-79% chained top-1 acceptance on clean text (the int8
+shards stay installed; the int4 originals were defective). Second, the
+conclusion inverts if routed-expert compute per extra position ever becomes
+cheap β€” e.g. Tensor Core grouped GEMM computing an S=4 verify batch at
+near-S=1 cost on GPU-resident experts. Revisit speculation only after that
+lands. Until then, benchmark and serve runs on this machine should set
+`DRAFT=0` explicitly: with a working MTP head present, the engine's
+`DRAFT=-1` default auto-enables 3 drafts and silently costs 10-37%.
+
+Logs: `/data/test/colibri-six-gpu-ab-mtp-*-20260712.log` (phase 1),
+`/data/test/colibri-six-gpu-ab-mtp8-*-20260713.log` (phase 2).
+
+## AVX-512 int4 kernel qualification (July 13)
+
+The candidate kernel from section "Candidate not yet merged" was put through
+the three measurement gates it owed. The working-tree build exposes it behind
+a runtime switch (`I4_ACC512=0/1`, default on), so every comparison below
+uses one binary.
+
+**Numeric error (passed, better than the incumbent).** A standalone harness
+compared the AVX-512 dual-accumulator kernel and the engine's sequential
+scalar-f32 order against a double-precision oracle over the real expert row
+shapes (I=6144 and I=2048), 70k+ random rows across activation scales from
+0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign
+cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the
+scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3,
+RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less
+rounding than sequential summation, so the "output divergence" is the new
+kernel being closer to exact, not further.
+
+**Quality (passed).** SCORE log-likelihood over 8 multi-domain passages (251
+continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off),
+perplexity 5.99 vs 5.98 β€” a 0.24% difference with per-passage signs split
+4/4. Statistically indistinguishable.
+
+**Throughput (+4% mean, +7% where it applies).** Three fixed greedy prompts,
+`TEMP=0 DRAFT=0`, 96 tokens, ABAB-interleaved on/off, two repeats each:
+
+| Prompt | on | off | gain |
+| --- | ---: | ---: | ---: |
+| sky | 6.08 / 6.08 | 5.75 / 5.62 | +7.0% |
+| water cycle | 6.51 / 6.50 | 6.53 / 6.52 | 0% |
+| rivers | 5.79 / 5.82 | 5.24 / 5.57 | +7.4% |
+
+The earlier +15% estimate came from a low 5.12 baseline and is retired. The
+gain is workload-dependent for a structural reason: the kernel only
+accelerates experts computed on CPU, and the water-cycle prompt's routing
+happens to be served almost entirely from the GPU hot tier. This is the
+clearest evidence yet that the next big lever is making GPU experts a
+compute tier (Tensor Core grouped GEMM), which would also shrink every
+CPU-side gain β€” and, by making multi-position verify batches cheap,
+would re-open the MTP speculation door (69-79% acceptance is already
+banked above).
+
+Verdict: the kernel is numerically superior, quality-neutral, and worth
++4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only
+open item is the policy call that outputs are not bit-identical to the old
+(worse) accumulation order. Logs:
+`/data/test/colibri-six-gpu-ab-avx512q-*-20260713.log`,
+`/data/test/score-acc{0,1}.out`, harness `/data/test/i4_numeric_test.c`.
+
+## Why six 5090s did not match two RTX PRO 6000s
+
+The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000
+Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32
+GB. Both machines may total 192 GB, but expert residency is constrained per
+rank and runtime state is replicated. Six smaller memory islands are not
+equivalent to two large ones.
+
+The 5090 has strong compute and is cost-effective for concurrency, but this
+workload is dominated by resident expert capacity and miss recovery. Changing
+TP/PP layout cannot repair a policy that replays an entire step after any
+expert miss.
+
+## Conclusions for colibri
+
+The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with
+different bandwidth and latency, so a model larger than VRAM can run. The
+full-resident experiments strengthened it: on this machine the whole int4
+model fits in VRAM+RAM, and once disk left the decode path colibri at
+6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s),
+because colibri never replays a whole step on an expert miss.
+
+The remaining bottleneck is no longer storage or scheduling β€” it is the int4
+expert matmul on CPU (56% of step time, 84.9% of CPU samples in `matmul_qt`),
+with attention (25%) next in line. Priorities:
+
+1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity
+ evaluation, fixed greedy repeat runs) and take its ~15% if it passes.
+2. Add finer counters to CPU expert execution (experts per layer, rows per
+ expert, per-matrix time) to guide kernel work.
+3. Keep all A/B tests on fixed `TEMP=0` prompts and fixed token counts; never
+ mix sampled runs into comparisons.
+4. Once the expert path is stably above about 7 tok/s, optimize attention
+ projection/RoPE and score-softmax-value.
+
+Twenty to thirty tokens per second has still not been demonstrated on this
+machine; 6.84 tok/s (256-token greedy) is the current ceiling.
+
+## Last known lab state
+
+The fastest configuration overall is colibri full-resident in
+`/data/test/colibri-full-resident` (6.28-6.84 tok/s; stop the
+`glm52-moet-real` service before benchmarking it and restore the service
+afterwards). The fastest vLLM-Moet configuration was the TP4 service on port
+8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle.
+Preserve both pack directories until the PP experiment no longer needs to be
+reproduced. Raw A/B logs live under `/data/test/colibri-six-gpu-ab-*.log`.
+These paths describe the lab machine and are not repository dependencies.