Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: GLM-5.2 6x RTX 5090 full-residency experiment β 6.84 tok/s, honest dead-ends log (#94)
ZacharyZcR3fb5a001060c
diff --git a/README.md b/README.mdindex b8d3dc6d..66bf1efe 100644--- a/README.md+++ b/README.md@@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)+Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) β full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.+| metric | value ||---|---|| model on disk (int4 container) | ~370 GB |diff --git a/docs/experiments/glm52-6x5090-2026-07-12.md b/docs/experiments/glm52-6x5090-2026-07-12.mdnew file mode 100644index 00000000..2c86c7cc--- /dev/null+++ b/docs/experiments/glm52-6x5090-2026-07-12.md@@ -0,0 +1,302 @@+# GLM-5.2 on 6x RTX 5090++This note records the July 12, 2026 experiments that compared colibri with+vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon+Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe+storage. It exists to preserve the measured facts and avoid repeating the same+dead ends.++The day had two halves. The morning runs (below) measured colibri at 0.12+tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on+colibri (see "colibri full-resident experiments") removed disk from the decode+path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark+and 6.84 tok/s on a 256-token run β the fastest single-request result measured+on this machine so far.++## Models and artifacts++- Official NVFP4 model: `/data/models/GLM-5.2-NVFP4` (434 GB, 47 shards)+- colibri INT4 model: `/data/models/GLM-5.2-colibri-int4` (144 shards)+- vLLM-Moet TP4 pack: `/data/models/GLM-5.2-NVFP4-packs` (190 GB)+- vLLM-Moet TP2/PP3 pack: `/data/models/GLM-5.2-NVFP4-packs-tp2pp3` (190 GB)+- vLLM-Moet checkout: `/data/test/vLLM-Moet`+- Built image: `vllm-moet-sm120:v024`++The official model was downloaded through `hf-mirror.com` with+`HF_HUB_DISABLE_XET=1`. Direct Hugging Face access failed and Xet/CAS stalled;+ordinary HTTP from the mirror sustained roughly 22-46 MB/s.++## Kernel validation++All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke+tests. Representative correctness results were:++| Path | Shape | Max relative error | Cosine similarity |+| --- | --- | ---: | ---: |+| Decode | T=2 | 2.086e-2 | 0.999878 |+| NVFP4 delta | T=2 | 1.818e-2 | 0.999884 |+| AFRAG prefill | T=128 | 2.591e-2 | 0.999883 |++The kernels work on SM120. The performance limitation below is therefore not+an unsupported-GPU fallback.++## Real generation results++All decode rates below are for a single request, not aggregate throughput.++| Runtime and layout | GPU use | TTFT | Decode rate | Important observation |+| --- | --- | ---: | ---: | --- |+| colibri INT4 (morning, partial residency) | 6 GPUs | about 42 s | 0.12 tok/s | GPU hot tier was active, but dense compute remained on CPU |+| colibri INT4 (afternoon, full residency) | 6 GPUs + 24 cores | β | 6.28-6.84 tok/s | all 19,456 experts resident in VRAM+RAM, zero disk waits |+| vLLM-Moet TP4, 12 GiB expert cache/card | GPUs 0-3 | 2.03 s | median 2.5 tok/s | 25.3% coverage; 100% whole-step replay |+| vLLM-Moet TP4, 14 GiB expert cache/card | GPUs 0-3 | not remeasured | median 2.6 tok/s | 29.5% coverage; replay still 100% |+| vLLM-Moet TP2 x PP3 | all 6 GPUs | 5.39 s | 1.78 tok/s | more local coverage, but serial PP and communication made latency worse |++The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only+about 0.1 tok/s because every decode step still missed experts and replayed the+whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not+reduce replay enough to offset pipeline and PYNCCL communication costs.++The morning colibri run generated 16 tokens in 132.46 seconds. Its measured+hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the+expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk+reads (29.31 s), and attention (14.28 s). The afternoon experiments below+showed the dominant problem was expert loading from disk, not the CUDA path+itself.++## colibri full-resident experiments (afternoon)++Test directory: `/data/test/colibri-full-resident`. Related PR:+[#80](https://github.com/JustVugg/colibri/pull/80) (on top of PR #68). All A/B+runs used the same binary flags, environment, prompt, `TEMP=0`, and token+count; only decode throughput is compared, never load or prefill time.++### The optimization ladder (each step cumulative)++| Change | Decode rate | Evidence |+| --- | ---: | --- |+| Baseline: 150 GB VRAM + 150 GB RAM fixed placement | 2.30 tok/s | 4.15 s of disk waits per 20 tokens |+| Full residency: all 19,456 experts in VRAM+RAM | 5.77 tok/s | 100% expert hit, zero disk waits |+| In-session dynamic GPU expert repinning (`REPIN=16`) | 6.00 tok/s | expert compute 6.76 s β 5.96 s |+| 24 physical cores, `OMP_PROC_BIND=spread OMP_PLACES=cores` | +39.6% vs unpinned | unpinned 3.64 β pinned 5.08 tok/s on the same prompt |+| Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT) | 6.05-6.08 tok/s | GPU expert calls 36,865 β 37,285 |+| Decode swap cap 32 β 16 per repin round | 6.10-6.28 tok/s | swap cost 0.18 β 0.09 s/round |++The final configuration measured 6.28 tok/s on the fixed 96-token greedy+benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed+costs; always state the token count when quoting a rate).++Winning layout, found automatically:++```text+GPU experts: 9,343 / 19,456 (176.73 GB across six cards)+RAM experts: 10,113 / 19,456 (about 191.3 GB)+Disk service/wait during decode: 0 s+```++Run configuration: `COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto+CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto`, with+`OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores` (now the default+generated by `resource_plan.py`).++### Directions tested and rejected (all reverted)++- Faster (`REPIN=8`) or slower (`REPIN=32`) repinning, and 64 swaps per+ round: swap/sync overhead or stale GPU expert sets lose more than they gain.+ 16 tokens between repins and 16 swaps per round is the current local optimum+ (8 swaps also regresses: fewer GPU calls outweigh the sync savings).+- A second prefill correction pass: overfits the short prompt's routing and+ disturbs the history-based layout (5.86 tok/s).+- Lazy demotion of evicted GPU experts (4.77 tok/s) and GPUβRAM D2H recovery+ (6.15 tok/s): both cost as much as re-reading from page cache.+- CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30+ tok/s); GPU expert groups are too small a fraction of step time, and the+ extra threads compete with the 24 CPU workers.+- OpenMP restructuring of RAM experts (row batching across experts,+ persistent teams): neutral or negative; libgomp already reuses hot teams,+ and interleaving experts in one static schedule destroys per-expert+ NUMA/cache locality.+- `numactl --interleave=all`, 2 MB transparent huge pages, 12-core runs,+ smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI+ int4Γint8 decode: all neutral or negative. Capacity and TLB are not the+ limit; raw history counts predict routing better than normalized ones.+- Next-layer expert prediction with GPU staging: the routing signal is real+ (70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the+ expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth+ revisiting only with dedicated streams that do not stall attention.++### Candidate not yet merged: AVX-512 int4 kernel++The lossless int4βfloat RAM expert kernel still uses 256-bit AVX2. A 512-bit+unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 β 5.89+tok/s, expert time β19.7%), and a dual-accumulator variant added a little more+(5.89 β 5.94 tok/s at 96 tokens). However, the changed accumulation order+produces small float differences and the 64-token outputs eventually diverge,+so it cannot be claimed bit-exact. It stays out of the tree until it has+numeric-error unit tests, a perplexity/token-consistency evaluation, and an+explicit decision on whether float-accumulation differences are acceptable.+Row blocking (2/4 output rows) looked good in short microbenchmarks but the+gain disappeared over 10,000 sustained iterations and in a 256-token run+(6.84 vs 6.83 tok/s) β a turbo/cache artifact, reverted.++### Where the time goes now++At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other+2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of+CPU time in `matmul_qt` and 8.8% in `rope_interleave`; a per-thread RoPE+sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention+β5.0%) with no math change. Scheduling-level CPU optimizations are exhausted β+the remaining expert time is the int4 matmul kernel itself.++## MTP speculative decoding at full residency (July 13)++Motivation: with all experts resident and disk waits at zero, the old+objection to speculation (a rejected draft wastes a disk read) no longer+applies, so the July 3 negative result deserved a retest. All runs used the+same binary and environment as the 16-swap configuration above, `TEMP=0`,+96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy+output degenerates into repetition, and a non-degenerate "water cycle"+prompt).++Phase 1 found the installed MTP head unusable: acceptance 0-4% at every+depth, exactly the known int4-head defect from issue #8 (the three+`out-mtp-*.safetensors` byte sizes matched the "unusable" fingerprint).+Speculation with the broken head costs 15-18% throughput β pure verification+tax.++Phase 2 replaced the head with the community int8 MTP shards+(`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`, ~10 GB; originals kept as+`.int4.bak`). The head then worked as designed β and speculation still lost+at every depth:++| Prompt | DRAFT | tok/s | tok/forward | Acceptance |+| --- | ---: | ---: | ---: | ---: |+| water cycle | 0 | 6.79 | 1.01 | β |+| water cycle | 1 | 6.45 | 1.81 | 79% |+| water cycle | 2 | 5.65 | 2.29 | 64% |+| water cycle | 3 | 6.13 | 3.10 | 69% |+| sky (degenerate) | 0 | 6.12 | 1.01 | β |+| sky (degenerate) | 1 | 5.51 | 1.75 | 73% |+| sky (degenerate) | 2 | 4.42 | 1.88 | 44% |+| sky (degenerate) | 3 | 3.86 / 3.81 | 2.13 | 38% |++The failure is structural, not a tuning problem. In a dense model a verify+batch of S tokens rereads the same weights once, so speculation amortizes+the dominant cost. In this MoE the dominant cost is per-unique-expert weight+processing, and the S positions of a verify batch route to mostly different+experts: measured per-forward expert time scaled almost linearly with S+(80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt).+There is no amortization to collect, and the MTP head's own forward (int8+experts, one extra layer per draft token) adds cost on top. Even 79%+acceptance at DRAFT=1 nets -5%.++Two useful facts survive the negative result. First, the int8 head is+genuinely good: 69-79% chained top-1 acceptance on clean text (the int8+shards stay installed; the int4 originals were defective). Second, the+conclusion inverts if routed-expert compute per extra position ever becomes+cheap β e.g. Tensor Core grouped GEMM computing an S=4 verify batch at+near-S=1 cost on GPU-resident experts. Revisit speculation only after that+lands. Until then, benchmark and serve runs on this machine should set+`DRAFT=0` explicitly: with a working MTP head present, the engine's+`DRAFT=-1` default auto-enables 3 drafts and silently costs 10-37%.++Logs: `/data/test/colibri-six-gpu-ab-mtp-*-20260712.log` (phase 1),+`/data/test/colibri-six-gpu-ab-mtp8-*-20260713.log` (phase 2).++## AVX-512 int4 kernel qualification (July 13)++The candidate kernel from section "Candidate not yet merged" was put through+the three measurement gates it owed. The working-tree build exposes it behind+a runtime switch (`I4_ACC512=0/1`, default on), so every comparison below+uses one binary.++**Numeric error (passed, better than the incumbent).** A standalone harness+compared the AVX-512 dual-accumulator kernel and the engine's sequential+scalar-f32 order against a double-precision oracle over the real expert row+shapes (I=6144 and I=2048), 70k+ random rows across activation scales from+0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign+cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the+scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3,+RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less+rounding than sequential summation, so the "output divergence" is the new+kernel being closer to exact, not further.++**Quality (passed).** SCORE log-likelihood over 8 multi-domain passages (251+continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off),+perplexity 5.99 vs 5.98 β a 0.24% difference with per-passage signs split+4/4. Statistically indistinguishable.++**Throughput (+4% mean, +7% where it applies).** Three fixed greedy prompts,+`TEMP=0 DRAFT=0`, 96 tokens, ABAB-interleaved on/off, two repeats each:++| Prompt | on | off | gain |+| --- | ---: | ---: | ---: |+| sky | 6.08 / 6.08 | 5.75 / 5.62 | +7.0% |+| water cycle | 6.51 / 6.50 | 6.53 / 6.52 | 0% |+| rivers | 5.79 / 5.82 | 5.24 / 5.57 | +7.4% |++The earlier +15% estimate came from a low 5.12 baseline and is retired. The+gain is workload-dependent for a structural reason: the kernel only+accelerates experts computed on CPU, and the water-cycle prompt's routing+happens to be served almost entirely from the GPU hot tier. This is the+clearest evidence yet that the next big lever is making GPU experts a+compute tier (Tensor Core grouped GEMM), which would also shrink every+CPU-side gain β and, by making multi-position verify batches cheap,+would re-open the MTP speculation door (69-79% acceptance is already+banked above).++Verdict: the kernel is numerically superior, quality-neutral, and worth++4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only+open item is the policy call that outputs are not bit-identical to the old+(worse) accumulation order. Logs:+`/data/test/colibri-six-gpu-ab-avx512q-*-20260713.log`,+`/data/test/score-acc{0,1}.out`, harness `/data/test/i4_numeric_test.c`.++## Why six 5090s did not match two RTX PRO 6000s++The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000+Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32+GB. Both machines may total 192 GB, but expert residency is constrained per+rank and runtime state is replicated. Six smaller memory islands are not+equivalent to two large ones.++The 5090 has strong compute and is cost-effective for concurrency, but this+workload is dominated by resident expert capacity and miss recovery. Changing+TP/PP layout cannot repair a policy that replays an entire step after any+expert miss.++## Conclusions for colibri++The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with+different bandwidth and latency, so a model larger than VRAM can run. The+full-resident experiments strengthened it: on this machine the whole int4+model fits in VRAM+RAM, and once disk left the decode path colibri at+6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s),+because colibri never replays a whole step on an expert miss.++The remaining bottleneck is no longer storage or scheduling β it is the int4+expert matmul on CPU (56% of step time, 84.9% of CPU samples in `matmul_qt`),+with attention (25%) next in line. Priorities:++1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity+ evaluation, fixed greedy repeat runs) and take its ~15% if it passes.+2. Add finer counters to CPU expert execution (experts per layer, rows per+ expert, per-matrix time) to guide kernel work.+3. Keep all A/B tests on fixed `TEMP=0` prompts and fixed token counts; never+ mix sampled runs into comparisons.+4. Once the expert path is stably above about 7 tok/s, optimize attention+ projection/RoPE and score-softmax-value.++Twenty to thirty tokens per second has still not been demonstrated on this+machine; 6.84 tok/s (256-token greedy) is the current ceiling.++## Last known lab state++The fastest configuration overall is colibri full-resident in+`/data/test/colibri-full-resident` (6.28-6.84 tok/s; stop the+`glm52-moet-real` service before benchmarking it and restore the service+afterwards). The fastest vLLM-Moet configuration was the TP4 service on port+8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle.+Preserve both pack directories until the PP experiment no longer needs to be+reproduced. Raw A/B logs live under `/data/test/colibri-six-gpu-ab-*.log`.+These paths describe the lab machine and are not repository dependencies.