Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: add GLM-5.2 experiment report on 4x RTX A6000 (Ampere) + EPYC 7402P
ThefloorMinere982047b192e
diff --git a/docs/experiments/glm52-4xa6000-2026-08-02.md b/docs/experiments/glm52-4xa6000-2026-08-02.mdnew file mode 100644index 00000000..1d9b7a15--- /dev/null+++ b/docs/experiments/glm52-4xa6000-2026-08-02.md@@ -0,0 +1,242 @@+# GLM-5.2 on 4Γ RTX A6000 (Ampere) + EPYC 7402P β 2026-08-02++**Engine:** colibri v1.3.0, commit `ecade075cfc2eae684097ea7de5570c3786ce199`+**Model:** `mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp` β 142 shards, 429.3 GB on disk+**Build:** `make glm CUDA=1 CUDA_ARCH=sm_86 CUDA_HOME=<cuda13>` (see *Build notes* below)++This report adds an **Ampere multi-GPU + pre-AVX-512 CPU** data point, which the existing+experiment set does not cover (6Γ5090 Blackwell, Metal M5 Max, Strix Halo, 9950X3D+5090).++Its main contribution is not a throughput number, however: it is a **methodological finding+that invalidates naive A/B benchmarking on this engine** (Β§6).++---++## 1. Hardware++| | |+|---|---|+| CPU | AMD EPYC 7402P β Zen 2, 24c/48t, socket SP3, **AVX2, no AVX-512**, single NUMA node |+| RAM | 8Γ32 GB DDR4-**2400** (native speed grade), 8/8 slots populated β 263.8 GB / 251.6 GiB, ~153 GB/s |+| GPU | 4Γ RTX A6000 48 GB (sm_86), all at **PCIe Gen4 x16**, driver 580.173.02 |+| NVMe A | Gen4-capable drive negotiated at **PCIe 3.0 x4** β `/` |+| NVMe B | same model, negotiated at **PCIe 3.0 x2** β model directory |+| OS | Ubuntu 24.04, kernel 6.8.0-136 |++Note: the GPUs negotiate Gen4 x16 while both NVMe sit on Gen3 links β a slot/routing+limitation, not a drive limitation.++## 2. Storage baseline++`fio --direct=1`, and `iobench` (19 MB blocks = expert-sized, O_DIRECT):++| | NVMe A (x4) | NVMe B (x2) |+|---|---|---|+| seq read 1M QD8 | 3232 MB/s | β |+| randread 4k QD1 | 9,880 IOPS / 0.090 ms | 4,605 IOPS / 0.207 ms |+| randread 4k QD32 Γ4 | 167,098 IOPS | 46,482 IOPS (saturated) |+| **randread 4M QD4** | **2619 MB/s / 5.79 ms** | **1521 MB/s / 10.44 ms** |+| `iobench` 19M, 16 threads | **2.86 GB/s / 7.0 ms per expert** | 1.60 GB/s / 12.2 ms |++## 3. Protocol++Full service restart β wait for API β 1 warm-up generation β **5 measured generations**.+Fixed prompt, greedy (`COLI_TEMP=0`), `max_tokens=64`, **single client**, `--ctx 32768`.+Median reported. RSS and VRAM sampled after load.++> **Caveat (important):** two measurement clients running concurrently against the same+> engine produce erratic results (0.43β1.04 tok/s) because the engine serialises requests.+> Any concurrent measurement must be discarded.++## 4. Configuration matrix (9 configs, same session, ~1 h)++| | Configuration | tok/s | RSS | VRAM |+|---|---|---|---|---|+| E1 | D1 + `RAM_GB=205` | **5.46** | 190 GB | 177 GB |+| D1 | C2 minus `URING`/`PILOT*` | 5.38 | 176 GB | 177 GB |+| D4 | C2 + `RAM_GB=205` | 4.45 | 190 GB | 177 GB |+| D2/D3/D5 | +`TC_INT4` / +explicit `OMP close` / no `CUDA_PIPE` | 4.30 | 176 GB | 177 GB |+| C2 | C1 + `CUDA_DENSE=1` | 4.26 | 176 GB | 177 GB |+| C0 | starting point (`cap 64`, fixed VRAM, no `PIN`) | 1.53 | 99 GB | 171 GB |+| C1 | max RAM/VRAM, **without** `CUDA_DENSE` | 1.41 | 176 GB | 188 GB |+| β | `SERVE_BATCH=1`, `KV_SLOTS=4`, 1/2/4 concurrent | 1.32 / 1.51 / 1.46 | 151 GB | β |++### Only two levers actually moved the number++**`CUDA_DENSE=1` β Γ2.8 (1.53 β 4.26).** By default the engine reports+`[CUDA] mode: routed experts only (resident dense on CPU)`. On a CPU-bound host the dense+path of all 78 layers runs on the CPU while 192 GB of VRAM serve experts only. This flag is+**not mentioned in the public README**; it was the single most profitable setting found.++**Removing `URING` + `PILOT` + `PILOT_REAL` + `PILOT_TWO` β +26 % (4.26 β 5.38).** These+hide disk latency. Once experts are resident the disk reads **0 MB/s during decode**, so the+prefetch/overlap machinery only consumes CPU β the scarce resource. `DIRECT=1 PIPE=1` kept.++### Neutral (Β±1 %, within noise)++`COLI_CUDA_TC_INT4` (the W4A4 path needs β₯16 rows per expert; S=1 decode provides one),+explicit `OMP_PROC_BIND=close` (the engine already sets it), and removing `COLI_CUDA_PIPE`+(its decode gate is already device-count aware on 4 devices).++### Counter-intuitive results++1. **Maximising RAM+VRAM *degrades* when the dense path stays on CPU**: C1 (176 GB RSS,+ 188 GB VRAM) β 1.41 tok/s vs C0 (99 GB RSS) β 1.53.+2. **Gains do not compose.** `RAM_GB` 190β205 is worth +4.5 % on the C2 base but only+ +1.5 % once `URING`/`PILOT` are removed β both targeted the same residual disk misses.+3. **Concurrency does not scale throughput**: 1.32 / 1.51 / 1.46 tok/s aggregate at 1/2/4.++## 5. Where the bottleneck actually is++Sampled during decode, after tuning:++| | before (C0) | after (D1) |+|---|---|---|+| CPU | 638 % | 292β490 % (4β5 cores of 24) |+| GPU | 2 % / 80 W | 20β22 % / 131 W |+| Disk | 6 MB/s | **0 MB/s** |++**Nothing is saturated.** The limiter is serialisation latency β 78 sequential layers,+CPUβGPU round trips, tiny matmuls at S=1 β not any single bandwidth. This explains both the+failure of batching and the uselessness of maximising memory.++### Prefill is a different regime entirely++Decode activates 8 of 256 experts per layer; consecutive tokens reuse them, so the learned+pinned set covers them. Prefill does not:++```+P(an expert is missed by N tokens) = (1 β 8/256)^N+P(512) β 1e-7 P(8595) β e^-273+```++**A 512-token slice already touches every expert of every layer.** Prefill therefore gets no+benefit from caching: it pays the non-resident fraction of the *entire model*, once per slice.++Measured: **2.6β4.3 tok/s prefill** at `--ctx 262144` with `COLI_PREFILL_CHUNK=512` β i.e.+35β55 minutes before first token on an 8,595-token prompt. Raising the chunk to 2048 reduces+the number of full-model sweeps from 17 to 5.++During prefill **only one GPU is active** (100 % / 200 W) while the other three sit at+0 % / ~80 W: aggregate PCIe and compute budget is unused while the engine waits on expert+loads.++Prefix caching itself works correctly in steady state (`prefix 8556/8562 token, prefill 6`):+the prefill cost is paid once per conversation, not per turn.++## 6. **The finding that invalidates naive A/B benchmarking**++A configuration **identical in memory placement** to the best result above+(VRAM 178 GB, RAM 190 GB) measured **2.56 tok/s instead of 5.46**, with no memory parameter+changed.++The hidden variable is `.coli_usage`, the persistent learned routing profile that seeds the+pinned hot store at startup:++| Moment | Recorded selections |+|---|---|+| first start | 58,240 |+| end of the 9-config matrix (~1 h later) | 238,840 |+| after a day of real conversational use | **17,697,640** |++The pinned set had re-tuned itself onto the real workload, so the synthetic benchmark prompt+became out-of-distribution for the cache.++**Consequences for anyone benchmarking this engine:**++- `.coli_usage` **persists across restarts** and grows monotonically. Two configurations+ measured hours apart are not comparable, even with identical flags.+- `CONTRIBUTING.md` asks for a *warm-up policy*, but warm-up alone is insufficient: the+ profile must be **snapshotted and restored** before each configuration under test.+- Absolute numbers in this report should be read as *within-session rankings*, not as+ portable throughput figures. The 9-config matrix ran back-to-back in one hour on one+ prompt, so its ordering is internally consistent; cross-session comparisons are not.++Two hypotheses were formed and then refuted by this: "total residency governs throughput"+(refuted: 98 % residency measured *slower* than 91 %) and "VRAM residency governs throughput"+(refuted by the identical-placement result). An arithmetic check also shows the VRAM/RAM+bandwidth differential accounts for ~1.3 ms of the 77 ms per-token gap β under 2 %.++**Suggested engine-side fix:** expose per-tier access counts (VRAM / RAM / disk hits) through+`/profile` in a scriptable form. The dashboard displays hit rate; making it queryable would+let contributors report the metric that actually decides placement trade-offs instead of+reporting occupied gigabytes.++## 7. Context vs residency trade-off++```+available = 192 GB VRAM + 251 GB RAM = 443 GB+required = 407 (experts) + 83 (KV @262144) + 12 (dense) + 10 (OS) = 512 GB+deficit = 69 GB+```++At `--ctx 32768` the KV costs 11 GB and everything fits. Measured: **5.46 tok/s at 32768 vs+0.86 tok/s at 262144**. The engine reports the collapse itself:++```+[RAM_GB=215.0] WARNING: cap=1 is the floor, projected peak 216.7 GB is 1.7 GB OVER+the budget (resident 132.2 GB + reserve 82.8 GB).+PIN_GB is inflating the resident set: lower it or drop it.+```++### KV cache size++MLA stores `kv_lora_rank 512 + rope 64 = 576` values per token per layer, Γ78 layers =+44,928 values/token.++| dtype | per token | @262,144 |+|---|---|---|+| fp32 | 179.7 KB | **47.1 GB** |+| fp16 | 89.9 KB | 23.6 GB |+| int8 | 44.9 KB | 11.8 GB |++The engine reports **47.7 GB**, i.e. fp32 β confirmed at `colibri.c:2277`+(`...*kvl*4`). On top of that, `kvb_all` at `colibri.c:2766` allocates+`Tk Γ HΓ(qk_nope+v_head) = Tk Γ 28,672` floats = 114.7 KB **per context position**, i.e.+**30.1 GB** at 262,144 β matching the reported `kvb 30.1`. That buffer materialises the full+per-head K/V, which is what MLA exists to avoid; `colibri.c:2568` already notes the+alternative cost `O(TΒ·kv_lora)` instead of `O(TΒ·HΒ·(nope+vh))`.++Comparison on the same model and same latent attention: llama.cpp with `-ctk f16` uses+**0.098 MB/token** vs colibri's **0.316 MB/token** β **Γ3.2**, i.e. ~57 GB avoidable at+262,144. That is 83 % of the 69 GB deficit above: with an fp16 KV and no `kvb_all`, long+context and full expert residency would be simultaneously reachable on this host.++## 8. Build notes (Linux, CUDA)++`nvcc` 12.6 + gcc 13 fails with `identifier "_Float32" is undefined`;+`-allow-unsupported-compiler` produces 100 errors. The `BUILD-cuda-glibc241.md` recipe works+and needs no root if the prefix is a user directory:++```sh+micromamba create -y -p ~/cuda13 -c nvidia -c conda-forge cuda-nvcc=13 cuda-cudart-dev=13+ln -sfn ~/cuda13/lib ~/cuda13/lib64+make glm CUDA=1 CUDA_ARCH=sm_86 CUDA_HOME=~/cuda13+```++The web dashboard needs Node β₯ 20.12 (`styleText` from `node:util`); Ubuntu 24.04 ships+18.19, so `npm run build` fails on a stock install.++## 9. Final configuration++```sh+CTX=32768 # 262144 measured 0.86 tok/s decode, 2.9 tok/s prefill+COLI_CUDA=1+COLI_GPUS=0,1,2,3+CUDA_DENSE=1 # Γ2.8 β the decisive lever+COLI_CUDA_ATTN=1+COLI_CUDA_PIPE=2+COLI_CUDA_TC_W4A16=1+CUDA_EXPERT_GB=188 # NOT "auto" β see issue on prefill scratch+PIN=auto+PIN_GB=all+RAM_GB=200+DIRECT=1+PIPE=1 # no URING, no PILOT* β +26 % on this host+COLI_PREFILL_CHUNK=2048+```++Operational notes: the engine does not release VRAM immediately on stop β restarting within+~30 s fails with `Failed with result 'exit-code'` (`Restart=on-failure` recovers). And+`coli plan` computes against *currently free* VRAM, so its output is meaningless while an+engine instance is running.