Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: add GLM-5.2 experiment report on 4x RTX A6000 (Ampere) + EPYC 7402P

ThefloorMinere982047b192e
diff --git a/docs/experiments/glm52-4xa6000-2026-08-02.md b/docs/experiments/glm52-4xa6000-2026-08-02.md
new file mode 100644
index 00000000..1d9b7a15
--- /dev/null
+++ b/docs/experiments/glm52-4xa6000-2026-08-02.md
@@ -0,0 +1,242 @@
+# GLM-5.2 on 4Γ— RTX A6000 (Ampere) + EPYC 7402P β€” 2026-08-02
+
+**Engine:** colibri v1.3.0, commit `ecade075cfc2eae684097ea7de5570c3786ce199`
+**Model:** `mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp` β€” 142 shards, 429.3 GB on disk
+**Build:** `make glm CUDA=1 CUDA_ARCH=sm_86 CUDA_HOME=<cuda13>` (see *Build notes* below)
+
+This report adds an **Ampere multi-GPU + pre-AVX-512 CPU** data point, which the existing
+experiment set does not cover (6Γ—5090 Blackwell, Metal M5 Max, Strix Halo, 9950X3D+5090).
+
+Its main contribution is not a throughput number, however: it is a **methodological finding
+that invalidates naive A/B benchmarking on this engine** (Β§6).
+
+---
+
+## 1. Hardware
+
+| | |
+|---|---|
+| CPU | AMD EPYC 7402P β€” Zen 2, 24c/48t, socket SP3, **AVX2, no AVX-512**, single NUMA node |
+| RAM | 8Γ—32 GB DDR4-**2400** (native speed grade), 8/8 slots populated β€” 263.8 GB / 251.6 GiB, ~153 GB/s |
+| GPU | 4Γ— RTX A6000 48 GB (sm_86), all at **PCIe Gen4 x16**, driver 580.173.02 |
+| NVMe A | Gen4-capable drive negotiated at **PCIe 3.0 x4** β†’ `/` |
+| NVMe B | same model, negotiated at **PCIe 3.0 x2** β†’ model directory |
+| OS | Ubuntu 24.04, kernel 6.8.0-136 |
+
+Note: the GPUs negotiate Gen4 x16 while both NVMe sit on Gen3 links β€” a slot/routing
+limitation, not a drive limitation.
+
+## 2. Storage baseline
+
+`fio --direct=1`, and `iobench` (19 MB blocks = expert-sized, O_DIRECT):
+
+| | NVMe A (x4) | NVMe B (x2) |
+|---|---|---|
+| seq read 1M QD8 | 3232 MB/s | β€” |
+| randread 4k QD1 | 9,880 IOPS / 0.090 ms | 4,605 IOPS / 0.207 ms |
+| randread 4k QD32 Γ—4 | 167,098 IOPS | 46,482 IOPS (saturated) |
+| **randread 4M QD4** | **2619 MB/s / 5.79 ms** | **1521 MB/s / 10.44 ms** |
+| `iobench` 19M, 16 threads | **2.86 GB/s / 7.0 ms per expert** | 1.60 GB/s / 12.2 ms |
+
+## 3. Protocol
+
+Full service restart β†’ wait for API β†’ 1 warm-up generation β†’ **5 measured generations**.
+Fixed prompt, greedy (`COLI_TEMP=0`), `max_tokens=64`, **single client**, `--ctx 32768`.
+Median reported. RSS and VRAM sampled after load.
+
+> **Caveat (important):** two measurement clients running concurrently against the same
+> engine produce erratic results (0.43–1.04 tok/s) because the engine serialises requests.
+> Any concurrent measurement must be discarded.
+
+## 4. Configuration matrix (9 configs, same session, ~1 h)
+
+| | Configuration | tok/s | RSS | VRAM |
+|---|---|---|---|---|
+| E1 | D1 + `RAM_GB=205` | **5.46** | 190 GB | 177 GB |
+| D1 | C2 minus `URING`/`PILOT*` | 5.38 | 176 GB | 177 GB |
+| D4 | C2 + `RAM_GB=205` | 4.45 | 190 GB | 177 GB |
+| D2/D3/D5 | +`TC_INT4` / +explicit `OMP close` / no `CUDA_PIPE` | 4.30 | 176 GB | 177 GB |
+| C2 | C1 + `CUDA_DENSE=1` | 4.26 | 176 GB | 177 GB |
+| C0 | starting point (`cap 64`, fixed VRAM, no `PIN`) | 1.53 | 99 GB | 171 GB |
+| C1 | max RAM/VRAM, **without** `CUDA_DENSE` | 1.41 | 176 GB | 188 GB |
+| β€” | `SERVE_BATCH=1`, `KV_SLOTS=4`, 1/2/4 concurrent | 1.32 / 1.51 / 1.46 | 151 GB | β€” |
+
+### Only two levers actually moved the number
+
+**`CUDA_DENSE=1` β€” Γ—2.8 (1.53 β†’ 4.26).** By default the engine reports
+`[CUDA] mode: routed experts only (resident dense on CPU)`. On a CPU-bound host the dense
+path of all 78 layers runs on the CPU while 192 GB of VRAM serve experts only. This flag is
+**not mentioned in the public README**; it was the single most profitable setting found.
+
+**Removing `URING` + `PILOT` + `PILOT_REAL` + `PILOT_TWO` β€” +26 % (4.26 β†’ 5.38).** These
+hide disk latency. Once experts are resident the disk reads **0 MB/s during decode**, so the
+prefetch/overlap machinery only consumes CPU β€” the scarce resource. `DIRECT=1 PIPE=1` kept.
+
+### Neutral (Β±1 %, within noise)
+
+`COLI_CUDA_TC_INT4` (the W4A4 path needs β‰₯16 rows per expert; S=1 decode provides one),
+explicit `OMP_PROC_BIND=close` (the engine already sets it), and removing `COLI_CUDA_PIPE`
+(its decode gate is already device-count aware on 4 devices).
+
+### Counter-intuitive results
+
+1. **Maximising RAM+VRAM *degrades* when the dense path stays on CPU**: C1 (176 GB RSS,
+ 188 GB VRAM) β†’ 1.41 tok/s vs C0 (99 GB RSS) β†’ 1.53.
+2. **Gains do not compose.** `RAM_GB` 190β†’205 is worth +4.5 % on the C2 base but only
+ +1.5 % once `URING`/`PILOT` are removed β€” both targeted the same residual disk misses.
+3. **Concurrency does not scale throughput**: 1.32 / 1.51 / 1.46 tok/s aggregate at 1/2/4.
+
+## 5. Where the bottleneck actually is
+
+Sampled during decode, after tuning:
+
+| | before (C0) | after (D1) |
+|---|---|---|
+| CPU | 638 % | 292–490 % (4–5 cores of 24) |
+| GPU | 2 % / 80 W | 20–22 % / 131 W |
+| Disk | 6 MB/s | **0 MB/s** |
+
+**Nothing is saturated.** The limiter is serialisation latency β€” 78 sequential layers,
+CPU↔GPU round trips, tiny matmuls at S=1 β€” not any single bandwidth. This explains both the
+failure of batching and the uselessness of maximising memory.
+
+### Prefill is a different regime entirely
+
+Decode activates 8 of 256 experts per layer; consecutive tokens reuse them, so the learned
+pinned set covers them. Prefill does not:
+
+```
+P(an expert is missed by N tokens) = (1 βˆ’ 8/256)^N
+P(512) β‰ˆ 1e-7 P(8595) β‰ˆ e^-273
+```
+
+**A 512-token slice already touches every expert of every layer.** Prefill therefore gets no
+benefit from caching: it pays the non-resident fraction of the *entire model*, once per slice.
+
+Measured: **2.6–4.3 tok/s prefill** at `--ctx 262144` with `COLI_PREFILL_CHUNK=512` β€” i.e.
+35–55 minutes before first token on an 8,595-token prompt. Raising the chunk to 2048 reduces
+the number of full-model sweeps from 17 to 5.
+
+During prefill **only one GPU is active** (100 % / 200 W) while the other three sit at
+0 % / ~80 W: aggregate PCIe and compute budget is unused while the engine waits on expert
+loads.
+
+Prefix caching itself works correctly in steady state (`prefix 8556/8562 token, prefill 6`):
+the prefill cost is paid once per conversation, not per turn.
+
+## 6. **The finding that invalidates naive A/B benchmarking**
+
+A configuration **identical in memory placement** to the best result above
+(VRAM 178 GB, RAM 190 GB) measured **2.56 tok/s instead of 5.46**, with no memory parameter
+changed.
+
+The hidden variable is `.coli_usage`, the persistent learned routing profile that seeds the
+pinned hot store at startup:
+
+| Moment | Recorded selections |
+|---|---|
+| first start | 58,240 |
+| end of the 9-config matrix (~1 h later) | 238,840 |
+| after a day of real conversational use | **17,697,640** |
+
+The pinned set had re-tuned itself onto the real workload, so the synthetic benchmark prompt
+became out-of-distribution for the cache.
+
+**Consequences for anyone benchmarking this engine:**
+
+- `.coli_usage` **persists across restarts** and grows monotonically. Two configurations
+ measured hours apart are not comparable, even with identical flags.
+- `CONTRIBUTING.md` asks for a *warm-up policy*, but warm-up alone is insufficient: the
+ profile must be **snapshotted and restored** before each configuration under test.
+- Absolute numbers in this report should be read as *within-session rankings*, not as
+ portable throughput figures. The 9-config matrix ran back-to-back in one hour on one
+ prompt, so its ordering is internally consistent; cross-session comparisons are not.
+
+Two hypotheses were formed and then refuted by this: "total residency governs throughput"
+(refuted: 98 % residency measured *slower* than 91 %) and "VRAM residency governs throughput"
+(refuted by the identical-placement result). An arithmetic check also shows the VRAM/RAM
+bandwidth differential accounts for ~1.3 ms of the 77 ms per-token gap β€” under 2 %.
+
+**Suggested engine-side fix:** expose per-tier access counts (VRAM / RAM / disk hits) through
+`/profile` in a scriptable form. The dashboard displays hit rate; making it queryable would
+let contributors report the metric that actually decides placement trade-offs instead of
+reporting occupied gigabytes.
+
+## 7. Context vs residency trade-off
+
+```
+available = 192 GB VRAM + 251 GB RAM = 443 GB
+required = 407 (experts) + 83 (KV @262144) + 12 (dense) + 10 (OS) = 512 GB
+deficit = 69 GB
+```
+
+At `--ctx 32768` the KV costs 11 GB and everything fits. Measured: **5.46 tok/s at 32768 vs
+0.86 tok/s at 262144**. The engine reports the collapse itself:
+
+```
+[RAM_GB=215.0] WARNING: cap=1 is the floor, projected peak 216.7 GB is 1.7 GB OVER
+the budget (resident 132.2 GB + reserve 82.8 GB).
+PIN_GB is inflating the resident set: lower it or drop it.
+```
+
+### KV cache size
+
+MLA stores `kv_lora_rank 512 + rope 64 = 576` values per token per layer, Γ—78 layers =
+44,928 values/token.
+
+| dtype | per token | @262,144 |
+|---|---|---|
+| fp32 | 179.7 KB | **47.1 GB** |
+| fp16 | 89.9 KB | 23.6 GB |
+| int8 | 44.9 KB | 11.8 GB |
+
+The engine reports **47.7 GB**, i.e. fp32 β€” confirmed at `colibri.c:2277`
+(`...*kvl*4`). On top of that, `kvb_all` at `colibri.c:2766` allocates
+`Tk Γ— HΓ—(qk_nope+v_head) = Tk Γ— 28,672` floats = 114.7 KB **per context position**, i.e.
+**30.1 GB** at 262,144 β€” matching the reported `kvb 30.1`. That buffer materialises the full
+per-head K/V, which is what MLA exists to avoid; `colibri.c:2568` already notes the
+alternative cost `O(TΒ·kv_lora)` instead of `O(TΒ·HΒ·(nope+vh))`.
+
+Comparison on the same model and same latent attention: llama.cpp with `-ctk f16` uses
+**0.098 MB/token** vs colibri's **0.316 MB/token** β€” **Γ—3.2**, i.e. ~57 GB avoidable at
+262,144. That is 83 % of the 69 GB deficit above: with an fp16 KV and no `kvb_all`, long
+context and full expert residency would be simultaneously reachable on this host.
+
+## 8. Build notes (Linux, CUDA)
+
+`nvcc` 12.6 + gcc 13 fails with `identifier "_Float32" is undefined`;
+`-allow-unsupported-compiler` produces 100 errors. The `BUILD-cuda-glibc241.md` recipe works
+and needs no root if the prefix is a user directory:
+
+```sh
+micromamba create -y -p ~/cuda13 -c nvidia -c conda-forge cuda-nvcc=13 cuda-cudart-dev=13
+ln -sfn ~/cuda13/lib ~/cuda13/lib64
+make glm CUDA=1 CUDA_ARCH=sm_86 CUDA_HOME=~/cuda13
+```
+
+The web dashboard needs Node β‰₯ 20.12 (`styleText` from `node:util`); Ubuntu 24.04 ships
+18.19, so `npm run build` fails on a stock install.
+
+## 9. Final configuration
+
+```sh
+CTX=32768 # 262144 measured 0.86 tok/s decode, 2.9 tok/s prefill
+COLI_CUDA=1
+COLI_GPUS=0,1,2,3
+CUDA_DENSE=1 # Γ—2.8 β€” the decisive lever
+COLI_CUDA_ATTN=1
+COLI_CUDA_PIPE=2
+COLI_CUDA_TC_W4A16=1
+CUDA_EXPERT_GB=188 # NOT "auto" β€” see issue on prefill scratch
+PIN=auto
+PIN_GB=all
+RAM_GB=200
+DIRECT=1
+PIPE=1 # no URING, no PILOT* β€” +26 % on this host
+COLI_PREFILL_CHUNK=2048
+```
+
+Operational notes: the engine does not release VRAM immediately on stop β€” restarting within
+~30 s fails with `Failed with result 'exit-code'` (`Restart=on-failure` recovers). And
+`coli plan` computes against *currently free* VRAM, so its output is meaningless while an
+engine instance is running.