Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs(experiments): the engine's own profiler, and the DRAFT re-test

Nicolas2ade55326c8f
diff --git a/docs/experiments/glm52-4xa6000-2026-08-02.md b/docs/experiments/glm52-4xa6000-2026-08-02.md
index a6c84a4b..277e3433 100644
--- a/docs/experiments/glm52-4xa6000-2026-08-02.md
+++ b/docs/experiments/glm52-4xa6000-2026-08-02.md
@@ -127,6 +127,44 @@ Sampled during decode, after tuning:
CPU↔GPU round trips, tiny matmuls at S=1 β€” not any single bandwidth. This explains both the
failure of batching and the uselessness of maximising memory.
+### And the engine's own profiler says where the latency goes
+
+External sampling could not attribute the time; the engine's per-phase counters can.
+`CTX=8192`, 48 tokens generated, greedy:
+
+```
+expert-disk 2.345 s ( 4.8 %) service 2.345 s / wait 2.046 s
+expert-matmul 9.074 s (18.6 %)
+attention 31.264 s (64.2 %) including kvb 0.000 s
+lm_head 0.000 s
+other 5.987 s (12.3 %)
+
+attention breakdown: projection/RoPE 4.592 s
+ score-softmax-value 20.398 s (41.9 % of total)
+ output projection 6.261 s
+```
+
+**Attention is 64 % of decode; expert compute is 19 %; disk is 5 %.** This reorders every
+placement question in this report. Expert residency, tiering and pinning together govern
+under a fifth of the time β€” a *perfect* expert placement, costing zero, would buy at most
+~19 %.
+
+Two consequences worth stating plainly:
+
+- **`CUDA_DENSE=1` being worth Γ—2.8 (Β§4) now has an explanation.** It moves the dense and
+ attention tensors onto the GPU. It was the decisive lever because it was the only one
+ aimed at the phase that dominates.
+- **`kvb 0.000 s`** β€” during decode the absorbed path runs and `kvb_all` is never built.
+ The 30.1 GB of Β§7 is a *prefill-only, transient* allocation that `cap_for_ram()`
+ nevertheless reserves permanently at every context length.
+
+`score-softmax-value` does not currently make arithmetic sense and is **not** presented here
+as a finding. At `CTX=8192` with a ~40-position KV cache it moves ~7 MB per token and spends
+425 ms doing it β€” about 17 MB/s, three orders of magnitude below this host's memory
+bandwidth. Either the counter attributes waiting time to attention, or there is a degenerate
+path at S=1. Resolving it needs a controlled repeat, which is the next measurement rather
+than a claim.
+
### Prefill is a different regime β€” but not for the reason stated in the first draft
Decode activates 8 of 256 experts per layer and consecutive tokens reuse them, so the learned
@@ -329,7 +367,43 @@ And the uniform per-layer cap the engine already applies is close to optimal:
*which* experts are pinned β€” it is how many fit, and how well the profile matches the current
workload (Β§6).
-## 9. Build notes (Linux, CUDA)
+## 9. Self-speculation (`DRAFT`) is not worth enabling here β€” on this workload
+
+`--auto-tier` forces `DRAFT=0` on this host. That decision was made against a plan with 46 %
+residency, and the configuration has since improved a lot, so it was worth re-testing.
+Same protocol, `.coli_usage` restored before each condition, two warm-ups then three
+measurements:
+
+| | mean |
+|---|---|
+| `DRAFT=0` (what `--auto-tier` imposes) | **1.70 tok/s** |
+| `DRAFT=3` | 0.85 tok/s (βˆ’50 %) |
+| `DRAFT=3` + `COLI_CUDA_MTP=1` | 0.96 tok/s (βˆ’44 %) |
+
+**`--auto-tier` is right.** Verifying up to 4 positions in one forward widens the per-layer
+expert union from 8 to as many as 32, and on this workload nothing is accepted in return.
+
+**The scope of that result is narrower than it looks, and the distinction matters.**
+`DRAFT=n` is not the MTP head β€” `colibri.c:944` describes it as *"n token auto-speculati per
+forward via n-gram lookup"*, and `colibri.c:6000` as lossless self-speculation whose
+candidates come **from the context**; the MTP head is only the fallback when the n-gram
+lookup returns nothing (`colibri.c:6093`). It is speculation by *copying*.
+
+The prompt used here asks for three advantages of expert routing and generates 64 tokens of
+entirely novel prose. There is nothing in the context to copy, so the lookup could never
+succeed β€” this measures the cost of speculation on its structural worst case. The honest
+conclusion is therefore:
+
+> On novel text generation, `DRAFT>0` costs 2Γ— and returns nothing, and disabling it by
+> default is correct. It is **untested here on the workloads this technique targets** β€”
+> code editing, summarisation, translation, structured output β€” where long spans are copied
+> from the context.
+
+That re-test needs a copy-heavy prompt and, unlike this one, needs the acceptance rate
+logged. This run did not capture it (`acceptance: not reported` in all three conditions),
+which is a defect of the harness, not of the engine.
+
+## 10. Build notes (Linux, CUDA)
`nvcc` 12.6 + gcc 13 fails with `identifier "_Float32" is undefined`;
`-allow-unsupported-compiler` produces 100 errors. The `BUILD-cuda-glibc241.md` recipe works
@@ -344,7 +418,7 @@ make glm CUDA=1 CUDA_ARCH=sm_86 CUDA_HOME=~/cuda13
The web dashboard needs Node β‰₯ 20.12 (`styleText` from `node:util`); Ubuntu 24.04 ships
18.19, so `npm run build` fails on a stock install.
-## 10. Final configuration
+## 11. Final configuration
```sh
CTX=32768 # 131072 also fine: βˆ’23 % decode, prefill unchanged (Β§5)