Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs(experiments): the engine's own profiler, and the DRAFT re-test
Nicolas2ade55326c8f
diff --git a/docs/experiments/glm52-4xa6000-2026-08-02.md b/docs/experiments/glm52-4xa6000-2026-08-02.mdindex a6c84a4b..277e3433 100644--- a/docs/experiments/glm52-4xa6000-2026-08-02.md+++ b/docs/experiments/glm52-4xa6000-2026-08-02.md@@ -127,6 +127,44 @@ Sampled during decode, after tuning:CPUβGPU round trips, tiny matmuls at S=1 β not any single bandwidth. This explains both thefailure of batching and the uselessness of maximising memory.+### And the engine's own profiler says where the latency goes++External sampling could not attribute the time; the engine's per-phase counters can.+`CTX=8192`, 48 tokens generated, greedy:++```+expert-disk 2.345 s ( 4.8 %) service 2.345 s / wait 2.046 s+expert-matmul 9.074 s (18.6 %)+attention 31.264 s (64.2 %) including kvb 0.000 s+lm_head 0.000 s+other 5.987 s (12.3 %)++attention breakdown: projection/RoPE 4.592 s+ score-softmax-value 20.398 s (41.9 % of total)+ output projection 6.261 s+```++**Attention is 64 % of decode; expert compute is 19 %; disk is 5 %.** This reorders every+placement question in this report. Expert residency, tiering and pinning together govern+under a fifth of the time β a *perfect* expert placement, costing zero, would buy at most+~19 %.++Two consequences worth stating plainly:++- **`CUDA_DENSE=1` being worth Γ2.8 (Β§4) now has an explanation.** It moves the dense and+ attention tensors onto the GPU. It was the decisive lever because it was the only one+ aimed at the phase that dominates.+- **`kvb 0.000 s`** β during decode the absorbed path runs and `kvb_all` is never built.+ The 30.1 GB of Β§7 is a *prefill-only, transient* allocation that `cap_for_ram()`+ nevertheless reserves permanently at every context length.++`score-softmax-value` does not currently make arithmetic sense and is **not** presented here+as a finding. At `CTX=8192` with a ~40-position KV cache it moves ~7 MB per token and spends+425 ms doing it β about 17 MB/s, three orders of magnitude below this host's memory+bandwidth. Either the counter attributes waiting time to attention, or there is a degenerate+path at S=1. Resolving it needs a controlled repeat, which is the next measurement rather+than a claim.+### Prefill is a different regime β but not for the reason stated in the first draftDecode activates 8 of 256 experts per layer and consecutive tokens reuse them, so the learned@@ -329,7 +367,43 @@ And the uniform per-layer cap the engine already applies is close to optimal:*which* experts are pinned β it is how many fit, and how well the profile matches the currentworkload (Β§6).-## 9. Build notes (Linux, CUDA)+## 9. Self-speculation (`DRAFT`) is not worth enabling here β on this workload++`--auto-tier` forces `DRAFT=0` on this host. That decision was made against a plan with 46 %+residency, and the configuration has since improved a lot, so it was worth re-testing.+Same protocol, `.coli_usage` restored before each condition, two warm-ups then three+measurements:++| | mean |+|---|---|+| `DRAFT=0` (what `--auto-tier` imposes) | **1.70 tok/s** |+| `DRAFT=3` | 0.85 tok/s (β50 %) |+| `DRAFT=3` + `COLI_CUDA_MTP=1` | 0.96 tok/s (β44 %) |++**`--auto-tier` is right.** Verifying up to 4 positions in one forward widens the per-layer+expert union from 8 to as many as 32, and on this workload nothing is accepted in return.++**The scope of that result is narrower than it looks, and the distinction matters.**+`DRAFT=n` is not the MTP head β `colibri.c:944` describes it as *"n token auto-speculati per+forward via n-gram lookup"*, and `colibri.c:6000` as lossless self-speculation whose+candidates come **from the context**; the MTP head is only the fallback when the n-gram+lookup returns nothing (`colibri.c:6093`). It is speculation by *copying*.++The prompt used here asks for three advantages of expert routing and generates 64 tokens of+entirely novel prose. There is nothing in the context to copy, so the lookup could never+succeed β this measures the cost of speculation on its structural worst case. The honest+conclusion is therefore:++> On novel text generation, `DRAFT>0` costs 2Γ and returns nothing, and disabling it by+> default is correct. It is **untested here on the workloads this technique targets** β+> code editing, summarisation, translation, structured output β where long spans are copied+> from the context.++That re-test needs a copy-heavy prompt and, unlike this one, needs the acceptance rate+logged. This run did not capture it (`acceptance: not reported` in all three conditions),+which is a defect of the harness, not of the engine.++## 10. Build notes (Linux, CUDA)`nvcc` 12.6 + gcc 13 fails with `identifier "_Float32" is undefined`;`-allow-unsupported-compiler` produces 100 errors. The `BUILD-cuda-glibc241.md` recipe works@@ -344,7 +418,7 @@ make glm CUDA=1 CUDA_ARCH=sm_86 CUDA_HOME=~/cuda13The web dashboard needs Node β₯ 20.12 (`styleText` from `node:util`); Ubuntu 24.04 ships18.19, so `npm run build` fails on a stock install.-## 10. Final configuration+## 11. Final configuration```shCTX=32768 # 131072 also fine: β23 % decode, prefill unchanged (Β§5)