Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: qwen38 GPU section -- measured decode figures, honest verdict
kreuzzelgeeff443b003f
diff --git a/docs/qwen38.md b/docs/qwen38.mdindex 6c2af438..3305de68 100644--- a/docs/qwen38.md+++ b/docs/qwen38.md@@ -193,8 +193,9 @@ heat file from an earlier run therefore does not help, and the engine does nowarmstart. Inside a run the locality is strong -- on the route trace of a315-token prompt plus 100 generated tokens, an LRU of 32 slots per layerserves 55 % of decode routes, 64 serve 79 %, 128 serve 90 % -- so the tier-earns its VRAM by promotion at touch time: with the RAM LRU at cap 128 and one-8 GB card, 49 % of decode routes were served from VRAM on the real checkpoint.+earns its VRAM by promotion at touch time: with the RAM LRU at cap 128, one+8 GB card served 45 % of decode routes from VRAM on the real checkpoint, two+cards 59 %.**VRAM per expert.** The tier charges what `cudaMalloc` takes, not the payload:an allocation above 1 MiB rounds up to a multiple of 2 MiB, so each 1.56 MiB@@ -204,13 +205,30 @@ are exactly 512 KiB and are served exactly, so its accounting was alreadyright.) Pooling experts into one arena per device would recover the 22 %; itis not done yet.-**Where the time goes** (this checkpoint, Threadripper 3945WX 12 cores, 12-threads, cap 32, `COLI_TIMERS=1`): 4.6 s per decode token were 1.4 s dense-kernels (BF16 trunk and DeltaNet projections), 1.8 s expert disk reads, 0.9 s-routed-expert GEMV on the CPU (1.9 ms per expert: e4m3 decode, not bandwidth),-0.3 s the rest. The VRAM tier attacks the routed-expert GEMV and, together-with a larger cap, the disk reads; the dense trunk is the largest item and is-not on the GPU yet.+**What it is worth on this machine -- measured, not projected.** Threadripper+PRO 3945WX (12 cores, `OMP_PLACES=cores`), 94 GB, checkpoint in the page cache,+prompt of 315 tokens plus 100 generated, cap 128, `COLI_TIMERS=1` decode bank+(the `Speed` line the engine prints divides by the whole generation time+including the prompt; the numbers below are decode only):++| | CPU only | tier, one 8 GB card | tier, two cards |+|---|---:|---:|---:|+| VRAM share of routed experts | -- | 45 % | 59 % |+| routed-expert GEMV on the CPU, ms/token | 192 | 126 | 93 |+| decode, ms/token | 806 | 808 | 808 |+| decode tok/s (three runs each) | 1.14 - 1.24 | 1.20 - 1.24 | 1.21 - 1.24 |+| greedy tokens | identical | identical | identical |++A decode token costs about 0.8 s here: 260 ms dense kernels (BF16 trunk and+the DeltaNet projections), 244 ms expert reads (page-cache copies of the cap+misses), 192 ms routed-expert GEMVs on the CPU (0.5 ms for the three FP8+matrices of one expert), 42 ms LM head, 24 ms QSA, 19 ms shared expert. The+tier removes 66 - 99 ms of GEMV time and spends about as much on its own+per-layer issue/take round trips and on staging the experts it promotes, so+the wall time does not move. The parity holds, the plumbing is exercised on a+real checkpoint, and the honest summary is: on a machine of this class the+expert tier is not where Qwen3.8's decode time is. The dense trunk (a third of+the token) and the expert reads (another third, a RAM-cap question) are.## Performance telemetry