Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
README: corrected 9800X3D benchmark row (#101) β 10.5 GB/s VHDX still disk-bound, CUDA tier ~0% when AVX-512 CPU matches the 5090; CUDA value is CPU-dependent
JustVugg2baac89feef8
diff --git a/README.md b/README.mdindex 8c3edecc..ac082b6f 100644--- a/README.md+++ b/README.md@@ -388,8 +388,9 @@ Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decodi| γ same machine, model moved to a Samsung 9100 PRO PCIe 5.0 ([#31](https://github.com/JustVugg/colibri/issues/31)) | **8.81 GB/s** O_DIRECT | γ (usage history retained) | **0.28 tok/s** Β· hit 57% Β· profile flips: 32% disk / **57% matmul** || Ryzen AI Max+ 395 (Framework Desktop) Β· Ubuntu Β· 128 GB LPDDR5x Β· Intel Optane 905p PCIe 3.0 ([#39](https://github.com/JustVugg/colibri/issues/39)) | 3.27 GB/s buffered | int8 MTP head Β· fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s Β· hit 57% Β· profile 49% disk / 47% matmul || γ five runs later β learned pin 47.6 GB ([#39](https://github.com/JustVugg/colibri/issues/39)) | γ | `--temp 0.7 --topp 0.7` | **0.40 tok/s** Β· hit 71% Β· fastest non-Apple datapoint |+| Ryzen 7 9800X3D (16T) Β· WSL2 Β· 70 GB RAM Β· Samsung 9100 PRO PCIe 5.0 Β· RTX 5090 ([#101](https://github.com/JustVugg/colibri/issues/101)) | **10.51 GB/s** O_DIRECT | MTP off Β· learned pin 24 GB Β· hit 54% Β· OMP hot-team on | **0.41 tok/s** Β· disk-bound (36.5 s disk vs 24.0 s matmul) Β· **CUDA expert tier β 0%** (AVX-512 CPU matches the 5090) Β· `--topp 0.7` β **0.52 tok/s** |-Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2β2.7Γ faster than the dev box β **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts; `--topp 0.7` alone bought a clean 1.6Γ end-to-end speedup. The M5 Max datapoint lands right on the table's second row: **~1 tok/s of a 744B model on a laptop SSD** β and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels. The Framework 13 rows are the cache thesis proven end-to-end on one machine: 0.29 β 0.37 tok/s (hit 28% β 66%, speculation finally engaging at 52% acceptance) just by giving the cache its RAM β int8 MTP head + a bigger cap + the learned pin. The cap part is now automatic (cap auto-raise, 2026-07-10). The 9950X pair is the cleanest bottleneck experiment yet β same machine, same history, only the disk swapped: Γ5.8 disk bandwidth bought Γ2.9 tokens, and the profile **flipped from 66% disk to 57% matmul**. Past ~5 GB/s the disk stops being the story and the CPU (or the CUDA expert tier) becomes it.+Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2β2.7Γ faster than the dev box β **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts; `--topp 0.7` alone bought a clean 1.6Γ end-to-end speedup. The M5 Max datapoint lands right on the table's second row: **~1 tok/s of a 744B model on a laptop SSD** β and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels. The Framework 13 rows are the cache thesis proven end-to-end on one machine: 0.29 β 0.37 tok/s (hit 28% β 66%, speculation finally engaging at 52% acceptance) just by giving the cache its RAM β int8 MTP head + a bigger cap + the learned pin. The cap part is now automatic (cap auto-raise, 2026-07-10). The 9950X pair is the cleanest bottleneck experiment yet β same machine, same history, only the disk swapped: Γ5.8 disk bandwidth bought Γ2.9 tokens, and the profile **flipped from 66% disk to 57% matmul**. But the crossover depends on the CPU kernel: the 9800X3D row ([#101](https://github.com/JustVugg/colibri/issues/101)) shows that with the OMP hot-team tuning on, the AVX-512 CPU matmul is fast enough that even a **10 GB/s NVMe stays disk-bound** β and there the **CUDA expert tier buys β 0%**, because the CPU already matches the 5090 on expert matmul. The GPU tier earns its VRAM only when the CPU is the weak link, not by default. (Honest correction from #101: an earlier version of that report ran with the OMP tuning off, which manufactured a false matmul-bound crossover and a false +14% for CUDA β neither survived a clean re-run.)## Quality benchmark β help wanted