Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
README honesty fixes: MTP not byte-identical to greedy (shape-dependent kernels, #100); VHDX ceiling was this drive not the virtualization layer (#101, community measured 10.5 GB/s)
JustVuggf1fbbca35281
diff --git a/README.md b/README.mdindex b8d3dc6d..8c3edecc 100644--- a/README.md+++ b/README.md@@ -26,7 +26,7 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B- **Faithful GLM-5.2 (`glm_moe_dsa`) forward** β validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).- **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57Γ smaller β GLM-5.2 has 64 heads and no GQA).- **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.-- **Native MTP speculative decoding** β GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0β4% and speculation never engages; at int8 it's 39β59% acceptance, **2.2β2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless β *and stays lossless under sampling* via rejection sampling. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 β ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up β the adaptive guard and `DRAFT=0` are there for that.+- **Native MTP speculative decoding** β GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0β4% and speculation never engages; at int8 it's 39β59% acceptance, **2.2β2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless *in exact arithmetic* β but **not byte-identical to non-speculative greedy in practice** ([#100](https://github.com/JustVugg/colibri/issues/100)): the batched verification forward (S>1) uses shape-dependent integer kernels that round differently from the S=1 path, so on a near-tie the greedy argmax can flip. Every emitted token is still the argmax of a valid forward β the continuation stays correct β it just isn't bit-for-bit the same stream; `DRAFT=0` (or the adaptive guard) gives byte-exact greedy. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 β ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up.- **Grammar-forced speculative drafts** (`GRAMMAR=file.gbnf`, [#48](https://github.com/JustVugg/colibri/issues/48)) β on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly **one** legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance β no draft head, no lookup table, and it engages even with the int4 MTP head from [#8](https://github.com/JustVugg/colibri/issues/8). It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output β worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes, `| ( ) ? * +`, comments); `GRAMMAR_DRAFT=n` caps the forced span per forward (default 24). Composes with `DRAFT`/MTP, which fill the free-text gaps between forced spans.- **True sampling** β temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).- **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4β2.5Γ faster (119 GFLOP/s measured), int4 1.8Γ in batch β routing decided per shape by measurement (int4 single-row stays f32: it measured slower).@@ -50,7 +50,7 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B| load time | ~30 s || peak RSS during chat | ~20 GB (auto-capped) || cold decode cost | ~11 GB disk reads/token (75 layers Γ 8 experts) |-| disk ceiling (VHDX random) | ~1 GB/s β ~0.05β0.1 tok/s cold |+| disk ceiling (this dev box's drive) | ~1 GB/s β ~0.05β0.1 tok/s cold || MTP speculation (int8 head) | 2.2β2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.@@ -331,7 +331,7 @@ thrashing. Persistent `.coli_usage` remains the long-term signal and is not deca## Got a better machine? Try it β here's what to expect-colibrΓ¬ was built on deliberately humble hardware (12 cores, 25 GB RAM, NVMe behind a WSL2 VHDX that caps random reads at ~1 GB/s). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, β₯16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS β never a network/9p mount).+colibrΓ¬ was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on *this* drive β note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, [#101](https://github.com/JustVugg/colibri/issues/101)). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, β₯16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS β never a network/9p mount).**How to test it, in order:**