Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

MTP head is int8 by default (int4 head: 0-4% acceptance, unusable β€” measured in #8); README: honest MTP numbers + M5 Max community datapoint (#4, #5)

JustVuggc035f417783a
diff --git a/README.md b/README.md
index be1e7094..63e7a3f4 100644
--- a/README.md
+++ b/README.md
@@ -26,7 +26,7 @@ The engine is a single C file (`c/glm.c`, ~1,300 lines) plus small headers. No B
- **Faithful GLM-5.2 (`glm_moe_dsa`) forward** β€” validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
- **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57Γ— smaller β€” GLM-5.2 has 64 heads and no GQA).
- **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
-- **Native MTP speculative decoding** β€” GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. Measured **2.00 tokens/forward (100% acceptance)** on structured text. Lossless β€” *and stays lossless under sampling* via rejection sampling (accept draft w.p. p(draft); on reject, resample with the draft banned).
+- **Native MTP speculative decoding** β€” GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless β€” *and stays lossless under sampling* via rejection sampling. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 β†’ ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up β€” the adaptive guard and `DRAFT=0` are there for that.
- **True sampling** β€” temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
- **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5Γ— faster (119 GFLOP/s measured), int4 1.8Γ— in batch β€” routing decided per shape by measurement (int4 single-row stays f32: it measured slower).
- **MLA weight absorption** (DeepSeek trick) for decode: no per-token k/v reconstruction β€” the query absorbs `kv_b`, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere.
@@ -48,7 +48,7 @@ The engine is a single C file (`c/glm.c`, ~1,300 lines) plus small headers. No B
| peak RSS during chat | ~20 GB (auto-capped) |
| cold decode cost | ~11 GB disk reads/token (75 layers Γ— 8 experts) |
| disk ceiling (VHDX random) | ~1 GB/s β†’ ~0.05–0.1 tok/s cold |
-| MTP speculation | 2.0 tok/forward measured |
+| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |
This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.
@@ -118,7 +118,7 @@ PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM
./coli bench
```
-**Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost; RAM turns cold reads into free cache hits):
+**Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost *once the cache is warm*; RAM turns cold reads into free cache hits):
| machine | expected |
|---|---|
@@ -138,8 +138,9 @@ Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decodi
|---|---|---|---|
| Intel Core Ultra 7 270K Plus (24 threads) Β· WSL2 Β· 24 GB RAM Β· NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered Β· 2.74 GB/s O_DIRECT | default | 0.07 tok/s Β· expert hit 3–4% Β· RSS 14.1 GB |
| 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** Β· expert hit 11% Β· RSS 14.7 GB |
+| Apple M5 Max (18 cores) Β· macOS Β· 128 GB unified Β· internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | 14.2 GB/s O_DIRECT | default, MTP off | **1.06 tok/s** Β· expert hit 23% Β· RSS 21.8 GB |
-Takeaways from this datapoint: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7Γ— faster than the dev box β€” **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts. And `--topp 0.7` alone bought a clean 1.6Γ— end-to-end speedup.
+Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7Γ— faster than the dev box β€” **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts; `--topp 0.7` alone bought a clean 1.6Γ— end-to-end speedup. The M5 Max datapoint lands right on the table's second row: **~1 tok/s of a 744B model on a laptop SSD** β€” and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels.
## Quality benchmark β€” help wanted
diff --git a/c/coli b/c/coli
index 36b75fc7..2cfdf408 100644
--- a/c/coli
+++ b/c/coli
@@ -409,9 +409,11 @@ def cmd_convert(a):
rc=subprocess.call(base)
if rc!=0: sys.exit(rc)
if a.no_mtp: sys.exit(0)
- # passo 2: testa MTP (layer 78) -> abilita la decodifica speculativa nativa (piu' veloce)
- print(f" {C.dim}[2/2] testa MTP (draft speculativi){C.r}")
- sys.exit(subprocess.call(base+["--mtp"]))
+ # passo 2: testa MTP (layer 78). SEMPRE int8: a int4 i draft sbagliano quasi sempre
+ # (acceptance 0-4% vs 39-59%, misurato β€” issue #8) e la speculazione non parte mai.
+ mtp_cmd=list(base); i=mtp_cmd.index("--ebits"); mtp_cmd[i+1]=str(max(8,a.ebits))
+ print(f" {C.dim}[2/2] testa MTP a int8 (draft speculativi){C.r}")
+ sys.exit(subprocess.call(mtp_cmd+["--mtp"]))
def main():
common=argparse.ArgumentParser(add_help=False)
diff --git a/c/convert_fp8_to_int4.py b/c/convert_fp8_to_int4.py
index f77e91e5..5785ee94 100644
--- a/c/convert_fp8_to_int4.py
+++ b/c/convert_fp8_to_int4.py
@@ -132,7 +132,7 @@ def main():
ap.add_argument("--repo", default=None)
ap.add_argument("--indir", default=None)
ap.add_argument("--outdir", required=False)
- ap.add_argument("--ebits", type=int, default=4) # bit residenti: attn/dense-mlp/shared
+ ap.add_argument("--ebits", type=int, default=None) # bit residenti (default 4; 8 per --mtp/--indexer)
ap.add_argument("--io-bits", type=int, default=8) # bit di embed/lm_head
ap.add_argument("--xbits", type=int, default=None) # bit degli expert ROUTED (streaming); default=ebits
ap.add_argument("--n-layers", type=int, default=78)
@@ -145,6 +145,10 @@ def main():
"i tensori indexer sono sparsi su ~tutti gli shard: ri-scarica l'intero repo (~756 GB "
"di traffico) per tenerne pochi GB. Resumabile shard per shard. Consigliato --ebits 8.")
a = ap.parse_args()
+ if a.ebits is None:
+ # testa MTP a int4 = acceptance ~0-4% (misurato, issue #8): il draft sbaglia sempre
+ # e la speculazione non parte mai. A int8: 39-59%, 2.2-2.8 token/forward.
+ a.ebits = 8 if (a.mtp or a.indexer) else 4
if a.xbits is None: a.xbits = a.ebits
if a.selftest: