Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
exact verify (#689): document the quantised-KV fallback and the untested near-tie flip
Ry Bruscoea13b3cd77545
diff --git a/README.md b/README.mdindex add77998..3a0ad7d9 100644--- a/README.md+++ b/README.md@@ -334,6 +334,15 @@ full forensic story). Grammar-forced draftsconstrained JSON output. Whether speculation is a net win depends on yourcache temperature — measure, and use `DRAFT=0` when it doesn't pay.+Verify batches can also opt into an **exact attention core** with+`COLI_EXACT_VERIFY=1` ([#689](https://github.com/JustVugg/colibri/issues/689)):+the CPU MLA-absorb score and context dots accumulate integer products and round+once, so a near-tie in a verify row resolves the same way on every host, at+roughly 0.6x tok/s on a tiny oracle (the dot itself is ~5–7x the float loop).+Two limits to know: with a quantised KV cache (`tq1`, TQ or int8 KV) the+context dot keeps the float path, so exactness there is not provided; and a+real near-tie flip has only been argued, not yet caught on GLM-5.2 at n=64.+## What it achieves<p align="center">diff --git a/c/colibri.c b/c/colibri.cindex 19d3e4b6..4f22a587 100644--- a/c/colibri.c+++ b/c/colibri.c@@ -4851,7 +4851,9 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int pfloat clat[512]; memset(clat,0,kvl*sizeof(float));if(exact_verify_on()&&g_spec_live&&!tq1&&!g_tq&&!g_kv8){/* #689 exact verify: clat[i] = sum_t sc[t]*Lt[i] as an exact dot over t per column- * (transposed walk: cache-unfriendly, verify rows only) */+ * (transposed walk: cache-unfriendly, verify rows only). NOT taken on the quantised+ * KV paths (tq1 / TQ / kv8): those keep the float context dot, so COLI_EXACT_VERIFY+ * does not provide exactness for the context dot with a quantised cache (README). */for(int i=0;i<kvl;i++){ exd_acc ea; exd_init(&ea);for(int jj=0;jj<nt;jj++){ int t = tlist ? tlist[jj] : st0+jj;exd_add_ff(&ea,sc[jj],coli_kv_row(ks->Lc[layer],t,kvl)[i]); }