Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

exact verify (#689): document the quantised-KV fallback and the untested near-tie flip

Ry Bruscoea13b3cd77545
diff --git a/README.md b/README.md
index add77998..3a0ad7d9 100644
--- a/README.md
+++ b/README.md
@@ -334,6 +334,15 @@ full forensic story). Grammar-forced drafts
constrained JSON output. Whether speculation is a net win depends on your
cache temperature — measure, and use `DRAFT=0` when it doesn't pay.
+Verify batches can also opt into an **exact attention core** with
+`COLI_EXACT_VERIFY=1` ([#689](https://github.com/JustVugg/colibri/issues/689)):
+the CPU MLA-absorb score and context dots accumulate integer products and round
+once, so a near-tie in a verify row resolves the same way on every host, at
+roughly 0.6x tok/s on a tiny oracle (the dot itself is ~5–7x the float loop).
+Two limits to know: with a quantised KV cache (`tq1`, TQ or int8 KV) the
+context dot keeps the float path, so exactness there is not provided; and a
+real near-tie flip has only been argued, not yet caught on GLM-5.2 at n=64.
+
## What it achieves
<p align="center">
diff --git a/c/colibri.c b/c/colibri.c
index 19d3e4b6..4f22a587 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -4851,7 +4851,9 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p
float clat[512]; memset(clat,0,kvl*sizeof(float));
if(exact_verify_on()&&g_spec_live&&!tq1&&!g_tq&&!g_kv8){
/* #689 exact verify: clat[i] = sum_t sc[t]*Lt[i] as an exact dot over t per column
- * (transposed walk: cache-unfriendly, verify rows only) */
+ * (transposed walk: cache-unfriendly, verify rows only). NOT taken on the quantised
+ * KV paths (tq1 / TQ / kv8): those keep the float context dot, so COLI_EXACT_VERIFY
+ * does not provide exactness for the context dot with a quantised cache (README). */
for(int i=0;i<kvl;i++){ exd_acc ea; exd_init(&ea);
for(int jj=0;jj<nt;jj++){ int t = tlist ? tlist[jj] : st0+jj;
exd_add_ff(&ea,sc[jj],coli_kv_row(ks->Lc[layer],t,kvl)[i]); }