Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
KV_TQ: refuse non-power-of-two row widths; KV_TQ<2 means the 4-bit tier
recvikingb7534d982c84
diff --git a/c/colibri.c b/c/colibri.cindex 4933e483..62545dd2 100644--- a/c/colibri.c+++ b/c/colibri.c@@ -10693,7 +10693,13 @@ int main(int argc, char **argv){{ int tqv = getenv("KV_TQ")?atoi(getenv("KV_TQ")):0;if(tqv){if(g_kv8){ fprintf(stderr,"[KV_TQ] KV8 and KV_TQ are mutually exclusive; KV_TQ wins (KV8 off)\n"); g_kv8=0; }- if(tqv<2) tqv=2; if(tqv>6) tqv=6; /* documented 3|4; clamp to the header's grid range */+ /* KV_TQ=1 used to clamp UP to 2, silently handing "just turn it on" the+ * most aggressive, lowest-quality tier. 4 is the recommended one, so a+ * bare/underspecified value lands there instead; >6 still clamps down to+ * the header's grid range. */+ if(tqv<2){ if(tqv!=4) fprintf(stderr,"[KV_TQ] KV_TQ=%d is below the 2..6 grid; using the recommended 4-bit tier\n",tqv);+ tqv=4; }+ if(tqv>6) tqv=6;g_tq=1; g_tq_bits=tqv;/* rotated int4 is a fixed 4-bit codec (best 4-bit for MLA); other bit widths and* KV_TQ_POLAR=1 use PolarQuant (variable bits, paper-faithful). */@@ -10714,6 +10720,23 @@ int main(int argc, char **argv){}#endifModel m; double t0=now_s(); model_init(&m,snap,cap,ebits,dbits);+ /* KV_TQ requires power-of-two row widths: both codecs rotate through a+ * radix-2 FWHT, and coli_kvq_quant_row returns an inert radius 0 for any+ * other width. On a model whose kv_lora/qk_rope are not powers of two that+ * would quantize EVERY latent row to zero and generate confident garbage+ * with no diagnostic -- the exact silent-misread failure the .coli_kv tier+ * magic exists to prevent. Refuse instead. GLM-5.2 (512/64) is unaffected;+ * this only fires on a model shape the codec cannot represent. */+ if(g_tq){+ int kl=m.c.kv_lora, kr=m.c.qk_rope;+ if(kl<2||kr<2||(kl&(kl-1))||(kr&(kr-1))){+ fprintf(stderr,"[KV_TQ] this model's latent rows are kv_lora=%d qk_rope=%d, but the "+ "rotation needs power-of-two widths (>=2): every row would quantize to zero. "+ "Refusing to run quantized -- unset KV_TQ (or use KV8=1, which has no width "+ "constraint).\n", kl, kr);+ return 2;+ }+ }if(!g_direct_heat_explicit){ /* COLI_DISKCLASS_WINDOW default, needs m.c (topk/n_layers) *//* CURRENT-STATE CALIBRATION: the "8" multiplier (recency window ~= the last 8* tokens' worth of routing) is a measured-config constant, not a derived truth.diff --git a/c/tests/test_kv_tq.c b/c/tests/test_kv_tq.cindex c65409e0..6cf1905a 100644--- a/c/tests/test_kv_tq.c+++ b/c/tests/test_kv_tq.c@@ -160,6 +160,17 @@ int main(void){/* subnormal-magnitude row: norm below the guard -> inert (like KV8) */for(int i=0;i<n;i++) x[i]=1e-38f;CHECK(coli_tq_quant_row(x,q,n,bits)==0.f, "subnormal row -> inert radius 0");+ /* NON-POWER-OF-TWO width: the radix-2 FWHT cannot represent it, so BOTH codecs+ * return an inert radius 0 for a perfectly healthy row. That is why colibri.c+ * refuses to start under KV_TQ when a model's kv_lora/qk_rope are not powers of+ * two -- without that guard every latent row would silently quantize to zero and+ * the engine would generate confident garbage. This pins the behavior the guard+ * is protecting against, so removing the guard cannot become quietly harmless. */+ for(int i=0;i<n;i++) x[i]=gauss();+ CHECK(coli_tq_quant_row(x,q,48,bits)==0.f, "non-power-of-two width -> inert (polar)");+ CHECK(coli_q4_quant_row(x,q,48)==0.f, "non-power-of-two width -> inert (int4)");+ CHECK(coli_kvq_quant_row(x,q,48,bits,0)==0.f, "non-power-of-two width -> inert (dispatch codec 0)");+ CHECK(coli_kvq_quant_row(x,q,48,bits,1)==0.f, "non-power-of-two width -> inert (dispatch codec 1)");/* a non-finite radius read from a corrupt file decodes to zero, never NaN */for(int i=0;i<n;i++) x[i]=gauss();float rad=coli_tq_quant_row(x,q,n,bits);diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 7177f301..4dadc877 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -50,7 +50,7 @@ Format: `VAR` β default β effect.| `KVSAVE` | `1` (on) | Persist the KV cache to `<model>/.coli_kv` so a conversation reopens warm. `KVSAVE=0` disables save+load (lossless round-trip; does not change output). || `KV_SLOTS` | `1` | Number of independent KV conversation slots (1β16), used in serve mode. || `KV8` | `0` (off) | Store the MLA latent KV cache in fp8 e4m3 with a per-row scale: ~3.9Γ less KV RAM, and `.coli_kv` shrinks ~4Γ (saved as the v2 format; f32 v1 files are quantized on resume and rewritten). Adds DeepSeek-V3-class KV quantization noise to attention. CPU attention path only for now: the CUDA/Metal fused-attention fast paths read f32 KV rows, so under KV8 they fall back to the CPU consumer (native fp8 decode; a one-time notice is printed under `COLI_CUDA_ATTN=1`). Forces `COLI_CUDA_PIPE=0`. Native CUDA/Metal fp8-KV kernels are follow-up PRs. |-| `KV_TQ` | `0` (off) | Sub-byte MLA latent KV quantization, mutually exclusive with `KV8` (`KV_TQ` wins). `KV_TQ=4` is the recommended tier: rotated-int4 codec (randomized-Hadamard rotation + Lloyd codebook, per-row radius as the scale), ~7.6Γ less KV RAM than f32. `KV_TQ=2|3|5|6` selects the PolarQuant codec at that bit width (`KV_TQ_POLAR=1` forces PolarQuant at 4 bits too). Requires power-of-two row widths (`kv_lora`/`qk_rope`; the GLM MLA shapes 512/64 qualify). `.coli_kv` is saved as the v3 format; a file saved under a different KV mode, codec, or bit width is refused with an explicit message and the cache restarts. Same CPU-only status as `KV8`: GPU fast paths fall back to the CPU consumer; native kernels are follow-up PRs. Forces `COLI_CUDA_PIPE=0`. |+| `KV_TQ` | `0` (off) | Sub-byte MLA latent KV quantization, mutually exclusive with `KV8` (`KV_TQ` wins). `KV_TQ=4` is the recommended tier: rotated-int4 codec (randomized-Hadamard rotation + Lloyd codebook, per-row radius as the scale), ~7.6Γ less KV RAM than f32. `KV_TQ=2|3|5|6` selects the PolarQuant codec at that bit width (`KV_TQ_POLAR=1` forces PolarQuant at 4 bits too). Requires power-of-two row widths (`kv_lora`/`qk_rope`; the GLM MLA shapes 512/64 qualify) β on a model whose shapes don't, the engine refuses to start rather than silently zeroing the cache. A value below the 2β6 grid (e.g. `KV_TQ=1`) is treated as the recommended `4` with a notice, not as the most aggressive tier. `.coli_kv` is saved as the v3 format; a file saved under a different KV mode, codec, or bit width is refused with an explicit message and the cache restarts. Same CPU-only status as `KV8`: GPU fast paths fall back to the CPU consumer; native kernels are follow-up PRs. Forces `COLI_CUDA_PIPE=0`. || `THINK` | `0` (off) | Emit a `<think>` reasoning block. `THINK=1` turns on visible reasoning. || `MTP` | on | Multi-Token Prediction (speculative draft head). `MTP=0` disables it. |