Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

KV_TQ: refuse non-power-of-two row widths; KV_TQ<2 means the 4-bit tier

recvikingb7534d982c84
diff --git a/c/colibri.c b/c/colibri.c
index 4933e483..62545dd2 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -10693,7 +10693,13 @@ int main(int argc, char **argv){
{ int tqv = getenv("KV_TQ")?atoi(getenv("KV_TQ")):0;
if(tqv){
if(g_kv8){ fprintf(stderr,"[KV_TQ] KV8 and KV_TQ are mutually exclusive; KV_TQ wins (KV8 off)\n"); g_kv8=0; }
- if(tqv<2) tqv=2; if(tqv>6) tqv=6; /* documented 3|4; clamp to the header's grid range */
+ /* KV_TQ=1 used to clamp UP to 2, silently handing "just turn it on" the
+ * most aggressive, lowest-quality tier. 4 is the recommended one, so a
+ * bare/underspecified value lands there instead; >6 still clamps down to
+ * the header's grid range. */
+ if(tqv<2){ if(tqv!=4) fprintf(stderr,"[KV_TQ] KV_TQ=%d is below the 2..6 grid; using the recommended 4-bit tier\n",tqv);
+ tqv=4; }
+ if(tqv>6) tqv=6;
g_tq=1; g_tq_bits=tqv;
/* rotated int4 is a fixed 4-bit codec (best 4-bit for MLA); other bit widths and
* KV_TQ_POLAR=1 use PolarQuant (variable bits, paper-faithful). */
@@ -10714,6 +10720,23 @@ int main(int argc, char **argv){
}
#endif
Model m; double t0=now_s(); model_init(&m,snap,cap,ebits,dbits);
+ /* KV_TQ requires power-of-two row widths: both codecs rotate through a
+ * radix-2 FWHT, and coli_kvq_quant_row returns an inert radius 0 for any
+ * other width. On a model whose kv_lora/qk_rope are not powers of two that
+ * would quantize EVERY latent row to zero and generate confident garbage
+ * with no diagnostic -- the exact silent-misread failure the .coli_kv tier
+ * magic exists to prevent. Refuse instead. GLM-5.2 (512/64) is unaffected;
+ * this only fires on a model shape the codec cannot represent. */
+ if(g_tq){
+ int kl=m.c.kv_lora, kr=m.c.qk_rope;
+ if(kl<2||kr<2||(kl&(kl-1))||(kr&(kr-1))){
+ fprintf(stderr,"[KV_TQ] this model's latent rows are kv_lora=%d qk_rope=%d, but the "
+ "rotation needs power-of-two widths (>=2): every row would quantize to zero. "
+ "Refusing to run quantized -- unset KV_TQ (or use KV8=1, which has no width "
+ "constraint).\n", kl, kr);
+ return 2;
+ }
+ }
if(!g_direct_heat_explicit){ /* COLI_DISKCLASS_WINDOW default, needs m.c (topk/n_layers) */
/* CURRENT-STATE CALIBRATION: the "8" multiplier (recency window ~= the last 8
* tokens' worth of routing) is a measured-config constant, not a derived truth.
diff --git a/c/tests/test_kv_tq.c b/c/tests/test_kv_tq.c
index c65409e0..6cf1905a 100644
--- a/c/tests/test_kv_tq.c
+++ b/c/tests/test_kv_tq.c
@@ -160,6 +160,17 @@ int main(void){
/* subnormal-magnitude row: norm below the guard -> inert (like KV8) */
for(int i=0;i<n;i++) x[i]=1e-38f;
CHECK(coli_tq_quant_row(x,q,n,bits)==0.f, "subnormal row -> inert radius 0");
+ /* NON-POWER-OF-TWO width: the radix-2 FWHT cannot represent it, so BOTH codecs
+ * return an inert radius 0 for a perfectly healthy row. That is why colibri.c
+ * refuses to start under KV_TQ when a model's kv_lora/qk_rope are not powers of
+ * two -- without that guard every latent row would silently quantize to zero and
+ * the engine would generate confident garbage. This pins the behavior the guard
+ * is protecting against, so removing the guard cannot become quietly harmless. */
+ for(int i=0;i<n;i++) x[i]=gauss();
+ CHECK(coli_tq_quant_row(x,q,48,bits)==0.f, "non-power-of-two width -> inert (polar)");
+ CHECK(coli_q4_quant_row(x,q,48)==0.f, "non-power-of-two width -> inert (int4)");
+ CHECK(coli_kvq_quant_row(x,q,48,bits,0)==0.f, "non-power-of-two width -> inert (dispatch codec 0)");
+ CHECK(coli_kvq_quant_row(x,q,48,bits,1)==0.f, "non-power-of-two width -> inert (dispatch codec 1)");
/* a non-finite radius read from a corrupt file decodes to zero, never NaN */
for(int i=0;i<n;i++) x[i]=gauss();
float rad=coli_tq_quant_row(x,q,n,bits);
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 7177f301..4dadc877 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -50,7 +50,7 @@ Format: `VAR` β€” default β€” effect.
| `KVSAVE` | `1` (on) | Persist the KV cache to `<model>/.coli_kv` so a conversation reopens warm. `KVSAVE=0` disables save+load (lossless round-trip; does not change output). |
| `KV_SLOTS` | `1` | Number of independent KV conversation slots (1–16), used in serve mode. |
| `KV8` | `0` (off) | Store the MLA latent KV cache in fp8 e4m3 with a per-row scale: ~3.9Γ— less KV RAM, and `.coli_kv` shrinks ~4Γ— (saved as the v2 format; f32 v1 files are quantized on resume and rewritten). Adds DeepSeek-V3-class KV quantization noise to attention. CPU attention path only for now: the CUDA/Metal fused-attention fast paths read f32 KV rows, so under KV8 they fall back to the CPU consumer (native fp8 decode; a one-time notice is printed under `COLI_CUDA_ATTN=1`). Forces `COLI_CUDA_PIPE=0`. Native CUDA/Metal fp8-KV kernels are follow-up PRs. |
-| `KV_TQ` | `0` (off) | Sub-byte MLA latent KV quantization, mutually exclusive with `KV8` (`KV_TQ` wins). `KV_TQ=4` is the recommended tier: rotated-int4 codec (randomized-Hadamard rotation + Lloyd codebook, per-row radius as the scale), ~7.6Γ— less KV RAM than f32. `KV_TQ=2|3|5|6` selects the PolarQuant codec at that bit width (`KV_TQ_POLAR=1` forces PolarQuant at 4 bits too). Requires power-of-two row widths (`kv_lora`/`qk_rope`; the GLM MLA shapes 512/64 qualify). `.coli_kv` is saved as the v3 format; a file saved under a different KV mode, codec, or bit width is refused with an explicit message and the cache restarts. Same CPU-only status as `KV8`: GPU fast paths fall back to the CPU consumer; native kernels are follow-up PRs. Forces `COLI_CUDA_PIPE=0`. |
+| `KV_TQ` | `0` (off) | Sub-byte MLA latent KV quantization, mutually exclusive with `KV8` (`KV_TQ` wins). `KV_TQ=4` is the recommended tier: rotated-int4 codec (randomized-Hadamard rotation + Lloyd codebook, per-row radius as the scale), ~7.6Γ— less KV RAM than f32. `KV_TQ=2|3|5|6` selects the PolarQuant codec at that bit width (`KV_TQ_POLAR=1` forces PolarQuant at 4 bits too). Requires power-of-two row widths (`kv_lora`/`qk_rope`; the GLM MLA shapes 512/64 qualify) β€” on a model whose shapes don't, the engine refuses to start rather than silently zeroing the cache. A value below the 2–6 grid (e.g. `KV_TQ=1`) is treated as the recommended `4` with a notice, not as the most aggressive tier. `.coli_kv` is saved as the v3 format; a file saved under a different KV mode, codec, or bit width is refused with an explicit message and the cache restarts. Same CPU-only status as `KV8`: GPU fast paths fall back to the CPU consumer; native kernels are follow-up PRs. Forces `COLI_CUDA_PIPE=0`. |
| `THINK` | `0` (off) | Emit a `<think>` reasoning block. `THINK=1` turns on visible reasoning. |
| `MTP` | on | Multi-Token Prediction (speculative draft head). `MTP=0` disables it. |