Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
fix(ablate): derive the per-item token cap from CTX, bound all three allocations
monotophic4925883b2e04
diff --git a/c/colibri.c b/c/colibri.cindex 2b84b666..17e41f73 100644--- a/c/colibri.c+++ b/c/colibri.c@@ -8053,12 +8053,63 @@ static void run_score(Model *m, const char *snap, const char *path){* never leak into the next). NLL/margin/correctness are exact; top-K is for a* paired approximate next-token KL on the host. */#define ABL_LOGIT_TOPK 32-/* Upper bound on one manifest item's declared token count. A teacher-forced- * ablation item is a prompt plus a short continuation; a million tokens is- * orders of magnitude above anything a study uses, and it keeps a hostile or- * corrupt manifest from asking the loader for an arbitrary allocation. */+/* Backstop ceiling on one manifest item's declared token count T. The+ * operative bound applied in ablate_manifest_load (below) is+ * ablate_item_token_limit() = min(CTX, ABLATE_MAX_ITEM_TOKENS): CTX+ * ("Maximum context length (tokens) the KV cache is sized for",+ * docs/ENVIRONMENT.md -- the same max_ctx parameter kv_pool_bytes and+ * expert_avail already take) is an existing engine limit, not an+ * invented one, and an item longer than the context this engine is+ * configured to hold cannot be processed regardless of what the+ * manifest asks for. This constant only bounds CTX itself, in case an+ * operator sets it to something absurd.+ *+ * That check runs before this item is allowed to contribute to any of+ * THREE allocations, largest first -- naming only the one the maintainer+ * raised, or getting the ranking backwards, is how an earlier draft of+ * this comment was wrong twice over:+ *+ * 1. kv_alloc's per-layer KV buffers, sized from the manifest-wide+ * maxT once parsing finishes (`(n_layers+1)*max_t*(kv_lora+qk_rope)`,+ * below; maxT can equal this item's own T).+ * 2. ablate_model_output_body's prefill buffer `x`, also sized from+ * maxT (`falloc((int64_t)maxT*D)`, D = c->hidden, further below).+ * 3. This loop's own resident bookkeeping: one AblateManifestItem plus+ * one AblateItemRef appended per accepted item (both defined+ * above) -- bounded per item by this same check, but unbounded in+ * ITEM COUNT across a whole manifest, a separate axis this cap+ * does not close.+ *+ * Worked at the backstop ceiling ABLATE_MAX_ITEM_TOKENS = 2^20 (reachable+ * only if CTX is itself set that high), with GLM-5.2's real config+ * (Cfg.hidden=6144, n_layers=78, kv_lora=512, qk_rope=64 -- all loaded+ * together in load_cfg in this file):+ *+ * 1. kv_alloc: (n_layers+1) * maxT * (kv_lora+qk_rope) * sizeof(float)+ * = 79 * 2^20 * 576 * 4 B = 190,857,609,216 B = 177.75 GiB+ * 2. prefill x: maxT * hidden * sizeof(float)+ * = 2^20 * 6144 * 4 B = 25,769,803,776 B = 24.00 GiB+ * 3. resident: sizeof(AblateManifestItem) + sizeof(AblateItemRef)+ * = 224 B + 16 B = 240 B per item (illustrative only --+ * unbounded in item count, not item length)+ *+ * kv_alloc is 7.41x the prefill buffer, not the smaller of the two. */#define ABLATE_MAX_ITEM_TOKENS (1<<20)+/* See the comment above: the bound actually applied to one manifest+ * item's T is the smaller of CTX and ABLATE_MAX_ITEM_TOKENS, not the+ * backstop constant alone. Reads CTX itself (not a cached max_ctx)+ * because ablate mode has no serving session to inherit one from; same+ * default (4096) and same unvalidated-atoi parsing as every other CTX+ * reader in this file, for one consistent meaning of "context length"+ * throughout. */+static int64_t ablate_item_token_limit(void){+ const char *env=getenv("CTX");+ int ctx = env ? atoi(env) : 4096;+ if(ctx<=0) ctx=4096;+ return ctx<ABLATE_MAX_ITEM_TOKENS ? (int64_t)ctx : (int64_t)ABLATE_MAX_ITEM_TOKENS;+}+static int logit_topk_count(int V, int requested){if(V<=0 || requested<=0) return 0;int k=requested;@@ -8260,6 +8311,7 @@ static int ablate_manifest_load(FILE *f, const Cfg *c,AblateItemRef *item_ids=NULL;size_t item_cap=0, item_count=0, row_cap=0;int64_t target_count=0;+ const int64_t item_token_limit=ablate_item_token_limit();int maxT=1, rc=0;char *line=NULL; size_t cap=0; ssize_t length; int64_t line_no=0, bad_line=0;int named=0; /* a specific reason was already printed */@@ -8292,14 +8344,17 @@ static int ablate_manifest_load(FILE *f, const Cfg *c,rc=1; break;}/* A manifest is host input: an item may not ask for an unbounded- * allocation by declaring an enormous length. The cap is far above any- * teacher-forced item a study runs -- the engine also allocates- * maxT * hidden floats for the prefill -- and it bounds the token array- * this loop is about to allocate to a few megabytes. */- if(T>ABLATE_MAX_ITEM_TOKENS){+ * allocation by declaring an enormous length. item_token_limit is+ * min(CTX, ABLATE_MAX_ITEM_TOKENS) -- see the comment on that+ * constant above -- and this check runs before the item can+ * contribute to any of the three allocations named there: this+ * loop's own resident bookkeeping (just below), and, once maxT is+ * known, kv_alloc's per-layer buffers and the prefill buffer `x`+ * (both sized from maxT, which this item's own T feeds). */+ if(T>item_token_limit){fprintf(stderr,"[ablate] INCOMPLETE: line %" PRId64 " declares %" PRId64- " tokens, above the %d-token limit for one item\n",- line_no,T,ABLATE_MAX_ITEM_TOKENS);+ " tokens, above the %" PRId64 "-token limit for one item\n",+ line_no,T,item_token_limit);named=1; rc=1; bad_line=line_no; break;}if(item<0 || T<2 || T>INT_MAX || np<1 || np>=T ||diff --git a/c/tests/test_ablate_mode.c b/c/tests/test_ablate_mode.cindex 39adb4f7..097e004f 100644--- a/c/tests/test_ablate_mode.c+++ b/c/tests/test_ablate_mode.c@@ -423,6 +423,59 @@ static void t_bounded_item_length(Cfg *c){remove(STDERR_PATH);}+/* "item T n_prompt mode ncells" (mode 0, ncells 0) followed by T real token+ * values, each within [0, vocab) for this file's CONFIG_JSON fixture+ * (vocab=64) -- unlike t_bounded_item_length's over-length manifests, this+ * one is fully well-formed and can actually run to completion. Caller+ * frees the result. */+static char *build_full_item_manifest(int64_t t, int64_t n_prompt){+ size_t cap=64+(size_t)t*4;+ char *buf=malloc(cap);+ if(!buf) return NULL;+ int off=snprintf(buf,cap,"0 %lld %lld 0 0",(long long)t,(long long)n_prompt);+ for(int64_t i=0;i<t && off>0;i++)+ off+=snprintf(buf+off,cap-(size_t)off," %d",(int)(i%60));+ if(off<=0 || (size_t)off+2>cap){ free(buf); return NULL; }+ buf[off]='\n'; buf[off+1]=0;+ return buf;+}++static void t_ctx_derived_item_limit(Cfg *c){+ printf("P9 the per-item token limit is derived from CTX, not a bare constant\n");+ setenv("CTX","4096",1);++ /* cap+1 (4097): refused by name, before the item can contribute to any+ * of the three allocations the limit exists to gate. */+ char *over=build_full_item_manifest(4097,2);+ char *text = over ? refusal_text_of(c,over) : NULL;+ free(over);+ CHECK(text && strstr(text,"above the 4096-token limit")!=NULL,+ "cap+1 (4097 tokens) is refused, naming the CTX=4096-derived limit");+ free(text);++ /* The real maximum (4096, CTX itself): a well-formed item at exactly the+ * derived limit is accepted and runs to completion. Without this case, a+ * limit that simply refused every manifest would also pass the case+ * above. */+ char *at_cap=build_full_item_manifest(4096,2);+ CHECK(at_cap!=NULL, "the at-cap fixture manifest was built");+ if(at_cap){+ manifest_write(at_cap);+ free(at_cap);+ remove(EVIDENCE_PATH);+ int rc=run_mode(c,MANIFEST_PATH,EVIDENCE_PATH);+ CHECK(rc==0, "a manifest at the real CTX-derived maximum (4096) loads and runs");+ char *ev=slurp(EVIDENCE_PATH,NULL);+ CHECK(ev && strstr(ev,"\"completed_items\":1")!=NULL &&+ strstr(ev,"\"completed_targets\":4094")!=NULL,+ "the accepted 4096-token item completes all 4094 of its targets");+ free(ev);+ remove(EVIDENCE_PATH);+ }+ remove(STDERR_PATH);+ unsetenv("CTX");+}+/* nll must come from a subtraction done wholly in double* (logZ - lo[gold]), not from a float-rounded intermediate. A row this* wide (max 1.0e7, gold logit 0.25, everything else far below the max)@@ -611,6 +664,7 @@ int main(int argc, char **argv){t_named_refusals(&c);t_refusal_names_the_offender(&c);t_bounded_item_length(&c);+ t_ctx_derived_item_limit(&c);t_nll_pin(&c);t_load_cfg_digest_binding(&c);remove(MANIFEST_PATH);diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex e6bee824..84239048 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -300,8 +300,8 @@ These are for testing, benchmarking, or internal use β not part of the everyda| `ORACLE_TF_MAX_MISMATCHES` | `0` | `colibri` only, env-only. Maximum token mismatches accepted with `ORACLE_STRICT=1` and `TF` set. Must be a nonnegative decimal integer smaller than the number of TF positions. CI uses `2` for the 32-position tiny fixture (30β32 matches); unset or `0` requires exact agreement. Mismatches remain visible in diagnostics. Ignored outside strict TF mode; cannot relax greedy comparison or non-finite-output checks. || `CHAT_TEMPLATE` | `1` | Apply the GLM chat template (`0` = raw prompt). || `PPL` | off (`olmoe.c` and `qwen38.c` only) | `PPL=1` enters teacher-forced NLL/perplexity meter mode in the OLMoE and Qwen3.8 sister engines. |-| `ABLATE_SCORE` | unset | Causal-ablation sweep over `ABLATE_SCORE=<file>`, with a per-target-position final-logit read-out. Runs before `SCORE` and exits when done. |-| `ABLATE_OUT` | unset | Where the ablation sweep writes its logit read-out. Pair with `ABLATE_SCORE`; an optional `ROUTE_TRACE` records the post-ablation router trace. |+| `ABLATE_SCORE` | unset | Causal-ablation sweep over the manifest named by `ABLATE_SCORE=<file>`: one teacher-forced prefill per item, with a final-logit read-out at every target position. Runs before `SCORE` and exits when done, returning 0 only if every item completed. One item per line, fields separated by single spaces: `item T n_prompt mode ncells (layer expert swap){ncells} token{T}`. `mode` is 0 baseline, 1 contribution, 2 route-around, 3 module-swap; `swap` is `-1` except in mode 3. Every field is checked against the loaded config before anything runs, and a manifest that does not fit the format is refused whole, with the offending line named on stderr β no part of it is executed. One item's declared token count is refused above `min(CTX, 1,048,576)` tokens β the same context length `CTX` already sizes the KV cache for, not a bare constant β so a corrupt or hostile manifest cannot size the loader's per-item token array, the run's KV allocation, or its prefill buffer beyond what this engine's configured context could ever actually need. CRLF line endings and a missing final newline are accepted. |+| `ABLATE_OUT` | unset | Where the ablation sweep writes its read-out, as JSON lines: a header naming the run's vocabulary, layer/expert geometry, the SHA-256 of the loaded `config.json` and of the manifest, and the item and target counts to expect; one record per item; one record per target position carrying the gold token's negative log-likelihood, logits, margin, argmax and top-k; and a final record with the counts actually completed, so a truncated file can be told from a complete one. The path must not already exist β an existing file, link or directory entry is refused rather than overwritten. Optional: without `ABLATE_OUT` the sweep runs but writes nothing, and an accompanying `ROUTE_TRACE` records the post-ablation router trace. `c/tools/check_ablate_evidence.py` validates a finished file against the manifest and config it names. || `DEBUG_LOGITS` | unset | In reference-comparison mode, dump per-position logit diagnostics. || `COLI_LOGIT_DUMP` | unset | `=1` prints the top-5 `id:logit` pairs per step to stderr β for comparing two engine configs on identical forced context (backend-exactness triage). || `I3_AVX512` | auto | Force the AVX-512 int3 kernel on (`1`) or off (`0`). |