Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

fix(ablate): derive the per-item token cap from CTX, bound all three allocations

monotophic4925883b2e04
diff --git a/c/colibri.c b/c/colibri.c
index 2b84b666..17e41f73 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -8053,12 +8053,63 @@ static void run_score(Model *m, const char *snap, const char *path){
* never leak into the next). NLL/margin/correctness are exact; top-K is for a
* paired approximate next-token KL on the host. */
#define ABL_LOGIT_TOPK 32
-/* Upper bound on one manifest item's declared token count. A teacher-forced
- * ablation item is a prompt plus a short continuation; a million tokens is
- * orders of magnitude above anything a study uses, and it keeps a hostile or
- * corrupt manifest from asking the loader for an arbitrary allocation. */
+/* Backstop ceiling on one manifest item's declared token count T. The
+ * operative bound applied in ablate_manifest_load (below) is
+ * ablate_item_token_limit() = min(CTX, ABLATE_MAX_ITEM_TOKENS): CTX
+ * ("Maximum context length (tokens) the KV cache is sized for",
+ * docs/ENVIRONMENT.md -- the same max_ctx parameter kv_pool_bytes and
+ * expert_avail already take) is an existing engine limit, not an
+ * invented one, and an item longer than the context this engine is
+ * configured to hold cannot be processed regardless of what the
+ * manifest asks for. This constant only bounds CTX itself, in case an
+ * operator sets it to something absurd.
+ *
+ * That check runs before this item is allowed to contribute to any of
+ * THREE allocations, largest first -- naming only the one the maintainer
+ * raised, or getting the ranking backwards, is how an earlier draft of
+ * this comment was wrong twice over:
+ *
+ * 1. kv_alloc's per-layer KV buffers, sized from the manifest-wide
+ * maxT once parsing finishes (`(n_layers+1)*max_t*(kv_lora+qk_rope)`,
+ * below; maxT can equal this item's own T).
+ * 2. ablate_model_output_body's prefill buffer `x`, also sized from
+ * maxT (`falloc((int64_t)maxT*D)`, D = c->hidden, further below).
+ * 3. This loop's own resident bookkeeping: one AblateManifestItem plus
+ * one AblateItemRef appended per accepted item (both defined
+ * above) -- bounded per item by this same check, but unbounded in
+ * ITEM COUNT across a whole manifest, a separate axis this cap
+ * does not close.
+ *
+ * Worked at the backstop ceiling ABLATE_MAX_ITEM_TOKENS = 2^20 (reachable
+ * only if CTX is itself set that high), with GLM-5.2's real config
+ * (Cfg.hidden=6144, n_layers=78, kv_lora=512, qk_rope=64 -- all loaded
+ * together in load_cfg in this file):
+ *
+ * 1. kv_alloc: (n_layers+1) * maxT * (kv_lora+qk_rope) * sizeof(float)
+ * = 79 * 2^20 * 576 * 4 B = 190,857,609,216 B = 177.75 GiB
+ * 2. prefill x: maxT * hidden * sizeof(float)
+ * = 2^20 * 6144 * 4 B = 25,769,803,776 B = 24.00 GiB
+ * 3. resident: sizeof(AblateManifestItem) + sizeof(AblateItemRef)
+ * = 224 B + 16 B = 240 B per item (illustrative only --
+ * unbounded in item count, not item length)
+ *
+ * kv_alloc is 7.41x the prefill buffer, not the smaller of the two. */
#define ABLATE_MAX_ITEM_TOKENS (1<<20)
+/* See the comment above: the bound actually applied to one manifest
+ * item's T is the smaller of CTX and ABLATE_MAX_ITEM_TOKENS, not the
+ * backstop constant alone. Reads CTX itself (not a cached max_ctx)
+ * because ablate mode has no serving session to inherit one from; same
+ * default (4096) and same unvalidated-atoi parsing as every other CTX
+ * reader in this file, for one consistent meaning of "context length"
+ * throughout. */
+static int64_t ablate_item_token_limit(void){
+ const char *env=getenv("CTX");
+ int ctx = env ? atoi(env) : 4096;
+ if(ctx<=0) ctx=4096;
+ return ctx<ABLATE_MAX_ITEM_TOKENS ? (int64_t)ctx : (int64_t)ABLATE_MAX_ITEM_TOKENS;
+}
+
static int logit_topk_count(int V, int requested){
if(V<=0 || requested<=0) return 0;
int k=requested;
@@ -8260,6 +8311,7 @@ static int ablate_manifest_load(FILE *f, const Cfg *c,
AblateItemRef *item_ids=NULL;
size_t item_cap=0, item_count=0, row_cap=0;
int64_t target_count=0;
+ const int64_t item_token_limit=ablate_item_token_limit();
int maxT=1, rc=0;
char *line=NULL; size_t cap=0; ssize_t length; int64_t line_no=0, bad_line=0;
int named=0; /* a specific reason was already printed */
@@ -8292,14 +8344,17 @@ static int ablate_manifest_load(FILE *f, const Cfg *c,
rc=1; break;
}
/* A manifest is host input: an item may not ask for an unbounded
- * allocation by declaring an enormous length. The cap is far above any
- * teacher-forced item a study runs -- the engine also allocates
- * maxT * hidden floats for the prefill -- and it bounds the token array
- * this loop is about to allocate to a few megabytes. */
- if(T>ABLATE_MAX_ITEM_TOKENS){
+ * allocation by declaring an enormous length. item_token_limit is
+ * min(CTX, ABLATE_MAX_ITEM_TOKENS) -- see the comment on that
+ * constant above -- and this check runs before the item can
+ * contribute to any of the three allocations named there: this
+ * loop's own resident bookkeeping (just below), and, once maxT is
+ * known, kv_alloc's per-layer buffers and the prefill buffer `x`
+ * (both sized from maxT, which this item's own T feeds). */
+ if(T>item_token_limit){
fprintf(stderr,"[ablate] INCOMPLETE: line %" PRId64 " declares %" PRId64
- " tokens, above the %d-token limit for one item\n",
- line_no,T,ABLATE_MAX_ITEM_TOKENS);
+ " tokens, above the %" PRId64 "-token limit for one item\n",
+ line_no,T,item_token_limit);
named=1; rc=1; bad_line=line_no; break;
}
if(item<0 || T<2 || T>INT_MAX || np<1 || np>=T ||
diff --git a/c/tests/test_ablate_mode.c b/c/tests/test_ablate_mode.c
index 39adb4f7..097e004f 100644
--- a/c/tests/test_ablate_mode.c
+++ b/c/tests/test_ablate_mode.c
@@ -423,6 +423,59 @@ static void t_bounded_item_length(Cfg *c){
remove(STDERR_PATH);
}
+/* "item T n_prompt mode ncells" (mode 0, ncells 0) followed by T real token
+ * values, each within [0, vocab) for this file's CONFIG_JSON fixture
+ * (vocab=64) -- unlike t_bounded_item_length's over-length manifests, this
+ * one is fully well-formed and can actually run to completion. Caller
+ * frees the result. */
+static char *build_full_item_manifest(int64_t t, int64_t n_prompt){
+ size_t cap=64+(size_t)t*4;
+ char *buf=malloc(cap);
+ if(!buf) return NULL;
+ int off=snprintf(buf,cap,"0 %lld %lld 0 0",(long long)t,(long long)n_prompt);
+ for(int64_t i=0;i<t && off>0;i++)
+ off+=snprintf(buf+off,cap-(size_t)off," %d",(int)(i%60));
+ if(off<=0 || (size_t)off+2>cap){ free(buf); return NULL; }
+ buf[off]='\n'; buf[off+1]=0;
+ return buf;
+}
+
+static void t_ctx_derived_item_limit(Cfg *c){
+ printf("P9 the per-item token limit is derived from CTX, not a bare constant\n");
+ setenv("CTX","4096",1);
+
+ /* cap+1 (4097): refused by name, before the item can contribute to any
+ * of the three allocations the limit exists to gate. */
+ char *over=build_full_item_manifest(4097,2);
+ char *text = over ? refusal_text_of(c,over) : NULL;
+ free(over);
+ CHECK(text && strstr(text,"above the 4096-token limit")!=NULL,
+ "cap+1 (4097 tokens) is refused, naming the CTX=4096-derived limit");
+ free(text);
+
+ /* The real maximum (4096, CTX itself): a well-formed item at exactly the
+ * derived limit is accepted and runs to completion. Without this case, a
+ * limit that simply refused every manifest would also pass the case
+ * above. */
+ char *at_cap=build_full_item_manifest(4096,2);
+ CHECK(at_cap!=NULL, "the at-cap fixture manifest was built");
+ if(at_cap){
+ manifest_write(at_cap);
+ free(at_cap);
+ remove(EVIDENCE_PATH);
+ int rc=run_mode(c,MANIFEST_PATH,EVIDENCE_PATH);
+ CHECK(rc==0, "a manifest at the real CTX-derived maximum (4096) loads and runs");
+ char *ev=slurp(EVIDENCE_PATH,NULL);
+ CHECK(ev && strstr(ev,"\"completed_items\":1")!=NULL &&
+ strstr(ev,"\"completed_targets\":4094")!=NULL,
+ "the accepted 4096-token item completes all 4094 of its targets");
+ free(ev);
+ remove(EVIDENCE_PATH);
+ }
+ remove(STDERR_PATH);
+ unsetenv("CTX");
+}
+
/* nll must come from a subtraction done wholly in double
* (logZ - lo[gold]), not from a float-rounded intermediate. A row this
* wide (max 1.0e7, gold logit 0.25, everything else far below the max)
@@ -611,6 +664,7 @@ int main(int argc, char **argv){
t_named_refusals(&c);
t_refusal_names_the_offender(&c);
t_bounded_item_length(&c);
+ t_ctx_derived_item_limit(&c);
t_nll_pin(&c);
t_load_cfg_digest_binding(&c);
remove(MANIFEST_PATH);
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index e6bee824..84239048 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -300,8 +300,8 @@ These are for testing, benchmarking, or internal use β€” not part of the everyda
| `ORACLE_TF_MAX_MISMATCHES` | `0` | `colibri` only, env-only. Maximum token mismatches accepted with `ORACLE_STRICT=1` and `TF` set. Must be a nonnegative decimal integer smaller than the number of TF positions. CI uses `2` for the 32-position tiny fixture (30–32 matches); unset or `0` requires exact agreement. Mismatches remain visible in diagnostics. Ignored outside strict TF mode; cannot relax greedy comparison or non-finite-output checks. |
| `CHAT_TEMPLATE` | `1` | Apply the GLM chat template (`0` = raw prompt). |
| `PPL` | off (`olmoe.c` and `qwen38.c` only) | `PPL=1` enters teacher-forced NLL/perplexity meter mode in the OLMoE and Qwen3.8 sister engines. |
-| `ABLATE_SCORE` | unset | Causal-ablation sweep over `ABLATE_SCORE=<file>`, with a per-target-position final-logit read-out. Runs before `SCORE` and exits when done. |
-| `ABLATE_OUT` | unset | Where the ablation sweep writes its logit read-out. Pair with `ABLATE_SCORE`; an optional `ROUTE_TRACE` records the post-ablation router trace. |
+| `ABLATE_SCORE` | unset | Causal-ablation sweep over the manifest named by `ABLATE_SCORE=<file>`: one teacher-forced prefill per item, with a final-logit read-out at every target position. Runs before `SCORE` and exits when done, returning 0 only if every item completed. One item per line, fields separated by single spaces: `item T n_prompt mode ncells (layer expert swap){ncells} token{T}`. `mode` is 0 baseline, 1 contribution, 2 route-around, 3 module-swap; `swap` is `-1` except in mode 3. Every field is checked against the loaded config before anything runs, and a manifest that does not fit the format is refused whole, with the offending line named on stderr β€” no part of it is executed. One item's declared token count is refused above `min(CTX, 1,048,576)` tokens β€” the same context length `CTX` already sizes the KV cache for, not a bare constant β€” so a corrupt or hostile manifest cannot size the loader's per-item token array, the run's KV allocation, or its prefill buffer beyond what this engine's configured context could ever actually need. CRLF line endings and a missing final newline are accepted. |
+| `ABLATE_OUT` | unset | Where the ablation sweep writes its read-out, as JSON lines: a header naming the run's vocabulary, layer/expert geometry, the SHA-256 of the loaded `config.json` and of the manifest, and the item and target counts to expect; one record per item; one record per target position carrying the gold token's negative log-likelihood, logits, margin, argmax and top-k; and a final record with the counts actually completed, so a truncated file can be told from a complete one. The path must not already exist β€” an existing file, link or directory entry is refused rather than overwritten. Optional: without `ABLATE_OUT` the sweep runs but writes nothing, and an accompanying `ROUTE_TRACE` records the post-ablation router trace. `c/tools/check_ablate_evidence.py` validates a finished file against the manifest and config it names. |
| `DEBUG_LOGITS` | unset | In reference-comparison mode, dump per-position logit diagnostics. |
| `COLI_LOGIT_DUMP` | unset | `=1` prints the top-5 `id:logit` pairs per step to stderr β€” for comparing two engine configs on identical forced context (backend-exactness triage). |
| `I3_AVX512` | auto | Force the AVX-512 int3 kernel on (`1`) or off (`0`). |