Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs+stamp: non-retroactivity, stamp-map cap, discovery-time abort, .qs scope

monotophic1a18eb6feede
diff --git a/c/st.h b/c/st.h
index b01ffb03..67aa126d 100644
--- a/c/st.h
+++ b/c/st.h
@@ -239,6 +239,18 @@ static void st_pread_full(int fd, void *buf, int64_t n, int64_t off, const char
}
}
+/* Stamps are a resident-tensor convention (see docs/FORMATS.md's "Stamp-map
+ * scan bound"): a handful to a few hundred entries per model
+ * (q_a/q_b/kv_a/kv_b_proj, o_proj, shared-expert and dense-MLP gate/up/down),
+ * NEVER the tens of thousands of routed-expert tensors a large MoE model
+ * carries (tools/repack_fp8_passthrough.py never stamps routed experts). A
+ * container whose combined __metadata__["colibri.fmt"] entries exceed this
+ * cap is not using the convention as designed -- CAP, not a switch to a hash
+ * table: refuse loudly rather than grow an unbounded array for an untrusted
+ * container that might be trying to force a large ingest allocation during
+ * header parsing, before any other validation has run. */
+#define ST_FMT_STAMP_MAX 4096
+
/* Parses one shard's __metadata__["colibri.fmt"] value (a safetensors metadata
* value is always a plain string, so colibri.fmt's VALUE is itself JSON text --
* a flat {tensor_name: format_name} object -- parsed a second time here) and
@@ -250,7 +262,22 @@ static void st_pread_full(int fd, void *buf, int64_t n, int64_t off, const char
* loudly -- same "untrusted container" discipline qt_resolve_fmt applies
* elsewhere: a stamp the engine cannot make sense of must not be silently
* ignored (that would be indistinguishable from a real mismatch going
- * unnoticed). */
+ * unnoticed).
+ *
+ * DISCOVERY-TIME ABORT SURFACE: every exit(1) below (malformed stamp value,
+ * malformed entry, or the ST_FMT_STAMP_MAX cap) fires from inside
+ * st_init_multi's shard-header-parse loop -- i.e. at CONTAINER DISCOVERY
+ * time, while the engine is still building its tensor index, before it has
+ * resolved a single tensor against the model's architecture or read one byte
+ * of weight data. This is coarser-grained and EARLIER than
+ * qt_resolve_fmt/qt_verify_fmt_stamp's own per-tensor refusals in colibri.c
+ * (which fire much later, during weight load, once a specific tensor's
+ * [O,I] shape and stamp are both known): a malformed stamp anywhere in any
+ * shard aborts the ENTIRE model load immediately, before the user ever sees
+ * which layer or tensor was implicated -- these messages name a shard FILE,
+ * never a tensor, which is how to tell this abort surface apart from the
+ * later per-tensor one at a glance. See docs/FORMATS.md's own section on
+ * this. */
static void st_fmt_stamp_ingest(shards *S, jval *root, const char *shard_path) {
jval *meta = json_get(root, "__metadata__");
if (!meta || meta->t != J_OBJ) return; /* no metadata object: unstamped, fine */
@@ -269,6 +296,12 @@ static void st_fmt_stamp_ingest(shards *S, jval *root, const char *shard_path) {
if (v->t != J_STR) {
fprintf(stderr, "%s: colibri.fmt entry '%s' is not a string -- malformed stamp, refusing (untrusted container)\n",
shard_path, inner->keys[i]); exit(1); }
+ if (S->fmt_n >= ST_FMT_STAMP_MAX) {
+ fprintf(stderr, "%s: __metadata__[\"colibri.fmt\"] stamps more than %d tensor names across "
+ "this container's shards -- stamps are a resident-tensor convention (docs/FORMATS.md), "
+ "not a bulk migration path; a container stamping this many names is malformed, "
+ "refusing (untrusted container)\n",
+ shard_path, ST_FMT_STAMP_MAX); exit(1); }
if (S->fmt_n == S->fmt_cap) {
S->fmt_cap = S->fmt_cap ? S->fmt_cap * 2 : 16;
S->fmt_name = realloc(S->fmt_name, S->fmt_cap * sizeof(char*));
@@ -286,7 +319,19 @@ static void st_fmt_stamp_ingest(shards *S, jval *root, const char *shard_path) {
/* Stamped format NAME for `name`, or NULL if this tensor carries no stamp
* (either because no shard stamped it, or the container predates this
* feature). Linear scan: S->fmt_n is a subset of S->n (only stamped tensors),
- * small in practice -- see the shards struct comment. */
+ * small in practice -- see the shards struct comment.
+ *
+ * SCOPE: .qs-BACKED TENSORS ONLY. This function itself will happily look up
+ * ANY name that got stamped -- st_fmt_stamp_ingest doesn't know or check
+ * whether a stamped name belongs to a quantized (.qs-backed) tensor -- but
+ * qt_from_disk (colibri.c) only ever CALLS this inside its `st_has(name+
+ * ".qs")` branch, i.e. only for a tensor that already carries a quantized
+ * scale sidecar. A colibri.fmt entry naming a raw f32/bf16 weight, a norm, a
+ * router, or embed/lm_head is stored here like any other entry but never
+ * looked up: it is silently ignored BY DESIGN, not an oversight -- the
+ * convention exists to disambiguate a byte-count collision among quantized
+ * formats, and only a .qs-backed tensor can have one. See docs/FORMATS.md's
+ * "Scope: .qs-backed tensors only". */
static const char *st_fmt_stamp(shards *S, const char *name) {
for (int i = 0; i < S->fmt_n; i++)
if (!strcmp(S->fmt_name[i], name)) return S->fmt_val[i];
diff --git a/c/tests/test_fp8_load.c b/c/tests/test_fp8_load.c
index a6bdd216..40c6b191 100644
--- a/c/tests/test_fp8_load.c
+++ b/c/tests/test_fp8_load.c
@@ -959,6 +959,114 @@ static void test_stamp_absent(void){
char p[300]; snprintf(p,sizeof p,"%s/model.safetensors",dir); unlink(p); rmdir(dir);
}
+/* ---- Stamp-map scan bound (maintainer review, #529): st_fmt_stamp_ingest
+ * (st.h) caps the number of stamped-tensor entries it will ingest across a
+ * container's shards at ST_FMT_STAMP_MAX and refuses (exit(1)) past it --
+ * see st.h's own comment and docs/FORMATS.md's "Stamp-map scan bound" for
+ * why (stamps are a resident-tensor convention, never a bulk migration path
+ * for the tens of thousands of routed-expert tensors a large MoE model
+ * carries). Writes a minimal ONE-tensor shard (the shard's actual tensor
+ * content is irrelevant to this check -- the cap fires during
+ * st_init_multi's header-parse loop, at CONTAINER DISCOVERY time, before any
+ * tensor is resolved against the model, see st_fmt_stamp_ingest's own
+ * "DISCOVERY-TIME ABORT SURFACE" comment) carrying an oversized
+ * __metadata__["colibri.fmt"] blob of N distinct, entirely fictitious tensor
+ * names -- st_fmt_stamp_ingest parses that blob independently of the
+ * shard's real tensor list, so the names never need to correspond to
+ * anything real for the cap to trigger. Tests BOTH sides of the boundary:
+ * exactly ST_FMT_STAMP_MAX entries must still load fine (no off-by-one
+ * false refusal), one more must refuse. */
+static void write_stamp_cap_fixture(const char *dir, int n_entries){
+#ifdef _WIN32
+ mkdir(dir);
+#else
+ mkdir(dir,0755);
+#endif
+ size_t inner_cap = (size_t)n_entries*40 + 64;
+ char *inner = malloc(inner_cap);
+ size_t p = 0; inner[p++]='{';
+ for(int i=0;i<n_entries;i++)
+ p += (size_t)snprintf(inner+p, inner_cap-p, "%s\"stamp_%d\":\"int8-row\"", i?",":"", i);
+ inner[p++]='}'; inner[p]=0;
+
+ /* JSON-escape `inner` into a quoted string literal for the OUTER header's
+ * colibri.fmt VALUE (double-JSON-encoding, same shape write_stamp_fixture's
+ * callers hand-escape for small fixtures -- built programmatically here
+ * since n_entries is too many to hand-escape). */
+ char *esc = malloc(p*2 + 4);
+ size_t q = 0; esc[q++]='"';
+ for(size_t i=0;i<p;i++){
+ char c = inner[i];
+ if(c=='"' || c=='\\') esc[q++]='\\';
+ esc[q++]=c;
+ }
+ esc[q++]='"'; esc[q]=0;
+ free(inner);
+
+ enum { O=1, I=1 };
+ static uint8_t q1[O*I]; static float s1[O];
+ q1[0]=1; s1[0]=0.01f;
+ char path[300]; snprintf(path,sizeof path,"%s/model.safetensors",dir);
+ int64_t nb=(int64_t)O*I, ns=(int64_t)O*4;
+ char *hdr = malloc(q + 512);
+ int hl = snprintf(hdr, q+512,
+ "{\"__metadata__\":{\"colibri.fmt\":%s},"
+ "\"w\":{\"dtype\":\"U8\",\"shape\":[%lld],\"data_offsets\":[0,%lld]},"
+ "\"w.qs\":{\"dtype\":\"F32\",\"shape\":[%lld],\"data_offsets\":[%lld,%lld]}}",
+ esc,
+ (long long)nb,(long long)nb,
+ (long long)(ns/4),(long long)nb,(long long)(nb+ns));
+ free(esc);
+ FILE *f=fopen(path,"wb");
+ if(!f){ printf("FAIL: cannot create %s (run from c/, like tools/run_tests.py does)\n", path); fails++; free(hdr); return; }
+ uint64_t hlen=(uint64_t)hl;
+ fwrite(&hlen,8,1,f); fwrite(hdr,1,(size_t)hl,f);
+ fwrite(q1,1,(size_t)nb,f); fwrite(s1,1,(size_t)ns,f);
+ fclose(f);
+ free(hdr);
+}
+
+static void test_stamp_map_cap_boundary_ok(void){
+ const char *dir="tests/tmp_fp8_stamp_cap_ok";
+ write_stamp_cap_fixture(dir, ST_FMT_STAMP_MAX); /* exactly at the cap: must NOT refuse */
+ static Model gm; memset(&gm,0,sizeof gm);
+ st_init(&gm.S, dir);
+ CHECK(gm.S.fmt_n == ST_FMT_STAMP_MAX);
+ char p[300]; snprintf(p,sizeof p,"%s/model.safetensors",dir); unlink(p); rmdir(dir);
+}
+
+static void test_stamp_map_cap_exceeded(void){
+ const char *dir="tests/tmp_fp8_stamp_cap_over";
+ write_stamp_cap_fixture(dir, ST_FMT_STAMP_MAX+1); /* one past the cap: must refuse */
+#ifndef _WIN32
+ int pipefd[2];
+ if(pipe(pipefd)==0){
+ pid_t pid = fork();
+ if(pid == 0){
+ dup2(pipefd[1],2); close(pipefd[0]); close(pipefd[1]);
+ static Model gm; memset(&gm,0,sizeof gm);
+ st_init(&gm.S, dir); /* must exit(1) inside st_fmt_stamp_ingest's cap check */
+ _exit(42); /* reaching here is the bug */
+ } else if(pid > 0){
+ close(pipefd[1]);
+ char err[1024]={0}; ssize_t n=read(pipefd[0],err,sizeof(err)-1); (void)n;
+ close(pipefd[0]);
+ int status=0; waitpid(pid,&status,0);
+ int ok = WIFEXITED(status) && WEXITSTATUS(status)==1;
+ if(!ok) printf("FAIL stamp-map cap exceeded: expected exit(1), got status=%d, stderr=%.200s\n", status, err);
+ CHECK(ok);
+ if(ok && !strstr(err,"refus")){
+ printf("FAIL stamp-map cap exceeded: exited(1) but message lacked a refusal explanation: %.200s\n", err);
+ fails++;
+ }
+ } else fails++;
+ } else fails++;
+#else
+ printf("skipped on Windows (no fork): stamp-map cap exceeded\n");
+#endif
+ char p[300]; snprintf(p,sizeof p,"%s/model.safetensors",dir); unlink(p); rmdir(dir);
+}
+
int main(void){
test_disambiguation();
test_fmt6_fp8_collision();
@@ -982,6 +1090,8 @@ int main(void){
test_stamp_mismatching();
test_stamp_unrecognized_name();
test_stamp_absent();
+ test_stamp_map_cap_boundary_ok();
+ test_stamp_map_cap_exceeded();
if(fails){ printf("fp8 loader-seam tests: %d FAILED\n", fails); return 1; }
printf("fp8 loader-seam tests: ok\n");
return 0;
diff --git a/docs/FORMATS.md b/docs/FORMATS.md
index 6c52379f..dd2f27b1 100644
--- a/docs/FORMATS.md
+++ b/docs/FORMATS.md
@@ -203,6 +203,80 @@ reference implementation of this proposal:
documentation for the exact rule, including the cases where even a
correct stamp still can't resolve one (the UE8M0 corners above).
+### Non-retroactivity
+
+The stamp is a **forward convention**, not a migration path. It describes
+containers written *going forward* by a tool that has been updated to write
+`__metadata__["colibri.fmt"]` β€” it says nothing about, and does nothing to,
+containers that already exist. A pre-convention container (anything written
+before this feature existed β€” including every container this repo's own
+tooling has produced to date, and every upstream Z.ai/DeepSeek checkpoint)
+is simply **unstamped**: `st_fmt_stamp` returns `NULL` for every tensor in
+it, `qt_verify_fmt_stamp` is a no-op, and `qt_resolve_fmt` resolves it by
+byte arithmetic alone, exactly as it did before this PR. There is no
+in-place upgrade path, no "adopt the stamp on an existing container"
+tooling, and none is implied by anything in this document β€” the only way a
+container gains a stamp is to be produced (or re-produced) by a tool that
+writes one. An absent stamp is therefore never itself a signal that
+anything is wrong with a container; it is the default, expected state for
+everything that predates this convention (which, as of this PR, is
+everything).
+
+### Stamp-map scan bound
+
+`st_fmt_stamp_ingest` (`c/st.h`) caps the total number of stamped-tensor
+entries it will ingest across a container's shards at `ST_FMT_STAMP_MAX`
+(4096) and refuses loudly (`exit(1)`) if a container's combined
+`__metadata__["colibri.fmt"]` entries exceed it. This is a **cap, not a
+switch to a hash table**: stamps are a resident-tensor convention β€” a
+handful to a few hundred tensors per model (`q_a`/`q_b`/`kv_a`/`kv_b_proj`,
+`o_proj`, shared-expert and dense-MLP gate/up/down; see the resident-role
+census in `c/tools/fp8_collision_census.py`), **never** the tens of
+thousands of routed-expert tensors a large MoE checkpoint carries. A
+container whose combined stamp map exceeds 4096 entries is not using this
+convention as it's designed to be used β€” most plausibly a malformed or
+adversarial container trying to force an unbounded ingest allocation during
+header parsing, before any size/shape validation has even run. Refusing
+loudly at that bound is the same "untrusted container, refuse rather than
+guess" discipline `qt_resolve_fmt` applies everywhere else in this feature.
+
+### Discovery-time abort surface
+
+`st_fmt_stamp_ingest`'s `exit(1)` calls (a `colibri.fmt` value that isn't a
+JSON string, one that doesn't parse to a JSON object, an entry whose value
+isn't a string, or the 4096-entry cap above) all fire from inside
+`st_init_multi`'s shard-header-parse loop β€” i.e. at **container discovery
+time**, while the engine is still building its tensor index, before it has
+resolved a single tensor against the model's architecture or read one byte
+of weight data. This is a coarser-grained, EARLIER abort surface than
+`qt_resolve_fmt`/`qt_verify_fmt_stamp`'s own per-tensor refusals (which fire
+much later, once a *specific* tensor's `[O,I]` shape and stamp are both
+known during weight load): a malformed stamp anywhere in any shard aborts
+the *entire* model load immediately, before the user ever sees which layer
+or tensor was implicated. Operationally, a stamp-related `exit(1)` whose
+message names a *file* (shard path) rather than a *tensor* is this
+discovery-time surface; one that names a tensor is the later, per-tensor
+one.
+
+### Scope: `.qs`-backed tensors only
+
+The stamp lookup (`st_fmt_stamp`, called from `qt_from_disk`) is consulted
+**only** inside `qt_from_disk`'s `st_has(&m->S, name+".qs")` branch β€” i.e.
+only for a tensor that already carries a quantized `.qs` scale sidecar. A
+`colibri.fmt` entry naming some *other* tensor (a raw f32/bf16 weight, a
+norm, a router, embed/lm_head) is parsed and stored exactly like any other
+entry β€” `st_fmt_stamp_ingest` has no way to know, and does not check,
+whether the name it's ingesting belongs to a `.qs`-backed tensor β€” but it is
+then **silently ignored by design**: `qt_from_disk`'s plain f32/bf16 read
+path never calls `st_fmt_stamp` for it, so nothing ever consults that entry.
+This is intentional, not an oversight this document is patching over: the
+whole point of the stamp is to disambiguate a *byte-count* collision among
+quantized formats, and only a `.qs`-backed tensor can have one β€” stamping
+anything else has no failure mode to guard against, so the convention simply
+doesn't extend its verification there. A future stamping tool that writes
+`colibri.fmt` entries for non-`.qs` tensors would not break anything, but
+those entries would have no effect.
+
This is offered as a **reference implementation**, not a mandate: a future
public format registry could standardize the metadata key and stamp shape
(`colibri.fmt` and its JSON-map-of-name convention are this PR's proposal,