Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: document full-family fp8 repack output and container declaration

monotophic40dd6ecf8b71
diff --git a/c/tools/README.md b/c/tools/README.md
index 659394ba..c6c1e93a 100644
--- a/c/tools/README.md
+++ b/c/tools/README.md
@@ -7,8 +7,9 @@ not runtime dependencies of the C engine.
- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)
re-quant of a GLM-5.2 container, for Metal-backend compatibility
(see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)
-- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8, resident kinds only;
- see the module docstring -- synthetic-fixture-tested only, no real-shard runs yet)
+- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8, resident kinds and
+ routed experts; `--mtp` for the multi-token-prediction head; see the module
+ docstring -- synthetic-fixture-tested only, no real-shard runs yet)
- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures
- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks
- `gen_unicode.py`: tokenizer table generation
diff --git a/docs/FORMATS.md b/docs/FORMATS.md
index ddfaa8f5..04ddccb3 100644
--- a/docs/FORMATS.md
+++ b/docs/FORMATS.md
@@ -22,8 +22,9 @@ kernel, and collision/refusal logic for `fmt=8`. **Post-INVERSION** (the
UNSTAMPED tensor at the fmt=1-vs-fmt=8 ambiguous shape (THE DESIGN LANDMINE)
no longer refuses β€” it resolves to `int8-row`, the incumbent,
already-on-disk, decodable format, and the writer (`repack_fp8_passthrough.py`)
-refuses to ever EMIT an `fmt=8` container at that same ambiguous shape, so an
-unstamped collision is never silently misread either way. This PR stacks on
+refuses to EMIT an `fmt=8` tensor at that same ambiguous shape **unless the
+reader will consult a stamp for it** (see "Emitting at the collision shape"
+below), so an unstamped collision is never silently misread either way. This PR stacks on
top of that and adds the stamp (writer + reader): a stamp's role here is
letting a genuinely-STAMPED `fmt=8` tensor at an ambiguous shape still be
read as `fmt=8` (overriding the unstamped-case default) β€” and, for every
@@ -318,6 +319,72 @@ anything is wrong with a container; it is the default, expected state for
everything that predates this convention (which, as of this PR, is
everything).
+### Emitting at the collision shape
+
+The writer-side rule the INVERSION made load bearing is **"never emit an
+`fmt=8` tensor at the collision shape that the reader cannot resolve"** β€” not
+"never emit at the collision shape". The two coincided while
+`repack_fp8_passthrough.py` emitted only tensors it stamped and refused the
+shape outright; they stop coinciding as soon as the tool emits anything the
+reader will not consult a stamp for.
+
+Which tensors those are is a **reader-side fact**, and the split runs exactly
+along the resident/routed line:
+
+- **Residents** load through `qt_from_disk`, which looks the tensor's name up
+ with `st_fmt_stamp` and passes the result into `qt_resolve_fmt` as
+ `stamped_name`. At the collision shape, `qt_resolve_fmt`'s *"THE STAMPED CASE
+ IS UNCHANGED by this inversion"* branch resolves such a tensor to `fmt=8`
+ through TRUST-VERIFY-REFUSE. **Emitting a stamped `fmt=8` resident at the
+ collision shape uses the stamp mechanism as designed; it does not relax
+ #528's invariant.** GLM-5.2's `self_attn.o_proj.weight` (`[6144,16384]`) is
+ the one shape family in that checkpoint where this matters.
+- **Routed experts** load through `expert_load_impl` and its two siblings,
+ every one of which calls `qt_resolve_fmt` with `stamped_name = NULL`. A stamp
+ on a routed expert is written and never read β€” and the `ST_FMT_STAMP_MAX` cap
+ below is sized on precisely that premise. So routed experts are emitted
+ **unstamped**, and for them the collision refusal stays unconditional.
+
+`repack_fp8_passthrough.py` enforces this twice, deliberately: once in
+`_check_geometry` (selection time, per tensor, with the `stampable` verdict) and
+once in `_emission_stamp_gate`, which re-derives the collision predicate from
+the **output tensors' own `[O,I]`** immediately before `save_file`. The second
+check is what makes the guarantee structural rather than procedural: a refactor
+that reclassifies a tensor or drops a stamp between selection and write trips a
+gate that does not share the selection code's assumptions.
+
+### Container-level format declaration
+
+Alongside the per-tensor stamp, a container may carry
+`__metadata__["colibri.container.formats"]` β€” JSON text holding a sorted array
+of format NAME strings (never ordinals, same rule as the stamp) that the
+container declares it carries residents in. `repack_fp8_passthrough.py` writes
+`["fp8-e4m3-b128"]` on every shard it emits.
+
+**No engine reads this key today.** It exists to close the one hole the
+per-tensor stamp cannot close by itself: a stamp lives in `__metadata__`, and
+any downstream tool that rewrites shards without preserving `__metadata__`
+turns a stamped `fmt=8` tensor at the collision shape into an unstamped one,
+which then resolves to `int8-row` silently β€” garbage weights, no error, the same
+failure class the INVERSION is otherwise safe against. A reader that knows a
+container **declares** fp8-e4m3 residents has the standing to treat an unstamped
+ambiguous tensor *inside that container* as an ERROR rather than an int8
+default, which converts a procedural guarantee back into a structural one.
+
+The declaration lands first and the check lands second on purpose: an unknown
+`__metadata__` key is inert to every existing engine (`st_fmt_stamp_ingest`
+looks up `"colibri.fmt"` alone and returns early on anything else;
+`st_init_multi` skips the whole `"__metadata__"` entry when indexing tensors),
+so containers minted now already carry the declaration before anything enforces
+it, and the enforcement has data to act on from the day it lands.
+`tests/test_fp8_e2e_repack_load.py` pins the inertness: a container carrying the
+key loads through the real `st_init`/`qt_from_disk` with no diagnostic.
+
+**Known window:** between a writer emitting the declaration and a reader
+enforcing it, containers declare and nothing checks β€” so the stamp-loss hole
+stays open in that interval. That is no worse than the status quo, which has no
+declaration at all.
+
### Duplicate claims, locality, coverage
Three rules complete the stamp's container-wide semantics (user-ratified