Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: document full-family fp8 repack output and container declaration
monotophic40dd6ecf8b71
diff --git a/c/tools/README.md b/c/tools/README.mdindex 659394ba..c6c1e93a 100644--- a/c/tools/README.md+++ b/c/tools/README.md@@ -7,8 +7,9 @@ not runtime dependencies of the C engine.- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)re-quant of a GLM-5.2 container, for Metal-backend compatibility(see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)-- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8, resident kinds only;- see the module docstring -- synthetic-fixture-tested only, no real-shard runs yet)+- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8, resident kinds and+ routed experts; `--mtp` for the multi-token-prediction head; see the module+ docstring -- synthetic-fixture-tested only, no real-shard runs yet)- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks- `gen_unicode.py`: tokenizer table generationdiff --git a/docs/FORMATS.md b/docs/FORMATS.mdindex ddfaa8f5..04ddccb3 100644--- a/docs/FORMATS.md+++ b/docs/FORMATS.md@@ -22,8 +22,9 @@ kernel, and collision/refusal logic for `fmt=8`. **Post-INVERSION** (theUNSTAMPED tensor at the fmt=1-vs-fmt=8 ambiguous shape (THE DESIGN LANDMINE)no longer refuses β it resolves to `int8-row`, the incumbent,already-on-disk, decodable format, and the writer (`repack_fp8_passthrough.py`)-refuses to ever EMIT an `fmt=8` container at that same ambiguous shape, so an-unstamped collision is never silently misread either way. This PR stacks on+refuses to EMIT an `fmt=8` tensor at that same ambiguous shape **unless the+reader will consult a stamp for it** (see "Emitting at the collision shape"+below), so an unstamped collision is never silently misread either way. This PR stacks ontop of that and adds the stamp (writer + reader): a stamp's role here isletting a genuinely-STAMPED `fmt=8` tensor at an ambiguous shape still beread as `fmt=8` (overriding the unstamped-case default) β and, for every@@ -318,6 +319,72 @@ anything is wrong with a container; it is the default, expected state foreverything that predates this convention (which, as of this PR, iseverything).+### Emitting at the collision shape++The writer-side rule the INVERSION made load bearing is **"never emit an+`fmt=8` tensor at the collision shape that the reader cannot resolve"** β not+"never emit at the collision shape". The two coincided while+`repack_fp8_passthrough.py` emitted only tensors it stamped and refused the+shape outright; they stop coinciding as soon as the tool emits anything the+reader will not consult a stamp for.++Which tensors those are is a **reader-side fact**, and the split runs exactly+along the resident/routed line:++- **Residents** load through `qt_from_disk`, which looks the tensor's name up+ with `st_fmt_stamp` and passes the result into `qt_resolve_fmt` as+ `stamped_name`. At the collision shape, `qt_resolve_fmt`'s *"THE STAMPED CASE+ IS UNCHANGED by this inversion"* branch resolves such a tensor to `fmt=8`+ through TRUST-VERIFY-REFUSE. **Emitting a stamped `fmt=8` resident at the+ collision shape uses the stamp mechanism as designed; it does not relax+ #528's invariant.** GLM-5.2's `self_attn.o_proj.weight` (`[6144,16384]`) is+ the one shape family in that checkpoint where this matters.+- **Routed experts** load through `expert_load_impl` and its two siblings,+ every one of which calls `qt_resolve_fmt` with `stamped_name = NULL`. A stamp+ on a routed expert is written and never read β and the `ST_FMT_STAMP_MAX` cap+ below is sized on precisely that premise. So routed experts are emitted+ **unstamped**, and for them the collision refusal stays unconditional.++`repack_fp8_passthrough.py` enforces this twice, deliberately: once in+`_check_geometry` (selection time, per tensor, with the `stampable` verdict) and+once in `_emission_stamp_gate`, which re-derives the collision predicate from+the **output tensors' own `[O,I]`** immediately before `save_file`. The second+check is what makes the guarantee structural rather than procedural: a refactor+that reclassifies a tensor or drops a stamp between selection and write trips a+gate that does not share the selection code's assumptions.++### Container-level format declaration++Alongside the per-tensor stamp, a container may carry+`__metadata__["colibri.container.formats"]` β JSON text holding a sorted array+of format NAME strings (never ordinals, same rule as the stamp) that the+container declares it carries residents in. `repack_fp8_passthrough.py` writes+`["fp8-e4m3-b128"]` on every shard it emits.++**No engine reads this key today.** It exists to close the one hole the+per-tensor stamp cannot close by itself: a stamp lives in `__metadata__`, and+any downstream tool that rewrites shards without preserving `__metadata__`+turns a stamped `fmt=8` tensor at the collision shape into an unstamped one,+which then resolves to `int8-row` silently β garbage weights, no error, the same+failure class the INVERSION is otherwise safe against. A reader that knows a+container **declares** fp8-e4m3 residents has the standing to treat an unstamped+ambiguous tensor *inside that container* as an ERROR rather than an int8+default, which converts a procedural guarantee back into a structural one.++The declaration lands first and the check lands second on purpose: an unknown+`__metadata__` key is inert to every existing engine (`st_fmt_stamp_ingest`+looks up `"colibri.fmt"` alone and returns early on anything else;+`st_init_multi` skips the whole `"__metadata__"` entry when indexing tensors),+so containers minted now already carry the declaration before anything enforces+it, and the enforcement has data to act on from the day it lands.+`tests/test_fp8_e2e_repack_load.py` pins the inertness: a container carrying the+key loads through the real `st_init`/`qt_from_disk` with no diagnostic.++**Known window:** between a writer emitting the declaration and a reader+enforcing it, containers declare and nothing checks β so the stamp-loss hole+stays open in that interval. That is no worse than the status quo, which has no+declaration at all.+### Duplicate claims, locality, coverageThree rules complete the stamp's container-wide semantics (user-ratified