Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: deliver the fp8 full-family and kv_b_proj decode-caveat documentation

monotophic6b7a13d8039e
diff --git a/c/tools/README.md b/c/tools/README.md
index c6c1e93a..7a7135e8 100644
--- a/c/tools/README.md
+++ b/c/tools/README.md
@@ -7,9 +7,18 @@ not runtime dependencies of the C engine.
- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)
re-quant of a GLM-5.2 container, for Metal-backend compatibility
(see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)
-- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8, resident kinds and
- routed experts; `--mtp` for the multi-token-prediction head; see the module
- docstring -- synthetic-fixture-tested only, no real-shard runs yet)
+- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8) minting a
+ standalone-loadable model directory: routed experts plus the full resident
+ family (attention projections including `kv_b_proj`, shared-expert and
+ dense-MLP weights), norms/router/embed/lm_head copied byte-identically in
+ their original BF16/F32, and the non-tensor files the loader opens at
+ runtime (`config.json`, `tokenizer.json`; `generation_config.json`
+ best-effort). `--mtp` repacks the multi-token-prediction head as a separate
+ pass into the same outdir. Caveat: the minted `kv_b_proj` loads today but
+ must not serve batched-path decode until the engine's fmt=8 absorb support
+ lands (branch `f8/absorb-fmt8`); the failure is a loud crash, not silent
+ corruption -- see the module docstring and `docs/FORMATS.md`.
+ Synthetic-fixture-tested only, no real-shard runs yet.
- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures
- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks
- `gen_unicode.py`: tokenizer table generation
diff --git a/docs/FORMATS.md b/docs/FORMATS.md
index 04ddccb3..388b99ab 100644
--- a/docs/FORMATS.md
+++ b/docs/FORMATS.md
@@ -385,6 +385,40 @@ enforcing it, containers declare and nothing checks β€” so the stamp-loss hole
stays open in that interval. That is no worse than the status quo, which has no
declaration at all.
+### Full-family emission and the `kv_b_proj` decode constraint
+
+`repack_fp8_passthrough.py` emits the **full tensor family**, making its
+`--outdir` a standalone-loadable model directory rather than a partial overlay:
+
+- **fp8-repacked** (byte-preserved weight + renamed `.qs` scale sidecar):
+ routed experts (unstamped, see above) and the resident projections β€”
+ `o_proj`, `q_a`/`q_b`/`kv_a`, `kv_b_proj`, shared-expert and dense-MLP
+ weights (stamped).
+- **Raw pass-through** (byte-identical copy, original BF16/F32 dtype, no
+ `.qs`, no stamp): norms, router (`mlp.gate.weight`), `e_score_correction_bias`,
+ `embed_tokens`, `lm_head`. This exercises nothing new on the read side β€” it
+ is exactly the plain `st_read_f32` fallback path `qt_from_disk` already takes
+ for any tensor without a `.qs` sidecar.
+- **Non-tensor files**: `config.json` and `tokenizer.json` are copied from
+ `--indir` (both are mandatory β€” the engine `exit(1)`s without either, so the
+ tool refuses to mint if `--indir` lacks them); `generation_config.json` is
+ copied best-effort, matching `load_cfg`'s own optional handling. No
+ `model.safetensors.index.json` is copied or needed: `st_init_multi` globs
+ `*.safetensors` and builds its own index, and a copied index would describe
+ the *source* shard layout, not the minted one.
+
+**Decode constraint on the minted `kv_b_proj`:** the container-side work is
+deliberately ahead of the engine-side work. A minted `kv_b_proj` **loads**
+today exactly like `o_proj` (same `qt_from_disk` path, same stamp), but the
+MLA-absorption paths that consume it β€” `qt_addrow`/`qt_matvec_rows` on CPU and
+the CUDA absorb kernels β€” have **no `fmt=8` case yet**; that support is being
+built in parallel on `f8/absorb-fmt8`. Until it lands, a container minted with
+`kv_b_proj` must not be used for decode that reaches the batched
+(`kvs`-non-NULL) serving path. The failure mode is a **loud crash, not a
+silent misread** (`ABSORB=0` cannot bypass absorb there), and sequencing β€”
+engine support before such a container serves decode β€” is release-gate work,
+not something the tool can enforce from the write side.
+
### Duplicate claims, locality, coverage
Three rules complete the stamp's container-wide semantics (user-ratified