Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: deliver the fp8 full-family and kv_b_proj decode-caveat documentation
monotophic6b7a13d8039e
diff --git a/c/tools/README.md b/c/tools/README.mdindex c6c1e93a..7a7135e8 100644--- a/c/tools/README.md+++ b/c/tools/README.md@@ -7,9 +7,18 @@ not runtime dependencies of the C engine.- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)re-quant of a GLM-5.2 container, for Metal-backend compatibility(see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)-- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8, resident kinds and- routed experts; `--mtp` for the multi-token-prediction head; see the module- docstring -- synthetic-fixture-tested only, no real-shard runs yet)+- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8) minting a+ standalone-loadable model directory: routed experts plus the full resident+ family (attention projections including `kv_b_proj`, shared-expert and+ dense-MLP weights), norms/router/embed/lm_head copied byte-identically in+ their original BF16/F32, and the non-tensor files the loader opens at+ runtime (`config.json`, `tokenizer.json`; `generation_config.json`+ best-effort). `--mtp` repacks the multi-token-prediction head as a separate+ pass into the same outdir. Caveat: the minted `kv_b_proj` loads today but+ must not serve batched-path decode until the engine's fmt=8 absorb support+ lands (branch `f8/absorb-fmt8`); the failure is a loud crash, not silent+ corruption -- see the module docstring and `docs/FORMATS.md`.+ Synthetic-fixture-tested only, no real-shard runs yet.- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks- `gen_unicode.py`: tokenizer table generationdiff --git a/docs/FORMATS.md b/docs/FORMATS.mdindex 04ddccb3..388b99ab 100644--- a/docs/FORMATS.md+++ b/docs/FORMATS.md@@ -385,6 +385,40 @@ enforcing it, containers declare and nothing checks β so the stamp-loss holestays open in that interval. That is no worse than the status quo, which has nodeclaration at all.+### Full-family emission and the `kv_b_proj` decode constraint++`repack_fp8_passthrough.py` emits the **full tensor family**, making its+`--outdir` a standalone-loadable model directory rather than a partial overlay:++- **fp8-repacked** (byte-preserved weight + renamed `.qs` scale sidecar):+ routed experts (unstamped, see above) and the resident projections β+ `o_proj`, `q_a`/`q_b`/`kv_a`, `kv_b_proj`, shared-expert and dense-MLP+ weights (stamped).+- **Raw pass-through** (byte-identical copy, original BF16/F32 dtype, no+ `.qs`, no stamp): norms, router (`mlp.gate.weight`), `e_score_correction_bias`,+ `embed_tokens`, `lm_head`. This exercises nothing new on the read side β it+ is exactly the plain `st_read_f32` fallback path `qt_from_disk` already takes+ for any tensor without a `.qs` sidecar.+- **Non-tensor files**: `config.json` and `tokenizer.json` are copied from+ `--indir` (both are mandatory β the engine `exit(1)`s without either, so the+ tool refuses to mint if `--indir` lacks them); `generation_config.json` is+ copied best-effort, matching `load_cfg`'s own optional handling. No+ `model.safetensors.index.json` is copied or needed: `st_init_multi` globs+ `*.safetensors` and builds its own index, and a copied index would describe+ the *source* shard layout, not the minted one.++**Decode constraint on the minted `kv_b_proj`:** the container-side work is+deliberately ahead of the engine-side work. A minted `kv_b_proj` **loads**+today exactly like `o_proj` (same `qt_from_disk` path, same stamp), but the+MLA-absorption paths that consume it β `qt_addrow`/`qt_matvec_rows` on CPU and+the CUDA absorb kernels β have **no `fmt=8` case yet**; that support is being+built in parallel on `f8/absorb-fmt8`. Until it lands, a container minted with+`kv_b_proj` must not be used for decode that reaches the batched+(`kvs`-non-NULL) serving path. The failure mode is a **loud crash, not a+silent misread** (`ABSORB=0` cannot bypass absorb there), and sequencing β+engine support before such a container serves decode β is release-gate work,+not something the tool can enforce from the write side.+### Duplicate claims, locality, coverageThree rules complete the stamp's container-wide semantics (user-ratified