Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: int4-rans256-g0 format specification (registry row proposed, not claimed)
monotophica77026230fa2
diff --git a/docs/int4-rans256-g0.md b/docs/int4-rans256-g0.mdnew file mode 100644index 00000000..328d0684--- /dev/null+++ b/docs/int4-rans256-g0.md@@ -0,0 +1,185 @@+# `int4-rans256-g0` β lossless entropy-coded container tier for per-row int4 experts++Status: format specification + offline tools, PR 1 of a 3-PR ladder. This PR+adds the codec (`c/rans.h`), the repack/verify tools+(`c/tools/repack_rans.py`, `c/tools/rans_verify.py`) and this document β+**no engine code paths change**. PR 2 (loader + CPU decode on expert load)+and PR 3 (Metal batched decode) build on it. A registry row is *proposed*+here, not claimed: the format's identity is the NAME string+`int4-rans256-g0`; the numeric `fmt` ordinal is left for the maintainer to+assign whenever a registry (FORMATS.md) lands.++## What it is, in one paragraph++`int4-rans256-g0` losslessly entropy-codes the packed int4 weight bytes of+routed-expert projections (`gate_proj`/`up_proj`/`down_proj`) with a+**single static rANS table shared per shard**, laid out as **256 independent+round-robin-interleaved streams per tensor** so wide decoders (SIMD lanes,+GPU threadgroups) get coalesced output. On a full-corpus census of a per-row+int4 GLM-5.2 container the achieved ratio is ~0.76 (β24% fewer bytes to+store and stream), with byte-exact reconstruction β this is a codec, not a+quantizer: decode reproduces the original packed bytes exactly, always.++Name breakdown: `int4` β source alphabet is 4-bit quantization codes+(nominally 16, effectively 15 symbols: the symmetric quantizer never emits+the -8 code point, and the table handles that automatically via a zero+frequency); `rans256` β rANS entropy coding, 256-way interleave; `g0` β+shared static table, generation 0.++## Container shape++Output is **ordinary safetensors shards**. Each entropy-coded weight tensor+keeps its **original, unmodified logical name** (so any name-indexed loader+finds it with zero special-casing), stored as dtype `U8`, shape+`[record_len]` β an opaque byte blob whose contents are the chunk record+below. The `.qs` per-row scale sidecars are **unchanged raw F32** (they are+~0.1% of the byte stream on per-row geometry; coding them is not worth a+second codec path in v1).++### Chunk record (one record = one weight tensor's bytes)++All integers little-endian. `N` = 256 streams.++```+offset 0: n_symbols u64 -- nibble count for this tensor+offset 8: packed_bytes u64 -- original packed-byte count (= ceil(n_symbols/2)),+ stored so a reader can size its output buffer+offset 16: stream_offsets[N+1] u32 x (N+1)+ -- offsets[i] = byte offset, relative to the start+ of `payload`, where stream i begins;+ offsets[N] = total payload length+then: zero-pad to the next 16-byte boundary (derived, never stored)+payload: N independent rANS byte streams, concatenated in stream order:+ stream i is payload[offsets[i] : offsets[i+1]]+then: zero-pad to the next 16-byte boundary (derived, never stored)+```++The two padding runs are computed identically by writer and reader+(`round16()`), never stored β a stored length could disagree with the+derived value. Validators refuse nonzero padding bytes. Alignment is+**record-relative**, not file-absolute: safetensors does not 16-align+tensor offsets, so a consumer that wants aligned u32/u64 access must read or+map the tensor's bytes into an aligned allocation (any `malloc`/GPU-buffer+allocation qualifies).++### Interleaving++Nibbles are unpacked low-nibble-first (`nib[2k] = byte[k] & 0xF`,+`nib[2k+1] = byte[k] >> 4`). Nibble at logical index `j` belongs to stream+`j % N` and is the `(j / N)`-th symbol of that stream. Every stream is an+ordinary, self-contained, single-state rANS stream over the same shared+table β no cross-stream coupling. Round-robin (not block) assignment is the+property that makes wide decode output-coalesced: lane `l` of a group based+at stream `b` emits logical position `r*N + b + l` at round `r`, so a+G-lane group writes G contiguous nibbles = G/2 whole packed bytes.++### Per-stream codec++ryg_rans-style construction (Fabian Giesen's public-domain reference+design): 32-bit state, byte renormalization, `L = 2^23`,+`scale_bits = 14` (`M = 16384`).++- encode walks the input backwards, writes bytes backwards, flushes the+ final 4-byte state; decode reads forward:+ `x = 4 bytes big-endian`, then per symbol:+ `slot = x & (M-1)`; `s = slot_to_symbol[slot]`;+ `x = freq[s]*(x >> scale_bits) + slot - start[s]`;+ `while (x < L) { if bytes remain: x = (x<<8)|next byte; else break; }`.+- The `else break` is load-bearing: a stream's final symbols legally decode+ with `x < L`; a decoder that reads past the stream end produces wrong+ tail symbols.+- Verifiable invariants of any genuine encoder output (validators use them+ to refuse corruption **without ground-truth bytes**): the initial 4-byte+ state lies in `[L, 256L)`; decode consumes the stream exactly; the final+ state is exactly `L`.++### Shared table, in shard metadata++The table ships **once per shard** in+`__metadata__["colibri.int4-rans256-g0.table"]`, a JSON string:++```json+{+ "table_id": "g0",+ "n_streams": 256,+ "scale_bits": 14,+ "M": 16384,+ "freq": [16 uint32, summing to M],+ "start": [16 uint32, the exclusive prefix sum of freq],+ "slot_to_symbol_b64": "<base64 of M x uint16, little-endian>",+ "table_crc32": "<hex crc32 of the raw slot_to_symbol bytes>"+}+```++Decode is always shard-local: every record in a shard decodes against that+shard's embedded copy, no cross-shard or external-file dependency. A+multi-shard writer must stamp the byte-identical table into every shard it+emits (the repack tool builds the table once, from the pooled histogram of+every selected tensor, and reuses the same JSON string). `table_crc32`+lets tooling confirm two shards used the identical table without diffing+the base64 blob. `table_id` is provenance metadata (a retuned future table+would ship as `g1`); readers never resolve it externally.++## THE STAMP IS MANDATORY (the load-bearing difference from fmt=7)++`__metadata__["colibri.fmt"]` maps each entropy-coded **weight** tensor+name (never `.qs`) to the string `int4-rans256-g0`, JSON-encoded with+sorted keys.++For the existing formats the engine can infer identity from byte arithmetic+(`weight bytes == formula(O, I)`), and a stamp is at most a cross-check.+**That inference is structurally impossible here**: entropy-coded size is+data-dependent β there is no `expected_bytes(O, I)` to compare against. The+stamp is therefore the **only** signal that a `U8` tensor is entropy-coded+at all. Consequences any consumer must respect:++- an *unstamped* `U8` tensor must never be presumed entropy-coded by any+ size heuristic;+- a future engine read path needs a stamp-gated dispatch **ahead of** the+ byte-arithmetic format inference: stamped `int4-rans256-g0` β rANS decode+ path; anything else β today's inference, completely unchanged. Getting+ that order wrong would misread a compressed blob as a fixed-ratio format+ and corrupt every weight with no error;+- validators refuse a shard whose stamps and table are not both present and+ coherent (`tools/rans_verify.py` is the reference validator, with a named+ refusal per corruption class).++## Scope (v1)++Per-row ("gs=0") int4 expert containers only. Per-group (g64-class)+containers are out of scope: their better-normalized codes measure a much+weaker ratio (~0.89 vs ~0.76) and their scale arrays are a two-orders-larger+share of the stream β a g64 round would need scale coding to be worth+doing, which is its own design. The repack tool *refuses* (by name) a+tensor whose weight-bytes/scale-rows ratio is the g64 signature rather than+silently coding it under this format's identity.++## Reference implementation map++- `c/rans.h` β dependency-free C99, header-only: scalar codec, record+ writer/reader with named refusals, batched decode arms (scalar, branch-free+ scalar, NEON, AVX-512 F+BW). Vector arms are byte-identical to scalar by+ contract and carry a compile-time ISA gate, a first-use round-trip+ selftest (a failing arm disables itself loudly) and env kill-switches+ (`RANS_NEON=0`, `RANS_AVX512=0`; `RANS_PATH=<arm>` forces one and refuses+ if unavailable β never a silent downgrade).+- `make rans` β builds `tools/librans_c.*`, the ctypes bridge the Python+ tools prefer; without it they fall back to a pure-Python codec that emits+ identical bytes, slowly.+- `c/tools/repack_rans.py` β writer. Deterministic: same input β+ byte-identical shards. Verifies every record byte-exact before writing.+- `c/tools/rans_verify.py` β validator (TRUST-VERIFY-REFUSE; the doc of+ refusal classes is its module docstring).+- `c/tests/test_rans.c`, `c/tests/test_rans_repack.py` β property suite,+ arm-identity sweep, framing/refusal battery, e2e round trip.++A third-party consumer needs only this document plus the scalar decode+description above to read records: parse the shard header, find the stamp+and table in `__metadata__`, locate a tensor's record by name, and decode+its 256 streams with the table (in any order or in parallel).++## Proposed registry row++| ordinal | name | weight bytes | scale layout | status |+|---|---|---|---|---|+| *(maintainer-assigned)* | `int4-rans256-g0` | data-dependent (chunk record above; **stamp mandatory**) | per-row `F32` `.qs`, raw (unchanged from int4-row) | proposed, offline tools only |