Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs+banner: DeepSeek V4 Flash REAP-150B is supported, and is not 284B
JustVugg1ac934bc3da3
diff --git a/README.md b/README.mdindex 126c5c6e..0f31e2e9 100644--- a/README.md+++ b/README.md@@ -409,7 +409,7 @@ the model's `config.json`):> | **GLM-5.3-Flash** | ~195 GB converted | 25 GB (12 GB weights at int4 + expert cache) | not needed |> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |-> | **DeepSeek V4 Flash** | ~167 GB | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |+> | **DeepSeek V4 Flash** | ~167 GB (REAP 150B: ~85 GB) | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |> | **Qwen3.8-Flash-Next** | ~185.5 GB (official FP8 checkpoint) | 16 GB min, 24 GB comfortable at the default context | not supported; CPU only |> | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU |>@@ -423,7 +423,7 @@ the model's `config.json`):| **Inkling** (Thinking Machines) | 975B / 41B | [`nbeerbower/Inkling-colibri-int4`](https://huggingface.co/nbeerbower/Inkling-colibri-int4) (469 GB) | `make -C c inkling` | [inkling.md](docs/inkling.md) || **GLM-5.3-Flash** (Z.ai) | 321B / 40B | [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) β converted to **int4-gs64** routed experts, dense stays BF16 and the precision is a load-time choice; vision included | `make -C c glm53` | [glm53-flash.md](docs/glm53-flash.md) || **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) β original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) |-| **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint β routed experts stay **native fp4**, dense stays fp8-e4m3 | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |+| **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint β routed experts stay **native fp4**, dense stays fp8-e4m3; the **REAP-pruned 150B** ([`puwaer/DeepSeek-V4-Flash-0731-reap-150b`](https://huggingface.co/puwaer/DeepSeek-V4-Flash-0731-reap-150b), 85 GB, 132 of 256 experts) loads with the same engine and no conversion | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) || **Qwen3.8-Flash-Next** (Alibaba) | 125B + 51B n-gram / 6B | [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) β original checkpoint; PLE stays pageable and experts stay **native block-FP8** | `make -C c qwen38` (CPU only) | [qwen38.md](docs/qwen38.md) || **Qwen3.6** (Alibaba) | 35B / 3B | [`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64) (~20 GB, **recommended**) β hybrid Gated Attention + Gated DeltaNet | `make -C c qwen36` (`CUDA=1` for the VRAM expert tier) | [qwen36.md](docs/qwen36.md) || **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` β **int8** container, ~7 GB | `make -C c olmoe` | β |diff --git a/c/coli b/c/coliindex ab9ee705..5fe9bc6d 100755--- a/c/coli+++ b/c/coli@@ -193,6 +193,13 @@ def model_banner_line(model):try:family = resolve_model(model).descriptorname, scale = family.display_name, family.display_scale+ # display_scale e' la taglia del checkpoint di riferimento. Un REAP ha+ # lo stesso model_type e meno esperti: "284B" su un 150B e' #1367+ # di nuovo. Quando la geometria non e' quella del riferimento si+ # lascia parlare la geometria, che e' misurata e non puo' mentire.+ experts = cfg.get("n_routed_experts")+ if family.reference_experts and experts and experts != family.reference_experts:+ scale = ""except (FamilyConfigError, UnknownFamilyError):# Unknown checkpoint: say so with its own words rather than guess.name = kind or "unknown model"diff --git a/c/family_registry.py b/c/family_registry.pyindex b86d36d8..eeddf2a8 100644--- a/c/family_registry.py+++ b/c/family_registry.py@@ -109,6 +109,14 @@ class FamilyDescriptor:converter: str = ""converter_accepts: tuple = ()converter_mtp_pass: bool = False+ # Gli esperti per layer del checkpoint con cui display_scale e' stato+ # scritto. display_scale e' un numero del checkpoint di riferimento, non+ # della famiglia: un checkpoint potato (REAP) ha la stessa architettura e+ # lo stesso model_type ma meno esperti, e annunciarlo con la taglia del+ # riferimento e' lo stesso errore di #1367 -- con la differenza che qui+ # la geometria lo rende visibile. 0 = nessun riferimento dichiarato, il+ # banner stampa display_scale come sempre.+ reference_experts: int = 0@dataclass(frozen=True, slots=True)@@ -1159,6 +1167,10 @@ FAMILIES = (model_types=("deepseek_v4",),display_name="DeepSeek V4 Flash",display_scale="284B",+ # deepseek-ai/DeepSeek-V4-Flash-0731 config.json: 43 layer, 256+ # esperti, top-k 6. Il REAP a 150B (#1310) ne ha 132 sugli stessi 43+ # layer: e' quello che fa scattare la geometria misurata nel banner.+ reference_experts=256,engine_artifact="deepseek_v4",engine_aliases=(),engine_group="deepseek_v4",diff --git a/c/tests/test_cli_output.py b/c/tests/test_cli_output.pyindex 84a1cda3..603e979c 100644--- a/c/tests/test_cli_output.py+++ b/c/tests/test_cli_output.py@@ -261,6 +261,24 @@ class BannerModelLineTest(unittest.TestCase):self.assertIn("unknown model", line)self.assertNotIn("GLM-5.2", line)+ def test_a_pruned_checkpoint_is_not_announced_with_the_reference_size(self):+ """#1310 made the REAP-pruned DeepSeek V4 Flash load: same model_type,+ same 43 layers, 132 routed experts instead of 256, 150B instead of+ 284B. The family's display_scale is the reference checkpoint's number,+ so printing it here is #1367 again -- except that this time the config+ can tell the two apart, so the banner must."""+ reap = self.line({"model_type": "deepseek_v4", "num_hidden_layers": 43,+ "n_routed_experts": 132})+ self.assertIn("DeepSeek V4 Flash", reap)+ self.assertNotIn("284B", reap, "a 150B checkpoint announced as 284B")+ self.assertIn("43L x 132E", reap)++ def test_the_reference_checkpoint_keeps_its_declared_size(self):+ official = self.line({"model_type": "deepseek_v4", "num_hidden_layers": 43,+ "n_routed_experts": 256})+ self.assertIn("284B", official)+ self.assertNotIn("43L x 256E", official)+def test_no_model_keeps_the_generic_tagline(self):"""What matters is that a tagline comes back and that it names nomodel. It used to assert the substring "GLM-5.2", which pinned thediff --git a/docs/deepseek-v4.md b/docs/deepseek-v4.mdindex f870988c..684216b9 100644--- a/docs/deepseek-v4.md+++ b/docs/deepseek-v4.md@@ -54,6 +54,21 @@ hf download deepseek-ai/DeepSeek-V4-Flash-0731 \--local-dir /path/to/DeepSeek-V4-Flash```+The REAP-pruned 150B variant loads with the same engine and needs no+conversion either. It keeps the official FP4 expert layout and the official+dense FP8, and drops 124 of the 256 routed experts per layer:++```bash+hf download puwaer/DeepSeek-V4-Flash-0731-reap-150b \+ --local-dir /path/to/DeepSeek-V4-Flash-reap-150b+```++85 GB on disk instead of 167. Its shards pack an expert's three weights and+three scales non-contiguously, sometimes across shards; the engine reads such+experts per matrix and takes the original contiguous fast path everywhere else+(#1310). The banner reports it by its measured geometry, `43L x 132E`, rather+than the 284B of the official checkpoint, because that number is not its.+A download can finish with a truncated shard even when the client reportssuccess. If `st.h` rejects a shard as out of bounds, compare every local shardsize with the Hugging Face repository before treating it as an engine failure.