Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
fix(convert): declare numpy in the OLMoE guard, and correct the OLMoE size in the README
Samiullah Shahfa8b211174b9
diff --git a/README.md b/README.mdindex 03b7e170..92068107 100644--- a/README.md+++ b/README.md@@ -370,7 +370,7 @@ the model's `config.json`):>> | Model | Disk for the weights | RAM | GPU |> |---|---|---|---|-> | **OLMoE** | ~4 GB | 8 GB | not needed |+> | **OLMoE** | ~7 GB (int8 container) | 8 GB | not needed |> | **GLM-5.2** | ~372 GB | 16 GB min, 24 GB comfortable | not needed |> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |@@ -386,7 +386,7 @@ the model's `config.json`):| **Inkling** (Thinking Machines) | 975B / 41B | [`nbeerbower/Inkling-colibri-int4`](https://huggingface.co/nbeerbower/Inkling-colibri-int4) (469 GB) | `make -C c inkling` | [inkling.md](docs/inkling.md) || **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) β original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) || **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint β routed experts stay **native fp4**, dense stays fp8-e4m3 | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |-| **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` | `make -C c olmoe` | β |+| **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` β **int8** container, ~7 GB | `make -C c olmoe` | β |Kimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight fromthe original Hugging Face shards, and the bf16 dense set is quantized at load time.diff --git a/c/tools/convert_olmoe_merged.py b/c/tools/convert_olmoe_merged.pyindex fe034ec3..be562c45 100644--- a/c/tools/convert_olmoe_merged.py+++ b/c/tools/convert_olmoe_merged.py@@ -25,11 +25,13 @@ import sysfrom pathlib import Pathtry:+ import numpy # noqa: F401 -- safetensors.torch.save_file imports it internallyimport torchfrom safetensors import safe_openfrom safetensors.torch import save_fileexcept ImportError as exc:- sys.exit(f"Missing dependencies: {exc}. Install: pip install torch safetensors huggingface_hub")+ sys.exit(f"Missing dependencies: {exc}. "+ f"Install: pip install numpy torch safetensors huggingface_hub")EXPERT_KEY_RE = re.compile(r"model\.layers\.(\d+)\.mlp\.experts\.(\d+)\.(gate_proj|up_proj|down_proj)\.weight"