Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

fix(convert): declare numpy in the OLMoE guard, and correct the OLMoE size in the README

Samiullah Shahfa8b211174b9
diff --git a/README.md b/README.md
index 03b7e170..92068107 100644
--- a/README.md
+++ b/README.md
@@ -370,7 +370,7 @@ the model's `config.json`):
>
> | Model | Disk for the weights | RAM | GPU |
> |---|---|---|---|
-> | **OLMoE** | ~4 GB | 8 GB | not needed |
+> | **OLMoE** | ~7 GB (int8 container) | 8 GB | not needed |
> | **GLM-5.2** | ~372 GB | 16 GB min, 24 GB comfortable | not needed |
> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |
> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |
@@ -386,7 +386,7 @@ the model's `config.json`):
| **Inkling** (Thinking Machines) | 975B / 41B | [`nbeerbower/Inkling-colibri-int4`](https://huggingface.co/nbeerbower/Inkling-colibri-int4) (469 GB) | `make -C c inkling` | [inkling.md](docs/inkling.md) |
| **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) β€” original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) |
| **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint β€” routed experts stay **native fp4**, dense stays fp8-e4m3 | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |
-| **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` | `make -C c olmoe` | β€” |
+| **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` β€” **int8** container, ~7 GB | `make -C c olmoe` | β€” |
Kimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight from
the original Hugging Face shards, and the bf16 dense set is quantized at load time.
diff --git a/c/tools/convert_olmoe_merged.py b/c/tools/convert_olmoe_merged.py
index fe034ec3..be562c45 100644
--- a/c/tools/convert_olmoe_merged.py
+++ b/c/tools/convert_olmoe_merged.py
@@ -25,11 +25,13 @@ import sys
from pathlib import Path
try:
+ import numpy # noqa: F401 -- safetensors.torch.save_file imports it internally
import torch
from safetensors import safe_open
from safetensors.torch import save_file
except ImportError as exc:
- sys.exit(f"Missing dependencies: {exc}. Install: pip install torch safetensors huggingface_hub")
+ sys.exit(f"Missing dependencies: {exc}. "
+ f"Install: pip install numpy torch safetensors huggingface_hub")
EXPERT_KEY_RE = re.compile(
r"model\.layers\.(\d+)\.mlp\.experts\.(\d+)\.(gate_proj|up_proj|down_proj)\.weight"