Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs/tools: document PILOT_EVICT_GUARD and add CLI overrides to test_olmoe_real

Egon Ruiterfbdb1099872f
diff --git a/c/olmoe.c b/c/olmoe.c
index 74df155b..a1cc0495 100644
--- a/c/olmoe.c
+++ b/c/olmoe.c
@@ -13,6 +13,7 @@
* WIDE=N : prefetch top-K*N candidates (default 1, try 2 or 3)
* SMOOTH=F : EMA coefficient for routing momentum (default 0.3, range 0.0-0.95)
* CONF_LIMIT=F : cumulative gate probability threshold for prefetch cutoff (default 0.92)
+ * PILOT_EVICT_GUARD=0/1 : 1=enable LFRU prefetch eviction guard (default), 0=disable
* (expert queue is sorted by eid for SSD read locality)
*/
#define _GNU_SOURCE
diff --git a/c/tools/test_olmoe_real.py b/c/tools/test_olmoe_real.py
index 1abf64d0..384b2f9d 100644
--- a/c/tools/test_olmoe_real.py
+++ b/c/tools/test_olmoe_real.py
@@ -30,8 +30,8 @@ BOOTSTRAP_REF = HERE / "ref_olmoe_bootstrap.json"
PROMPT_IDS = [510, 5347, 273, 6181, 310] # "The capital of France is"
MAX_NEW = 12
-CACHE_SIZE = 32 # experts cached per layer
-QUANT_BITS = 8 # engine supports 2-8; 8 = int8 (lossless vs our quant)
+CACHE_SIZE = int(sys.argv[1]) if len(sys.argv) > 1 else 32 # experts cached per layer
+QUANT_BITS = int(sys.argv[2]) if len(sys.argv) > 2 else 8 # engine supports 2-8; 8 = int8 (lossless vs our quant)
# ── Step 1: Write bootstrap ref with dummy full_ids = prompt_ids ──────────
# olmoe.exe needs full_ids to know how many tokens to generate (nfull - np).