Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: highlight inference engine techniques

ZacharyZcR04bdc86ca283
diff --git a/README.it.md b/README.it.md
index 3d16a62c..41d68584 100644
--- a/README.it.md
+++ b/README.it.md
@@ -75,6 +75,23 @@ lo rende possibile. Non noleggiare intelligenza dietro un'API, ma possederla,
analizzarla, misurarla e migliorarla. Il motore resta volutamente abbastanza piccolo
perché la prossima ottimizzazione utile possa arrivare da chiunque sia disposto a misurarla.
+## Punti di forza tecnici
+
+- **Una gerarchia, non una soglia di memoria.** VRAM, RAM e NVMe sono livelli di
+ piazzamento degli stessi pesi; poca memoria veloce cambia la velocità, non il modello.
+- **Un JIT per i pesi.** Il calore di routing misurato alimenta una LRU per layer,
+ un hot-store appreso e il prefetch del layer successivo senza caricare tutti gli expert.
+- **L'I/O fa parte del motore.** Unione degli expert per batch, letture sovrapposte
+ al calcolo, `O_DIRECT` e striping pesato su due SSD ottimizzano direttamente lo streaming.
+- **Esecuzione eterogenea.** CPU, CUDA, Metal, memoria NUMA e residenza parziale o
+ completa degli expert condividono un runtime e si combinano in base alla macchina.
+- **Stato compresso senza cambiare modello.** Validazione token-exact, stato MLA KV
+ 57× più piccolo, conversazioni persistenti e DSA fedele vincolano l'ottimizzazione
+ alla correttezza.
+- **La speculazione deve meritarsi il costo.** MTP nativo e draft vincolati da
+ grammatica sono misurati end-to-end e si disattivano quando l'accettazione non
+ ripaga la verifica.
+
## L'idea
Un modello Mixture-of-Experts da 744B attiva solo ~40B parametri per token — e
diff --git a/README.md b/README.md
index 44fe511a..d8a66229 100644
--- a/README.md
+++ b/README.md
@@ -81,6 +81,23 @@ code that does it. Not renting intelligence behind an API — *holding* it:
probing it, measuring it, improving it. The engine is deliberately small enough
that the next useful optimization can come from anyone willing to measure it.
+## Technical highlights
+
+- **One hierarchy, not one memory threshold.** VRAM, RAM, and NVMe are placement
+ tiers for the same weights; limited fast memory changes speed, not model semantics.
+- **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learned
+ pinned hot-store, and one-layer-ahead prefetch instead of loading every expert.
+- **I/O is part of the engine.** Batched expert unions, overlapped reads and
+ compute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path
+ rather than pretending storage latency is free.
+- **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full
+ expert residency share one runtime and can be combined according to the machine.
+- **Compressed state without a different model.** Token-exact forward validation,
+ 57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep
+ optimization tied to correctness.
+- **Speculation that must earn its keep.** Native MTP and grammar-forced drafts
+ are measured end to end and can be disabled when acceptance does not repay verification.
+
## The idea
A 744B Mixture-of-Experts model activates only ~40B parameters per token — and
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 0229f735..0d96e903 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -62,6 +62,21 @@ VRAM/RAM/磁盘层级条,以及角落的实时迷你大脑。</em></p>
不是从 API 租用智能,而是持有、探测、测量和改进它。引擎刻意保持足够小,让任何愿意测量
的人都可能贡献下一项有效优化。
+## 技术亮点
+
+- **统一层级,而非单一内存门槛。**VRAM、RAM 与 NVMe 是同一份权重的不同放置层级;
+ 高速内存不足只影响速度,不改变模型语义。
+- **权重的 JIT。**实测路由热度驱动逐层 LRU、学习型热门专家固定区和提前一层的预取,
+ 无需加载所有专家。
+- **I/O 本身就是引擎的一部分。**专家批次并集、读算重叠、`O_DIRECT` 与加权双 SSD
+ 分流直接优化流式路径,而不是假装存储延迟不存在。
+- **异构执行。**CPU、CUDA、Metal、NUMA 内存以及专家的部分或全部常驻共用一个运行时,
+ 可按机器条件组合。
+- **压缩状态,不篡改模型。**逐 token 精确的前向验证、缩小 57 倍的 MLA KV 状态、
+ 持久化热会话与忠实 DSA,让优化始终受正确性约束。
+- **必须证明收益的推测解码。**原生 MTP 与语法强制草稿均接受端到端测量;
+ 接受率无法覆盖验证成本时可以关闭。
+
## 核心概念
744B 的专家混合(Mixture-of-Experts)模型,每个 token 只会激活约 40B 参数——
diff --git a/README.zh-TW.md b/README.zh-TW.md
index 6ad5a972..ac679d2b 100644
--- a/README.zh-TW.md
+++ b/README.zh-TW.md
@@ -62,6 +62,21 @@ VRAM/RAM/硬碟層級長條,以及角落的即時迷你大腦。</em></p>
不是從 API 租用智慧,而是持有、探測、測量和改進它。引擎刻意維持足夠小,讓任何願意測量
的人都可能貢獻下一項有效最佳化。
+## 技術亮點
+
+- **統一階層,而非單一記憶體門檻。**VRAM、RAM 與 NVMe 是同一份權重的不同配置階層;
+ 高速記憶體不足只影響速度,不改變模型語意。
+- **權重的 JIT。**實測路由熱度驅動逐層 LRU、學習型熱門專家固定區和提前一層的預先載入,
+ 無需載入所有專家。
+- **I/O 本身就是引擎的一部分。**專家批次聯集、讀算重疊、`O_DIRECT` 與加權雙 SSD
+ 分流直接最佳化串流路徑,而不是假裝儲存延遲不存在。
+- **異質執行。**CPU、CUDA、Metal、NUMA 記憶體以及專家的部分或全部常駐共用一個執行環境,
+ 可依機器條件組合。
+- **壓縮狀態,不竄改模型。**逐 token 精確的前向驗證、縮小 57 倍的 MLA KV 狀態、
+ 持久化熱會話與忠實 DSA,讓最佳化始終受正確性約束。
+- **必須證明收益的推測解碼。**原生 MTP 與文法強制草稿均接受端到端測量;
+ 接受率無法覆蓋驗證成本時可以關閉。
+
## 核心概念
744B 的專家混合(Mixture-of-Experts)模型,每個 token 只會啟用約 40B 參數——