Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: highlight inference engine techniques
ZacharyZcR04bdc86ca283
diff --git a/README.it.md b/README.it.mdindex 3d16a62c..41d68584 100644--- a/README.it.md+++ b/README.it.md@@ -75,6 +75,23 @@ lo rende possibile. Non noleggiare intelligenza dietro un'API, ma possederla,analizzarla, misurarla e migliorarla. Il motore resta volutamente abbastanza piccoloperché la prossima ottimizzazione utile possa arrivare da chiunque sia disposto a misurarla.+## Punti di forza tecnici++- **Una gerarchia, non una soglia di memoria.** VRAM, RAM e NVMe sono livelli di+ piazzamento degli stessi pesi; poca memoria veloce cambia la velocità, non il modello.+- **Un JIT per i pesi.** Il calore di routing misurato alimenta una LRU per layer,+ un hot-store appreso e il prefetch del layer successivo senza caricare tutti gli expert.+- **L'I/O fa parte del motore.** Unione degli expert per batch, letture sovrapposte+ al calcolo, `O_DIRECT` e striping pesato su due SSD ottimizzano direttamente lo streaming.+- **Esecuzione eterogenea.** CPU, CUDA, Metal, memoria NUMA e residenza parziale o+ completa degli expert condividono un runtime e si combinano in base alla macchina.+- **Stato compresso senza cambiare modello.** Validazione token-exact, stato MLA KV+ 57× più piccolo, conversazioni persistenti e DSA fedele vincolano l'ottimizzazione+ alla correttezza.+- **La speculazione deve meritarsi il costo.** MTP nativo e draft vincolati da+ grammatica sono misurati end-to-end e si disattivano quando l'accettazione non+ ripaga la verifica.+## L'ideaUn modello Mixture-of-Experts da 744B attiva solo ~40B parametri per token — ediff --git a/README.md b/README.mdindex 44fe511a..d8a66229 100644--- a/README.md+++ b/README.md@@ -81,6 +81,23 @@ code that does it. Not renting intelligence behind an API — *holding* it:probing it, measuring it, improving it. The engine is deliberately small enoughthat the next useful optimization can come from anyone willing to measure it.+## Technical highlights++- **One hierarchy, not one memory threshold.** VRAM, RAM, and NVMe are placement+ tiers for the same weights; limited fast memory changes speed, not model semantics.+- **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learned+ pinned hot-store, and one-layer-ahead prefetch instead of loading every expert.+- **I/O is part of the engine.** Batched expert unions, overlapped reads and+ compute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path+ rather than pretending storage latency is free.+- **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full+ expert residency share one runtime and can be combined according to the machine.+- **Compressed state without a different model.** Token-exact forward validation,+ 57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep+ optimization tied to correctness.+- **Speculation that must earn its keep.** Native MTP and grammar-forced drafts+ are measured end to end and can be disabled when acceptance does not repay verification.+## The ideaA 744B Mixture-of-Experts model activates only ~40B parameters per token — anddiff --git a/README.zh-CN.md b/README.zh-CN.mdindex 0229f735..0d96e903 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -62,6 +62,21 @@ VRAM/RAM/磁盘层级条,以及角落的实时迷你大脑。</em></p>不是从 API 租用智能,而是持有、探测、测量和改进它。引擎刻意保持足够小,让任何愿意测量的人都可能贡献下一项有效优化。+## 技术亮点++- **统一层级,而非单一内存门槛。**VRAM、RAM 与 NVMe 是同一份权重的不同放置层级;+ 高速内存不足只影响速度,不改变模型语义。+- **权重的 JIT。**实测路由热度驱动逐层 LRU、学习型热门专家固定区和提前一层的预取,+ 无需加载所有专家。+- **I/O 本身就是引擎的一部分。**专家批次并集、读算重叠、`O_DIRECT` 与加权双 SSD+ 分流直接优化流式路径,而不是假装存储延迟不存在。+- **异构执行。**CPU、CUDA、Metal、NUMA 内存以及专家的部分或全部常驻共用一个运行时,+ 可按机器条件组合。+- **压缩状态,不篡改模型。**逐 token 精确的前向验证、缩小 57 倍的 MLA KV 状态、+ 持久化热会话与忠实 DSA,让优化始终受正确性约束。+- **必须证明收益的推测解码。**原生 MTP 与语法强制草稿均接受端到端测量;+ 接受率无法覆盖验证成本时可以关闭。+## 核心概念744B 的专家混合(Mixture-of-Experts)模型,每个 token 只会激活约 40B 参数——diff --git a/README.zh-TW.md b/README.zh-TW.mdindex 6ad5a972..ac679d2b 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -62,6 +62,21 @@ VRAM/RAM/硬碟層級長條,以及角落的即時迷你大腦。</em></p>不是從 API 租用智慧,而是持有、探測、測量和改進它。引擎刻意維持足夠小,讓任何願意測量的人都可能貢獻下一項有效最佳化。+## 技術亮點++- **統一階層,而非單一記憶體門檻。**VRAM、RAM 與 NVMe 是同一份權重的不同配置階層;+ 高速記憶體不足只影響速度,不改變模型語意。+- **權重的 JIT。**實測路由熱度驅動逐層 LRU、學習型熱門專家固定區和提前一層的預先載入,+ 無需載入所有專家。+- **I/O 本身就是引擎的一部分。**專家批次聯集、讀算重疊、`O_DIRECT` 與加權雙 SSD+ 分流直接最佳化串流路徑,而不是假裝儲存延遲不存在。+- **異質執行。**CPU、CUDA、Metal、NUMA 記憶體以及專家的部分或全部常駐共用一個執行環境,+ 可依機器條件組合。+- **壓縮狀態,不竄改模型。**逐 token 精確的前向驗證、縮小 57 倍的 MLA KV 狀態、+ 持久化熱會話與忠實 DSA,讓最佳化始終受正確性約束。+- **必須證明收益的推測解碼。**原生 MTP 與文法強制草稿均接受端到端測量;+ 接受率無法覆蓋驗證成本時可以關閉。+## 核心概念744B 的專家混合(Mixture-of-Experts)模型,每個 token 只會啟用約 40B 參數——