Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: avoid mmap jargon in core mechanisms
linyubupabac00ac568eb
diff --git a/README.md b/README.mdindex 6a219bf..c64c377 100644--- a/README.md+++ b/README.md@@ -69,8 +69,9 @@ are co-located with each checkpoint and load automatically, so## Core mechanisms-- **SSD expert offload**: expert weights are mmapped on demand; peak- memory is bounded by the active set, not the parameter count.+- **SSD expert offload**: expert weights are streamed from storage on+ demand; peak memory is bounded by the active set, not the parameter+ count.- **Prerouter**: a trained head predicts expert routing one stepahead, so expert loads overlap the forward pass instead of stallingit — **up to +59%** decode throughput; the gain grows with storagediff --git a/README_zh.md b/README_zh.mdindex e537387..296bd83 100644--- a/README_zh.md+++ b/README_zh.md@@ -60,8 +60,8 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后## 核心机制-- **SSD 专家 offload**:专家权重按需 mmap,峰值内存由激活集而非参数量- 决定;+- **SSD 专家 offload**:专家权重按需从存储流式加载,峰值内存由+ 激活集而非参数量决定;- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由宽度 *K* 增大;