Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: tighten core mechanisms
linyubupaee7cb74aa075
diff --git a/README.md b/README.mdindex 0c93717..6a219bf 100644--- a/README.md+++ b/README.md@@ -69,31 +69,16 @@ are co-located with each checkpoint and load automatically, so## Core mechanisms-edge0 reduces serving a large MoE on memory-constrained hardware to-three composable mechanisms:--- **SSD expert offload.** Expert weights live on disk and are mmapped- on demand; an LRU keeps the active set resident and long-tail- experts are prefetched per layer. Peak memory is bounded by the- active set, not the parameter count.-- **Prerouter.** In standard MoE decode, expert selection at layer- *N* depends on the layer *N−1* output, so each expert load begins- only after the routing decision and stalls the step. The prerouter- is a lightweight trained head that predicts layer-*N* routing from- the token-*t−1* hidden state — one layer and one token ahead — so- loads are issued at the step boundary and I/O latency overlaps the- forward pass. Against the same model with native routing (same- adapters, same load), this yields **up to +59%** decode throughput;- the gain grows with storage latency and with the fraction of the- expert working set that exceeds residency, i.e. with model size,- routed width *K*, and memory pressure.-- **Recover-LoRA.** Quantization loss is repaired post hoc: freeze- the int4 base, attach LoRA adapters, and train them with a- distillation loss against the FP teacher on real and synthetic- corpora (on-policy). Adapters stay unmerged at inference — a- side path over a read-only, mmap-shared base — so one base serves- many adapter sets. On the released checkpoints this holds 4-bit- quality within a few points of fp16 (see [Quality](#quality)).+- **SSD expert offload**: expert weights are mmapped on demand; peak+ memory is bounded by the active set, not the parameter count.+- **Prerouter**: a trained head predicts expert routing one step+ ahead, so expert loads overlap the forward pass instead of stalling+ it — **up to +59%** decode throughput; the gain grows with storage+ latency, model size, and routed width *K*.+- **Recover-LoRA**: the int4 base is frozen and LoRA adapters are+ trained by distillation from the FP teacher, recovering most of the+ quantization loss at 4-bit (see [Quality](#quality)). Adapters stay+ unmerged: one read-only base serves multiple adapter sets.## Quick startdiff --git a/README_zh.md b/README_zh.mdindex 7482c24..e537387 100644--- a/README_zh.md+++ b/README_zh.md@@ -60,22 +60,14 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后## 核心机制-edge0 把「在内存受限硬件上服务大 MoE」归约为三个可组合的机制:--- **SSD 专家 offload**。专家权重驻留磁盘、按需 mmap;LRU 维持激活集- 常驻,长尾专家按层预取。峰值内存由激活集而非参数量决定。-- **prerouter**。标准 MoE 解码中,第 *N* 层的专家选择依赖第 *N−1*- 层输出,每次专家装载都要等到路由决策之后、阻塞当前 step。prerouter- 是一个轻量训练头:用 token *t−1* 的隐层状态提前一层、提前一个 token- 预测第 *N* 层路由,使装载在 step 边界发出、I/O 延迟与前向计算重叠。- 对照同模型原生路由(同适配器、同负载),解码吞吐**最高 +59%**;- 收益随存储延迟与超出常驻容量的工作集占比增大——即随模型规模、- 路由宽度 *K* 与内存压力增大。-- **Recover-LoRA**。量化损失事后修复:冻结 int4 基模,挂载 LoRA 适配器,- 以 FP teacher 为目标、在真实与合成语料上(on-policy)用蒸馏损失训练。- 推理时适配器不合并——在只读、可多份 mmap 共享的基模上旁路叠加,- 一份基模服务多套适配器。在发布的 checkpoint 上,这使 4bit 质量与- fp16 相差仅几分(见[质量](#质量))。+- **SSD 专家 offload**:专家权重按需 mmap,峰值内存由激活集而非参数量+ 决定;+- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠+ 而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由+ 宽度 *K* 增大;+- **Recover-LoRA**:冻结 int4 基模,用 FP teacher 蒸馏训练 LoRA,+ 在 4bit 下恢复绝大部分量化损失(见[质量](#质量))。适配器不合并,+ 一份只读基模服务多套适配器。## 快速开始