Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: rewrite core mechanisms in technical register
linyubupaf17d09cc53e4
diff --git a/README.md b/README.mdindex a67e7e7..0c93717 100644--- a/README.md+++ b/README.md@@ -69,34 +69,31 @@ are co-located with each checkpoint and load automatically, so## Core mechanisms-- **SSD expert offload**: MoE expert weights are mmapped from disk and- streamed on demand; the active set stays resident in an LRU and- long-tail experts are prefetched per layer — large models run in- modest memory;-- **Prerouter routing prediction**: MoE decode waits on expert loads —- routing depends on the previous layer's output, so by the time the- router picks the experts, their SSD load has not even started. A- lightweight trained head breaks this serialization: it predicts the- next token's expert routing from the previous token's hidden state- one step ahead (double shift: prev-layer + prev-token), so expert- loads are submitted at the step boundary and SSD read latency hides- completely behind the forward pass (`start_layer=7` on both tiers).- Measured A/B decode speedup vs the same model with native routing- (identical adapters and load, alternating rounds): **up to +59%** on- this test machine. The slower the storage, the bigger the win: the- mechanism removes exactly the cold-read wait that dominates when the- expert working set exceeds what stays resident, so the gain scales- with model size, routed width (K), and memory pressure;-- **Recover-LoRA**: quantization costs accuracy; we- recover it after the fact. The recipe: quantize the base model to- int4 and **freeze it** → insert LoRA adapters → distill training data- from the FP teacher (the original unquantized model, on real and- synthetic corpora, with on-policy distillation) → train the LoRA on- the distillation loss. At inference the trained LoRA stays resident- and is applied as a side path at forward time instead of being- merged — the base stays a read-only mmap and multiple adapter sets- share one base. This is what keeps the released checkpoints within- a few points of their fp16 base models (see Quality) at 4-bit.+edge0 reduces serving a large MoE on memory-constrained hardware to+three composable mechanisms:++- **SSD expert offload.** Expert weights live on disk and are mmapped+ on demand; an LRU keeps the active set resident and long-tail+ experts are prefetched per layer. Peak memory is bounded by the+ active set, not the parameter count.+- **Prerouter.** In standard MoE decode, expert selection at layer+ *N* depends on the layer *N−1* output, so each expert load begins+ only after the routing decision and stalls the step. The prerouter+ is a lightweight trained head that predicts layer-*N* routing from+ the token-*t−1* hidden state — one layer and one token ahead — so+ loads are issued at the step boundary and I/O latency overlaps the+ forward pass. Against the same model with native routing (same+ adapters, same load), this yields **up to +59%** decode throughput;+ the gain grows with storage latency and with the fraction of the+ expert working set that exceeds residency, i.e. with model size,+ routed width *K*, and memory pressure.+- **Recover-LoRA.** Quantization loss is repaired post hoc: freeze+ the int4 base, attach LoRA adapters, and train them with a+ distillation loss against the FP teacher on real and synthetic+ corpora (on-policy). Adapters stay unmerged at inference — a+ side path over a read-only, mmap-shared base — so one base serves+ many adapter sets. On the released checkpoints this holds 4-bit+ quality within a few points of fp16 (see [Quality](#quality)).## Quick startdiff --git a/README_zh.md b/README_zh.mdindex 9725819..7482c24 100644--- a/README_zh.md+++ b/README_zh.md@@ -60,23 +60,22 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后## 核心机制-- **SSD 专家 offload**:MoE 专家权重从磁盘 mmap 按需流式加载,- 激活集驻留 LRU,长尾专家按层预取 —— 大模型小显存跑得动;-- **prerouter 路由预判**:MoE 解码的固有瓶颈是「等专家装载」——路由依赖- 前一层输出,等路由选出专家时,SSD 装载甚至还没开始。edge0 用一个轻量- 训练头打破这条串行链:基于上一 token 的隐层状态提前一步预测下一 token- 的专家路由(双移位:层移位 + token 移位),专家装载在 step 边界即提前- 提交,SSD 读取延迟完全隐藏在前向计算之下(`start_layer=7`)。同模型- 原生路由 A/B 交替实测(同适配器、同负载):**最高 +59%**(本测试机)。- **存储越慢,收益越大**:该机制消除的正是专家工作集超出常驻容量时占- 主导的冷读等待,因此收益随模型规模、路由宽度(K)与内存压力增大;-- **Recover-LoRA**:量化有精度损失,我们在量化之后- 把它补回来。流程:基模 int4 量化并**冻结** → 插入 LoRA 适配器 →- 由 FP teacher(原未量化模型,真实 + 合成语料,on-policy 蒸馏)- 产出蒸馏数据 → 用蒸馏损失训练 LoRA。推理时训练好的 LoRA 常驻、- 前向时旁路叠加而不合并——基模保持只读 mmap、多套适配器共享同一份- 基模。这正是 4bit 下发布的 checkpoint 仍能贴近 fp16 基座几分之内- 的原因(见[质量](#质量))。+edge0 把「在内存受限硬件上服务大 MoE」归约为三个可组合的机制:++- **SSD 专家 offload**。专家权重驻留磁盘、按需 mmap;LRU 维持激活集+ 常驻,长尾专家按层预取。峰值内存由激活集而非参数量决定。+- **prerouter**。标准 MoE 解码中,第 *N* 层的专家选择依赖第 *N−1*+ 层输出,每次专家装载都要等到路由决策之后、阻塞当前 step。prerouter+ 是一个轻量训练头:用 token *t−1* 的隐层状态提前一层、提前一个 token+ 预测第 *N* 层路由,使装载在 step 边界发出、I/O 延迟与前向计算重叠。+ 对照同模型原生路由(同适配器、同负载),解码吞吐**最高 +59%**;+ 收益随存储延迟与超出常驻容量的工作集占比增大——即随模型规模、+ 路由宽度 *K* 与内存压力增大。+- **Recover-LoRA**。量化损失事后修复:冻结 int4 基模,挂载 LoRA 适配器,+ 以 FP teacher 为目标、在真实与合成语料上(on-policy)用蒸馏损失训练。+ 推理时适配器不合并——在只读、可多份 mmap 共享的基模上旁路叠加,+ 一份基模服务多套适配器。在发布的 checkpoint 上,这使 4bit 质量与+ fp16 相差仅几分(见[质量](#质量))。## 快速开始