Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: drop redundant 'parallel LoRA' wording
linyubupa87ad83afd79b
diff --git a/README.md b/README.mdindex 24a1883..a67e7e7 100644--- a/README.md+++ b/README.md@@ -4,7 +4,7 @@# edge0-**An open-source streaming MoE inference framework — SSD expert offload + parallel LoRA + prerouter routing prediction.**+**An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction.**[](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)[](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)@@ -17,7 +17,7 @@ English | [中文](README_zh.md)**edge0** is an open-source streaming MoE inference framework. Itgeneralizes the production-proven recipe — **SSD expert offload +-parallel LoRA + prerouter routing prediction** — into an extensible+Recover-LoRA + prerouter routing prediction** — into an extensibleframework. The backend is isolated by design: the current MLX backendruns on Apple Silicon, and additional platforms (CUDA, …) plug into thesame core abstractions.@@ -87,7 +87,7 @@ are co-located with each checkpoint and load automatically, somechanism removes exactly the cold-read wait that dominates when theexpert working set exceeds what stays resident, so the gain scaleswith model size, routed width (K), and memory pressure;-- **Recover-LoRA** (parallel LoRA): quantization costs accuracy; we+- **Recover-LoRA**: quantization costs accuracy; werecover it after the fact. The recipe: quantize the base model toint4 and **freeze it** → insert LoRA adapters → distill training datafrom the FP teacher (the original unquantized model, on real anddiff --git a/README_zh.md b/README_zh.mdindex 17fe5d4..9725819 100644--- a/README_zh.md+++ b/README_zh.md@@ -4,7 +4,7 @@# edge0-**开源流式 MoE 推理框架 —— SSD 专家 offload + 并行 LoRA + prerouter 路由预判**+**开源流式 MoE 推理框架 —— SSD 专家 offload + Recover-LoRA + prerouter 路由预判**[](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)[](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)@@ -16,7 +16,7 @@</div>**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload +-并行 LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,+Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)即将接入。框架随附两个模型档位。每个档位是一个端到端发布:发布的 checkpoint、@@ -70,7 +70,7 @@原生路由 A/B 交替实测(同适配器、同负载):**最高 +59%**(本测试机)。**存储越慢,收益越大**:该机制消除的正是专家工作集超出常驻容量时占主导的冷读等待,因此收益随模型规模、路由宽度(K)与内存压力增大;-- **Recover-LoRA**(并行 LoRA):量化有精度损失,我们在量化之后+- **Recover-LoRA**:量化有精度损失,我们在量化之后把它补回来。流程:基模 int4 量化并**冻结** → 插入 LoRA 适配器 →由 FP teacher(原未量化模型,真实 + 合成语料,on-policy 蒸馏)产出蒸馏数据 → 用蒸馏损失训练 LoRA。推理时训练好的 LoRA 常驻、