Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

README: tighten core mechanisms

linyubupaee7cb74aa075
diff --git a/README.md b/README.md
index 0c93717..6a219bf 100644
--- a/README.md
+++ b/README.md
@@ -69,31 +69,16 @@ are co-located with each checkpoint and load automatically, so
## Core mechanisms
-edge0 reduces serving a large MoE on memory-constrained hardware to
-three composable mechanisms:
-
-- **SSD expert offload.** Expert weights live on disk and are mmapped
- on demand; an LRU keeps the active set resident and long-tail
- experts are prefetched per layer. Peak memory is bounded by the
- active set, not the parameter count.
-- **Prerouter.** In standard MoE decode, expert selection at layer
- *N* depends on the layer *N−1* output, so each expert load begins
- only after the routing decision and stalls the step. The prerouter
- is a lightweight trained head that predicts layer-*N* routing from
- the token-*t−1* hidden state — one layer and one token ahead — so
- loads are issued at the step boundary and I/O latency overlaps the
- forward pass. Against the same model with native routing (same
- adapters, same load), this yields **up to +59%** decode throughput;
- the gain grows with storage latency and with the fraction of the
- expert working set that exceeds residency, i.e. with model size,
- routed width *K*, and memory pressure.
-- **Recover-LoRA.** Quantization loss is repaired post hoc: freeze
- the int4 base, attach LoRA adapters, and train them with a
- distillation loss against the FP teacher on real and synthetic
- corpora (on-policy). Adapters stay unmerged at inference — a
- side path over a read-only, mmap-shared base — so one base serves
- many adapter sets. On the released checkpoints this holds 4-bit
- quality within a few points of fp16 (see [Quality](#quality)).
+- **SSD expert offload**: expert weights are mmapped on demand; peak
+ memory is bounded by the active set, not the parameter count.
+- **Prerouter**: a trained head predicts expert routing one step
+ ahead, so expert loads overlap the forward pass instead of stalling
+ it — **up to +59%** decode throughput; the gain grows with storage
+ latency, model size, and routed width *K*.
+- **Recover-LoRA**: the int4 base is frozen and LoRA adapters are
+ trained by distillation from the FP teacher, recovering most of the
+ quantization loss at 4-bit (see [Quality](#quality)). Adapters stay
+ unmerged: one read-only base serves multiple adapter sets.
## Quick start
diff --git a/README_zh.md b/README_zh.md
index 7482c24..e537387 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -60,22 +60,14 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后
## 核心机制
-edge0 把「在内存受限硬件上服务大 MoE」归约为三个可组合的机制:
-
-- **SSD 专家 offload**。专家权重驻留磁盘、按需 mmap;LRU 维持激活集
- 常驻,长尾专家按层预取。峰值内存由激活集而非参数量决定。
-- **prerouter**。标准 MoE 解码中,第 *N* 层的专家选择依赖第 *N−1*
- 层输出,每次专家装载都要等到路由决策之后、阻塞当前 step。prerouter
- 是一个轻量训练头:用 token *t−1* 的隐层状态提前一层、提前一个 token
- 预测第 *N* 层路由,使装载在 step 边界发出、I/O 延迟与前向计算重叠。
- 对照同模型原生路由(同适配器、同负载),解码吞吐**最高 +59%**;
- 收益随存储延迟与超出常驻容量的工作集占比增大——即随模型规模、
- 路由宽度 *K* 与内存压力增大。
-- **Recover-LoRA**。量化损失事后修复:冻结 int4 基模,挂载 LoRA 适配器,
- 以 FP teacher 为目标、在真实与合成语料上(on-policy)用蒸馏损失训练。
- 推理时适配器不合并——在只读、可多份 mmap 共享的基模上旁路叠加,
- 一份基模服务多套适配器。在发布的 checkpoint 上,这使 4bit 质量与
- fp16 相差仅几分(见[质量](#质量))。
+- **SSD 专家 offload**:专家权重按需 mmap,峰值内存由激活集而非参数量
+ 决定;
+- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠
+ 而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由
+ 宽度 *K* 增大;
+- **Recover-LoRA**:冻结 int4 基模,用 FP teacher 蒸馏训练 LoRA,
+ 在 4bit 下恢复绝大部分量化损失(见[质量](#质量))。适配器不合并,
+ 一份只读基模服务多套适配器。
## 快速开始