Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

README: describe Recover-LoRA (post-quantization distillation training)

linyubupaac4a25616b71
diff --git a/README.md b/README.md
index 39dd2b5..24a1883 100644
--- a/README.md
+++ b/README.md
@@ -87,9 +87,16 @@ are co-located with each checkpoint and load automatically, so
mechanism removes exactly the cold-read wait that dominates when the
expert working set exceeds what stays resident, so the gain scales
with model size, routed width (K), and memory pressure;
-- **Parallel LoRA**: adapters are applied as a side path at forward
- time instead of being merged — the base stays a read-only mmap and
- multiple adapter sets share one base.
+- **Recover-LoRA** (parallel LoRA): quantization costs accuracy; we
+ recover it after the fact. The recipe: quantize the base model to
+ int4 and **freeze it** → insert LoRA adapters → distill training data
+ from the FP teacher (the original unquantized model, on real and
+ synthetic corpora, with on-policy distillation) → train the LoRA on
+ the distillation loss. At inference the trained LoRA stays resident
+ and is applied as a side path at forward time instead of being
+ merged — the base stays a read-only mmap and multiple adapter sets
+ share one base. This is what keeps the released checkpoints within
+ a few points of their fp16 base models (see Quality) at 4-bit.
## Quick start
diff --git a/README_zh.md b/README_zh.md
index b1f8232..17fe5d4 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -70,8 +70,13 @@
原生路由 A/B 交替实测(同适配器、同负载):**最高 +59%**(本测试机)。
**存储越慢,收益越大**:该机制消除的正是专家工作集超出常驻容量时占
主导的冷读等待,因此收益随模型规模、路由宽度(K)与内存压力增大;
-- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,
- 基模保持只读 mmap、多套适配器共享同一份基模。
+- **Recover-LoRA**(并行 LoRA):量化有精度损失,我们在量化之后
+ 把它补回来。流程:基模 int4 量化并**冻结** → 插入 LoRA 适配器 →
+ 由 FP teacher(原未量化模型,真实 + 合成语料,on-policy 蒸馏)
+ 产出蒸馏数据 → 用蒸馏损失训练 LoRA。推理时训练好的 LoRA 常驻、
+ 前向时旁路叠加而不合并——基模保持只读 mmap、多套适配器共享同一份
+ 基模。这正是 4bit 下发布的 checkpoint 仍能贴近 fp16 基座几分之内
+ 的原因(见[质量](#质量))。
## 快速开始