Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: describe Recover-LoRA (post-quantization distillation training)
linyubupaac4a25616b71
diff --git a/README.md b/README.mdindex 39dd2b5..24a1883 100644--- a/README.md+++ b/README.md@@ -87,9 +87,16 @@ are co-located with each checkpoint and load automatically, somechanism removes exactly the cold-read wait that dominates when theexpert working set exceeds what stays resident, so the gain scaleswith model size, routed width (K), and memory pressure;-- **Parallel LoRA**: adapters are applied as a side path at forward- time instead of being merged — the base stays a read-only mmap and- multiple adapter sets share one base.+- **Recover-LoRA** (parallel LoRA): quantization costs accuracy; we+ recover it after the fact. The recipe: quantize the base model to+ int4 and **freeze it** → insert LoRA adapters → distill training data+ from the FP teacher (the original unquantized model, on real and+ synthetic corpora, with on-policy distillation) → train the LoRA on+ the distillation loss. At inference the trained LoRA stays resident+ and is applied as a side path at forward time instead of being+ merged — the base stays a read-only mmap and multiple adapter sets+ share one base. This is what keeps the released checkpoints within+ a few points of their fp16 base models (see Quality) at 4-bit.## Quick startdiff --git a/README_zh.md b/README_zh.mdindex b1f8232..17fe5d4 100644--- a/README_zh.md+++ b/README_zh.md@@ -70,8 +70,13 @@原生路由 A/B 交替实测(同适配器、同负载):**最高 +59%**(本测试机)。**存储越慢,收益越大**:该机制消除的正是专家工作集超出常驻容量时占主导的冷读等待,因此收益随模型规模、路由宽度(K)与内存压力增大;-- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,- 基模保持只读 mmap、多套适配器共享同一份基模。+- **Recover-LoRA**(并行 LoRA):量化有精度损失,我们在量化之后+ 把它补回来。流程:基模 int4 量化并**冻结** → 插入 LoRA 适配器 →+ 由 FP teacher(原未量化模型,真实 + 合成语料,on-policy 蒸馏)+ 产出蒸馏数据 → 用蒸馏损失训练 LoRA。推理时训练好的 LoRA 常驻、+ 前向时旁路叠加而不合并——基模保持只读 mmap、多套适配器共享同一份+ 基模。这正是 4bit 下发布的 checkpoint 仍能贴近 fp16 基座几分之内+ 的原因(见[质量](#质量))。## 快速开始