Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: report short-context peak memory (35b 2.9 GiB, 10b 1.0 GiB)
linyubupad426e65991df
diff --git a/README.md b/README.mdindex e26ddf9..28b4b51 100644--- a/README.md+++ b/README.md@@ -30,10 +30,9 @@ are co-located with each checkpoint and load automatically, so(M1/M2/M3/M4). The CUDA backend is on the roadmap — no otherplatforms are supported yet.- **Python**: 3.10+ (3.12 recommended).-- **Memory**: ~3.3 GB peak active memory for `edge0-35b` and ~3.1 GB for- `edge0-10b` at 3.3k-token contexts (see [Benchmark](#benchmark); short- conversations on `edge0-10b` stay under ~1.4 GB). Add headroom for the- OS and tokenizer.+- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for+ `edge0-10b` (short contexts; see [Benchmark](#benchmark)). Add+ headroom for the OS, tokenizer, and long-context KV growth.- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB(`edge0-10b`); expert weights are mmapped and read on demand, they arenot loaded into RAM up front.@@ -179,18 +178,18 @@ multiple prompts):Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampledwarmup steps → 200 timed sampled decode tokens, 2 runs per tier):-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory** | Test machine ||---|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 3.3 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 3.1 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |*Cold = first request after process start (expert weights fault in fromSSD); warm = subsequent requests (page cache resident). Prefill numbersare throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*-*Peak active memory is the MLX allocator's peak (model weights + KV cache +-expert working set), not RSS: expert weights stream from SSD via mmap and the-OS page cache is not counted.*+**Peak active memory at short contexts (MLX allocator peak; expert weights+stream from SSD via mmap and are not resident). Long contexts add KV+cache: ~3.3 GiB on `edge0-10b` at 3.3k tokens.*Reproduce:diff --git a/README_zh.md b/README_zh.mdindex 193496c..8561692 100644--- a/README_zh.md+++ b/README_zh.md@@ -24,9 +24,9 @@- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS(M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。- **Python**:3.10+(推荐 3.12)。-- **内存**:3.3k token 上下文下 `edge0-35b` ≈3.3 GB、`edge0-10b`- ≈3.1 GB 峰值激活内存(见[性能实测](#性能实测);10b 短对话在- ≈1.4 GB 内);另需为系统与 tokenizer 预留余量。+- **内存**:短上下文下 `edge0-35b` ≈2.9 GB、`edge0-10b` ≈1.0 GB+ 峰值激活内存(见[性能实测](#性能实测));另为系统、tokenizer 与+ 长上下文 KV 增长预留余量。- **磁盘**:4bit checkpoint 约 23 GB(`edge0-35b`)/ 4.2 GB(`edge0-10b`);专家权重 mmap 按需读取,不一次性载入内存。@@ -153,16 +153,14 @@ engine.close() # 释放 mmap / 专家缓存`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →200 token 计时段,每档 2 轮):-| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 |+| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存** | 测试机器 ||---|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 3.3 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 3.1 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |-*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。-Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。*+*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。-*峰值 active 内存为 MLX allocator 的峰值(权重 + KV cache + 专家工作集),-不含 RSS:专家权重经 mmap 从 SSD 流式读取,OS 页缓存不计入。*+**短上下文下的峰值 active 内存(MLX allocator 峰值;专家权重经 mmap 流式读取、不常驻内存)。长上下文增加 KV cache:3.3k token 下 `edge0-10b` ≈3.3 GiB。**复现: