Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
README: add prefill timing (cold/warm) to the benchmark table
linyubupa918d61c932e7
diff --git a/README.md b/README.mdindex 67a046b..2dc9a6d 100644--- a/README.md+++ b/README.md@@ -168,10 +168,14 @@ multiple prompts):Measured with `examples/bench.py` (prefill → 10 sampled warmup steps → 200timed sampled decode tokens, 2 runs per tier):-| Tier | Decode speed | Peak active memory | Test machine |-|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-10b` | 25.3 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |+| Tier | Decode speed | Prefill (cold / warm)* | Peak active memory | Test machine |+|---|---|---|---|---|+| `edge0-35b` | 14.9–17.7 tok/s | 4.4 s / 0.8 s (27 tok) | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 25.3 tok/s | 0.15 s / 0.25 s (37 tok) | 1.0 GiB | Mac mini M4 Pro, 24 GB |++*Cold = first request after process start (expert weights fault in from+SSD); warm = subsequent requests (page cache resident). Prompt length in+parentheses.**Peak active memory is the MLX allocator's peak (model weights + KV cache +expert working set), not RSS: expert weights stream from SSD via mmap and thediff --git a/README_zh.md b/README_zh.mdindex b4095d0..91e0d58 100644--- a/README_zh.md+++ b/README_zh.md@@ -144,10 +144,13 @@ engine.close() # 释放 mmap / 专家缓存`examples/bench.py` 实测(prefill → 10 步采样 warmup → 200 token 计时段,每档 2 轮):-| 档位 | 解码速度 | 峰值 active 内存 | 测试机器 |-|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-10b` | 25.3 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |+| 档位 | 解码速度 | Prefill(冷/热)* | 峰值 active 内存 | 测试机器 |+|---|---|---|---|---|+| `edge0-35b` | 14.9–17.7 tok/s | 4.4 s / 0.8 s(27 tok) | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 25.3 tok/s | 0.15 s / 0.25 s(37 tok) | 1.0 GiB | Mac mini M4 Pro, 24 GB |++*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。+括号内为 prompt 长度。**峰值 active 内存为 MLX allocator 的峰值(权重 + KV cache + 专家工作集),不含 RSS:专家权重经 mmap 从 SSD 流式读取,OS 页缓存不计入。*