Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: drop the peak-memory footnote boilerplate
linyubupadd633cd9ea8a
diff --git a/README.md b/README.mdindex d016535..6787426 100644--- a/README.md+++ b/README.md@@ -223,7 +223,7 @@ edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampledwarmup steps → 200 timed sampled decode tokens, 2 runs per tier):-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory** | Test machine |+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine ||---|---|---|---|---|| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB || `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |@@ -232,9 +232,6 @@ warmup steps → 200 timed sampled decode tokens, 2 runs per tier):SSD); warm = subsequent requests (page cache resident). Prefill numbersare throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*-**Peak active memory at short contexts (MLX allocator peak; expert weights-stream from SSD via mmap and are not resident).*-Reproduce:```bashdiff --git a/README_zh.md b/README_zh.mdindex 3fd8f8a..7765792 100644--- a/README_zh.md+++ b/README_zh.md@@ -205,15 +205,13 @@ engine.close() # 释放 mmap / 专家缓存`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →200 token 计时段,每档 2 轮):-| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存** | 测试机器 |+| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 ||---|---|---|---|---|| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB || `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。-**短上下文下的峰值 active 内存(MLX allocator 峰值;专家权重经 mmap 流式读取、不常驻内存)。**-复现:```bashdiff --git a/docs/models/edge0-8b.md b/docs/models/edge0-8b.mdindex 5b83a06..f44792c 100644--- a/docs/models/edge0-8b.md+++ b/docs/models/edge0-8b.md@@ -24,7 +24,7 @@ The performance profile is based on benchmarks of the current release adapter ve| Streaming prefetch history | on (`prefetch_history=True`) || Serving port | 8083 || Measured throughput | 23.9–25.3 tok/s (M4 Pro) |-| Measured peak activation memory | ≈ 1.0 GiB (short context) |+| Measured peak activation memory | ≈ 1.0 GiB |> Note: all values are taken from `Ling8BConfig._defaults()` and `LayerOptions.prod_k8()`. The head count comes from the explicit `owners=range(7, 23)` — 16 heads in total (the current release head distribution: prediction is consumed from L7 onward, and L1–6 use the original router); `feature_topk="executed"`.