Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
bench: long-prompt prefill measurement (BENCH_LONG=1)
linyubupa0117e3c8751e
diff --git a/README.md b/README.mdindex 2dc9a6d..b4810eb 100644--- a/README.md+++ b/README.md@@ -165,17 +165,17 @@ multiple prompts):## Benchmark-Measured with `examples/bench.py` (prefill → 10 sampled warmup steps → 200-timed sampled decode tokens, 2 runs per tier):+Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled+warmup steps → 200 timed sampled decode tokens, 2 runs per tier):-| Tier | Decode speed | Prefill (cold / warm)* | Peak active memory | Test machine |+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine ||---|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 4.4 s / 0.8 s (27 tok) | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-10b` | 25.3 tok/s | 0.15 s / 0.25 s (37 tok) | 1.0 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 3.3–4.5 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 3.1 GiB | Mac mini M4 Pro, 24 GB |*Cold = first request after process start (expert weights fault in from-SSD); warm = subsequent requests (page cache resident). Prompt length in-parentheses.*+SSD); warm = subsequent requests (page cache resident). Prefill numbers+are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).**Peak active memory is the MLX allocator's peak (model weights + KV cache +expert working set), not RSS: expert weights stream from SSD via mmap and thediff --git a/README_zh.md b/README_zh.mdindex 91e0d58..9d7da1c 100644--- a/README_zh.md+++ b/README_zh.md@@ -142,15 +142,16 @@ engine.close() # 释放 mmap / 专家缓存## 性能实测-`examples/bench.py` 实测(prefill → 10 步采样 warmup → 200 token 计时段,每档 2 轮):+`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →+200 token 计时段,每档 2 轮):-| 档位 | 解码速度 | Prefill(冷/热)* | 峰值 active 内存 | 测试机器 |+| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 ||---|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 4.4 s / 0.8 s(27 tok) | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-10b` | 25.3 tok/s | 0.15 s / 0.25 s(37 tok) | 1.0 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 3.3–4.5 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 3.1 GiB | Mac mini M4 Pro, 24 GB |*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。-括号内为 prompt 长度。*+Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。**峰值 active 内存为 MLX allocator 的峰值(权重 + KV cache + 专家工作集),不含 RSS:专家权重经 mmap 从 SSD 流式读取,OS 页缓存不计入。*diff --git a/examples/bench.py b/examples/bench.pyindex e9e71eb..5b4d977 100644--- a/examples/bench.py+++ b/examples/bench.py@@ -9,9 +9,13 @@ from the checkpoint (or forced via the model name).Usage:python examples/bench.py /path/to/model [--ntok 200] [--warmup 10]python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL+ BENCH_LONG=1 python examples/bench.py edge0-35b # ~3k-token prefillEnv (sampling knobs, defaults follow the tier's GenerationConfig):BENCH_TEMP / BENCH_PROMPT / BENCH_SEED+ BENCH_LONG=1 use a ~3k-token synthetic prompt so the prefill timing+ is meaningful (short prompts only measure fixed+ overhead; set BENCH_PROMPT for your own long text)"""from __future__ import annotations@@ -31,10 +35,38 @@ PROMPTS = {}DEFAULT_PROMPT = "什么是混合专家模型(MoE)?简单介绍一下。"+# Synthetic long-context prompt (~3k tokens after tokenization): two+# non-repetitive paragraphs alternating 40 times, prefixed with a task+# instruction. Prefill timing on a few dozen tokens only measures fixed+# overhead; a long prompt exercises the real chunked prefill path+# (per-layer expert loading, KV growth).+_PARA_ZH = (+ "流式推理框架的设计要点在于把磁盘带宽、缓存层级与计算核心三者重叠"+ "起来。混合专家模型每一层只在少数专家上激活,路由器给出的稀疏选择"+ "恰好为按需加载提供了天然的批粒度。若能在前一步的隐状态基础上预判"+ "下一步的专家集合,固态硬盘的读取延迟就能被计算完全掩盖,这对边缘"+ "设备上的大模型部署具有直接意义。")+_PARA_EN = (+ "The design of a streaming inference framework hinges on overlapping "+ "disk bandwidth, cache hierarchy, and compute. A mixture-of-experts "+ "layer activates only a handful of experts per token, and the router "+ "sparsity provides a natural granularity for on-demand loading. If "+ "the expert set of the next step can be predicted from the previous "+ "hidden state, SSD read latency hides behind compute entirely, which "+ "matters for large-model deployment on edge hardware.")+++def _long_prompt() -> str:+ paras = [_PARA_ZH if i % 2 == 0 else _PARA_EN for i in range(40)]+ return "请仔细阅读以下材料并用一句话总结其核心思想:" + " ".join(paras)+def _prompt_for(engine) -> str:- return os.environ.get("BENCH_PROMPT") or PROMPTS.get(engine.name,- DEFAULT_PROMPT)+ if os.environ.get("BENCH_PROMPT"):+ return os.environ["BENCH_PROMPT"]+ if os.environ.get("BENCH_LONG") == "1":+ return _long_prompt()+ return PROMPTS.get(engine.name, DEFAULT_PROMPT)def run_bench(engine, ntok: int, warmup: int) -> dict:@@ -89,7 +121,9 @@ def run_bench(engine, ntok: int, warmup: int) -> dict:results.append(dict(prefill_s=t_prefill, decode_s=t_decode,ntok=len(out), peak_gib=peak_gib,tok_s=len(out) / t_decode if t_decode else 0.0))- print(f"prompt={len(ids)} tok prefill={t_prefill:.2f}s "+ pf_tps = (len(ids) / t_prefill) if t_prefill else 0.0+ print(f"prompt={len(ids)} tok prefill={t_prefill:.2f}s "+ f"({pf_tps:.0f} tok/s) "f"decode={len(out)}/{t_decode:.2f}s "f"tok/s={results[-1]['tok_s']:.1f} "f"peak_active={peak_gib:.2f} GiB", flush=True)