Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

bench: long-prompt prefill measurement (BENCH_LONG=1)

linyubupa0117e3c8751e
diff --git a/README.md b/README.md
index 2dc9a6d..b4810eb 100644
--- a/README.md
+++ b/README.md
@@ -165,17 +165,17 @@ multiple prompts):
## Benchmark
-Measured with `examples/bench.py` (prefill → 10 sampled warmup steps → 200
-timed sampled decode tokens, 2 runs per tier):
+Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled
+warmup steps → 200 timed sampled decode tokens, 2 runs per tier):
-| Tier | Decode speed | Prefill (cold / warm)* | Peak active memory | Test machine |
+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |
|---|---|---|---|---|
-| `edge0-35b` | 14.9–17.7 tok/s | 4.4 s / 0.8 s (27 tok) | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |
-| `edge0-10b` | 25.3 tok/s | 0.15 s / 0.25 s (37 tok) | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 3.3–4.5 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 3.1 GiB | Mac mini M4 Pro, 24 GB |
*Cold = first request after process start (expert weights fault in from
-SSD); warm = subsequent requests (page cache resident). Prompt length in
-parentheses.*
+SSD); warm = subsequent requests (page cache resident). Prefill numbers
+are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*
*Peak active memory is the MLX allocator's peak (model weights + KV cache +
expert working set), not RSS: expert weights stream from SSD via mmap and the
diff --git a/README_zh.md b/README_zh.md
index 91e0d58..9d7da1c 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -142,15 +142,16 @@ engine.close() # 释放 mmap / 专家缓存
## 性能实测
-`examples/bench.py` 实测(prefill → 10 步采样 warmup → 200 token 计时段,每档 2 轮):
+`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →
+200 token 计时段,每档 2 轮):
-| 档位 | 解码速度 | Prefill(冷/热)* | 峰值 active 内存 | 测试机器 |
+| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 |
|---|---|---|---|---|
-| `edge0-35b` | 14.9–17.7 tok/s | 4.4 s / 0.8 s(27 tok) | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |
-| `edge0-10b` | 25.3 tok/s | 0.15 s / 0.25 s(37 tok) | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 3.3–4.5 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-10b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 3.1 GiB | Mac mini M4 Pro, 24 GB |
*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。
-括号内为 prompt 长度。*
+Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。*
*峰值 active 内存为 MLX allocator 的峰值(权重 + KV cache + 专家工作集),
不含 RSS:专家权重经 mmap 从 SSD 流式读取,OS 页缓存不计入。*
diff --git a/examples/bench.py b/examples/bench.py
index e9e71eb..5b4d977 100644
--- a/examples/bench.py
+++ b/examples/bench.py
@@ -9,9 +9,13 @@ from the checkpoint (or forced via the model name).
Usage:
python examples/bench.py /path/to/model [--ntok 200] [--warmup 10]
python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL
+ BENCH_LONG=1 python examples/bench.py edge0-35b # ~3k-token prefill
Env (sampling knobs, defaults follow the tier's GenerationConfig):
BENCH_TEMP / BENCH_PROMPT / BENCH_SEED
+ BENCH_LONG=1 use a ~3k-token synthetic prompt so the prefill timing
+ is meaningful (short prompts only measure fixed
+ overhead; set BENCH_PROMPT for your own long text)
"""
from __future__ import annotations
@@ -31,10 +35,38 @@ PROMPTS = {
}
DEFAULT_PROMPT = "什么是混合专家模型(MoE)?简单介绍一下。"
+# Synthetic long-context prompt (~3k tokens after tokenization): two
+# non-repetitive paragraphs alternating 40 times, prefixed with a task
+# instruction. Prefill timing on a few dozen tokens only measures fixed
+# overhead; a long prompt exercises the real chunked prefill path
+# (per-layer expert loading, KV growth).
+_PARA_ZH = (
+ "流式推理框架的设计要点在于把磁盘带宽、缓存层级与计算核心三者重叠"
+ "起来。混合专家模型每一层只在少数专家上激活,路由器给出的稀疏选择"
+ "恰好为按需加载提供了天然的批粒度。若能在前一步的隐状态基础上预判"
+ "下一步的专家集合,固态硬盘的读取延迟就能被计算完全掩盖,这对边缘"
+ "设备上的大模型部署具有直接意义。")
+_PARA_EN = (
+ "The design of a streaming inference framework hinges on overlapping "
+ "disk bandwidth, cache hierarchy, and compute. A mixture-of-experts "
+ "layer activates only a handful of experts per token, and the router "
+ "sparsity provides a natural granularity for on-demand loading. If "
+ "the expert set of the next step can be predicted from the previous "
+ "hidden state, SSD read latency hides behind compute entirely, which "
+ "matters for large-model deployment on edge hardware.")
+
+
+def _long_prompt() -> str:
+ paras = [_PARA_ZH if i % 2 == 0 else _PARA_EN for i in range(40)]
+ return "请仔细阅读以下材料并用一句话总结其核心思想:" + " ".join(paras)
+
def _prompt_for(engine) -> str:
- return os.environ.get("BENCH_PROMPT") or PROMPTS.get(engine.name,
- DEFAULT_PROMPT)
+ if os.environ.get("BENCH_PROMPT"):
+ return os.environ["BENCH_PROMPT"]
+ if os.environ.get("BENCH_LONG") == "1":
+ return _long_prompt()
+ return PROMPTS.get(engine.name, DEFAULT_PROMPT)
def run_bench(engine, ntok: int, warmup: int) -> dict:
@@ -89,7 +121,9 @@ def run_bench(engine, ntok: int, warmup: int) -> dict:
results.append(dict(prefill_s=t_prefill, decode_s=t_decode,
ntok=len(out), peak_gib=peak_gib,
tok_s=len(out) / t_decode if t_decode else 0.0))
- print(f"prompt={len(ids)} tok prefill={t_prefill:.2f}s "
+ pf_tps = (len(ids) / t_prefill) if t_prefill else 0.0
+ print(f"prompt={len(ids)} tok prefill={t_prefill:.2f}s "
+ f"({pf_tps:.0f} tok/s) "
f"decode={len(out)}/{t_decode:.2f}s "
f"tok/s={results[-1]['tok_s']:.1f} "
f"peak_active={peak_gib:.2f} GiB", flush=True)