Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs: name peak memory as the MLX allocator peak

kevin93277e4954b7cd8d
diff --git a/README.md b/README.md
index f600408..c4e1c07 100644
--- a/README.md
+++ b/README.md
@@ -120,14 +120,16 @@ edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:
Measured with `python/examples/bench.py` (3.3k-token prompt prefill → 10
sampled warmup steps → 200 timed sampled decode tokens, 2 runs per tier):
-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |
+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak MLX allocator memory | Test machine |
|---|---|---|---|---|
| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
*Cold = first request after process start (expert weights fault in from
SSD); warm = subsequent requests (page cache resident). Prefill numbers
-are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*
+are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`). Peak MLX
+allocator memory is `mlx.core.get_peak_memory()` after decode — not
+process RSS.*
Reproduce:
@@ -151,9 +153,12 @@ python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL
`python/pyproject.toml`). Garbled, mixed-language output on Apple A18 /
A18 Pro means an older `mlx`: `pip install 'mlx==0.30.6'
'mlx-metal==0.30.6'` ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).
-- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for
- `edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add
- headroom for the OS, tokenizer, and long-context KV growth.
+- **Memory**: ~2.9 GB **MLX allocator** peak for `edge0-35b`, ~1.0 GB for
+ `edge0-8b` (short contexts; see [Benchmark](#benchmark)). That figure is
+ `mlx.core.get_peak_memory()`, not process RSS — file-backed expert pages,
+ the MLX buffer cache, tokenizer, and interpreter are extra. The paper
+ reports process footprints roughly 2× the allocator peak on a 16 GB M2.
+ Add headroom for the OS, tokenizer, and long-context KV growth.
- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB
(`edge0-8b`); expert weights are mmapped and read on demand, they are
not loaded into RAM up front.
diff --git a/python/examples/bench.py b/python/examples/bench.py
index 971c459..5ebba37 100644
--- a/python/examples/bench.py
+++ b/python/examples/bench.py
@@ -3,7 +3,7 @@
Runs the same measurement shape as the deployment's bench: prefill ->
sampled warmup steps -> timed sampled decode -> report tok/s and MLX
-peak active memory. Works for both tiers; the tier is auto-detected
+MLX allocator peak memory (`mlx.core.get_peak_memory`). Works for both tiers; the tier is auto-detected
from the checkpoint (or forced via the model name).
Usage:
@@ -126,10 +126,10 @@ def run_bench(engine, ntok: int, warmup: int) -> dict:
f"({pf_tps:.0f} tok/s) "
f"decode={len(out)}/{t_decode:.2f}s "
f"tok/s={results[-1]['tok_s']:.1f} "
- f"peak_active={peak_gib:.2f} GiB", flush=True)
+ f"mlx_allocator_peak={peak_gib:.2f} GiB", flush=True)
mean_ts = sum(r["tok_s"] for r in results) / len(results)
max_peak = max(r["peak_gib"] for r in results)
- print(f"[bench] mean tok/s={mean_ts:.1f} peak_active<={max_peak:.2f} GiB",
+ print(f"[bench] mean tok/s={mean_ts:.1f} mlx_allocator_peak<={max_peak:.2f} GiB",
flush=True)
return {"runs": results, "mean_tok_s": mean_ts, "peak_gib": max_peak}
diff --git a/python/tests/test_repo_hygiene.py b/python/tests/test_repo_hygiene.py
index 9ea9b7c..9032505 100644
--- a/python/tests/test_repo_hygiene.py
+++ b/python/tests/test_repo_hygiene.py
@@ -146,3 +146,11 @@ def test_notice_link_resolves():
assert m, "README should link the NOTICE file"
target = ROOT / m.group(1)
assert target.is_file(), f"README NOTICE link target missing: {target}"
+
+
+def test_readme_names_peak_memory_as_mlx_allocator():
+ """Readers mistook "peak active memory" for process RSS (#106)."""
+ readme = (ROOT / "README.md").read_text(encoding="utf-8")
+ assert "Peak MLX allocator memory" in readme
+ assert "mlx.core.get_peak_memory()" in readme
+ assert "not process RSS" in readme