Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: name peak memory as the MLX allocator peak
kevin93277e4954b7cd8d
diff --git a/README.md b/README.mdindex f600408..c4e1c07 100644--- a/README.md+++ b/README.md@@ -120,14 +120,16 @@ edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:Measured with `python/examples/bench.py` (3.3k-token prompt prefill → 10sampled warmup steps → 200 timed sampled decode tokens, 2 runs per tier):-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak MLX allocator memory | Test machine ||---|---|---|---|---|| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB || `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |*Cold = first request after process start (expert weights fault in fromSSD); warm = subsequent requests (page cache resident). Prefill numbers-are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*+are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`). Peak MLX+allocator memory is `mlx.core.get_peak_memory()` after decode — not+process RSS.*Reproduce:@@ -151,9 +153,12 @@ python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL`python/pyproject.toml`). Garbled, mixed-language output on Apple A18 /A18 Pro means an older `mlx`: `pip install 'mlx==0.30.6''mlx-metal==0.30.6'` ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).-- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for- `edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add- headroom for the OS, tokenizer, and long-context KV growth.+- **Memory**: ~2.9 GB **MLX allocator** peak for `edge0-35b`, ~1.0 GB for+ `edge0-8b` (short contexts; see [Benchmark](#benchmark)). That figure is+ `mlx.core.get_peak_memory()`, not process RSS — file-backed expert pages,+ the MLX buffer cache, tokenizer, and interpreter are extra. The paper+ reports process footprints roughly 2× the allocator peak on a 16 GB M2.+ Add headroom for the OS, tokenizer, and long-context KV growth.- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB(`edge0-8b`); expert weights are mmapped and read on demand, they arenot loaded into RAM up front.diff --git a/python/examples/bench.py b/python/examples/bench.pyindex 971c459..5ebba37 100644--- a/python/examples/bench.py+++ b/python/examples/bench.py@@ -3,7 +3,7 @@Runs the same measurement shape as the deployment's bench: prefill ->sampled warmup steps -> timed sampled decode -> report tok/s and MLX-peak active memory. Works for both tiers; the tier is auto-detected+MLX allocator peak memory (`mlx.core.get_peak_memory`). Works for both tiers; the tier is auto-detectedfrom the checkpoint (or forced via the model name).Usage:@@ -126,10 +126,10 @@ def run_bench(engine, ntok: int, warmup: int) -> dict:f"({pf_tps:.0f} tok/s) "f"decode={len(out)}/{t_decode:.2f}s "f"tok/s={results[-1]['tok_s']:.1f} "- f"peak_active={peak_gib:.2f} GiB", flush=True)+ f"mlx_allocator_peak={peak_gib:.2f} GiB", flush=True)mean_ts = sum(r["tok_s"] for r in results) / len(results)max_peak = max(r["peak_gib"] for r in results)- print(f"[bench] mean tok/s={mean_ts:.1f} peak_active<={max_peak:.2f} GiB",+ print(f"[bench] mean tok/s={mean_ts:.1f} mlx_allocator_peak<={max_peak:.2f} GiB",flush=True)return {"runs": results, "mean_tok_s": mean_ts, "peak_gib": max_peak}diff --git a/python/tests/test_repo_hygiene.py b/python/tests/test_repo_hygiene.pyindex 9ea9b7c..9032505 100644--- a/python/tests/test_repo_hygiene.py+++ b/python/tests/test_repo_hygiene.py@@ -146,3 +146,11 @@ def test_notice_link_resolves():assert m, "README should link the NOTICE file"target = ROOT / m.group(1)assert target.is_file(), f"README NOTICE link target missing: {target}"+++def test_readme_names_peak_memory_as_mlx_allocator():+ """Readers mistook "peak active memory" for process RSS (#106)."""+ readme = (ROOT / "README.md").read_text(encoding="utf-8")+ assert "Peak MLX allocator memory" in readme+ assert "mlx.core.get_peak_memory()" in readme+ assert "not process RSS" in readme