Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: address review on MLX allocator peak wording
kevin93272924af9a35a4
diff --git a/README.md b/README.mdindex c4e1c07..485f234 100644--- a/README.md+++ b/README.md@@ -156,9 +156,10 @@ python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL- **Memory**: ~2.9 GB **MLX allocator** peak for `edge0-35b`, ~1.0 GB for`edge0-8b` (short contexts; see [Benchmark](#benchmark)). That figure is`mlx.core.get_peak_memory()`, not process RSS — file-backed expert pages,- the MLX buffer cache, tokenizer, and interpreter are extra. The paper- reports process footprints roughly 2× the allocator peak on a 16 GB M2.- Add headroom for the OS, tokenizer, and long-context KV growth.+ the MLX buffer cache, tokenizer, and interpreter are extra, but that+ extra RSS is reclaimable and does not set the machine requirement (the+ same build runs on a 16 GB M2). Add headroom for the OS, tokenizer, and+ long-context KV growth.- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB(`edge0-8b`); expert weights are mmapped and read on demand, they arenot loaded into RAM up front.diff --git a/python/README.md b/python/README.mdindex 3a323f4..2809923 100644--- a/python/README.md+++ b/python/README.md@@ -31,7 +31,7 @@ All commands below run from this directory (`python/`).| OS / hardware | macOS on Apple Silicon (M1/M2/M3/M4) — the MLX backend is Apple-Silicon-only; a CUDA backend is on the roadmap || Python | 3.10+ (3.12 recommended) || MLX | `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see `pyproject.toml`) |-| Memory | ~2.9 GB peak active (edge0-35b), ~1.0 GB (edge0-8b), short contexts |+| Memory | ~2.9 GB MLX allocator peak (edge0-35b), ~1.0 GB (edge0-8b), short contexts; this is `mlx.core.get_peak_memory()`, not process RSS || Disk | ~23 GB (edge0-35b) / ~4.2 GB (edge0-8b); expert weights are mmapped and read on demand, not loaded into RAM up front |> Garbled, mixed-language output on Apple A18 / A18 Pro means an older@@ -110,13 +110,15 @@ exact path).Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampledwarmup steps → 200 timed sampled decode tokens, 2 runs per tier):-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak MLX allocator memory | Test machine ||---|---|---|---|---|| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB || `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |*Cold = first request after process start (expert weights fault in from-SSD); warm = subsequent requests (page cache resident).*+SSD); warm = subsequent requests (page cache resident). Peak MLX+allocator memory is `mlx.core.get_peak_memory()` after decode — not+process RSS.*```bashpython examples/bench.py edge0-8b # via $EDGE0_8B_MODELdiff --git a/python/examples/bench.py b/python/examples/bench.pyindex 5ebba37..e2b5058 100644--- a/python/examples/bench.py+++ b/python/examples/bench.py@@ -3,8 +3,9 @@Runs the same measurement shape as the deployment's bench: prefill ->sampled warmup steps -> timed sampled decode -> report tok/s and MLX-MLX allocator peak memory (`mlx.core.get_peak_memory`). Works for both tiers; the tier is auto-detected-from the checkpoint (or forced via the model name).+allocator peak memory (`mlx.core.get_peak_memory`). Works for both+tiers; the tier is auto-detected from the checkpoint (or forced via the+model name).Usage:python examples/bench.py /path/to/model [--ntok 200] [--warmup 10]diff --git a/python/tests/test_repo_hygiene.py b/python/tests/test_repo_hygiene.pyindex 9032505..0c329be 100644--- a/python/tests/test_repo_hygiene.py+++ b/python/tests/test_repo_hygiene.py@@ -17,6 +17,8 @@ import pathlibimport reimport subprocess+import pytest+# Repo root (this file lives at <root>/python/tests/): the hygiene guards# scan the whole multi-platform repo, not just the Python subproject.ROOT = pathlib.Path(__file__).resolve().parents[2]@@ -148,9 +150,12 @@ def test_notice_link_resolves():assert target.is_file(), f"README NOTICE link target missing: {target}"-def test_readme_names_peak_memory_as_mlx_allocator():[email protected]("readme", ["README.md", "python/README.md"])+def test_readme_names_peak_memory_as_mlx_allocator(readme):"""Readers mistook "peak active memory" for process RSS (#106)."""- readme = (ROOT / "README.md").read_text(encoding="utf-8")- assert "Peak MLX allocator memory" in readme- assert "mlx.core.get_peak_memory()" in readme- assert "not process RSS" in readme+ text = (ROOT / readme).read_text(encoding="utf-8")+ assert "Peak MLX allocator memory" in text+ assert "mlx.core.get_peak_memory()" in text+ assert "not process RSS" in text+ assert "peak active" not in text.lower(), (+ f"{readme} still labels the allocator peak as 'peak active'")