Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs: address review on MLX allocator peak wording

kevin93272924af9a35a4
diff --git a/README.md b/README.md
index c4e1c07..485f234 100644
--- a/README.md
+++ b/README.md
@@ -156,9 +156,10 @@ python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL
- **Memory**: ~2.9 GB **MLX allocator** peak for `edge0-35b`, ~1.0 GB for
`edge0-8b` (short contexts; see [Benchmark](#benchmark)). That figure is
`mlx.core.get_peak_memory()`, not process RSS — file-backed expert pages,
- the MLX buffer cache, tokenizer, and interpreter are extra. The paper
- reports process footprints roughly 2× the allocator peak on a 16 GB M2.
- Add headroom for the OS, tokenizer, and long-context KV growth.
+ the MLX buffer cache, tokenizer, and interpreter are extra, but that
+ extra RSS is reclaimable and does not set the machine requirement (the
+ same build runs on a 16 GB M2). Add headroom for the OS, tokenizer, and
+ long-context KV growth.
- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB
(`edge0-8b`); expert weights are mmapped and read on demand, they are
not loaded into RAM up front.
diff --git a/python/README.md b/python/README.md
index 3a323f4..2809923 100644
--- a/python/README.md
+++ b/python/README.md
@@ -31,7 +31,7 @@ All commands below run from this directory (`python/`).
| OS / hardware | macOS on Apple Silicon (M1/M2/M3/M4) — the MLX backend is Apple-Silicon-only; a CUDA backend is on the roadmap |
| Python | 3.10+ (3.12 recommended) |
| MLX | `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see `pyproject.toml`) |
-| Memory | ~2.9 GB peak active (edge0-35b), ~1.0 GB (edge0-8b), short contexts |
+| Memory | ~2.9 GB MLX allocator peak (edge0-35b), ~1.0 GB (edge0-8b), short contexts; this is `mlx.core.get_peak_memory()`, not process RSS |
| Disk | ~23 GB (edge0-35b) / ~4.2 GB (edge0-8b); expert weights are mmapped and read on demand, not loaded into RAM up front |
> Garbled, mixed-language output on Apple A18 / A18 Pro means an older
@@ -110,13 +110,15 @@ exact path).
Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled
warmup steps → 200 timed sampled decode tokens, 2 runs per tier):
-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |
+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak MLX allocator memory | Test machine |
|---|---|---|---|---|
| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
*Cold = first request after process start (expert weights fault in from
-SSD); warm = subsequent requests (page cache resident).*
+SSD); warm = subsequent requests (page cache resident). Peak MLX
+allocator memory is `mlx.core.get_peak_memory()` after decode — not
+process RSS.*
```bash
python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL
diff --git a/python/examples/bench.py b/python/examples/bench.py
index 5ebba37..e2b5058 100644
--- a/python/examples/bench.py
+++ b/python/examples/bench.py
@@ -3,8 +3,9 @@
Runs the same measurement shape as the deployment's bench: prefill ->
sampled warmup steps -> timed sampled decode -> report tok/s and MLX
-MLX allocator peak memory (`mlx.core.get_peak_memory`). Works for both tiers; the tier is auto-detected
-from the checkpoint (or forced via the model name).
+allocator peak memory (`mlx.core.get_peak_memory`). Works for both
+tiers; the tier is auto-detected from the checkpoint (or forced via the
+model name).
Usage:
python examples/bench.py /path/to/model [--ntok 200] [--warmup 10]
diff --git a/python/tests/test_repo_hygiene.py b/python/tests/test_repo_hygiene.py
index 9032505..0c329be 100644
--- a/python/tests/test_repo_hygiene.py
+++ b/python/tests/test_repo_hygiene.py
@@ -17,6 +17,8 @@ import pathlib
import re
import subprocess
+import pytest
+
# Repo root (this file lives at <root>/python/tests/): the hygiene guards
# scan the whole multi-platform repo, not just the Python subproject.
ROOT = pathlib.Path(__file__).resolve().parents[2]
@@ -148,9 +150,12 @@ def test_notice_link_resolves():
assert target.is_file(), f"README NOTICE link target missing: {target}"
-def test_readme_names_peak_memory_as_mlx_allocator():
[email protected]("readme", ["README.md", "python/README.md"])
+def test_readme_names_peak_memory_as_mlx_allocator(readme):
"""Readers mistook "peak active memory" for process RSS (#106)."""
- readme = (ROOT / "README.md").read_text(encoding="utf-8")
- assert "Peak MLX allocator memory" in readme
- assert "mlx.core.get_peak_memory()" in readme
- assert "not process RSS" in readme
+ text = (ROOT / readme).read_text(encoding="utf-8")
+ assert "Peak MLX allocator memory" in text
+ assert "mlx.core.get_peak_memory()" in text
+ assert "not process RSS" in text
+ assert "peak active" not in text.lower(), (
+ f"{readme} still labels the allocator peak as 'peak active'")