Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: add community hardware benchmarks skeleton
JiahaoWang76e04779f595
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdnew file mode 100644index 0000000..3c583fb--- /dev/null+++ b/docs/benchmarks/README.md@@ -0,0 +1,59 @@+# Community Hardware Benchmarks++This directory collects real-world performance numbers for edge0 across different+hardware, contributed by the community. If you run edge0 on your machine, your+numbers are welcome here — **every reporter is credited by handle**.++## How to contribute a result++1. Run the benchmark on your device (macOS / Apple Silicon is supported today):++ ```bash+ cd python+ python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'+ export EDGE0_8B_MODEL=$PWD/models/edge0-8b # or EDGE0_35B_MODEL+ BENCH_LONG=1 .venv/bin/python examples/bench.py edge0-8b+ ```++2. Post your numbers in the pinned+ [Hardware Benchmarks discussion](https://github.com/Edge0-AI/Edge0/discussions/142),+ using the submission block there.++3. Either a maintainer curates your reply into [`results.csv`](./results.csv),+ or you open a PR adding your own row — we'll help review it.++> Other platforms (Windows / iOS / Android) don't have a one-command benchmark+> yet. Help build them! See the good-first-issues linked from the discussion:+> [#139](https://github.com/Edge0-AI/Edge0/issues/139) (Windows),+> [#140](https://github.com/Edge0-AI/Edge0/issues/140) (iOS),+> [#141](https://github.com/Edge0-AI/Edge0/issues/141) (Android).++## Data++Results live in [`results.csv`](./results.csv) (machine-readable; GitHub renders+it as a table). Columns:++| column | meaning |+| --- | --- |+| `date` | submission date (YYYY-MM-DD) |+| `tier` | `edge0-8b` or `edge0-35b` |+| `runtime` | backend used, e.g. `python-mlx` |+| `device` | device / machine name |+| `chip` | SoC, e.g. `Apple M2` |+| `ram_gb` | unified / system memory in GiB |+| `os` | OS version |+| `storage` | `internal SSD` / `external USB-SSD` (matters for expert paging) |+| `mlx_version` | `pip show mlx` version |+| `decode_tok_s` | decode throughput (tokens/sec) |+| `prefill_tok_s` | prefill throughput, prefer `BENCH_LONG` long-context |+| `peak_active_gib` | MLX peak active memory (GiB) |+| `on_battery` | `yes` / `no` |+| `reporter` | GitHub handle, e.g. `@you` |+| `notes` | anything unusual (throttling, short vs long, background load) |++## Why this matters++edge0 streams experts from SSD on demand, so throughput depends heavily on RAM+and storage speed — low-memory machines evict and re-read expert pages more+often. Collecting numbers across real hardware helps us (and users) understand+the practical performance envelope on each device.diff --git a/docs/benchmarks/results.csv b/docs/benchmarks/results.csvnew file mode 100644index 0000000..3b3c1de--- /dev/null+++ b/docs/benchmarks/results.csv@@ -0,0 +1,2 @@+date,tier,runtime,device,chip,ram_gb,os,storage,mlx_version,decode_tok_s,prefill_tok_s,peak_active_gib,on_battery,reporter,notes+2026-10-15,edge0-8b,python-mlx,MacBook Pro,Apple M5,16,macOS 26.3.2,internal SSD,0.30.6,11.8,314,3.19,no,@shuk777,"decode 13.5 short / 11.8 long; prefill 231 cold / 314 warm (BENCH_LONG)"