Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

add notice and so on

Binea37f258194d
diff --git a/NOTICE b/NOTICE
index ddee8a9..2b044cc 100644
--- a/NOTICE
+++ b/NOTICE
@@ -4,7 +4,7 @@ edge0
This project bundles or adapts the following third-party works:
mlx-lm (MIT License)
- - `src/edge0/backends/mlx/_impl/qwen3_next.py` is vendored from mlx-lm
+ - `python/src/edge0/backends/mlx/_impl/qwen3_next.py` is vendored from mlx-lm
0.31.0 (https://github.com/ml-explore/mlx-lm), with additive hooks for
prerouter routing and layer callbacks. Non-prerouter paths are kept
byte-identical to the upstream file.
@@ -12,10 +12,24 @@ mlx-lm (MIT License)
Copyright (c) 2023-2024 Apple Inc. and the mlx-lm contributors.
bailing_hybrid (Apache License 2.0)
- - `src/edge0/backends/mlx/_impl/bailing_hybrid.py` is vendored from the
+ - `python/src/edge0/backends/mlx/_impl/bailing_hybrid.py` is vendored from the
Ling 3.0 MLX implementation (rapid-mlx / Ling-3.0-tiny, Apache-2.0), with
prerouter hook identifiers renamed for consistency with the framework.
Copyright (c) the bailing / Ling MLX contributors.
+llama.cpp (MIT License)
+ - The Android and Windows engines build their inference core from a
+ pristine upstream checkout of llama.cpp
+ (https://github.com/ggml-org/llama.cpp), pinned at commit 7ab4ee7
+ (release b11100) via `vendor.llama.pin` and materialized into the
+ gitignored `vendor/llama.cpp` by the platform build scripts.
+ - It is never patched in place: all modifications ship as replayable
+ patch bands under `patches/llama.cpp/{common,android,windows}` and are
+ replayed into an isolated build worktree at compile time.
+ - ggml (Apache License 2.0) is developed within the same upstream tree.
+ - Not vendored as files in this repo; built from source by the consumer
+ (see `android/NOTICE` and `windows/NOTICE` for the per-platform detail).
+ Copyright (c) 2023-2024 The ggml authors.
+
All other files are Copyright (c) 2025 edge0 contributors, licensed under
the Apache License, Version 2.0 (see LICENSE).
diff --git a/macos/README.md b/macos/README.md
index cb5b40d..aff7c0f 100644
--- a/macos/README.md
+++ b/macos/README.md
@@ -95,9 +95,3 @@ edge0.app (Tauri 2 + React 19)
- **Engine:** C++/Metal on mlx 0.30.6. Int4 streaming experts (mmap slices + LRU hot stack + SSD offload), prerouter prefetch, unmerged LoRA, KV prefix cache. NAX GEMM is off by default (`MLX_METAL_NO_NAX`) because mlx 0.30.6 is numerically wrong for edge0-8b on M5.
`EDGE0_HOME` defaults to `~/.edge0`.
-
-## License
-
-Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
-
-Models: [Edge0-8B-A1B-preview](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) and [Edge0-35B-A3B-preview](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) (Apache-2.0). Engine math library: [mlx](https://github.com/ml-explore/mlx) (MIT). Upstream framework: [Edge0-AI/edge0](https://github.com/Edge0-AI/edge0).
diff --git a/python/README.md b/python/README.md
index 2bee612..c3fe3bc 100644
--- a/python/README.md
+++ b/python/README.md
@@ -1,24 +1,202 @@
# edge0 — Python framework
-The Python implementation of the edge0 streaming MoE inference framework
-(SSD expert offload + Recover-LoRA + prerouter routing prediction),
-shipping the MLX backend for Apple Silicon.
+This document is for **developers installing and running the edge0 Python
+framework from source**. It covers: quick start (install → models → run),
+measured performance and quality, and the design behind the stack.
-This directory is one platform of the edge0 multi-platform repo — see the
-[repository README](../README.md) for the full picture (macOS / iOS /
-Android runtimes and the unified-framework roadmap).
+edge0 is published as a **monorepo** —
+[`Edge0-AI/edge0`](https://github.com/Edge0-AI/edge0) — whose top level
+holds the shared docs (`docs/`) and the platform subprojects (`python/` =
+this framework, plus `macos/`, `ios/`, `android/`, `windows/`). The Python
+framework is the reference implementation of the edge0 recipe — **SSD
+expert offload + Recover-LoRA + prerouter routing prediction** — running
+large sparse-MoE models on Apple Silicon via MLX, with peak memory bounded
+by the *active* expert set rather than the parameter count.
-## Quick start
+Third-party attribution: see [`NOTICE`](../NOTICE) at the repo root. Model
+cards: [Edge0/Edge0-8B-A1B-preview](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)
+· [Edge0/Edge0-35B-A3B-preview](https://huggingface.co/Edge0/Edge0-35B-A3B-preview).
+All commands below run from this directory (`python/`).
+
+---
+
+## Quick Start
+
+### Prerequisites
+
+| component | requirement |
+|---|---|
+| OS / hardware | macOS on Apple Silicon (M1/M2/M3/M4) — the MLX backend is Apple-Silicon-only; a CUDA backend is on the roadmap |
+| Python | 3.10+ (3.12 recommended) |
+| MLX | `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see `pyproject.toml`) |
+| Memory | ~2.9 GB peak active (edge0-35b), ~1.0 GB (edge0-8b), short contexts |
+| Disk | ~23 GB (edge0-35b) / ~4.2 GB (edge0-8b); expert weights are mmapped and read on demand, not loaded into RAM up front |
+
+> Garbled, mixed-language output on Apple A18 / A18 Pro means an older
+> `mlx`: `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`
+> ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).
+
+### Install
```bash
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
-edge0 demo edge0-8b # after downloading a model; see ../README.md
```
-- Docs: [`docs/`](../docs/) at the repo root (architecture, attention, MoE, streaming, prerouter)
-- Examples: [`examples/`](examples/) · Scripts: [`scripts/`](scripts/)
-- Tests: `pytest` (unit) · `pytest -m slow` (real weights)
+### Models
+
+Both tiers are published on Hugging Face and ModelScope; each repo bundles
+the base checkpoint and the trained LoRA + prerouter adapters in **one
+directory**, so a single download is a ready-to-run model:
+
+```bash
+# the repo's helper (defaults to the two published tiers):
+.venv/bin/python scripts/fetch_models.py --tier edge0-8b --target-dir models
+.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models
+
+# or directly with the CLI:
+.venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview --local-dir models/edge0-8b
+.venv/bin/huggingface-cli download Edge0/Edge0-35B-A3B-preview --local-dir models/edge0-35b
+```
+
+Point edge0 at a tier through an environment variable, or pass the
+directory directly (the tier is auto-detected from `config.json`):
+
+```bash
+export EDGE0_8B_MODEL=$PWD/models/edge0-8b
+```
+
+### Run
+
+```bash
+edge0 demo edge0-8b # one-shot generation demo
+edge0 chat edge0-8b --prompt "Explain streaming inference in one sentence."
+edge0 serve edge0-8b # OpenAI-compatible server on http://127.0.0.1:8000
+```
+
+```bash
+curl http://127.0.0.1:8000/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"edge0-8b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
+```
+
+`python -m edge0 ...` is equivalent to `edge0 ...`.
+
+### Python API
+
+```python
+from edge0 import AutoEngine
+from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
+
+engine = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected
+req = ChatRequest(
+ model=engine.name,
+ messages=[ChatMessage(role="user", content="Hello!")],
+ max_tokens=64,
+)
+tokens, meta = ChatSession(engine, req).run()
+print(engine._tok.decode(tokens))
+engine.close() # release mmaps / expert cache
+```
+
+`examples/demo.py` is the same minimal walkthrough (`edge0 demo` runs this
+exact path).
+
+---
+
+## Performance
+
+Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled
+warmup steps → 200 timed sampled decode tokens, 2 runs per tier):
+
+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |
+|---|---|---|---|---|
+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+
+*Cold = first request after process start (expert weights fault in from
+SSD); warm = subsequent requests (page cache resident).*
+
+```bash
+python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL
+python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL
+```
+
+## Quality
+
+Run with [OpenCompass](https://github.com/open-compass/opencompass) under
+identical settings for the edge0 models (int4 + trained adapters +
+prerouter) and the original fp16 bases. The edge0 pipeline loses little:
+**3.9 points on average for edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even
+above the base). Max 100:
+
+| Benchmark | edge0-35b (int4) | Qwen3.6-35B-A3B (fp16) | edge0-8b (int4) | Ling 3.0 tiny (fp16) |
+|---|---:|---:|---:|---:|
+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
+| **Average** | **79.2** | **83.2** | **69.9** | **72.7** |
+
+---
+
+## Technical Details
-## License
+### Core mechanisms
-Apache-2.0, including vendored third-party code (see [NOTICE](../NOTICE)).
+- **SSD expert offload** — expert weights stream from storage on demand;
+ peak memory is bounded by the active set, not the parameter count.
+- **Prerouter** — a trained head predicts expert routing one step ahead, so
+ expert loads overlap the forward pass instead of stalling it (**up to
+ +59%** decode throughput; the gain grows with storage latency, model
+ size, and routed width *K*).
+- **Recover-LoRA** — the int4 base is frozen and LoRA adapters are trained
+ by distillation from the FP teacher, recovering most of the 4-bit
+ quantization loss. Adapters stay unmerged: one read-only base serves
+ multiple adapter sets.
+
+### Design
+
+- **transformers-style usage** — `AutoModel` / `AutoConfig` / `AutoEngine`
+ resolve the tier from the model name.
+- **Backend isolation** — all MLX code lives under
+ `src/edge0/backends/mlx/`; the core logic (model specs, prerouter,
+ streaming expert pool, server) depends only on the backend facade
+ (`backends/base.py`), so a new backend implements the same facade
+ (`backends/cuda/` is a reserved slot) with zero changes to core code.
+- **Adapters as safetensors** — LoRA and prerouter weights are
+ `.safetensors` files with provenance metadata (source, version, owner
+ layers), resolved from the model directory or the gitignored `artifacts/`
+ fallback.
+- **Model + adapters in one directory** — a model directory holds the base
+ checkpoint and that model's adapters; upgrading adapters swaps adapter
+ files only — the base stays read-only and is never merged.
+
+### Package layout
+
+```
+src/edge0/
+├── backends/mlx/ # MLX backend (isolation boundary; cuda/ reserved)
+├── engine/ models/ moe/ prerouter/ streaming/
+├── adapters/ attention/ server/
+└── cli.py registry.py sampling.py config.py
+```
+
+---
+
+## Documentation
+
+- [Architecture](../docs/architecture.md)
+- [Attention](../docs/attention.md) / [MoE](../docs/moe.md) / [SSD streaming](../docs/streaming.md) / [prerouter](../docs/prerouter.md)
+- [Adding a model](../docs/adding-a-model.md)
+- [edge0-35b](../docs/models/edge0-35b.md) / [edge0-8b](../docs/models/edge0-8b.md)
+- Technical report: [The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063) ([PDF](../paper/main.pdf))
+
+## Tests
+
+```bash
+pytest # unit tests (no real weights)
+EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q # real-weight generation
+.venv/bin/python scripts/e2e_smoke.py \
+ --qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b
+```
diff --git a/python/tests/test_repo_hygiene.py b/python/tests/test_repo_hygiene.py
index a6ed8ef..9ea9b7c 100644
--- a/python/tests/test_repo_hygiene.py
+++ b/python/tests/test_repo_hygiene.py
@@ -15,13 +15,15 @@ from __future__ import annotations
import pathlib
import re
+import subprocess
# Repo root (this file lives at <root>/python/tests/): the hygiene guards
# scan the whole multi-platform repo, not just the Python subproject.
ROOT = pathlib.Path(__file__).resolve().parents[2]
_EXCLUDE_DIRS = {".git", ".venv", ".pytest_cache", "__pycache__", ".agents",
- ".codex"}
+ ".codex", "vendor", "wt", ".edge0", "models", "node_modules",
+ ".gradle", "build-dl", "target", "dist", "build"}
_EXCLUDE_PARTS = {"egg-info"}
_EXCLUDE_SUFFIXES = {".pyc", ".so", ".dylib", ".bin", ".safetensors", ".npz",
".png", ".jpg", ".jpeg", ".gif", ".ico", ".pdf"}
@@ -53,7 +55,35 @@ _SECRET_PATTERNS = [
]
+def _git_ls_files() -> list[str] | None:
+ """Repo-relative paths of the files that are (or would be) committed:
+ tracked files plus untracked-but-not-gitignored ones. Materialized
+ build areas (``vendor/``, ``wt/``, ``models/``, ``.venv/``) are
+ gitignored, so this naturally excludes them — the hygiene guards check
+ *shipped* code, not downloaded third-party dependencies. Returns None
+ if git is unavailable, so the caller can fall back to a directory walk.
+ """
+ try:
+ out = subprocess.run(
+ ["git", "ls-files", "--cached", "--others", "--exclude-standard",
+ "-z"],
+ cwd=ROOT, capture_output=True, check=True,
+ ).stdout
+ except (OSError, subprocess.CalledProcessError):
+ return None
+ return [p for p in out.decode("utf-8", "replace").split("\0") if p]
+
+
def _iter_text_files():
+ shipped = _git_ls_files()
+ if shipped is not None:
+ for rel in shipped:
+ path = ROOT / rel
+ if path.suffix in _EXCLUDE_SUFFIXES or not path.is_file():
+ continue
+ yield path
+ return
+ # Fallback (no git): walk the tree, skipping vcs / build / vendor dirs.
for path in ROOT.rglob("*"):
rel = path.relative_to(ROOT)
if any(part in _EXCLUDE_DIRS for part in rel.parts):