Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs(readme): restructure EN/ZH READMEs in sglang/vllm style

Bincafd81be4e24
diff --git a/README.md b/README.md
index 5e0240a..110e96c 100644
--- a/README.md
+++ b/README.md
@@ -6,6 +6,8 @@
**An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction.**
+**Python** · **macOS** · **iOS** · **Android** — one recipe, every device.
+
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Edge0--35B--A3B--preview-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Edge0--8B--A1B--preview-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)
[![ModelScope](https://img.shields.io/badge/ModelScope-Edge0--35B--A3B--preview-624AFF?style=for-the-badge)](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)
@@ -18,12 +20,48 @@ English | [中文](README_zh.md)
</div>
+## 📰 News
+
+- **[2026-09-30]** We released the **edge0 inference engines for three platforms — iOS, macOS and Android** — so users get the best inference experience across architectures and platforms. The source is open-sourced in this repo ([`ios/`](ios/) · [`macos/`](macos/) · [`android/`](android/)) — see each directory's README for details. The **unified inference framework** follows by the **end of October 2026**; see the [Roadmap](#-roadmap).
+- **[2026-09-20]** `--prefill-ondemand` lands for `edge0 demo / chat / serve` ([#112](https://github.com/Edge0-AI/Edge0/pull/112)).
+- **[2026-09-16]** Our technical report is on arXiv: [The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction](https://arxiv.org/abs/2609.18063).
+- **[2026-09-08]** Initial open-source release of **edge0**, together with both model tiers — [`Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) and [`Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) — on Hugging Face and ModelScope.
+
+## 🤔 About
+
**edge0** is an open-source streaming MoE inference framework. It
generalizes the production-proven recipe — **SSD expert offload +
Recover-LoRA + prerouter routing prediction** — into an extensible
-framework. The backend is isolated by design: the current MLX backend
-runs on Apple Silicon, and additional platforms (CUDA, …) plug into the
-same core abstractions.
+framework that runs large sparse-MoE models on consumer hardware: peak
+memory is bounded by the *active* expert set, not the parameter count.
+
+### Core mechanisms
+
+- **SSD expert offload**: expert weights are streamed from storage on
+ demand; peak memory is bounded by the active set, not the parameter
+ count.
+- **Prerouter**: a trained head predicts expert routing one step
+ ahead, so expert loads overlap the forward pass instead of stalling
+ it — **up to +59%** decode throughput; the gain grows with storage
+ latency, model size, and routed width *K*.
+- **Recover-LoRA**: the int4 base is frozen and LoRA adapters are
+ trained by distillation from the FP teacher, recovering most of the
+ quantization loss at 4-bit (see [Quality](#quality)). Adapters stay
+ unmerged: one read-only base serves multiple adapter sets.
+
+### Platforms
+
+One repo, one recipe, per-platform runtimes:
+
+| Platform | Directory | Stack | Status |
+|---|---|---|---|
+| **Python** (macOS · Apple Silicon) | [`python/`](python/) | Python + MLX | ✅ Available now |
+| **macOS** app & CLI | [`macos/`](macos/) | Rust | ✅ Open-sourced (2026-09-30) |
+| **iOS** app | [`ios/`](ios/) | Swift + MLX Swift | ✅ Open-sourced (2026-09-30) |
+| **Android** app & engine | [`android/`](android/) | Kotlin + native engine | ✅ Open-sourced (2026-09-30) |
+| **Windows** | — | — | 🗺️ On the roadmap |
+
+### Models
Two model tiers ship with the framework. Each tier is an end-to-end
release: the released checkpoint, the trained LoRA adapters, and the
@@ -40,32 +78,19 @@ LoRA and prerouter training done for this framework — the adapter files
are co-located with each checkpoint and load automatically, so
`edge0 serve <tier>` runs the trained pipeline out of the box.
-## Requirements
-
-- **OS / hardware**: the MLX backend runs on macOS with Apple Silicon
- (M1/M2/M3/M4). The CUDA backend is on the roadmap — no other
- platforms are supported yet.
-- **Python**: 3.10+ (3.12 recommended).
-- **MLX**: `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see
- `pyproject.toml`). Garbled, mixed-language output on Apple A18 / A18 Pro
- means an older `mlx`: `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`
- ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).
-- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for
- `edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add
- headroom for the OS, tokenizer, and long-context KV growth.
-- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB
- (`edge0-8b`); expert weights are mmapped and read on demand, they are
- not loaded into RAM up front.
-
-## Design
+### Design
- **transformers-style usage**: `AutoModel` / `AutoConfig` / `AutoEngine`
resolve the tier from the model name;
-- **Backend isolation**: all MLX code lives under `edge0/backends/mlx/`;
- the core logic (model specs, prerouter, streaming expert pool, server)
- depends only on the backend facade (`edge0/backends/base.py`), so a new
- backend implements the same facade (`backends/cuda/` is a reserved
- slot) with zero changes to core code;
+- **Backend isolation (Python framework)**: within the Python framework,
+ all MLX code lives under `python/src/edge0/backends/mlx/`; the core
+ logic (model specs, prerouter, streaming expert pool, server) depends
+ only on the backend facade (`backends/base.py`), so a new backend
+ implements the same facade (`backends/cuda/` is a reserved slot) with
+ zero changes to core code. The iOS / macOS / Android engines ship
+ platform-native stacks today — bringing every platform under one
+ access layer is exactly what the unified inference framework
+ (see [Roadmap](#-roadmap)) will deliver;
- **Adapters as safetensors**: LoRA and prerouter weights are
`.safetensors` files with provenance metadata (source, version, owner
layers), resolved from the model directory or `artifacts/`;
@@ -74,34 +99,79 @@ are co-located with each checkpoint and load automatically, so
and that model's adapters; upgrading adapters swaps adapter
files only — the base stays read-only and is never merged.
-## Core mechanisms
+### Quality
-- **SSD expert offload**: expert weights are streamed from storage on
- demand; peak memory is bounded by the active set, not the parameter
- count.
-- **Prerouter**: a trained head predicts expert routing one step
- ahead, so expert loads overlap the forward pass instead of stalling
- it — **up to +59%** decode throughput; the gain grows with storage
- latency, model size, and routed width *K*.
-- **Recover-LoRA**: the int4 base is frozen and LoRA adapters are
- trained by distillation from the FP teacher, recovering most of the
- quantization loss at 4-bit (see [Quality](#quality)). Adapters stay
- unmerged: one read-only base serves multiple adapter sets.
+All benchmarks were run by us with [OpenCompass](https://github.com/open-compass/opencompass)
+under identical settings and parameters for both the edge0 models (int4 +
+trained adapters + prerouter routing) and the original fp16 base models.
+The loss of the edge0 pipeline is small: **3.9 points on average for
+edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:
+
+| Benchmark | edge0-35b (int4) | Qwen3.6-35B-A3B (fp16) | edge0-8b (int4) | Ling 3.0 tiny (fp16) |
+|---|---:|---:|---:|---:|
+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
+| **Average** | **79.2** | **83.2** | **69.9** | **72.7** |
+
+### Benchmark
+
+Measured with `python/examples/bench.py` (3.3k-token prompt prefill → 10
+sampled warmup steps → 200 timed sampled decode tokens, 2 runs per tier):
+
+| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |
+|---|---|---|---|---|
+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+
+*Cold = first request after process start (expert weights fault in from
+SSD); warm = subsequent requests (page cache resident). Prefill numbers
+are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*
-## Quick start
+Reproduce:
-### 1) Install
+```bash
+cd python
+python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL
+python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL
+```
+
+## 🚀 Getting Started
+
+### Python (macOS · Apple Silicon)
+
+#### Requirements
+
+- **OS / hardware**: the MLX backend runs on macOS with Apple Silicon
+ (M1/M2/M3/M4). The CUDA backend is on the roadmap — no other
+ platforms are supported by the Python framework yet.
+- **Python**: 3.10+ (3.12 recommended).
+- **MLX**: `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see
+ `python/pyproject.toml`). Garbled, mixed-language output on Apple A18 /
+ A18 Pro means an older `mlx`: `pip install 'mlx==0.30.6'
+ 'mlx-metal==0.30.6'` ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).
+- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for
+ `edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add
+ headroom for the OS, tokenizer, and long-context KV growth.
+- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB
+ (`edge0-8b`); expert weights are mmapped and read on demand, they are
+ not loaded into RAM up front.
+
+#### 1) Install
```bash
+cd python
# Python >= 3.10; the MLX backend requires macOS with Apple Silicon
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
```
-### 2) Download a model
+#### 2) Download a model
-The two tiers are published on Hugging Face and ModelScope — each repo bundles the
-base checkpoint and the trained LoRA + prerouter adapters in **one
-directory**, so a single download is a ready-to-run model:
+The two tiers are published on Hugging Face and ModelScope — each repo
+bundles the base checkpoint and the trained LoRA + prerouter adapters in
+**one directory**, so a single download is a ready-to-run model:
- [`Edge0/Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) (~23 GB) · [ModelScope mirror](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)
- [`Edge0/Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) (~4.2 GB) · [ModelScope mirror](https://www.modelscope.cn/models/Edge0/Edge0-8B-A1B-preview)
@@ -125,7 +195,7 @@ models/edge0-35b/
└── prerouter_edge0_35b.safetensors # trained prerouter heads
```
-### 3) Point edge0 at it
+#### 3) Point edge0 at it
Tier names resolve to local directories via environment variables
(where you put the download is up to you):
@@ -143,7 +213,7 @@ edge0 demo models/edge0-35b
edge0 serve models/edge0-8b
```
-### 4) Run
+#### 4) Run
```bash
# quick demo
@@ -165,7 +235,7 @@ edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."
`python -m edge0 ...` is equivalent to `edge0 ...`.
-### Python API
+#### Python API
```python
from edge0 import AutoEngine
@@ -182,10 +252,10 @@ print(engine._tok.decode(tokens))
engine.close() # release mmaps / expert cache
```
-`examples/demo.py` is the same minimal walkthrough (`edge0 demo` runs
-this exact path).
+`python/examples/demo.py` is the same minimal walkthrough (`edge0 demo`
+runs this exact path).
-### Models and adapters
+#### Models and adapters
- **Checkpoint**: the original model directory (`config.json`,
`model*.safetensors`, tokenizer). `edge0 serve <dir>` /
@@ -195,72 +265,111 @@ this exact path).
location automatically:
- the model directory (recommended): side by side with the base, e.g.
`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
- - `artifacts/` (repo root, gitignored): convert once from
- training-side npz exports via `edge0 convert-adapters --npz-dir ...`.
+ - `artifacts/` (gitignored): convert once from training-side npz
+ exports via `edge0 convert-adapters --npz-dir ...`.
- The published model repos bundle both the base checkpoint and the
current default adapter release, so `scripts/fetch_models.py` produces
- a ready-to-run model directory. Check each model's doc page for its
+ a ready-to-run model directory. Check each model's doc page for its
adapter provenance (training data, owner-layer layout).
- Both adapters are required for the prerouter + LoRA pipeline; if a
file is missing, `edge0` fails with a clear message (or pass
`--no-prerouter` / `--no-lora` to run the plain base model).
-## Quality
-All benchmarks were run by us with [OpenCompass](https://github.com/open-compass/opencompass)
-under identical settings and parameters for both the edge0 models (int4 +
-trained adapters + prerouter routing) and the original fp16 base models.
-The loss of the edge0 pipeline is small: **3.9 points on average for
-edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:
+#### Documentation
-| Benchmark | edge0-35b (int4) | Qwen3.6-35B-A3B (fp16) | edge0-8b (int4) | Ling 3.0 tiny (fp16) |
-|---|---:|---:|---:|---:|
-| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
-| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
-| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
-| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
-| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
-| **Average** | **79.2** | **83.2** | **69.9** | **72.7** |
+- [Architecture](docs/architecture.md)
+- [Attention](docs/attention.md) / [MoE](docs/moe.md) / [SSD streaming](docs/streaming.md) / [prerouter](docs/prerouter.md)
+- [Adding a model](docs/adding-a-model.md)
+- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)
+- Technical report: [The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063) ([PDF](paper/main.pdf))
-## Benchmark
+### macOS / iOS / Android
-Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled
-warmup steps → 200 timed sampled decode tokens, 2 runs per tier):
+The three platform engines are open-sourced in this repo — more details
+in each directory's README:
-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |
-|---|---|---|---|---|
-| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
-| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+- **macOS**: local CLI / daemon / desktop app (Rust) — see [`macos/README.md`](macos/README.md)
+- **iOS**: on-device iPhone app (Swift + MLX Swift) — see [`ios/README.md`](ios/README.md)
+- **Android**: on-device app + native engine (Kotlin) — see [`android/README.md`](android/README.md)
-*Cold = first request after process start (expert weights fault in from
-SSD); warm = subsequent requests (page cache resident). Prefill numbers
-are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*
+The **unified inference framework** — one access layer, runtime
+auto-adapting to iOS / macOS / Android / Windows / Python — arrives by
+the **end of October 2026**; see the [Roadmap](#-roadmap).
-Reproduce:
+## 🗺️ Roadmap
-```bash
-python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL
-python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL
-```
+- **[End of Oct 2026] edge0 unified inference framework** — we will
+ open-source a unified inference framework: **one unified access
+ layer** (a single API across chat / serve / on-device use), with the
+ **runtime automatically adapting to the hardware platform** — iOS,
+ macOS, Android, Windows and Python. It builds on the three platform
+ engines already open-sourced in this repo (`ios/` · `macos/` ·
+ `android/`).
+- **Windows** platform support (via the unified framework).
+- **CUDA backend** for the Python framework — reserved slot at
+ `python/src/edge0/backends/cuda/`, core code needs zero changes.
+- More model tiers and adapter releases on the existing pipeline.
+
+## 🤝 Contributing
+
+Contributions are welcome — issues, PRs, benchmark reports and model
+ports all count.
-## Tests
+**Python framework** (available now):
```bash
+cd python
+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
+
pytest # unit tests (no real weights)
EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q
- # real-weight generation; missing tiers are skipped
+ # real-weight generation; missing tiers are skipped
.venv/bin/python scripts/e2e_smoke.py \
--qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b
- # staged vs exact consistency + generation smoke
+ # staged vs exact consistency + generation smoke
scripts/generate_example.py # full-pipeline API example
-examples/demo.py # minimal API walkthrough
+examples/demo.py # minimal API walkthrough
```
-## Documentation
+CI runs unit tests (macOS + MLX) and a repo-hygiene suite (no hardcoded
+paths, backend-boundary and secret checks) on every PR.
-- [Architecture](docs/architecture.md)
-- [Attention](docs/attention.md) / [MoE](docs/moe.md) / [SSD streaming](docs/streaming.md) / [prerouter](docs/prerouter.md)
-- [Adding a model](docs/adding-a-model.md)
-- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)
+**Platform runtimes** (macOS / iOS / Android): the directories are
+reserved placeholders for now; contribution guides will land together
+with the unified framework code.
+
+Workflow: fork → feature branch → PR against `main`. Please keep the
+hygiene suite green and add tests for new behavior.
+
+## 📖 Citation
+
+If you find edge0 useful, please cite our technical report:
+
+```bibtex
+@article{lin2026other,
+ title = {The Other Half of the Memory Wall: Serving 35B MoEs from SSD
+ with Trained Routing Prediction},
+ author = {Lin, Yu and Wang, Yiming and Cai, Runyuan and Liu, Hanze and
+ Zeng, Xiaodong},
+ journal = {arXiv preprint arXiv:2609.18063},
+ year = {2026},
+ url = {https://arxiv.org/abs/2609.18063}
+}
+```
+
+## 📧 Contact Us
+
+<!-- TODO(edge0 team): fill in the official contact channels before publish. -->
+
+Community and support channels are coming soon — this section will list
+the official ways to reach us:
+
+- **Email**: _to be added_
+- **Discord**: _to be added_
+- **WeChat group**: _to be added_
+
+For bugs and feature requests, please use
+[GitHub Issues](https://github.com/Edge0-AI/Edge0/issues).
## License
diff --git a/README_zh.md b/README_zh.md
index 09f0d5e..6631ef3 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -6,6 +6,8 @@
**开源流式 MoE 推理框架 —— SSD 专家 offload + Recover-LoRA + prerouter 路由预判**
+**Python** · **macOS** · **iOS** · **Android** —— 一套配方,全端落地。
+
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Edge0--35B--A3B--preview-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Edge0--8B--A1B--preview-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)
[![ModelScope](https://img.shields.io/badge/ModelScope-Edge0--35B--A3B--preview-624AFF?style=for-the-badge)](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)
@@ -18,9 +20,53 @@
</div>
+## 📰 新闻
+
+- **【2026-09-30】** 我们发布了**三端推理引擎**,适配 iOS、macOS、Android
+ 平台,让用户在不同架构、不同平台上都有最佳的推理体验。三端源码已
+ 开源至本仓库([`ios/`](ios/) · [`macos/`](macos/) ·
+ [`android/`](android/)),更多细节见各目录 README;**统一推理框架**
+ 将于 **2026 年 10 月底**发布,详见[路线图](#-路线图)。
+- **【2026-09-20】** `edge0 demo / chat / serve` 新增 `--prefill-ondemand`
+ ([#112](https://github.com/Edge0-AI/Edge0/pull/112))。
+- **【2026-09-16】** 技术报告上线 arXiv:[The Other Half of the Memory
+ Wall: Serving 35B MoEs from SSD with Trained Routing
+ Prediction](https://arxiv.org/abs/2609.18063)。
+- **【2026-09-08】** **edge0** 首次开源发布,两个模型档位
+ [`Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)
+ 与 [`Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)
+ 同步登陆 Hugging Face 与 ModelScope。
+
+## 🤔 关于 edge0
+
**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload +
-Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,
-当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)即将接入。
+Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架,让大型稀疏
+MoE 模型跑在消费级硬件上——峰值内存由**激活**专家集而非参数量决定。
+
+### 核心机制
+
+- **SSD 专家 offload**:专家权重按需从存储流式加载,峰值内存由
+ 激活集而非参数量决定;
+- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠
+ 而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由
+ 宽度 *K* 增大;
+- **Recover-LoRA**:冻结 int4 基模,用 FP teacher 蒸馏训练 LoRA,
+ 在 4bit 下恢复绝大部分量化损失(见[质量](#质量))。适配器不合并,
+ 一份只读基模服务多套适配器。
+
+### 平台
+
+一个仓库、一套配方、各平台独立 runtime:
+
+| 平台 | 目录 | 技术栈 | 状态 |
+|---|---|---|---|
+| **Python**(macOS · Apple Silicon) | [`python/`](python/) | Python + MLX | ✅ 现已可用 |
+| **macOS** 桌面 App 与 CLI | [`macos/`](macos/) | Rust | ✅ 已开源(2026-09-30) |
+| **iOS** App | [`ios/`](ios/) | Swift + MLX Swift | ✅ 已开源(2026-09-30) |
+| **Android** App 与引擎 | [`android/`](android/) | Kotlin + 原生引擎 | ✅ 已开源(2026-09-30) |
+| **Windows** | — | — | 🗺️ 路线图中 |
+
+### 模型
框架随附两个模型档位。每个档位是一个端到端发布:发布的 checkpoint、
训练好的 LoRA 适配器与训练好的 prerouter 头作为整体协同工作。
@@ -35,13 +81,71 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后
适配器文件与 checkpoint 同目录、自动加载,`edge0 serve <tier>` 开箱即跑
训练好的完整管线。
-## 环境要求
+### 设计
+
+- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`
+ 按模型名自动选类;
+- **后端隔离(Python 框架)**:在 Python 框架内,全部 MLX 代码收在
+ `python/src/edge0/backends/mlx/`,核心逻辑(模型 spec / prerouter /
+ 流式专家池 / server)只依赖后端门面(`backends/base.py` 的 `core` /
+ `nn` 门面),新增后端实现同一门面即可平级接入(`backends/cuda/`
+ 预留插槽),核心代码零改动。iOS / macOS / Android 引擎目前是各自的
+ 平台原生技术栈——把所有平台收进统一接入层,正是统一推理框架
+ (见[路线图](#-路线图))要交付的内容;
+- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据
+ (来源、版本、owner 层)的 `.safetensors`,放模型目录或 `artifacts/`
+ 均可自动解析;
+- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /
+ `model*.safetensors` / tokenizer)和该模型的适配器,升级适配器只换
+ 适配器文件,基模不动、不 merge。
+
+### 质量
+
+全部评测由我们使用 [OpenCompass](https://github.com/open-compass/opencompass)、
+在完全相同的设置与参数下对 edge0 模型(int4 + 训练适配器 + prerouter 路由)
+与原 fp16 基座模型测得。edge0 管线的损失很小:**edge0-35b 平均仅落后
+3.9 分、edge0-8b 落后 2.8 分**(MMLU-Pro 甚至反超基座)。满分 100:
+
+| 评测集 | edge0-35b(int4) | Qwen3.6-35B-A3B(fp16) | edge0-8b(int4) | Ling 3.0 tiny(fp16) |
+|---|---:|---:|---:|---:|
+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
+| **平均** | **79.2** | **83.2** | **69.9** | **72.7** |
+
+### 性能实测
+
+`python/examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样
+warmup → 200 token 计时段,每档 2 轮):
+
+| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 |
+|---|---|---|---|---|
+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+
+*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。*
+
+复现:
+
+```bash
+cd python
+python examples/bench.py edge0-35b # 经 $EDGE0_35B_MODEL
+python examples/bench.py edge0-8b # 经 $EDGE0_8B_MODEL
+```
+
+## 🚀 快速开始
+
+### Python(macOS · Apple Silicon)
+
+#### 环境要求
- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS
- (M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。
+ (M1/M2/M3/M4);CUDA 后端在路线图中,Python 框架暂不支持其他平台。
- **Python**:3.10+(推荐 3.12)。
- **MLX**:`mlx==0.30.6` / `mlx-metal==0.30.6`(`mlx-lm==0.31.0`,见
- `pyproject.toml`)。Apple A18 / A18 Pro 上输出乱码 = mlx 版本旧:
+ `python/pyproject.toml`)。Apple A18 / A18 Pro 上输出乱码 = mlx 版本旧:
`pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`
([#8](https://github.com/Edge0-AI/Edge0/issues/8))。
- **内存**:短上下文下 `edge0-35b` ≈2.9 GB、`edge0-8b` ≈1.0 GB
@@ -50,45 +154,19 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后
- **磁盘**:4bit checkpoint 约 23 GB(`edge0-35b`)/ 4.2 GB
(`edge0-8b`);专家权重 mmap 按需读取,不一次性载入内存。
-## 设计
-
-- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`
- 按模型名自动选类;
-- **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,核心逻辑
- (模型 spec / prerouter / 流式专家池 / server)只依赖后端门面
- (`edge0/backends/base.py` 的 `core` / `nn` 门面),新增后端实现同一门面
- 即可平级接入(`backends/cuda/` 预留插槽),核心代码零改动;
-- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据
- (来源、版本、owner 层)的 `.safetensors`,放模型目录或 `artifacts/`
- 均可自动解析;
-- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /
- `model*.safetensors` / tokenizer)和该模型的适配器,升级适配器只换适配器文件
- 文件,基模不动、不 merge。
-
-## 核心机制
-
-- **SSD 专家 offload**:专家权重按需从存储流式加载,峰值内存由
- 激活集而非参数量决定;
-- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠
- 而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由
- 宽度 *K* 增大;
-- **Recover-LoRA**:冻结 int4 基模,用 FP teacher 蒸馏训练 LoRA,
- 在 4bit 下恢复绝大部分量化损失(见[质量](#质量))。适配器不合并,
- 一份只读基模服务多套适配器。
-
-## 快速开始
-
-### 1) 安装
+#### 1) 安装
```bash
+cd python
# Python ≥3.10;MLX 后端需 macOS + Apple Silicon
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
```
-### 2) 下载模型
+#### 2) 下载模型
-两个档位发布在 Hugging Face 与 ModelScope——每个仓库把基模 checkpoint 与训练好的
-LoRA + prerouter 适配器打包在**同一目录**,一次下载即为可运行的模型:
+两个档位发布在 Hugging Face 与 ModelScope——每个仓库把基模 checkpoint
+与训练好的 LoRA + prerouter 适配器打包在**同一目录**,一次下载即为
+可运行的模型:
- [`Edge0/Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)(约 23 GB) · [ModelScope 镜像](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)
- [`Edge0/Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)(约 4.2 GB) · [ModelScope 镜像](https://www.modelscope.cn/models/Edge0/Edge0-8B-A1B-preview)
@@ -114,7 +192,7 @@ models/edge0-35b/
└── prerouter_edge0_35b.safetensors # 训练好的 prerouter 头
```
-### 3) 指向模型目录
+#### 3) 指向模型目录
档位名经环境变量解析到本地目录(放哪由你决定):
@@ -131,7 +209,7 @@ edge0 demo models/edge0-35b
edge0 serve models/edge0-8b
```
-### 4) 运行
+#### 4) 运行
```bash
# 快速演示
@@ -152,7 +230,7 @@ edge0 chat edge0-35b --prompt "用一句话解释流式推理。"
`python -m edge0 ...` 等价于 `edge0 ...`。
-### Python API
+#### Python API
```python
from edge0 import AutoEngine
@@ -169,9 +247,9 @@ print(engine._tok.decode(tokens))
engine.close() # 释放 mmap / 专家缓存
```
-`examples/demo.py` 就是这条最小路径(`edge0 demo` 内部等价运行)。
+`python/examples/demo.py` 就是这条最小路径(`edge0 demo` 内部等价运行)。
-### 模型与适配器
+#### 模型与适配器
- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、
tokenizer)。`edge0 serve <dir>` / `AutoEngine.from_pretrained(<dir>)`
@@ -179,68 +257,104 @@ engine.close() # 释放 mmap / 专家缓存
- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:
- **模型目录内**(推荐):与基模同目录,如
`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
- - `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从
- 训练侧 npz 一次性转换。
+ - `artifacts/`(gitignored):`edge0 convert-adapters` 从训练侧 npz
+ 一次性转换。
- 发布的模型仓库同时包含基模与当前默认适配器版本,
`scripts/fetch_models.py` 下载后即为可运行的模型目录。适配器来源
(训练数据、owner 层分布)见各模型文档页。
- prerouter + LoRA 两条适配器都必需;缺文件时 `edge0` 会给出明确报错
(也可加 `--no-prerouter` / `--no-lora` 直接跑裸基模)。
-## 质量
+#### 文档
-全部评测由我们使用 [OpenCompass](https://github.com/open-compass/opencompass)、
-在完全相同的设置与参数下对 edge0 模型(int4 + 训练适配器 + prerouter 路由)
-与原 fp16 基座模型测得。edge0 管线的损失很小:**edge0-35b 平均仅落后
-3.9 分、edge0-8b 落后 2.8 分**(MMLU-Pro 甚至反超基座)。满分 100:
+- [架构总览](docs/architecture.md)
+- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)
+- [如何接入新模型](docs/adding-a-model.md)
+- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)
+- 技术报告:[The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063)([PDF](paper/main.pdf))
-| 评测集 | edge0-35b(int4) | Qwen3.6-35B-A3B(fp16) | edge0-8b(int4) | Ling 3.0 tiny(fp16) |
-|---|---:|---:|---:|---:|
-| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
-| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
-| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
-| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
-| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
-| **平均** | **79.2** | **83.2** | **69.9** | **72.7** |
+### macOS / iOS / Android
-## 性能实测
+三端推理引擎已开源在本仓库,更多细节见各目录 README:
-`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →
-200 token 计时段,每档 2 轮):
+- **macOS**:本地 CLI / daemon / 桌面 App(Rust)—— 详见
+ [`macos/README.md`](macos/README.md)
+- **iOS**:iPhone 端侧 App(Swift + MLX Swift)—— 详见
+ [`ios/README.md`](ios/README.md)
+- **Android**:端侧 App + 原生引擎(Kotlin)—— 详见
+ [`android/README.md`](android/README.md)
-| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 |
-|---|---|---|---|---|
-| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |
-| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+**统一推理框架**(接入层统一,runtime 自动适配 iOS / macOS / Android /
+Windows / Python)将于 **2026 年 10 月底**发布,见[路线图](#-路线图)。
-*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。
+## 🗺️ 路线图
-复现:
+- **【2026 年 10 月底】edge0 统一推理框架开源** —— 发布统一推理框架:
+ **接入层统一**为一套 API(chat / serve / 端侧共用),**runtime 自动
+ 适配不同硬件平台**——iOS、macOS、Android、Windows、Python;在本仓库
+ 已开源的三端引擎(`ios/` · `macos/` · `android/`)之上统一收编。
+- **Windows** 平台支持(经统一框架)。
+- **CUDA 后端**(Python 框架)—— 插槽已预留在
+ `python/src/edge0/backends/cuda/`,核心代码零改动。
+- 更多模型档位与适配器版本在现有管线上持续发布。
-```bash
-python examples/bench.py edge0-35b # 经 $EDGE0_35B_MODEL
-python examples/bench.py edge0-8b # 经 $EDGE0_8B_MODEL
-```
+## 🤝 贡献
-## 验证
+欢迎各种形式的贡献——issue、PR、性能实测报告、新模型接入都算。
+
+**Python 框架**(现已可用):
```bash
+cd python
+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
+
pytest # 单元测试(不含真实权重)
EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q
- # 真实权重生成测试;缺少的档位会明确 skip
+ # 真实权重生成测试;缺少的档位会明确 skip
.venv/bin/python scripts/e2e_smoke.py \
--qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b
- # staged vs exact 一致性 + 生成冒烟
+ # staged vs exact 一致性 + 生成冒烟
scripts/generate_example.py # 完整 API 上手例子(prefill→生成→解码全链路)
-examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)
+examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)
```
-## 文档
+每个 PR 都会跑 CI:单元测试(macOS + MLX)与仓库卫生检查(无硬编码
+路径、后端边界、无密钥)。
-- [架构总览](docs/architecture.md)
-- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)
-- [如何接入新模型](docs/adding-a-model.md)
-- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)
+**平台 runtime**(macOS / iOS / Android):目前是占位目录,贡献指南将
+随统一框架代码一并发布。
+
+流程:fork → 功能分支 → 向 `main` 发 PR。请保持卫生检查通过,并为新
+行为补测试。
+
+## 📖 引用
+
+如果 edge0 对你有帮助,请引用我们的技术报告:
+
+```bibtex
+@article{lin2026other,
+ title = {The Other Half of the Memory Wall: Serving 35B MoEs from SSD
+ with Trained Routing Prediction},
+ author = {Lin, Yu and Wang, Yiming and Cai, Runyuan and Liu, Hanze and
+ Zeng, Xiaodong},
+ journal = {arXiv preprint arXiv:2609.18063},
+ year = {2026},
+ url = {https://arxiv.org/abs/2609.18063}
+}
+```
+
+## 📧 联系我们
+
+<!-- TODO(edge0 团队):发布前填写官方联系渠道。 -->
+
+社区与支持渠道即将上线,本节将列出联系我们的官方方式:
+
+- **邮箱**:_待补充_
+- **Discord**:_待补充_
+- **微信群**:_待补充_
+
+Bug 与功能建议请直接提
+[GitHub Issues](https://github.com/Edge0-AI/Edge0/issues)。
## License