Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs(readme): restructure EN/ZH READMEs in sglang/vllm style
Bincafd81be4e24
diff --git a/README.md b/README.mdindex 5e0240a..110e96c 100644--- a/README.md+++ b/README.md@@ -6,6 +6,8 @@**An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction.**+**Python** · **macOS** · **iOS** · **Android** — one recipe, every device.+[](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)[](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)[](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)@@ -18,12 +20,48 @@ English | [中文](README_zh.md)</div>+## 📰 News++- **[2026-09-30]** We released the **edge0 inference engines for three platforms — iOS, macOS and Android** — so users get the best inference experience across architectures and platforms. The source is open-sourced in this repo ([`ios/`](ios/) · [`macos/`](macos/) · [`android/`](android/)) — see each directory's README for details. The **unified inference framework** follows by the **end of October 2026**; see the [Roadmap](#-roadmap).+- **[2026-09-20]** `--prefill-ondemand` lands for `edge0 demo / chat / serve` ([#112](https://github.com/Edge0-AI/Edge0/pull/112)).+- **[2026-09-16]** Our technical report is on arXiv: [The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction](https://arxiv.org/abs/2609.18063).+- **[2026-09-08]** Initial open-source release of **edge0**, together with both model tiers — [`Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) and [`Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) — on Hugging Face and ModelScope.++## 🤔 About+**edge0** is an open-source streaming MoE inference framework. Itgeneralizes the production-proven recipe — **SSD expert offload +Recover-LoRA + prerouter routing prediction** — into an extensible-framework. The backend is isolated by design: the current MLX backend-runs on Apple Silicon, and additional platforms (CUDA, …) plug into the-same core abstractions.+framework that runs large sparse-MoE models on consumer hardware: peak+memory is bounded by the *active* expert set, not the parameter count.++### Core mechanisms++- **SSD expert offload**: expert weights are streamed from storage on+ demand; peak memory is bounded by the active set, not the parameter+ count.+- **Prerouter**: a trained head predicts expert routing one step+ ahead, so expert loads overlap the forward pass instead of stalling+ it — **up to +59%** decode throughput; the gain grows with storage+ latency, model size, and routed width *K*.+- **Recover-LoRA**: the int4 base is frozen and LoRA adapters are+ trained by distillation from the FP teacher, recovering most of the+ quantization loss at 4-bit (see [Quality](#quality)). Adapters stay+ unmerged: one read-only base serves multiple adapter sets.++### Platforms++One repo, one recipe, per-platform runtimes:++| Platform | Directory | Stack | Status |+|---|---|---|---|+| **Python** (macOS · Apple Silicon) | [`python/`](python/) | Python + MLX | ✅ Available now |+| **macOS** app & CLI | [`macos/`](macos/) | Rust | ✅ Open-sourced (2026-09-30) |+| **iOS** app | [`ios/`](ios/) | Swift + MLX Swift | ✅ Open-sourced (2026-09-30) |+| **Android** app & engine | [`android/`](android/) | Kotlin + native engine | ✅ Open-sourced (2026-09-30) |+| **Windows** | — | — | 🗺️ On the roadmap |++### ModelsTwo model tiers ship with the framework. Each tier is an end-to-endrelease: the released checkpoint, the trained LoRA adapters, and the@@ -40,32 +78,19 @@ LoRA and prerouter training done for this framework — the adapter filesare co-located with each checkpoint and load automatically, so`edge0 serve <tier>` runs the trained pipeline out of the box.-## Requirements--- **OS / hardware**: the MLX backend runs on macOS with Apple Silicon- (M1/M2/M3/M4). The CUDA backend is on the roadmap — no other- platforms are supported yet.-- **Python**: 3.10+ (3.12 recommended).-- **MLX**: `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see- `pyproject.toml`). Garbled, mixed-language output on Apple A18 / A18 Pro- means an older `mlx`: `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`- ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).-- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for- `edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add- headroom for the OS, tokenizer, and long-context KV growth.-- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB- (`edge0-8b`); expert weights are mmapped and read on demand, they are- not loaded into RAM up front.--## Design+### Design- **transformers-style usage**: `AutoModel` / `AutoConfig` / `AutoEngine`resolve the tier from the model name;-- **Backend isolation**: all MLX code lives under `edge0/backends/mlx/`;- the core logic (model specs, prerouter, streaming expert pool, server)- depends only on the backend facade (`edge0/backends/base.py`), so a new- backend implements the same facade (`backends/cuda/` is a reserved- slot) with zero changes to core code;+- **Backend isolation (Python framework)**: within the Python framework,+ all MLX code lives under `python/src/edge0/backends/mlx/`; the core+ logic (model specs, prerouter, streaming expert pool, server) depends+ only on the backend facade (`backends/base.py`), so a new backend+ implements the same facade (`backends/cuda/` is a reserved slot) with+ zero changes to core code. The iOS / macOS / Android engines ship+ platform-native stacks today — bringing every platform under one+ access layer is exactly what the unified inference framework+ (see [Roadmap](#-roadmap)) will deliver;- **Adapters as safetensors**: LoRA and prerouter weights are`.safetensors` files with provenance metadata (source, version, ownerlayers), resolved from the model directory or `artifacts/`;@@ -74,34 +99,79 @@ are co-located with each checkpoint and load automatically, soand that model's adapters; upgrading adapters swaps adapterfiles only — the base stays read-only and is never merged.-## Core mechanisms+### Quality-- **SSD expert offload**: expert weights are streamed from storage on- demand; peak memory is bounded by the active set, not the parameter- count.-- **Prerouter**: a trained head predicts expert routing one step- ahead, so expert loads overlap the forward pass instead of stalling- it — **up to +59%** decode throughput; the gain grows with storage- latency, model size, and routed width *K*.-- **Recover-LoRA**: the int4 base is frozen and LoRA adapters are- trained by distillation from the FP teacher, recovering most of the- quantization loss at 4-bit (see [Quality](#quality)). Adapters stay- unmerged: one read-only base serves multiple adapter sets.+All benchmarks were run by us with [OpenCompass](https://github.com/open-compass/opencompass)+under identical settings and parameters for both the edge0 models (int4 ++trained adapters + prerouter routing) and the original fp16 base models.+The loss of the edge0 pipeline is small: **3.9 points on average for+edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:++| Benchmark | edge0-35b (int4) | Qwen3.6-35B-A3B (fp16) | edge0-8b (int4) | Ling 3.0 tiny (fp16) |+|---|---:|---:|---:|---:|+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |+| **Average** | **79.2** | **83.2** | **69.9** | **72.7** |++### Benchmark++Measured with `python/examples/bench.py` (3.3k-token prompt prefill → 10+sampled warmup steps → 200 timed sampled decode tokens, 2 runs per tier):++| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |+|---|---|---|---|---|+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |++*Cold = first request after process start (expert weights fault in from+SSD); warm = subsequent requests (page cache resident). Prefill numbers+are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*-## Quick start+Reproduce:-### 1) Install+```bash+cd python+python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL+python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL+```++## 🚀 Getting Started++### Python (macOS · Apple Silicon)++#### Requirements++- **OS / hardware**: the MLX backend runs on macOS with Apple Silicon+ (M1/M2/M3/M4). The CUDA backend is on the roadmap — no other+ platforms are supported by the Python framework yet.+- **Python**: 3.10+ (3.12 recommended).+- **MLX**: `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see+ `python/pyproject.toml`). Garbled, mixed-language output on Apple A18 /+ A18 Pro means an older `mlx`: `pip install 'mlx==0.30.6'+ 'mlx-metal==0.30.6'` ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).+- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for+ `edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add+ headroom for the OS, tokenizer, and long-context KV growth.+- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB+ (`edge0-8b`); expert weights are mmapped and read on demand, they are+ not loaded into RAM up front.++#### 1) Install```bash+cd python# Python >= 3.10; the MLX backend requires macOS with Apple Siliconpython3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'```-### 2) Download a model+#### 2) Download a model-The two tiers are published on Hugging Face and ModelScope — each repo bundles the-base checkpoint and the trained LoRA + prerouter adapters in **one-directory**, so a single download is a ready-to-run model:+The two tiers are published on Hugging Face and ModelScope — each repo+bundles the base checkpoint and the trained LoRA + prerouter adapters in+**one directory**, so a single download is a ready-to-run model:- [`Edge0/Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) (~23 GB) · [ModelScope mirror](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)- [`Edge0/Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) (~4.2 GB) · [ModelScope mirror](https://www.modelscope.cn/models/Edge0/Edge0-8B-A1B-preview)@@ -125,7 +195,7 @@ models/edge0-35b/└── prerouter_edge0_35b.safetensors # trained prerouter heads```-### 3) Point edge0 at it+#### 3) Point edge0 at itTier names resolve to local directories via environment variables(where you put the download is up to you):@@ -143,7 +213,7 @@ edge0 demo models/edge0-35bedge0 serve models/edge0-8b```-### 4) Run+#### 4) Run```bash# quick demo@@ -165,7 +235,7 @@ edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."`python -m edge0 ...` is equivalent to `edge0 ...`.-### Python API+#### Python API```pythonfrom edge0 import AutoEngine@@ -182,10 +252,10 @@ print(engine._tok.decode(tokens))engine.close() # release mmaps / expert cache```-`examples/demo.py` is the same minimal walkthrough (`edge0 demo` runs-this exact path).+`python/examples/demo.py` is the same minimal walkthrough (`edge0 demo`+runs this exact path).-### Models and adapters+#### Models and adapters- **Checkpoint**: the original model directory (`config.json`,`model*.safetensors`, tokenizer). `edge0 serve <dir>` /@@ -195,72 +265,111 @@ this exact path).location automatically:- the model directory (recommended): side by side with the base, e.g.`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- - `artifacts/` (repo root, gitignored): convert once from- training-side npz exports via `edge0 convert-adapters --npz-dir ...`.+ - `artifacts/` (gitignored): convert once from training-side npz+ exports via `edge0 convert-adapters --npz-dir ...`.- The published model repos bundle both the base checkpoint and thecurrent default adapter release, so `scripts/fetch_models.py` produces- a ready-to-run model directory. Check each model's doc page for its+ a ready-to-run model directory. Check each model's doc page for itsadapter provenance (training data, owner-layer layout).- Both adapters are required for the prerouter + LoRA pipeline; if afile is missing, `edge0` fails with a clear message (or pass`--no-prerouter` / `--no-lora` to run the plain base model).-## Quality-All benchmarks were run by us with [OpenCompass](https://github.com/open-compass/opencompass)-under identical settings and parameters for both the edge0 models (int4 +-trained adapters + prerouter routing) and the original fp16 base models.-The loss of the edge0 pipeline is small: **3.9 points on average for-edge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:+#### Documentation-| Benchmark | edge0-35b (int4) | Qwen3.6-35B-A3B (fp16) | edge0-8b (int4) | Ling 3.0 tiny (fp16) |-|---|---:|---:|---:|---:|-| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |-| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |-| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |-| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |-| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |-| **Average** | **79.2** | **83.2** | **69.9** | **72.7** |+- [Architecture](docs/architecture.md)+- [Attention](docs/attention.md) / [MoE](docs/moe.md) / [SSD streaming](docs/streaming.md) / [prerouter](docs/prerouter.md)+- [Adding a model](docs/adding-a-model.md)+- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)+- Technical report: [The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063) ([PDF](paper/main.pdf))-## Benchmark+### macOS / iOS / Android-Measured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled-warmup steps → 200 timed sampled decode tokens, 2 runs per tier):+The three platform engines are open-sourced in this repo — more details+in each directory's README:-| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory | Test machine |-|---|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |+- **macOS**: local CLI / daemon / desktop app (Rust) — see [`macos/README.md`](macos/README.md)+- **iOS**: on-device iPhone app (Swift + MLX Swift) — see [`ios/README.md`](ios/README.md)+- **Android**: on-device app + native engine (Kotlin) — see [`android/README.md`](android/README.md)-*Cold = first request after process start (expert weights fault in from-SSD); warm = subsequent requests (page cache resident). Prefill numbers-are throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*+The **unified inference framework** — one access layer, runtime+auto-adapting to iOS / macOS / Android / Windows / Python — arrives by+the **end of October 2026**; see the [Roadmap](#-roadmap).-Reproduce:+## 🗺️ Roadmap-```bash-python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL-python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL-```+- **[End of Oct 2026] edge0 unified inference framework** — we will+ open-source a unified inference framework: **one unified access+ layer** (a single API across chat / serve / on-device use), with the+ **runtime automatically adapting to the hardware platform** — iOS,+ macOS, Android, Windows and Python. It builds on the three platform+ engines already open-sourced in this repo (`ios/` · `macos/` ·+ `android/`).+- **Windows** platform support (via the unified framework).+- **CUDA backend** for the Python framework — reserved slot at+ `python/src/edge0/backends/cuda/`, core code needs zero changes.+- More model tiers and adapter releases on the existing pipeline.++## 🤝 Contributing++Contributions are welcome — issues, PRs, benchmark reports and model+ports all count.-## Tests+**Python framework** (available now):```bash+cd python+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'+pytest # unit tests (no real weights)EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q- # real-weight generation; missing tiers are skipped+ # real-weight generation; missing tiers are skipped.venv/bin/python scripts/e2e_smoke.py \--qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b- # staged vs exact consistency + generation smoke+ # staged vs exact consistency + generation smokescripts/generate_example.py # full-pipeline API example-examples/demo.py # minimal API walkthrough+examples/demo.py # minimal API walkthrough```-## Documentation+CI runs unit tests (macOS + MLX) and a repo-hygiene suite (no hardcoded+paths, backend-boundary and secret checks) on every PR.-- [Architecture](docs/architecture.md)-- [Attention](docs/attention.md) / [MoE](docs/moe.md) / [SSD streaming](docs/streaming.md) / [prerouter](docs/prerouter.md)-- [Adding a model](docs/adding-a-model.md)-- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)+**Platform runtimes** (macOS / iOS / Android): the directories are+reserved placeholders for now; contribution guides will land together+with the unified framework code.++Workflow: fork → feature branch → PR against `main`. Please keep the+hygiene suite green and add tests for new behavior.++## 📖 Citation++If you find edge0 useful, please cite our technical report:++```bibtex+@article{lin2026other,+ title = {The Other Half of the Memory Wall: Serving 35B MoEs from SSD+ with Trained Routing Prediction},+ author = {Lin, Yu and Wang, Yiming and Cai, Runyuan and Liu, Hanze and+ Zeng, Xiaodong},+ journal = {arXiv preprint arXiv:2609.18063},+ year = {2026},+ url = {https://arxiv.org/abs/2609.18063}+}+```++## 📧 Contact Us++<!-- TODO(edge0 team): fill in the official contact channels before publish. -->++Community and support channels are coming soon — this section will list+the official ways to reach us:++- **Email**: _to be added_+- **Discord**: _to be added_+- **WeChat group**: _to be added_++For bugs and feature requests, please use+[GitHub Issues](https://github.com/Edge0-AI/Edge0/issues).## Licensediff --git a/README_zh.md b/README_zh.mdindex 09f0d5e..6631ef3 100644--- a/README_zh.md+++ b/README_zh.md@@ -6,6 +6,8 @@**开源流式 MoE 推理框架 —— SSD 专家 offload + Recover-LoRA + prerouter 路由预判**+**Python** · **macOS** · **iOS** · **Android** —— 一套配方,全端落地。+[](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)[](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)[](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)@@ -18,9 +20,53 @@</div>+## 📰 新闻++- **【2026-09-30】** 我们发布了**三端推理引擎**,适配 iOS、macOS、Android+ 平台,让用户在不同架构、不同平台上都有最佳的推理体验。三端源码已+ 开源至本仓库([`ios/`](ios/) · [`macos/`](macos/) ·+ [`android/`](android/)),更多细节见各目录 README;**统一推理框架**+ 将于 **2026 年 10 月底**发布,详见[路线图](#-路线图)。+- **【2026-09-20】** `edge0 demo / chat / serve` 新增 `--prefill-ondemand`+ ([#112](https://github.com/Edge0-AI/Edge0/pull/112))。+- **【2026-09-16】** 技术报告上线 arXiv:[The Other Half of the Memory+ Wall: Serving 35B MoEs from SSD with Trained Routing+ Prediction](https://arxiv.org/abs/2609.18063)。+- **【2026-09-08】** **edge0** 首次开源发布,两个模型档位+ [`Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)+ 与 [`Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)+ 同步登陆 Hugging Face 与 ModelScope。++## 🤔 关于 edge0+**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload +-Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,-当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)即将接入。+Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架,让大型稀疏+MoE 模型跑在消费级硬件上——峰值内存由**激活**专家集而非参数量决定。++### 核心机制++- **SSD 专家 offload**:专家权重按需从存储流式加载,峰值内存由+ 激活集而非参数量决定;+- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠+ 而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由+ 宽度 *K* 增大;+- **Recover-LoRA**:冻结 int4 基模,用 FP teacher 蒸馏训练 LoRA,+ 在 4bit 下恢复绝大部分量化损失(见[质量](#质量))。适配器不合并,+ 一份只读基模服务多套适配器。++### 平台++一个仓库、一套配方、各平台独立 runtime:++| 平台 | 目录 | 技术栈 | 状态 |+|---|---|---|---|+| **Python**(macOS · Apple Silicon) | [`python/`](python/) | Python + MLX | ✅ 现已可用 |+| **macOS** 桌面 App 与 CLI | [`macos/`](macos/) | Rust | ✅ 已开源(2026-09-30) |+| **iOS** App | [`ios/`](ios/) | Swift + MLX Swift | ✅ 已开源(2026-09-30) |+| **Android** App 与引擎 | [`android/`](android/) | Kotlin + 原生引擎 | ✅ 已开源(2026-09-30) |+| **Windows** | — | — | 🗺️ 路线图中 |++### 模型框架随附两个模型档位。每个档位是一个端到端发布:发布的 checkpoint、训练好的 LoRA 适配器与训练好的 prerouter 头作为整体协同工作。@@ -35,13 +81,71 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后适配器文件与 checkpoint 同目录、自动加载,`edge0 serve <tier>` 开箱即跑训练好的完整管线。-## 环境要求+### 设计++- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`+ 按模型名自动选类;+- **后端隔离(Python 框架)**:在 Python 框架内,全部 MLX 代码收在+ `python/src/edge0/backends/mlx/`,核心逻辑(模型 spec / prerouter /+ 流式专家池 / server)只依赖后端门面(`backends/base.py` 的 `core` /+ `nn` 门面),新增后端实现同一门面即可平级接入(`backends/cuda/`+ 预留插槽),核心代码零改动。iOS / macOS / Android 引擎目前是各自的+ 平台原生技术栈——把所有平台收进统一接入层,正是统一推理框架+ (见[路线图](#-路线图))要交付的内容;+- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据+ (来源、版本、owner 层)的 `.safetensors`,放模型目录或 `artifacts/`+ 均可自动解析;+- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /+ `model*.safetensors` / tokenizer)和该模型的适配器,升级适配器只换+ 适配器文件,基模不动、不 merge。++### 质量++全部评测由我们使用 [OpenCompass](https://github.com/open-compass/opencompass)、+在完全相同的设置与参数下对 edge0 模型(int4 + 训练适配器 + prerouter 路由)+与原 fp16 基座模型测得。edge0 管线的损失很小:**edge0-35b 平均仅落后+3.9 分、edge0-8b 落后 2.8 分**(MMLU-Pro 甚至反超基座)。满分 100:++| 评测集 | edge0-35b(int4) | Qwen3.6-35B-A3B(fp16) | edge0-8b(int4) | Ling 3.0 tiny(fp16) |+|---|---:|---:|---:|---:|+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |+| **平均** | **79.2** | **83.2** | **69.9** | **72.7** |++### 性能实测++`python/examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样+warmup → 200 token 计时段,每档 2 轮):++| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 |+|---|---|---|---|---|+| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |++*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。*++复现:++```bash+cd python+python examples/bench.py edge0-35b # 经 $EDGE0_35B_MODEL+python examples/bench.py edge0-8b # 经 $EDGE0_8B_MODEL+```++## 🚀 快速开始++### Python(macOS · Apple Silicon)++#### 环境要求- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS- (M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。+ (M1/M2/M3/M4);CUDA 后端在路线图中,Python 框架暂不支持其他平台。- **Python**:3.10+(推荐 3.12)。- **MLX**:`mlx==0.30.6` / `mlx-metal==0.30.6`(`mlx-lm==0.31.0`,见- `pyproject.toml`)。Apple A18 / A18 Pro 上输出乱码 = mlx 版本旧:+ `python/pyproject.toml`)。Apple A18 / A18 Pro 上输出乱码 = mlx 版本旧:`pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`([#8](https://github.com/Edge0-AI/Edge0/issues/8))。- **内存**:短上下文下 `edge0-35b` ≈2.9 GB、`edge0-8b` ≈1.0 GB@@ -50,45 +154,19 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后- **磁盘**:4bit checkpoint 约 23 GB(`edge0-35b`)/ 4.2 GB(`edge0-8b`);专家权重 mmap 按需读取,不一次性载入内存。-## 设计--- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`- 按模型名自动选类;-- **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,核心逻辑- (模型 spec / prerouter / 流式专家池 / server)只依赖后端门面- (`edge0/backends/base.py` 的 `core` / `nn` 门面),新增后端实现同一门面- 即可平级接入(`backends/cuda/` 预留插槽),核心代码零改动;-- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据- (来源、版本、owner 层)的 `.safetensors`,放模型目录或 `artifacts/`- 均可自动解析;-- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /- `model*.safetensors` / tokenizer)和该模型的适配器,升级适配器只换适配器文件- 文件,基模不动、不 merge。--## 核心机制--- **SSD 专家 offload**:专家权重按需从存储流式加载,峰值内存由- 激活集而非参数量决定;-- **prerouter**:训练头提前一步预测专家路由,专家装载与前向计算重叠- 而非阻塞——解码吞吐**最高 +59%**,收益随存储延迟、模型规模与路由- 宽度 *K* 增大;-- **Recover-LoRA**:冻结 int4 基模,用 FP teacher 蒸馏训练 LoRA,- 在 4bit 下恢复绝大部分量化损失(见[质量](#质量))。适配器不合并,- 一份只读基模服务多套适配器。--## 快速开始--### 1) 安装+#### 1) 安装```bash+cd python# Python ≥3.10;MLX 后端需 macOS + Apple Siliconpython3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'```-### 2) 下载模型+#### 2) 下载模型-两个档位发布在 Hugging Face 与 ModelScope——每个仓库把基模 checkpoint 与训练好的-LoRA + prerouter 适配器打包在**同一目录**,一次下载即为可运行的模型:+两个档位发布在 Hugging Face 与 ModelScope——每个仓库把基模 checkpoint+与训练好的 LoRA + prerouter 适配器打包在**同一目录**,一次下载即为+可运行的模型:- [`Edge0/Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview)(约 23 GB) · [ModelScope 镜像](https://www.modelscope.cn/models/Edge0/Edge0-35B-A3B-preview)- [`Edge0/Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview)(约 4.2 GB) · [ModelScope 镜像](https://www.modelscope.cn/models/Edge0/Edge0-8B-A1B-preview)@@ -114,7 +192,7 @@ models/edge0-35b/└── prerouter_edge0_35b.safetensors # 训练好的 prerouter 头```-### 3) 指向模型目录+#### 3) 指向模型目录档位名经环境变量解析到本地目录(放哪由你决定):@@ -131,7 +209,7 @@ edge0 demo models/edge0-35bedge0 serve models/edge0-8b```-### 4) 运行+#### 4) 运行```bash# 快速演示@@ -152,7 +230,7 @@ edge0 chat edge0-35b --prompt "用一句话解释流式推理。"`python -m edge0 ...` 等价于 `edge0 ...`。-### Python API+#### Python API```pythonfrom edge0 import AutoEngine@@ -169,9 +247,9 @@ print(engine._tok.decode(tokens))engine.close() # 释放 mmap / 专家缓存```-`examples/demo.py` 就是这条最小路径(`edge0 demo` 内部等价运行)。+`python/examples/demo.py` 就是这条最小路径(`edge0 demo` 内部等价运行)。-### 模型与适配器+#### 模型与适配器- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、tokenizer)。`edge0 serve <dir>` / `AutoEngine.from_pretrained(<dir>)`@@ -179,68 +257,104 @@ engine.close() # 释放 mmap / 专家缓存- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:- **模型目录内**(推荐):与基模同目录,如`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- - `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从- 训练侧 npz 一次性转换。+ - `artifacts/`(gitignored):`edge0 convert-adapters` 从训练侧 npz+ 一次性转换。- 发布的模型仓库同时包含基模与当前默认适配器版本,`scripts/fetch_models.py` 下载后即为可运行的模型目录。适配器来源(训练数据、owner 层分布)见各模型文档页。- prerouter + LoRA 两条适配器都必需;缺文件时 `edge0` 会给出明确报错(也可加 `--no-prerouter` / `--no-lora` 直接跑裸基模)。-## 质量+#### 文档-全部评测由我们使用 [OpenCompass](https://github.com/open-compass/opencompass)、-在完全相同的设置与参数下对 edge0 模型(int4 + 训练适配器 + prerouter 路由)-与原 fp16 基座模型测得。edge0 管线的损失很小:**edge0-35b 平均仅落后-3.9 分、edge0-8b 落后 2.8 分**(MMLU-Pro 甚至反超基座)。满分 100:+- [架构总览](docs/architecture.md)+- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)+- [如何接入新模型](docs/adding-a-model.md)+- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)+- 技术报告:[The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063)([PDF](paper/main.pdf))-| 评测集 | edge0-35b(int4) | Qwen3.6-35B-A3B(fp16) | edge0-8b(int4) | Ling 3.0 tiny(fp16) |-|---|---:|---:|---:|---:|-| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |-| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |-| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |-| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |-| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |-| **平均** | **79.2** | **83.2** | **69.9** | **72.7** |+### macOS / iOS / Android-## 性能实测+三端推理引擎已开源在本仓库,更多细节见各目录 README:-`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →-200 token 计时段,每档 2 轮):+- **macOS**:本地 CLI / daemon / 桌面 App(Rust)—— 详见+ [`macos/README.md`](macos/README.md)+- **iOS**:iPhone 端侧 App(Swift + MLX Swift)—— 详见+ [`ios/README.md`](ios/README.md)+- **Android**:端侧 App + 原生引擎(Kotlin)—— 详见+ [`android/README.md`](android/README.md)-| 档位 | 解码速度 | Prefill 吞吐(冷/热)* | 峰值 active 内存 | 测试机器 |-|---|---|---|---|---|-| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB |-| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |+**统一推理框架**(接入层统一,runtime 自动适配 iOS / macOS / Android /+Windows / Python)将于 **2026 年 10 月底**发布,见[路线图](#-路线图)。-*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。+## 🗺️ 路线图-复现:+- **【2026 年 10 月底】edge0 统一推理框架开源** —— 发布统一推理框架:+ **接入层统一**为一套 API(chat / serve / 端侧共用),**runtime 自动+ 适配不同硬件平台**——iOS、macOS、Android、Windows、Python;在本仓库+ 已开源的三端引擎(`ios/` · `macos/` · `android/`)之上统一收编。+- **Windows** 平台支持(经统一框架)。+- **CUDA 后端**(Python 框架)—— 插槽已预留在+ `python/src/edge0/backends/cuda/`,核心代码零改动。+- 更多模型档位与适配器版本在现有管线上持续发布。-```bash-python examples/bench.py edge0-35b # 经 $EDGE0_35B_MODEL-python examples/bench.py edge0-8b # 经 $EDGE0_8B_MODEL-```+## 🤝 贡献-## 验证+欢迎各种形式的贡献——issue、PR、性能实测报告、新模型接入都算。++**Python 框架**(现已可用):```bash+cd python+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'+pytest # 单元测试(不含真实权重)EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q- # 真实权重生成测试;缺少的档位会明确 skip+ # 真实权重生成测试;缺少的档位会明确 skip.venv/bin/python scripts/e2e_smoke.py \--qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b- # staged vs exact 一致性 + 生成冒烟+ # staged vs exact 一致性 + 生成冒烟scripts/generate_example.py # 完整 API 上手例子(prefill→生成→解码全链路)-examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)+examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)```-## 文档+每个 PR 都会跑 CI:单元测试(macOS + MLX)与仓库卫生检查(无硬编码+路径、后端边界、无密钥)。-- [架构总览](docs/architecture.md)-- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)-- [如何接入新模型](docs/adding-a-model.md)-- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)+**平台 runtime**(macOS / iOS / Android):目前是占位目录,贡献指南将+随统一框架代码一并发布。++流程:fork → 功能分支 → 向 `main` 发 PR。请保持卫生检查通过,并为新+行为补测试。++## 📖 引用++如果 edge0 对你有帮助,请引用我们的技术报告:++```bibtex+@article{lin2026other,+ title = {The Other Half of the Memory Wall: Serving 35B MoEs from SSD+ with Trained Routing Prediction},+ author = {Lin, Yu and Wang, Yiming and Cai, Runyuan and Liu, Hanze and+ Zeng, Xiaodong},+ journal = {arXiv preprint arXiv:2609.18063},+ year = {2026},+ url = {https://arxiv.org/abs/2609.18063}+}+```++## 📧 联系我们++<!-- TODO(edge0 团队):发布前填写官方联系渠道。 -->++社区与支持渠道即将上线,本节将列出联系我们的官方方式:++- **邮箱**:_待补充_+- **Discord**:_待补充_+- **微信群**:_待补充_++Bug 与功能建议请直接提+[GitHub Issues](https://github.com/Edge0-AI/Edge0/issues)。## License