Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
refactor(repo): adopt the multi-platform layout
Bincad29a1e1d32
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 9c3a3ad..78cfea2 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -9,6 +9,9 @@ jobs:test-macos:name: Unit tests (macOS + MLX)runs-on: macos-14+ defaults:+ run:+ working-directory: pythonsteps:- uses: actions/checkout@v4- uses: actions/setup-python@v5@@ -22,6 +25,9 @@ jobs:hygiene:name: Repo hygiene (pure Python, no MLX)runs-on: ubuntu-latest+ defaults:+ run:+ working-directory: pythonsteps:- uses: actions/checkout@v4- uses: actions/setup-python@v5diff --git a/.gitignore b/.gitignoreindex c3b7432..b41702c 100644--- a/.gitignore+++ b/.gitignore@@ -18,4 +18,4 @@ artifacts/# Bench / runtime logs*.log-scripts/alignment_results.json+python/scripts/alignment_results.jsondiff --git a/android/README.md b/android/README.mdnew file mode 100644index 0000000..6068ea4--- /dev/null+++ b/android/README.md@@ -0,0 +1,23 @@+# edge0 — Android runtime++> ✅ Open-sourced on **2026-09-30**. This README will be updated with+> full build & usage docs together with the source drop.++The Android runtime runs the edge0 models fully on-device on Android+(Kotlin app + native inference engine): local streaming MoE inference+with the same recipe as the rest of the framework — SSD expert offload,+Recover-LoRA, and prerouter routing prediction.++The next milestone is the **unified inference framework** — one access+layer, runtime auto-adapting to the hardware platform (iOS / macOS /+Android / Windows / Python) — targeted for the **end of October 2026**;+see the [roadmap](../README.md#-roadmap).++See the [repository README](../README.md) for the multi-platform picture.++---++> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建+> 与使用文档。Android runtime:Kotlin App + 原生推理引擎的端侧推理。+> **统一推理框架**将于 **2026 年 10 月底**发布,详见+> [根 README 路线图](../README_zh.md)。diff --git a/docs/adding-a-model.md b/docs/adding-a-model.mdindex 6af69b4..f14ac8c 100644--- a/docs/adding-a-model.md+++ b/docs/adding-a-model.md@@ -42,7 +42,7 @@ The subclass only needs to implement the class method `_defaults(model_dir) -> CKey points:-- The repository root ships `scripts/convert_adapters_legacy.py`, which one-shot converts the training npz exports into safetensors artifacts under `artifacts/`; `ModelConfig.artifact(name)` (`models/base.py:28–30`) returns the absolute paths of those artifacts, and the adapter uses them to fill in the LoRA / prerouter weight paths.+- The Python project root (`python/`) ships `scripts/convert_adapters_legacy.py`, which one-shot converts the training npz exports into safetensors artifacts under `artifacts/`; `ModelConfig.artifact(name)` (`models/base.py:28–30`) returns the absolute paths of those artifacts, and the adapter uses them to fill in the LoRA / prerouter weight paths.- LoRA overrides go through `resolve_lora` (`models/base.py:88–95`): a bare model name resolves to that tier's artifacts, `"model_dir"` means "keep the training weights in place", and an empty string disables it.## Step 3: `build_model` / `build_engine`diff --git a/docs/prerouter.md b/docs/prerouter.mdindex 460a1da..8744fa4 100644--- a/docs/prerouter.md+++ b/docs/prerouter.md@@ -73,7 +73,7 @@ per `start_layer`/`owners`. Family differences are expressed through three hooks| weight file | `prerouter_edge0_35b.safetensors` | `prerouter_edge0_8b.safetensors` |Both weight files are resolved from the model directory by default, with-`artifacts/` (repo root, gitignored) as the fallback — see the README's+`artifacts/` (Python project root, gitignored) as the fallback — see the README's"Models and adapters" section.## Tests and regressiondiff --git a/ios/README.md b/ios/README.mdnew file mode 100644index 0000000..389fa70--- /dev/null+++ b/ios/README.md@@ -0,0 +1,23 @@+# edge0 — iOS runtime++> ✅ Open-sourced on **2026-09-30**. This README will be updated with+> full build & usage docs together with the source drop.++The iOS runtime runs the edge0 models fully on-device on iPhone+(Swift / SwiftUI + MLX Swift): local streaming MoE inference with the+same recipe as the rest of the framework — SSD expert offload,+Recover-LoRA, and prerouter routing prediction.++The next milestone is the **unified inference framework** — one access+layer, runtime auto-adapting to the hardware platform (iOS / macOS /+Android / Windows / Python) — targeted for the **end of October 2026**;+see the [roadmap](../README.md#-roadmap).++See the [repository README](../README.md) for the multi-platform picture.++---++> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建+> 与使用文档。iOS runtime:Swift / SwiftUI + MLX Swift 的 iPhone 端侧+> 推理。**统一推理框架**将于 **2026 年 10 月底**发布,详见+> [根 README 路线图](../README_zh.md)。diff --git a/macos/README.md b/macos/README.mdnew file mode 100644index 0000000..35e7ddd--- /dev/null+++ b/macos/README.md@@ -0,0 +1,24 @@+# edge0 — macOS runtime++> ✅ Open-sourced on **2026-09-30**. This README will be updated with+> full build & usage docs together with the source drop.++The macOS runtime is the local CLI / daemon / desktop app for edge0 on+Apple Silicon (Rust + Tauri): an Ollama-style UX that pulls the edge0+models ([Edge0-35B-A3B-preview](https://huggingface.co/Edge0/Edge0-35B-A3B-preview),+[Edge0-8B-A1B-preview](https://huggingface.co/Edge0/Edge0-8B-A1B-preview))+and serves them locally behind an OpenAI-compatible API.++The next milestone is the **unified inference framework** — one access+layer, runtime auto-adapting to the hardware platform (iOS / macOS /+Android / Windows / Python) — targeted for the **end of October 2026**;+see the [roadmap](../README.md#-roadmap).++See the [repository README](../README.md) for the multi-platform picture.++---++> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建+> 与使用文档。macOS runtime:Rust + Tauri 的本地 CLI / daemon / 桌面+> App,Ollama 式体验 + OpenAI 兼容 API。**统一推理框架**将于 **2026 年+> 10 月底**发布,详见[根 README 路线图](../README_zh.md)。diff --git a/python/README.md b/python/README.mdnew file mode 100644index 0000000..2bee612--- /dev/null+++ b/python/README.md@@ -0,0 +1,24 @@+# edge0 — Python framework++The Python implementation of the edge0 streaming MoE inference framework+(SSD expert offload + Recover-LoRA + prerouter routing prediction),+shipping the MLX backend for Apple Silicon.++This directory is one platform of the edge0 multi-platform repo — see the+[repository README](../README.md) for the full picture (macOS / iOS /+Android runtimes and the unified-framework roadmap).++## Quick start++```bash+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'+edge0 demo edge0-8b # after downloading a model; see ../README.md+```++- Docs: [`docs/`](../docs/) at the repo root (architecture, attention, MoE, streaming, prerouter)+- Examples: [`examples/`](examples/) · Scripts: [`scripts/`](scripts/)+- Tests: `pytest` (unit) · `pytest -m slow` (real weights)++## License++Apache-2.0, including vendored third-party code (see [NOTICE](../NOTICE)).diff --git a/examples/bench.py b/python/examples/bench.pysimilarity index 100%rename from examples/bench.pyrename to python/examples/bench.pydiff --git a/examples/demo.py b/python/examples/demo.pysimilarity index 100%rename from examples/demo.pyrename to python/examples/demo.pydiff --git a/pyproject.toml b/python/pyproject.tomlsimilarity index 100%rename from pyproject.tomlrename to python/pyproject.tomldiff --git a/scripts/convert_adapters_legacy.py b/python/scripts/convert_adapters_legacy.pysimilarity index 100%rename from scripts/convert_adapters_legacy.pyrename to python/scripts/convert_adapters_legacy.pydiff --git a/scripts/e2e_smoke.py b/python/scripts/e2e_smoke.pysimilarity index 100%rename from scripts/e2e_smoke.pyrename to python/scripts/e2e_smoke.pydiff --git a/scripts/fetch_models.py b/python/scripts/fetch_models.pysimilarity index 100%rename from scripts/fetch_models.pyrename to python/scripts/fetch_models.pydiff --git a/scripts/generate_example.py b/python/scripts/generate_example.pysimilarity index 100%rename from scripts/generate_example.pyrename to python/scripts/generate_example.pydiff --git a/scripts/strip_vision_weights.py b/python/scripts/strip_vision_weights.pysimilarity index 100%rename from scripts/strip_vision_weights.pyrename to python/scripts/strip_vision_weights.pydiff --git a/scripts/upload_hf.py b/python/scripts/upload_hf.pysimilarity index 100%rename from scripts/upload_hf.pyrename to python/scripts/upload_hf.pydiff --git a/scripts/verify_alignment.py b/python/scripts/verify_alignment.pysimilarity index 100%rename from scripts/verify_alignment.pyrename to python/scripts/verify_alignment.pydiff --git a/src/edge0/__init__.py b/python/src/edge0/__init__.pysimilarity index 100%rename from src/edge0/__init__.pyrename to python/src/edge0/__init__.pydiff --git a/src/edge0/__main__.py b/python/src/edge0/__main__.pysimilarity index 100%rename from src/edge0/__main__.pyrename to python/src/edge0/__main__.pydiff --git a/src/edge0/adapters/__init__.py b/python/src/edge0/adapters/__init__.pysimilarity index 100%rename from src/edge0/adapters/__init__.pyrename to python/src/edge0/adapters/__init__.pydiff --git a/src/edge0/adapters/lora.py b/python/src/edge0/adapters/lora.pysimilarity index 100%rename from src/edge0/adapters/lora.pyrename to python/src/edge0/adapters/lora.pydiff --git a/src/edge0/attention/spec.py b/python/src/edge0/attention/spec.pysimilarity index 100%rename from src/edge0/attention/spec.pyrename to python/src/edge0/attention/spec.pydiff --git a/src/edge0/backends/__init__.py b/python/src/edge0/backends/__init__.pysimilarity index 100%rename from src/edge0/backends/__init__.pyrename to python/src/edge0/backends/__init__.pydiff --git a/src/edge0/backends/base.py b/python/src/edge0/backends/base.pysimilarity index 100%rename from src/edge0/backends/base.pyrename to python/src/edge0/backends/base.pydiff --git a/src/edge0/backends/mlx/__init__.py b/python/src/edge0/backends/mlx/__init__.pysimilarity index 100%rename from src/edge0/backends/mlx/__init__.pyrename to python/src/edge0/backends/mlx/__init__.pydiff --git a/src/edge0/backends/mlx/_impl/__init__.py b/python/src/edge0/backends/mlx/_impl/__init__.pysimilarity index 100%rename from src/edge0/backends/mlx/_impl/__init__.pyrename to python/src/edge0/backends/mlx/_impl/__init__.pydiff --git a/src/edge0/backends/mlx/_impl/bailing_hybrid.py b/python/src/edge0/backends/mlx/_impl/bailing_hybrid.pysimilarity index 100%rename from src/edge0/backends/mlx/_impl/bailing_hybrid.pyrename to python/src/edge0/backends/mlx/_impl/bailing_hybrid.pydiff --git a/src/edge0/backends/mlx/_impl/qwen3_5.py b/python/src/edge0/backends/mlx/_impl/qwen3_5.pysimilarity index 100%rename from src/edge0/backends/mlx/_impl/qwen3_5.pyrename to python/src/edge0/backends/mlx/_impl/qwen3_5.pydiff --git a/src/edge0/backends/mlx/_impl/qwen3_5_moe.py b/python/src/edge0/backends/mlx/_impl/qwen3_5_moe.pysimilarity index 100%rename from src/edge0/backends/mlx/_impl/qwen3_5_moe.pyrename to python/src/edge0/backends/mlx/_impl/qwen3_5_moe.pydiff --git a/src/edge0/backends/mlx/_impl/qwen3_next.py b/python/src/edge0/backends/mlx/_impl/qwen3_next.pysimilarity index 100%rename from src/edge0/backends/mlx/_impl/qwen3_next.pyrename to python/src/edge0/backends/mlx/_impl/qwen3_next.pydiff --git a/src/edge0/backends/mlx/backend.py b/python/src/edge0/backends/mlx/backend.pysimilarity index 100%rename from src/edge0/backends/mlx/backend.pyrename to python/src/edge0/backends/mlx/backend.pydiff --git a/src/edge0/backends/mlx/io.py b/python/src/edge0/backends/mlx/io.pysimilarity index 100%rename from src/edge0/backends/mlx/io.pyrename to python/src/edge0/backends/mlx/io.pydiff --git a/src/edge0/backends/mlx/quant.py b/python/src/edge0/backends/mlx/quant.pysimilarity index 100%rename from src/edge0/backends/mlx/quant.pyrename to python/src/edge0/backends/mlx/quant.pydiff --git a/src/edge0/cli.py b/python/src/edge0/cli.pysimilarity index 99%rename from src/edge0/cli.pyrename to python/src/edge0/cli.pyindex 97555d8..d2cbb73 100644--- a/src/edge0/cli.py+++ b/python/src/edge0/cli.py@@ -211,9 +211,9 @@ def cmd_serve(args) -> int:def cmd_convert(args) -> int:import runpysys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve()- .parents[1] / "scripts"))+ .parents[2] / "scripts"))runpy.run_path(- str(__import__("pathlib").Path(__file__).resolve().parents[1]+ str(__import__("pathlib").Path(__file__).resolve().parents[2]/ "scripts" / "convert_adapters_legacy.py"),run_name="__main__",)diff --git a/src/edge0/config.py b/python/src/edge0/config.pysimilarity index 100%rename from src/edge0/config.pyrename to python/src/edge0/config.pydiff --git a/src/edge0/engine/__init__.py b/python/src/edge0/engine/__init__.pysimilarity index 100%rename from src/edge0/engine/__init__.pyrename to python/src/edge0/engine/__init__.pydiff --git a/src/edge0/engine/base.py b/python/src/edge0/engine/base.pysimilarity index 100%rename from src/edge0/engine/base.pyrename to python/src/edge0/engine/base.pydiff --git a/src/edge0/engine/hooks.py b/python/src/edge0/engine/hooks.pysimilarity index 100%rename from src/edge0/engine/hooks.pyrename to python/src/edge0/engine/hooks.pydiff --git a/src/edge0/engine/ling.py b/python/src/edge0/engine/ling.pysimilarity index 100%rename from src/edge0/engine/ling.pyrename to python/src/edge0/engine/ling.pydiff --git a/src/edge0/engine/qwen.py b/python/src/edge0/engine/qwen.pysimilarity index 100%rename from src/edge0/engine/qwen.pyrename to python/src/edge0/engine/qwen.pydiff --git a/src/edge0/models/__init__.py b/python/src/edge0/models/__init__.pysimilarity index 100%rename from src/edge0/models/__init__.pyrename to python/src/edge0/models/__init__.pydiff --git a/src/edge0/models/base.py b/python/src/edge0/models/base.pysimilarity index 99%rename from src/edge0/models/base.pyrename to python/src/edge0/models/base.pyindex d3dead5..bb1d73d 100644--- a/src/edge0/models/base.py+++ b/python/src/edge0/models/base.py@@ -19,7 +19,7 @@ from edge0.moe.spec import MoESpecfrom edge0.prerouter.spec import PrerouterSpecfrom edge0.streaming.options import LayerOptions-# Repository-root artifacts (gitignored; produced once by+# Project-root artifacts (gitignored; produced once by# scripts/convert_adapters_legacy.py from the training npz exports, then# the source npz are discarded).ARTIFACTS_DIR = Path(__file__).resolve().parents[3] / "artifacts"diff --git a/src/edge0/models/edge0_35b/__init__.py b/python/src/edge0/models/edge0_35b/__init__.pysimilarity index 100%rename from src/edge0/models/edge0_35b/__init__.pyrename to python/src/edge0/models/edge0_35b/__init__.pydiff --git a/src/edge0/models/edge0_8b/__init__.py b/python/src/edge0/models/edge0_8b/__init__.pysimilarity index 100%rename from src/edge0/models/edge0_8b/__init__.pyrename to python/src/edge0/models/edge0_8b/__init__.pydiff --git a/src/edge0/moe/__init__.py b/python/src/edge0/moe/__init__.pysimilarity index 100%rename from src/edge0/moe/__init__.pyrename to python/src/edge0/moe/__init__.pydiff --git a/src/edge0/moe/routing.py b/python/src/edge0/moe/routing.pysimilarity index 100%rename from src/edge0/moe/routing.pyrename to python/src/edge0/moe/routing.pydiff --git a/src/edge0/moe/spec.py b/python/src/edge0/moe/spec.pysimilarity index 100%rename from src/edge0/moe/spec.pyrename to python/src/edge0/moe/spec.pydiff --git a/src/edge0/prerouter/__init__.py b/python/src/edge0/prerouter/__init__.pysimilarity index 100%rename from src/edge0/prerouter/__init__.pyrename to python/src/edge0/prerouter/__init__.pydiff --git a/src/edge0/prerouter/heads.py b/python/src/edge0/prerouter/heads.pysimilarity index 100%rename from src/edge0/prerouter/heads.pyrename to python/src/edge0/prerouter/heads.pydiff --git a/src/edge0/prerouter/install.py b/python/src/edge0/prerouter/install.pysimilarity index 100%rename from src/edge0/prerouter/install.pyrename to python/src/edge0/prerouter/install.pydiff --git a/src/edge0/prerouter/spec.py b/python/src/edge0/prerouter/spec.pysimilarity index 100%rename from src/edge0/prerouter/spec.pyrename to python/src/edge0/prerouter/spec.pydiff --git a/src/edge0/prerouter/stager.py b/python/src/edge0/prerouter/stager.pysimilarity index 100%rename from src/edge0/prerouter/stager.pyrename to python/src/edge0/prerouter/stager.pydiff --git a/src/edge0/prerouter/state.py b/python/src/edge0/prerouter/state.pysimilarity index 100%rename from src/edge0/prerouter/state.pyrename to python/src/edge0/prerouter/state.pydiff --git a/src/edge0/registry.py b/python/src/edge0/registry.pysimilarity index 100%rename from src/edge0/registry.pyrename to python/src/edge0/registry.pydiff --git a/src/edge0/sampling.py b/python/src/edge0/sampling.pysimilarity index 100%rename from src/edge0/sampling.pyrename to python/src/edge0/sampling.pydiff --git a/src/edge0/server/__init__.py b/python/src/edge0/server/__init__.pysimilarity index 100%rename from src/edge0/server/__init__.pyrename to python/src/edge0/server/__init__.pydiff --git a/src/edge0/server/app.py b/python/src/edge0/server/app.pysimilarity index 100%rename from src/edge0/server/app.pyrename to python/src/edge0/server/app.pydiff --git a/src/edge0/server/chat.py b/python/src/edge0/server/chat.pysimilarity index 100%rename from src/edge0/server/chat.pyrename to python/src/edge0/server/chat.pydiff --git a/src/edge0/streaming/__init__.py b/python/src/edge0/streaming/__init__.pysimilarity index 100%rename from src/edge0/streaming/__init__.pyrename to python/src/edge0/streaming/__init__.pydiff --git a/src/edge0/streaming/cache.py b/python/src/edge0/streaming/cache.pysimilarity index 100%rename from src/edge0/streaming/cache.pyrename to python/src/edge0/streaming/cache.pydiff --git a/src/edge0/streaming/install.py b/python/src/edge0/streaming/install.pysimilarity index 100%rename from src/edge0/streaming/install.pyrename to python/src/edge0/streaming/install.pydiff --git a/src/edge0/streaming/layer.py b/python/src/edge0/streaming/layer.pysimilarity index 100%rename from src/edge0/streaming/layer.pyrename to python/src/edge0/streaming/layer.pydiff --git a/src/edge0/streaming/mmap.py b/python/src/edge0/streaming/mmap.pysimilarity index 100%rename from src/edge0/streaming/mmap.pyrename to python/src/edge0/streaming/mmap.pydiff --git a/src/edge0/streaming/options.py b/python/src/edge0/streaming/options.pysimilarity index 100%rename from src/edge0/streaming/options.pyrename to python/src/edge0/streaming/options.pydiff --git a/tests/test_cli_flags.py b/python/tests/test_cli_flags.pysimilarity index 100%rename from tests/test_cli_flags.pyrename to python/tests/test_cli_flags.pydiff --git a/tests/test_e2e_slow.py b/python/tests/test_e2e_slow.pysimilarity index 100%rename from tests/test_e2e_slow.pyrename to python/tests/test_e2e_slow.pydiff --git a/tests/test_moe_spec.py b/python/tests/test_moe_spec.pysimilarity index 100%rename from tests/test_moe_spec.pyrename to python/tests/test_moe_spec.pydiff --git a/tests/test_prefill_hook.py b/python/tests/test_prefill_hook.pysimilarity index 100%rename from tests/test_prefill_hook.pyrename to python/tests/test_prefill_hook.pydiff --git a/tests/test_registry.py b/python/tests/test_registry.pysimilarity index 100%rename from tests/test_registry.pyrename to python/tests/test_registry.pydiff --git a/tests/test_repo_hygiene.py b/python/tests/test_repo_hygiene.pysimilarity index 91%rename from tests/test_repo_hygiene.pyrename to python/tests/test_repo_hygiene.pyindex c8ef1d0..a6ed8ef 100644--- a/tests/test_repo_hygiene.py+++ b/python/tests/test_repo_hygiene.py@@ -16,13 +16,15 @@ from __future__ import annotationsimport pathlibimport re-ROOT = pathlib.Path(__file__).resolve().parents[1]+# Repo root (this file lives at <root>/python/tests/): the hygiene guards+# scan the whole multi-platform repo, not just the Python subproject.+ROOT = pathlib.Path(__file__).resolve().parents[2]_EXCLUDE_DIRS = {".git", ".venv", ".pytest_cache", "__pycache__", ".agents",".codex"}_EXCLUDE_PARTS = {"egg-info"}_EXCLUDE_SUFFIXES = {".pyc", ".so", ".dylib", ".bin", ".safetensors", ".npz",- ".png", ".jpg", ".jpeg", ".gif", ".ico"}+ ".png", ".jpg", ".jpeg", ".gif", ".ico", ".pdf"}# NOTE: patterns are assembled with + so this file never contains the# literal string it is looking for.@@ -85,7 +87,7 @@ def test_no_hardcoded_local_paths():def test_mlx_imports_stay_inside_the_backend():bad = []- for path in (ROOT / "src" / "edge0").rglob("*.py"):+ for path in (ROOT / "python" / "src" / "edge0").rglob("*.py"):rel = path.relative_to(ROOT)text = path.read_text(encoding="utf-8")if _MLX_IMPORT.search(text) and "backends/mlx" not in rel.as_posix():diff --git a/tests/test_sampling.py b/python/tests/test_sampling.pysimilarity index 100%rename from tests/test_sampling.pyrename to python/tests/test_sampling.pydiff --git a/tests/test_server.py b/python/tests/test_server.pysimilarity index 100%rename from tests/test_server.pyrename to python/tests/test_server.pydiff --git a/tests/test_streaming_math.py b/python/tests/test_streaming_math.pysimilarity index 100%rename from tests/test_streaming_math.pyrename to python/tests/test_streaming_math.py