Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
adjust readme
Binaa8100808953
diff --git a/README.md b/README.mdindex 1d5b87b..845e8d2 100644--- a/README.md+++ b/README.md@@ -22,7 +22,7 @@ English | [中文](README_zh.md)## News-- **[2026-09-30]** We released the **edge0 inference engines for three platforms — iOS, macOS and Android** — so users get the best inference experience across architectures and platforms. The source is open-sourced in this repo ([`ios/`](ios/) · [`macos/`](macos/) · [`android/`](android/)) — see each directory's README for details. The **unified inference framework** follows by the **end of October 2026**; see the [Roadmap](#roadmap).+- **[2026-09-30]** We released the **edge0 inference engines for four platforms — iOS, macOS, Android and Windows** — so users get the best inference experience across architectures and platforms. The source is open-sourced in this repo ([`ios/`](ios/) · [`macos/`](macos/) · [`android/`](android/) · [`windows/`](windows/)) — see each directory's README for details. The **unified inference framework** follows in **Q4 2026**; see the [Roadmap](#roadmap).- **[2026-09-20]** `--prefill-ondemand` lands for `edge0 demo / chat / serve` ([#112](https://github.com/Edge0-AI/Edge0/pull/112)).- **[2026-09-16]** Our technical report is on arXiv: [The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction](https://arxiv.org/abs/2609.18063).- **[2026-09-08]** Initial open-source release of **edge0**, together with both model tiers — [`Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) and [`Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) — on Hugging Face and ModelScope.@@ -59,7 +59,7 @@ One repo, one recipe, per-platform runtimes:| **macOS** app & CLI | [`macos/`](macos/) | Rust | ✅ Open-sourced (2026-09-30) || **iOS** app | [`ios/`](ios/) | Swift + MLX Swift | ✅ Open-sourced (2026-09-30) || **Android** app & engine | [`android/`](android/) | Kotlin + native engine | ✅ Open-sourced (2026-09-30) |-| **Windows** | — | — | 🗺️ On the roadmap |+| **Windows** app & engine | [`windows/`](windows/) | C++ + Vulkan | ✅ Open-sourced (2026-09-30) |### Models@@ -283,35 +283,37 @@ runs this exact path).- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)- Technical report: [The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063) ([PDF](paper/main.pdf))-### macOS / iOS / Android+### macOS / iOS / Android / Windows-The three platform engines are open-sourced in this repo — more details+The four platform engines are open-sourced in this repo — more detailsin each directory's README:- **macOS**: local CLI / daemon / desktop app (Rust) — see [`macos/README.md`](macos/README.md)- **iOS**: on-device iPhone app (Swift + MLX Swift) — see [`ios/README.md`](ios/README.md)- **Android**: on-device app + native engine (Kotlin) — see [`android/README.md`](android/README.md)+- **Windows**: desktop app + native engine (C++ + Vulkan) — see [`windows/README.md`](windows/README.md)The **unified inference framework** — one access layer, runtime-auto-adapting to iOS / macOS / Android / Windows / Python — arrives by-the **end of October 2026**; see the [Roadmap](#roadmap).+auto-adapting to iOS / macOS / Android / Windows / Python — arrives in+**Q4 2026**; see the [Roadmap](#roadmap).## Roadmap+### Q4 2026+**Platforms & systems**-- **[End of Oct 2026] edge0 unified inference framework** — we will+- **edge0 unified inference framework** — we willopen-source a unified inference framework: **one unified accesslayer** (a single API across chat / serve / on-device use), with the**runtime automatically adapting to the hardware platform** — iOS,- macOS, Android, Windows and Python. It builds on the three platform+ macOS, Android, Windows and Python. It builds on the platformengines already open-sourced in this repo (`ios/` · `macos/` ·- `android/`).-- **Windows** platform support (via the unified framework).+ `android/` · `windows/`).- **CUDA backend** for the Python framework — reserved slot at`python/src/edge0/backends/cuda/`, core code needs zero changes.-**Models & algorithms — Q4 2026**+**Models & algorithms**Q4 works two fronts: bringing a next-generation architecture into theframework, and turning latent reasoning into a real latency saving rather@@ -383,8 +385,6 @@ If you find edge0 useful, please cite our technical report:## Contact Us-<!-- TODO(edge0 team): fill in the official contact channels before publish. -->-Community and support channels are coming soon — this section will listthe official ways to reach us:diff --git a/README_zh.md b/README_zh.mdindex 6b76609..da25752 100644--- a/README_zh.md+++ b/README_zh.md@@ -22,11 +22,11 @@## 新闻-- **【2026-09-30】** 我们发布了**三端推理引擎**,适配 iOS、macOS、Android- 平台,让用户在不同架构、不同平台上都有最佳的推理体验。三端源码已- 开源至本仓库([`ios/`](ios/) · [`macos/`](macos/) ·- [`android/`](android/)),更多细节见各目录 README;**统一推理框架**- 将于 **2026 年 10 月底**发布,详见[路线图](#路线图)。+- **【2026-09-30】** 我们发布了**四端推理引擎**,适配 iOS、macOS、+ Android、Windows 平台,让用户在不同架构、不同平台上都有最佳的推理+ 体验。四端源码已开源至本仓库([`ios/`](ios/) · [`macos/`](macos/) ·+ [`android/`](android/) · [`windows/`](windows/)),更多细节见各目录+ README;**统一推理框架**将于 **2026 Q4** 发布,详见[路线图](#路线图)。- **【2026-09-20】** `edge0 demo / chat / serve` 新增 `--prefill-ondemand`([#112](https://github.com/Edge0-AI/Edge0/pull/112))。- **【2026-09-16】** 技术报告上线 arXiv:[The Other Half of the Memory@@ -64,7 +64,7 @@ MoE 模型跑在消费级硬件上——峰值内存由**激活**专家集而非| **macOS** 桌面 App 与 CLI | [`macos/`](macos/) | Rust | ✅ 已开源(2026-09-30) || **iOS** App | [`ios/`](ios/) | Swift + MLX Swift | ✅ 已开源(2026-09-30) || **Android** App 与引擎 | [`android/`](android/) | Kotlin + 原生引擎 | ✅ 已开源(2026-09-30) |-| **Windows** | — | — | 🗺️ 路线图中 |+| **Windows** App 与引擎 | [`windows/`](windows/) | C++ + Vulkan | ✅ 已开源(2026-09-30) |### 模型@@ -273,9 +273,9 @@ engine.close() # 释放 mmap / 专家缓存- [edge0-35b](docs/models/edge0-35b.md) / [edge0-8b](docs/models/edge0-8b.md)- 技术报告:[The Other Half of the Memory Wall](https://arxiv.org/abs/2609.18063)([PDF](paper/main.pdf))-### macOS / iOS / Android+### macOS / iOS / Android / Windows-三端推理引擎已开源在本仓库,更多细节见各目录 README:+四端推理引擎已开源在本仓库,更多细节见各目录 README:- **macOS**:本地 CLI / daemon / 桌面 App(Rust)—— 详见[`macos/README.md`](macos/README.md)@@ -283,23 +283,26 @@ engine.close() # 释放 mmap / 专家缓存[`ios/README.md`](ios/README.md)- **Android**:端侧 App + 原生引擎(Kotlin)—— 详见[`android/README.md`](android/README.md)+- **Windows**:桌面 App + 原生引擎(C++ + Vulkan)—— 详见+ [`windows/README.md`](windows/README.md)**统一推理框架**(接入层统一,runtime 自动适配 iOS / macOS / Android /-Windows / Python)将于 **2026 年 10 月底**发布,见[路线图](#路线图)。+Windows / Python)将于 **2026 Q4** 发布,见[路线图](#路线图)。## 路线图+### 2026 Q4+**平台与系统**-- **【2026 年 10 月底】edge0 统一推理框架开源** —— 发布统一推理框架:- **接入层统一**为一套 API(chat / serve / 端侧共用),**runtime 自动- 适配不同硬件平台**——iOS、macOS、Android、Windows、Python;在本仓库- 已开源的三端引擎(`ios/` · `macos/` · `android/`)之上统一收编。-- **Windows** 平台支持(经统一框架)。+- **edge0 统一推理框架开源** —— 发布统一推理框架:**接入层统一**为一套+ API(chat / serve / 端侧共用),**runtime 自动适配不同硬件平台**——+ iOS、macOS、Android、Windows、Python;在本仓库已开源的平台引擎+ (`ios/` · `macos/` · `android/` · `windows/`)之上统一收编。- **CUDA 后端**(Python 框架)—— 插槽已预留在`python/src/edge0/backends/cuda/`,核心代码零改动。-**模型与算法 —— 2026 Q4**+**模型与算法**Q4 推进两条主线:把下一代架构引入框架,以及把潜在推理(latentthinking)变成真正的延迟收益、而不只是算力节省。@@ -364,8 +367,6 @@ examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代## 联系我们-<!-- TODO(edge0 团队):如需补充更多渠道(Discord / 微信群等),在此添加。 -->-社区与支持渠道即将上线,本节将列出联系我们的官方方式:- **邮箱**:[email protected]diff --git a/android/README.md b/android/README.mdindex 9be16bd..bb891b6 100644--- a/android/README.md+++ b/android/README.md@@ -10,7 +10,7 @@ Recover-LoRA, and prerouter routing prediction.The next milestone is the **unified inference framework** — one accesslayer, runtime auto-adapting to the hardware platform (iOS / macOS /-Android / Windows / Python) — targeted for the **end of October 2026**;+Android / Windows / Python) — targeted for **Q4 2026**;see the [roadmap](../README.md#roadmap).See the [repository README](../README.md) for the multi-platform picture.@@ -19,5 +19,5 @@ See the [repository README](../README.md) for the multi-platform picture.> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建> 与使用文档。Android runtime:Kotlin App + 原生推理引擎的端侧推理。-> **统一推理框架**将于 **2026 年 10 月底**发布,详见+> **统一推理框架**将于 **2026 Q4** 发布,详见> [根 README 路线图](../README_zh.md)。diff --git a/ios/README.md b/ios/README.mdindex 7fc05a3..292de71 100644--- a/ios/README.md+++ b/ios/README.md@@ -10,7 +10,7 @@ Recover-LoRA, and prerouter routing prediction.The next milestone is the **unified inference framework** — one accesslayer, runtime auto-adapting to the hardware platform (iOS / macOS /-Android / Windows / Python) — targeted for the **end of October 2026**;+Android / Windows / Python) — targeted for **Q4 2026**;see the [roadmap](../README.md#roadmap).See the [repository README](../README.md) for the multi-platform picture.@@ -19,5 +19,5 @@ See the [repository README](../README.md) for the multi-platform picture.> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建> 与使用文档。iOS runtime:Swift / SwiftUI + MLX Swift 的 iPhone 端侧-> 推理。**统一推理框架**将于 **2026 年 10 月底**发布,详见+> 推理。**统一推理框架**将于 **2026 Q4** 发布,详见> [根 README 路线图](../README_zh.md)。diff --git a/macos/README.md b/macos/README.mdindex e4efb5c..fbfdf84 100644--- a/macos/README.md+++ b/macos/README.md@@ -11,7 +11,7 @@ and serves them locally behind an OpenAI-compatible API.The next milestone is the **unified inference framework** — one accesslayer, runtime auto-adapting to the hardware platform (iOS / macOS /-Android / Windows / Python) — targeted for the **end of October 2026**;+Android / Windows / Python) — targeted for **Q4 2026**;see the [roadmap](../README.md#roadmap).See the [repository README](../README.md) for the multi-platform picture.@@ -20,5 +20,5 @@ See the [repository README](../README.md) for the multi-platform picture.> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建> 与使用文档。macOS runtime:Rust + Tauri 的本地 CLI / daemon / 桌面-> App,Ollama 式体验 + OpenAI 兼容 API。**统一推理框架**将于 **2026 年-> 10 月底**发布,详见[根 README 路线图](../README_zh.md)。+> App,Ollama 式体验 + OpenAI 兼容 API。**统一推理框架**将于 **2026 Q4**+> 发布,详见[根 README 路线图](../README_zh.md)。diff --git a/windows/README.md b/windows/README.mdnew file mode 100644index 0000000..4296800--- /dev/null+++ b/windows/README.md@@ -0,0 +1,25 @@+# edge0 — Windows runtime++> ✅ Open-sourced on **2026-09-30**. This README will be updated with+> full build & usage docs together with the source drop.++The Windows runtime is a native C++17 port of the edge0 streaming core+(CMake build) with a **Vulkan** compute backend: the `edge0` core+(streaming mmap / cache / layer / MoE) is mirrored from the upstream+Python tree, running the same recipe as the rest of the framework — SSD+expert offload, Recover-LoRA, and prerouter routing prediction.++The next milestone is the **unified inference framework** — one access+layer, runtime auto-adapting to the hardware platform (iOS / macOS /+Android / Windows / Python) — targeted for **Q4 2026**; see the+[roadmap](../README.md#roadmap).++See the [repository README](../README.md) for the multi-platform picture.++---++> ✅ 源码已于 **2026-09-30** 开源,本 README 将随源码落库补全完整的构建+> 与使用文档。Windows runtime:edge0 流式核心的原生 C++17 移植(CMake+> 构建)+ **Vulkan** 计算后端,`edge0` 核心(streaming mmap / cache /+> layer / MoE)镜像自上游 Python 树,沿用同一套配方。**统一推理框架**+> 将于 **2026 Q4** 发布,详见[根 README 路线图](../README_zh.md)。