Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: sync the Chinese README and fix stale docs claims
linyubupac6b6d599b533
diff --git a/README_zh.md b/README_zh.mdindex 296bd83..b7d3d36 100644--- a/README_zh.md+++ b/README_zh.md@@ -134,6 +134,17 @@ edge0 demo edge0-35bedge0 serve edge0-35b```+```bash+curl http://127.0.0.1:8000/v1/chat/completions \+ -H 'Content-Type: application/json' \+ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'++# 5) 单轮对话(--max-new 限制生成长度;加 --show-thinking 会一并打印思考块)+edge0 chat edge0-35b --prompt "用一句话解释流式推理。"+```++`python -m edge0 ...` 等价于 `edge0 ...`。+### Python API```python@@ -176,6 +187,15 @@ engine.close() # 释放 mmap / 专家缓存与原 fp16 基座模型测得。edge0 管线的损失很小:**edge0-35b 平均仅落后3.9 分、edge0-8b 落后 2.8 分**(MMLU-Pro 甚至反超基座)。满分 100:+| 评测集 | edge0-35b(int4) | Qwen3.5-MoE 35B-A3B(fp16) | edge0-8b(int4) | Ling 3.0 tiny(fp16) |+|---|---:|---:|---:|---:|+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |+| **平均** | **79.2** | **83.2** | **69.9** | **72.7** |+## 性能实测`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →diff --git a/docs/adding-a-model.md b/docs/adding-a-model.mdindex f56a746..6af69b4 100644--- a/docs/adding-a-model.md+++ b/docs/adding-a-model.md@@ -12,7 +12,7 @@ The contract of `edge0.registry` is: **one adapter module exposes three things**- `build_model(model_dir, **overrides)` — the assembled model skeleton;- `build_engine(model_dir, **overrides)` — a generation-capable engine.-`register_model(name, adapter)` performs the registration; `TYPE_ALIASES` maps a checkpoint's `model_type` to a registered name. See `_resolve_name` and `AutoConfig.from_pretrained` in `registry.py` (registry.py:73–101).+`register_model(name, adapter)` performs the registration; `TYPE_ALIASES` maps a checkpoint's `model_type` to a registered name. See `_resolve_name` (registry.py:43) and `AutoConfig.from_pretrained` (registry.py:87).## Step 1: Create the adapter module and directory@@ -36,7 +36,7 @@ Create a package under `src/edge0/models/` named after your tier, e.g. `edge0_35| `port` | `int` | serving port || `target_tok_s` / `peak_active_mem_mb` | `float` | acceptance metrics (measured on the benchmark environment) |-The subclass only needs to implement the class method `_defaults(model_dir) -> Config`, which returns the family's default configuration. The edge0-35b implementation lives at `edge0_35b/__init__.py:31–69`: `_defaults` builds a `Qwen35Config` from a complete `MoESpec`, the `LayerOptions` preset `staged_k4()`, a `PrerouterSpec`, sampling defaults, and acceptance metrics.+The subclass only needs to implement the class method `_defaults(model_dir) -> Config`, which returns the family's default configuration. The edge0-35b implementation lives at `edge0_35b/__init__.py:31–70`: `_defaults` builds a `Qwen35Config` from a complete `MoESpec`, the `LayerOptions` preset `staged_k4()`, a `PrerouterSpec`, sampling defaults, and acceptance metrics.`from_pretrained(model_dir=None, **overrides)` is a template method provided by `ModelConfig` (`models/base.py:60–73`): it calls `_defaults` to obtain the base configuration, then validates each override before applying it via `replace`; unknown fields raise a `TypeError` that lists the known fields. As a result, **every public attribute can be overridden by the user**.@@ -49,7 +49,7 @@ Key points:Two module-level functions that reuse the engine's own construction path (DRY).-`edge0_35b/__init__.py:72–79`:+`edge0_35b/__init__.py:73–80`:```pythondef build_model(model_dir=None, **overrides):@@ -59,7 +59,7 @@ def build_model(model_dir=None, **overrides):return model```-`edge0_35b/__init__.py:82–87`:+`edge0_35b/__init__.py:83–87`:```pythondef build_engine(model_dir=None, **overrides):@@ -116,8 +116,8 @@ Refer to `tests/test_moe_spec.py` and `tests/test_registry.py` (pure-logic unit- **Registration and aliases** (test_registry.py:12–20): assert that `MODEL_REGISTRY` contains your registered name and that `TYPE_ALIASES` resolves correctly.- **Config defaults** (test_registry.py:23–33): the `moe_spec` / `options` / `prerouter` / `prerouter_top_k` fields of `AutoConfig.from_pretrained(name=...)` match expectations.- **Profile refinement** (test_registry.py:35–71): assert your tier's values field by field (expert count, top_k, quant, port, acceptance metrics, etc.).-- **Overrides and rejection** (test_registry.py:74–81): `from_pretrained(..., prerouter=None, lora="")` takes effect; unknown fields raise `TypeError`.-- **Adapter contract** (test_registry.py:89–92): assert that every `MODEL_REGISTRY` entry exposes `Config` / `build_model` / `build_engine`.+- **Overrides and rejection** (test_registry.py:82–89): `from_pretrained(..., prerouter=None, lora="")` takes effect; unknown fields raise `TypeError`.+- **Adapter contract** (test_registry.py:97): assert that every `MODEL_REGISTRY` entry exposes `Config` / `build_model` / `build_engine`.- **Path resolution** (test_moe_spec.py): the `keys` template, `block_of` numeric-field indexing, the `layer_of` default derived from `block_path`, and `bundle_projs` switching with the layout.- **Math parity**: routing math must match the vendored model bit-for-bit (pinned by parity tests in `moe/routing.py`); if your model introduces a new routing kind, add the corresponding comparison.diff --git a/docs/architecture.md b/docs/architecture.mdindex 4b8cc01..67ac6d3 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -1,7 +1,7 @@# Architecture Overview**edge0** is an open-source streaming MoE inference framework that abstracts the-"SSD expert offload + parallel LoRA + prerouter routing prediction" scheme —+"SSD expert offload + Recover-LoRA + prerouter routing prediction" scheme —validated in production deployments — into an extensible, general-purposeframework. Backends are isolated by design: the current release implements theMLX backend (Apple Silicon), core logic is decoupled from the backend, and other@@ -22,7 +22,9 @@ box: `edge0-35b` (Qwen3.5-MoE, K=4) and `edge0-8b` (Ling 3.0 hybrid, K=8).`.safetensors` files carrying metadata; legacy npz training exports areconverted by a one-off migration script and then deprecated;4. **Unified terminology**: the pre-routing head is always called the- **prerouter** — zero legacy-term residue in code and docs.+ **prerouter** — zero legacy-term residue in code and docs. The legacy npz+ exports are migrated by `scripts/convert_adapters_legacy.py`, which+ normalizes their key namespace to `layers.<N>.<part>.weight`.## Layered Structure@@ -54,7 +56,7 @@ src/edge0/│ ├── base.py # shared loop (via backends.core)│ ├── qwen.py # Qwen35Engine (K=4 tier)│ └── ling.py # Ling8BEngine (K=8 tier)-├── adapters/lora.py # parallel LoRA installation (backends.nn)+├── adapters/lora.py # unmerged LoRA installation (parallel delta path)├── models/ # model adapter layer (transformers-style)│ ├── base.py # ModelConfig base class + artifact path resolution│ ├── edge0_35b/ # edge0-35b tier (registered name "edge0-35b")diff --git a/docs/models/edge0-35b.md b/docs/models/edge0-35b.mdindex ac227c2..b95a24a 100644--- a/docs/models/edge0-35b.md+++ b/docs/models/edge0-35b.md@@ -24,7 +24,7 @@ The performance profile is based on benchmarks of the current release adapter ve| Streaming prefetch history | on (`prefetch_history=True`) || Serving port | 8085 || Measured throughput | 14.9–17.7 tok/s (M4 Pro) |-| Measured peak activation memory | ≈ 3.3 GB |+| Measured peak activation memory | ≈ 2.9 GiB |> Note: all values are taken from `Qwen35Config._defaults()` and `LayerOptions.staged_k4()`. The head count comes from the explicit `owners` list (6 through 38, 33 heads in total); `feature_topk="executed"` means the top-k features fed to the heads are exactly the set actually routed at decode time.diff --git a/docs/models/edge0-8b.md b/docs/models/edge0-8b.mdindex 2a4097b..470e2cc 100644--- a/docs/models/edge0-8b.md+++ b/docs/models/edge0-8b.md@@ -24,7 +24,7 @@ The performance profile is based on benchmarks of the current release adapter ve| Streaming prefetch history | on (`prefetch_history=True`) || Serving port | 8083 || Measured throughput | 23.9–25.3 tok/s (M4 Pro) |-| Measured peak activation memory | ≈ 1.0 GB (short context) / 3.1 GB (3.3k-token context) |+| Measured peak activation memory | ≈ 1.0 GiB (short context) / 3.3 GiB (3.3k-token context) |> Note: all values are taken from `Ling8BConfig._defaults()` and `LayerOptions.prod_k8()`. The head count comes from the explicit `owners=range(7, 23)` — 16 heads in total (the current release head distribution: prediction is consumed from L7 onward, and L1–6 use the original router); `feature_topk="executed"`.diff --git a/docs/prerouter.md b/docs/prerouter.mdindex 6bf2d84..460a1da 100644--- a/docs/prerouter.md+++ b/docs/prerouter.md@@ -54,7 +54,7 @@ per `start_layer`/`owners`. Family differences are expressed through three hooks- `LayerOptions.staged_replace=True`: the routed set == the staged set (prerouter mode,no drop);-- Production profiles (`staged_k4`/`staged_k8`) use **explicit-index** routing: prerouter+- Production profiles (`staged_k4`/`prod_k8`) use **explicit-index** routing: prerouterpredictions directly serve as the staged fill source, `staged_replace=False` but theslot table maps exactly — likewise zero drop;- `pin_bonus`: experts predicted by the prerouter get an extra bonus during hot pin@@ -65,12 +65,16 @@ per `start_layer`/`owners`. Family differences are expressed through three hooks| | edge0-35b | edge0-8b ||---|---|---|| Routing family | SOFTMAX_TOPK (precise softmax → top-k → normalization) | SIGMOID_GROUP (sigmoid + group-limited top-k, n_group 8 / topk_group 4, routed_scaling 2.5) |-| start_layer | 7 (first consumer layer) | 1 |-| heads (owners) | 33 (6..38) | 22 (1..22) |+| start_layer | 7 (first consumer layer) | 7 |+| heads (owners) | 33 (6..38) | 16 (7..22) || hidden | 512 (fp16) | 512 (fp16) || feature_topk | executed | executed || patch_call | True (qwen3_next needs the patch) | False (built-in model hook) |-| weight file | `artifacts/prerouter_edge0_35b_k4.safetensors` | `artifacts/prerouter_edge0_8b.safetensors` |+| weight file | `prerouter_edge0_35b.safetensors` | `prerouter_edge0_8b.safetensors` |++Both weight files are resolved from the model directory by default, with+`artifacts/` (repo root, gitignored) as the fallback — see the README's+"Models and adapters" section.## Tests and regressiondiff --git a/docs/streaming.md b/docs/streaming.mdindex 8ab41a7..149502f 100644--- a/docs/streaming.md+++ b/docs/streaming.md@@ -50,7 +50,7 @@ Deduplicate the routing indices → `_get_bundles(unique)` builds per expert (re- `load_full_layer()`: loads the layer's 9 tensors directly (the checkpoint already stores each layer as a single stacked tensor; the mmap dtype conversion costs ≈9ms per layer, and the CPU load is hidden under the previous layer's GPU execution);- `_gather_sort` folds the batch into the token dimension → sorted gather → `_scatter_unsort` un-sorts and restores the batch dimension;-- `prefill_full_layers` loads whole layers only for the leading layers (12 for Qwen); the remaining layers go through hot/exact;+- `full_layer_prefill` turns whole-layer prefill on per tier, and `prefill_full_layers` limits it to the leading N layers (0 = every layer, the edge0-8b setting); layers outside that range go through hot/exact;- After use, `clear_full_layer()` frees the GPU copy, and the page cache carries the hot data.## Sorting and Compilation@@ -75,7 +75,7 @@ With `use_compile`, the staged/exact paths are wrapped in `mx.compile`:| `prefill_hot` | hot stack size during prefill | 0 || `use_compile` / `top_k` | compile wrapping / routing top-k override | True / None |-Presets: `staged_k4()` (edge0-35b: staged with 4 slots + 32 hot + whole-layer prefill over 12 layers), `staged_k8()` (edge0-8b: staged with 8 slots, no hot residency). Both tiers share `cache_slots=64`.+Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile — staged decode off, E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.## Why Whole-Layer Loading Is Also Fast