Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs: sync the Chinese README and fix stale docs claims

linyubupac6b6d599b533
diff --git a/README_zh.md b/README_zh.md
index 296bd83..b7d3d36 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -134,6 +134,17 @@ edge0 demo edge0-35b
edge0 serve edge0-35b
```
+```bash
+curl http://127.0.0.1:8000/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
+
+# 5) 单轮对话(--max-new 限制生成长度;加 --show-thinking 会一并打印思考块)
+edge0 chat edge0-35b --prompt "用一句话解释流式推理。"
+```
+
+`python -m edge0 ...` 等价于 `edge0 ...`。
+
### Python API
```python
@@ -176,6 +187,15 @@ engine.close() # 释放 mmap / 专家缓存
与原 fp16 基座模型测得。edge0 管线的损失很小:**edge0-35b 平均仅落后
3.9 分、edge0-8b 落后 2.8 分**(MMLU-Pro 甚至反超基座)。满分 100:
+| 评测集 | edge0-35b(int4) | Qwen3.5-MoE 35B-A3B(fp16) | edge0-8b(int4) | Ling 3.0 tiny(fp16) |
+|---|---:|---:|---:|---:|
+| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
+| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
+| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
+| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
+| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
+| **平均** | **79.2** | **83.2** | **69.9** | **72.7** |
+
## 性能实测
`examples/bench.py` 实测(3.3k token prompt prefill → 10 步采样 warmup →
diff --git a/docs/adding-a-model.md b/docs/adding-a-model.md
index f56a746..6af69b4 100644
--- a/docs/adding-a-model.md
+++ b/docs/adding-a-model.md
@@ -12,7 +12,7 @@ The contract of `edge0.registry` is: **one adapter module exposes three things**
- `build_model(model_dir, **overrides)` — the assembled model skeleton;
- `build_engine(model_dir, **overrides)` — a generation-capable engine.
-`register_model(name, adapter)` performs the registration; `TYPE_ALIASES` maps a checkpoint's `model_type` to a registered name. See `_resolve_name` and `AutoConfig.from_pretrained` in `registry.py` (registry.py:73–101).
+`register_model(name, adapter)` performs the registration; `TYPE_ALIASES` maps a checkpoint's `model_type` to a registered name. See `_resolve_name` (registry.py:43) and `AutoConfig.from_pretrained` (registry.py:87).
## Step 1: Create the adapter module and directory
@@ -36,7 +36,7 @@ Create a package under `src/edge0/models/` named after your tier, e.g. `edge0_35
| `port` | `int` | serving port |
| `target_tok_s` / `peak_active_mem_mb` | `float` | acceptance metrics (measured on the benchmark environment) |
-The subclass only needs to implement the class method `_defaults(model_dir) -> Config`, which returns the family's default configuration. The edge0-35b implementation lives at `edge0_35b/__init__.py:31–69`: `_defaults` builds a `Qwen35Config` from a complete `MoESpec`, the `LayerOptions` preset `staged_k4()`, a `PrerouterSpec`, sampling defaults, and acceptance metrics.
+The subclass only needs to implement the class method `_defaults(model_dir) -> Config`, which returns the family's default configuration. The edge0-35b implementation lives at `edge0_35b/__init__.py:31–70`: `_defaults` builds a `Qwen35Config` from a complete `MoESpec`, the `LayerOptions` preset `staged_k4()`, a `PrerouterSpec`, sampling defaults, and acceptance metrics.
`from_pretrained(model_dir=None, **overrides)` is a template method provided by `ModelConfig` (`models/base.py:60–73`): it calls `_defaults` to obtain the base configuration, then validates each override before applying it via `replace`; unknown fields raise a `TypeError` that lists the known fields. As a result, **every public attribute can be overridden by the user**.
@@ -49,7 +49,7 @@ Key points:
Two module-level functions that reuse the engine's own construction path (DRY).
-`edge0_35b/__init__.py:72–79`:
+`edge0_35b/__init__.py:73–80`:
```python
def build_model(model_dir=None, **overrides):
@@ -59,7 +59,7 @@ def build_model(model_dir=None, **overrides):
return model
```
-`edge0_35b/__init__.py:82–87`:
+`edge0_35b/__init__.py:83–87`:
```python
def build_engine(model_dir=None, **overrides):
@@ -116,8 +116,8 @@ Refer to `tests/test_moe_spec.py` and `tests/test_registry.py` (pure-logic unit
- **Registration and aliases** (test_registry.py:12–20): assert that `MODEL_REGISTRY` contains your registered name and that `TYPE_ALIASES` resolves correctly.
- **Config defaults** (test_registry.py:23–33): the `moe_spec` / `options` / `prerouter` / `prerouter_top_k` fields of `AutoConfig.from_pretrained(name=...)` match expectations.
- **Profile refinement** (test_registry.py:35–71): assert your tier's values field by field (expert count, top_k, quant, port, acceptance metrics, etc.).
-- **Overrides and rejection** (test_registry.py:74–81): `from_pretrained(..., prerouter=None, lora="")` takes effect; unknown fields raise `TypeError`.
-- **Adapter contract** (test_registry.py:89–92): assert that every `MODEL_REGISTRY` entry exposes `Config` / `build_model` / `build_engine`.
+- **Overrides and rejection** (test_registry.py:82–89): `from_pretrained(..., prerouter=None, lora="")` takes effect; unknown fields raise `TypeError`.
+- **Adapter contract** (test_registry.py:97): assert that every `MODEL_REGISTRY` entry exposes `Config` / `build_model` / `build_engine`.
- **Path resolution** (test_moe_spec.py): the `keys` template, `block_of` numeric-field indexing, the `layer_of` default derived from `block_path`, and `bundle_projs` switching with the layout.
- **Math parity**: routing math must match the vendored model bit-for-bit (pinned by parity tests in `moe/routing.py`); if your model introduces a new routing kind, add the corresponding comparison.
diff --git a/docs/architecture.md b/docs/architecture.md
index 4b8cc01..67ac6d3 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -1,7 +1,7 @@
# Architecture Overview
**edge0** is an open-source streaming MoE inference framework that abstracts the
-"SSD expert offload + parallel LoRA + prerouter routing prediction" scheme —
+"SSD expert offload + Recover-LoRA + prerouter routing prediction" scheme —
validated in production deployments — into an extensible, general-purpose
framework. Backends are isolated by design: the current release implements the
MLX backend (Apple Silicon), core logic is decoupled from the backend, and other
@@ -22,7 +22,9 @@ box: `edge0-35b` (Qwen3.5-MoE, K=4) and `edge0-8b` (Ling 3.0 hybrid, K=8).
`.safetensors` files carrying metadata; legacy npz training exports are
converted by a one-off migration script and then deprecated;
4. **Unified terminology**: the pre-routing head is always called the
- **prerouter** — zero legacy-term residue in code and docs.
+ **prerouter** — zero legacy-term residue in code and docs. The legacy npz
+ exports are migrated by `scripts/convert_adapters_legacy.py`, which
+ normalizes their key namespace to `layers.<N>.<part>.weight`.
## Layered Structure
@@ -54,7 +56,7 @@ src/edge0/
│ ├── base.py # shared loop (via backends.core)
│ ├── qwen.py # Qwen35Engine (K=4 tier)
│ └── ling.py # Ling8BEngine (K=8 tier)
-├── adapters/lora.py # parallel LoRA installation (backends.nn)
+├── adapters/lora.py # unmerged LoRA installation (parallel delta path)
├── models/ # model adapter layer (transformers-style)
│ ├── base.py # ModelConfig base class + artifact path resolution
│ ├── edge0_35b/ # edge0-35b tier (registered name "edge0-35b")
diff --git a/docs/models/edge0-35b.md b/docs/models/edge0-35b.md
index ac227c2..b95a24a 100644
--- a/docs/models/edge0-35b.md
+++ b/docs/models/edge0-35b.md
@@ -24,7 +24,7 @@ The performance profile is based on benchmarks of the current release adapter ve
| Streaming prefetch history | on (`prefetch_history=True`) |
| Serving port | 8085 |
| Measured throughput | 14.9–17.7 tok/s (M4 Pro) |
-| Measured peak activation memory | ≈ 3.3 GB |
+| Measured peak activation memory | ≈ 2.9 GiB |
> Note: all values are taken from `Qwen35Config._defaults()` and `LayerOptions.staged_k4()`. The head count comes from the explicit `owners` list (6 through 38, 33 heads in total); `feature_topk="executed"` means the top-k features fed to the heads are exactly the set actually routed at decode time.
diff --git a/docs/models/edge0-8b.md b/docs/models/edge0-8b.md
index 2a4097b..470e2cc 100644
--- a/docs/models/edge0-8b.md
+++ b/docs/models/edge0-8b.md
@@ -24,7 +24,7 @@ The performance profile is based on benchmarks of the current release adapter ve
| Streaming prefetch history | on (`prefetch_history=True`) |
| Serving port | 8083 |
| Measured throughput | 23.9–25.3 tok/s (M4 Pro) |
-| Measured peak activation memory | ≈ 1.0 GB (short context) / 3.1 GB (3.3k-token context) |
+| Measured peak activation memory | ≈ 1.0 GiB (short context) / 3.3 GiB (3.3k-token context) |
> Note: all values are taken from `Ling8BConfig._defaults()` and `LayerOptions.prod_k8()`. The head count comes from the explicit `owners=range(7, 23)` — 16 heads in total (the current release head distribution: prediction is consumed from L7 onward, and L1–6 use the original router); `feature_topk="executed"`.
diff --git a/docs/prerouter.md b/docs/prerouter.md
index 6bf2d84..460a1da 100644
--- a/docs/prerouter.md
+++ b/docs/prerouter.md
@@ -54,7 +54,7 @@ per `start_layer`/`owners`. Family differences are expressed through three hooks
- `LayerOptions.staged_replace=True`: the routed set == the staged set (prerouter mode,
no drop);
-- Production profiles (`staged_k4`/`staged_k8`) use **explicit-index** routing: prerouter
+- Production profiles (`staged_k4`/`prod_k8`) use **explicit-index** routing: prerouter
predictions directly serve as the staged fill source, `staged_replace=False` but the
slot table maps exactly — likewise zero drop;
- `pin_bonus`: experts predicted by the prerouter get an extra bonus during hot pin
@@ -65,12 +65,16 @@ per `start_layer`/`owners`. Family differences are expressed through three hooks
| | edge0-35b | edge0-8b |
|---|---|---|
| Routing family | SOFTMAX_TOPK (precise softmax → top-k → normalization) | SIGMOID_GROUP (sigmoid + group-limited top-k, n_group 8 / topk_group 4, routed_scaling 2.5) |
-| start_layer | 7 (first consumer layer) | 1 |
-| heads (owners) | 33 (6..38) | 22 (1..22) |
+| start_layer | 7 (first consumer layer) | 7 |
+| heads (owners) | 33 (6..38) | 16 (7..22) |
| hidden | 512 (fp16) | 512 (fp16) |
| feature_topk | executed | executed |
| patch_call | True (qwen3_next needs the patch) | False (built-in model hook) |
-| weight file | `artifacts/prerouter_edge0_35b_k4.safetensors` | `artifacts/prerouter_edge0_8b.safetensors` |
+| weight file | `prerouter_edge0_35b.safetensors` | `prerouter_edge0_8b.safetensors` |
+
+Both weight files are resolved from the model directory by default, with
+`artifacts/` (repo root, gitignored) as the fallback — see the README's
+"Models and adapters" section.
## Tests and regression
diff --git a/docs/streaming.md b/docs/streaming.md
index 8ab41a7..149502f 100644
--- a/docs/streaming.md
+++ b/docs/streaming.md
@@ -50,7 +50,7 @@ Deduplicate the routing indices → `_get_bundles(unique)` builds per expert (re
- `load_full_layer()`: loads the layer's 9 tensors directly (the checkpoint already stores each layer as a single stacked tensor; the mmap dtype conversion costs ≈9ms per layer, and the CPU load is hidden under the previous layer's GPU execution);
- `_gather_sort` folds the batch into the token dimension → sorted gather → `_scatter_unsort` un-sorts and restores the batch dimension;
-- `prefill_full_layers` loads whole layers only for the leading layers (12 for Qwen); the remaining layers go through hot/exact;
+- `full_layer_prefill` turns whole-layer prefill on per tier, and `prefill_full_layers` limits it to the leading N layers (0 = every layer, the edge0-8b setting); layers outside that range go through hot/exact;
- After use, `clear_full_layer()` frees the GPU copy, and the page cache carries the hot data.
## Sorting and Compilation
@@ -75,7 +75,7 @@ With `use_compile`, the staged/exact paths are wrapped in `mx.compile`:
| `prefill_hot` | hot stack size during prefill | 0 |
| `use_compile` / `top_k` | compile wrapping / routing top-k override | True / None |
-Presets: `staged_k4()` (edge0-35b: staged with 4 slots + 32 hot + whole-layer prefill over 12 layers), `staged_k8()` (edge0-8b: staged with 8 slots, no hot residency). Both tiers share `cache_slots=64`.
+Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile — staged decode off, E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.
## Why Whole-Layer Loading Is Also Fast