Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
clean up convert_adapters command
Bind464ceb737f1
diff --git a/README.md b/README.mdindex f1cb73c..1d5b87b 100644--- a/README.md+++ b/README.md@@ -265,8 +265,8 @@ runs this exact path).location automatically:- the model directory (recommended): side by side with the base, e.g.`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- - `artifacts/` (gitignored): convert once from training-side npz- exports via `edge0 convert-adapters --npz-dir ...`.+ - `artifacts/` at the Python project root (gitignored): an optional+ fallback cache for adapter safetensors not co-located with the model.- The published model repos bundle both the base checkpoint and thecurrent default adapter release, so `scripts/fetch_models.py` producesa ready-to-run model directory. Check each model's doc page for itsdiff --git a/README_zh.md b/README_zh.mdindex 37f615f..6b76609 100644--- a/README_zh.md+++ b/README_zh.md@@ -257,8 +257,8 @@ engine.close() # 释放 mmap / 专家缓存- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:- **模型目录内**(推荐):与基模同目录,如`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- - `artifacts/`(gitignored):`edge0 convert-adapters` 从训练侧 npz- 一次性转换。+ - `artifacts/`(Python 项目根,gitignored):可选的回退缓存,放置未+ 与模型同目录的适配器 safetensors。- 发布的模型仓库同时包含基模与当前默认适配器版本,`scripts/fetch_models.py` 下载后即为可运行的模型目录。适配器来源(训练数据、owner 层分布)见各模型文档页。diff --git a/docs/adding-a-model.md b/docs/adding-a-model.mdindex f14ac8c..be304bb 100644--- a/docs/adding-a-model.md+++ b/docs/adding-a-model.md@@ -42,7 +42,7 @@ The subclass only needs to implement the class method `_defaults(model_dir) -> CKey points:-- The Python project root (`python/`) ships `scripts/convert_adapters_legacy.py`, which one-shot converts the training npz exports into safetensors artifacts under `artifacts/`; `ModelConfig.artifact(name)` (`models/base.py:28–30`) returns the absolute paths of those artifacts, and the adapter uses them to fill in the LoRA / prerouter weight paths.+- `ModelConfig.artifact(name)` (`models/base.py`) resolves an adapter's absolute path — from the model directory first, falling back to the gitignored `artifacts/` cache — and the adapter uses it to fill in the LoRA / prerouter weight paths.- LoRA overrides go through `resolve_lora` (`models/base.py:88–95`): a bare model name resolves to that tier's artifacts, `"model_dir"` means "keep the training weights in place", and an empty string disables it.## Step 3: `build_model` / `build_engine`@@ -105,7 +105,7 @@ The lookup order of `_resolve_name` (registry.py:42–56): explicit `name` → t- `config.json` exists, and its `model_type` is registered (or covered by your `TYPE_ALIASES`);- the weights are in safetensors format, with key prefixes matching `moe_spec.key_template` (e.g. `language_model.model.layers.N.mlp.switch_mlp`);-- the LoRA / prerouter weights are `.safetensors` files carrying metadata (see the comment at the top of `models/base.py`: they are converted one-shot from the training npz by `convert_adapters_legacy.py`).+- the LoRA / prerouter weights are `.safetensors` files carrying metadata (see the comment at the top of `models/base.py`).The directory basename is the last-resort resolution fallback (`_model_type_from_dir`), so aligning the directory name with the registered name is recommended, but not required.diff --git a/docs/architecture.md b/docs/architecture.mdindex 3a07c42..c5adc22 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -19,12 +19,10 @@ box: `edge0-35b` (Qwen3.6-35B-A3B, K=4) and `edge0-8b` (Ling 3.0 hybrid, K=8).future CUDA backend (`backends/cuda/`, selected via the `EDGE0_BACKEND`environment variable);3. **Adapters unified on safetensors**: LoRA and prerouter weights are both- `.safetensors` files carrying metadata; legacy npz training exports are- converted by a one-off migration script and then deprecated;+ `.safetensors` files carrying metadata;4. **Unified terminology**: the pre-routing head is always called the- **prerouter** — zero legacy-term residue in code and docs. The legacy npz- exports are migrated by `scripts/convert_adapters_legacy.py`, which- normalizes their key namespace to `layers.<N>.<part>.weight`.+ **prerouter** — zero legacy-term residue in code and docs. Prerouter+ weight keys are normalized to `layers.<N>.<part>.weight`.## Layered Structurediff --git a/python/scripts/convert_adapters_legacy.py b/python/scripts/convert_adapters_legacy.pydeleted file mode 100644index 09c1b96..0000000--- a/python/scripts/convert_adapters_legacy.py+++ /dev/null@@ -1,159 +0,0 @@-#!/usr/bin/env python3-"""One-shot legacy adapter converter: npz -> safetensors.--The training pipelines exported adapters as ``.npz`` (qwen round-7 and-the training side). edge0 consumes safetensors ONLY — this script converts each-legacy pair once into ``artifacts/`` (gitignored), writing the-conversion provenance (model, kind, K, r/alpha, source md5, format-version) into the safetensors ``__metadata__``. The npz sources are-NOT deleted (see the conversion note in README); they are only needed-to regenerate the artifacts.--Usage:- python scripts/convert_adapters_legacy.py [--force]--Idempotent: existing artifacts are skipped unless ``--force``.-"""--from __future__ import annotations--import argparse-import hashlib-import os-import json-import sys-from datetime import datetime, timezone-from pathlib import Path--import numpy as np--ROOT = Path(__file__).resolve().parents[1]-ARTIFACTS = ROOT / "artifacts"--# (npz path, tier, kind) — every legacy source this script understands.-# Training-side npz exports discovered per machine; pass --npz-dir (or-# edit this list) to point at your own exports.-NPZ_DIRS = [- Path(os.environ.get("EDGE0_NPZ_DIR", "/path/to/npz/exports")),-]-SOURCES = [- ("prerouter_<family>.npz", "edge0-35b", "prerouter"),- ("lora_<family>.npz", "edge0-35b", "lora"),- ("prerouter_<family>.npz", "edge0-8b", "prerouter"),- ("lora_<family>.npz", "edge0-8b", "lora"),-]---def _find_source(name: str) -> Path | None:- for d in NPZ_DIRS:- p = d / name- if p.is_file():- return p- return None---def md5(path: Path) -> str:- h = hashlib.md5()- with open(path, "rb") as f:- for chunk in iter(lambda: f.read(1 << 20), b""):- h.update(chunk)- return h.hexdigest()---def rewrite_key(key: str, tier: str, kind: str) -> str:- """Map a legacy npz key to the edge0 artifact key.-- prerouter heads are normalized to ``layers.<N>.<part>.weight``;- qwen lora gains the ``language_model.`` prefix (the checkpoint- namespace the adapter is applied into).- """- if kind == "prerouter":- # Some families use layers.N.*; others namespace the layer index- # with a prefix segment (<ns>.<N>.<part>.weight). Both are- # normalized to layers.<N>.<part>.weight.- parts = key.split(".")- if (len(parts) >= 4 and parts[0] != "layers"- and parts[1].isdigit()):- return f"layers.{parts[1]}.{parts[2]}.{parts[3]}"- return key- if kind == "lora" and tier == "edge0-35b" and key.startswith("model.layers."):- return "language_model." + key- return key---def convert(source: Path, tier: str, kind: str, force: bool = False) -> Path:- name = {- ("edge0-35b", "prerouter"): "prerouter_edge0_35b_k4.safetensors",- ("edge0-35b", "lora"): "lora_edge0_35b_k4.safetensors",- ("edge0-8b", "prerouter"): "prerouter_edge0_8b.safetensors",- ("edge0-8b", "lora"): "lora_edge0_8b.safetensors",- }[(tier, kind)]- out = ARTIFACTS / name- if out.exists() and not force:- print(f"skip {out.name} (exists)")- return out-- data = np.load(source)- tensors = {}- for key in data.files:- tensors[rewrite_key(key, tier, kind)] = data[key]-- # provenance: rank from the A matrix, alpha from the training run- # (32 for both shipped pairs), K from the tier profile.- meta = {- "model": tier,- "kind": kind,- "K": "4" if tier == "edge0-35b" else "8",- "r": "16", "alpha": "32",- "source": str(source),- "source_md5": md5(source),- "converted": datetime.now(timezone.utc).isoformat(timespec="seconds"),- "format_version": "1",- }- if kind == "prerouter":- owners = sorted({int(k.split(".")[1])- for k in tensors if k.startswith("layers.")})- meta["owners"] = json.dumps(owners)-- from safetensors.numpy import save_file as save_np- save_np(tensors, str(out), metadata={"__metadata__": json.dumps(meta)})- print(f"write {out.name} ({len(tensors)} tensors, "- f"{out.stat().st_size >> 20} MiB)")- return out---def main(argv: list[str] | None = None) -> int:- ap = argparse.ArgumentParser(description=__doc__)- ap.add_argument("--force", action="store_true",- help="reconvert even if the artifact exists")- ap.add_argument("--npz-dir", default=None,- help="directory holding the legacy npz exports "- "(default: $EDGE0_NPZ_DIR)")- args = ap.parse_args(argv)-- if args.npz_dir:- NPZ_DIRS.insert(0, Path(args.npz_dir))- ARTIFACTS.mkdir(parents=True, exist_ok=True)- missing = []- resolved = []- for name, tier, kind in SOURCES:- p = _find_source(name)- if p is None:- missing.append(name)- else:- resolved.append((p, tier, kind))- if missing:- print("missing legacy sources (skip their conversion):")- for m in missing:- print(" -", m)- for path, tier, kind in resolved:- try:- convert(path, tier, kind, force=args.force)- except Exception as exc: # noqa: BLE001 — report and continue- print(f"FAIL {path.name}: {exc}", file=sys.stderr)- return 1- return 0---if __name__ == "__main__":- raise SystemExit(main())diff --git a/python/src/edge0/adapters/lora.py b/python/src/edge0/adapters/lora.pyindex b13bf32..1103dd8 100644--- a/python/src/edge0/adapters/lora.py+++ b/python/src/edge0/adapters/lora.py@@ -16,9 +16,7 @@ Safetensors key naming (framework contract)::where ``<target>`` is the module path inside the loaded model (e.g.``model.layers.3.attention.q_proj`` or-``language_model.model.layers.0.linear_attn.in_proj_qkv``). The legacy-npz files used the same A/B naming, so conversion is a pure format-migration (see ``scripts/convert_adapters_legacy.py``).+``language_model.model.layers.0.linear_attn.in_proj_qkv``)."""from __future__ import annotationsdiff --git a/python/src/edge0/cli.py b/python/src/edge0/cli.pyindex 5b3ec54..097442c 100644--- a/python/src/edge0/cli.py+++ b/python/src/edge0/cli.py@@ -6,7 +6,6 @@ Commands:edge0 serve start the HTTP server (one model, queued generations)edge0 chat one-shot prompt -> answer on the terminaledge0 models list registered tiers and their default profiles- edge0 convert-adapters one-shot legacy npz -> safetensors migration"""from __future__ import annotations@@ -208,25 +207,6 @@ def cmd_serve(args) -> int:return 0-def cmd_convert(args) -> int:- from pathlib import Path-- scripts_dir = Path(__file__).resolve().parents[2] / "scripts"- if str(scripts_dir) not in sys.path:- sys.path.insert(0, str(scripts_dir))- # Delegate to the legacy converter's own main(argv) with a clean argv,- # so the subcommand name never leaks into its argparse (runpy would- # re-parse sys.argv and choke on "convert-adapters").- from convert_adapters_legacy import main as _convert_main-- argv: list[str] = []- if getattr(args, "force", False):- argv.append("--force")- if getattr(args, "npz_dir", None):- argv += ["--npz-dir", args.npz_dir]- return _convert_main(argv)--def _add_engine_flags(p) -> None:"""Engine-construction flags shared by demo / chat / serve."""p.add_argument("--no-prerouter", action="store_true")@@ -295,15 +275,6 @@ def _build_parser() -> argparse.ArgumentParser:_add_engine_flags(p)p.set_defaults(fn=cmd_serve)- p = sub.add_parser("convert-adapters",- help="one-shot legacy npz -> safetensors migration")- p.add_argument("--force", action="store_true",- help="reconvert even if the artifact exists")- p.add_argument("--npz-dir", default=None,- help="directory holding the legacy npz exports "- "(default: $EDGE0_NPZ_DIR)")- p.set_defaults(fn=cmd_convert)-return apdiff --git a/python/src/edge0/models/base.py b/python/src/edge0/models/base.pyindex bb1d73d..d6ede85 100644--- a/python/src/edge0/models/base.py+++ b/python/src/edge0/models/base.py@@ -19,16 +19,16 @@ from edge0.moe.spec import MoESpecfrom edge0.prerouter.spec import PrerouterSpecfrom edge0.streaming.options import LayerOptions-# Project-root artifacts (gitignored; produced once by-# scripts/convert_adapters_legacy.py from the training npz exports, then-# the source npz are discarded).+# Project-root artifacts (gitignored): an optional local cache of adapter+# safetensors. Adapters normally ship inside the model directory; this is+# only the fallback location artifact() returns when they are not there.ARTIFACTS_DIR = Path(__file__).resolve().parents[3] / "artifacts"def artifact(name: str, model_dir: str | None = None) -> str:"""Path of one adapter file: the model's own directory first(model and its adapters side by side in one- directory), falling back to ``artifacts/`` (conversion cache)."""+ directory), falling back to ``artifacts/`` (optional local cache)."""if model_dir:cand = Path(model_dir) / nameif cand.is_file():