Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
Open-source readiness: EN README, zero hardcoded paths, demo-profile parity
linyubupa2de963b20122
diff --git a/README.md b/README.mdindex 7533ab2..f28e7df 100644--- a/README.md+++ b/README.md@@ -1,118 +1,146 @@# edge0-**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload +-并行 LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,-当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)按同一套核心抽象接入。+[English](README.md) | [中文](README_zh.md)-开箱支持两个模型 tier:+**edge0** is an open-source streaming MoE inference framework. It+generalizes the production-proven recipe — **SSD expert offload ++parallel LoRA + prerouter routing prediction** — into an extensible+framework. The backend is isolated by design: the current MLX backend+runs on Apple Silicon, and additional platforms (CUDA, …) plug into the+same core abstractions.-| 模型 | 说明 | 推理档 |+Two model tiers work out of the box:++| Tier | Model | Inference profile ||---|---|---|-| `edge0-35b` | Qwen3.6-35B-A3B 4bit(40 层,256 专家) | prerouter K=4 |-| `edge0-10b` | Ling-10B bailing_hybrid(24 层,128 专家) | prerouter K=8 |--## 设计--- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`- 按模型名自动选类;-- **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,核心逻辑- (模型 spec / prerouter / 流式专家池 / server)只依赖后端门面- (`edge0/backends/base.py` 的 `core` / `nn` 门面),新增后端实现同一门面- 即可平级接入(`backends/cuda/` 预留插槽),核心代码零改动;-- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据- (来源、轮次、owners)的 `.safetensors`,放模型目录或 `artifacts/`- 均可自动解析;-- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /- `model*.safetensors` / tokenizer)和该模型的适配器,换轮次只换适配器- 文件,基模不动、不 merge。--## 核心机制--- **SSD 专家 offload**:MoE 专家权重从磁盘 mmap 按需流式加载,- 激活集驻留 LRU,长尾专家按层预取 —— 大模型小显存跑得动;-- **prerouter 路由预判**:前一 token 的隐层状态经轻量头预测下一 token- 的专家路由,SSD 预取与下一层前向重叠,路由零等待- (qwen 系 `start_layer=7`,ling 系同);-- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,- 基模保持只读 mmap、多套适配器共享同一份基模;-- **数值防护**:per-layer hidden clip(`LING_HIDDEN_CLIP`,默认 1000)- 阻断 fp16 溢出 → 全 NaN logits → token-0 死循环的塌缩链。--## 快速开始+| `edge0-35b` | Qwen3.6-35B-A3B 4-bit (40 layers, 256 experts) | prerouter K=4 |+| `edge0-10b` | Ling-10B 4-bit bailing hybrid (24 layers, 128 experts) | prerouter K=8 |++## Design++- **transformers-style usage**: `AutoModel` / `AutoConfig` / `AutoEngine`+ resolve the tier from the model name;+- **Backend isolation**: all MLX code lives under `edge0/backends/mlx/`;+ the core logic (model specs, prerouter, streaming expert pool, server)+ depends only on the backend facade (`edge0/backends/base.py`), so a new+ backend implements the same facade (`backends/cuda/` is a reserved+ slot) with zero changes to core code;+- **Adapters as safetensors**: LoRA and prerouter weights are+ `.safetensors` files with provenance metadata (source, round, owners),+ resolved from the model directory or `artifacts/`;+- **Model + adapters in one directory**: a model directory holds both+ the base checkpoint (`config.json` / `model*.safetensors` / tokenizer)+ and that model's adapters; switching training rounds swaps adapter+ files only — the base stays read-only and is never merged.++## Core mechanisms++- **SSD expert offload**: MoE expert weights are mmapped from disk and+ streamed on demand; the active set stays resident in an LRU and+ long-tail experts are prefetched per layer — large models run in+ modest memory;+- **Prerouter routing prediction**: a lightweight head predicts the next+ token's expert routing from the previous token's hidden state, so SSD+ prefetch overlaps the next forward pass with zero routing latency+ (`start_layer=7` on both tiers);+- **Parallel LoRA**: adapters are applied as a side path at forward+ time instead of being merged — the base stays a read-only mmap and+ multiple adapter sets share one base;+- **Numerical guard**: per-layer hidden clipping (`LING_HIDDEN_CLIP`,+ default 1000) breaks the fp16 overflow → all-NaN logits → token-0+ death-spiral collapse chain.++## Quick start```bash-# 1) 安装(Python ≥3.10;MLX 后端需 mlx 环境当前支持的平台)+# 1) Install (Python >= 3.10; MLX backend requires an mlx-supported platform)python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'-# 2) 一键试玩:自动探测本机 checkpoint;也可以显式传路径或 tier 名-edge0 demo+# 2) Quick demo: pass a checkpoint directory or tier nameedge0 demo /path/to/qwen35/model-edge0 demo edge0-10b+edge0 demo edge0-10b # checkpoint resolved via env vars (below)-# 3) vLLM 风格起服务:模型是位置参数,checkpoint 类型自动识别+# 3) Serve (OpenAI-compatible /v1/chat/completions; the model is a+# positional argument, tier auto-detected from config.json)edge0 serve /path/to/qwen35/model-edge0 serve edge0-35b # tier 名:EDGE0_35B_MODEL 环境变量可覆盖默认路径+edge0 serve edge0-35b+```++`python -m edge0 ...` is equivalent to `edge0 ...`.+Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint+directories through environment variables; without them, the tier is+auto-detected from the checkpoint's `config.json`:++```bash+export EDGE0_35B_MODEL=/path/to/qwen35/model+export EDGE0_10B_MODEL=/path/to/ling/model+```++```bashcurl http://127.0.0.1:8000/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'-# 4) 命令行一问一答(同样接受路径或 tier 名)-edge0 chat /path/to/model --prompt "你好,用一句话介绍珠海。"+# 4) One-shot chat+edge0 chat /path/to/model --prompt "Explain streaming inference in one sentence."```-`python -m edge0 ...` 与 `edge0 ...` 完全等价。-### Python API```pythonfrom edge0 import AutoEngine-eng = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别-ids = eng.encode_chat([{"role": "user", "content": "你好"}], think=True)+eng = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected+ids = eng.encode_chat([{"role": "user", "content": "Hello"}], think=True)tokens = eng.generate(ids, max_new_tokens=512)print(eng._tok.decode(tokens))-eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)+eng.reset() # clear per-request state (cross-request KV / prerouter cache)```-### 模型与适配器+### Models and adapters-- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、- tokenizer)。`edge0 serve <dir>` / `AutoEngine.from_pretrained(<dir>)`- 按 `config.json` 自动识别 tier。-- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:- - **模型目录内**(推荐):与基模同目录,如- `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- - `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从- 训练侧 npz 一次性转换。-- 当前默认适配器轮次:35b = round9,10b = round6(pgstart-sel1m,- owners L7–22 与 `start_layer=7` 一致)。+- **Checkpoint**: the original model directory (`config.json`,+ `model*.safetensors`, tokenizer). `edge0 serve <dir>` /+ `AutoEngine.from_pretrained(<dir>)` detect the tier from+ `config.json`.+- **Adapters** (LoRA + prerouter, safetensors) are resolved from either+ location automatically:+ - the model directory (recommended): side by side with the base, e.g.+ `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;+ - `artifacts/` (repo root, gitignored): convert once from+ training-side npz exports via `edge0 convert-adapters --npz-dir ...`.+- Current default adapter rounds: 35b = round9, 10b = round6+ (pgstart-sel1m, owners L7–22 matching `start_layer=7`).-### 稳定性验证+### Stability validation-两个 tier 均通过长连测(temp 0.7 采样、think 开关混合、多 prompt):+Both tiers passed long-run sampling tests (temp 0.7, thinking on/off,+multiple prompts):-- `!` 死循环塌缩 0/N(NaN clip 防护生效);-- 碎片退化 0/N(低层 prerouter 噪声通过 `start_layer=7` 消除);-- 跨请求状态污染 0/N(每请求 `reset()` + 首 token greedy)。+- `!` death-spiral collapse: 0/N (NaN clipping in effect);+- fragment degeneration: 0/N (low-layer prerouter noise eliminated via+ `start_layer=7`);+- cross-request state pollution: 0/N (per-request `reset()` ++ first-token greedy).-## 验证+## Tests```bash-pytest # 单元测试(不含真实权重)-pytest -m slow # 真实 checkpoint 端到端(qwen/ling 生成 + HTTP)-scripts/e2e_smoke.py # 两档模型 staged vs exact 数值一致性冒烟-scripts/zhuhai_example.py # 完整 API 上手例子(--thinking 开关演示)-examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)+pytest # unit tests (no real weights)+pytest -m slow # end-to-end with real checkpoints (generation + HTTP)+scripts/e2e_smoke.py # staged vs exact numerical consistency smoke+scripts/generate_example.py # full-pipeline API example+examples/demo.py # minimal API walkthrough```-## 文档+## Documentation-- [架构总览](docs/architecture.md)-- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)-- [如何接入新模型](docs/adding-a-model.md)+- [Architecture](docs/architecture.md)+- [Attention](docs/attention.md) / [MoE](docs/moe.md) / [SSD streaming](docs/streaming.md) / [prerouter](docs/prerouter.md)+- [Adding a model](docs/adding-a-model.md)- [edge0-35b](docs/models/edge0-35b.md) / [edge0-10b](docs/models/edge0-10b.md)## License-Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE.md))。+Apache-2.0, including vendored third-party code (see [NOTICE](NOTICE.md)).diff --git a/README_zh.md b/README_zh.mdnew file mode 100644index 0000000..8e28fe9--- /dev/null+++ b/README_zh.md@@ -0,0 +1,128 @@+# edge0++[English](README.md) | 中文++**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload ++并行 LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,+当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)即将接入。++开箱支持两个模型 tier:++| 模型 | 说明 | 推理档 |+|---|---|---|+| `edge0-35b` | Qwen3.6-35B-A3B 4bit(40 层,256 专家) | prerouter K=4 |+| `edge0-10b` | Ling-10B 4bit(24 层,128 专家) | prerouter K=8 |++## 设计++- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`+ 按模型名自动选类;+- **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,核心逻辑+ (模型 spec / prerouter / 流式专家池 / server)只依赖后端门面+ (`edge0/backends/base.py` 的 `core` / `nn` 门面),新增后端实现同一门面+ 即可平级接入(`backends/cuda/` 预留插槽),核心代码零改动;+- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据+ (来源、轮次、owners)的 `.safetensors`,放模型目录或 `artifacts/`+ 均可自动解析;+- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /+ `model*.safetensors` / tokenizer)和该模型的适配器,换轮次只换适配器+ 文件,基模不动、不 merge。++## 核心机制++- **SSD 专家 offload**:MoE 专家权重从磁盘 mmap 按需流式加载,+ 激活集驻留 LRU,长尾专家按层预取 —— 大模型小显存跑得动;+- **prerouter 路由预判**:前一 token 的隐层状态经轻量头预测下一 token+ 的专家路由,SSD 预取与下一层前向重叠,路由零等待+ (qwen 系 `start_layer=7`,ling 系同);+- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,+ 基模保持只读 mmap、多套适配器共享同一份基模;+- **数值防护**:per-layer hidden clip(`LING_HIDDEN_CLIP`,默认 1000)+ 阻断 fp16 溢出 → 全 NaN logits → token-0 死循环的塌缩链。++## 快速开始++```bash+# 1) 安装(Python ≥3.10;MLX 后端需 mlx 环境当前支持的平台)+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'++# 2) 快速演示:指定 checkpoint 目录或 tier 名(tier 路径见环境变量)+edge0 demo /path/to/qwen35/model+edge0 demo edge0-10b # 由 EDGE0_10B_MODEL 指向的 checkpoint++# 3) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,+# checkpoint 类型按 config.json 自动识别)+edge0 serve /path/to/qwen35/model+edge0 serve edge0-35b # tier 名,checkpoint 路径同上由环境变量解析++curl http://127.0.0.1:8000/v1/chat/completions \+ -H 'Content-Type: application/json' \+ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'++# 4) 命令行一问一答(同样接受路径或 tier 名)+edge0 chat /path/to/model --prompt "用一句话介绍流式推理。"+```++`python -m edge0 ...` 与 `edge0 ...` 完全等价。++tier 名(`edge0-35b` / `edge0-10b`)通过环境变量解析到本机 checkpoint+目录,二者都未设置时按目录内 `config.json` 自动识别:++```bash+export EDGE0_35B_MODEL=/path/to/qwen35/model+export EDGE0_10B_MODEL=/path/to/ling/model+```++### Python API++```python+from edge0 import AutoEngine++eng = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别+ids = eng.encode_chat([{"role": "user", "content": "你好"}], think=True)+tokens = eng.generate(ids, max_new_tokens=512)+print(eng._tok.decode(tokens))+eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)+```++### 模型与适配器++- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、+ tokenizer)。`edge0 serve <dir>` / `AutoEngine.from_pretrained(<dir>)`+ 按 `config.json` 自动识别 tier。+- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:+ - **模型目录内**(推荐):与基模同目录,如+ `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;+ - `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从+ 训练侧 npz 一次性转换。+- 当前默认适配器轮次:35b = round9,10b = round6(pgstart-sel1m,+ owners L7–22 与 `start_layer=7` 一致)。++### 稳定性验证++两个 tier 均通过长连测(temp 0.7 采样、think 开关混合、多 prompt):++- `!` 死循环塌缩 0/N(NaN clip 防护生效);+- 碎片退化 0/N(低层 prerouter 噪声通过 `start_layer=7` 消除);+- 跨请求状态污染 0/N(每请求 `reset()` + 首 token greedy)。++## 验证++```bash+pytest # 单元测试(不含真实权重)+pytest -m slow # 真实 checkpoint 端到端(qwen/ling 生成 + HTTP)+scripts/e2e_smoke.py # 两档模型 staged vs exact 数值一致性冒烟+scripts/generate_example.py # 完整 API 上手例子(prefill→生成→解码全链路)+examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)+```++## 文档++- [架构总览](docs/architecture.md)+- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)+- [如何接入新模型](docs/adding-a-model.md)+- [edge0-35b](docs/models/edge0-35b.md) / [edge0-10b](docs/models/edge0-10b.md)++## License++Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE.md))。diff --git a/docs/adding-a-model.md b/docs/adding-a-model.mdindex 5e784ec..7d38642 100644--- a/docs/adding-a-model.md+++ b/docs/adding-a-model.md@@ -42,7 +42,7 @@ edge0_35b/__init__.py`)作为逐步示例,逐处标注真实代码行号。`| `gen` | `GenerationConfig` | 采样默认(温度 / top-p / top-k / eos 等) || `prefill_chunk` / `hot_window` / `intra_staging` / `prefetch_history` | — | 流式与预取行为 || `port` | `int` | 服务端口 |-| `target_tok_s` / `peak_active_mem_mb` | `float` | 验收指标(生产机实测) |+| `target_tok_s` / `peak_active_mem_mb` | `float` | 验收指标(基准环境实测) |子类只需实现类方法 `_defaults(model_dir) -> Config`,返回族默认配置。edge0-35b 的实现见 `edge0_35b/__init__.py:31–69`:`_defaults` 用一份完整的 `MoESpec`、`LayerOptions`diff --git a/docs/architecture.md b/docs/architecture.mdindex 5be2094..e7ee01e 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -129,7 +129,7 @@ MLX 后端是参考实现;未来 CUDA 后端实现同一表面即可复用全dequant 参考逐元素对比(相对 L2 < 1%),防 gate/up 交换、行错位类回归;- `tests/test_moe_spec.py`、`tests/test_registry.py`、`tests/test_sampling.py` ——纯逻辑层单测,无需真实 checkpoint;-- 验收指标:`target_tok_s` 与 `peak_active_mem_mb` 记录在生产机器上实测的+- 验收指标:`target_tok_s` 与 `peak_active_mem_mb` 记录的基准环境实测吞吐与峰值活跃内存,作为回归标尺。## Licensediff --git a/docs/models/edge0-10b.md b/docs/models/edge0-10b.mdindex ec37560..c5c1397 100644--- a/docs/models/edge0-10b.md+++ b/docs/models/edge0-10b.md@@ -2,7 +2,7 @@edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)的 10B 级稀疏混合专家模型。相比 35b 档牺牲一定的参数量,换来更低的峰值内存与更高的生成速率,适合对延迟与显存敏感的场景。-生产档案基于 v7 checkpoint 实测,默认由 `Ling10BConfig`(`src/edge0/models/edge0_10b/__init__.py`)固定。+性能档案基于 round6(pgstart-sel1m)adapter 组合的基准实测,默认由 `Ling10BConfig`(`src/edge0/models/edge0_10b/__init__.py`)固定。## 模型档案@@ -16,7 +16,7 @@ edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)| 专家量化 | 4-bit affine,group 64 || 权重布局 | `WeightLayout.SEPARATE`(gate/up/down 分离张量堆叠) || 专家权重路径 | `model.layers.N.mlp.experts` |-| 前置路由(prerouter) | 22 个头(显式 owners 1..22),start_layer 1,hidden 512,fp16 |+| 前置路由(prerouter) | 16 个头(显式 owners 7..22),start_layer 7,hidden 512,fp16 || 解码调用方式 | `patch_call=False`(头内建在 `BailingSparseMoE` 中,从 `prerouter_cache` logits 消费) || LoRA | `r=16, alpha=32.0` || 预填分块 | 2048 |@@ -26,23 +26,24 @@ edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)| 验收吞吐 | ≈ 33 tok/s || 验收峰值激活内存 | ≈ 1.4 GB |-> 说明:数值全部取自 `Ling10BConfig._defaults()` 与 `LayerOptions.staged_k8()`。头部数量来自显式 `owners=range(1, 23)`,共 22 个头;`feature_topk="executed"`。+> 说明:数值全部取自 `Ling10BConfig._defaults()` 与 `LayerOptions.prod_k8()`。头部数量来自显式 `owners=range(7, 23)`,共 16 个头(round6 训练头部分布,L7 起消费预测,L1–6 走原始 router);`feature_topk="executed"`。## 分阶段解码(staged decode)-该档使用 `LayerOptions.staged_k8()` 预设:+该档使用 `LayerOptions.prod_k8()` 预设(对齐 ling v7 部署的生产开关):-- 固定槽分阶段解码(`staged=True`,`staged_n=8`,`staged_trigger=8`,`staged_sync=True`),逐层无 host 同步。-- `staged_replace=False`:路由由内建的混合 prerouter 提供,分阶段集合与路由集合一致,槽表映射零丢弃。-- 不使用热专家钉住(`hot_per_layer=0`),`prefill_hot=0`。-- 整层 E3b prefill(`full_layer_prefill=True`)。+- 分阶段解码关闭(`staged=False`,`staged_sync=False`,`staged_n=8`)——部署验证+ 该档上 staged decode 会劣化输出,prerouter 直接驱动下一 token 的专家+ 预取(step 边界 `stage_all` + prefill 尾部各一次)。+- 专家缓存 `cache_slots=64`,热专家钉住关闭(`hot_per_layer=0`)。+- 整层 E3b prefill(`full_layer_prefill=True`,`prefill_chunk=2048`)。## 使用方式### CLI 起服务```bash-edge0 serve --name edge0-10b --model-dir /path/to/checkpoint --host 127.0.0.1 --port 8083+edge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8083```可选参数:diff --git a/docs/models/edge0-35b.md b/docs/models/edge0-35b.mdindex c9b8a14..bd391d3 100644--- a/docs/models/edge0-35b.md+++ b/docs/models/edge0-35b.md@@ -1,8 +1,8 @@# edge0-35b-edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B 级稀疏混合专家模型。通过流式专家加载(streaming experts)、训练前置路由(prerouter)与 LoRA 适配,把整份权重量化后驻留在磁盘、按需装载,在一块消费级设备上即可服务。+edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B 级稀疏混合专家模型。通过流式专家加载(streaming experts)、训练前置路由(prerouter)与 LoRA 适配,把整份权重量化后驻留在磁盘、按需装载,在单台设备上即可服务。-生产档案基于 round-7 checkpoint 实测,默认由 `Qwen35Config`(`src/edge0/models/edge0_35b/__init__.py`)固定。+性能档案基于 round9 adapter 组合的基准实测,默认由 `Qwen35Config`(`src/edge0/models/edge0_35b/__init__.py`)固定。## 模型档案@@ -42,7 +42,7 @@ edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B### CLI 起服务```bash-edge0 serve --name edge0-35b --model-dir /path/to/checkpoint --host 127.0.0.1 --port 8085+edge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8085```可选参数:diff --git a/examples/bench.py b/examples/bench.pynew file mode 100644index 0000000..e9e71eb--- /dev/null+++ b/examples/bench.py@@ -0,0 +1,130 @@+#!/usr/bin/env python3+"""edge0 speed + memory benchmark (any tier, bench_v7-equivalent protocol).++Runs the same measurement shape as the deployment's bench: prefill ->+sampled warmup steps -> timed sampled decode -> report tok/s and MLX+peak active memory. Works for both tiers; the tier is auto-detected+from the checkpoint (or forced via the model name).++Usage:+ python examples/bench.py /path/to/model [--ntok 200] [--warmup 10]+ python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL++Env (sampling knobs, defaults follow the tier's GenerationConfig):+ BENCH_TEMP / BENCH_PROMPT / BENCH_SEED+"""++from __future__ import annotations++import argparse+import os+import sys+import time++from edge0 import AutoEngine+from edge0.backends import core+from edge0.sampling import sample++PROMPTS = {+ "edge0-35b": "什么是混合专家模型(MoE)?它和普通 Transformer 有什么区别?",+ "edge0-10b": "9.11 和 9.8 哪个大?请仔细比较。",+}+DEFAULT_PROMPT = "什么是混合专家模型(MoE)?简单介绍一下。"+++def _prompt_for(engine) -> str:+ return os.environ.get("BENCH_PROMPT") or PROMPTS.get(engine.name,+ DEFAULT_PROMPT)+++def run_bench(engine, ntok: int, warmup: int) -> dict:+ tok = engine._tok+ prompt = _prompt_for(engine)+ if hasattr(engine, "encode_chat"):+ ids = engine.encode_chat(+ [{"role": "user", "content": prompt}], think=False)+ else:+ ids = list(tok(tok.apply_chat_template(+ [{"role": "user", "content": prompt}],+ tokenize=False, add_generation_prompt=True,+ enable_thinking=False))["input_ids"])++ gen = getattr(engine.cfg, "gen", None)+ temperature = float(os.environ.get(+ "BENCH_TEMP", getattr(gen, "temperature", 0.7)))+ top_k = getattr(gen, "top_k", 64)+ top_p = getattr(gen, "top_p", 0.95)+ rp = getattr(gen, "repetition_penalty", 1.0)+ seed = os.environ.get("BENCH_SEED")++ results = []+ for run in range(2):+ engine.reset()+ core.reset_peak_memory()+ t0 = time.perf_counter()+ engine.prefill(ids)+ t_prefill = time.perf_counter() - t0+ logits = engine.next_logits()+ if seed is not None:+ core.random.seed(int(seed))++ history = list(ids)+ # warmup: sampled, untimed (first tokens are dominated by expert+ # cold-starts; the timed window is the steady state)+ for _ in range(warmup):+ tid = int(core.argmax(logits, axis=-1).item())+ history.append(tid)+ logits = engine.step(tid)+ out: list[int] = []+ t0 = time.perf_counter()+ for _ in range(ntok):+ tid = sample(logits, temperature=temperature, top_k=top_k,+ top_p=top_p, repetition_penalty=rp,+ history=history, seed=None)+ history.append(tid)+ out.append(tid)+ logits = engine.step(tid)+ t_decode = time.perf_counter() - t0+ peak_gib = core.get_peak_memory() / (1024 ** 3)+ results.append(dict(prefill_s=t_prefill, decode_s=t_decode,+ ntok=len(out), peak_gib=peak_gib,+ tok_s=len(out) / t_decode if t_decode else 0.0))+ print(f"prompt={len(ids)} tok prefill={t_prefill:.2f}s "+ f"decode={len(out)}/{t_decode:.2f}s "+ f"tok/s={results[-1]['tok_s']:.1f} "+ f"peak_active={peak_gib:.2f} GiB", flush=True)+ mean_ts = sum(r["tok_s"] for r in results) / len(results)+ max_peak = max(r["peak_gib"] for r in results)+ print(f"[bench] mean tok/s={mean_ts:.1f} peak_active<={max_peak:.2f} GiB",+ flush=True)+ return {"runs": results, "mean_tok_s": mean_ts, "peak_gib": max_peak}+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__)+ ap.add_argument("model", help="checkpoint dir or tier name")+ ap.add_argument("--ntok", type=int,+ default=int(os.environ.get("BENCH_NTOK", "200")))+ ap.add_argument("--warmup", type=int, default=10)+ args = ap.parse_args()+ # tier name -> checkpoint dir via $EDGE0_<TIER>_MODEL (cli parity)+ model = args.model+ name = None+ if model.startswith("edge0-"):+ from edge0.cli import TIER_ENV+ env = os.environ.get(TIER_ENV.get(model, ""))+ if not env:+ raise SystemExit(+ f"set {TIER_ENV.get(model, 'EDGE0_<TIER>_MODEL')} to the "+ f"checkpoint directory for {model}, or pass the dir itself")+ model = env+ engine = AutoEngine.from_pretrained(model, name=name)+ print(f"[bench] tier={engine.name} ntok={args.ntok} warmup={args.warmup}",+ flush=True)+ run_bench(engine, args.ntok, args.warmup)+ engine.close()+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/examples/demo.py b/examples/demo.pyindex 5fc3b0a..0a4efb0 100644--- a/examples/demo.py+++ b/examples/demo.py@@ -28,7 +28,7 @@ def main() -> int:ap.add_argument("--model-dir", default=None,help="checkpoint directory (tier auto-detected)")ap.add_argument("--prompt",- default="Hello! Write one short sentence about Zhuhai.")+ default="Hello! Write one short sentence about the seaside.")ap.add_argument("--max-new", type=int, default=24)args = ap.parse_args()diff --git a/scripts/convert_adapters_legacy.py b/scripts/convert_adapters_legacy.pyindex e8a01de..879b320 100644--- a/scripts/convert_adapters_legacy.py+++ b/scripts/convert_adapters_legacy.py@@ -19,6 +19,7 @@ from __future__ import annotationsimport argparseimport hashlib+import osimport jsonimport sysfrom datetime import datetime, timezone@@ -30,18 +31,27 @@ ROOT = Path(__file__).resolve().parents[1]ARTIFACTS = ROOT / "artifacts"# (npz path, tier, kind) — every legacy source this script understands.+# Training-side npz exports discovered per machine; pass --npz-dir (or+# edit this list) to point at your own exports.+NPZ_DIRS = [+ Path(os.environ.get("EDGE0_NPZ_DIR", "/path/to/npz/exports")),+]SOURCES = [- ("/Users/linyu/Documents/qwen35-v7-deploy/pregate_qwen35_v7_round7.npz",- "edge0-35b", "prerouter"),- ("/Users/linyu/Documents/qwen35-v7-deploy/lora_qwen35_v7_round7.npz",- "edge0-35b", "lora"),- ("/Users/linyu/Documents/ling-mlx-server/v7-deploy/pregate_v7_fp16.npz",- "edge0-10b", "prerouter"),- ("/Users/linyu/Documents/ling-mlx-server/v7-deploy/lora_v7_fp16.npz",- "edge0-10b", "lora"),+ ("pregate_qwen35_v7_round7.npz", "edge0-35b", "prerouter"),+ ("lora_qwen35_v7_round7.npz", "edge0-35b", "lora"),+ ("pregate_v7_fp16.npz", "edge0-10b", "prerouter"),+ ("lora_v7_fp16.npz", "edge0-10b", "lora"),]+def _find_source(name: str) -> Path | None:+ for d in NPZ_DIRS:+ p = d / name+ if p.is_file():+ return p+ return None++def md5(path: Path) -> str:h = hashlib.md5()with open(path, "rb") as f:@@ -113,21 +123,31 @@ def main(argv: list[str] | None = None) -> int:ap = argparse.ArgumentParser(description=__doc__)ap.add_argument("--force", action="store_true",help="reconvert even if the artifact exists")+ ap.add_argument("--npz-dir", default=None,+ help="directory holding the legacy npz exports "+ "(default: $EDGE0_NPZ_DIR)")args = ap.parse_args(argv)+ if args.npz_dir:+ NPZ_DIRS.insert(0, Path(args.npz_dir))ARTIFACTS.mkdir(parents=True, exist_ok=True)- missing = [s[0] for s in SOURCES if not Path(s[0]).is_file()]+ missing = []+ resolved = []+ for name, tier, kind in SOURCES:+ p = _find_source(name)+ if p is None:+ missing.append(name)+ else:+ resolved.append((p, tier, kind))if missing:print("missing legacy sources (skip their conversion):")for m in missing:print(" -", m)- for path, tier, kind in SOURCES:- if not Path(path).is_file():- continue+ for path, tier, kind in resolved:try:- convert(Path(path), tier, kind, force=args.force)+ convert(path, tier, kind, force=args.force)except Exception as exc: # noqa: BLE001 — report and continue- print(f"FAIL {Path(path).name}: {exc}", file=sys.stderr)+ print(f"FAIL {path.name}: {exc}", file=sys.stderr)return 1return 0diff --git a/scripts/e2e_smoke.py b/scripts/e2e_smoke.pyindex 216abe2..65a0074 100644--- a/scripts/e2e_smoke.py+++ b/scripts/e2e_smoke.py@@ -20,8 +20,8 @@ from edge0.backends import iofrom edge0.config import GenerationConfigfrom edge0.streaming.options import LayerOptions-QWEN_DEFAULT = "/Users/linyu/Documents/qwen35-v7-deploy/model"-LING_DEFAULT = "/Users/linyu/Documents/ling-mlx-server/v7-deploy/model"+QWEN_DEFAULT = os.environ.get("EDGE0_35B_MODEL", "/path/to/qwen35/model")+LING_DEFAULT = os.environ.get("EDGE0_10B_MODEL", "/path/to/ling/model")def _prompt_ids(model_dir: str, text: str) -> list[int]:diff --git a/scripts/zhuhai_example.py b/scripts/generate_example.pysimilarity index 89%rename from scripts/zhuhai_example.pyrename to scripts/generate_example.pyindex 77f0ebd..5d1262d 100644--- a/scripts/zhuhai_example.py+++ b/scripts/generate_example.py@@ -1,9 +1,9 @@-"""Complete end-to-end example: generate an English introduction to-Zhuhai with edge0-35b (prerouter + LoRA + SSD offload, default tier-profile — staged decode on, K=4).+"""Complete end-to-end example: generate an English introduction to a+seaside city with edge0-35b (prerouter + LoRA + SSD offload, default+tier profile — staged decode on, K=4).Usage:- .venv/bin/python scripts/zhuhai_example.py [--max-new N] [--temp T]+ .venv/bin/python scripts/generate_example.py [--max-new N] [--temp T]The example walks through the full pipeline a user of the frameworkwould hit:@@ -19,6 +19,7 @@ would hit:from __future__ import annotationsimport argparse+import osimport sysimport time@@ -28,7 +29,7 @@ from edge0.config import GenerationConfigPROMPT = (- "Write a short introduction to Zhuhai, a city in southern China, "+ "Write a short introduction to a seaside city in southern China, ""in English. Keep it to 3-4 sentences.")@@ -41,7 +42,7 @@ def main() -> int:help="enable the <think> block (enable_thinking=True)")args = ap.parse_args()- model_dir = "/Users/linyu/Documents/qwen35-v7-deploy/model"+ model_dir = os.environ.get("EDGE0_35B_MODEL", "/path/to/qwen35/model")print(f"[1] loading tokenizer + edge0-35b from {model_dir}", flush=True)tok = io.load_tokenizer(model_dir)t0 = time.time()diff --git a/scripts/verify_alignment.py b/scripts/verify_alignment.pyindex 27f5355..a4d54f2 100644--- a/scripts/verify_alignment.py+++ b/scripts/verify_alignment.py@@ -22,13 +22,13 @@ import sysN = 14 # max new tokens (greedy)PROMPTS = {- "qwen": "Hello! Write one short sentence about Zhuhai.",- "ling": "你好,用一句话介绍珠海。",+ "qwen": "Hello! Write one short sentence about the seaside.",+ "ling": "你好,用一句话介绍海滨城市。",}-QMODEL = "/Users/linyu/Documents/qwen35-v7-deploy/model"-LMODEL = "/Users/linyu/Documents/ling-mlx-server/v7-deploy/model"-QDIR = "/Users/linyu/Documents/qwen35-v7-deploy"-LDIR = "/Users/linyu/Documents/ling-mlx-server/v7-deploy"+QMODEL = os.environ.get("EDGE0_35B_MODEL", "/path/to/qwen35/model")+LMODEL = os.environ.get("EDGE0_10B_MODEL", "/path/to/ling/model")+QDIR = os.path.dirname(QMODEL)+LDIR = os.path.dirname(LMODEL)RESULT = os.path.join(os.path.dirname(os.path.abspath(__file__)),"alignment_results.json")diff --git a/src/edge0/cli.py b/src/edge0/cli.pyindex 0598d26..b8a863a 100644--- a/src/edge0/cli.py+++ b/src/edge0/cli.py@@ -18,16 +18,16 @@ import sysfrom edge0 import models # noqa: F401 (populates MODEL_REGISTRY)from edge0.registry import MODEL_REGISTRY-# Checkpoint locations the demo command probes when no --model-dir is-# given (the dev boxes these tiers were validated on).-DEMO_DEFAULTS = {- "edge0-35b": "/Users/linyu/Documents/qwen35-v7-deploy/model",- "edge0-10b": "/Users/linyu/Documents/ling-mlx-server/v7-deploy/model",+# Tier name -> environment variable that locates that tier's checkpoint+# (no built-in paths: every machine resolves its own checkpoints).+TIER_ENV = {+ "edge0-35b": "EDGE0_35B_MODEL",+ "edge0-10b": "EDGE0_10B_MODEL",}DEMO_PROMPTS = {- "edge0-35b": "Hello! Write one short sentence about Zhuhai.",- "edge0-10b": "你好,用一句话介绍珠海。",+ "edge0-35b": "Hello! Write one short sentence about the seaside.",+ "edge0-10b": "你好,用一句话介绍海滨城市。",}@@ -69,50 +69,45 @@ def _engine_kwargs(args) -> dict:def _resolve_model(args) -> tuple[str | None, str | None]:- """vLLM-style model argument resolution.+ """Resolve the positional ``model`` argument.``edge0 serve <model>`` accepts either a registered tier name- (``edge0-35b`` / ``edge0-10b`` — checkpoint found via the matching- env var, else the known dev-box default) or a checkpoint path- (``edge0 serve /root/edge0-35b`` — tier auto-detected from the- checkpoint's ``config.json``).+ (``edge0-35b`` / ``edge0-10b`` — checkpoint located via the matching+ ``EDGE0_<TIER>_MODEL`` environment variable) or a checkpoint path+ (tier auto-detected from the checkpoint's ``config.json``).Returns ``(model_dir, name)``; either may stay None to keep the- legacy ``--model-dir`` / ``--name`` behaviour.+ ``--model-dir`` / ``--name`` behaviour."""model = getattr(args, "model", None)if not model:return args.model_dir, args.nameif model in MODEL_REGISTRY:- short = (model.split("-", 1)[-1] if model.startswith("edge0-")- else model)- env = os.environ.get(f"EDGE0_{short.upper()}_MODEL")- return env or DEMO_DEFAULTS.get(model), model+ env = os.environ.get(TIER_ENV.get(model, ""))+ return env, modelreturn model, None # a path: tier auto-detected by the registry+def _missing_model_help(name) -> str:+ tier_env = TIER_ENV.get(name, "EDGE0_<TIER>_MODEL")+ return (+ f"[edge0] no checkpoint for {name or 'model'}.\n"+ f"Set {tier_env} to the checkpoint directory, e.g.\n"+ f" export {tier_env}=/path/to/model\n"+ "or pass the checkpoint explicitly:\n"+ " edge0 demo /path/to/model\n"+ "or point at any compatible checkpoint (tier auto-detected from"+ " config.json)."+ )++def cmd_demo(args) -> int:from edge0 import AutoEnginefrom edge0.server.chat import ChatMessage, ChatRequest, ChatSessionmodel_dir, name = _resolve_model(args)- if not model_dir:- # No explicit model: probe the known dev-box checkpoints.- for d in DEMO_DEFAULTS.values():- if os.path.isdir(d):- model_dir = d- breakif not model_dir or not os.path.isdir(model_dir):- print(- f"[edge0] no checkpoint for {name or 'model'}.\n"- "Pass the model explicitly, vLLM-style:\n"- f" edge0 demo {DEMO_DEFAULTS.get(name, '<checkpoint-dir>')}\n"- "or point at your own checkpoint:\n"- " edge0 demo /path/to/qwen35/model\n"- "Tier names are edge0-35b / edge0-10b (env EDGE0_<TIER>_MODEL "- "overrides the default search path).",- file=sys.stderr,- )+ print(_missing_model_help(name), file=sys.stderr)return 2engine = AutoEngine.from_pretrained(model_dir, name=name,**_engine_kwargs(args))@@ -204,7 +199,7 @@ def main(argv: list[str] | None = None) -> int:p = sub.add_parser("demo",- help="one-command quickstart (vLLM-style model argument)")+ help="one-shot generation demo")p.add_argument("model", nargs="?", default=None,help="tier name (edge0-35b) or checkpoint dir")p.add_argument("--model-dir", default=None)@@ -216,7 +211,7 @@ def main(argv: list[str] | None = None) -> int:p = sub.add_parser("chat",- help="one-shot prompt answering (vLLM-style model argument)")+ help="one-shot prompt answering")p.add_argument("model", nargs="?", default=None,help="tier name (edge0-35b) or checkpoint dir")p.add_argument("--model-dir", default=None)@@ -228,7 +223,7 @@ def main(argv: list[str] | None = None) -> int:p = sub.add_parser("serve",- help="run the HTTP server, vLLM-style: edge0 serve <model>")+ help="run the OpenAI-compatible HTTP server: edge0 serve <model>")p.add_argument("model", nargs="?", default=None,help="tier name (edge0-35b) or checkpoint dir")p.add_argument("--model-dir", default=None)diff --git a/src/edge0/models/edge0_35b/__init__.py b/src/edge0/models/edge0_35b/__init__.pyindex ff39fb5..d60b86c 100644--- a/src/edge0/models/edge0_35b/__init__.py+++ b/src/edge0/models/edge0_35b/__init__.py@@ -1,6 +1,6 @@"""edge0-35b adapter: Qwen3.5-MoE (K=4 tier, 35B-class).-Family facts (production profile, verified against the round-7+Family facts (production profile, verified against the round-9checkpoint):* 40 layers, 256 routed experts (4-bit affine, group 64), K=4,diff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.pyindex 115130e..fd31ec8 100644--- a/src/edge0/streaming/options.py+++ b/src/edge0/streaming/options.py@@ -91,10 +91,10 @@ class LayerOptions:return cls(staged=True, staged_replace=False, staged_n=4,staged_trigger=4, staged_sync=True, history_prefetch=True,- hot_per_layer=32, hot_update_interval=4, hot_decay=0.75,+ hot_per_layer=0, hot_update_interval=4, hot_decay=0.75,cache_slots=64, prefetch_cap=48, load_threads=8,prefetch_threads=4, use_compile=True, top_k=4,- full_layer_prefill=True, prefill_full_layers=12,+ full_layer_prefill=False, prefill_full_layers=0,prefill_hot=32, **overrides)@classmethoddiff --git a/tests/test_registry.py b/tests/test_registry.pyindex adf9998..285c533 100644--- a/tests/test_registry.py+++ b/tests/test_registry.py@@ -48,8 +48,11 @@ def test_qwen35_profile():assert cfg.moe_spec.layout.value == "separate"assert "language_model.model.layers" in cfg.moe_spec.key_templateassert cfg.options.staged_n == 4- assert cfg.options.prefill_full_layers == 12- assert cfg.options.hot_per_layer == 32+ # demo production profile: on-demand prefill (QWEN_PREFILL_FULL=0),+ # no resident hot pins (QWEN_HOT=0)+ assert cfg.options.prefill_full_layers == 0+ assert cfg.options.full_layer_prefill is False+ assert cfg.options.hot_per_layer == 0assert cfg.prerouter.start_layer == 7assert cfg.prerouter.hidden == 512assert cfg.prerouter.dtype == "fp16"@@ -68,9 +71,10 @@ def test_ling10b_profile():assert cfg.moe_spec.n_group == 8assert cfg.moe_spec.topk_group == 4assert cfg.options.staged_n == 8- assert cfg.prerouter.start_layer == 1- assert cfg.prerouter.owners == tuple(range(1, 23))+ assert cfg.prerouter.start_layer == 7+ assert cfg.prerouter.owners == tuple(range(7, 23))assert cfg.prerouter.patch_call is False+ assert cfg.gen.repetition_penalty == 1.1assert 156895 in cfg.gen.eos_idsassert cfg.port == 8083diff --git a/tests/test_server.py b/tests/test_server.pyindex 7c7a373..82061bf 100644--- a/tests/test_server.py+++ b/tests/test_server.py@@ -62,10 +62,17 @@ class FakeTok:class PlainTok(FakeTok):- """Tokenizer without apply_chat_template (stdlib fallback path)."""+ """Tokenizer without apply_chat_template (stdlib fallback path).- def apply_chat_template(self, *a, **k): # pragma: no cover - unused- raise AttributeError+ Deleting the method (vs raising) keeps hasattr() False so the+ server's plain-text fallback renders instead."""++ def __init__(self):+ super().__init__()+ try:+ del self.apply_chat_template+ except AttributeError:+ passclass FakeEngine:@@ -91,6 +98,10 @@ class FakeEngine:def stats(self):return {"ok": 1}+ def reset(self):+ """Per-request state clear (server parity with the real engine)."""+ self.pos = 0+def _req(**kw) -> ChatRequest:payload = {@@ -225,8 +236,8 @@ def test_queue_server_health():def test_queue_server_chat_serializes():eng = FakeEngine()srv = QueueServer(eng)- out = srv.chat(_req())- assert out["usage"]["completion_tokens"] == 3+ tokens, meta = srv.chat(_req())+ assert meta["usage"]["completion_tokens"] == 3assert len(eng.generated) == 1gen = eng.generated[0][1]assert gen.temperature == 0.7@@ -270,11 +281,13 @@ def test_chat_stream_events_sequence():})body = raw if isinstance(raw, bytes) else raw.get_data()events = [e for e in body.decode("utf-8").split("\n\n") if e]+ assert events[-1] == "data: [DONE]"chunks = [json.loads(e[6:]) for e in events[:-1]] # drop [DONE]- deltas = [c["choices"][0]["delta"]["content"] for c in chunks]+ deltas = [c["choices"][0]["delta"]["content"]+ for c in chunks if c["choices"][0]["delta"]]assert deltas == ["T11", "T12", "T13"]assert chunks[-1]["choices"][0]["finish_reason"] == "stop"- assert events[-1] == "data: [DONE]"+ assert chunks[-1]["choices"][0]["delta"] == {}# ---- stdlib HTTP transport ------------------------------------------------