Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: translate docs to English; README prerouter A/B numbers and HF checkpoint quick start
linyubupab4e45ab13e2c
diff --git a/README.md b/README.mdindex 28b4b51..7f3e699 100644--- a/README.md+++ b/README.md@@ -60,10 +60,20 @@ are co-located with each checkpoint and load automatically, sostreamed on demand; the active set stays resident in an LRU andlong-tail experts are prefetched per layer — large models run inmodest memory;-- **Prerouter routing prediction**: a lightweight head predicts the next- token's expert routing from the previous token's hidden state, so SSD- prefetch overlaps the next forward pass with zero routing latency- (`start_layer=7` on both tiers);+- **Prerouter routing prediction**: MoE decode waits on expert loads —+ routing depends on the previous layer's output, so by the time the+ router picks the experts, their SSD load has not even started. A+ lightweight trained head breaks this serialization: it predicts the+ next token's expert routing from the previous token's hidden state+ one step ahead (double shift: prev-layer + prev-token), so expert+ loads are submitted at the step boundary and SSD read latency hides+ completely behind the forward pass (`start_layer=7` on both tiers).+ Measured A/B decode speedup vs the same model with native routing+ (identical adapters and load, alternating rounds): **up to +59%** on+ this test machine. The slower the storage, the bigger the win: the+ mechanism removes exactly the cold-read wait that dominates when the+ expert working set exceeds what stays resident, so the gain scales+ with model size, routed width (K), and memory pressure;- **Parallel LoRA**: adapters are applied as a side path at forwardtime instead of being merged — the base stays a read-only mmap andmultiple adapter sets share one base;@@ -73,24 +83,66 @@ are co-located with each checkpoint and load automatically, so## Quick start+### 1) Install+```bash-# 1) Install (Python >= 3.10; MLX backend requires macOS + Apple Silicon)+# Python >= 3.10; the MLX backend requires macOS with Apple Siliconpython3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'+```++### 2) Download a model++The two tiers are published on Hugging Face — each repo bundles the+base checkpoint and the trained LoRA + prerouter adapters in **one+directory**, so a single download is a ready-to-run model:++- [`Edge0/Edge0-35b-a3b-preview`](https://huggingface.co/Edge0/Edge0-35b-a3b-preview) (~23 GB)+- [`Edge0/Edge0-10b-a1b-preview`](https://huggingface.co/Edge0/Edge0-10b-a1b-preview) (~4.2 GB)++```bash+# with the repo's helper (defaults to the two repos above):+.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models+.venv/bin/python scripts/fetch_models.py --tier edge0-10b --target-dir models++# or directly with the CLI:+.venv/bin/huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir models/edge0-35b+.venv/bin/huggingface-cli download Edge0/Edge0-10b-a1b-preview --local-dir models/edge0-10b+```++Either way you end up with a directory like:++```+models/edge0-35b/+├── config.json, model-*.safetensors, tokenizer files # base checkpoint+├── lora_edge0_35b.safetensors # trained LoRA adapters+└── prerouter_edge0_35b.safetensors # trained prerouter heads+```++### 3) Point edge0 at it-# 2) Get a model: base checkpoint + trained LoRA/prerouter adapters in ONE dir.-# (Set EDGE0_35B_REPO / EDGE0_10B_REPO to the published Hugging Face-# repo ids, then:)-.venv/bin/python scripts/fetch_models.py --tier edge0-35b-.venv/bin/python scripts/fetch_models.py --tier edge0-10b+Tier names resolve to local directories via environment variables+(where you put the download is up to you):++```bashexport EDGE0_35B_MODEL=$PWD/models/edge0-35bexport EDGE0_10B_MODEL=$PWD/models/edge0-10b+```++Or skip the env vars entirely and pass the directory directly — the+tier is auto-detected from the checkpoint's `config.json`:++```bash+edge0 demo models/edge0-35b+edge0 serve models/edge0-10b+```++### 4) Run-# 3) Quick demo: pass a tier name or a checkpoint directory+```bash+# quick demoedge0 demo edge0-35b-edge0 demo /path/to/qwen35/model-# 4) Serve (OpenAI-compatible /v1/chat/completions; the model is a-# positional argument, tier auto-detected from config.json)+# serve (OpenAI-compatible /v1/chat/completions)edge0 serve edge0-35b```@@ -106,22 +158,6 @@ edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."`python -m edge0 ...` is equivalent to `edge0 ...`.-If you already have a checkpoint on disk, just point at it — the tier is-auto-detected from the checkpoint's `config.json`:--```bash-edge0 demo /path/to/model-edge0 serve /path/to/model-```--Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint-directories through environment variables:--```bash-export EDGE0_35B_MODEL=/path/to/qwen35/model-export EDGE0_10B_MODEL=/path/to/ling/model-```-### Python API```pythondiff --git a/README_zh.md b/README_zh.mdindex 8561692..f2502a8 100644--- a/README_zh.md+++ b/README_zh.md@@ -49,9 +49,14 @@- **SSD 专家 offload**:MoE 专家权重从磁盘 mmap 按需流式加载,激活集驻留 LRU,长尾专家按层预取 —— 大模型小显存跑得动;-- **prerouter 路由预判**:前一 token 的隐层状态经轻量头预测下一 token- 的专家路由,SSD 预取与下一层前向重叠,路由零等待- (qwen 系 `start_layer=7`,ling 系同);+- **prerouter 路由预判**:MoE 解码的固有瓶颈是「等专家装载」——路由依赖+ 前一层输出,等路由选出专家时,SSD 装载甚至还没开始。edge0 用一个轻量+ 训练头打破这条串行链:基于上一 token 的隐层状态提前一步预测下一 token+ 的专家路由(双移位:层移位 + token 移位),专家装载在 step 边界即提前+ 提交,SSD 读取延迟完全隐藏在前向计算之下(`start_layer=7`)。同模型+ 原生路由 A/B 交替实测(同适配器、同负载):**最高 +59%**(本测试机)。+ **存储越慢,收益越大**:该机制消除的正是专家工作集超出常驻容量时占+ 主导的冷读等待,因此收益随模型规模、路由宽度(K)与内存压力增大;- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,基模保持只读 mmap、多套适配器共享同一份基模;- **数值防护**:per-layer hidden clip(`LING_HIDDEN_CLIP`,默认 1000)@@ -59,50 +64,67 @@## 快速开始+### 1) 安装+```bash-# 1) 安装(Python ≥3.10;MLX 后端需 macOS + Apple Silicon)+# Python ≥3.10;MLX 后端需 macOS + Apple Siliconpython3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'+```-# 2) 下载模型:基模 + 训练好的 LoRA/prerouter 适配器在一个目录里-# (把 EDGE0_35B_REPO / EDGE0_10B_REPO 设为发布的 Hugging Face 仓库 id)-.venv/bin/python scripts/fetch_models.py --tier edge0-35b-.venv/bin/python scripts/fetch_models.py --tier edge0-10b-export EDGE0_35B_MODEL=$PWD/models/edge0-35b-export EDGE0_10B_MODEL=$PWD/models/edge0-10b+### 2) 下载模型-# 3) 快速演示:指定 tier 名或 checkpoint 目录-edge0 demo edge0-35b-edge0 demo /path/to/qwen35/model+两个档位发布在 Hugging Face——每个仓库把基模 checkpoint 与训练好的+LoRA + prerouter 适配器打包在**同一目录**,一次下载即为可运行的模型:-# 4) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,-# checkpoint 类型按 config.json 自动识别)-edge0 serve edge0-35b-```+- [`Edge0/Edge0-35b-a3b-preview`](https://huggingface.co/Edge0/Edge0-35b-a3b-preview)(约 23 GB)+- [`Edge0/Edge0-10b-a1b-preview`](https://huggingface.co/Edge0/Edge0-10b-a1b-preview)(约 4.2 GB)```bash-curl http://127.0.0.1:8000/v1/chat/completions \- -H 'Content-Type: application/json' \- -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'+# 用仓库自带脚本(默认即上述两个仓库):+.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models+.venv/bin/python scripts/fetch_models.py --tier edge0-10b --target-dir models++# 或直接用 CLI:+.venv/bin/huggingface-cli download Edge0/Edge0-35b-a3b-preview \+ --local-dir models/edge0-35b+.venv/bin/huggingface-cli download Edge0/Edge0-10b-a1b-preview \+ --local-dir models/edge0-10b+```-# 5) 命令行一问一答(用 --max-new 控制长度,加 --show-thinking 打印思考过程)-edge0 chat edge0-35b --prompt "用一句话介绍流式推理。"+下载完成后目录结构:++```+models/edge0-35b/+├── config.json, model-*.safetensors, tokenizer 文件 # 基模 checkpoint+├── lora_edge0_35b.safetensors # 训练好的 LoRA 适配器+└── prerouter_edge0_35b.safetensors # 训练好的 prerouter 头```-`python -m edge0 ...` 与 `edge0 ...` 完全等价。+### 3) 指向模型目录-如果本机已有 checkpoint,直接传路径即可,tier 按 `config.json` 自动识别:+档位名经环境变量解析到本地目录(放哪由你决定):```bash-edge0 demo /path/to/model-edge0 serve /path/to/model+export EDGE0_35B_MODEL=$PWD/models/edge0-35b+export EDGE0_10B_MODEL=$PWD/models/edge0-10b```-tier 名(`edge0-35b` / `edge0-10b`)通过环境变量解析到本机 checkpoint-目录:+也可以不用环境变量,直接传目录——框架从 checkpoint 的 `config.json`+自动识别档位:```bash-export EDGE0_35B_MODEL=/path/to/qwen35/model-export EDGE0_10B_MODEL=/path/to/ling/model+edge0 demo models/edge0-35b+edge0 serve models/edge0-10b+```++### 4) 运行++```bash+# 快速演示+edge0 demo edge0-35b++# 起服务(OpenAI 兼容 /v1/chat/completions)+edge0 serve edge0-35b```### Python APIdiff --git a/docs/adding-a-model.md b/docs/adding-a-model.mdindex 7d38642..eb9df68 100644--- a/docs/adding-a-model.md+++ b/docs/adding-a-model.md@@ -1,71 +1,55 @@-# 接入一个新模型+# Adding a New Model-本文档给出向 edge0 接入一个新模型族的分步指南,用 `edge0-35b`(`src/edge0/models/-edge0_35b/__init__.py`)作为逐步示例,逐处标注真实代码行号。`edge0-10b`-(`src/edge0/models/edge0_10b/__init__.py`)是第二个活例子,路径解析与契约完全一致。+This document is a step-by-step guide for adding a new model family to edge0, using `edge0-35b` (`src/edge0/models/edge0_35b/__init__.py`) as the running example, with real code line numbers noted throughout. `edge0-10b` (`src/edge0/models/edge0_10b/__init__.py`) is a second live example; its path resolution and contract are identical.-接入目标:让 `AutoConfig` / `AutoModel` / `AutoEngine` 能按模型名(或 checkpoint 的-`config.json` 的 `model_type`)解析你的模型,且流式 MoE 层能由一个通用的-`StreamingSwitchGLU` 驱动。+Integration goal: let `AutoConfig` / `AutoModel` / `AutoEngine` resolve your model by name (or by the `model_type` in the checkpoint's `config.json`), and have the streaming MoE layers driven by a common `StreamingSwitchGLU`.-## 接入总览(契约)+## Integration overview (the contract)-`edge0.registry` 的契约是:**一个适配器模块暴露三件套**:+The contract of `edge0.registry` is: **one adapter module exposes three things**:-- `Config` —— 一个 `ModelConfig` 子类(含 `from_pretrained`);-- `build_model(model_dir, **overrides)` —— 装配好的模型骨架;-- `build_engine(model_dir, **overrides)` —— 可生成引擎。+- `Config` — a `ModelConfig` subclass (with `from_pretrained`);+- `build_model(model_dir, **overrides)` — the assembled model skeleton;+- `build_engine(model_dir, **overrides)` — a generation-capable engine.-`register_model(name, adapter)` 注册;`TYPE_ALIASES` 把 checkpoint 的 `model_type`-映射到注册名。见 `registry.py` 的 `_resolve_name` 与 `AutoConfig.from_pretrained`-(registry.py:73–101)。+`register_model(name, adapter)` performs the registration; `TYPE_ALIASES` maps a checkpoint's `model_type` to a registered name. See `_resolve_name` and `AutoConfig.from_pretrained` in `registry.py` (registry.py:73–101).-## 第 1 步:创建适配器模块与目录+## Step 1: Create the adapter module and directory-在 `src/edge0/models/` 下建一个包,名取你的档位,例如 `edge0_35b/`。目录内至少一个-`__init__.py` 作为适配器。模型目录布局要求见「模型目录布局」一节。+Create a package under `src/edge0/models/` named after your tier, e.g. `edge0_35b/`. The directory needs at least one `__init__.py`, which acts as the adapter. For the model directory layout requirements, see the "Model directory layout requirements" section.-## 第 2 步:定义 `Config` 子类与 `_defaults`+## Step 2: Define the `Config` subclass and `_defaults`-`Config` 是 `ModelConfig`(`models/base.py`)的子类。`ModelConfig` 的字段即「跑起一个-模型档位需要的一切」:+`Config` is a subclass of `ModelConfig` (`models/base.py`). The fields of `ModelConfig` are "everything needed to run a model tier":-| 字段 | 类型 | 语义 |+| Field | Type | Semantics || --- | --- | --- |-| `name` | `str` | 注册名 |-| `model_dir` | `str` | checkpoint 目录 |-| `moe_spec` | `MoESpec` | MoE 规格(见 moe.md) |-| `options` | `LayerOptions` | 流式层选项 |-| `prerouter` | `PrerouterSpec \| None` | 预路由头规格 |-| `prerouter_top_k` | `int` | 预路由宽度(0 → 用 `options.top_k`) |-| `lora` / `lora_r` / `lora_alpha` | `str` / `int` / `float` | LoRA 权重路径与超参(`""` 禁用) |-| `gen` | `GenerationConfig` | 采样默认(温度 / top-p / top-k / eos 等) |-| `prefill_chunk` / `hot_window` / `intra_staging` / `prefetch_history` | — | 流式与预取行为 |-| `port` | `int` | 服务端口 |-| `target_tok_s` / `peak_active_mem_mb` | `float` | 验收指标(基准环境实测) |+| `name` | `str` | registered name |+| `model_dir` | `str` | checkpoint directory |+| `moe_spec` | `MoESpec` | MoE spec (see moe.md) |+| `options` | `LayerOptions` | streaming layer options |+| `prerouter` | `PrerouterSpec \| None` | prerouter head spec |+| `prerouter_top_k` | `int` | prerouter width (0 → use `options.top_k`) |+| `lora` / `lora_r` / `lora_alpha` | `str` / `int` / `float` | LoRA weight path and hyperparameters (`""` disables) |+| `gen` | `GenerationConfig` | sampling defaults (temperature / top-p / top-k / eos, etc.) |+| `prefill_chunk` / `hot_window` / `intra_staging` / `prefetch_history` | — | streaming and prefetch behavior |+| `port` | `int` | serving port |+| `target_tok_s` / `peak_active_mem_mb` | `float` | acceptance metrics (measured on the benchmark environment) |-子类只需实现类方法 `_defaults(model_dir) -> Config`,返回族默认配置。edge0-35b 的-实现见 `edge0_35b/__init__.py:31–69`:`_defaults` 用一份完整的 `MoESpec`、`LayerOptions`-预设 `staged_k4()`、`PrerouterSpec`、采样默认与验收指标构造 `Qwen35Config`。+The subclass only needs to implement the class method `_defaults(model_dir) -> Config`, which returns the family's default configuration. The edge0-35b implementation lives at `edge0_35b/__init__.py:31–69`: `_defaults` builds a `Qwen35Config` from a complete `MoESpec`, the `LayerOptions` preset `staged_k4()`, a `PrerouterSpec`, sampling defaults, and acceptance metrics.-`from_pretrained(model_dir=None, **overrides)` 是 `ModelConfig` 提供的模板方法-(`models/base.py:60–73`):它调用 `_defaults` 得到基配置,再对每个 override 校验后-`replace` 覆盖;未知字段抛 `TypeError` 并列出已知字段。因此**每个 public 属性都能被-用户 override**。+`from_pretrained(model_dir=None, **overrides)` is a template method provided by `ModelConfig` (`models/base.py:60–73`): it calls `_defaults` to obtain the base configuration, then validates each override before applying it via `replace`; unknown fields raise a `TypeError` that lists the known fields. As a result, **every public attribute can be overridden by the user**.-要点:+Key points:-- 仓库根目录有 `scripts/convert_adapters_legacy.py` 一次性把训练 npz 导出转成- safetensors 产物到 `artifacts/`;`ModelConfig.artifact(name)`(`models/base.py:28–30`)- 返回这些产物的绝对路径,适配器用它填 LoRA / prerouter 权重路径。-- LoRA override 走 `resolve_lora`(`models/base.py:88–95`):裸模型名解析为该档产物,- `"model_dir"` 表示「保留训练权重在原位」,空串禁用。+- The repository root ships `scripts/convert_adapters_legacy.py`, which one-shot converts the training npz exports into safetensors artifacts under `artifacts/`; `ModelConfig.artifact(name)` (`models/base.py:28–30`) returns the absolute paths of those artifacts, and the adapter uses them to fill in the LoRA / prerouter weight paths.+- LoRA overrides go through `resolve_lora` (`models/base.py:88–95`): a bare model name resolves to that tier's artifacts, `"model_dir"` means "keep the training weights in place", and an empty string disables it.-## 第 3 步:`build_model` / `build_engine`+## Step 3: `build_model` / `build_engine`-两个模块级函数,复用引擎自己的构建路径(DRY)。+Two module-level functions that reuse the engine's own construction path (DRY).-`edge0_35b/__init__.py:72–79`:+`edge0_35b/__init__.py:72–79`:```pythondef build_model(model_dir=None, **overrides):@@ -75,7 +59,7 @@ def build_model(model_dir=None, **overrides):return model```-`edge0_35b/__init__.py:82–87`:+`edge0_35b/__init__.py:82–87`:```pythondef build_engine(model_dir=None, **overrides):@@ -84,14 +68,11 @@ def build_engine(model_dir=None, **overrides):return Qwen35Engine(cfg.model_dir, cfg)```-你的模型若族数学不同(如 edge0-10b 的 `SIGMOID_GROUP` + 混合 MLA),在-`src/edge0/engine/` 写一个对应引擎(参照 `engine/qwen.py` / `engine/ling.py`),-`build_model` / `build_engine` 引用它。`load_installed` 负责装配流式孪生 / LoRA /-prerouter,是框架给引擎提供的构建路径。+If your model family's math differs (e.g. edge0-10b's `SIGMOID_GROUP` + hybrid MLA), write a matching engine under `src/edge0/engine/` (model it on `engine/qwen.py` / `engine/ling.py`) and reference it from `build_model` / `build_engine`. `load_installed` assembles the streaming twins / LoRA / prerouter; it is the construction path the framework provides to engines.-## 第 4 步:注册到注册表+## Step 4: Register with the registry-文件末尾调用 `register_model`,并把 `Config` 暴露为模块属性(契约要求):+At the end of the file, call `register_model` and expose `Config` as a module attribute (required by the contract):```python# edge0_35b/__init__.py:90–93@@ -99,17 +80,13 @@ register_model("edge0-35b", sys.modules[__name__])Config = Qwen35Config # registry contract: adapter.Config```-- `register_model`(`registry.py:25–28`):重名抛 `ValueError`。-- 适配器模块必须被 import 才会注册。`edge0.models.__init__`(models/__init__.py:15)- 显式 import 各档包来 populate 注册表,`AutoConfig.from_pretrained` 里也会- `from edge0 import models` 兜底。-- 你的模型包应加进 `edge0/models/__init__.py` 的 import 列表。+- `register_model` (`registry.py:25–28`): a duplicate name raises `ValueError`.+- An adapter module must be imported for it to register. `edge0.models.__init__` (models/__init__.py:15) explicitly imports each tier's package to populate the registry, and `AutoConfig.from_pretrained` also runs `from edge0 import models` as a fallback.+- Your model package should be added to the import list in `edge0/models/__init__.py`.-## 第 5 步:`TYPE_ALIASES`+## Step 5: `TYPE_ALIASES`-`TYPE_ALIASES`(`registry.py:15–22`)把 checkpoint `config.json` 的 `model_type`-映射到注册名,使 `AutoEngine.from_pretrained(model_dir=...)` 能**直接从目录解析**而无需-显式传 name。例:+`TYPE_ALIASES` (`registry.py:15–22`) maps the `model_type` in a checkpoint's `config.json` to a registered name, so that `AutoEngine.from_pretrained(model_dir=...)` can **resolve directly from the directory** without passing a name explicitly. For example:```pythonTYPE_ALIASES = {@@ -120,46 +97,31 @@ TYPE_ALIASES = {}```-`_resolve_name`(registry.py:42–56)的查找顺序:显式 `name` → 目录的 `model_type`-(`_model_type_from_dir` 读 `config.json`,失败则退回目录 basename)→ 匹配-`MODEL_REGISTRY` → 匹配 `TYPE_ALIASES` → 否则 `KeyError` 列出已注册模型。-未知名会因 `test_unknown_name_lists_registry` 之类的断言被测试钉住。+The lookup order of `_resolve_name` (registry.py:42–56): explicit `name` → the directory's `model_type` (`_model_type_from_dir` reads `config.json`, falling back to the directory basename on failure) → match against `MODEL_REGISTRY` → match against `TYPE_ALIASES` → otherwise a `KeyError` listing the registered models. Unknown names are pinned by tests through assertions such as `test_unknown_name_lists_registry`.-## 模型目录布局要求+## Model directory layout requirements-`AutoConfig.from_pretrained(model_dir=...)` 会读 `model_dir/config.json` 的-`model_type` 字段(`registry.py:58–70`)。因此 checkpoint 目录至少要满足:+`AutoConfig.from_pretrained(model_dir=...)` reads the `model_type` field of `model_dir/config.json` (`registry.py:58–70`). The checkpoint directory must therefore at least satisfy:-- `config.json` 存在,且 `model_type` 已登记(或你的 `TYPE_ALIASES` 覆盖该值);-- 权重为 safetensors 格式,key 前缀与 `moe_spec.key_template` 一致(如- `language_model.model.layers.N.mlp.switch_mlp`);-- LoRA / prerouter 权重为带元数据的 `.safetensors`(`models/base.py` 顶部注释:由- `convert_adapters_legacy.py` 从训练 npz 一次性转换)。+- `config.json` exists, and its `model_type` is registered (or covered by your `TYPE_ALIASES`);+- the weights are in safetensors format, with key prefixes matching `moe_spec.key_template` (e.g. `language_model.model.layers.N.mlp.switch_mlp`);+- the LoRA / prerouter weights are `.safetensors` files carrying metadata (see the comment at the top of `models/base.py`: they are converted one-shot from the training npz by `convert_adapters_legacy.py`).-目录 basename 是最后兜底的解析手段(`_model_type_from_dir`),所以目录名最好与-注册名对齐,但不是必须。+The directory basename is the last-resort resolution fallback (`_model_type_from_dir`), so aligning the directory name with the registered name is recommended, but not required.-## 第 6 步:测试建议+## Step 6: Testing recommendations-参考 `tests/test_moe_spec.py` 与 `tests/test_registry.py`(纯逻辑单测,无需真实-checkpoint)。+Refer to `tests/test_moe_spec.py` and `tests/test_registry.py` (pure-logic unit tests; no real checkpoint needed).-- **注册与别名**(test_registry.py:12–20):断言 `MODEL_REGISTRY` 含你的注册名、- `TYPE_ALIASES` 解析正确。-- **Config 默认值**(test_registry.py:23–33):`AutoConfig.from_pretrained(name=...)`- 的 `moe_spec` / `options` / `prerouter` / `prerouter_top_k` 等字段符合预期。-- **profile 细化**(test_registry.py:35–71):逐字段断言你的档位数值(专家数、top_k、- quant、端口、验收指标等)。-- **override 与拒绝**(test_registry.py:74–81):`from_pretrained(..., prerouter=None,- lora="")` 生效;未知字段抛 `TypeError`。-- **适配器契约**(test_registry.py:89–92):断言 `MODEL_REGISTRY` 每个条目都暴露- `Config` / `build_model` / `build_engine`。-- **路径解析**(test_moe_spec.py):`keys` 模板、`block_of` 数字段下标、`layer_of`- 从 `block_path` 推出的默认、`bundle_projs` 随布局切换。-- **数学 parity**:路由数学必须与 vendored 模型 bit 级一致(`moe/routing.py` 用 parity- 测试钉死),若你的模型引入新路由种类需补对照。+- **Registration and aliases** (test_registry.py:12–20): assert that `MODEL_REGISTRY` contains your registered name and that `TYPE_ALIASES` resolves correctly.+- **Config defaults** (test_registry.py:23–33): the `moe_spec` / `options` / `prerouter` / `prerouter_top_k` fields of `AutoConfig.from_pretrained(name=...)` match expectations.+- **Profile refinement** (test_registry.py:35–71): assert your tier's values field by field (expert count, top_k, quant, port, acceptance metrics, etc.).+- **Overrides and rejection** (test_registry.py:74–81): `from_pretrained(..., prerouter=None, lora="")` takes effect; unknown fields raise `TypeError`.+- **Adapter contract** (test_registry.py:89–92): assert that every `MODEL_REGISTRY` entry exposes `Config` / `build_model` / `build_engine`.+- **Path resolution** (test_moe_spec.py): the `keys` template, `block_of` numeric-field indexing, the `layer_of` default derived from `block_path`, and `bundle_projs` switching with the layout.+- **Math parity**: routing math must match the vendored model bit-for-bit (pinned by parity tests in `moe/routing.py`); if your model introduces a new routing kind, add the corresponding comparison.-## 完整最小骨架(对照 edge0-35b)+## Complete minimal skeleton (modeled on edge0-35b)```python# src/edge0/models/my_model/__init__.py@@ -183,25 +145,25 @@ class MyConfig(ModelConfig):block_path="model.layers.{layer}.mlp",layer_path="model.layers.{layer}",),- options=LayerOptions.staged_k4(), # 你的档位用 staged_k4 / staged_k8 等预设+ options=LayerOptions.staged_k4(), # use your tier's preset, e.g. staged_k4 / staged_k8prerouter=...,)def build_model(model_dir=None, **overrides):cfg = MyConfig.from_pretrained(model_dir, **overrides)- return ... # load_installed 或你的装配路径+ return ... # load_installed or your own assembly pathdef build_engine(model_dir=None, **overrides):cfg = MyConfig.from_pretrained(model_dir, **overrides)- return ... # 你的引擎+ return ... # your engineregister_model("my-model", sys.modules[__name__])Config = MyConfig```-## 参考+## References-- `src/edge0/registry.py` —— 注册表 + Auto 三件套-- `src/edge0/models/base.py` —— `ModelConfig` / `from_pretrained` / artifact 解析-- `src/edge0/models/edge0_35b/__init__.py`、`edge0_10b/__init__.py` —— 两个活例子-- `tests/test_registry.py`、`tests/test_moe_spec.py` —— 行为示例+- `src/edge0/registry.py` — the registry + the Auto trio+- `src/edge0/models/base.py` — `ModelConfig` / `from_pretrained` / artifact resolution+- `src/edge0/models/edge0_35b/__init__.py`, `edge0_10b/__init__.py` — the two live examples+- `tests/test_registry.py`, `tests/test_moe_spec.py` — behavior examplesdiff --git a/docs/architecture.md b/docs/architecture.mdindex e7ee01e..b1a6df3 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -1,138 +1,156 @@-# 架构总览--**edge0** 是一个开源的流式 MoE 推理框架,把生产部署中验证过的-「SSD 专家 offload + 并行 LoRA + prerouter 路由预判」方案抽象成可扩展的通用框架。-后端隔离设计:当前实现 MLX 后端(Apple Silicon),核心逻辑与后端解耦,-其它平台按同一套门面接入。-开箱支持两个模型:`edge0-35b`(Qwen3.5-MoE,K=4)与 `edge0-10b`(Ling 3.0-hybrid,K=8)。--## 核心设计目标--1. **像 transformers 一样使用**:`AutoConfig` / `AutoModel` / `AutoEngine` 按模型- 名(或 checkpoint 的 `model_type`)自动解析适配器;-2. **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,框架代码只通过- `edge0.backends` 暴露的 `core` / `nn` / `io` / `quant` 命名空间接触后端,- 为未来 CUDA 后端预留平级插槽(`backends/cuda/`,由 `EDGE0_BACKEND` 环境变量- 选择);-3. **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据的- `.safetensors`,旧 npz 训练导出只经一次性迁移脚本转换后即弃用;-4. **术语统一**:预路由头一律称 **prerouter**,代码与文档零旧术语残留。--## 分层结构+# Architecture Overview++**edge0** is an open-source streaming MoE inference framework that abstracts the+"SSD expert offload + parallel LoRA + prerouter routing prediction" scheme —+validated in production deployments — into an extensible, general-purpose+framework. Backends are isolated by design: the current release implements the+MLX backend (Apple Silicon), core logic is decoupled from the backend, and other+platforms plug in through the same facade. Two models are supported out of the+box: `edge0-35b` (Qwen3.5-MoE, K=4) and `edge0-10b` (Ling 3.0 hybrid, K=8).++## Core Design Goals++1. **Use it like transformers**: `AutoConfig` / `AutoModel` / `AutoEngine`+ automatically resolve the adapter by model name (or by the checkpoint's+ `model_type`);+2. **Backend isolation**: all MLX code lives under `edge0/backends/mlx/`, and+ framework code touches the backend only through the `core` / `nn` / `io` /+ `quant` namespaces exposed by `edge0.backends`, reserving a peer slot for a+ future CUDA backend (`backends/cuda/`, selected via the `EDGE0_BACKEND`+ environment variable);+3. **Adapters unified on safetensors**: LoRA and prerouter weights are both+ `.safetensors` files carrying metadata; legacy npz training exports are+ converted by a one-off migration script and then deprecated;+4. **Unified terminology**: the pre-routing head is always called the+ **prerouter** — zero legacy-term residue in code and docs.++## Layered Structure```src/edge0/-├── backends/ # 后端抽象(唯一允许接触 MLX 的边界)-│ ├── base.py # TensorStore 协议 + open_tensor_store()-│ ├── __init__.py # 按 EDGE0_BACKEND 装配 core/nn/io/quant-│ └── mlx/ # MLX 参考实现(_impl/ 为 vendored 模型)-├── config.py # GenerationConfig(采样参数)-├── sampling.py # 采样器(温度/top-k/top-p/重复惩罚,经 backends.core)-├── registry.py # 模型注册表 + AutoConfig/AutoModel/AutoEngine-├── attention/spec.py # 注意力规格(MHA/MLA 差异的抽象层)-├── moe/ # MoE 抽象+├── backends/ # backend abstraction (the only boundary allowed to touch MLX)+│ ├── base.py # TensorStore protocol + open_tensor_store()+│ ├── __init__.py # assembles core/nn/io/quant per EDGE0_BACKEND+│ └── mlx/ # MLX reference implementation (_impl/ holds the vendored models)+├── config.py # GenerationConfig (sampling parameters)+├── sampling.py # sampler (temperature/top-k/top-p/repetition penalty, via backends.core)+├── registry.py # model registry + AutoConfig/AutoModel/AutoEngine+├── attention/spec.py # attention spec (abstraction layer over MHA/MLA differences)+├── moe/ # MoE abstractions│ ├── spec.py # MoESpec/QuantSpec/RouterKind/WeightLayout-│ └── routing.py # 路由数学(与 vendored 模型 bit 级一致)-├── streaming/ # SSD 流式专家层-│ ├── options.py # LayerOptions(typed 替换部署期 env 旋钮)-│ ├── layer.py # StreamingSwitchGLU(每层状态机,全部执行路径)-│ ├── mmap.py # SafetensorsMmap(byte-range mmap)-│ └── cache.py # SharedExpertCache(跨层 LRU)-├── prerouter/ # 路由预判-│ ├── spec.py # PrerouterSpec(纯配置)-│ ├── install.py # 头权重安装-│ ├── heads.py # 头数学(fc1 -> erf gelu -> fc2 + linear_init)-│ ├── stager.py # 跨 token 预测保存/提交(每家族子类)-│ └── state.py # 跨 token 状态-├── engine/ # 推理编排(prefill/decode 循环,按模型子类化)-│ ├── base.py # 共享循环(经 backends.core)-│ ├── qwen.py # Qwen35Engine(K=4 档)-│ └── ling.py # Ling10BEngine(K=8 档)-├── adapters/lora.py # 并行 LoRA 安装(backends.nn)-├── models/ # 模型适配层(transformers 风格)-│ ├── base.py # ModelConfig 基类 + artifact 路径解析-│ ├── edge0_35b/ # edge0-35b 档(注册名 "edge0-35b")-│ └── edge0_10b/ # edge0-10b 档(注册名 "edge0-10b")-└── server/ # OpenAI 兼容 HTTP 服务(纯 stdlib)- ├── app.py # ThreadingHTTPServer + 路由- └── chat.py # /v1/chat/completions 会话状态+│ └── routing.py # routing math (bit-identical to the vendored models)+├── streaming/ # SSD streaming expert layers+│ ├── options.py # LayerOptions (typed replacement for deployment-time env knobs)+│ ├── layer.py # StreamingSwitchGLU (per-layer state machine, all execution paths)+│ ├── mmap.py # SafetensorsMmap (byte-range mmap)+│ └── cache.py # SharedExpertCache (cross-layer LRU)+├── prerouter/ # routing prediction+│ ├── spec.py # PrerouterSpec (pure config)+│ ├── install.py # head weight installation+│ ├── heads.py # head math (fc1 -> erf gelu -> fc2 + linear_init)+│ ├── stager.py # cross-token prediction save/commit (per-family subclasses)+│ └── state.py # cross-token state+├── engine/ # inference orchestration (prefill/decode loop, subclassed per model)+│ ├── base.py # shared loop (via backends.core)+│ ├── qwen.py # Qwen35Engine (K=4 tier)+│ └── ling.py # Ling10BEngine (K=8 tier)+├── adapters/lora.py # parallel LoRA installation (backends.nn)+├── models/ # model adapter layer (transformers-style)+│ ├── base.py # ModelConfig base class + artifact path resolution+│ ├── edge0_35b/ # edge0-35b tier (registered name "edge0-35b")+│ └── edge0_10b/ # edge0-10b tier (registered name "edge0-10b")+└── server/ # OpenAI-compatible HTTP service (pure stdlib)+ ├── app.py # ThreadingHTTPServer + routing+ └── chat.py # /v1/chat/completions session state```-### 依赖方向+### Dependency Direction```server / cli└─ engine ──> streaming / prerouter / adapters ──> moe / attention spec└──────────────> backends.{core,nn,io,quant}- └─> backends/mlx/(MLX 唯一入口)+ └─> backends/mlx/ (the only entry point to MLX)```-纯配置层(`config.py`、`moe/spec.py`、`prerouter/spec.py`、`streaming/options.py`、-`registry.py`、`models/*`、`server/`)不 import 任何后端包;运行时逻辑一律通过-`edge0.backends` 命名空间。CI 用 grep 强制:MLX 包不得在 `backends/mlx/` 之外被-import。+Pure configuration layers (`config.py`, `moe/spec.py`, `prerouter/spec.py`,+`streaming/options.py`, `registry.py`, `models/*`, `server/`) import no backend+package; runtime logic always goes through the `edge0.backends` namespace.+CI enforces this with grep: MLX packages must not be imported outside+`backends/mlx/`.-## 关键数据流+## Key Data Flows-### Prefill(整层装载)+### Prefill (full-layer loading)-对前 `prefill_full_layers` 个层:`load_full_layer()` 把整层 9 个张量(gate/up/down-× weight/scales/biases)按 bundle 布局装入,`_gather_sort` + 排序 gather + 反排序-一次前向算完,随后 `clear_full_layer()` 释放,页缓存承担热数据(E3b 策略)。-其余层走 hot-stack(LRU 常驻 top-N 专家)或 on-demand exact 路径。+For the first `prefill_full_layers` layers: `load_full_layer()` loads all 9+tensors of the layer (gate/up/down × weight/scales/biases) in the bundle+layout, the forward pass is computed in one shot via `_gather_sort` + sorted+gather + unsort, and `clear_full_layer()` then releases it, with the page cache+carrying the hot data (E3b strategy). The remaining layers go through the+hot-stack (LRU-resident top-N experts) or the on-demand exact path.-### Decode(staged 双缓冲)+### Decode (staged double buffering)-1. prerouter 头在 token t-1 预测 token t 的专家集;-2. `stager` 把预测写入双缓冲槽;step 边界同步/异步填充槽位;-3. `StreamingSwitchGLU` 用槽位表 + 增量栈(`incr_stack`)把 9 个- `mx.stack` 图节点换成本地 `put_along_axis` 行写入;-4. 专家权重经 `quant.gather_qmm` 按槽位 gather,路由索引全程不离开 GPU。+1. the prerouter head predicts the expert set for token t at token t-1;+2. the `stager` writes predictions into double-buffered slots; slots are filled+ synchronously/asynchronously at step boundaries;+3. `StreamingSwitchGLU` uses a slot table + incremental stack (`incr_stack`) to+ replace the 9 `mx.stack` graph nodes with local `put_along_axis` row writes;+4. expert weights are gathered by slot via `quant.gather_qmm`, and routing+ indices never leave the GPU.-### 采样与生成+### Sampling and Generation-`engine/base.py` 的共享循环驱动 prefill → decode,逐 token 调-`sampling.sample()`(温度/top-k/top-p/重复惩罚,向量化历史惩罚,单次-host-sync categorical 抽样)。+The shared loop in `engine/base.py` drives prefill → decode, calling+`sampling.sample()` per token (temperature/top-k/top-p/repetition penalty,+vectorized history penalty, a single host-sync categorical draw).-## 注册表与模型接入+## Registry and Model Integration-适配器模块暴露三件套(契约):+Adapter modules expose a three-part contract:-- `Config` —— `ModelConfig` 子类(`from_pretrained` 合并 override);-- `build_model(model_dir, **overrides)` —— 装配好的模型骨架;-- `build_engine(model_dir, **overrides)` —— 可生成引擎。+- `Config` — a `ModelConfig` subclass (`from_pretrained` merges overrides);+- `build_model(model_dir, **overrides)` — the assembled model skeleton;+- `build_engine(model_dir, **overrides)` — a generation-ready engine.-`register_model(name, module)` 注册;`TYPE_ALIASES` 把 checkpoint-`config.json` 的 `model_type`(如 `qwen3_5_moe`、`bailing_hybrid`)映射到注册名,-因此 `AutoEngine.from_pretrained(model_dir=...)` 可以直接从目录解析。-详见 [adding-a-model.md](adding-a-model.md)。+`register_model(name, module)` performs the registration; `TYPE_ALIASES` maps+the `model_type` in the checkpoint's `config.json` (e.g. `qwen3_5_moe`,+`bailing_hybrid`) to registered names, so+`AutoEngine.from_pretrained(model_dir=...)` can resolve directly from a+directory. See [adding-a-model.md](adding-a-model.md) for details.-## 后端隔离的落法+## Backend Isolation in Practice-`edge0.backends` 是框架唯一允许出现的后端 import 面:+`edge0.backends` is the only backend import surface allowed in the framework:-- `core` —— 数组与算子命名空间(matmul/softmax/topk/take/eval/compile/random 等);-- `nn` —— 模块工厂(Module/Linear/RMSNorm/silu/gelu);-- `io` —— 模型/分词器/张量库加载;-- `quant` —— 量化 gather 内核(`gather_qmm`)。+- `core` — the array and operator namespace (matmul/softmax/topk/take/eval/+ compile/random, etc.);+- `nn` — module factory (Module/Linear/RMSNorm/silu/gelu);+- `io` — model/tokenizer/tensor-store loading;+- `quant` — the quantized gather kernel (`gather_qmm`).-MLX 后端是参考实现;未来 CUDA 后端实现同一表面即可复用全部框架代码。-注意:**每个模型的引擎胶水**(`engine/qwen.py`、`engine/ling.py`)因绑定 vendored-后端模型而引用 `edge0.backends.mlx._impl`,这是有意的例外——模型本身就是后端-资产,编排循环(`engine/base.py`)才是跨后端共享的。+The MLX backend is the reference implementation; a future CUDA backend that+implements the same surface can reuse all of the framework code.+Note: **the per-model engine glue** (`engine/qwen.py`, `engine/ling.py`)+references `edge0.backends.mlx._impl` because it binds to the vendored backend+models — this is a deliberate exception: the models themselves are backend+assets, while the orchestration loop (`engine/base.py`) is what is shared+across backends.-## 测试策略+## Testing Strategy-- `tests/test_streaming_math.py` —— 数学契约:exact/staged/hot/full 各路径与- dequant 参考逐元素对比(相对 L2 < 1%),防 gate/up 交换、行错位类回归;-- `tests/test_moe_spec.py`、`tests/test_registry.py`、`tests/test_sampling.py` ——- 纯逻辑层单测,无需真实 checkpoint;-- 验收指标:`target_tok_s` 与 `peak_active_mem_mb` 记录的基准环境实测- 吞吐与峰值活跃内存,作为回归标尺。+- `tests/test_streaming_math.py` — math contract: the exact/staged/hot/full+ paths are each compared element-wise against the dequant reference (relative+ L2 < 1%), guarding against gate/up-swap and row-misalignment regressions;+- `tests/test_moe_spec.py`, `tests/test_registry.py`, `tests/test_sampling.py`+ — pure-logic-layer unit tests that need no real checkpoint;+- Acceptance metrics: throughput and peak active memory measured on the+ benchmark environment and recorded by `target_tok_s` and+ `peak_active_mem_mb`, used as the regression yardstick.## License-Apache-2.0,含 vendored 第三方模型代码(mlx-lm 的 Qwen3.5-MoE、生产部署的-ling backbone),详见根目录 [NOTICE](../NOTICE)。+Apache-2.0, including vendored third-party model code (Qwen3.5-MoE from mlx-lm+and the Ling backbone from the reference deployment); see the root+[NOTICE](../NOTICE) for details.diff --git a/docs/attention.md b/docs/attention.mdindex 84fa02e..0a29371 100644--- a/docs/attention.md+++ b/docs/attention.md@@ -1,19 +1,14 @@-# 注意力规格(AttentionSpec)+# Attention Specification (`AttentionSpec`)-`edge0` 不重新实现注意力内核:具体的序列混合计算由 **vendored 基础模型**携带-(`edge0.backends.mlx._impl` 中的 GQA / GatedDeltaNet 对应 edge0-35b,-MLA / DeltaNet 对应 edge0-10b)。注意力内核属于**后端模型资产**,不属于框架。+`edge0` does not re-implement attention kernels: the concrete sequence-mixing computation is carried by the **vendored base model** (GQA / GatedDeltaNet in `edge0.backends.mlx._impl` for edge0-35b; MLA / DeltaNet for edge0-10b). Attention kernels are **backend model assets**, not part of the framework.-那为什么还要一个 `AttentionSpec`?因为框架需要在**不依赖具体实现**的情况下-**审视(introspect)**一个模型:缓存尺寸、层角色、文档、未来的内核适配。-`AttentionSpec` 就是这层「注意力分类学」的载体。本文档说明它是什么、字段语义、-引擎如何使用,以及为什么单独抽象(MLA/MHA 的差异)。+So why have an `AttentionSpec` at all? Because the framework needs to **introspect** a model **without depending on a concrete implementation**: cache sizing, layer roles, documentation, and future kernel adaptation. `AttentionSpec` is the carrier for this "attention taxonomy" layer. This document explains what it is, the field semantics, how the engine uses it, and why it is a separate abstraction (the MLA/MHA differences).-源码:`src/edge0/attention/spec.py`。+Source: `src/edge0/attention/spec.py`.-## 设计动机+## Design Motivation-模块头部的 docstring 明确了边界:+The docstring at the top of the module draws the boundary explicitly:```edge0 does not re-implement attention kernels: the vendored base model@@ -24,103 +19,87 @@ sizing, layer roles, documentation, and future kernels — without knowingthe concrete implementation.```-新增一种注意力类型 = 一个新的 `AttentionKind` 成员 + 一份 `AttentionSpec` 描述 +-一个内核实现(在 vendored 基础模型或某个后端的 `_impl` 模块里)。+Adding a new attention type = a new `AttentionKind` member + an `AttentionSpec` description + a kernel implementation (in the vendored base model or in a backend's `_impl` module).-## 注意力种类:`AttentionKind`+## Attention Kinds: `AttentionKind`-`AttentionKind` 是 `str` 枚举,标识 edge0 支持的注意力 / 序列混合家族。+`AttentionKind` is a `str` enum identifying the attention / sequence-mixing families edge0 supports.-| 成员 | 值 | 含义 |+| Member | Value | Meaning || --- | --- | --- |-| `GQA` | `"gqa"` | 分组查询注意力,softmax + KV cache |-| `MLA` | `"mla"` | 多头潜在注意力(压缩 KV) |-| `DELTANET` | `"deltanet"` | DeltaNet 线性注意力(门控 delta 规则,无 KV cache) |-| `GATED_DELTANET` | `"gated_deltanet"` | GatedDeltaNet(带衰减的门控 delta / 线性注意力) |-| `DENSE_MLP` | `"dense_mlp"` | 无序列混合(纯 MLP 层) |+| `GQA` | `"gqa"` | Grouped-query attention, softmax + KV cache |+| `MLA` | `"mla"` | Multi-head latent attention (compressed KV) |+| `DELTANET` | `"deltanet"` | DeltaNet linear attention (gated delta rule, no KV cache) |+| `GATED_DELTANET` | `"gated_deltanet"` | GatedDeltaNet (gated delta / linear attention with decay) |+| `DENSE_MLP` | `"dense_mlp"` | No sequence mixing (pure MLP layer) |-`DENSE_MLP` 的存在说明:一个模型的某些层可能根本没有注意力块(例如 edge0-10b 的-layer 0 是 dense 层),分类学要能显式表达「这一层不做序列混合」。+The existence of `DENSE_MLP` says that some layers of a model may have no attention block at all (for example, layer 0 of edge0-10b is a dense layer), and the taxonomy must be able to state explicitly that "this layer does no sequence mixing".-## 规格:`AttentionSpec`+## The Spec: `AttentionSpec`-`AttentionSpec` 是一个 `frozen=True` 的 dataclass,描述**一个层**的序列混合块。+`AttentionSpec` is a `frozen=True` dataclass describing the sequence-mixing block of **a single layer**.-| 字段 | 类型 | 默认值 | 语义 |+| Field | Type | Default | Semantics || --- | --- | --- | --- |-| `kind` | `AttentionKind` | (必填) | 注意力家族 |-| `layer_indices` | `tuple[int, ...]` | (必填) | 该规格覆盖的 0 基层号 |-| `num_heads` | `int \| None` | `None` | 头数(仅 GQA/MLA;否则为 `None`) |-| `num_kv_heads` | `int \| None` | `None` | KV 头数(仅 GQA/MLA) |-| `head_dim` | `int \| None` | `None` | 每头维度(仅 GQA/MLA) |-| `cache` | `bool` | `True` | 该块是否维护 KV 式缓存(线性注意力层为 `False`) |-| `notes` | `str` | `""` | 自由说明(例如混合调度计划) |+| `kind` | `AttentionKind` | (required) | Attention family |+| `layer_indices` | `tuple[int, ...]` | (required) | 0-based layer indices covered by this spec |+| `num_heads` | `int \| None` | `None` | Number of heads (GQA/MLA only; otherwise `None`) |+| `num_kv_heads` | `int \| None` | `None` | Number of KV heads (GQA/MLA only) |+| `head_dim` | `int \| None` | `None` | Per-head dimension (GQA/MLA only) |+| `cache` | `bool` | `True` | Whether this block maintains a KV-style cache (`False` for linear-attention layers) |+| `notes` | `str` | `""` | Free-form notes (e.g. a hybrid scheduling plan) |-要点:+Key points:-- `cache` 字段区分了「softmax 注意力(要 KV cache)」与「线性注意力(无 KV cache)」,- 这是缓存尺寸计算的关键分叉。-- `layer_indices` 是一个元组而非单层,因为**一个规格可以覆盖多个层**(例如一个- GatedDeltaNet 规格可以描述连续的若干层),也便于表达混合调度。-- `__repr__` 被设计为**紧凑单行**,便于打日志 / 文档,例如:- `AttentionSpec(gated_deltanet, layers=0..19, cache=False)`。+- The `cache` field distinguishes "softmax attention (needs a KV cache)" from "linear attention (no KV cache)" — the key fork in cache-size computation.+- `layer_indices` is a tuple rather than a single layer because **one spec can cover multiple layers** (for example, a single GatedDeltaNet spec can describe a run of consecutive layers), and it also makes hybrid scheduling easy to express.+- `__repr__` is designed to be a **compact single line**, convenient for logging / documentation, e.g. `AttentionSpec(gated_deltanet, layers=0..19, cache=False)`.-### 汇总:`summarize()`+### Summary: `summarize()````pythondef summarize(specs: list[AttentionSpec]) -> str```-把一份规格列表压缩成一行人类可读摘要,例如用于 CLI 启动横幅:+Compresses a list of specs into a one-line human-readable summary, e.g. for the CLI startup banner:```gated_deltanetx20, dense_mlpx1```-即 `{kind.value}x{len(layer_indices)}` 的逗号拼接。文档标注它用于 CLI banner。+That is, comma-joined `{kind.value}x{len(layer_indices)}`. The documentation notes it is used for the CLI banner.-## 引擎如何使用+## How the Engine Uses It-当前源码里,`AttentionSpec` 及其辅助函数**尚未被任何引擎 / 模型代码引用**——它是一份-「先定义、后接线」的分类学模块(`grep` 全仓仅命中 `attention/spec.py` 自身)。-这一点需要诚实标注:`AttentionKind` / `AttentionSpec` / `summarize` 目前的消费方只有-本模块,属于面向未来的框架设施,其设计意图(缓存尺寸、层角色、文档、未来内核)由-docstring 与字段定义承载。+In the current source, `AttentionSpec` and its helpers **are not yet referenced by any engine / model code** — it is a "define first, wire up later" taxonomy module (a repo-wide `grep` only hits `attention/spec.py` itself). This deserves an honest note: the current consumers of `AttentionKind` / `AttentionSpec` / `summarize` are only this module itself; it is forward-looking framework infrastructure whose design intent (cache sizing, layer roles, documentation, future kernels) is carried by the docstring and the field definitions.-即便如此,抽象方向已经明确:+Even so, the direction of the abstraction is already clear:-- **缓存尺寸**:`cache` 字段让框架无需理解内核细节即可判断该层要不要分配 KV cache、- 以及(对 GQA/MLA)按 `num_heads` / `num_kv_heads` / `head_dim` 计算尺寸。-- **层角色**:`kind` 把「softmax 注意力 / 线性注意力 / 无混合」区分开,供调度、文档与- 性能剖析参考。-- **未来内核**:新增注意力类型无需改动框架其余部分,只要新加枚举成员 + 规格 + 内核。+- **Cache sizing**: the `cache` field lets the framework decide — without understanding kernel internals — whether a layer needs a KV cache allocated, and (for GQA/MLA) compute its size from `num_heads` / `num_kv_heads` / `head_dim`.+- **Layer roles**: `kind` separates "softmax attention / linear attention / no mixing", for scheduling, documentation, and profiling to refer to.+- **Future kernels**: adding a new attention type requires no changes to the rest of the framework — just a new enum member + spec + kernel.-这与后端抽象(`edge0.backends`)是一致的思想:框架代码只依赖抽象的规格 / 命名空间,-具体数学由后端或 vendored 模型实现。+This is the same idea as the backend abstraction (`edge0.backends`): framework code depends only on abstract specs / namespaces, while the concrete math is implemented by a backend or the vendored model.-## 为什么单独抽象(MLA / MHA 差异)+## Why a Separate Abstraction (MLA / MHA Differences)-与其说「引擎在跑 attention」,不如说引擎在编排「一层序列混合」。把注意力显式抽象出来,-是为了容纳差异巨大的实现而不改框架:+Rather than "the engine runs attention", it is more accurate to say the engine orchestrates "a layer of sequence mixing". Making attention an explicit abstraction exists to accommodate wildly different implementations without changing the framework:-| 维度 | MHA / GQA(softmax) | MLA / 线性注意力 |+| Dimension | MHA / GQA (softmax) | MLA / linear attention || --- | --- | --- |-| KV 表示 | 完整缓存,随层增长 | 压缩潜在 KV |-| 缓存需求 | 需要 KV cache(`cache=True`) | 无 / 极简缓存(`cache=False`) |-| 头几何 | `num_heads` / `num_kv_heads` / `head_dim` 有意义 | 通常不适用(`None`) |-| 混合层级 | 几乎每层 | 可能只有部分层(含 dense 层) |+| KV representation | Full cache, growing with layers | Compressed latent KV |+| Cache requirement | Needs a KV cache (`cache=True`) | None / minimal cache (`cache=False`) |+| Head geometry | `num_heads` / `num_kv_heads` / `head_dim` are meaningful | Usually not applicable (`None`) |+| Mixing layer coverage | Nearly every layer | Possibly only some layers (incl. dense layers) |-`AttentionSpec` 用统一的字段集表达这两类差异:`kind` 表达家族,`cache` 表达缓存有无,-头几何字段仅在 GQA/MLA 下填充,`layer_indices` 表达混合层级与调度。这样框架对-edge0-35b 的 GatedDeltaNet 与 edge0-10b 的 MLA 可以走同一套审视逻辑,而不必各自-硬编码。+`AttentionSpec` expresses both families with a single, uniform field set: `kind` carries the family, `cache` carries cache presence, the head-geometry fields are populated only for GQA/MLA, and `layer_indices` carries the mixing layer coverage and scheduling. The framework can then run the same introspection logic for edge0-35b's GatedDeltaNet and edge0-10b's MLA, without hardcoding each of them.-## 新增一种注意力的步骤+## Steps to Add a New Attention Type-按模块 docstring,新增 `Xxx` 注意力:+Per the module docstring, adding an `Xxx` attention:-1. 在 `AttentionKind` 加一个成员;-2. 写一份 `AttentionSpec` 描述;-3. 提供内核实现(vendored 基础模型或某后端 `_impl` 模块)。+1. Add a member to `AttentionKind`;+2. Write an `AttentionSpec` description;+3. Provide a kernel implementation (in the vendored base model or a backend's `_impl` module).-以上即 `src/edge0/attention/spec.py` 的全部内容与意图。+The above is the entire content and intent of `src/edge0/attention/spec.py`.diff --git a/docs/models/edge0-10b.md b/docs/models/edge0-10b.mdindex 2067bb4..4a66a0c 100644--- a/docs/models/edge0-10b.md+++ b/docs/models/edge0-10b.md@@ -1,64 +1,62 @@# edge0-10b-edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)的 10B 级稀疏混合专家模型。相比 35b 档牺牲一定的参数量,换来更低的峰值内存与更高的生成速率,适合对延迟与显存敏感的场景。+The lightweight, high-throughput tier of the edge0 platform: a 10B-class sparse mixture-of-experts model built on the Ling 3.0 hybrid architecture (MLA + MoE). It trades some parameter count versus the 35b tier for lower peak memory and a higher generation rate, making it a good fit for latency- and memory-sensitive scenarios.-性能档案基于当前发布 adapter 版本的基准实测,默认由 `Ling10BConfig`(`src/edge0/models/edge0_10b/__init__.py`)固定。+The performance profile is based on benchmarks of the current release adapter version and is pinned by default in `Ling10BConfig` (`src/edge0/models/edge0_10b/__init__.py`).-## 模型档案+## Model profile-| 项目 | 值 |+| Item | Value || --- | --- |-| 参数量级 | 10B 级 |-| 层数 | 24(第 0 层为 dense) |-| 专家数 | 128(路由专家)+ 1 常驻共享专家 |-| top_k(K) | 8(原生路由宽度) |-| 路由方式 | `SIGMOID_GROUP`(sigmoid + group 限制 top-k:`n_group=8, topk_group=4, routed_scaling=2.5, norm_topk_prob=True`) |-| 专家量化 | 4-bit affine,group 64 |-| 权重布局 | `WeightLayout.SEPARATE`(gate/up/down 分离张量堆叠) |-| 专家权重路径 | `model.layers.N.mlp.experts` |-| 前置路由(prerouter) | 16 个头(显式 owners 7..22),start_layer 7,hidden 512,fp16 |-| 解码调用方式 | `patch_call=False`(头内建在 `BailingSparseMoE` 中,从 `prerouter_cache` logits 消费) |+| Parameter scale | 10B-class |+| Number of layers | 24 (layer 0 is dense) |+| Number of experts | 128 routed experts + 1 always-resident shared expert |+| top_k (K) | 8 (native routing width) |+| Routing | `SIGMOID_GROUP` (sigmoid + group-constrained top-k: `n_group=8, topk_group=4, routed_scaling=2.5, norm_topk_prob=True`) |+| Expert quantization | 4-bit affine, group 64 |+| Weight layout | `WeightLayout.SEPARATE` (gate/up/down stacked as separate tensors) |+| Expert weight path | `model.layers.N.mlp.experts` |+| Prerouter | 16 heads (explicit owners 7..22), start_layer 7, hidden 512, fp16 |+| Decode invocation | `patch_call=False` (heads are built into `BailingSparseMoE` and consume logits from `prerouter_cache`) || LoRA | `r=16, alpha=32.0` |-| 预填分块 | 2048 |-| 热窗 | 1 |-| 流式预取历史 | 开(`prefetch_history=True`) |-| 服务端口 | 8083 |-| 实测吞吐 | 23.9–25.3 tok/s(M4 Pro) |-| 实测峰值激活内存 | ≈ 1.0 GB(短上下文)/ 3.1 GB(3.3k token 上下文) |+| Prefill chunk | 2048 |+| Hot window | 1 |+| Streaming prefetch history | on (`prefetch_history=True`) |+| Serving port | 8083 |+| Measured throughput | 23.9–25.3 tok/s (M4 Pro) |+| Measured peak activation memory | ≈ 1.0 GB (short context) / 3.1 GB (3.3k-token context) |-> 说明:数值全部取自 `Ling10BConfig._defaults()` 与 `LayerOptions.prod_k8()`。头部数量来自显式 `owners=range(7, 23)`,共 16 个头(当前发布头部分布,L7 起消费预测,L1–6 走原始 router);`feature_topk="executed"`。+> Note: all values are taken from `Ling10BConfig._defaults()` and `LayerOptions.prod_k8()`. The head count comes from the explicit `owners=range(7, 23)` — 16 heads in total (the current release head distribution: prediction is consumed from L7 onward, and L1–6 use the original router); `feature_topk="executed"`.-## 分阶段解码(staged decode)+## Staged decode-该档使用 `LayerOptions.prod_k8()` 预设(对齐参考部署的生产开关):+This tier uses the `LayerOptions.prod_k8()` preset (aligned with the reference deployment's production switches):-- 分阶段解码关闭(`staged=False`,`staged_sync=False`,`staged_n=8`)——部署验证- 该档上 staged decode 会劣化输出,prerouter 直接驱动下一 token 的专家- 预取(step 边界 `stage_all` + prefill 尾部各一次)。-- 专家缓存 `cache_slots=64`,热专家钉住关闭(`hot_per_layer=0`)。-- 整层 E3b prefill(`full_layer_prefill=True`,`prefill_chunk=2048`)。+- Staged decode is off (`staged=False`, `staged_sync=False`, `staged_n=8`) — deployment verification showed that staged decode degrades output on this tier, so the prerouter directly drives expert prefetch for the next token (one `stage_all` at the step boundary and one at the prefill tail).+- Expert cache `cache_slots=64`, hot-expert pinning off (`hot_per_layer=0`).+- Full-layer E3b prefill (`full_layer_prefill=True`, `prefill_chunk=2048`).-## 使用方式+## Usage-### CLI 起服务+### Serving via the CLI```bashedge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8083```-可选参数:+Optional arguments:-- `--no-prerouter`:禁用前置路由(`prerouter=None`)。-- `--no-lora`:禁用 LoRA(`lora=""`)。-- `--flask`:改用 Flask 传输(需要安装 flask,支持 SSE 流式)。+- `--no-prerouter`: disable the prerouter (`prerouter=None`).+- `--no-lora`: disable LoRA (`lora=""`).+- `--flask`: switch to the Flask transport (requires flask to be installed; supports SSE streaming).-单轮对话(终端)可改用 `chat`:+For single-turn chat in the terminal, use `chat` instead:```bash-edge0 chat --name edge0-10b --model-dir /path/to/checkpoint --prompt "你好"+edge0 chat --name edge0-10b --model-dir /path/to/checkpoint --prompt "Hello"```-查看该档默认档案:+To view this tier's default profile:```bashedge0 models@@ -69,48 +67,48 @@ edge0 models```pythonfrom edge0 import AutoEngine, AutoModel, AutoConfig-# 直接可生成的引擎+# an engine ready to generateengine = AutoEngine.from_pretrained("/path/to/checkpoint", name="edge0-10b",)-ids = engine.generate([156895]) # 内部使用配置里的默认采样+ids = engine.generate([156895]) # uses the default sampling from the configtext = engine._tok.decode(ids)engine.close()-# 仅加载权重(流式专家 + prerouter + LoRA 已装配)+# weights only (streaming experts + prerouter + LoRA installed)model = AutoModel.from_pretrained("/path/to/checkpoint", name="edge0-10b")-# 仅拿配置+# config onlycfg = AutoConfig.from_pretrained("/path/to/checkpoint", name="edge0-10b")```-`AutoEngine` / `AutoModel` / `AutoConfig` 三者也可省略 `name`,从 checkpoint 的 `config.json` 的 `model_type` 或目录 basename 自动解析(见 `src/edge0/registry.py`)。+`AutoEngine` / `AutoModel` / `AutoConfig` can all omit `name` and resolve automatically from the `model_type` in the checkpoint's `config.json` or from the directory basename (see `src/edge0/registry.py`).## HTTP API-`edge0 serve` 暴露一个 OpenAI 兼容的单模型端点。引擎一次独占一个请求,生成以 FIFO 队列串行执行。+`edge0 serve` exposes a single OpenAI-compatible model endpoint. The engine serves one exclusive request at a time; generation is serialized through a FIFO queue.-| 方法 | 路径 | 说明 |+| Method | Path | Description || --- | --- | --- |-| `GET` | `/healthz` | 健康检查 |-| `GET` | `/v1/models` | 列出已加载模型 |-| `POST` | `/v1/chat/completions` | 对话补全(支持 `stream`) |-| `POST` | `/v1/completions` | 不支持,返回 400 |+| `GET` | `/healthz` | Health check |+| `GET` | `/v1/models` | List loaded models |+| `POST` | `/v1/chat/completions` | Chat completions (supports `stream`) |+| `POST` | `/v1/completions` | Not supported; returns 400 |-### 非流式对话+### Non-streaming chat```bashcurl -s http://127.0.0.1:8083/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"model": "edge0-10b",- "messages": [{"role": "user", "content": "介绍一下 Ling"}],+ "messages": [{"role": "user", "content": "Introduce Ling"}],"temperature": 0.7,"max_tokens": 256}'```-响应字段(非流式):+Response fields (non-streaming):```json{@@ -127,25 +125,25 @@ curl -s http://127.0.0.1:8083/v1/chat/completions \}```-请求可选字段:`model`、`messages`(含 `role`/`content`,content 支持多段文本自动拼接)、`temperature`、`top_p`、`top_k`、`max_tokens`、`seed`、`stream`。+Optional request fields: `model`, `messages` (with `role`/`content`; content supports multiple text segments that are concatenated automatically), `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`, `stream`.-### 流式对话(需 Flask)+### Streaming chat (requires Flask)```bashcurl -N http://127.0.0.1:8083/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"model": "edge0-10b",- "messages": [{"role": "user", "content": "数到五"}],+ "messages": [{"role": "user", "content": "Count to five"}],"stream": true}'```-每个 token 输出一段 `data: {"object":"chat.completion.chunk", ...}` SSE 事件,结束以 `data: [DONE]` 收尾。+Each token emits one `data: {"object":"chat.completion.chunk", ...}` SSE event, and the stream ends with `data: [DONE]`.-## 配置覆盖+## Configuration overrides-`from_pretrained` 支持对任意公开字段做覆盖(未知字段会抛 `TypeError`)。+`from_pretrained` supports overriding any public field (unknown fields raise a `TypeError`).```pythonfrom edge0 import AutoEngine@@ -153,12 +151,12 @@ from edge0 import AutoEngineengine = AutoEngine.from_pretrained("/path/to/checkpoint",name="edge0-10b",- port=9083, # 覆盖默认端口 8083- target_tok_s=35.0, # 覆盖验收吞吐目标- prerouter=None, # 关闭前置路由- lora="", # 关闭 LoRA- prefill_chunk=1024, # 缩小预填分块+ port=9083, # override the default port 8083+ target_tok_s=35.0, # override the acceptance throughput target+ prerouter=None, # disable the prerouter+ lora="", # disable LoRA+ prefill_chunk=1024, # smaller prefill chunk)```-CLI 里对应的覆盖是 `--no-prerouter` / `--no-lora`(见 `src/edge0/cli.py` 的 `_engine_kwargs`)。引擎参数覆盖请直接走 `Ling10BConfig.from_pretrained(model_dir, **overrides)`。+The corresponding CLI overrides are `--no-prerouter` / `--no-lora` (see `_engine_kwargs` in `src/edge0/cli.py`). To override engine parameters, call `Ling10BConfig.from_pretrained(model_dir, **overrides)` directly.diff --git a/docs/models/edge0-35b.md b/docs/models/edge0-35b.mdindex 9e16024..ac227c2 100644--- a/docs/models/edge0-35b.md+++ b/docs/models/edge0-35b.md@@ -1,64 +1,63 @@# edge0-35b-edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B 级稀疏混合专家模型。通过流式专家加载(streaming experts)、训练前置路由(prerouter)与 LoRA 适配,把整份权重量化后驻留在磁盘、按需装载,在单台设备上即可服务。+The flagship first tier of the edge0 platform: a 35B-class sparse mixture-of-experts model built on Qwen3.5-MoE (K=4 tier). Through streaming expert loading, a trained prerouter, and LoRA adaptation, the full quantized weight set resides on disk and is loaded on demand, so the model can be served on a single device.-性能档案基于当前发布 adapter 版本的基准实测,默认由 `Qwen35Config`(`src/edge0/models/edge0_35b/__init__.py`)固定。+The performance profile is based on benchmarks of the current release adapter version and is pinned by default in `Qwen35Config` (`src/edge0/models/edge0_35b/__init__.py`).-## 模型档案+## Model profile-| 项目 | 值 |+| Item | Value || --- | --- |-| 参数量级 | 35B 级 |-| 层数 | 40 |-| 专家数 | 256(路由专家)+ 1 常驻共享专家 |-| top_k(K) | 4 |-| 路由方式 | `SOFTMAX_TOPK`(softmax → top-k → renormalize,`norm_topk_prob=True`) |-| 专家量化 | 4-bit affine,group 64 |-| 权重布局 | `WeightLayout.SEPARATE`(gate/up/down 分离张量堆叠) |-| 专家权重路径 | `language_model.model.layers.N.mlp.switch_mlp` |-| 前置路由(prerouter) | 33 个头(owners 6..38),start_layer 7,hidden 512,fp16 |-| 解码调用方式 | `patch_call=True`,跨 token 分阶段解码,K=4 |+| Parameter scale | 35B-class |+| Number of layers | 40 |+| Number of experts | 256 routed experts + 1 always-resident shared expert |+| top_k (K) | 4 |+| Routing | `SOFTMAX_TOPK` (softmax → top-k → renormalize, `norm_topk_prob=True`) |+| Expert quantization | 4-bit affine, group 64 |+| Weight layout | `WeightLayout.SEPARATE` (gate/up/down stacked as separate tensors) |+| Expert weight path | `language_model.model.layers.N.mlp.switch_mlp` |+| Prerouter | 33 heads (owners 6..38), start_layer 7, hidden 512, fp16 |+| Decode invocation | `patch_call=True`, staged decode across tokens, K=4 || LoRA | `r=16, alpha=32.0` |-| 预填分块 | 2048 |-| 热窗 | 4 |-| 流式预取历史 | 开(`prefetch_history=True`) |-| 服务端口 | 8085 |-| 实测吞吐 | 14.9–17.7 tok/s(M4 Pro) |-| 实测峰值激活内存 | ≈ 3.3 GB |+| Prefill chunk | 2048 |+| Hot window | 4 |+| Streaming prefetch history | on (`prefetch_history=True`) |+| Serving port | 8085 |+| Measured throughput | 14.9–17.7 tok/s (M4 Pro) |+| Measured peak activation memory | ≈ 3.3 GB |-> 说明:数值全部取自 `Qwen35Config._defaults()` 与 `LayerOptions.staged_k4()`。头部数量来自显式 `owners` 列表(6 到 38,共 33 个头);`feature_topk="executed"` 表示喂入头的 top-k 特征即解码时实际路由的集合。+> Note: all values are taken from `Qwen35Config._defaults()` and `LayerOptions.staged_k4()`. The head count comes from the explicit `owners` list (6 through 38, 33 heads in total); `feature_topk="executed"` means the top-k features fed to the heads are exactly the set actually routed at decode time.-## 分阶段解码(staged decode)+## Staged decode-该档使用 `LayerOptions.staged_k4()` 预设:+This tier uses the `LayerOptions.staged_k4()` preset:-- 固定槽分阶段解码(`staged=True`,`staged_n=4`,`staged_sync=True`),逐层无 host 同步。-- `staged_replace=False`:路由由训练好的 prerouter 头提供(MoE 块经由 prerouter logits 路由),因此分阶段集合与路由集合完全一致,槽表映射零丢弃。-- 按需 prefill(`full_layer_prefill=False`):prefill 与解码走同一条按需- 专家装载路径(生产档 `QWEN_PREFILL_FULL=0` 对齐),峰值内存与部署一致;-- 常驻热专家钉住关闭(`hot_per_layer=0`,生产档 `QWEN_HOT=0` 对齐)。+- Fixed-slot staged decode (`staged=True`, `staged_n=4`, `staged_sync=True`), with no per-layer host synchronization.+- `staged_replace=False`: routing is supplied by the trained prerouter heads (the MoE blocks route via prerouter logits), so the staged set and the routed set are exactly identical and the slot-table mapping discards nothing.+- On-demand prefill (`full_layer_prefill=False`): prefill and decode take the same on-demand expert loading path (aligned with the production profile's `QWEN_PREFILL_FULL=0`), so peak memory matches deployment;+- Always-resident hot-expert pinning is off (`hot_per_layer=0`, aligned with the production profile's `QWEN_HOT=0`).-## 使用方式+## Usage-### CLI 起服务+### Serving via the CLI```bashedge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8085```-可选参数:+Optional arguments:-- `--no-prerouter`:禁用前置路由(`prerouter=None`)。-- `--no-lora`:禁用 LoRA(`lora=""`)。-- `--flask`:改用 Flask 传输(需要安装 flask,支持 SSE 流式)。+- `--no-prerouter`: disable the prerouter (`prerouter=None`).+- `--no-lora`: disable LoRA (`lora=""`).+- `--flask`: switch to the Flask transport (requires flask to be installed; supports SSE streaming).-单轮对话(终端)可改用 `chat`:+For single-turn chat in the terminal, use `chat` instead:```bash-edge0 chat --name edge0-35b --model-dir /path/to/checkpoint --prompt "你好"+edge0 chat --name edge0-35b --model-dir /path/to/checkpoint --prompt "Hello"```-查看该档默认档案:+To view this tier's default profile:```bashedge0 models@@ -69,48 +68,48 @@ edge0 models```pythonfrom edge0 import AutoEngine, AutoModel, AutoConfig-# 直接可生成的引擎+# an engine ready to generateengine = AutoEngine.from_pretrained("/path/to/checkpoint", name="edge0-35b",)-ids = engine.generate([248044]) # 内部使用配置里的默认采样+ids = engine.generate([248044]) # uses the default sampling from the configtext = engine._tok.decode(ids)engine.close()-# 仅加载权重(流式专家 + prerouter + LoRA 已装配)+# weights only (streaming experts + prerouter + LoRA installed)model = AutoModel.from_pretrained("/path/to/checkpoint", name="edge0-35b")-# 仅拿配置+# config onlycfg = AutoConfig.from_pretrained("/path/to/checkpoint", name="edge0-35b")```-`AutoEngine` / `AutoModel` / `AutoConfig` 三者也可省略 `name`,从 checkpoint 的 `config.json` 的 `model_type` 或目录 basename 自动解析(见 `src/edge0/registry.py`)。+`AutoEngine` / `AutoModel` / `AutoConfig` can all omit `name` and resolve automatically from the `model_type` in the checkpoint's `config.json` or from the directory basename (see `src/edge0/registry.py`).## HTTP API-`edge0 serve` 暴露一个 OpenAI 兼容的单模型端点。引擎一次独占一个请求,生成以 FIFO 队列串行执行。+`edge0 serve` exposes a single OpenAI-compatible model endpoint. The engine serves one exclusive request at a time; generation is serialized through a FIFO queue.-| 方法 | 路径 | 说明 |+| Method | Path | Description || --- | --- | --- |-| `GET` | `/healthz` | 健康检查 |-| `GET` | `/v1/models` | 列出已加载模型 |-| `POST` | `/v1/chat/completions` | 对话补全(支持 `stream`) |-| `POST` | `/v1/completions` | 不支持,返回 400 |+| `GET` | `/healthz` | Health check |+| `GET` | `/v1/models` | List loaded models |+| `POST` | `/v1/chat/completions` | Chat completions (supports `stream`) |+| `POST` | `/v1/completions` | Not supported; returns 400 |-### 非流式对话+### Non-streaming chat```bashcurl -s http://127.0.0.1:8085/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"model": "edge0-35b",- "messages": [{"role": "user", "content": "介绍你自己"}],+ "messages": [{"role": "user", "content": "Introduce yourself"}],"temperature": 0.6,"max_tokens": 256}'```-响应字段(非流式):+Response fields (non-streaming):```json{@@ -127,25 +126,25 @@ curl -s http://127.0.0.1:8085/v1/chat/completions \}```-请求可选字段:`model`、`messages`(含 `role`/`content`,content 支持多段文本自动拼接)、`temperature`、`top_p`、`top_k`、`max_tokens`、`seed`、`stream`。+Optional request fields: `model`, `messages` (with `role`/`content`; content supports multiple text segments that are concatenated automatically), `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`, `stream`.-### 流式对话(需 Flask)+### Streaming chat (requires Flask)```bashcurl -N http://127.0.0.1:8085/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"model": "edge0-35b",- "messages": [{"role": "user", "content": "数到五"}],+ "messages": [{"role": "user", "content": "Count to five"}],"stream": true}'```-每个 token 输出一段 `data: {"object":"chat.completion.chunk", ...}` SSE 事件,结束以 `data: [DONE]` 收尾。+Each token emits one `data: {"object":"chat.completion.chunk", ...}` SSE event, and the stream ends with `data: [DONE]`.-## 配置覆盖+## Configuration overrides-`from_pretrained` 支持对任意公开字段做覆盖(未知字段会抛 `TypeError`)。+`from_pretrained` supports overriding any public field (unknown fields raise a `TypeError`).```pythonfrom edge0 import AutoEngine@@ -153,12 +152,12 @@ from edge0 import AutoEngineengine = AutoEngine.from_pretrained("/path/to/checkpoint",name="edge0-35b",- port=9090, # 覆盖默认端口 8085- target_tok_s=14.0, # 覆盖验收吞吐目标- prerouter=None, # 关闭前置路由- lora="", # 关闭 LoRA- prefill_chunk=1024, # 缩小预填分块+ port=9090, # override the default port 8085+ target_tok_s=14.0, # override the acceptance throughput target+ prerouter=None, # disable the prerouter+ lora="", # disable LoRA+ prefill_chunk=1024, # smaller prefill chunk)```-CLI 里对应的覆盖是 `--no-prerouter` / `--no-lora`(见 `src/edge0/cli.py` 的 `_engine_kwargs`)。引擎参数覆盖请直接走 `Qwen35Config.from_pretrained(model_dir, **overrides)`。+The corresponding CLI overrides are `--no-prerouter` / `--no-lora` (see `_engine_kwargs` in `src/edge0/cli.py`). To override engine parameters, call `Qwen35Config.from_pretrained(model_dir, **overrides)` directly.diff --git a/docs/moe.md b/docs/moe.mdindex 8c74172..fc88f97 100644--- a/docs/moe.md+++ b/docs/moe.md@@ -1,81 +1,86 @@-# MoE 规格与路由(MoESpec / Routing)+# MoE Spec and Routing (MoESpec / Routing)-MoE 抽象位于 `src/edge0/moe/`,两个文件:+The MoE abstraction lives in `src/edge0/moe/`, in two files:-- `spec.py` —— 路由种类(`RouterKind`)、量化规格(`QuantSpec`)、权重布局- (`WeightLayout`),以及驱动**驻留数学与流式层**的 `MoESpec`;-- `routing.py` —— 两种路由数学,与 vendored 模型 **bit 级一致**(从它们逐字抽取)。+- `spec.py` — routing kinds (`RouterKind`), the quantization spec (`QuantSpec`), the weight layout (`WeightLayout`), and the `MoESpec` that drives **resident math and streaming layers**;+- `routing.py` — the two routing math implementations, **bit-identical** to the vendored models (extracted verbatim from them).-设计核心:**每个 MoE 块由一份 `MoESpec` 描述**,流式子系统消费这份规格(布局 +-key 模板),因此**一个通用的 `StreamingSwitchGLU` 就能服务所有模型**。+Design core: **every MoE block is described by a single `MoESpec`**, and the streaming+subsystem consumes that spec (layout + key templates), so **one generic+`StreamingSwitchGLU` can serve all models**.-## 路由种类:`RouterKind`+## Routing kinds: `RouterKind`-`RouterKind` 是 `str` 枚举,edge0 支持两类路由数学家族。+`RouterKind` is a `str` enum; edge0 supports two families of routing math.-| 成员 | 值 | 语义 |+| Member | Value | Semantics || --- | --- | --- |-| `SOFTMAX_TOPK` | `"softmax_topk"` | 精确 softmax → top-k → 重新归一化(Qwen3.5-MoE 风格,`norm_topk_prob=True`) |-| `SIGMOID_GROUP` | `"sigmoid_group"` | sigmoid 分数 + 分组限 top-k + routed 缩放(DeepSeek-V3 / Bailing 风格) |+| `SOFTMAX_TOPK` | `"softmax_topk"` | exact softmax → top-k → re-normalization (Qwen3.5-MoE style, `norm_topk_prob=True`) |+| `SIGMOID_GROUP` | `"sigmoid_group"` | sigmoid scores + group-limited top-k + routed scaling (DeepSeek-V3 / Bailing style) |-两个家族对应两份模型:edge0-35b 用 `SOFTMAX_TOPK`,edge0-10b 用 `SIGMOID_GROUP`。-对应数学实现见「路由函数」一节。+The two families correspond to the two models: edge0-35b uses `SOFTMAX_TOPK`, edge0-10b+uses `SIGMOID_GROUP`. See the "Routing functions" section for the matching math+implementations.-## 量化规格:`QuantSpec`+## Quantization spec: `QuantSpec`-`QuantSpec` 是 `frozen` dataclass,描述**逐张量的默认**权重量化。+`QuantSpec` is a `frozen` dataclass describing the **per-tensor default** weight+quantization.-| 字段 | 类型 | 默认值 | 语义 |+| Field | Type | Default | Semantics || --- | --- | --- | --- |-| `bits` | `int` | `4` | 量化位宽 |-| `group_size` | `int` | `64` | 量化组大小 |-| `mode` | `str` | `"affine"` | 量化模式(仿射) |+| `bits` | `int` | `4` | quantization bit width |+| `group_size` | `int` | `64` | quantization group size |+| `mode` | `str` | `"affine"` | quantization mode (affine) |-两个生产模型都使用 `bits=4, group_size=64, mode="affine"`。流式层的量化 gather-(`quant.gather_qmm`)消费这三个字段。+Both production models use `bits=4, group_size=64, mode="affine"`. The streaming layer's+quantized gather (`quant.gather_qmm`) consumes these three fields.-## 权重布局:`WeightLayout`+## Weight layout: `WeightLayout`-`WeightLayout` 枚举描述专家投影在 checkpoint 里的存储方式。+The `WeightLayout` enum describes how expert projections are stored in the checkpoint.-| 成员 | 值 | 语义 |+| Member | Value | Semantics || --- | --- | --- |-| `SEPARATE` | `"separate"` | `gate_proj` / `up_proj` / `down_proj` 三份堆叠张量 |-| `FUSED_GATE_UP` | `"fused_gate_up"` | `gate_proj + up_proj` 融合成一份堆叠张量(out 轴行数加倍);`down_proj` 独立 |+| `SEPARATE` | `"separate"` | three stacked tensors: `gate_proj` / `up_proj` / `down_proj` |+| `FUSED_GATE_UP` | `"fused_gate_up"` | `gate_proj + up_proj` fused into a single stacked tensor (double the rows along the out axis); `down_proj` separate |-当前两个模型都用 `SEPARATE`;`FUSED_GATE_UP` 为支持融合 checkpoint 的模型预留。+Both current models use `SEPARATE`; `FUSED_GATE_UP` is reserved for models with+fused-checkpoint support.-## 规格:`MoESpec`+## Spec: `MoESpec`-`MoESpec` 是 `frozen` dataclass,描述一个模型所有 MoE 块需要的一切。+`MoESpec` is a `frozen` dataclass describing everything all MoE blocks of a model need.-| 字段 | 类型 | 默认值 | 语义 |+| Field | Type | Default | Semantics || --- | --- | --- | --- |-| `num_experts` | `int` | (必填) | 每层被路由的专家数 |-| `top_k` | `int` | (必填) | 每 token 选择的专家数(路由宽度) |-| `intermediate_size` | `int` | (必填) | 专家 FFN 隐层尺寸(**仅文档**;数学从权重张量推导尺寸) |-| `router` | `RouterKind` | `SOFTMAX_TOPK` | 路由数学家族 |-| `norm_topk_prob` | `bool` | `True` | 是否对选择权重重新归一化 |-| `routed_scaling` | `float \| None` | `None` | 施加在选择权重上的缩放系数(sigmoid-group 路由);softmax-topk 为 `None` |-| `n_group` | `int \| None` | `None` | 分组路由的组数(仅 sigmoid-group) |-| `topk_group` | `int \| None` | `None` | 存活的分组数(仅 sigmoid-group) |-| `shared_experts` | `int` | `0` | 常驻共享专家数(**从不被流式**) |-| `quant` | `QuantSpec` | `QuantSpec()` | 权重量化 |-| `layout` | `WeightLayout` | `SEPARATE` | 权重布局 |-| `key_template` | `str` | `""` | safetensors key 前缀,含 `{layer}`;`{proj}` / `{part}` 占位由消费方追加 |-| `block_path` | `str` | `""` | 从加载的模型对象到 MoE 块的点分属性路径,含 `{layer}` |-| `layer_path` | `str` | `""` | 到 **decoder 层对象**(块的宿主)的点分路径,含 `{layer}`;prerouter stager 用它读每层缓存;默认由 `block_path` 去掉末尾属性段推出 |-| `expert_row_axis` | `int` | `0` | 堆叠 `[num_experts, ...]` 张量中,一个专家是连续切片的轴(受支持布局恒为 0,仅为文档保留) |--关键差异标注:--- `intermediate_size` 仅供文档——**数学不依赖它**,尺寸一律从权重张量推导。-- `shared_experts` 的语义是「常驻、永不流式」:共享专家留在基础模型里,只有路由专家被- `StreamingSwitchGLU` 流式处理。--### Bundle 布局与 up-first 顺序(重点)--`bundle_projs` 属性返回**缓存专家 bundle 所含投影,按堆叠顺序**:+| `num_experts` | `int` | (required) | number of routed experts per layer |+| `top_k` | `int` | (required) | number of experts selected per token (routing width) |+| `intermediate_size` | `int` | (required) | expert FFN hidden size (**documentation only**; the math derives sizes from the weight tensors) |+| `router` | `RouterKind` | `SOFTMAX_TOPK` | routing math family |+| `norm_topk_prob` | `bool` | `True` | whether to re-normalize the selection weights |+| `routed_scaling` | `float \| None` | `None` | scaling factor applied to the selection weights (sigmoid-group routing); `None` for softmax-topk |+| `n_group` | `int \| None` | `None` | number of groups for grouped routing (sigmoid-group only) |+| `topk_group` | `int \| None` | `None` | number of surviving groups (sigmoid-group only) |+| `shared_experts` | `int` | `0` | number of resident shared experts (**never streamed**) |+| `quant` | `QuantSpec` | `QuantSpec()` | weight quantization |+| `layout` | `WeightLayout` | `SEPARATE` | weight layout |+| `key_template` | `str` | `""` | safetensors key prefix, containing `{layer}`; the `{proj}` / `{part}` placeholders are appended by the consumer |+| `block_path` | `str` | `""` | dotted attribute path from the loaded model object to the MoE block, containing `{layer}` |+| `layer_path` | `str` | `""` | dotted path to the **decoder layer object** (the block's host), containing `{layer}`; the prerouter stager uses it to read each layer's cache; defaults to `block_path` with its trailing attribute segment removed |+| `expert_row_axis` | `int` | `0` | in a stacked `[num_experts, ...]` tensor, the axis on which one expert is a contiguous slice (always 0 for supported layouts; kept for documentation only) |++Key differences worth noting:++- `intermediate_size` is documentation only — **the math does not depend on it**; sizes+ are always derived from the weight tensors.+- The semantics of `shared_experts` are "resident, never streamed": shared experts stay+ in the base model, and only routed experts are streamed by `StreamingSwitchGLU`.++### Bundle layout and the up-first order (important)++The `bundle_projs` property returns **the projections contained in a cached expert+bundle, in stacking order**:```pythonif self.layout is WeightLayout.FUSED_GATE_UP:@@ -83,69 +88,82 @@ if self.layout is WeightLayout.FUSED_GATE_UP:return ("up_proj", "gate_proj", "down_proj")```-**unfused 顺序是 `("up_proj", "gate_proj", "down_proj")`——up 在前。**+**The unfused order is `("up_proj", "gate_proj", "down_proj")` — up first.**-这个顺序不是随意的,它**直接对齐数学参数的顺序**。GLU 激活的数学是:+This order is not arbitrary; it **aligns directly with the order of the math's+parameters**. The GLU activation math is:```pythondef _swiglu(up, gate):return nn.silu(gate) * up```-即 `_swiglu(up, gate)`。因此权重栈把 `up` 放在 `gate` 之前,正好对应函数签名-`(up, gate)`——消费方只需按顺序喂参数,**不需要做任何重排**。`StreamingSwitchGLU`-的 bundle 构建与数学路径正是按 `self._bundle_projs` 的顺序展开的。+i.e. `_swiglu(up, gate)`. The weight stacks therefore place `up` before `gate`, exactly+matching the function signature `(up, gate)` — the consumer only needs to feed the+arguments in order, **with no reordering whatsoever**. `StreamingSwitchGLU`'s bundle+construction and math path expand in exactly the order of `self._bundle_projs`.-**fused 布局则把 gate 行叠在 up 行之上**(out 轴方向),所以+**The fused layout instead stacks gate rows above up rows** (along the out axis), so```pythonx_gate, x_up = core.split(x_gu, 2, axis=-1)```-一次 `split(x_gu, 2)` 恰好得到 `(gate, up)`。行序是「gate 在上、up 在下」。构建 fused-bundle 时,shard 里 gate/up 是两份独立张量,必须**按专家逐个交错**拼接(gate 切片在前、-up 切片在后),而非整体拼接,否则会行错位(见 `streaming/layer.py` 的 `_build`)。+one `split(x_gu, 2)` yields exactly `(gate, up)`. The row order is "gate on top, up+below". When building a fused bundle, gate/up in the shard are two separate tensors and+must be concatenated **interleaved expert by expert** (gate slice first, up slice+second), not concatenated wholesale, otherwise the rows misalign (see `_build` in+`streaming/layer.py`).-`fuse_gu` 属性是 `layout is FUSED_GATE_UP` 的快捷判断,供流式层分支使用。+The `fuse_gu` property is a shortcut check for `layout is FUSED_GATE_UP`, used by+streaming-layer branches.-### 路径解析:`keys()` / `block_of()` / `layer_of()`+### Path resolution: `keys()` / `block_of()` / `layer_of()`-`MoESpec` 用三条路径字符串把「层号」翻译成真实对象 / 张量 key。+`MoESpec` uses three path strings to translate a "layer number" into real objects /+tensor keys.-**`keys(layer, proj, part)`** —— 解析某层某投影某部分的一个 safetensors key:+**`keys(layer, proj, part)`** — resolves one safetensors key for a given layer,+projection, and part:```pythonprefix = self.key_template.format(layer=layer)return f"{prefix}.{proj}.{part}"```-例(edge0-35b):`keys(3, "gate_proj", "weight")` 得到-`language_model.model.layers.3.mlp.switch_mlp.gate_proj.weight`。+Example (edge0-35b): `keys(3, "gate_proj", "weight")` yields+`language_model.model.layers.3.mlp.switch_mlp.gate_proj.weight`.-**`block_of(model, layer)`** —— 解析 MoE 块对象。沿 `block_path` 逐段 `getattr`,-遇到纯数字段则按下标取(多数家族里 layer 是普通 list)。+**`block_of(model, layer)`** — resolves the MoE block object. It walks `block_path`+segment by segment with `getattr`, indexing into purely numeric segments instead (in+most families the layers are a plain list).-**`layer_of(model, layer)`** —— 解析 decoder 层对象(块的宿主)。若给了 `layer_path`-用它的模板;否则按约定回退:块位于 `<layer>.<mlp>.<block>`,层就是去掉末尾**两段**-属性后的路径(`split(".")[:-2]`)。prerouter stager 用 `layer_of` 读取每层缓存。+**`layer_of(model, layer)`** — resolves the decoder layer object (the block's host). If+`layer_path` is given, its template is used; otherwise the convention fallback applies:+the block lives at `<layer>.<mlp>.<block>`, so the layer is the path with the trailing+**two** attribute segments removed (`split(".")[:-2]`). The prerouter stager uses+`layer_of` to read each layer's cache.-测试(`tests/test_moe_spec.py`)用假模型验证了这些路径解析:数字段走下标、`layer_path`-缺失时从 `block_path` 正确推出层路径、`bundle_projs` 随布局切换、`QuantSpec` 默认值、-两种 `RouterKind` 的值。+The tests (`tests/test_moe_spec.py`) validate these path resolutions with a fake model:+numeric segments go through indexing, the layer path is correctly derived from+`block_path` when `layer_path` is missing, `bundle_projs` switches with the layout,+`QuantSpec` defaults hold, and both `RouterKind` values are covered.-## 路由函数:`routing.py`+## Routing functions: `routing.py`-两个路由函数共享**驻留**与**流式**两条 MoE 路径。模块 docstring 强调:+The two routing functions are shared by the **resident** and **streaming** MoE paths.+The module docstring stresses:> These functions must stay bit-identical to the vendored base> implementations (they are extracted from them verbatim); parity tests pin> this.-即它们是从 vendored 基础实现**逐字抽取**的,必须与之一致,由 parity 测试钉死。+That is, they are **extracted verbatim** from the vendored base implementations and must+stay identical to them — pinned by parity tests.### `select_from_logits(logits, top_k, norm=True)`-Softmax-topk 路由(Qwen3.5-MoE / `norm_topk_prob=True` 语义):+Softmax-topk routing (Qwen3.5-MoE / `norm_topk_prob=True` semantics):```pythongates = core.softmax(logits, axis=-1, precise=True)@@ -156,33 +174,38 @@ if norm:return inds, scores```-流程:**精确 softmax → top-k → (可选)重新归一化**。返回 `(inds [..., k], scores [..., k])`。+Flow: **exact softmax → top-k → (optional) re-normalization**. Returns+`(inds [..., k], scores [..., k])`.### `group_select_from_logits(logits, top_k, n_group, topk_group, routed_scaling, norm=True, expert_bias=None)`-Sigmoid + 分组限 top-k 路由(DeepSeek-V3 / Bailing 规则):+Sigmoid + group-limited top-k routing (DeepSeek-V3 / Bailing rules):```pythonscores = core.sigmoid(logits.astype(core.float32))select = scores + expert_bias if expert_bias is not None else scores```-- **选择分数**是 `sigmoid(logits) + expert_bias`;-- 按每组**前二**选择分数之和排序,保留 `topk_group` 个最优组,其余组分数置 `-inf`- 排除(`k_drop = n_group - topk_group`;`topk_group == n_group` 时无组被丢);-- 在存活组内按选择分数取 top-k 专家;-- **权重**是所选专家的**原始 sigmoid 分数**(不含 bias),可选归一化- (`w / (w.sum + 1e-20)`)后乘以 `routed_scaling`。--返回 `(inds [..., k], scores [..., k])`。注意「选谁」用选择分数(含 bias),「权重」用-raw sigmoid(不含 bias)——这是 DeepSeek 式路由的关键区分。--两份模型各取其一:edge0-35b 用 `select_from_logits`,edge0-10b 用-`group_select_from_logits`;prerouter stager 也复用这两个函数做跨 token 的教师路由。--## 参考--- `src/edge0/moe/spec.py` —— `MoESpec` / `QuantSpec` / `RouterKind` / `WeightLayout`-- `src/edge0/moe/routing.py` —— 两个路由函数-- `src/edge0/moe/__init__.py` —— 重新导出以上符号-- `tests/test_moe_spec.py` —— 路径解析与布局契约测试+- The **selection score** is `sigmoid(logits) + expert_bias`;+- groups are ranked by the sum of their **top-two** selection scores, the best+ `topk_group` groups are kept, and the remaining groups' scores are set to `-inf` to+ exclude them (`k_drop = n_group - topk_group`; no group is dropped when+ `topk_group == n_group`);+- within the surviving groups, the top-k experts are picked by selection score;+- the **weights** are the selected experts' **raw sigmoid scores** (bias excluded),+ optionally normalized (`w / (w.sum + 1e-20)`) and then multiplied by `routed_scaling`.++Returns `(inds [..., k], scores [..., k])`. Note that "who is selected" uses the+selection score (bias included) while the "weights" use the raw sigmoid (bias excluded) —+the key distinction of DeepSeek-style routing.++The two models each use one: edge0-35b uses `select_from_logits`, edge0-10b uses+`group_select_from_logits`; the prerouter stager also reuses these two functions for+cross-token teacher routing.++## References++- `src/edge0/moe/spec.py` — `MoESpec` / `QuantSpec` / `RouterKind` / `WeightLayout`+- `src/edge0/moe/routing.py` — the two routing functions+- `src/edge0/moe/__init__.py` — re-exports the symbols above+- `tests/test_moe_spec.py` — path resolution and layout contract testsdiff --git a/docs/prerouter.md b/docs/prerouter.mdindex 8a39e0a..cba0e2f 100644--- a/docs/prerouter.md+++ b/docs/prerouter.md@@ -1,71 +1,81 @@-# prerouter:跨 token 路由预判+# prerouter: cross-token routing prediction-## 动机+## Motivation-MoE decode 每步都需要「本层输出 → 路由 → 下层的专家选择」,而路由依赖前一层-输出——SSD 流式下,等路由结果出来再加载专家意味着每步都要等权重装载。解法:-**用一层小网络提前一个 token 预测路由**,让专家装载与生成并行,staged 槽位-永远先于需求就绪。+Every MoE decode step needs "this layer's output → routing → next layer's expert+selection", and routing depends on the previous layer's output — under SSD streaming,+waiting for the routing result before loading experts means waiting on weight loading at+every step. The fix: **a small per-layer network predicts the routing one token ahead**,+so expert loading runs in parallel with generation and staged slots are always ready+before they are needed.-## 双移位:prev-layer + prev-token+## Double shift: prev-layer + prev-token-- prerouter 头 **OWNED by 层 N**:在 token t 用层 N 的 MoE 输入(attention 后- norm 输出)预测**层 N+1** 的路由(层移位);-- 层 N+1 消费的是层 N 的头在 **token t-1** 的预测(token 移位)。+- The prerouter head **owned by layer N**: at token t it takes layer N's MoE input (the+ post-attention norm output) to predict **layer N+1**'s routing (layer shift);+- Layer N+1 consumes the prediction that layer N's head made at **token t-1** (token+ shift).-所以一个 decode step 的 staged 专家集就是 prerouter 的预测本身——**按构造零-drop**(`staged_replace` 语义)。+So a decode step's staged expert set is the prerouter's prediction itself — **zero drop+by construction** (`staged_replace` semantics).-## 头结构(`prerouter/heads.py`)+## Head structure (`prerouter/heads.py`)-特征向量:`concat[hidden, 本 token top-k one-hot, 上 token top-k one-hot]`-(`feature_topk` 决定 one-hot 来源:"executed" = 块实际路由的 top-k,qwen 训练-语义;"teacher" = 原 gate 重算的 top-k,ling v7 训练语义)。+Feature vector: `concat[hidden, current-token top-k one-hot, prev-token top-k one-hot]`+(`feature_topk` decides the one-hot source: "executed" = the top-k the block actually+routed to, qwen training semantics; "teacher" = the top-k recomputed from the original+gate, the reference deployment's training semantics).```-head: fc1 -> exact(erf) gelu -> fc2 + linear_init (与训练导出 bit 级一致)+head: fc1 -> exact(erf) gelu -> fc2 + linear_init (bit-identical to the training export)```-## 安装与接线(`prerouter/install.py`)+## Installation and wiring (`prerouter/install.py`)-`install_prerouter(model, spec, store)`:+`install_prerouter(model, spec, store)`:-- 从 safetensors 读每个 owner 的头权重(键形如 `layers.<N>.fc1.weight`);-- 替换/注入模型中的 prerouter 模块;-- **patch_call**:qwen 的 vendored MoE 块不是 prerouter-aware,安装类级- `__call__` 补丁让 decode 路由走 `pred_inds`;ling 的 `bailing_hybrid` 内置了- 消费钩子,`patch_call=False`。+- reads each owner's head weights from safetensors (keys shaped like+ `layers.<N>.fc1.weight`);+- replaces/injects the prerouter modules in the model;+- **patch_call**: qwen's vendored MoE blocks are not prerouter-aware, so a class-level+ `__call__` patch is installed to route decode through `pred_inds`; ling's+ `bailing_hybrid` has a built-in consumption hook, so `patch_call=False`.-## 跨 token 分派(`prerouter/stager.py`)+## Cross-token dispatch (`prerouter/stager.py`)-`CrossTokenStager`(qwen)/ `LingPrerouterStager`(ling)在 step 边界把预测-**提交**给下一层:先保存本轮 logits,step 后按 `start_layer`/`owners` 把每个-owner 的预测写入对应 consumer 的 staged 槽位。家族差异通过三个 hook 表达:-`_features`(特征组装)、`_select`(logits → 专家选择)、`_store`(提交)。+`CrossTokenStager` (qwen) / `LingPrerouterStager` (ling) **commit** the predictions to+the next layer at step boundaries: they first save the current step's logits, then after+the step write each owner's predictions into the corresponding consumer's staged slots+per `start_layer`/`owners`. Family differences are expressed through three hooks:+`_features` (feature assembly), `_select` (logits → expert selection), `_store`+(commit).-## 与 streaming 的配合+## Working with streaming-- `LayerOptions.staged_replace=True`:路由集 == staged 集(prerouter 模式,- 无 drop);-- 生产 profile(`staged_k4`/`staged_k8`)使用**显式索引**路由:prerouter 预测- 直接作为 staged 填充来源,`staged_replace=False` 但槽表精确映射,同样零 drop;-- `pin_bonus`:prerouter 预测的专家在 hot 驻留选择时额外加分,让「即将用到」- 的专家优先常驻。+- `LayerOptions.staged_replace=True`: the routed set == the staged set (prerouter mode,+ no drop);+- Production profiles (`staged_k4`/`staged_k8`) use **explicit-index** routing: prerouter+ predictions directly serve as the staged fill source, `staged_replace=False` but the+ slot table maps exactly — likewise zero drop;+- `pin_bonus`: experts predicted by the prerouter get an extra bonus during hot pin+ selection, so experts that are about to be used are preferentially kept resident.-## 模型档+## Model tiers| | edge0-35b | edge0-10b ||---|---|---|-| 路由家族 | SOFTMAX_TOPK(precise softmax → top-k → 归一化) | SIGMOID_GROUP(sigmoid + 组限制 top-k,n_group 8 / topk_group 4,routed_scaling 2.5) |-| start_layer | 7(首个消费层) | 1 |-| heads(owners) | 33 个(6..38) | 22 个(1..22) |-| hidden | 512(fp16) | 512(fp16) |+| Routing family | SOFTMAX_TOPK (precise softmax → top-k → normalization) | SIGMOID_GROUP (sigmoid + group-limited top-k, n_group 8 / topk_group 4, routed_scaling 2.5) |+| start_layer | 7 (first consumer layer) | 1 |+| heads (owners) | 33 (6..38) | 22 (1..22) |+| hidden | 512 (fp16) | 512 (fp16) || feature_topk | executed | executed |-| patch_call | True(qwen3_next 需补丁) | False(模型内置钩子) |-| 权重文件 | `artifacts/prerouter_edge0_35b_k4.safetensors` | `artifacts/prerouter_edge0_10b.safetensors` |+| patch_call | True (qwen3_next needs the patch) | False (built-in model hook) |+| weight file | `artifacts/prerouter_edge0_35b_k4.safetensors` | `artifacts/prerouter_edge0_10b.safetensors` |-## 测试与回归+## Tests and regression-`tests/test_streaming_math.py` 的 staged 路径在「无 prerouter」下用显式索引-模拟槽位填充,保证 staged 数学与 exact 等价;prerouter 头本身的数值由-`heads.py` 的 bit 级移植保证(源自训练实现),引擎 e2e 冒烟验证真实生成链路。+The staged path in `tests/test_streaming_math.py` simulates slot filling with explicit+indices under "no prerouter", guaranteeing the staged math is equivalent to the exact+math; the prerouter head's own numerics are guaranteed by the bit-level port in+`heads.py` (sourced from the training implementation), and the engine e2e smoke test+validates the real generation chain.diff --git a/docs/streaming.md b/docs/streaming.mdindex 7779b0a..94ea65c 100644--- a/docs/streaming.md+++ b/docs/streaming.md@@ -1,109 +1,84 @@-# SSD 流式专家层(streaming)+# SSD Streaming Expert Layers (streaming)-## 问题+## The Problem-Qwen3.5-MoE 有 40 层 × 256 专家,每层 MoE 权重约 310MB(4-bit 量化后)——全量-常驻远超 Apple Silicon 统一内存预算。edge0 的方案:**权重留在 SSD,按需 mmap-加载进 GPU,用 LRU + 预取 + 固定槽把「每步只动几十 MB」变成「每步几乎不动」**。+Qwen3.5-MoE has 40 layers × 256 experts, with each layer's MoE weights around 310MB after 4-bit quantization — keeping them all resident far exceeds the Apple Silicon unified-memory budget. edge0's approach: **keep the weights on SSD, mmap them into the GPU on demand, and use LRU + prefetch + fixed slots to turn "tens of MB moved per step" into "almost nothing moved per step"**.-核心部件(`edge0/streaming/`):+Core components (`edge0/streaming/`):-| 模块 | 职责 |+| Module | Responsibility ||---|---|-| `mmap.py` | `SafetensorsMmap`:单文件 byte-range mmap,按张量名惰性读原始字节 |-| `cache.py` | `SharedExpertCache`:跨层共享的 LRU,容量 `cache_slots` |-| `layer.py` | `StreamingSwitchGLU`:每层一个实例,持有全部执行路径与状态 |-| `options.py` | `LayerOptions`:typed 配置(部署期 env 旋钮的替代品) |+| `mmap.py` | `SafetensorsMmap`: single-file byte-range mmap, lazily reads raw bytes by tensor name |+| `cache.py` | `SharedExpertCache`: cross-layer shared LRU with capacity `cache_slots` |+| `layer.py` | `StreamingSwitchGLU`: one instance per layer, holding all execution paths and state |+| `options.py` | `LayerOptions`: typed configuration (a replacement for deployment-time env knobs) |-## 数学契约(必须与 vendored 模型 bit 级一致)+## Math Contract (must be bit-identical to the vendored model)-MoE 块数学为 `down(silu(gate(x)) * up(x))`,即-`_swiglu(up, gate) = nn.silu(gate) * up`。**bundle 顺序直接对应数学参数顺序**,-权重栈不需要任何重排:+The MoE block math is `down(silu(gate(x)) * up(x))`, i.e. `_swiglu(up, gate) = nn.silu(gate) * up`. **The bundle order maps directly onto the order of the math parameters**, so the weight stack needs no reordering:-- **separate(未融合)**:bundle 顺序为- `("up_proj", "gate_proj", "down_proj")` —— up 在前!堆栈后的 wargs 顺序与- `_swiglu(up, gate)` 签名逐位对应:`(w_u, s_u, b_u, w_g, s_g, b_g, w_d, s_d, b_d)`。-- **fused gate+up**:bundle 顺序为 `("gate_up_proj", "down_proj")`;gate_up 行内- 布局为 **gate 行在上、up 行在下**(沿输出特征轴拼接),`mx.split(x_gu, 2, -1)`- 得到 `(gate, up)`。+- **separate (unfused)**: the bundle order is `("up_proj", "gate_proj", "down_proj")` — up first! After stacking, the wargs order matches the `_swiglu(up, gate)` signature position by position: `(w_u, s_u, b_u, w_g, s_g, b_g, w_d, s_d, b_d)`.+- **fused gate+up**: the bundle order is `("gate_up_proj", "down_proj")`; the row layout of gate_up is **gate rows on top, up rows below** (concatenated along the output feature axis), and `mx.split(x_gu, 2, -1)` yields `(gate, up)`.-任何一处重排(如历史上把 bundle 写成 gate-first)都会在-`tests/test_streaming_math.py` 的参考对比中暴露(相对 L2 从 ≈0.2% 跳到 ≈100%)。+Any reordering (e.g. the historical gate-first bundle order) is exposed by the reference comparison in `tests/test_streaming_math.py` (relative L2 jumps from ≈0.2% to ≈100%).-### 量化布局+### Quantized Layout-- `switch_mlp.<proj>.weight`:打包 u32 `[E, out, in/8]`(4-bit affine,组 64);-- `scales` / `biases`:bf16 位型,按组 `[E, out, in/64]`;-- 内核:`backends.quant.gather_qmm`(MLX 量化 gather matmul),dequant 后 bf16- 内部精度(相对 L2 ≈0.24%,测试容差按此标定)。+- `switch_mlp.<proj>.weight`: packed u32 `[E, out, in/8]` (4-bit affine, group size 64);+- `scales` / `biases`: bf16 dtype, per group `[E, out, in/64]`;+- Kernel: `backends.quant.gather_qmm` (MLX quantized gather matmul), with bf16 internal precision after dequantization (relative L2 ≈0.24%; the test tolerance is calibrated against this).-## 执行路径(由快到全)+## Execution Paths (from fast to full)-### 1. exact(on-demand bundle)+### 1. exact (on-demand bundle)-路由索引去重 → `_get_bundles(unique)` 逐专家构建(从 LRU 或 mmap 读)→ 堆栈 →-`gather_qmm`。最慢但最通用,是其他路径的正确性基准。+Deduplicate the routing indices → `_get_bundles(unique)` builds per expert (read from the LRU or mmap) → stack → `gather_qmm`. The slowest but most general path; it is the correctness baseline for the other paths.-### 2. staged(固定槽双缓冲,decode 主力)+### 2. staged (fixed-slot double buffering, the decode workhorse)-- `staged_n` 个固定槽位 + 溢出零槽;-- 路由索引 → `mx.take` 槽位表,**索引不离开 GPU**(每层每步零 host 同步);-- `staged_sync`:step 边界同步填充;`asm_cache`:按专家集缓存槽表与堆栈图节点,- 重复集不重建;`incr_stack`:把 9 个 `mx.stack` 节点换成增量栈的- `put_along_axis` 行写入(staged 期间从 LRU 去重,`incr_writeback` 归还);-- 缺专家映射到溢出零槽、贡献丢弃;槽未就绪时回退 exact 路径。+- `staged_n` fixed slots plus an overflow zero slot;+- Routing indices → `mx.take` on the slot table, **indices never leave the GPU** (zero host synchronization per layer per step);+- `staged_sync`: fill synchronized at step boundaries; `asm_cache`: caches the slot table and the stacked graph nodes per expert set, so repeated sets are not rebuilt; `incr_stack`: replaces the 9 `mx.stack` nodes with incremental-stack `put_along_axis` row writes (deduplicated against the LRU while staged is active, and returned by `incr_writeback`);+- Missing experts map to the overflow zero slot and their contribution is dropped; if a slot is not ready, the path falls back to exact.-### 3. hot(LRU 常驻 top-N)+### 3. hot (LRU-resident top-N)-- 每层按衰减计数 `_hot_counts` 选 top-N(`hot_per_layer`),LRU 命中的专家以- 常驻堆栈形式存在(`load_hot_layer` / `materialize_hot`,滑动窗口- `hot_window` 控制同时驻留的层数);-- 命中走堆栈 gather,未命中走 exact 并 scatter-add。+- Each layer picks its top-N (`hot_per_layer`) from the decayed counts `_hot_counts`; experts that hit in the LRU live as a resident stack (`load_hot_layer` / `materialize_hot`, with the sliding window `hot_window` controlling how many layers stay resident at once);+- Hits take the stacked gather; misses take exact with a scatter-add.-### 4. full-layer(E3b 整层装载,prefill 主力)+### 4. full-layer (whole-layer loading for E3b, the prefill workhorse)-- `load_full_layer()`:整层 9 个张量直接装入(checkpoint 本来就是逐层堆叠的- 单张量,mmap 位型转换 ≈9ms/层,CPU 装载隐藏在上一层的 GPU 执行下);-- `_gather_sort` 把 batch 并入 token 维 → 排序 gather → `_scatter_unsort`- 反排序并恢复 batch 维;-- `prefill_full_layers` 只对前导层整层装载(qwen 为 12 层),其余层走 hot/exact;-- 用后 `clear_full_layer()` 释放 GPU 副本,页缓存承担热数据。+- `load_full_layer()`: loads the layer's 9 tensors directly (the checkpoint already stores each layer as a single stacked tensor; the mmap dtype conversion costs ≈9ms per layer, and the CPU load is hidden under the previous layer's GPU execution);+- `_gather_sort` folds the batch into the token dimension → sorted gather → `_scatter_unsort` un-sorts and restores the batch dimension;+- `prefill_full_layers` loads whole layers only for the leading layers (12 for Qwen); the remaining layers go through hot/exact;+- After use, `clear_full_layer()` frees the GPU copy, and the page cache carries the hot data.-## 排序与编译+## Sorting and Compilation-`use_compile` 时,staged/exact 路径包进 `mx.compile`:+With `use_compile`, the staged/exact paths are wrapped in `mx.compile`:-- 集合规模 ≥64 时用排序变体 `_moe_math_sorted`(`_gather_sort` 预处理 +- 排序 gather + `_scatter_unsort` 还原);-- 否则用直接 gather 变体 `_moe_math`;-- 两者数学等价,测试对每条路径都做了 exact 对比。+- set sizes ≥64 use the sorted variant `_moe_math_sorted` (`_gather_sort` preprocessing + sorted gather + `_scatter_unsort` restore);+- otherwise the direct gather variant `_moe_math` is used;+- the two are mathematically equivalent, and the tests compare every path against exact.-## 关键选项(`LayerOptions`)+## Key Options (`LayerOptions`)-| 字段 | 含义 | 默认 |+| Field | Meaning | Default ||---|---|---|-| `staged` / `staged_n` / `staged_trigger` | 固定槽 decode 开关/槽数/触发 top-k | False / 8 / 8 |-| `staged_replace` | staged 集**取代**路由(配合 prerouter,零 drop) | False |-| `staged_sync` / `asm_cache` / `incr_stack` / `incr_writeback` | 填充同步 / 图节点缓存 / 增量栈 / 归还 LRU | True / True / False / False |-| `hot_per_layer` / `hot_update_interval` / `hot_decay` / `pin_bonus` | 热专家驻留数与刷新 | 0 / 4 / 0.75 / 2.0 |-| `cache_slots` / `prefetch_cap` | LRU 容量 / 预取缓冲 | 64 / 48 |-| `load_threads` / `prefetch_threads` | 构建 / 预取线程数 | 8 / 4 |-| `full_layer_prefill` / `prefill_full_layers` | 整层装载 prefill / 前导层数 | False / 0 |-| `prefill_hot` | prefill 期 hot 栈规模 | 0 |-| `use_compile` / `top_k` | 编译包装 / 路由 top-k 覆盖 | True / None |--预设:`staged_k4()`(edge0-35b:staged 4 槽 + hot 32 + 整层 prefill 12 层)、-`staged_k8()`(edge0-10b:staged 8 槽,无 hot 驻留)。两档共用-`cache_slots=64`。--## 为什么「整层」加载也快--checkpoint 的 MoE 权重本来就是逐层堆叠的单张量(`[256, 512, 256]` 这类),-整层装载是 9 次直接读 + 位型 view,不是 256×9 次逐专家构建。CPU 装载与 GPU-执行通过 `before_layer_cb` + `async_eval_per_layer` 流水线重叠。--> 注意 fused gate_up 的整层装载必须**逐专家交错** gate/up 行(与 `_build`-> 的逐专家拼接一致);把两个 `[E, ...]` 张量整体拼接再 reshape 会产生行错位-> (早期版本的真实 bug,已由 `test_full_layer_prefill_matches_exact[fused]`-> 钉死)。+| `staged` / `staged_n` / `staged_trigger` | Fixed-slot decode toggle / slot count / trigger top-k | False / 8 / 8 |+| `staged_replace` | The staged set **replaces** routing (paired with the prerouter; zero drops) | False |+| `staged_sync` / `asm_cache` / `incr_stack` / `incr_writeback` | Fill synchronization / graph-node cache / incremental stack / writeback to the LRU | True / True / False / False |+| `hot_per_layer` / `hot_update_interval` / `hot_decay` / `pin_bonus` | Hot-expert residency count and refresh | 0 / 4 / 0.75 / 2.0 |+| `cache_slots` / `prefetch_cap` | LRU capacity / prefetch buffer | 64 / 48 |+| `load_threads` / `prefetch_threads` | Build / prefetch thread counts | 8 / 4 |+| `full_layer_prefill` / `prefill_full_layers` | Whole-layer prefill loading / number of leading layers | False / 0 |+| `prefill_hot` | hot stack size during prefill | 0 |+| `use_compile` / `top_k` | compile wrapping / routing top-k override | True / None |++Presets: `staged_k4()` (edge0-35b: staged with 4 slots + 32 hot + whole-layer prefill over 12 layers), `staged_k8()` (edge0-10b: staged with 8 slots, no hot residency). Both tiers share `cache_slots=64`.++## Why Whole-Layer Loading Is Also Fast++The MoE weights in the checkpoint are already single per-layer stacked tensors (such as `[256, 512, 256]`), so a whole-layer load is 9 direct reads + a dtype view, not 256×9 per-expert builds. CPU loading and GPU execution are overlapped through the `before_layer_cb` + `async_eval_per_layer` pipeline.++> Note that a fused gate_up whole-layer load must **interleave the gate/up rows per expert** (consistent with the per-expert concatenation in `_build`); concatenating the two `[E, ...]` tensors wholesale and then reshaping produces misaligned rows (a real bug in an early version, now pinned down by `test_full_layer_prefill_matches_exact[fused]`).