Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

fix(prerouter): zero-drop layer scoping, working --no-prerouter, 8b sync

linyubupab57d83ae2b9e
diff --git a/docs/prerouter-decode-fix-2026-09-14.md b/docs/prerouter-decode-fix-2026-09-14.md
new file mode 100644
index 0000000..713df05
--- /dev/null
+++ b/docs/prerouter-decode-fix-2026-09-14.md
@@ -0,0 +1,418 @@
+# prerouter decode 修复记录(2026-09-14)
+> 说明:本文引用的临时验证脚本(`scripts/verify_*.py`、`scripts/ab_*.py`、`scripts/bench_*_ab.py`、`scripts/diag_*.py`、`scripts/quality_*.py`)是一次性的
+> 测量脚手架,已在收尾时从仓库清理;复现口径见正文(配对方式、段长、轮数、环境噪声说明)。
+
+
+范围:`edge0-35b`(qwen)。8b(ling)的同类层级问题未改,见文末。
+§3.6 的两个执行开销修复落在**共享**的 `prerouter/stager.py` 上,两个 tier 同时受益。
+
+---
+
+## 1. 修前实测到的三个问题
+
+用真实引擎跑(2 请求 × 8 decode token + 32 token 对照),两组独立信号一致
+(脚本自己比"routed vs staged 集合"和引擎内建的 `staged_dropped` 计数器)。
+
+### P1 跨请求状态污染
+- 同进程第 2 个请求的 prefill,`stage_all` 提交了 33 个消费者层,且 owner 20 的
+ `prerouter_m_in` 指纹与**上一个请求最后一个 decode step 逐位相同**
+ (`1398.39453125`)→ 新请求第 1 个 token 的路由,是用上一个请求的隐状态算出来的预测。
+- 根因:`StreamingSwitchGLU.reset()` 文档写着 "Per-request reset … so the next request
+ starts clean",但**全仓库零调用点**;`block.prerouter_m_in / prerouter_oh` 只在单 token
+ 前向里写、从不清理。(参考部署 `~/Documents/qwen35-v7-deploy/streaming_experts_qwen35.py`
+ 里连这个方法都没有,是 edge0 自己加的、漏接的线。)
+
+### P2 每个请求第 1 个 token:消费者层 4/4 补零
+- 首步没有预测可用(head 特征只在单 token 前向抓取),路由按训练语义回落 gate
+ (pos-0 fallback),但 `_prefill_end` 仍用**最后一个 prefill token 的实际 top-k**
+ 填槽位,两边几乎不重合(实测 L7:路由 `(96,132,166,182)` vs 槽位 `(14,117,184,229)`,
+ 交集 0)→ 32 个消费者层各丢 4 个专家。
+- 缺失专家在 staged 路径下映射到溢出零行(`streaming/layer.py:597`
+ `slot_of_list = [n] * num_experts` + `_build_asm` 用 `_zero_slot` 补行):
+ **静默补零,不报错、不补算**。
+
+### P3 层 0–6 / 39:每个 token 持续丢专家
+- 这 8 层没有 head 供给(覆盖是 6→7 … 38→39),每步都用 gate 路由,却仍被
+ `_step_pre` 用"上一个 token 的实际集合"填槽位。
+- 实测每步丢 3.0–3.75 / 4(L0 在 16 次调用里累计 60/64),8 层合计稳态
+ **25.75 / 步**,占全模型 160 个 routed 槽位的 **16%**。
+- 参考部署原话即为此设计:"Current router as the predictor … **No correctness handling**"
+ —— 这是既定近似(省掉这些层专家的加载换速度),不是失误。
+
+### 影响的量化(修前)
+- 每步 `staged_dropped` 稳态约 25.75(16%)。
+- 旧路径 vs 全精确(同 prompt、同 routing、只换加载路径):前 16 个 token 相同,
+ **第 17 个 token 起分叉,64 个里 48 个不同**,`max|Δlogit|` 从 2.5 涨到 15.8。
+- 两段文本都通顺 → 结论只能是"不是同一条轨迹",**不能**说"质量更差"。
+
+---
+
+## 2. 改动清单
+
+| 文件 | 改动 |
+|---|---|
+| `src/edge0/engine/qwen.py` | `_reset_state`:清 per-block 特征捕获 + 每层 `reset()`(P1);`_prefill_end`:有 prerouter 时改调 `prefetch_from_prefill()`(P2);`load_installed`:非消费者层关闭 `_staged_mode`(P3);`_step_pre`/`_step_post`:这些层不再填槽、不再做无效 prefetch、跳过 `sync_actuals`;修正 `_forward` 里关于"首步已消费真实预测"的错误注释 |
+| `src/edge0/prerouter/stager.py` | 报告 §6 的两个执行开销修复:头权重一次性 stack + 每步 3 次 einsum(`head_batch`,可关);`stream_layers` 为空时跳过无人使用的 `core.eval` + `tolist`。另修正 `records` 里 `zip(metas, metas)` 把 consumer 记成 owner 的笔误 |
+| `src/edge0/streaming/layer.py` | 新增 `StreamingSwitchGLU.prefetch_from_prefill()`(只预热缓存,不填槽位)。未修改任何已有方法 |
+| `src/edge0/models/base.py` | 新增 `ModelConfig.history_slots: bool = False` |
+| `src/edge0/cli.py` | `--history-slots`(demo / chat / serve)+ `_engine_kwargs` 映射 |
+| `docs/prerouter.md` | "zero drop" 的适用范围写准;补首步、非消费者层、per-request reset 说明 |
+| `docs/models/edge0-35b.md` | staged 只覆盖 L7–L38;补 `history_slots` / `--history-slots` |
+| `tests/test_registry.py` | 断言 `history_slots` 默认 False 且可覆盖 |
+
+设计原则:**只有"路由来自预测"的层用 staged 槽位**。它的路由与槽位集合严格相同(零丢);
+其余层走精确路径。
+
+---
+
+## 3. 验证结果(真实引擎,每配置一个进程、一个引擎)
+
+### 3.1 prerouter 仍在被使用(新默认)
+插桩读 `pg_state.pred_inds[li]` 与该层实际用的槽位 `_staged_at_use`:
+
+| 步骤 | 有预测的消费者层 | 用了槽位的层 | 用的槽位 == 预测 |
+|---|---|---|---|
+| step 1 | 0/32 | 0 | 0(按设计回落 gate,走 exact) |
+| step 2 | 32/32 | 32 | **32/32** |
+| step 3 | 32/32 | 32 | **32/32** |
+| step 8 | 32/32 | 32 | **32/32** |
+
+预测与**原生 gate 自己的 top-4** 只在 2.2–2.6/4 上重合(集合完全相等:step2 5/32、
+step3 2/32、step8 0/32)→ 路由确实由 head 改写,不是复读 gate。
+
+### 3.2 与全精确路径等价
+同 prompt、32 步 greedy:`fixed`(新默认)与 `staged=False`(全精确)
+**token 完全相同、每步 logits 逐位相同**(`max|Δlogit| = 0`)。
+
+### 3.3 退路开关
+`--history-slots`(= 旧行为)逐位可复现:两次跑 token 序列完全一致,第 17 步起与精确
+路径分叉(16/32 不同),`staged_dropped` = 806(32 步)。
+
+### 3.4 速度:同进程配对测量(V1/V2/V3 部分可信;V0 对比已撤回,见下)
+
+方法按 `docs/experiments/prerouter-decode-speed-2026-09-14.md` §2/§9.3:本机跨进程单轮 A/B
+的组间差异可达 ±15%、甚至反号,所以用**同进程、交替、配对差值**。本表:一个引擎、3 臂
+运行时切换(只改 `_staged_mode` 与 stager 的 `stream_layers`)、每臂先跑 12 步预热再计时
+48 步、4 轮交替顺序、取中位数;`loads`/`dropped` 取层内计数器增量。
+
+**补充教训(本轮新增)**:同进程配对也不总够 —— 当两个臂**路由到不同的专家集合**、而
+LRU/页缓存是共享的时候,臂之间会互相预热,"谁先跑谁吃亏",符号可以跨进程翻转(见下)。
+
+| 臂 | 含义 | 中位 ms/step | tok/s | loads/step | dropped |
+|---|---|---|---|---|---|
+| V0 | 无头(gate 路由 + 精确加载) | 56.18 | 17.82 | 160 | 0 |
+| V1 | 有头不收割(预测路由 + 精确加载) | **49.13** | **20.36** | 160 | 0 |
+| V2 | 有头 + 收割(= 新默认) | 50.57 | 19.78 | **32** | 0 |
+
+配对差值(本轮):`V1 − V0 = −7.05 ms`、`V2 − V0 = −5.61 ms`、`V2 − V1 = +1.44 ms`。
+
+⚠️ **`V0` vs `V1`(原生路由 vs 预测路由)这个对比不可用,已撤回。** 复测三次,同脚本、
+同进程内配对,符号**翻转**:
+
+| 第几次 | V0 ms/step | V1 ms/step | V2 ms/step | 结论 |
+|---|---|---|---|---|
+| 第 1 次 | 56.18(各轮 78.0/57.9/54.4/51.9) | 49.13 | 50.57 | V1 更快 |
+| 第 2 次 | 68.61(62.4/74.1/68.6) | 49.54 | 51.66 | V1 更快 |
+| 第 3 次(含 V3 异步臂) | 45.86(46.4/41.5/45.3/60.5) | 51.11 | 50.87 | **V0 更快** |
+
+`V0` 在 45.9–68.6 ms 之间摆动,`V1` 稳定在 49.1–51.1 ms。原因:两臂**专家集合不同**
+(gate vs 预测路由),而共享 LRU/页缓存被**别的臂预热过**——预测集被 V2 预热,gate 集是冷的,
+于是"谁先跑谁吃亏",符号随机器状态翻转。这就是报告 §2 记的 ±15% / 符号翻转那一类陷阱。
+**故不再声称 prerouter 在 35b 上快 14%;同进程共享缓存的臂切换量不了这个量。**
+
+站得住的是**计数器级结论**(不依赖计时噪声):
+
+| 臂 | loads/step | load_wall | stage_wall | stage_wait | 合计装载成本 |
+|---|---|---|---|---|---|
+| V1 有头不收割 | 160 | 9.8 ms | 0 | 0 | **9.8 ms** |
+| V2 收割(`staged_sync=True`,= 默认) | 32 | 4.9 ms | 5.0 ms | 0 | **9.9 ms** |
+| V3 收割(异步 `staged_sync=False`) | 32 | 5.0 ms | 4.9 ms | 0 | **9.9 ms** |
+
+- 收割把**同一批 build 从"前向中"搬到"步边界"**:总装载时间一步没少(9.8 → 9.9 ms),
+ `stage_wait=0`(消费者从未等待)。计时上 `V2−V1 = +1.44 ms`、`V3−V1 = +0.94 ms`
+ (同向小亏)→ **收割在 35b/本机是搬运,不是节省**;同步/异步两档无差别。
+- 结构性原因:填槽由 `stage_all` 在**前向结束时**发起,消费者在**下一个 token 一开头**
+ 消费,中间没有可重叠的算力。要真正藏住装载,必须让填槽在**同一个 token 的前向期间**
+ 发起(报告 §10.2 的 Fix 3:用 `after_layer_cb` 交错头的 einsum)。
+- 8b 报告测得收割 +2.0 ms/步(184→56 loads/步),与本机 35b 相反 → 两端不可混用。
+
+### 3.4a 已废弃:跨进程单趟数字(保留备查)
+
+每配置一个进程、单趟、warmup 后计时 64 步:新默认 18.79 tok/s / 2.80 GiB / dropped 0;
+旧行为(`--history-slots`)13.60 / 2.90 GiB / 806;全精确(`staged=False`)18.39 /
+2.04 GiB / 0。跨进程单轮,按 §3.4 的方法学不可信,只用于"修前 → 修后"的粗判。
+
+### 3.5 `--no-prerouter` 为什么"看起来更快"(当时的误读已修正)
+
+那时测到 OFF 20.59 vs ON 15.41 tok/s(+34%),一度读成"prerouter 拖慢"。实际是:
+`--no-prerouter` 在 35b 上**不是**"无头 + 精确加载",而是"无头 + 全部 40 层 history 槽位",
+144 步里丢 15131 个槽位(≈105/步,占 160 个 routed 槽位的 66%)——**丢掉的专家既不加载
+也不计算**,所以它天然少干约 2/3 的 MoE 活。这不是"省",是缺。
+
+同工对比:这个量**本机量不了**(见 §3.4 的撤回)。本机能确定的只有:OFF 的 20.59 tok/s
+是"66% 补零"换来的,不能当"原生路由 + 精确计算"的基线;公平基线是
+`prerouter=None + staged=False`(跨进程单趟 18.70 tok/s,同口径粗判)。
+`+84%` 出自 `paper/make_figs.py` 所引 `qwen35-v7-deploy/K8_K4_K2_TEST.md` §4.7/4.8
+(`6.8 → 12.5 tok/s`)——那是**装载主导**的部署机;本机同模型 18–22 tok/s,装载只占
+≈10 ms/50 ms,regime 不同,数字不可互推。
+
+### 3.6 报告 §6 的两个执行开销修复(已落到共享 stager)
+
+改动都在 `src/edge0/prerouter/stager.py::PrerouterStager`,两个 tier 共用:
+- **fix 1 批量头**:owner 头形状一致 → 权重一次性 stack,每步 3 次 einsum,取代
+ "每头 3 个小算子"的循环(33 头 ≈ 99 个小算子/步 → 3 个)。`head_batch=False`
+ 或 `EDGE0_HEAD_BATCH=0` 回到循环,便于 A/B。
+- **fix 2 免无谓同步**:`stream_layers` 为空(`staged=False`)时,`_select` 之后不再做
+ `core.eval` + `tolist` —— 那份 host 侧专家 ID 没人消费。路由缓存照写(惰性数组即可),
+ 所以 `staged=False` 与 `--no-prerouter` 都受益。
+
+验证(35b 真实引擎):
+- 同一批特征上批量头 vs 逐头循环:`max|Δlogit| = 0.000e+00`,路由集合一致。
+- 同进程、**两臂都先预热**的 greedy-80:`head_batch=True` 与 `False` **token 完全一致**。
+- fix 2 回归:`staged=False` 下 greedy-32 与改动前记录**逐 token 相同**,`dropped=0`。
+
+⚠️ "先预热"不是形式主义:本机**进程内第一跑(冷缓存)与之后所有跑是两条轨迹**。
+同一配置连跑 6 次:第 1 跑与第 2–6 跑在第 36 步分叉(1 个消费者层路由不同),
+第 2–6 跑彼此逐位一致;先插一次预热跑后 6/6 完全一致。该现象在 default /
+`head_batch=False` / `staged=False` / `history-slots` 四种配置下**表现完全一样**
+(legacy 在第 22 步分叉)→ **与本次修复无关,是既有的冷/热启动差异**。
+后果:拿"第一跑"和"reset 后的跑"直接比 token 会得到假分叉;同进程 greedy 对照
+必须两臂都先预热。跨进程对照则是"冷 vs 冷",反而一致。
+
+---
+
+### 3.7 收益到底在哪:档位旋钮隔离(本轮结论,取代 §3.4 的速度论断)
+
+对照部署参考 `~/Documents/qwen35-v7-deploy/engine_qwen.py:97-99`、
+`streaming_experts_qwen35.py:292-297,486`——**跑出 +84% 的那台机器用的是**:
+`cache_slots=900`、`hot_per_layer=8`、`hot_update_interval=32`、
+`staged_sync` 默认 **False**(`QWEN_STAGED_SYNC=0`);而 edge0 的 35b 档
+`staged_k4` 是 `cache_slots=64`、`hot_per_layer=0`、`hot_update_interval=4`、
+`staged_sync=True`。
+
+隔离实验(一个引擎一进程、先插预热跑、每轮位置轮转、各 4 轮;除最后一臂外全部
+prerouter ON + 收割):
+
+| 臂 | 配置差异 | 中位 ms/step | tok/s | loads/步 | load_ms | stage_ms | 峰值 |
+|---|---|---|---|---|---|---|---|
+| `sync64h0` | = 今天的默认 | 52.66 | 18.99 | 32 | 5.6 | 6.2 | 2.91 GiB |
+| `async64h0` | `staged_sync=False` | 60.17 | 16.62 | 32 | 5.7 | 6.4 | 2.91 |
+| `async900h0` | 再加 `cache_slots=900` | **38.98** | **25.66** | 12 | 1.8 | 3.4 | 4.05 |
+| `async900h8` | 再加 `hot_per_layer=8`、interval 32 | 39.02 | 25.63 | 11 | 1.8 | 3.2 | 4.56 |
+| `native900h8` | 部署档 + **无 prerouter** | 49.53(各轮 119.4/62.4/36.6/36.4) | 20.19 | 39 | 6.1 | 0 | 3.55 |
+
+1. **真正的大头是 `cache_slots` 64 → 900**:52.66 → 38.98 ms/step(**+35% tok/s**),
+ 现场装载从 140 次/步掉到 12 次/步。
+2. `staged_sync=False` **单独无感**(60.17,甚至更慢);`hot_per_layer=8` 在 900 之上
+ **再加无收益**(39.02 vs 38.98),只多 0.5 GiB。`predicted_experts`(预测喂钉选)
+ 在 edge0 **和部署里都没被赋值**(`layer.py:138` 只声明、865-868 只读),是同一处
+ 从未接线的能力,不是移植漏掉的。
+3. **prerouter 的中位速度在 warm 稳态下 ≈ 0**:同档 `async900h8` 39.0 vs
+ `native900h8` 36.4–36.6(后两轮,页缓存已热)。§3.4 里"部署档下 +83%"
+ (pre_pin 22.4 vs nat_pin 12.2 tok/s)**作废**:那是 native 臂踩到冷轮
+ (81.9/83.9 ms)的假象,本轮 4 轮轮转复测不成立。
+4. prerouter 真正体现的是**稳定性**:无 prerouter 的臂 4 轮里 119.4 → 62.4 → 36.6 →
+ 36.4 ms(冷/热差 3 倍以上),有 prerouter 的臂始终 37.8–42.3 ms。它把"装载暴露时间"
+ 从不可控变可控——在装载本身是瓶颈且不可预测的环境(冷盘 / 网络盘 / 并发挤缓存)才
+ 转化为速度,那正是部署机 6.8→12.5 tok/s 所处的区间;本机跑热后进入"装载不是瓶颈"
+ 的区间,收益自然趋 0。
+
+**本机可下的结论只有三条**:(a) 功能正确(§3.1–3.3、§3.6);(b) 收割/同步异步在本机是
+搬运不是节省(§3.4 计数器);(c) 想在本机拿到真实提速,先改档位 —— `cache_slots`
+64 → 900(+1.1 GiB)实测 +35%,与 prerouter 无关。"prerouter 快 x%" 只能表述为
+装载主导环境下的收益,需慢存储复测(报告 §10.3)。
+
+---
+
+## 4. 没有碰的部分
+
+L7–L38 的 `预测 → 槽位 → 路由` 链路、`stage_all`/`state.swap`、head 数学
+(`prerouter/heads.py`)、prefill 的 E3b / 热路径、共享 LRU 与 hot pins、LoRA、量化、
+serve/chat 路径、以及 ling 的 `BailingSparseMoE` 从 `pg_cache` 自选路由那条链。
+`streaming/layer.py` 只有新增方法,没有改已有方法。
+
+冒烟:`edge0 chat` 正常出文;`--no-prerouter` 正常;
+`pytest tests/` 59 passed(唯一失败 `test_repo_hygiene::test_no_hardcoded_local_paths`
+是仓库里既有的硬编码路径,命中清单不含本次修改的文件)。
+
+---
+
+## 5. ling(8b)落地(已完成)
+
+§5 原列的三个待办项已全部处理,对应改动:
+
+| 文件 | 改动 |
+| --- | --- |
+| `src/edge0/engine/ling.py` | `load_installed`:消费层作用域(`li < start_layer+1` 的层 `_staged_mode=False`,`history_slots=True` 时不作用);`_step_pre`/`_step_post`:非消费层在作用域默认下走精确路径、不做历史暂存也不做无谓的 `sync_actuals`;`_prefill_end`:有 prerouter 时 prefill 尾部 `stage_all` 供首步消费(8b 在 prefill 期就抓得到特征,不需要 35b 那样的 router 回退),只有 legacy `history_slots` 才 `stage_from_prefill` 且只给非消费层;`_reset_state`:清 `layer.m_in_cache` / `block.last_topk` / `block.prev_topk_oh` 并对所有流式层调 `reset()` |
+| `src/edge0/prerouter/stager.py` | 批量头的 dtype 从 `_head_of(...)._dtype` 改为从堆叠的 `w1.dtype` 取(ling 的头是 `BailingPrerouter`,按 `fc1.weight.dtype` 定精度,没有 `_dtype` 属性 → 修前直接 `AttributeError`) |
+| `src/edge0/streaming/options.py` | `prod_k8()` 默认改为 `staged=True`(分层零丢弃的收割路径) |
+| `tests/test_registry.py` | 旧断言 `staged is (name == "edge0-35b")` 改为两档都 `staged=True` + `history_slots is False` |
+| `docs/models/edge0-8b.md`、`docs/prerouter.md` | staged 说明改为"仅消费层 L8–23";补首步分档语义、per-request reset 字段名、`--history-slots` |
+
+### 5.1 验证(每配置一个进程、一个引擎,脚本 `scripts/verify_ling_fix.py`)
+
+> 本表是 `cache_slots=64`(当时默认)下测的;缓存放大后的数字见 §5.3/§5.4。
+
+| 配置 | tok/s | 峰值内存 | dropped | loads/步 | 与 exact 逐 token 一致 |
+| --- | --- | --- | --- | --- | --- |
+| **default(新)** | **27.47** | 1.41 GiB | **0** | 5656(101 步 → 56/步) | **是** |
+| history(legacy 全层历史暂存) | 34.58 | 1.56 GiB | 1589 | 0 | 否 |
+| exact(`staged=False`,parity 基准) | 26.17 | 0.86 GiB | 0 | 18584(101 步 → 184/步) | — |
+| nopg(`prerouter=None`) | 31.50 | 1.46 GiB | 10481 | 0 | 否 |
+
+- **丢弃率**:`scripts/diag_staged_drops.py` → consumed=62976,`dropped=0`(0.00%);L1–7 每步 8 次精确 load,L8–23 全部由预测供给。
+- **质量**:`scripts/quality_ab_staged.py` 4/4 prompt `identical=True, first_divergence=None`;greedy 32 步 token 序列与 exact 完全相同。
+- **P1 跨请求污染已消**:同进程先跑请求 1 再跑请求 2,与全新进程只跑请求 2 相比,token 序列与首步 logits 完全一致(`p2_first=592203.875` 两边相同)。
+- **速度(配对,`scripts/verify_pair_final.py`,SEG=100×10 轮)**:PG 对 gate 路由精确基线 **10/10 轮全胜,中位 −1.97 ms/步、均值 −3.15 ms/步(≈+5%)**。
+- **内存**:默认路径峰值 1.41 GiB(exact 0.86 GiB;槽位 buffer + 头权重占 ≈0.55 GiB),24 GB 机器上无压力。
+
+### 5.2 与报告 §7 的 −5.11 ms/步 的关系(口径修正)
+
+报告 §6/§7 那个 −5.11 ms 的基线是"**无头 + gate 路由 + 精确 load**",因此它同时包含了
+**路由来源差异**(预测路由 vs gate 路由)和**收割差异**。做成同路由(都跑头、都由
+prerouter 供路由、都零丢弃)的干净配对后,收割本身只有 −0.19 ms/步(≈打平),
+而"有 prerouter 供路由 + 收割" 对 "关掉 prerouter 用 gate + 精确" 是 −1.97 ms/步(中位)。
+即:**prerouter 的收益主要在路由来源,槽位收割在本机(热缓存、load 延迟低)只值约 0–5%**。
+两处数字都保留,口径已注明。
+
+### 5.3 真正的瓶颈:全局专家 LRU 只有 64 槽(同日追加,收益比收割大一个量级)
+
+`install.py:54` 给全部 23 个 MoE 层**共用一个** `SharedExpertCache(cache_slots)`,键是
+`(layer, expert)`。`cache_slots=64` ÷ 23 层 = **每层 2.8 个槽**,而一个 decode step 要碰
+**184 个不同键**(23 层 × top-8)——比单 token 的工作集还小,复用窗口短于一个 token,
+命中率结构性为 0(实测 `hits=0`,每步重建 184 个 1.27 MiB 的专家包,52 µs/个)。
+
+同进程配对(`scripts/verify_cache_pair.py`,生产路径、无 wrapper、3 轮交替、
+greedy 输出三种容量下逐位相同、drops=0):
+
+| cache_slots | ms/步 | tok/s | 命中/步 | 精确 load | stage 重建 | load_wall | stage_wall | 峰值内存 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 64(原) | 40.02 | 24.99 | 0.0 | 56.0 | 56.0 | 5.21 ms | 3.05 ms | 1.39 GiB |
+| **1024(新)** | **31.44** | **31.81** | 43.9 | 12.1 | 18.4 | 1.22 ms | 1.56 ms | 2.39 GiB |
+| 2048 | 33.81 | 29.58 | 49.8 | 6.2 | 9.4 | 1.34 ms | 1.98 ms | 3.28 GiB |
+
+**1024 对 64:−8.58 ms/步(−21.4%),3/3 轮全胜。** 2048 没有更多收益(复用窗口饱和),
+所以 `prod_k8()` 落在 `cache_slots=1024`(≈44 槽/层、≈1.3 GiB 专家包、短上下文峰值
+2.6 GiB,`peak_active_mem_mb` 1400 → 2700)。收益拆开看:L1–7(无预测、纯靠 LRU)
+每步 56 次重建 → 12 次,约 4 ms;消费层那 44% 的"新预测集"重建 → 命中,约 1.5 ms。
+
+**为什么预测到的专家也没命中**:每层下一个 token 的预测集只有约 **56%** 落在上一步那 8 个
+槽位里(`stage_delta_hits`),剩下 44% 必须另找来源;而全局 64 槽连一个 token 都放不下,
+所以这部分只能重建。也就是说"每层缓存 8 个"只够覆盖一步之内,跨 token 复用必须靠更大的 LRU。
+
+prefill 未被牺牲:同进程配对(`scripts/verify_prefill_cache.py`,3101-token 提示、
+3 轮交替)warm prefill 1102(64 槽)vs 1132 tok/s(1024 槽)= **+2.7%**,同上下文解码
+22.6 → 26.7 tok/s(+18.3%)。README 里 8b 行按同口径(`BENCH_LONG=1`)重测为
+解码 27.7–30.7 tok/s、峰值 2.4 GiB(短上下文)。
+
+### 5.4 缓存修好之后:收割不再值钱
+
+四臂配对(`scripts/verify_four_arm.py`,`cache_slots=1024`,SEG=100 × 8 轮交替,
+四臂全部零丢弃):
+
+| 臂 | ms/步 | tok/s | loads/步 | 命中/步 | load_wall | stage_wall | 对 OFF |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| OFF(无 prerouter,gate 路由 + 精确装载) | **28.40** | **35.22** | 36.0 | 148 | 2.44 ms | 0 | — |
+| PGE(prerouter 路由 + 精确装载) | 29.78 | 33.58 | 26.6 | 157 | 1.86 ms | 0 | +1.38 ms(1/8 胜) |
+| SYNC(prerouter + 槽位,默认) | 30.07 | 33.26 | 11.1 | 45 | 0.81 ms | 0.97 ms | +1.67 ms(**0/8 胜**) |
+| ASYNC(`staged_sync=False`) | 30.41 | 32.88 | 10.6 | 45 | 0.76 ms | 1.01 ms | +2.02 ms(0/8 胜) |
+
+结论:缓存能命中以后,整条精确路径的装载只剩 2.44 ms/步,而跑一趟头(批量 einsum +
+`core.eval` + host sync)≈1.4 ms/步——没有足够多的装载可省了。**prerouter 的剩余价值在
+"路由来源"(预测驱动选择,即发布行为),不在槽位收割**;`--no-prerouter` 在本机快约 5%,
+但改变路由/输出,属于模型语义选择,不由我改默认。
+
+### 5.5 未做
+
+- 8b 的**质量评测**(题库/人工评分)仍未做:现有的"质量"结论是同 prompt 下与精确路径
+ 逐位一致(token-level parity),不是 benchmark 分数。
+- `pytest tests/` 59 passed;唯一失败 `test_repo_hygiene::test_no_hardcoded_local_paths`
+ 的命中现在只剩环境文件(`artifacts/html-deleted-20260911.txt`、`.venv-mlx0305/pyvenv.cfg`、
+ `paper/.tools/**`),仓库源码与 `scripts/` 下的路径已清干净。
+- `staged_k4()`(35b,40 层 × 256 专家)与 `staged_k8()` 仍是 `cache_slots=64`:同样的
+ 超订问题更严重(35b 一步要碰 40×4=160 个键),但改 35b 属于另一个 tier,未动。
+
+---
+
+## 6. 第二轮(干净 clone `edge0_new`,HEAD `ae1ee2d`):开关语义 + ling 同步 + 四组实测
+
+### 6.1 开关修复:关掉 prerouter 必须是精确路径
+
+`--no-prerouter` 原来在**两个 tier 上都会留下 staged 槽位**(`options.staged=True` 时
+`stream_layers` 仍含所有层),于是关掉预测头后走的不是精确路径,而是"上一 token 实际
+top-k 填槽"的历史路径:edge0-8b 一次请求实测 `staged_dropped = 3593`;35b 早先
+"`--no-prerouter` 快 34%"也是同一原因(它少算了约 2/3 的专家)。§3.5 的修正说的就是这件事,
+但当时只改了消费层作用域、没修"关"这一侧。
+
+修法(`src/edge0/engine/{ling,qwen}.py::load_installed`)——作用域统一成一条规则:
+**槽位只给"路由来自预测"的层**(consumer `start_layer+1` 起);`prerouter is None`
+时没有任何预测,所以**没有任何层保留槽位**,走精确装载。`history_slots=True`
+(`--history-slots`)是显式 opt-in 回旧行为。
+
+| 配置 | staged 层 | `staged_dropped` |
+|---|---|---|
+| edge0-8b prerouter ON(作用域默认) | 16(L8–23) | 0 |
+| edge0-8b `--no-prerouter`(修复后) | **0** | **0** |
+| edge0-35b prerouter ON | 32(L7–38) | 0 |
+| edge0-35b `--no-prerouter`(修复后) | **0** | **0** |
+
+开关只有一个:`--no-prerouter`(demo / chat / serve 三处同名同义),没有反向 flag。
+
+### 6.2 ling(edge0-8b)同步 P1/P2/P3
+
+`src/edge0/engine/ling.py` 做了与 35b 同口径的三件事:`_reset_state` 清
+`layer.m_in_cache` / `block.last_topk` / `block.prev_topk_oh` + 每层 `reset()`(P1);
+`_prefill_end` 有 prerouter 时只 `stage_all()`,历史填槽只作用于非消费层且仅
+`history_slots=True` 时(P2);`_step_pre`/`_step_post` 按 `first_pg_consumer`
+作用域化,非消费层不填槽、不做无效 prefetch(P3)。`prod_k8()` 默认 `staged=True`
+(`cache_slots=64` 保持低内存档;收割关掉时预测不参与装载,收益结构性为 0)。
+
+### 6.3 四组实测(同机、先预热、每轮轮转、3 轮取中位)
+
+| 臂 | ms/步 | tok/s | MLX 峰值 | 进程 RSS | staged 层 | dropped |
+|---|---|---|---|---|---|---|
+| qwen prerouter ON | 49.06 | 20.38 | 2.89 GiB | 11.69 GiB | 32/40 | 0 |
+| qwen `--no-prerouter` | 50.21 | 19.92 | 2.07 GiB | 11.78 GiB | 0/40 | 0 |
+| ling prerouter ON | **35.66** | **28.04** | 1.48 GiB | 4.94 GiB | 16/23 | 0 |
+| ling `--no-prerouter` | 43.99 | 22.73 | 1.03 GiB | 4.86 GiB | 0/23 | 0 |
+
+- **速度**:ling 开 prerouter **快 8.33 ms/步(+23%,配对 3/3 轮一致 −6.0/−9.3/−8.6)**;
+ qwen 两边打平(−1.15 ms,配对 −11.6/+1.2/−3.0),与 §5.4 结论一致(35b 本机装载不是瓶颈)。
+- **内存**:prerouter 多占 +0.82 GiB(35b)/ +0.45 GiB(8b)MLX 峰值(槽位 + 头);
+ 进程 RSS 基本持平(11.69 vs 11.78 / 4.94 vs 4.86 GiB)。
+- **质量**:收割开 vs 关(prerouter 都开,只有装载调度不同)**token 逐位一致**——
+ qwen 两个 prompt 各 32 token IDENTICAL;ling 同样 IDENTICAL。
+- **跨请求重置**:同进程 req1→req2 与冷进程单跑 req2 的 token 序列 IDENTICAL;
+ ling 首步 logits digest 完全相同(−390314.312)。qwen 的 digest 有数值级差异
+ (−307200.0 vs −299008.0)但 32 token 全同,属"进程首跑与后续跑存在数值抖动"这一
+ 既有现象,不是跨请求残留。
+- 与 no-prerouter 对比时 token 会不同(路由来源从 gate 换成预测头,属模型语义,不是回归);
+ 两边文本都通顺。
+
+### 6.4 ling 的 demo 默认
+
+`Ling8BConfig.demo_no_prerouter = True` + `edge0.registry.demo_kwargs()`:
+`edge0 demo` / `examples/demo.py` 对 edge0-8b 默认走 gate 路由的精确路径;
+`chat` / `serve` 保持档位默认(预测路径)。冒烟(各 8 token):`demo`(ling) 无
+`prerouter installed` 行、`demo --no-prerouter` 无、`chat`(ling) 有、
+`chat --no-prerouter` 无、`examples/demo.py`(ling) 无。
+`pytest tests/` = 65 passed, 1 skipped。
+
+### 6.5 逐缺陷复核(同一 clone 上的实测签名)
+
+| 缺陷 | 判据 | qwen(35b) | ling(8b) |
+|---|---|---|---|
+| **P1 跨请求残留** | 同进程 req1→req2 的 token 序列 vs 冷进程单跑 req2 | 32 token 全同(首步 logits digest 有数值级差异,属"进程首跑"既有抖动) | 32 token 全同,digest 完全相同(−390314.312) |
+| **P2 首步** | prefill 结束有没有把"最后一个 prefill token 的实际 top-k"当预测填槽;首步是否零专家 | `prefetch_submitted=128`、`stage_builds=0`(改为预取预测集,不填槽);首步 `staged_used=0`、`staged_fallback=32`(32 层走精确路径)、`staged_dropped=0` | `stage_builds=128`(预测在 prefill 就绪);首步 `staged_used=128`、`staged_fallback=0`、`staged_dropped=0` |
+| **P3 非消费层丢专家** | staged 层集合 + `staged_dropped` | L7–38(32 层),L0–6/L39 不填槽,`dropped=0` | L8–23(16 层),L1–7 不填槽(owner 6 不存在 ⇒ L7 永远 gate,已知边界),`dropped=0` |
+
+两个 tier 的"首步正确签名"本来就不同:qwen 的特征只在单 token 前向捕获,首步按训练态走
+gate(pos-0 fallback)并由 prefill 预取预测集;ling 的特征在 prefill 内就绪,首步直接消费
+预测。这是数据流决定的,不是两个 tier 不一致。
+
+诚实附注:qwen 首步预取命中仅 5/128(首步路由是 gate、预取的是预测集,重合度低),即首步
+基本还是按需装载 —— 正确性优先的取舍,也是 35b 在本机 prerouter 提速 ≈0 的原因之一。
+
+
+
+
diff --git a/docs/prerouter-switch-semantics-2026-09-14.md b/docs/prerouter-switch-semantics-2026-09-14.md
new file mode 100644
index 0000000..e465cc1
--- /dev/null
+++ b/docs/prerouter-switch-semantics-2026-09-14.md
@@ -0,0 +1,146 @@
+# prerouter 开关语义与生效判据(edge0-8b / ling)
+
+> 只看一件事:**开关打开时必须真的走 prerouter,关掉时必须真的完全不走**。
+> 本文不讨论收益、不提速、不比快慢;只定义"开/关"的语义、生效的判据、以及
+> 会让开关名不副实的故障。
+>
+> 日期:2026-09-14;对象:`edge0-8b`(Ling 3.0 hybrid);代码以当前工作树为准。
+
+---
+
+## 1. 有三个不同层次的"开关",先别混
+
+| 层次 | 开关 | 位置 | 作用 |
+|---|---|---|---|
+| 总开关 | `--no-prerouter`(默认**开**) | `src/edge0/cli.py:65-66` → `kw["prerouter"] = None` | 决定是否构建/使用预测头 |
+| 模型内派生 | `prerouter_enabled` | `src/edge0/engine/ling.py:53`(`cfg.prerouter is not None`) | 传给 vendored 模型,逐层派生下面两个标志 |
+| 逐层角色 | `has_prerouter`(本层**有头**,预测下一层)<br>`use_prerouter`(本层**消费**预测) | `backends/mlx/_impl/bailing_hybrid.py:712-724` | 决定谁跑头、谁从预测里选专家 |
+| 装载侧 | `staged` / `staged_n` / `staged_sync` | `src/edge0/streaming/options.py`(`prod_k8()`) | 是否把预测出的专家**提前装进槽**;**不改变路由来源** |
+
+逐层派生规则(8b 的 `start_layer=7`, 24 层):
+
+```python
+has_prerouter = enabled and li >= start_layer - 1 and li < n_layers - 1 # li in [6, 22]
+use_prerouter = enabled and li >= start_layer and li < n_layers # li in [7, 23]
+```
+
+- **谁提供预测**:`owners`,显式写死在 `src/edge0/models/edge0_8b/__init__.py:53`
+ → `owners=tuple(range(7, 23))` = **16 个头(7..22)**,owner `li` 预测 `li+1`。
+ 实测烘好的头文件 `~/Documents/edge0-8b/prerouter_edge0_8b.safetensors`:48 个数组
+ (每头 `fc1/fc2/linear_init`),layers = 7..22,元数据 `owners=[7..22]`。
+- **谁消费预测**:`use_prerouter` → L7..23;实际有供给的是 **L8..23**(见 §7 边界)。
+- stager 只遍历 `state.owners`(`src/edge0/prerouter/stager.py:176`),头集合由 `owners` 决定,
+ 与 `has_prerouter` 的派生范围无关。
+
+**路由判定点**(唯一一处,`bailing_hybrid.py:682-693`):
+
+```python
+if self.use_prerouter and prev_prerouter_logits is not None:
+ idx, w = self._select_from_logits(prev_prerouter_logits) # ON:路由来自预测
+else:
+ idx, w = self.gate(x) # 否则:本层自己的 gate
+```
+
+`prev_prerouter_logits` 在模型前向循环里取值(`bailing_hybrid.py:808-813`):
+
+```python
+if layer.use_prerouter and prerouter_cache is not None and li in prerouter_cache:
+ prev_prerouter_logits = prerouter_cache[li]
+```
+
+→ **注意这三个条件是与关系:标志为真但缓存里没有这一层的条目时,会静默回退到 gate**
+(不报错、不打日志)。这是开关最容易名不副实的地方,见 §4。
+
+---
+
+## 2. 开:必须成立的五个条件
+
+| # | 条件 | 代码位置 | 判据(怎么看出真的开了) |
+|---|---|---|---|
+| 1 | 头被构建:`owners` 16 个 | `ling.py:95-105`(`install_prerouter` / `len(heads)`) | 启动打印 `prerouter installed: 16 heads, start=7, K=8` |
+| 2 | 每个 consumer 每步都拿到预测:`pg_cache[li]` 已写入 | `stager.py:302-308`(`_store`) | `pg_cache` 条目数 == 有供给的 consumer 数(8..23 = 16) |
+| 3 | 首步就有预测(消费层从 token 1 就消费) | `ling.py` 的 `_prefill_end` → `stage_all()` | 第一步输出与 gate 路由不同;`consumed` 从第 1 步起增长 |
+| 4 | 每个请求重置干净 | `ling.py` 的 `_reset_state`(`pg_cache` + `m_in_cache`/`last_topk`/`prev_topk_oh`) | 同一进程连续两请求 == 冷进程单跑(见 §5) |
+| 5 | 装载侧作用域正确:只有 `li >= start_layer + 1` 用槽 | `ling.py:78-80` 作用域 `_staged_mode` | 消费层 `slots == routing`;`dropped == 0` |
+
+条件 3 的特征来源(`stager.py:276-295`(`_features`)):`layer.m_in_cache`(本层 post-attention 后的
+MLP 输入,取最后位置)+ `block.last_topk` 的 one-hot(当前 token)+ `block.prev_topk_oh`
+的 one-hot(上一 token);prefill 结束的首次 staging 用倒数第 2 个位置当"上一 token"。
+
+---
+
+## 3. 关:`--no-prerouter` 必须做到的全停清单
+
+`--no-prerouter` → `prerouter=None` → `prerouter_enabled=False` → 两个派生标志全为 False。
+
+| 必须为"零"的项 | 位置 | 判据 |
+|---|---|---|
+| 不构建任何头 | `ling.py:95-105`(`if cfg.prerouter and ...`) | 启动**没有** `prerouter installed` 那行 |
+| stager 不存在 | `ling.py:95`(`pg_state = pg_stager = None`) | 无 `pg_cache` 写入 |
+| 前向缓存传 None | `ling.py:211`(`prerouter_cache=self._pg_stager.pg_cache` 只在存在时) | 模型循环里 `prerouter_cache is None` → 永不取值 |
+| 逐层 `use_prerouter=False` | `bailing_hybrid.py:717-724` | 所有 MoE 层走 `self.gate(x)` |
+| 无头前向、无预测槽装载 | 头不存在 → 无从执行 | 头/预测相关计数全 0 |
+| 不保留任何 staged 槽位 | `ling.py:80-89`(`prerouter is None` ⇒ 全部层 `_staged_mode=False`) | `stream_layers` 为空、`staged_dropped == 0`、装载按需发生(`loads` 仍 > 0) |
+
+**关的等价物**:与 `exact`(精确装载)档在路由来源上一致 —— 两者都是 gate 路由,
+且(修复后)都不保留 staged 槽位,即 `--no-prerouter` **就是**精确路径本身。
+修复前它反而会留下全部层的槽位(走历史路径,见 §4"关掉仍填槽"一行)。
+
+---
+
+## 4. 六种"开关开着/关着但没生效"的故障(均已修复,列出以防回归)
+
+| 故障 | 现象 | 根因 | 修法 |
+|---|---|---|---|
+| **静默回退** | 部分层悄悄用 gate,得到既不 ON 也不 OFF 的**第三态**(混合路由) | `pg_cache` 缺该层条目时 `if` 条件不成立,无日志无异常(`bailing_hybrid.py:808-813`) | 必须保证"每个有供给的 consumer 每步都有条目";用 `consumed/dropped` 判据守住 |
+| **首步被覆盖** | 第一步路由错一位 | `_prefill_end` 里 `stage_from_prefill()` 用"最后一个 prefill token 的实际 top-k"顶掉了刚算出的预测 | 有 prerouter 时只跑 `stage_all()`,不再 `stage_from_prefill` |
+| **跨请求残留** | 第 2 个请求第一步用第 1 个请求的 hidden state 跑头 | `_reset_state` 未清 `layer.m_in_cache` / `block.last_topk` / `block.prev_topk_oh` | 逐 owner 清这三个字段 + 清 `pg_cache` |
+| **非消费层错位** | L1-7 丢 17-26% 真实专家(`history_slots` 档实测 `dropped=1589`),输出劣化 | L1-7 没有预测,却用"上一 token 实际 top-k"填槽并当预测消费 | 作用域 `_staged_mode`:只有 `li >= start_layer+1` 走槽 |
+| **关掉仍填槽** | 关掉 prerouter 后走的不是精确路径,而是同一条"上一 token 实际 top-k"历史路径(8b 实测 `dropped=3593`;35b 少算约 2/3 专家),输出劣化 | 作用域只对"非消费层"关槽(`li < first_pg_consumer`),而 `prerouter=None` 时 `first_pg_consumer=0` → 一层都没关 | `prerouter is None` ⇒ 全部层 `_staged_mode=False`(精确装载);`history_slots=True` 才回旧行为 |
+| **头批次崩溃** | 开了直接 `AttributeError`(等于开关不可用) | 批量头路径读 `_head_of(...)._dtype`,而 ling 的头是 `BailingPrerouter`(无该属性) | 从堆叠后的权重取 dtype(`w1.dtype`) |
+
+判定口径:`dropped` = 槽里装了、但该层路由没用上的专家数(应当恒为 0);
+`consumed` = 槽被消费的次数。
+
+---
+
+## 5. 生效验证(实测)
+
+| 检查项 | 结果 | 含义 |
+|---|---|---|
+| 槽位与路由一致性 | `consumed=62976`,`dropped=0`,rate 0.00% | 预测装进去的专家 100% 被该层路由消费,无错位 |
+| 跨请求重置 | 同进程(请求 1 后跑请求 2)== 冷进程(只跑请求 2):token 序列一致,首步 logits digest 均为 `p2_first=592203.875` | 条件 4 成立,无残留污染 |
+| 与精确路径 parity | 4/4 prompt `identical=True, first_divergence=None`;greedy-32 token 序列一致 | ON 的价值路径与"精确路径"逐 token 相同 |
+| 四档对照(路由来源 / `dropped` / parity) | `default`(预测路由)= 0 / True;`history`(L1-7 也用历史槽)= 1589 / False;`exact`(无 prerouter,gate 路由)= 0 / 不适用;`nopg`(预测算了但不消费)= 10481 / False | 只有"预测路由 + 作用域正确"这一档是生效态 |
+
+---
+
+## 6. 30 秒自检
+
+```bash
+# 开(默认):应打印 prerouter installed: 16 heads, start=7, K=8
+edge0 chat --name edge0-8b --model-dir ~/Documents/edge0-8b \
+ --prompt "用一句话说明什么是MoE。" --max-new 32
+
+# 关:不应出现上面那行
+edge0 chat --name edge0-8b --model-dir ~/Documents/edge0-8b --no-prerouter \
+ --prompt "用一句话说明什么是MoE。" --max-new 32
+```
+
+判定:**开**=有 `installed` 行 + `dropped == 0` + 输出与 `--no-prerouter` 不同;
+**关**=无 `installed` 行 + 无 `pg_cache` 写入 + 输出等于 gate 路由(`exact` 档)。
+
+---
+
+## 7. 已知边界(不是 bug,是配置事实,但会被误读成"开关没生效")
+
+1. **L7 是"有消费无供给"的缝隙。** consumer 从 `start_layer=7` 起算(L7..23),而 `owners`
+ 显式从 7 开始(预测 8..23)。因此 **L7 永远拿不到预测、永远走 gate**;`has_prerouter`
+ 的派生规则会从 `start_layer-1 = 6` 起算,但 8b 的显式 `owners` 覆盖了它(没有 owner 6,
+ 烘好的文件里也确实没有 layer 6)。想消除这条缝隙要么把 `start_layer` 写 8,要么让
+ `owners` 含 6,属于配置决策。
+2. **16 个头 ≠ 22。** `src/edge0/models/edge0_8b/__init__.py:10-13` 的文件头注释写的是
+ "22 heads (explicit owners 1..22), start_layer 1",与同文件 `:49-58` 的 spec
+ (`start_layer=7, owners=7..22`)和烘好的 artifact 元数据(`owners=[7..22]`,48 个数组、
+ layers 7..22)**都不一致**。那行注释描述的是 start-1 变体;按 §1 的代码事实,正确写法是
+ **16 heads(owners 7..22)、start_layer 7**。
diff --git a/examples/demo.py b/examples/demo.py
index e4e0641..686700f 100644
--- a/examples/demo.py
+++ b/examples/demo.py
@@ -10,6 +10,11 @@ Run (checkpoint path or registered tier name):
Three lines do everything: build the engine (prerouter + LoRA + SSD
offload installed automatically from the tier config), template the
prompt, generate.
+
+The demo runs each tier's showcase configuration: ``edge0-8b`` defaults
+to the gate-routed exact path (``prerouter=None``). Use ``edge0 chat``
+(or ``AutoEngine.from_pretrained(..., prerouter=...)``) for the
+prediction path.
"""
from __future__ import annotations
@@ -17,7 +22,7 @@ from __future__ import annotations
import argparse
import sys
-from edge0 import AutoEngine
+from edge0 import AutoEngine, demo_kwargs
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
@@ -30,9 +35,14 @@ def main() -> int:
ap.add_argument("--prompt",
default="Hello! Write one short sentence about the seaside.")
ap.add_argument("--max-new", type=int, default=24)
+ ap.add_argument("--no-prerouter", action="store_true",
+ help="route with the layers' own gates (edge0-8b demos "
+ "already run that path)")
args = ap.parse_args()
- engine = AutoEngine.from_pretrained(args.model_dir, name=args.model)
+ kw = ({"prerouter": None} if args.no_prerouter
+ else demo_kwargs(args.model_dir, args.model))
+ engine = AutoEngine.from_pretrained(args.model_dir, name=args.model, **kw)
req = ChatRequest(
model=engine.name,
messages=[ChatMessage(role="user", content=args.prompt)],
diff --git a/src/edge0/__init__.py b/src/edge0/__init__.py
index 432e6ca..bf81bd0 100644
--- a/src/edge0/__init__.py
+++ b/src/edge0/__init__.py
@@ -23,13 +23,14 @@ __all__ = [
"AutoConfig",
"AutoModel",
"AutoEngine",
+ "demo_kwargs",
]
def __getattr__(name: str):
# Deferred imports so `import edge0` stays light and backend selection
# (EDGE0_BACKEND) happens on first use.
- if name in ("AutoConfig", "AutoModel", "AutoEngine"):
+ if name in ("AutoConfig", "AutoModel", "AutoEngine", "demo_kwargs"):
from edge0 import registry as _reg
return getattr(_reg, name)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
diff --git a/src/edge0/cli.py b/src/edge0/cli.py
index cefa632..0e025dc 100644
--- a/src/edge0/cli.py
+++ b/src/edge0/cli.py
@@ -66,6 +66,8 @@ def _engine_kwargs(args) -> dict:
kw["prerouter"] = None
if getattr(args, "no_lora", False):
kw["lora"] = ""
+ if getattr(args, "history_slots", False):
+ kw["history_slots"] = True
return kw
@@ -122,14 +124,19 @@ def _display_text(text: str, show_thinking: bool) -> str:
def cmd_demo(args) -> int:
from edge0 import AutoEngine
+ from edge0.registry import demo_kwargs
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
model_dir, name = _resolve_model(args)
if not model_dir or not os.path.isdir(model_dir):
print(_missing_model_help(name), file=sys.stderr)
return 2
- engine = AutoEngine.from_pretrained(model_dir, name=name,
- **_engine_kwargs(args))
+ kw = _engine_kwargs(args)
+ if "prerouter" not in kw:
+ # No explicit --no-prerouter: the tier's demo default applies
+ # (edge0-8b demos run the gate-routed exact path).
+ kw = demo_kwargs(model_dir, name, **kw)
+ engine = AutoEngine.from_pretrained(model_dir, name=name, **kw)
tok = engine._tok
if tok is None:
engine.close()
@@ -232,6 +239,10 @@ def main(argv: list[str] | None = None) -> int:
help="print the model's reasoning block too")
p.add_argument("--no-prerouter", action="store_true")
p.add_argument("--no-lora", action="store_true")
+ p.add_argument("--history-slots", action="store_true",
+ help="legacy staging: also fill staged slots from history "
+ "for layers whose route is not a prerouter prediction "
+ "(zeroes routed experts outside the slot set)")
p.set_defaults(fn=cmd_demo)
p = sub.add_parser(
@@ -248,6 +259,10 @@ def main(argv: list[str] | None = None) -> int:
help="print the model's reasoning block too")
p.add_argument("--no-prerouter", action="store_true")
p.add_argument("--no-lora", action="store_true")
+ p.add_argument("--history-slots", action="store_true",
+ help="legacy staging: also fill staged slots from history "
+ "for layers whose route is not a prerouter prediction "
+ "(zeroes routed experts outside the slot set)")
p.set_defaults(fn=cmd_chat)
p = sub.add_parser(
@@ -263,6 +278,10 @@ def main(argv: list[str] | None = None) -> int:
help="use the Flask transport (needs flask installed)")
p.add_argument("--no-prerouter", action="store_true")
p.add_argument("--no-lora", action="store_true")
+ p.add_argument("--history-slots", action="store_true",
+ help="legacy staging: also fill staged slots from history "
+ "for layers whose route is not a prerouter prediction "
+ "(zeroes routed experts outside the slot set)")
p.set_defaults(fn=cmd_serve)
p = sub.add_parser("convert-adapters",
diff --git a/src/edge0/engine/ling.py b/src/edge0/engine/ling.py
index b8a0344..87229b9 100644
--- a/src/edge0/engine/ling.py
+++ b/src/edge0/engine/ling.py
@@ -64,6 +64,32 @@ def load_installed(model_dir: str, cfg):
installed = install_streaming_experts(
model, shards, spec, options=opts, num_layers=n_layers)
all_stream = {li: t for li, t in enumerate(installed) if t is not None}
+ # Zero-drop layer scoping (quality invariant): staged slots are only
+ # coherent for layers whose ROUTE comes from the prerouter (consumer
+ # li >= start_layer + 1: owner li-1 predicts li, and the consuming
+ # block re-selects from the SAME cached logits the stager staged
+ # from — set == routing, drops == 0, output bit-parity with the
+ # exact path). Layers at/below start_layer route with their own
+ # gate every token; their slots cannot be predicted, so they run
+ # the exact path. History-staging them instead (qwen-style
+ # actuals-as-prediction) misaligns one step behind the gate and
+ # measurably degrades output (17% drop rate, garbled text).
+ first_pg_consumer = (
+ getattr(cfg.prerouter, "start_layer", 7) + 1
+ if cfg.prerouter else 0)
+ if not getattr(cfg, "history_slots", False):
+ # Scoped default: only the consumers keep staged slots. Switch-off
+ # takes the same rule to its limit -- with ``--no-prerouter`` there is
+ # no prediction anywhere, so NO layer keeps slots and the off path is
+ # the exact path (gate routing, on-demand loads), not history staging
+ # (measured ``staged_dropped = 3593`` over a whole run when the slots
+ # stayed on with the prerouter off). Legacy ``history_slots=True``
+ # deliberately leaves every layer staged (slots filled from the
+ # previous token's actuals), which is the behaviour that silently
+ # zeroes the routed experts outside the slot set.
+ for li, t in all_stream.items():
+ if cfg.prerouter is None or li < first_pg_consumer:
+ t._staged_mode = False
stream_layers = {li: t for li, t in all_stream.items() if t._staged_mode}
if cfg.lora:
@@ -83,6 +109,7 @@ def load_installed(model_dir: str, cfg):
f"K={cfg.prerouter_top_k}", flush=True)
installs = dict(all_stream_layers=all_stream,
stream_layers=stream_layers,
+ first_pg_consumer=first_pg_consumer,
pg_state=pg_state, pg_stager=pg_stager)
return model, model_config, shards, installs
@@ -112,6 +139,7 @@ class Ling8BEngine(Edge0Engine):
inst) = load_installed(cfg.model_dir, cfg)
self._all_stream_layers = inst["all_stream_layers"]
self._stream_layers = inst["stream_layers"]
+ self._first_pg_consumer = inst["first_pg_consumer"]
self._pg_state = inst["pg_state"]
self._pg_stager = inst["pg_stager"]
opts = cfg.options
@@ -188,7 +216,10 @@ class Ling8BEngine(Edge0Engine):
if self._pg_stager is not None else None))
logits = self.model.lm_head(h[0, -1])
core.eval(logits)
- # Step boundary: ONE stacked head batch -> ONE tolist -> fills.
+ # Step boundary: ONE head batch (stacked einsum) -> fills. The
+ # host-side tolist only happens when a consumer actually needs the
+ # expert IDs (``stream_layers`` non-empty); with staged loading off
+ # the routing cache is filled lazily and the sync is skipped.
if (self._pg_stager is not None and not self._prefill_active
and len(ids) == 1):
self._pg_stager.stage_all()
@@ -197,19 +228,46 @@ class Ling8BEngine(Edge0Engine):
# ---- step hooks -------------------------------------------------------
def _step_pre(self, token_id: int) -> None:
- if self._pg_stager is None and self._history_prefetch is not None:
+ if self._pg_stager is not None:
+ # Legacy ``history_slots=True``: the layers without an incoming
+ # prediction (li < start+1) are still staged, so fill them from
+ # their previous actuals. In the scoped default they are
+ # ``_staged_mode=False`` and run the exact path here -- no
+ # staging and no prefetch (the previous token's set covers only a
+ # small fraction of the next route, so the reads are wasted).
+ for li, exp in self._all_stream_layers.items():
+ if li < self._first_pg_consumer:
+ if exp._staged_mode and exp.last_used:
+ exp.stage_experts(list(exp.last_used))
+ elif self._history_prefetch is not None:
self._history_prefetch()
def _step_post(self, token_id: int) -> None:
+ if self._pg_stager is not None:
+ # Consumers: the prerouter's ``stage_all`` already submitted
+ # their next set; only record actuals (drop accounting + hot
+ # pins). Non-consumers run the exact path in the scoped default
+ # (``_staged_mode=False``), where this bookkeeping is dead
+ # weight; legacy ``history_slots`` stages them from their
+ # actuals like the no-prerouter path.
+ for li, exp in self._all_stream_layers.items():
+ if li < self._first_pg_consumer:
+ if not exp._staged_mode:
+ continue
+ exp.sync_actuals()
+ if exp.last_used:
+ exp.stage_experts(list(exp.last_used))
+ exp.swap_staged()
+ elif exp._staged_mode:
+ exp.sync_actuals()
+ return
+ # No prerouter: history staging -- each layer's next set is its
+ # previous actuals (adjacent-token expert locality).
for exp in self._stream_layers.values():
exp.sync_actuals()
- if self._pg_stager is None:
- # History staging: each layer's next set is its previous
- # actuals (adjacent-token expert locality).
- for exp in self._stream_layers.values():
- if exp.last_used:
- exp.stage_experts(list(exp.last_used))
- exp.swap_staged()
+ if exp.last_used:
+ exp.stage_experts(list(exp.last_used))
+ exp.swap_staged()
# ---- prefill / reset --------------------------------------------------
@@ -218,21 +276,48 @@ class Ling8BEngine(Edge0Engine):
exp.clear_full_layer()
if self._pg_stager is not None:
# Deployment parity: prefill's tail stages the FIRST decode
- # token's expert sets (engine.py:919 calls stage_all right
- # after prefill). Without this the first decode step has no
- # prerouter logits, falls back to the true gate for one step,
- # and the whole prediction chain shifts by one token
- # (observed: [23982, 2862, ...] vs correct [23982, 4264, ...]).
+ # token's expert sets for the prerouter consumers (engine.py:919
+ # calls stage_all right after prefill). Without this the first
+ # decode step has no prerouter logits, falls back to the true
+ # gate for one step, and the whole prediction chain shifts by one
+ # token (observed: [23982, 2862, ...] vs correct [23982, 4264, ...]).
self._pg_stager.stage_all()
- if self._stream_layers:
+ if getattr(self.cfg, "history_slots", False):
+ # Legacy: the non-consumer layers are staged too and get the
+ # last prefill token's actual top-k. MUST NOT touch the
+ # consumers -- ``stage_from_prefill`` would overwrite the
+ # prerouter's submissions and misalign the first step.
+ for li, exp in self._stream_layers.items():
+ if li < self._first_pg_consumer:
+ exp.stage_from_prefill()
+ elif self._stream_layers:
+ # No prerouter: history staging is the only predictor (legacy).
for exp in self._stream_layers.values():
exp.stage_from_prefill()
def _reset_state(self) -> None:
self.cache = self.model.make_cache()
- if self._pg_stager is not None:
+ if self._pg_state is not None:
self._pg_stager.reset()
self._pg_state.reset()
+ # Drop the per-block/per-layer feature captures as well, or the
+ # NEXT request's first ``stage_all`` runs the heads on this
+ # request's last token: ``prev_topk_oh`` is only ever written by
+ # ``_note`` at the step boundary, so it would feed the previous
+ # request's last top-k as the "prev token" feature and stage
+ # stale predictions that the first decode step then routes with.
+ for owner in self._pg_state.owners:
+ block = self.cfg.moe_spec.block_of(self.model, owner)
+ layer = self.cfg.moe_spec.layer_of(self.model, owner)
+ block.last_topk = None
+ block.prev_topk_oh = None
+ layer.m_in_cache = None
+ # Per-request reset of the streaming layers' staged state: a leftover
+ # ``_staged_state`` from the previous request defeats the staged
+ # path's "fill not ready -> exact path" fallback (``st`` would never
+ # be None) and would serve the previous request's expert bundles.
+ for exp in getattr(self, "_all_stream_layers", {}).values():
+ exp.reset()
def _lm_logits(self, h: core.array) -> core.array:
return self.model.lm_head(h[0, -1])
diff --git a/src/edge0/engine/qwen.py b/src/edge0/engine/qwen.py
index 12e0953..96ce7f2 100644
--- a/src/edge0/engine/qwen.py
+++ b/src/edge0/engine/qwen.py
@@ -60,6 +60,31 @@ def load_installed(model_dir: str, cfg):
installed = install_streaming_experts(
model, shards, spec, options=opts, num_layers=n_layers)
all_stream = {li: t for li, t in enumerate(installed) if t is not None}
+ # Zero-drop layer scoping (quality invariant). Staged slots are only
+ # coherent for a layer whose ROUTE is a prerouter prediction (consumer
+ # ``start_layer+1 .. n_layers-2``): the head's submission is exactly the
+ # set that consumer itself re-selects from the same cached state, so
+ # slots == routing and ``staged_dropped == 0``. Every other layer routes
+ # with its own gate, and slots filled from history (the previous token's
+ # actuals) silently zero every routed expert outside that set -- measured
+ # 3.0-3.75 of 4 per step for L0-6/L39, and the same class of misalignment
+ # in the 8b tier. Those layers run the exact path (``_step_pre`` /
+ # ``_step_post``) instead.
+ #
+ # Switch-off is the same rule taken to its limit: with
+ # ``--no-prerouter`` there is no prediction anywhere, so no layer keeps
+ # slots and the off path is the exact path (gate routing, on-demand
+ # loads), not history staging. ``history_slots=True`` opts back into the
+ # legacy behaviour on purpose (``--history-slots``).
+ if not getattr(cfg, "history_slots", False):
+ if cfg.prerouter is None:
+ for t in all_stream.values():
+ t._staged_mode = False
+ else:
+ lo, hi = cfg.prerouter.start_layer, n_layers - 2
+ for li, t in all_stream.items():
+ if not lo <= li <= hi:
+ t._staged_mode = False
stream_layers = {li: t for li, t in all_stream.items() if t._staged_mode}
if cfg.lora:
@@ -151,11 +176,11 @@ class Qwen35Engine(Edge0Engine):
# Step boundary: ONE stacked head batch -> ONE tolist -> fills.
# Demo parity (engine_qwen._forward: the deployment's pre-routing /
# trained-state condition and not full_layer): stage_all+swap run
- # after EVERY forward, including prefill chunks — so the last
- # prefill token's predictions fill pred_inds and oh_prev, and the
- # FIRST decode step consumes a real prediction at L7 instead of
- # falling back to the router (a different, cleaner-but-divergent
- # trajectory).
+ # after EVERY forward, including prefill chunks. During prefill the
+ # per-layer features (block.prerouter_m_in/oh) are not captured (the
+ # patch only fills them on single-token forwards), so stage_all
+ # submits nothing and the first decode step consumes no prediction --
+ # it routes with the router, the pos-0 fallback used in training.
if (self._pg_stager is not None and not full_layer):
self._pg_stager.stage_all()
self._pg_state.swap()
@@ -165,11 +190,18 @@ class Qwen35Engine(Edge0Engine):
def _step_pre(self, token_id: int) -> None:
if self._pg_stager is not None:
- # Router layers (no owner): fill from their previous actuals.
+ # Layers without an incoming prediction (L0-6, L39, and every
+ # layer switched off by the history_slots scoping): they route
+ # with their own gate this step, so a staged slot set filled from
+ # history would silently zero every routed expert outside it.
+ # They run the exact path -- staging them is the legacy
+ # history_slots=True behaviour. No prefetch either: the previous
+ # token's set overlaps the next route by only ~6%, so the reads
+ # are wasted (measured 11.7 -> 16.7 tok/s when removed).
st = self._pg_state
for li, exp in self._all_stream_layers.items():
if li < st.start or li >= st.n - 1:
- if exp.last_used:
+ if exp._staged_mode and exp.last_used:
exp.stage_experts(list(exp.last_used))
elif self._history_prefetch is not None:
self._history_prefetch()
@@ -179,7 +211,13 @@ class Qwen35Engine(Edge0Engine):
st = self._pg_state
for li, exp in self._all_stream_layers.items():
if li < st.start or li >= st.n - 1:
- exp.sync_actuals()
+ # Non-staged layers stage nothing from their actuals, so
+ # the bookkeeping (and its per-layer host sync) is dead
+ # weight -- unless the layer keeps hot pins, whose counts
+ # it feeds.
+ if (exp._staged_mode
+ or getattr(exp, "hot_per_layer", 0) > 0):
+ exp.sync_actuals()
return
if self._stream_layers:
for exp in self._all_stream_layers.values():
@@ -195,13 +233,41 @@ class Qwen35Engine(Edge0Engine):
for exp in self._all_stream_layers.values():
exp.refresh_hot_pins()
if self._stream_layers:
- for exp in self._stream_layers.values():
- exp.stage_from_prefill()
+ if self._pg_stager is None or getattr(self.cfg, "history_slots",
+ False):
+ for exp in self._stream_layers.values():
+ exp.stage_from_prefill()
+ else:
+ # With a prerouter the first decode step routes with the
+ # router (pos-0 fallback, as in training): staging the last
+ # prefill token's actual top-k there zeroes every routed
+ # expert outside that set (measured 4/4 for L7). Prefetch
+ # them instead; the step then runs the exact path.
+ for exp in self._stream_layers.values():
+ exp.prefetch_from_prefill()
def _reset_state(self) -> None:
self.cache = self._lm.make_cache()
if self._pg_state is not None:
self._pg_state.reset()
+ # Drop the per-block feature captures too. The MoE patch writes
+ # ``prerouter_m_in`` / ``prerouter_oh`` only on single-token
+ # forwards and nothing else touches them, so without this the
+ # NEXT request's prefill would re-run every head on the previous
+ # request's last hidden state: its stage_all would submit stale
+ # predictions that the first decode step then routes with (the
+ # fresh-process behaviour is the gate fallback at pos 0).
+ for owner in self._pg_state.owners:
+ block = self.cfg.moe_spec.block_of(self.model, owner)
+ block.prerouter_m_in = None
+ block.prerouter_oh = None
+ # Per-request reset of the streaming layers' staged state
+ # (``StreamingSwitchGLU.reset``): a leftover ``_staged_state`` from
+ # the previous request defeats the staged path's "fill not ready ->
+ # exact path" fallback (``st`` would never be None) and would serve
+ # the previous request's expert bundles.
+ for exp in getattr(self, "_all_stream_layers", {}).values():
+ exp.reset()
def _lm_logits(self, h: core.array) -> core.array:
return self._lm.lm_head(h[0, -1])
diff --git a/src/edge0/models/base.py b/src/edge0/models/base.py
index 09a14b9..7677182 100644
--- a/src/edge0/models/base.py
+++ b/src/edge0/models/base.py
@@ -50,6 +50,16 @@ class ModelConfig:
options: LayerOptions
prerouter: PrerouterSpec | None = None
prerouter_top_k: int = 0 # 0 -> options.top_k
+ history_slots: bool = False # False (default): only the layers whose
+ # ROUTE is a prerouter prediction keep
+ # staged slots; every other layer
+ # prefetches its history top-k and runs
+ # the exact path. True = deployment
+ # legacy: staged slots are also filled
+ # from history (previous token's
+ # actuals / last prefill token), which
+ # zeroes every routed expert outside
+ # the slot set.
lora: str = "" # safetensors path; "" disables
lora_r: int = 16
lora_alpha: float = 32.0
diff --git a/src/edge0/models/edge0_8b/__init__.py b/src/edge0/models/edge0_8b/__init__.py
index 7c8f4db..7fcc724 100644
--- a/src/edge0/models/edge0_8b/__init__.py
+++ b/src/edge0/models/edge0_8b/__init__.py
@@ -29,6 +29,12 @@ from edge0.streaming.options import LayerOptions
class Ling8BConfig(ModelConfig):
"""edge0-8b family config (Ling 3.0 hybrid, K=8)."""
+ #: ``edge0 demo`` / ``examples/demo.py`` default to the gate-routed exact
+ #: path for this tier (``prerouter=None``); ``edge0 chat`` / ``serve``
+ #: keep the tier default (prediction path). Consumed by
+ #: ``edge0.registry.demo_kwargs``.
+ demo_no_prerouter = True
+
@classmethod
def _defaults(cls, model_dir: str) -> "Ling8BConfig":
return cls(
diff --git a/src/edge0/prerouter/stager.py b/src/edge0/prerouter/stager.py
index 2c8a613..c380ca5 100644
--- a/src/edge0/prerouter/stager.py
+++ b/src/edge0/prerouter/stager.py
@@ -23,11 +23,13 @@ implemented here:
from __future__ import annotations
+import os
+
from edge0.backends import core
from edge0.moe.routing import group_select_from_logits, select_from_logits
from edge0.moe.spec import MoESpec, RouterKind
-from edge0.prerouter.heads import topk_onehot
+from edge0.prerouter.heads import gelu_erf, topk_onehot
from edge0.prerouter.spec import PrerouterSpec
from edge0.prerouter.state import PrerouterState
from edge0.streaming.layer import StreamingSwitchGLU
@@ -54,7 +56,20 @@ def _topk_onehot_pos(idx, num_experts: int, pos: int = -1):
class PrerouterStager:
"""Base: one-batch collect -> stack -> select -> eval -> tolist ->
- per-consumer stage_experts."""
+ per-consumer stage_experts.
+
+ Two cost fixes on top of the naive loop (measured on the 8b tier,
+ docs/experiments/prerouter-decode-speed-2026-09-14.md §6):
+
+ * ``head_batch`` (default on): the owner heads share one shape, so their
+ weights are stacked once and each step runs 3 ``einsum`` ops instead of
+ 3 small ops per head. Mathematically identical up to fp16 rounding;
+ ``head_batch=False`` (or ``EDGE0_HEAD_BATCH=0``) keeps the per-head loop
+ for A/B.
+ * when nothing consumes host-side expert IDs (``stream_layers`` empty,
+ i.e. ``staged=False``) the ``core.eval`` + ``tolist`` are skipped: the
+ routing cache holds lazy arrays, so the sync produces nothing usable.
+ """
def __init__(
self,
@@ -75,6 +90,9 @@ class PrerouterStager:
self.num_experts = spec.num_experts
self.records: list | None = None
self.cur_step = 0
+ self.head_batch = os.environ.get("EDGE0_HEAD_BATCH", "1") != "0"
+ self._stacked_key: tuple | None = None
+ self._stacked: tuple | None = None
# ---- feature hooks (overridden per family) ----------------------------
@@ -87,13 +105,67 @@ class PrerouterStager:
raise NotImplementedError
def _store(self, consumer: int, logits_i, inds_i, scores_i,
- experts: list[int]) -> None:
- """Record this consumer's prediction + submit its staged fill."""
+ experts: list[int] | None) -> None:
+ """Record this consumer's prediction + submit its staged fill
+ (``experts=None`` when no consumer needs host-side IDs)."""
raise NotImplementedError
def _note(self, owner: int, cur_oh) -> None:
"""Per-owner side effects (e.g. roll the prev-token feature)."""
+ # ---- batched head execution (fix 1) -----------------------------------
+
+ def _stacked_weights(self, metas: list[int]):
+ """Stack the owner heads' weights once (all heads share one shape).
+
+ Returns None (and stays on the per-head loop) if anything about the
+ shapes/dtypes is not uniform."""
+ key = tuple(metas)
+ if not self.head_batch:
+ return None
+ if self._stacked_key == key:
+ return self._stacked
+ self._stacked_key = key
+ self._stacked = None
+ try:
+ heads = [self._head_of(o) for o in metas]
+ t = lambda w: core.transpose( # noqa: E731
+ core.stack([h.weight for h in w]), (0, 2, 1))
+ w1 = t([h.fc1 for h in heads])
+ w2 = t([h.fc2 for h in heads])
+ wl = t([h.linear_init for h in heads])
+ core.eval(w1, w2, wl)
+ self._stacked = (w1, w2, wl)
+ except Exception: # noqa: BLE001 — any shape mismatch: per-head loop
+ self._stacked = None
+ return self._stacked
+
+ def _logits(self, metas: list[int], inputs: list[core.array],
+ cur_ohs: list[core.array], prev_ohs: list[core.array]):
+ """Stacked head logits [n,1,1,E] (batched, else per-head loop)."""
+ w = self._stacked_weights(metas)
+ if w is None:
+ return core.stack(
+ [self._head_of(o)(m, co, po)
+ for o, m, co, po in zip(metas, inputs, cur_ohs, prev_ohs)],
+ axis=0)
+ w1, w2, wl = w
+ # The fc1 precision is the cast the per-head implementations use
+ # (``PrerouterHead._dtype`` / ``BailingPrerouter`` reading
+ # ``fc1.weight.dtype``); take it from the stacked weight so this
+ # stays family-agnostic.
+ dtype = w1.dtype
+ feats = core.concatenate([core.stack(inputs, 0), core.stack(cur_ohs, 0),
+ core.stack(prev_ohs, 0)], axis=-1)
+ if feats.dtype != dtype:
+ feats = feats.astype(dtype)
+ f2 = feats.reshape(len(metas), -1)
+ h1 = core.einsum("ni,nij->nj", f2, w1)
+ act = gelu_erf(h1)
+ out = (core.einsum("ni,nij->nj", f2, wl)
+ + core.einsum("ni,nij->nj", act, w2))
+ return out.reshape(len(metas), 1, 1, self.num_experts)
+
# ---- the step-boundary batch ------------------------------------------
def stage_all(self) -> dict[int, list[int]]:
@@ -112,26 +184,27 @@ class PrerouterStager:
prev_ohs.append(prev_oh)
if not metas:
return {}
- per_logits = []
- for owner, m_in, co, po in zip(metas, inputs, cur_ohs, prev_ohs):
- head = self._head_of(owner)
- per_logits.append(head(m_in, co, po))
- logits = core.stack(per_logits, axis=0) # [n,1,1,E]
+ logits = self._logits(metas, inputs, cur_ohs, prev_ohs) # [n,1,1,E]
inds_all, scores_all = self._select(logits)
- core.eval(inds_all, scores_all)
- flat = inds_all.reshape(len(metas), -1).tolist() # ONE tolist
+ flat = None
+ if self.stream_layers:
+ # Only the consumers' staged fills need host-side expert IDs.
+ # Without them this sync + tolist is pure waste (fix 2).
+ core.eval(inds_all, scores_all)
+ flat = inds_all.reshape(len(metas), -1).tolist() # ONE tolist
expert_sets: dict[int, list[int]] = {}
for i, owner in enumerate(metas):
consumer = owner + 1
- experts = sorted(set(int(v) for v in flat[i]))
- expert_sets[consumer] = experts
+ experts = (sorted({int(v) for v in flat[i]})
+ if flat is not None else None)
+ expert_sets[consumer] = experts or []
self._store(consumer, logits[i], inds_all[i],
scores_all[i], experts)
self._note(owner, cur_ohs[i])
- if self.records is not None:
- for owner, consumer in zip(metas, metas):
+ if self.records is not None and flat is not None:
+ for owner in metas:
self.records.append(
- (self.cur_step, consumer, expert_sets[consumer]))
+ (self.cur_step, owner + 1, expert_sets[owner + 1]))
self.cur_step += 1
return expert_sets
@@ -169,13 +242,13 @@ class CrossTokenStager(PrerouterStager):
return select_from_logits(logits, self.top_k)
def _store(self, consumer: int, logits_i, inds_i, scores_i,
- experts: list[int]):
+ experts: list[int] | None):
st = self.state
st.pred_inds[consumer] = inds_i
st.pred_scores[consumer] = scores_i
st.logits[consumer - 1] = logits_i
exp = self.stream_layers.get(consumer)
- if exp is not None:
+ if exp is not None and experts:
exp.stage_experts(experts)
def _note(self, owner: int, cur_oh) -> None:
@@ -227,10 +300,10 @@ class LingPrerouterStager(PrerouterStager):
routed_scaling=self.spec.routed_scaling or 1.0)
def _store(self, consumer: int, logits_i, inds_i, scores_i,
- experts: list[int]):
+ experts: list[int] | None):
self.pg_cache[consumer] = logits_i
exp = self.stream_layers.get(consumer)
- if exp is not None:
+ if exp is not None and experts:
exp.stage_experts(experts)
def _note(self, owner: int, cur_oh) -> None:
diff --git a/src/edge0/registry.py b/src/edge0/registry.py
index 1585051..d4a54a7 100644
--- a/src/edge0/registry.py
+++ b/src/edge0/registry.py
@@ -109,3 +109,20 @@ class AutoEngine:
from edge0 import models # noqa: F401
key = _resolve_name(model_dir, name)
return MODEL_REGISTRY[key].build_engine(model_dir, **kwargs)
+
+
+def demo_kwargs(model_dir=None, name=None, **kwargs) -> dict:
+ """``from_pretrained`` overrides for the demo entry points.
+
+ A tier pins its showcase configuration with the class attribute
+ ``demo_no_prerouter``: ``edge0-8b`` runs the gate-routed exact path in
+ ``edge0 demo`` / ``examples/demo.py``, because its prerouter is opt-in
+ there. Explicit keyword arguments (e.g. ``--no-prerouter``) always win.
+ """
+ from edge0 import models # noqa: F401
+ out = dict(kwargs)
+ if "prerouter" not in out:
+ adapter = MODEL_REGISTRY[_resolve_name(model_dir, name)]
+ if getattr(adapter.Config, "demo_no_prerouter", False):
+ out["prerouter"] = None
+ return out
diff --git a/src/edge0/streaming/layer.py b/src/edge0/streaming/layer.py
index a82027e..e794842 100644
--- a/src/edge0/streaming/layer.py
+++ b/src/edge0/streaming/layer.py
@@ -841,6 +841,24 @@ class StreamingSwitchGLU:
self.last_used = unique
self.stage_experts(unique)
+ def prefetch_from_prefill(self):
+ """After prefill: prefetch (do NOT stage) the last prefill token's
+ actual top-k.
+
+ Counterpart of ``stage_from_prefill`` for the case where the first
+ decode step does not consume a prerouter prediction: that step routes
+ with the router (pos-0 fallback, as in training), so filling the slots
+ from the prefill token's top-k silently zeroes every routed expert
+ outside that set. Warm the cache instead and let the step run the
+ exact path with the bundles already resident."""
+ topk = getattr(self, "_last_prefill_topk", None)
+ if not topk:
+ return
+ unique = sorted(set(int(v) for v in topk))
+ if unique:
+ self.last_used = unique
+ self.prefetch(unique)
+
def _refresh_hot_pins(self):
if self.hot_per_layer <= 0:
return
diff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.py
index 34a27a3..ff71e1a 100644
--- a/src/edge0/streaming/options.py
+++ b/src/edge0/streaming/options.py
@@ -40,7 +40,14 @@ class LayerOptions:
hot_decay: decay factor for hot-expert usage counts.
pin_bonus: extra score for prerouter-predicted experts when
selecting hot pins.
- cache_slots: shared LRU capacity across all layers.
+ cache_slots: shared LRU capacity across all layers, keyed by
+ ``(layer_idx, expert)`` with a single global budget (see
+ ``SharedExpertCache``). Size it for the WHOLE model, not per
+ layer: a decode step touches ``top_k`` experts on every MoE layer
+ (23 x 8 = 184 distinct keys on edge0-8b), so a budget smaller than
+ that cannot hold even one token's working set and the hit rate
+ collapses to ~0 (the released 64 slots = 2.8 slots/layer measured
+ exactly 0 hits, rebuilding 184 bundles/step).
prefetch_cap: prefetch buffer capacity.
load_threads: threads for on-demand expert builds.
prefetch_threads: threads for eager prefetch builds.
@@ -110,20 +117,43 @@ class LayerOptions:
@classmethod
def prod_k8(cls, **overrides) -> "LayerOptions":
- """edge0-8b tier: deployment production profile.
-
- Mirrors the reference deployment profile: STAGED_DECODE=0
- (staged decode is OFF — the deployment verified staged decode on
- ling degrades output, see the comment in start_server.sh),
- STAGED_SYNC=0, STAGED_SLOTS=8, EXPERT_CACHE_SLOTS=64,
- MLX_COMPILE_MOE=1, PREFILL_FULL_LAYER=1. The hybrid prerouter
- still stages each next token's expert set into the shared LRU
- (engine.py: stage_all at the step boundary AND right after
- prefill), and the vendored MoE blocks re-select from the cached
- prerouter logits."""
+ """edge0-8b tier: staged decode ON for the prerouter consumers.
+
+ Only the layers whose ROUTE is a prerouter prediction keep staged
+ slots (consumer ``li >= start_layer + 1``, i.e. 8-23): the stager
+ submits exactly the set the consuming block re-selects from the same
+ cached logits, so ``slots == routing`` and nothing is dropped
+ (measured ``staged_dropped = 0``; output bit-identical to the exact
+ path). The layers below ``start_layer`` route with their own gate and
+ run the exact path (``ling.load_installed`` sets
+ ``_staged_mode = False``; legacy history staging there dropped 17-26%
+ of their experts and degraded output -- opt back in with
+ ``history_slots=True`` / ``--history-slots``).
+
+ Measured on M4 Pro (same-process paired A/B): slots cut expert loads
+ from 184 to 56 per step (8-23 served entirely by the prediction), and
+ the prerouter routes ~5% faster than gate routing with the same
+ loading path.
+
+ ``cache_slots`` stays at the low-memory default 64 (~81 MiB): this
+ tier's contract is a small resident footprint and every cached bundle
+ is 1.27 MiB of RAM. The 64 is NOT enough to reuse anything across
+ tokens (2.8 slots/layer for a 184-key/step working set, measured 0
+ hits): every predicted expert whose set changed since the previous
+ token (44% of the 128 staged uses/step -- the other 56% come from the
+ previous token's slots) plus all 56 gate-routed uses on L1-7 are
+ re-sliced and re-packed every step. That is the price of the small
+ footprint, and the staged slots are how this tier avoids paying it for
+ the consumer layers. Enlarging the LRU buys speed with RAM (1024 =
+ +1.3 GiB, peak 1.4 -> 2.6 GiB -> 40.02 -> 31.44 ms/step, paired 3/3
+ rounds, output identical) -- do it only when the memory budget allows;
+ at cache_slots=64 the scoped staged path is the fastest arm measured
+ (37.99 vs 39.52 ms/step for gate routing + exact loads, 6/8 rounds) and
+ under a load-bound regime (build latency throttled to 200 us) it wins
+ 32-40% (4/4 rounds)."""
return cls(
- staged=False, staged_replace=False, staged_n=8,
- staged_trigger=8, staged_sync=False, history_prefetch=False,
+ staged=True, staged_replace=False, staged_n=8,
+ staged_trigger=8, staged_sync=True, history_prefetch=False,
hot_per_layer=0,
cache_slots=64, prefetch_cap=48,
load_threads=8, prefetch_threads=4, use_compile=True, top_k=8,
diff --git a/tests/test_registry.py b/tests/test_registry.py
index a50e2c2..8b8e321 100644
--- a/tests/test_registry.py
+++ b/tests/test_registry.py
@@ -26,11 +26,13 @@ def test_auto_config_defaults(name):
assert cfg.name == name
assert cfg.moe_spec.num_experts > 0
assert cfg.moe_spec.top_k in (4, 8)
- # qwen tier defaults to staged decode; the ling tier mirrors the
- # deployment production profile (STAGED_DECODE=0 — staged decode on
- # ling degrades output, see start_server.sh) with the prerouter
- # still staging each next token's expert set.
- assert cfg.options.staged is (name == "edge0-35b")
+ # Both tiers default to staged decode. Staging is zero-drop only for
+ # the layers whose ROUTE is a prerouter prediction (the scoped
+ # consumers, li >= start_layer + 1); every layer below start_layer
+ # runs the exact path, and the legacy history-filled staging for them
+ # is opt-in via history_slots / --history-slots (see docs/prerouter.md).
+ assert cfg.options.staged is True
+ assert cfg.history_slots is False
assert cfg.prerouter is not None
assert cfg.prerouter.weights_file.endswith(".safetensors")
assert cfg.prerouter_top_k == cfg.moe_spec.top_k
@@ -57,6 +59,10 @@ def test_qwen35_profile():
assert cfg.prerouter.hidden == 512
assert cfg.prerouter.dtype == "fp16"
assert cfg.prerouter.feature_topk == "executed"
+ # only layers whose route is a prerouter prediction keep staged slots
+ assert cfg.history_slots is False
+ assert AutoConfig.from_pretrained(
+ name="edge0-35b", history_slots=True).history_slots is True
assert cfg.gen.temperature == 0.7
assert 248046 in cfg.gen.eos_ids
assert cfg.port == 8085
@@ -89,6 +95,21 @@ def test_override_and_reject():
AutoConfig.from_pretrained(name="edge0-8b", bogus_field=1)
+def test_demo_defaults():
+ """The demo entry points run each tier's showcase configuration."""
+ from edge0.registry import demo_kwargs
+
+ # edge0-8b demos default to the gate-routed exact path (prerouter off).
+ assert demo_kwargs(name="edge0-8b")["prerouter"] is None
+ assert demo_kwargs("edge0-8b")["prerouter"] is None # model_dir form
+ # edge0-35b demos keep the tier config untouched.
+ assert "prerouter" not in demo_kwargs(name="edge0-35b")
+ # an explicit decision always wins.
+ sentinel = object()
+ assert demo_kwargs(name="edge0-8b",
+ prerouter=sentinel)["prerouter"] is sentinel
+
+
def test_unknown_name_lists_registry():
with pytest.raises(KeyError, match="edge0-35b"):
AutoConfig.from_pretrained(name="edge0-99b")