Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
fix(prerouter): zero-drop layer scoping, working --no-prerouter, 8b sync
linyubupab57d83ae2b9e
diff --git a/docs/prerouter-decode-fix-2026-09-14.md b/docs/prerouter-decode-fix-2026-09-14.mdnew file mode 100644index 0000000..713df05--- /dev/null+++ b/docs/prerouter-decode-fix-2026-09-14.md@@ -0,0 +1,418 @@+# prerouter decode 修复记录(2026-09-14)+> 说明:本文引用的临时验证脚本(`scripts/verify_*.py`、`scripts/ab_*.py`、`scripts/bench_*_ab.py`、`scripts/diag_*.py`、`scripts/quality_*.py`)是一次性的+> 测量脚手架,已在收尾时从仓库清理;复现口径见正文(配对方式、段长、轮数、环境噪声说明)。+++范围:`edge0-35b`(qwen)。8b(ling)的同类层级问题未改,见文末。+§3.6 的两个执行开销修复落在**共享**的 `prerouter/stager.py` 上,两个 tier 同时受益。++---++## 1. 修前实测到的三个问题++用真实引擎跑(2 请求 × 8 decode token + 32 token 对照),两组独立信号一致+(脚本自己比"routed vs staged 集合"和引擎内建的 `staged_dropped` 计数器)。++### P1 跨请求状态污染+- 同进程第 2 个请求的 prefill,`stage_all` 提交了 33 个消费者层,且 owner 20 的+ `prerouter_m_in` 指纹与**上一个请求最后一个 decode step 逐位相同**+ (`1398.39453125`)→ 新请求第 1 个 token 的路由,是用上一个请求的隐状态算出来的预测。+- 根因:`StreamingSwitchGLU.reset()` 文档写着 "Per-request reset … so the next request+ starts clean",但**全仓库零调用点**;`block.prerouter_m_in / prerouter_oh` 只在单 token+ 前向里写、从不清理。(参考部署 `~/Documents/qwen35-v7-deploy/streaming_experts_qwen35.py`+ 里连这个方法都没有,是 edge0 自己加的、漏接的线。)++### P2 每个请求第 1 个 token:消费者层 4/4 补零+- 首步没有预测可用(head 特征只在单 token 前向抓取),路由按训练语义回落 gate+ (pos-0 fallback),但 `_prefill_end` 仍用**最后一个 prefill token 的实际 top-k**+ 填槽位,两边几乎不重合(实测 L7:路由 `(96,132,166,182)` vs 槽位 `(14,117,184,229)`,+ 交集 0)→ 32 个消费者层各丢 4 个专家。+- 缺失专家在 staged 路径下映射到溢出零行(`streaming/layer.py:597`+ `slot_of_list = [n] * num_experts` + `_build_asm` 用 `_zero_slot` 补行):+ **静默补零,不报错、不补算**。++### P3 层 0–6 / 39:每个 token 持续丢专家+- 这 8 层没有 head 供给(覆盖是 6→7 … 38→39),每步都用 gate 路由,却仍被+ `_step_pre` 用"上一个 token 的实际集合"填槽位。+- 实测每步丢 3.0–3.75 / 4(L0 在 16 次调用里累计 60/64),8 层合计稳态+ **25.75 / 步**,占全模型 160 个 routed 槽位的 **16%**。+- 参考部署原话即为此设计:"Current router as the predictor … **No correctness handling**"+ —— 这是既定近似(省掉这些层专家的加载换速度),不是失误。++### 影响的量化(修前)+- 每步 `staged_dropped` 稳态约 25.75(16%)。+- 旧路径 vs 全精确(同 prompt、同 routing、只换加载路径):前 16 个 token 相同,+ **第 17 个 token 起分叉,64 个里 48 个不同**,`max|Δlogit|` 从 2.5 涨到 15.8。+- 两段文本都通顺 → 结论只能是"不是同一条轨迹",**不能**说"质量更差"。++---++## 2. 改动清单++| 文件 | 改动 |+|---|---|+| `src/edge0/engine/qwen.py` | `_reset_state`:清 per-block 特征捕获 + 每层 `reset()`(P1);`_prefill_end`:有 prerouter 时改调 `prefetch_from_prefill()`(P2);`load_installed`:非消费者层关闭 `_staged_mode`(P3);`_step_pre`/`_step_post`:这些层不再填槽、不再做无效 prefetch、跳过 `sync_actuals`;修正 `_forward` 里关于"首步已消费真实预测"的错误注释 |+| `src/edge0/prerouter/stager.py` | 报告 §6 的两个执行开销修复:头权重一次性 stack + 每步 3 次 einsum(`head_batch`,可关);`stream_layers` 为空时跳过无人使用的 `core.eval` + `tolist`。另修正 `records` 里 `zip(metas, metas)` 把 consumer 记成 owner 的笔误 |+| `src/edge0/streaming/layer.py` | 新增 `StreamingSwitchGLU.prefetch_from_prefill()`(只预热缓存,不填槽位)。未修改任何已有方法 |+| `src/edge0/models/base.py` | 新增 `ModelConfig.history_slots: bool = False` |+| `src/edge0/cli.py` | `--history-slots`(demo / chat / serve)+ `_engine_kwargs` 映射 |+| `docs/prerouter.md` | "zero drop" 的适用范围写准;补首步、非消费者层、per-request reset 说明 |+| `docs/models/edge0-35b.md` | staged 只覆盖 L7–L38;补 `history_slots` / `--history-slots` |+| `tests/test_registry.py` | 断言 `history_slots` 默认 False 且可覆盖 |++设计原则:**只有"路由来自预测"的层用 staged 槽位**。它的路由与槽位集合严格相同(零丢);+其余层走精确路径。++---++## 3. 验证结果(真实引擎,每配置一个进程、一个引擎)++### 3.1 prerouter 仍在被使用(新默认)+插桩读 `pg_state.pred_inds[li]` 与该层实际用的槽位 `_staged_at_use`:++| 步骤 | 有预测的消费者层 | 用了槽位的层 | 用的槽位 == 预测 |+|---|---|---|---|+| step 1 | 0/32 | 0 | 0(按设计回落 gate,走 exact) |+| step 2 | 32/32 | 32 | **32/32** |+| step 3 | 32/32 | 32 | **32/32** |+| step 8 | 32/32 | 32 | **32/32** |++预测与**原生 gate 自己的 top-4** 只在 2.2–2.6/4 上重合(集合完全相等:step2 5/32、+step3 2/32、step8 0/32)→ 路由确实由 head 改写,不是复读 gate。++### 3.2 与全精确路径等价+同 prompt、32 步 greedy:`fixed`(新默认)与 `staged=False`(全精确)+**token 完全相同、每步 logits 逐位相同**(`max|Δlogit| = 0`)。++### 3.3 退路开关+`--history-slots`(= 旧行为)逐位可复现:两次跑 token 序列完全一致,第 17 步起与精确+路径分叉(16/32 不同),`staged_dropped` = 806(32 步)。++### 3.4 速度:同进程配对测量(V1/V2/V3 部分可信;V0 对比已撤回,见下)++方法按 `docs/experiments/prerouter-decode-speed-2026-09-14.md` §2/§9.3:本机跨进程单轮 A/B+的组间差异可达 ±15%、甚至反号,所以用**同进程、交替、配对差值**。本表:一个引擎、3 臂+运行时切换(只改 `_staged_mode` 与 stager 的 `stream_layers`)、每臂先跑 12 步预热再计时+48 步、4 轮交替顺序、取中位数;`loads`/`dropped` 取层内计数器增量。++**补充教训(本轮新增)**:同进程配对也不总够 —— 当两个臂**路由到不同的专家集合**、而+LRU/页缓存是共享的时候,臂之间会互相预热,"谁先跑谁吃亏",符号可以跨进程翻转(见下)。++| 臂 | 含义 | 中位 ms/step | tok/s | loads/step | dropped |+|---|---|---|---|---|---|+| V0 | 无头(gate 路由 + 精确加载) | 56.18 | 17.82 | 160 | 0 |+| V1 | 有头不收割(预测路由 + 精确加载) | **49.13** | **20.36** | 160 | 0 |+| V2 | 有头 + 收割(= 新默认) | 50.57 | 19.78 | **32** | 0 |++配对差值(本轮):`V1 − V0 = −7.05 ms`、`V2 − V0 = −5.61 ms`、`V2 − V1 = +1.44 ms`。++⚠️ **`V0` vs `V1`(原生路由 vs 预测路由)这个对比不可用,已撤回。** 复测三次,同脚本、+同进程内配对,符号**翻转**:++| 第几次 | V0 ms/step | V1 ms/step | V2 ms/step | 结论 |+|---|---|---|---|---|+| 第 1 次 | 56.18(各轮 78.0/57.9/54.4/51.9) | 49.13 | 50.57 | V1 更快 |+| 第 2 次 | 68.61(62.4/74.1/68.6) | 49.54 | 51.66 | V1 更快 |+| 第 3 次(含 V3 异步臂) | 45.86(46.4/41.5/45.3/60.5) | 51.11 | 50.87 | **V0 更快** |++`V0` 在 45.9–68.6 ms 之间摆动,`V1` 稳定在 49.1–51.1 ms。原因:两臂**专家集合不同**+(gate vs 预测路由),而共享 LRU/页缓存被**别的臂预热过**——预测集被 V2 预热,gate 集是冷的,+于是"谁先跑谁吃亏",符号随机器状态翻转。这就是报告 §2 记的 ±15% / 符号翻转那一类陷阱。+**故不再声称 prerouter 在 35b 上快 14%;同进程共享缓存的臂切换量不了这个量。**++站得住的是**计数器级结论**(不依赖计时噪声):++| 臂 | loads/step | load_wall | stage_wall | stage_wait | 合计装载成本 |+|---|---|---|---|---|---|+| V1 有头不收割 | 160 | 9.8 ms | 0 | 0 | **9.8 ms** |+| V2 收割(`staged_sync=True`,= 默认) | 32 | 4.9 ms | 5.0 ms | 0 | **9.9 ms** |+| V3 收割(异步 `staged_sync=False`) | 32 | 5.0 ms | 4.9 ms | 0 | **9.9 ms** |++- 收割把**同一批 build 从"前向中"搬到"步边界"**:总装载时间一步没少(9.8 → 9.9 ms),+ `stage_wait=0`(消费者从未等待)。计时上 `V2−V1 = +1.44 ms`、`V3−V1 = +0.94 ms`+ (同向小亏)→ **收割在 35b/本机是搬运,不是节省**;同步/异步两档无差别。+- 结构性原因:填槽由 `stage_all` 在**前向结束时**发起,消费者在**下一个 token 一开头**+ 消费,中间没有可重叠的算力。要真正藏住装载,必须让填槽在**同一个 token 的前向期间**+ 发起(报告 §10.2 的 Fix 3:用 `after_layer_cb` 交错头的 einsum)。+- 8b 报告测得收割 +2.0 ms/步(184→56 loads/步),与本机 35b 相反 → 两端不可混用。++### 3.4a 已废弃:跨进程单趟数字(保留备查)++每配置一个进程、单趟、warmup 后计时 64 步:新默认 18.79 tok/s / 2.80 GiB / dropped 0;+旧行为(`--history-slots`)13.60 / 2.90 GiB / 806;全精确(`staged=False`)18.39 /+2.04 GiB / 0。跨进程单轮,按 §3.4 的方法学不可信,只用于"修前 → 修后"的粗判。++### 3.5 `--no-prerouter` 为什么"看起来更快"(当时的误读已修正)++那时测到 OFF 20.59 vs ON 15.41 tok/s(+34%),一度读成"prerouter 拖慢"。实际是:+`--no-prerouter` 在 35b 上**不是**"无头 + 精确加载",而是"无头 + 全部 40 层 history 槽位",+144 步里丢 15131 个槽位(≈105/步,占 160 个 routed 槽位的 66%)——**丢掉的专家既不加载+也不计算**,所以它天然少干约 2/3 的 MoE 活。这不是"省",是缺。++同工对比:这个量**本机量不了**(见 §3.4 的撤回)。本机能确定的只有:OFF 的 20.59 tok/s+是"66% 补零"换来的,不能当"原生路由 + 精确计算"的基线;公平基线是+`prerouter=None + staged=False`(跨进程单趟 18.70 tok/s,同口径粗判)。+`+84%` 出自 `paper/make_figs.py` 所引 `qwen35-v7-deploy/K8_K4_K2_TEST.md` §4.7/4.8+(`6.8 → 12.5 tok/s`)——那是**装载主导**的部署机;本机同模型 18–22 tok/s,装载只占+≈10 ms/50 ms,regime 不同,数字不可互推。++### 3.6 报告 §6 的两个执行开销修复(已落到共享 stager)++改动都在 `src/edge0/prerouter/stager.py::PrerouterStager`,两个 tier 共用:+- **fix 1 批量头**:owner 头形状一致 → 权重一次性 stack,每步 3 次 einsum,取代+ "每头 3 个小算子"的循环(33 头 ≈ 99 个小算子/步 → 3 个)。`head_batch=False`+ 或 `EDGE0_HEAD_BATCH=0` 回到循环,便于 A/B。+- **fix 2 免无谓同步**:`stream_layers` 为空(`staged=False`)时,`_select` 之后不再做+ `core.eval` + `tolist` —— 那份 host 侧专家 ID 没人消费。路由缓存照写(惰性数组即可),+ 所以 `staged=False` 与 `--no-prerouter` 都受益。++验证(35b 真实引擎):+- 同一批特征上批量头 vs 逐头循环:`max|Δlogit| = 0.000e+00`,路由集合一致。+- 同进程、**两臂都先预热**的 greedy-80:`head_batch=True` 与 `False` **token 完全一致**。+- fix 2 回归:`staged=False` 下 greedy-32 与改动前记录**逐 token 相同**,`dropped=0`。++⚠️ "先预热"不是形式主义:本机**进程内第一跑(冷缓存)与之后所有跑是两条轨迹**。+同一配置连跑 6 次:第 1 跑与第 2–6 跑在第 36 步分叉(1 个消费者层路由不同),+第 2–6 跑彼此逐位一致;先插一次预热跑后 6/6 完全一致。该现象在 default /+`head_batch=False` / `staged=False` / `history-slots` 四种配置下**表现完全一样**+(legacy 在第 22 步分叉)→ **与本次修复无关,是既有的冷/热启动差异**。+后果:拿"第一跑"和"reset 后的跑"直接比 token 会得到假分叉;同进程 greedy 对照+必须两臂都先预热。跨进程对照则是"冷 vs 冷",反而一致。++---++### 3.7 收益到底在哪:档位旋钮隔离(本轮结论,取代 §3.4 的速度论断)++对照部署参考 `~/Documents/qwen35-v7-deploy/engine_qwen.py:97-99`、+`streaming_experts_qwen35.py:292-297,486`——**跑出 +84% 的那台机器用的是**:+`cache_slots=900`、`hot_per_layer=8`、`hot_update_interval=32`、+`staged_sync` 默认 **False**(`QWEN_STAGED_SYNC=0`);而 edge0 的 35b 档+`staged_k4` 是 `cache_slots=64`、`hot_per_layer=0`、`hot_update_interval=4`、+`staged_sync=True`。++隔离实验(一个引擎一进程、先插预热跑、每轮位置轮转、各 4 轮;除最后一臂外全部+prerouter ON + 收割):++| 臂 | 配置差异 | 中位 ms/step | tok/s | loads/步 | load_ms | stage_ms | 峰值 |+|---|---|---|---|---|---|---|---|+| `sync64h0` | = 今天的默认 | 52.66 | 18.99 | 32 | 5.6 | 6.2 | 2.91 GiB |+| `async64h0` | `staged_sync=False` | 60.17 | 16.62 | 32 | 5.7 | 6.4 | 2.91 |+| `async900h0` | 再加 `cache_slots=900` | **38.98** | **25.66** | 12 | 1.8 | 3.4 | 4.05 |+| `async900h8` | 再加 `hot_per_layer=8`、interval 32 | 39.02 | 25.63 | 11 | 1.8 | 3.2 | 4.56 |+| `native900h8` | 部署档 + **无 prerouter** | 49.53(各轮 119.4/62.4/36.6/36.4) | 20.19 | 39 | 6.1 | 0 | 3.55 |++1. **真正的大头是 `cache_slots` 64 → 900**:52.66 → 38.98 ms/step(**+35% tok/s**),+ 现场装载从 140 次/步掉到 12 次/步。+2. `staged_sync=False` **单独无感**(60.17,甚至更慢);`hot_per_layer=8` 在 900 之上+ **再加无收益**(39.02 vs 38.98),只多 0.5 GiB。`predicted_experts`(预测喂钉选)+ 在 edge0 **和部署里都没被赋值**(`layer.py:138` 只声明、865-868 只读),是同一处+ 从未接线的能力,不是移植漏掉的。+3. **prerouter 的中位速度在 warm 稳态下 ≈ 0**:同档 `async900h8` 39.0 vs+ `native900h8` 36.4–36.6(后两轮,页缓存已热)。§3.4 里"部署档下 +83%"+ (pre_pin 22.4 vs nat_pin 12.2 tok/s)**作废**:那是 native 臂踩到冷轮+ (81.9/83.9 ms)的假象,本轮 4 轮轮转复测不成立。+4. prerouter 真正体现的是**稳定性**:无 prerouter 的臂 4 轮里 119.4 → 62.4 → 36.6 →+ 36.4 ms(冷/热差 3 倍以上),有 prerouter 的臂始终 37.8–42.3 ms。它把"装载暴露时间"+ 从不可控变可控——在装载本身是瓶颈且不可预测的环境(冷盘 / 网络盘 / 并发挤缓存)才+ 转化为速度,那正是部署机 6.8→12.5 tok/s 所处的区间;本机跑热后进入"装载不是瓶颈"+ 的区间,收益自然趋 0。++**本机可下的结论只有三条**:(a) 功能正确(§3.1–3.3、§3.6);(b) 收割/同步异步在本机是+搬运不是节省(§3.4 计数器);(c) 想在本机拿到真实提速,先改档位 —— `cache_slots`+64 → 900(+1.1 GiB)实测 +35%,与 prerouter 无关。"prerouter 快 x%" 只能表述为+装载主导环境下的收益,需慢存储复测(报告 §10.3)。++---++## 4. 没有碰的部分++L7–L38 的 `预测 → 槽位 → 路由` 链路、`stage_all`/`state.swap`、head 数学+(`prerouter/heads.py`)、prefill 的 E3b / 热路径、共享 LRU 与 hot pins、LoRA、量化、+serve/chat 路径、以及 ling 的 `BailingSparseMoE` 从 `pg_cache` 自选路由那条链。+`streaming/layer.py` 只有新增方法,没有改已有方法。++冒烟:`edge0 chat` 正常出文;`--no-prerouter` 正常;+`pytest tests/` 59 passed(唯一失败 `test_repo_hygiene::test_no_hardcoded_local_paths`+是仓库里既有的硬编码路径,命中清单不含本次修改的文件)。++---++## 5. ling(8b)落地(已完成)++§5 原列的三个待办项已全部处理,对应改动:++| 文件 | 改动 |+| --- | --- |+| `src/edge0/engine/ling.py` | `load_installed`:消费层作用域(`li < start_layer+1` 的层 `_staged_mode=False`,`history_slots=True` 时不作用);`_step_pre`/`_step_post`:非消费层在作用域默认下走精确路径、不做历史暂存也不做无谓的 `sync_actuals`;`_prefill_end`:有 prerouter 时 prefill 尾部 `stage_all` 供首步消费(8b 在 prefill 期就抓得到特征,不需要 35b 那样的 router 回退),只有 legacy `history_slots` 才 `stage_from_prefill` 且只给非消费层;`_reset_state`:清 `layer.m_in_cache` / `block.last_topk` / `block.prev_topk_oh` 并对所有流式层调 `reset()` |+| `src/edge0/prerouter/stager.py` | 批量头的 dtype 从 `_head_of(...)._dtype` 改为从堆叠的 `w1.dtype` 取(ling 的头是 `BailingPrerouter`,按 `fc1.weight.dtype` 定精度,没有 `_dtype` 属性 → 修前直接 `AttributeError`) |+| `src/edge0/streaming/options.py` | `prod_k8()` 默认改为 `staged=True`(分层零丢弃的收割路径) |+| `tests/test_registry.py` | 旧断言 `staged is (name == "edge0-35b")` 改为两档都 `staged=True` + `history_slots is False` |+| `docs/models/edge0-8b.md`、`docs/prerouter.md` | staged 说明改为"仅消费层 L8–23";补首步分档语义、per-request reset 字段名、`--history-slots` |++### 5.1 验证(每配置一个进程、一个引擎,脚本 `scripts/verify_ling_fix.py`)++> 本表是 `cache_slots=64`(当时默认)下测的;缓存放大后的数字见 §5.3/§5.4。++| 配置 | tok/s | 峰值内存 | dropped | loads/步 | 与 exact 逐 token 一致 |+| --- | --- | --- | --- | --- | --- |+| **default(新)** | **27.47** | 1.41 GiB | **0** | 5656(101 步 → 56/步) | **是** |+| history(legacy 全层历史暂存) | 34.58 | 1.56 GiB | 1589 | 0 | 否 |+| exact(`staged=False`,parity 基准) | 26.17 | 0.86 GiB | 0 | 18584(101 步 → 184/步) | — |+| nopg(`prerouter=None`) | 31.50 | 1.46 GiB | 10481 | 0 | 否 |++- **丢弃率**:`scripts/diag_staged_drops.py` → consumed=62976,`dropped=0`(0.00%);L1–7 每步 8 次精确 load,L8–23 全部由预测供给。+- **质量**:`scripts/quality_ab_staged.py` 4/4 prompt `identical=True, first_divergence=None`;greedy 32 步 token 序列与 exact 完全相同。+- **P1 跨请求污染已消**:同进程先跑请求 1 再跑请求 2,与全新进程只跑请求 2 相比,token 序列与首步 logits 完全一致(`p2_first=592203.875` 两边相同)。+- **速度(配对,`scripts/verify_pair_final.py`,SEG=100×10 轮)**:PG 对 gate 路由精确基线 **10/10 轮全胜,中位 −1.97 ms/步、均值 −3.15 ms/步(≈+5%)**。+- **内存**:默认路径峰值 1.41 GiB(exact 0.86 GiB;槽位 buffer + 头权重占 ≈0.55 GiB),24 GB 机器上无压力。++### 5.2 与报告 §7 的 −5.11 ms/步 的关系(口径修正)++报告 §6/§7 那个 −5.11 ms 的基线是"**无头 + gate 路由 + 精确 load**",因此它同时包含了+**路由来源差异**(预测路由 vs gate 路由)和**收割差异**。做成同路由(都跑头、都由+prerouter 供路由、都零丢弃)的干净配对后,收割本身只有 −0.19 ms/步(≈打平),+而"有 prerouter 供路由 + 收割" 对 "关掉 prerouter 用 gate + 精确" 是 −1.97 ms/步(中位)。+即:**prerouter 的收益主要在路由来源,槽位收割在本机(热缓存、load 延迟低)只值约 0–5%**。+两处数字都保留,口径已注明。++### 5.3 真正的瓶颈:全局专家 LRU 只有 64 槽(同日追加,收益比收割大一个量级)++`install.py:54` 给全部 23 个 MoE 层**共用一个** `SharedExpertCache(cache_slots)`,键是+`(layer, expert)`。`cache_slots=64` ÷ 23 层 = **每层 2.8 个槽**,而一个 decode step 要碰+**184 个不同键**(23 层 × top-8)——比单 token 的工作集还小,复用窗口短于一个 token,+命中率结构性为 0(实测 `hits=0`,每步重建 184 个 1.27 MiB 的专家包,52 µs/个)。++同进程配对(`scripts/verify_cache_pair.py`,生产路径、无 wrapper、3 轮交替、+greedy 输出三种容量下逐位相同、drops=0):++| cache_slots | ms/步 | tok/s | 命中/步 | 精确 load | stage 重建 | load_wall | stage_wall | 峰值内存 |+| --- | --- | --- | --- | --- | --- | --- | --- | --- |+| 64(原) | 40.02 | 24.99 | 0.0 | 56.0 | 56.0 | 5.21 ms | 3.05 ms | 1.39 GiB |+| **1024(新)** | **31.44** | **31.81** | 43.9 | 12.1 | 18.4 | 1.22 ms | 1.56 ms | 2.39 GiB |+| 2048 | 33.81 | 29.58 | 49.8 | 6.2 | 9.4 | 1.34 ms | 1.98 ms | 3.28 GiB |++**1024 对 64:−8.58 ms/步(−21.4%),3/3 轮全胜。** 2048 没有更多收益(复用窗口饱和),+所以 `prod_k8()` 落在 `cache_slots=1024`(≈44 槽/层、≈1.3 GiB 专家包、短上下文峰值+2.6 GiB,`peak_active_mem_mb` 1400 → 2700)。收益拆开看:L1–7(无预测、纯靠 LRU)+每步 56 次重建 → 12 次,约 4 ms;消费层那 44% 的"新预测集"重建 → 命中,约 1.5 ms。++**为什么预测到的专家也没命中**:每层下一个 token 的预测集只有约 **56%** 落在上一步那 8 个+槽位里(`stage_delta_hits`),剩下 44% 必须另找来源;而全局 64 槽连一个 token 都放不下,+所以这部分只能重建。也就是说"每层缓存 8 个"只够覆盖一步之内,跨 token 复用必须靠更大的 LRU。++prefill 未被牺牲:同进程配对(`scripts/verify_prefill_cache.py`,3101-token 提示、+3 轮交替)warm prefill 1102(64 槽)vs 1132 tok/s(1024 槽)= **+2.7%**,同上下文解码+22.6 → 26.7 tok/s(+18.3%)。README 里 8b 行按同口径(`BENCH_LONG=1`)重测为+解码 27.7–30.7 tok/s、峰值 2.4 GiB(短上下文)。++### 5.4 缓存修好之后:收割不再值钱++四臂配对(`scripts/verify_four_arm.py`,`cache_slots=1024`,SEG=100 × 8 轮交替,+四臂全部零丢弃):++| 臂 | ms/步 | tok/s | loads/步 | 命中/步 | load_wall | stage_wall | 对 OFF |+| --- | --- | --- | --- | --- | --- | --- | --- |+| OFF(无 prerouter,gate 路由 + 精确装载) | **28.40** | **35.22** | 36.0 | 148 | 2.44 ms | 0 | — |+| PGE(prerouter 路由 + 精确装载) | 29.78 | 33.58 | 26.6 | 157 | 1.86 ms | 0 | +1.38 ms(1/8 胜) |+| SYNC(prerouter + 槽位,默认) | 30.07 | 33.26 | 11.1 | 45 | 0.81 ms | 0.97 ms | +1.67 ms(**0/8 胜**) |+| ASYNC(`staged_sync=False`) | 30.41 | 32.88 | 10.6 | 45 | 0.76 ms | 1.01 ms | +2.02 ms(0/8 胜) |++结论:缓存能命中以后,整条精确路径的装载只剩 2.44 ms/步,而跑一趟头(批量 einsum ++`core.eval` + host sync)≈1.4 ms/步——没有足够多的装载可省了。**prerouter 的剩余价值在+"路由来源"(预测驱动选择,即发布行为),不在槽位收割**;`--no-prerouter` 在本机快约 5%,+但改变路由/输出,属于模型语义选择,不由我改默认。++### 5.5 未做++- 8b 的**质量评测**(题库/人工评分)仍未做:现有的"质量"结论是同 prompt 下与精确路径+ 逐位一致(token-level parity),不是 benchmark 分数。+- `pytest tests/` 59 passed;唯一失败 `test_repo_hygiene::test_no_hardcoded_local_paths`+ 的命中现在只剩环境文件(`artifacts/html-deleted-20260911.txt`、`.venv-mlx0305/pyvenv.cfg`、+ `paper/.tools/**`),仓库源码与 `scripts/` 下的路径已清干净。+- `staged_k4()`(35b,40 层 × 256 专家)与 `staged_k8()` 仍是 `cache_slots=64`:同样的+ 超订问题更严重(35b 一步要碰 40×4=160 个键),但改 35b 属于另一个 tier,未动。++---++## 6. 第二轮(干净 clone `edge0_new`,HEAD `ae1ee2d`):开关语义 + ling 同步 + 四组实测++### 6.1 开关修复:关掉 prerouter 必须是精确路径++`--no-prerouter` 原来在**两个 tier 上都会留下 staged 槽位**(`options.staged=True` 时+`stream_layers` 仍含所有层),于是关掉预测头后走的不是精确路径,而是"上一 token 实际+top-k 填槽"的历史路径:edge0-8b 一次请求实测 `staged_dropped = 3593`;35b 早先+"`--no-prerouter` 快 34%"也是同一原因(它少算了约 2/3 的专家)。§3.5 的修正说的就是这件事,+但当时只改了消费层作用域、没修"关"这一侧。++修法(`src/edge0/engine/{ling,qwen}.py::load_installed`)——作用域统一成一条规则:+**槽位只给"路由来自预测"的层**(consumer `start_layer+1` 起);`prerouter is None`+时没有任何预测,所以**没有任何层保留槽位**,走精确装载。`history_slots=True`+(`--history-slots`)是显式 opt-in 回旧行为。++| 配置 | staged 层 | `staged_dropped` |+|---|---|---|+| edge0-8b prerouter ON(作用域默认) | 16(L8–23) | 0 |+| edge0-8b `--no-prerouter`(修复后) | **0** | **0** |+| edge0-35b prerouter ON | 32(L7–38) | 0 |+| edge0-35b `--no-prerouter`(修复后) | **0** | **0** |++开关只有一个:`--no-prerouter`(demo / chat / serve 三处同名同义),没有反向 flag。++### 6.2 ling(edge0-8b)同步 P1/P2/P3++`src/edge0/engine/ling.py` 做了与 35b 同口径的三件事:`_reset_state` 清+`layer.m_in_cache` / `block.last_topk` / `block.prev_topk_oh` + 每层 `reset()`(P1);+`_prefill_end` 有 prerouter 时只 `stage_all()`,历史填槽只作用于非消费层且仅+`history_slots=True` 时(P2);`_step_pre`/`_step_post` 按 `first_pg_consumer`+作用域化,非消费层不填槽、不做无效 prefetch(P3)。`prod_k8()` 默认 `staged=True`+(`cache_slots=64` 保持低内存档;收割关掉时预测不参与装载,收益结构性为 0)。++### 6.3 四组实测(同机、先预热、每轮轮转、3 轮取中位)++| 臂 | ms/步 | tok/s | MLX 峰值 | 进程 RSS | staged 层 | dropped |+|---|---|---|---|---|---|---|+| qwen prerouter ON | 49.06 | 20.38 | 2.89 GiB | 11.69 GiB | 32/40 | 0 |+| qwen `--no-prerouter` | 50.21 | 19.92 | 2.07 GiB | 11.78 GiB | 0/40 | 0 |+| ling prerouter ON | **35.66** | **28.04** | 1.48 GiB | 4.94 GiB | 16/23 | 0 |+| ling `--no-prerouter` | 43.99 | 22.73 | 1.03 GiB | 4.86 GiB | 0/23 | 0 |++- **速度**:ling 开 prerouter **快 8.33 ms/步(+23%,配对 3/3 轮一致 −6.0/−9.3/−8.6)**;+ qwen 两边打平(−1.15 ms,配对 −11.6/+1.2/−3.0),与 §5.4 结论一致(35b 本机装载不是瓶颈)。+- **内存**:prerouter 多占 +0.82 GiB(35b)/ +0.45 GiB(8b)MLX 峰值(槽位 + 头);+ 进程 RSS 基本持平(11.69 vs 11.78 / 4.94 vs 4.86 GiB)。+- **质量**:收割开 vs 关(prerouter 都开,只有装载调度不同)**token 逐位一致**——+ qwen 两个 prompt 各 32 token IDENTICAL;ling 同样 IDENTICAL。+- **跨请求重置**:同进程 req1→req2 与冷进程单跑 req2 的 token 序列 IDENTICAL;+ ling 首步 logits digest 完全相同(−390314.312)。qwen 的 digest 有数值级差异+ (−307200.0 vs −299008.0)但 32 token 全同,属"进程首跑与后续跑存在数值抖动"这一+ 既有现象,不是跨请求残留。+- 与 no-prerouter 对比时 token 会不同(路由来源从 gate 换成预测头,属模型语义,不是回归);+ 两边文本都通顺。++### 6.4 ling 的 demo 默认++`Ling8BConfig.demo_no_prerouter = True` + `edge0.registry.demo_kwargs()`:+`edge0 demo` / `examples/demo.py` 对 edge0-8b 默认走 gate 路由的精确路径;+`chat` / `serve` 保持档位默认(预测路径)。冒烟(各 8 token):`demo`(ling) 无+`prerouter installed` 行、`demo --no-prerouter` 无、`chat`(ling) 有、+`chat --no-prerouter` 无、`examples/demo.py`(ling) 无。+`pytest tests/` = 65 passed, 1 skipped。++### 6.5 逐缺陷复核(同一 clone 上的实测签名)++| 缺陷 | 判据 | qwen(35b) | ling(8b) |+|---|---|---|---|+| **P1 跨请求残留** | 同进程 req1→req2 的 token 序列 vs 冷进程单跑 req2 | 32 token 全同(首步 logits digest 有数值级差异,属"进程首跑"既有抖动) | 32 token 全同,digest 完全相同(−390314.312) |+| **P2 首步** | prefill 结束有没有把"最后一个 prefill token 的实际 top-k"当预测填槽;首步是否零专家 | `prefetch_submitted=128`、`stage_builds=0`(改为预取预测集,不填槽);首步 `staged_used=0`、`staged_fallback=32`(32 层走精确路径)、`staged_dropped=0` | `stage_builds=128`(预测在 prefill 就绪);首步 `staged_used=128`、`staged_fallback=0`、`staged_dropped=0` |+| **P3 非消费层丢专家** | staged 层集合 + `staged_dropped` | L7–38(32 层),L0–6/L39 不填槽,`dropped=0` | L8–23(16 层),L1–7 不填槽(owner 6 不存在 ⇒ L7 永远 gate,已知边界),`dropped=0` |++两个 tier 的"首步正确签名"本来就不同:qwen 的特征只在单 token 前向捕获,首步按训练态走+gate(pos-0 fallback)并由 prefill 预取预测集;ling 的特征在 prefill 内就绪,首步直接消费+预测。这是数据流决定的,不是两个 tier 不一致。++诚实附注:qwen 首步预取命中仅 5/128(首步路由是 gate、预取的是预测集,重合度低),即首步+基本还是按需装载 —— 正确性优先的取舍,也是 35b 在本机 prerouter 提速 ≈0 的原因之一。++++diff --git a/docs/prerouter-switch-semantics-2026-09-14.md b/docs/prerouter-switch-semantics-2026-09-14.mdnew file mode 100644index 0000000..e465cc1--- /dev/null+++ b/docs/prerouter-switch-semantics-2026-09-14.md@@ -0,0 +1,146 @@+# prerouter 开关语义与生效判据(edge0-8b / ling)++> 只看一件事:**开关打开时必须真的走 prerouter,关掉时必须真的完全不走**。+> 本文不讨论收益、不提速、不比快慢;只定义"开/关"的语义、生效的判据、以及+> 会让开关名不副实的故障。+>+> 日期:2026-09-14;对象:`edge0-8b`(Ling 3.0 hybrid);代码以当前工作树为准。++---++## 1. 有三个不同层次的"开关",先别混++| 层次 | 开关 | 位置 | 作用 |+|---|---|---|---|+| 总开关 | `--no-prerouter`(默认**开**) | `src/edge0/cli.py:65-66` → `kw["prerouter"] = None` | 决定是否构建/使用预测头 |+| 模型内派生 | `prerouter_enabled` | `src/edge0/engine/ling.py:53`(`cfg.prerouter is not None`) | 传给 vendored 模型,逐层派生下面两个标志 |+| 逐层角色 | `has_prerouter`(本层**有头**,预测下一层)<br>`use_prerouter`(本层**消费**预测) | `backends/mlx/_impl/bailing_hybrid.py:712-724` | 决定谁跑头、谁从预测里选专家 |+| 装载侧 | `staged` / `staged_n` / `staged_sync` | `src/edge0/streaming/options.py`(`prod_k8()`) | 是否把预测出的专家**提前装进槽**;**不改变路由来源** |++逐层派生规则(8b 的 `start_layer=7`, 24 层):++```python+has_prerouter = enabled and li >= start_layer - 1 and li < n_layers - 1 # li in [6, 22]+use_prerouter = enabled and li >= start_layer and li < n_layers # li in [7, 23]+```++- **谁提供预测**:`owners`,显式写死在 `src/edge0/models/edge0_8b/__init__.py:53`+ → `owners=tuple(range(7, 23))` = **16 个头(7..22)**,owner `li` 预测 `li+1`。+ 实测烘好的头文件 `~/Documents/edge0-8b/prerouter_edge0_8b.safetensors`:48 个数组+ (每头 `fc1/fc2/linear_init`),layers = 7..22,元数据 `owners=[7..22]`。+- **谁消费预测**:`use_prerouter` → L7..23;实际有供给的是 **L8..23**(见 §7 边界)。+- stager 只遍历 `state.owners`(`src/edge0/prerouter/stager.py:176`),头集合由 `owners` 决定,+ 与 `has_prerouter` 的派生范围无关。++**路由判定点**(唯一一处,`bailing_hybrid.py:682-693`):++```python+if self.use_prerouter and prev_prerouter_logits is not None:+ idx, w = self._select_from_logits(prev_prerouter_logits) # ON:路由来自预测+else:+ idx, w = self.gate(x) # 否则:本层自己的 gate+```++`prev_prerouter_logits` 在模型前向循环里取值(`bailing_hybrid.py:808-813`):++```python+if layer.use_prerouter and prerouter_cache is not None and li in prerouter_cache:+ prev_prerouter_logits = prerouter_cache[li]+```++→ **注意这三个条件是与关系:标志为真但缓存里没有这一层的条目时,会静默回退到 gate**+(不报错、不打日志)。这是开关最容易名不副实的地方,见 §4。++---++## 2. 开:必须成立的五个条件++| # | 条件 | 代码位置 | 判据(怎么看出真的开了) |+|---|---|---|---|+| 1 | 头被构建:`owners` 16 个 | `ling.py:95-105`(`install_prerouter` / `len(heads)`) | 启动打印 `prerouter installed: 16 heads, start=7, K=8` |+| 2 | 每个 consumer 每步都拿到预测:`pg_cache[li]` 已写入 | `stager.py:302-308`(`_store`) | `pg_cache` 条目数 == 有供给的 consumer 数(8..23 = 16) |+| 3 | 首步就有预测(消费层从 token 1 就消费) | `ling.py` 的 `_prefill_end` → `stage_all()` | 第一步输出与 gate 路由不同;`consumed` 从第 1 步起增长 |+| 4 | 每个请求重置干净 | `ling.py` 的 `_reset_state`(`pg_cache` + `m_in_cache`/`last_topk`/`prev_topk_oh`) | 同一进程连续两请求 == 冷进程单跑(见 §5) |+| 5 | 装载侧作用域正确:只有 `li >= start_layer + 1` 用槽 | `ling.py:78-80` 作用域 `_staged_mode` | 消费层 `slots == routing`;`dropped == 0` |++条件 3 的特征来源(`stager.py:276-295`(`_features`)):`layer.m_in_cache`(本层 post-attention 后的+MLP 输入,取最后位置)+ `block.last_topk` 的 one-hot(当前 token)+ `block.prev_topk_oh`+的 one-hot(上一 token);prefill 结束的首次 staging 用倒数第 2 个位置当"上一 token"。++---++## 3. 关:`--no-prerouter` 必须做到的全停清单++`--no-prerouter` → `prerouter=None` → `prerouter_enabled=False` → 两个派生标志全为 False。++| 必须为"零"的项 | 位置 | 判据 |+|---|---|---|+| 不构建任何头 | `ling.py:95-105`(`if cfg.prerouter and ...`) | 启动**没有** `prerouter installed` 那行 |+| stager 不存在 | `ling.py:95`(`pg_state = pg_stager = None`) | 无 `pg_cache` 写入 |+| 前向缓存传 None | `ling.py:211`(`prerouter_cache=self._pg_stager.pg_cache` 只在存在时) | 模型循环里 `prerouter_cache is None` → 永不取值 |+| 逐层 `use_prerouter=False` | `bailing_hybrid.py:717-724` | 所有 MoE 层走 `self.gate(x)` |+| 无头前向、无预测槽装载 | 头不存在 → 无从执行 | 头/预测相关计数全 0 |+| 不保留任何 staged 槽位 | `ling.py:80-89`(`prerouter is None` ⇒ 全部层 `_staged_mode=False`) | `stream_layers` 为空、`staged_dropped == 0`、装载按需发生(`loads` 仍 > 0) |++**关的等价物**:与 `exact`(精确装载)档在路由来源上一致 —— 两者都是 gate 路由,+且(修复后)都不保留 staged 槽位,即 `--no-prerouter` **就是**精确路径本身。+修复前它反而会留下全部层的槽位(走历史路径,见 §4"关掉仍填槽"一行)。++---++## 4. 六种"开关开着/关着但没生效"的故障(均已修复,列出以防回归)++| 故障 | 现象 | 根因 | 修法 |+|---|---|---|---|+| **静默回退** | 部分层悄悄用 gate,得到既不 ON 也不 OFF 的**第三态**(混合路由) | `pg_cache` 缺该层条目时 `if` 条件不成立,无日志无异常(`bailing_hybrid.py:808-813`) | 必须保证"每个有供给的 consumer 每步都有条目";用 `consumed/dropped` 判据守住 |+| **首步被覆盖** | 第一步路由错一位 | `_prefill_end` 里 `stage_from_prefill()` 用"最后一个 prefill token 的实际 top-k"顶掉了刚算出的预测 | 有 prerouter 时只跑 `stage_all()`,不再 `stage_from_prefill` |+| **跨请求残留** | 第 2 个请求第一步用第 1 个请求的 hidden state 跑头 | `_reset_state` 未清 `layer.m_in_cache` / `block.last_topk` / `block.prev_topk_oh` | 逐 owner 清这三个字段 + 清 `pg_cache` |+| **非消费层错位** | L1-7 丢 17-26% 真实专家(`history_slots` 档实测 `dropped=1589`),输出劣化 | L1-7 没有预测,却用"上一 token 实际 top-k"填槽并当预测消费 | 作用域 `_staged_mode`:只有 `li >= start_layer+1` 走槽 |+| **关掉仍填槽** | 关掉 prerouter 后走的不是精确路径,而是同一条"上一 token 实际 top-k"历史路径(8b 实测 `dropped=3593`;35b 少算约 2/3 专家),输出劣化 | 作用域只对"非消费层"关槽(`li < first_pg_consumer`),而 `prerouter=None` 时 `first_pg_consumer=0` → 一层都没关 | `prerouter is None` ⇒ 全部层 `_staged_mode=False`(精确装载);`history_slots=True` 才回旧行为 |+| **头批次崩溃** | 开了直接 `AttributeError`(等于开关不可用) | 批量头路径读 `_head_of(...)._dtype`,而 ling 的头是 `BailingPrerouter`(无该属性) | 从堆叠后的权重取 dtype(`w1.dtype`) |++判定口径:`dropped` = 槽里装了、但该层路由没用上的专家数(应当恒为 0);+`consumed` = 槽被消费的次数。++---++## 5. 生效验证(实测)++| 检查项 | 结果 | 含义 |+|---|---|---|+| 槽位与路由一致性 | `consumed=62976`,`dropped=0`,rate 0.00% | 预测装进去的专家 100% 被该层路由消费,无错位 |+| 跨请求重置 | 同进程(请求 1 后跑请求 2)== 冷进程(只跑请求 2):token 序列一致,首步 logits digest 均为 `p2_first=592203.875` | 条件 4 成立,无残留污染 |+| 与精确路径 parity | 4/4 prompt `identical=True, first_divergence=None`;greedy-32 token 序列一致 | ON 的价值路径与"精确路径"逐 token 相同 |+| 四档对照(路由来源 / `dropped` / parity) | `default`(预测路由)= 0 / True;`history`(L1-7 也用历史槽)= 1589 / False;`exact`(无 prerouter,gate 路由)= 0 / 不适用;`nopg`(预测算了但不消费)= 10481 / False | 只有"预测路由 + 作用域正确"这一档是生效态 |++---++## 6. 30 秒自检++```bash+# 开(默认):应打印 prerouter installed: 16 heads, start=7, K=8+edge0 chat --name edge0-8b --model-dir ~/Documents/edge0-8b \+ --prompt "用一句话说明什么是MoE。" --max-new 32++# 关:不应出现上面那行+edge0 chat --name edge0-8b --model-dir ~/Documents/edge0-8b --no-prerouter \+ --prompt "用一句话说明什么是MoE。" --max-new 32+```++判定:**开**=有 `installed` 行 + `dropped == 0` + 输出与 `--no-prerouter` 不同;+**关**=无 `installed` 行 + 无 `pg_cache` 写入 + 输出等于 gate 路由(`exact` 档)。++---++## 7. 已知边界(不是 bug,是配置事实,但会被误读成"开关没生效")++1. **L7 是"有消费无供给"的缝隙。** consumer 从 `start_layer=7` 起算(L7..23),而 `owners`+ 显式从 7 开始(预测 8..23)。因此 **L7 永远拿不到预测、永远走 gate**;`has_prerouter`+ 的派生规则会从 `start_layer-1 = 6` 起算,但 8b 的显式 `owners` 覆盖了它(没有 owner 6,+ 烘好的文件里也确实没有 layer 6)。想消除这条缝隙要么把 `start_layer` 写 8,要么让+ `owners` 含 6,属于配置决策。+2. **16 个头 ≠ 22。** `src/edge0/models/edge0_8b/__init__.py:10-13` 的文件头注释写的是+ "22 heads (explicit owners 1..22), start_layer 1",与同文件 `:49-58` 的 spec+ (`start_layer=7, owners=7..22`)和烘好的 artifact 元数据(`owners=[7..22]`,48 个数组、+ layers 7..22)**都不一致**。那行注释描述的是 start-1 变体;按 §1 的代码事实,正确写法是+ **16 heads(owners 7..22)、start_layer 7**。diff --git a/examples/demo.py b/examples/demo.pyindex e4e0641..686700f 100644--- a/examples/demo.py+++ b/examples/demo.py@@ -10,6 +10,11 @@ Run (checkpoint path or registered tier name):Three lines do everything: build the engine (prerouter + LoRA + SSDoffload installed automatically from the tier config), template theprompt, generate.++The demo runs each tier's showcase configuration: ``edge0-8b`` defaults+to the gate-routed exact path (``prerouter=None``). Use ``edge0 chat``+(or ``AutoEngine.from_pretrained(..., prerouter=...)``) for the+prediction path."""from __future__ import annotations@@ -17,7 +22,7 @@ from __future__ import annotationsimport argparseimport sys-from edge0 import AutoEngine+from edge0 import AutoEngine, demo_kwargsfrom edge0.server.chat import ChatMessage, ChatRequest, ChatSession@@ -30,9 +35,14 @@ def main() -> int:ap.add_argument("--prompt",default="Hello! Write one short sentence about the seaside.")ap.add_argument("--max-new", type=int, default=24)+ ap.add_argument("--no-prerouter", action="store_true",+ help="route with the layers' own gates (edge0-8b demos "+ "already run that path)")args = ap.parse_args()- engine = AutoEngine.from_pretrained(args.model_dir, name=args.model)+ kw = ({"prerouter": None} if args.no_prerouter+ else demo_kwargs(args.model_dir, args.model))+ engine = AutoEngine.from_pretrained(args.model_dir, name=args.model, **kw)req = ChatRequest(model=engine.name,messages=[ChatMessage(role="user", content=args.prompt)],diff --git a/src/edge0/__init__.py b/src/edge0/__init__.pyindex 432e6ca..bf81bd0 100644--- a/src/edge0/__init__.py+++ b/src/edge0/__init__.py@@ -23,13 +23,14 @@ __all__ = ["AutoConfig","AutoModel","AutoEngine",+ "demo_kwargs",]def __getattr__(name: str):# Deferred imports so `import edge0` stays light and backend selection# (EDGE0_BACKEND) happens on first use.- if name in ("AutoConfig", "AutoModel", "AutoEngine"):+ if name in ("AutoConfig", "AutoModel", "AutoEngine", "demo_kwargs"):from edge0 import registry as _regreturn getattr(_reg, name)raise AttributeError(f"module {__name__!r} has no attribute {name!r}")diff --git a/src/edge0/cli.py b/src/edge0/cli.pyindex cefa632..0e025dc 100644--- a/src/edge0/cli.py+++ b/src/edge0/cli.py@@ -66,6 +66,8 @@ def _engine_kwargs(args) -> dict:kw["prerouter"] = Noneif getattr(args, "no_lora", False):kw["lora"] = ""+ if getattr(args, "history_slots", False):+ kw["history_slots"] = Truereturn kw@@ -122,14 +124,19 @@ def _display_text(text: str, show_thinking: bool) -> str:def cmd_demo(args) -> int:from edge0 import AutoEngine+ from edge0.registry import demo_kwargsfrom edge0.server.chat import ChatMessage, ChatRequest, ChatSessionmodel_dir, name = _resolve_model(args)if not model_dir or not os.path.isdir(model_dir):print(_missing_model_help(name), file=sys.stderr)return 2- engine = AutoEngine.from_pretrained(model_dir, name=name,- **_engine_kwargs(args))+ kw = _engine_kwargs(args)+ if "prerouter" not in kw:+ # No explicit --no-prerouter: the tier's demo default applies+ # (edge0-8b demos run the gate-routed exact path).+ kw = demo_kwargs(model_dir, name, **kw)+ engine = AutoEngine.from_pretrained(model_dir, name=name, **kw)tok = engine._tokif tok is None:engine.close()@@ -232,6 +239,10 @@ def main(argv: list[str] | None = None) -> int:help="print the model's reasoning block too")p.add_argument("--no-prerouter", action="store_true")p.add_argument("--no-lora", action="store_true")+ p.add_argument("--history-slots", action="store_true",+ help="legacy staging: also fill staged slots from history "+ "for layers whose route is not a prerouter prediction "+ "(zeroes routed experts outside the slot set)")p.set_defaults(fn=cmd_demo)p = sub.add_parser(@@ -248,6 +259,10 @@ def main(argv: list[str] | None = None) -> int:help="print the model's reasoning block too")p.add_argument("--no-prerouter", action="store_true")p.add_argument("--no-lora", action="store_true")+ p.add_argument("--history-slots", action="store_true",+ help="legacy staging: also fill staged slots from history "+ "for layers whose route is not a prerouter prediction "+ "(zeroes routed experts outside the slot set)")p.set_defaults(fn=cmd_chat)p = sub.add_parser(@@ -263,6 +278,10 @@ def main(argv: list[str] | None = None) -> int:help="use the Flask transport (needs flask installed)")p.add_argument("--no-prerouter", action="store_true")p.add_argument("--no-lora", action="store_true")+ p.add_argument("--history-slots", action="store_true",+ help="legacy staging: also fill staged slots from history "+ "for layers whose route is not a prerouter prediction "+ "(zeroes routed experts outside the slot set)")p.set_defaults(fn=cmd_serve)p = sub.add_parser("convert-adapters",diff --git a/src/edge0/engine/ling.py b/src/edge0/engine/ling.pyindex b8a0344..87229b9 100644--- a/src/edge0/engine/ling.py+++ b/src/edge0/engine/ling.py@@ -64,6 +64,32 @@ def load_installed(model_dir: str, cfg):installed = install_streaming_experts(model, shards, spec, options=opts, num_layers=n_layers)all_stream = {li: t for li, t in enumerate(installed) if t is not None}+ # Zero-drop layer scoping (quality invariant): staged slots are only+ # coherent for layers whose ROUTE comes from the prerouter (consumer+ # li >= start_layer + 1: owner li-1 predicts li, and the consuming+ # block re-selects from the SAME cached logits the stager staged+ # from — set == routing, drops == 0, output bit-parity with the+ # exact path). Layers at/below start_layer route with their own+ # gate every token; their slots cannot be predicted, so they run+ # the exact path. History-staging them instead (qwen-style+ # actuals-as-prediction) misaligns one step behind the gate and+ # measurably degrades output (17% drop rate, garbled text).+ first_pg_consumer = (+ getattr(cfg.prerouter, "start_layer", 7) + 1+ if cfg.prerouter else 0)+ if not getattr(cfg, "history_slots", False):+ # Scoped default: only the consumers keep staged slots. Switch-off+ # takes the same rule to its limit -- with ``--no-prerouter`` there is+ # no prediction anywhere, so NO layer keeps slots and the off path is+ # the exact path (gate routing, on-demand loads), not history staging+ # (measured ``staged_dropped = 3593`` over a whole run when the slots+ # stayed on with the prerouter off). Legacy ``history_slots=True``+ # deliberately leaves every layer staged (slots filled from the+ # previous token's actuals), which is the behaviour that silently+ # zeroes the routed experts outside the slot set.+ for li, t in all_stream.items():+ if cfg.prerouter is None or li < first_pg_consumer:+ t._staged_mode = Falsestream_layers = {li: t for li, t in all_stream.items() if t._staged_mode}if cfg.lora:@@ -83,6 +109,7 @@ def load_installed(model_dir: str, cfg):f"K={cfg.prerouter_top_k}", flush=True)installs = dict(all_stream_layers=all_stream,stream_layers=stream_layers,+ first_pg_consumer=first_pg_consumer,pg_state=pg_state, pg_stager=pg_stager)return model, model_config, shards, installs@@ -112,6 +139,7 @@ class Ling8BEngine(Edge0Engine):inst) = load_installed(cfg.model_dir, cfg)self._all_stream_layers = inst["all_stream_layers"]self._stream_layers = inst["stream_layers"]+ self._first_pg_consumer = inst["first_pg_consumer"]self._pg_state = inst["pg_state"]self._pg_stager = inst["pg_stager"]opts = cfg.options@@ -188,7 +216,10 @@ class Ling8BEngine(Edge0Engine):if self._pg_stager is not None else None))logits = self.model.lm_head(h[0, -1])core.eval(logits)- # Step boundary: ONE stacked head batch -> ONE tolist -> fills.+ # Step boundary: ONE head batch (stacked einsum) -> fills. The+ # host-side tolist only happens when a consumer actually needs the+ # expert IDs (``stream_layers`` non-empty); with staged loading off+ # the routing cache is filled lazily and the sync is skipped.if (self._pg_stager is not None and not self._prefill_activeand len(ids) == 1):self._pg_stager.stage_all()@@ -197,19 +228,46 @@ class Ling8BEngine(Edge0Engine):# ---- step hooks -------------------------------------------------------def _step_pre(self, token_id: int) -> None:- if self._pg_stager is None and self._history_prefetch is not None:+ if self._pg_stager is not None:+ # Legacy ``history_slots=True``: the layers without an incoming+ # prediction (li < start+1) are still staged, so fill them from+ # their previous actuals. In the scoped default they are+ # ``_staged_mode=False`` and run the exact path here -- no+ # staging and no prefetch (the previous token's set covers only a+ # small fraction of the next route, so the reads are wasted).+ for li, exp in self._all_stream_layers.items():+ if li < self._first_pg_consumer:+ if exp._staged_mode and exp.last_used:+ exp.stage_experts(list(exp.last_used))+ elif self._history_prefetch is not None:self._history_prefetch()def _step_post(self, token_id: int) -> None:+ if self._pg_stager is not None:+ # Consumers: the prerouter's ``stage_all`` already submitted+ # their next set; only record actuals (drop accounting + hot+ # pins). Non-consumers run the exact path in the scoped default+ # (``_staged_mode=False``), where this bookkeeping is dead+ # weight; legacy ``history_slots`` stages them from their+ # actuals like the no-prerouter path.+ for li, exp in self._all_stream_layers.items():+ if li < self._first_pg_consumer:+ if not exp._staged_mode:+ continue+ exp.sync_actuals()+ if exp.last_used:+ exp.stage_experts(list(exp.last_used))+ exp.swap_staged()+ elif exp._staged_mode:+ exp.sync_actuals()+ return+ # No prerouter: history staging -- each layer's next set is its+ # previous actuals (adjacent-token expert locality).for exp in self._stream_layers.values():exp.sync_actuals()- if self._pg_stager is None:- # History staging: each layer's next set is its previous- # actuals (adjacent-token expert locality).- for exp in self._stream_layers.values():- if exp.last_used:- exp.stage_experts(list(exp.last_used))- exp.swap_staged()+ if exp.last_used:+ exp.stage_experts(list(exp.last_used))+ exp.swap_staged()# ---- prefill / reset --------------------------------------------------@@ -218,21 +276,48 @@ class Ling8BEngine(Edge0Engine):exp.clear_full_layer()if self._pg_stager is not None:# Deployment parity: prefill's tail stages the FIRST decode- # token's expert sets (engine.py:919 calls stage_all right- # after prefill). Without this the first decode step has no- # prerouter logits, falls back to the true gate for one step,- # and the whole prediction chain shifts by one token- # (observed: [23982, 2862, ...] vs correct [23982, 4264, ...]).+ # token's expert sets for the prerouter consumers (engine.py:919+ # calls stage_all right after prefill). Without this the first+ # decode step has no prerouter logits, falls back to the true+ # gate for one step, and the whole prediction chain shifts by one+ # token (observed: [23982, 2862, ...] vs correct [23982, 4264, ...]).self._pg_stager.stage_all()- if self._stream_layers:+ if getattr(self.cfg, "history_slots", False):+ # Legacy: the non-consumer layers are staged too and get the+ # last prefill token's actual top-k. MUST NOT touch the+ # consumers -- ``stage_from_prefill`` would overwrite the+ # prerouter's submissions and misalign the first step.+ for li, exp in self._stream_layers.items():+ if li < self._first_pg_consumer:+ exp.stage_from_prefill()+ elif self._stream_layers:+ # No prerouter: history staging is the only predictor (legacy).for exp in self._stream_layers.values():exp.stage_from_prefill()def _reset_state(self) -> None:self.cache = self.model.make_cache()- if self._pg_stager is not None:+ if self._pg_state is not None:self._pg_stager.reset()self._pg_state.reset()+ # Drop the per-block/per-layer feature captures as well, or the+ # NEXT request's first ``stage_all`` runs the heads on this+ # request's last token: ``prev_topk_oh`` is only ever written by+ # ``_note`` at the step boundary, so it would feed the previous+ # request's last top-k as the "prev token" feature and stage+ # stale predictions that the first decode step then routes with.+ for owner in self._pg_state.owners:+ block = self.cfg.moe_spec.block_of(self.model, owner)+ layer = self.cfg.moe_spec.layer_of(self.model, owner)+ block.last_topk = None+ block.prev_topk_oh = None+ layer.m_in_cache = None+ # Per-request reset of the streaming layers' staged state: a leftover+ # ``_staged_state`` from the previous request defeats the staged+ # path's "fill not ready -> exact path" fallback (``st`` would never+ # be None) and would serve the previous request's expert bundles.+ for exp in getattr(self, "_all_stream_layers", {}).values():+ exp.reset()def _lm_logits(self, h: core.array) -> core.array:return self.model.lm_head(h[0, -1])diff --git a/src/edge0/engine/qwen.py b/src/edge0/engine/qwen.pyindex 12e0953..96ce7f2 100644--- a/src/edge0/engine/qwen.py+++ b/src/edge0/engine/qwen.py@@ -60,6 +60,31 @@ def load_installed(model_dir: str, cfg):installed = install_streaming_experts(model, shards, spec, options=opts, num_layers=n_layers)all_stream = {li: t for li, t in enumerate(installed) if t is not None}+ # Zero-drop layer scoping (quality invariant). Staged slots are only+ # coherent for a layer whose ROUTE is a prerouter prediction (consumer+ # ``start_layer+1 .. n_layers-2``): the head's submission is exactly the+ # set that consumer itself re-selects from the same cached state, so+ # slots == routing and ``staged_dropped == 0``. Every other layer routes+ # with its own gate, and slots filled from history (the previous token's+ # actuals) silently zero every routed expert outside that set -- measured+ # 3.0-3.75 of 4 per step for L0-6/L39, and the same class of misalignment+ # in the 8b tier. Those layers run the exact path (``_step_pre`` /+ # ``_step_post``) instead.+ #+ # Switch-off is the same rule taken to its limit: with+ # ``--no-prerouter`` there is no prediction anywhere, so no layer keeps+ # slots and the off path is the exact path (gate routing, on-demand+ # loads), not history staging. ``history_slots=True`` opts back into the+ # legacy behaviour on purpose (``--history-slots``).+ if not getattr(cfg, "history_slots", False):+ if cfg.prerouter is None:+ for t in all_stream.values():+ t._staged_mode = False+ else:+ lo, hi = cfg.prerouter.start_layer, n_layers - 2+ for li, t in all_stream.items():+ if not lo <= li <= hi:+ t._staged_mode = Falsestream_layers = {li: t for li, t in all_stream.items() if t._staged_mode}if cfg.lora:@@ -151,11 +176,11 @@ class Qwen35Engine(Edge0Engine):# Step boundary: ONE stacked head batch -> ONE tolist -> fills.# Demo parity (engine_qwen._forward: the deployment's pre-routing /# trained-state condition and not full_layer): stage_all+swap run- # after EVERY forward, including prefill chunks — so the last- # prefill token's predictions fill pred_inds and oh_prev, and the- # FIRST decode step consumes a real prediction at L7 instead of- # falling back to the router (a different, cleaner-but-divergent- # trajectory).+ # after EVERY forward, including prefill chunks. During prefill the+ # per-layer features (block.prerouter_m_in/oh) are not captured (the+ # patch only fills them on single-token forwards), so stage_all+ # submits nothing and the first decode step consumes no prediction --+ # it routes with the router, the pos-0 fallback used in training.if (self._pg_stager is not None and not full_layer):self._pg_stager.stage_all()self._pg_state.swap()@@ -165,11 +190,18 @@ class Qwen35Engine(Edge0Engine):def _step_pre(self, token_id: int) -> None:if self._pg_stager is not None:- # Router layers (no owner): fill from their previous actuals.+ # Layers without an incoming prediction (L0-6, L39, and every+ # layer switched off by the history_slots scoping): they route+ # with their own gate this step, so a staged slot set filled from+ # history would silently zero every routed expert outside it.+ # They run the exact path -- staging them is the legacy+ # history_slots=True behaviour. No prefetch either: the previous+ # token's set overlaps the next route by only ~6%, so the reads+ # are wasted (measured 11.7 -> 16.7 tok/s when removed).st = self._pg_statefor li, exp in self._all_stream_layers.items():if li < st.start or li >= st.n - 1:- if exp.last_used:+ if exp._staged_mode and exp.last_used:exp.stage_experts(list(exp.last_used))elif self._history_prefetch is not None:self._history_prefetch()@@ -179,7 +211,13 @@ class Qwen35Engine(Edge0Engine):st = self._pg_statefor li, exp in self._all_stream_layers.items():if li < st.start or li >= st.n - 1:- exp.sync_actuals()+ # Non-staged layers stage nothing from their actuals, so+ # the bookkeeping (and its per-layer host sync) is dead+ # weight -- unless the layer keeps hot pins, whose counts+ # it feeds.+ if (exp._staged_mode+ or getattr(exp, "hot_per_layer", 0) > 0):+ exp.sync_actuals()returnif self._stream_layers:for exp in self._all_stream_layers.values():@@ -195,13 +233,41 @@ class Qwen35Engine(Edge0Engine):for exp in self._all_stream_layers.values():exp.refresh_hot_pins()if self._stream_layers:- for exp in self._stream_layers.values():- exp.stage_from_prefill()+ if self._pg_stager is None or getattr(self.cfg, "history_slots",+ False):+ for exp in self._stream_layers.values():+ exp.stage_from_prefill()+ else:+ # With a prerouter the first decode step routes with the+ # router (pos-0 fallback, as in training): staging the last+ # prefill token's actual top-k there zeroes every routed+ # expert outside that set (measured 4/4 for L7). Prefetch+ # them instead; the step then runs the exact path.+ for exp in self._stream_layers.values():+ exp.prefetch_from_prefill()def _reset_state(self) -> None:self.cache = self._lm.make_cache()if self._pg_state is not None:self._pg_state.reset()+ # Drop the per-block feature captures too. The MoE patch writes+ # ``prerouter_m_in`` / ``prerouter_oh`` only on single-token+ # forwards and nothing else touches them, so without this the+ # NEXT request's prefill would re-run every head on the previous+ # request's last hidden state: its stage_all would submit stale+ # predictions that the first decode step then routes with (the+ # fresh-process behaviour is the gate fallback at pos 0).+ for owner in self._pg_state.owners:+ block = self.cfg.moe_spec.block_of(self.model, owner)+ block.prerouter_m_in = None+ block.prerouter_oh = None+ # Per-request reset of the streaming layers' staged state+ # (``StreamingSwitchGLU.reset``): a leftover ``_staged_state`` from+ # the previous request defeats the staged path's "fill not ready ->+ # exact path" fallback (``st`` would never be None) and would serve+ # the previous request's expert bundles.+ for exp in getattr(self, "_all_stream_layers", {}).values():+ exp.reset()def _lm_logits(self, h: core.array) -> core.array:return self._lm.lm_head(h[0, -1])diff --git a/src/edge0/models/base.py b/src/edge0/models/base.pyindex 09a14b9..7677182 100644--- a/src/edge0/models/base.py+++ b/src/edge0/models/base.py@@ -50,6 +50,16 @@ class ModelConfig:options: LayerOptionsprerouter: PrerouterSpec | None = Noneprerouter_top_k: int = 0 # 0 -> options.top_k+ history_slots: bool = False # False (default): only the layers whose+ # ROUTE is a prerouter prediction keep+ # staged slots; every other layer+ # prefetches its history top-k and runs+ # the exact path. True = deployment+ # legacy: staged slots are also filled+ # from history (previous token's+ # actuals / last prefill token), which+ # zeroes every routed expert outside+ # the slot set.lora: str = "" # safetensors path; "" disableslora_r: int = 16lora_alpha: float = 32.0diff --git a/src/edge0/models/edge0_8b/__init__.py b/src/edge0/models/edge0_8b/__init__.pyindex 7c8f4db..7fcc724 100644--- a/src/edge0/models/edge0_8b/__init__.py+++ b/src/edge0/models/edge0_8b/__init__.py@@ -29,6 +29,12 @@ from edge0.streaming.options import LayerOptionsclass Ling8BConfig(ModelConfig):"""edge0-8b family config (Ling 3.0 hybrid, K=8)."""+ #: ``edge0 demo`` / ``examples/demo.py`` default to the gate-routed exact+ #: path for this tier (``prerouter=None``); ``edge0 chat`` / ``serve``+ #: keep the tier default (prediction path). Consumed by+ #: ``edge0.registry.demo_kwargs``.+ demo_no_prerouter = True+@classmethoddef _defaults(cls, model_dir: str) -> "Ling8BConfig":return cls(diff --git a/src/edge0/prerouter/stager.py b/src/edge0/prerouter/stager.pyindex 2c8a613..c380ca5 100644--- a/src/edge0/prerouter/stager.py+++ b/src/edge0/prerouter/stager.py@@ -23,11 +23,13 @@ implemented here:from __future__ import annotations+import os+from edge0.backends import corefrom edge0.moe.routing import group_select_from_logits, select_from_logitsfrom edge0.moe.spec import MoESpec, RouterKind-from edge0.prerouter.heads import topk_onehot+from edge0.prerouter.heads import gelu_erf, topk_onehotfrom edge0.prerouter.spec import PrerouterSpecfrom edge0.prerouter.state import PrerouterStatefrom edge0.streaming.layer import StreamingSwitchGLU@@ -54,7 +56,20 @@ def _topk_onehot_pos(idx, num_experts: int, pos: int = -1):class PrerouterStager:"""Base: one-batch collect -> stack -> select -> eval -> tolist ->- per-consumer stage_experts."""+ per-consumer stage_experts.++ Two cost fixes on top of the naive loop (measured on the 8b tier,+ docs/experiments/prerouter-decode-speed-2026-09-14.md §6):++ * ``head_batch`` (default on): the owner heads share one shape, so their+ weights are stacked once and each step runs 3 ``einsum`` ops instead of+ 3 small ops per head. Mathematically identical up to fp16 rounding;+ ``head_batch=False`` (or ``EDGE0_HEAD_BATCH=0``) keeps the per-head loop+ for A/B.+ * when nothing consumes host-side expert IDs (``stream_layers`` empty,+ i.e. ``staged=False``) the ``core.eval`` + ``tolist`` are skipped: the+ routing cache holds lazy arrays, so the sync produces nothing usable.+ """def __init__(self,@@ -75,6 +90,9 @@ class PrerouterStager:self.num_experts = spec.num_expertsself.records: list | None = Noneself.cur_step = 0+ self.head_batch = os.environ.get("EDGE0_HEAD_BATCH", "1") != "0"+ self._stacked_key: tuple | None = None+ self._stacked: tuple | None = None# ---- feature hooks (overridden per family) ----------------------------@@ -87,13 +105,67 @@ class PrerouterStager:raise NotImplementedErrordef _store(self, consumer: int, logits_i, inds_i, scores_i,- experts: list[int]) -> None:- """Record this consumer's prediction + submit its staged fill."""+ experts: list[int] | None) -> None:+ """Record this consumer's prediction + submit its staged fill+ (``experts=None`` when no consumer needs host-side IDs)."""raise NotImplementedErrordef _note(self, owner: int, cur_oh) -> None:"""Per-owner side effects (e.g. roll the prev-token feature)."""+ # ---- batched head execution (fix 1) -----------------------------------++ def _stacked_weights(self, metas: list[int]):+ """Stack the owner heads' weights once (all heads share one shape).++ Returns None (and stays on the per-head loop) if anything about the+ shapes/dtypes is not uniform."""+ key = tuple(metas)+ if not self.head_batch:+ return None+ if self._stacked_key == key:+ return self._stacked+ self._stacked_key = key+ self._stacked = None+ try:+ heads = [self._head_of(o) for o in metas]+ t = lambda w: core.transpose( # noqa: E731+ core.stack([h.weight for h in w]), (0, 2, 1))+ w1 = t([h.fc1 for h in heads])+ w2 = t([h.fc2 for h in heads])+ wl = t([h.linear_init for h in heads])+ core.eval(w1, w2, wl)+ self._stacked = (w1, w2, wl)+ except Exception: # noqa: BLE001 — any shape mismatch: per-head loop+ self._stacked = None+ return self._stacked++ def _logits(self, metas: list[int], inputs: list[core.array],+ cur_ohs: list[core.array], prev_ohs: list[core.array]):+ """Stacked head logits [n,1,1,E] (batched, else per-head loop)."""+ w = self._stacked_weights(metas)+ if w is None:+ return core.stack(+ [self._head_of(o)(m, co, po)+ for o, m, co, po in zip(metas, inputs, cur_ohs, prev_ohs)],+ axis=0)+ w1, w2, wl = w+ # The fc1 precision is the cast the per-head implementations use+ # (``PrerouterHead._dtype`` / ``BailingPrerouter`` reading+ # ``fc1.weight.dtype``); take it from the stacked weight so this+ # stays family-agnostic.+ dtype = w1.dtype+ feats = core.concatenate([core.stack(inputs, 0), core.stack(cur_ohs, 0),+ core.stack(prev_ohs, 0)], axis=-1)+ if feats.dtype != dtype:+ feats = feats.astype(dtype)+ f2 = feats.reshape(len(metas), -1)+ h1 = core.einsum("ni,nij->nj", f2, w1)+ act = gelu_erf(h1)+ out = (core.einsum("ni,nij->nj", f2, wl)+ + core.einsum("ni,nij->nj", act, w2))+ return out.reshape(len(metas), 1, 1, self.num_experts)+# ---- the step-boundary batch ------------------------------------------def stage_all(self) -> dict[int, list[int]]:@@ -112,26 +184,27 @@ class PrerouterStager:prev_ohs.append(prev_oh)if not metas:return {}- per_logits = []- for owner, m_in, co, po in zip(metas, inputs, cur_ohs, prev_ohs):- head = self._head_of(owner)- per_logits.append(head(m_in, co, po))- logits = core.stack(per_logits, axis=0) # [n,1,1,E]+ logits = self._logits(metas, inputs, cur_ohs, prev_ohs) # [n,1,1,E]inds_all, scores_all = self._select(logits)- core.eval(inds_all, scores_all)- flat = inds_all.reshape(len(metas), -1).tolist() # ONE tolist+ flat = None+ if self.stream_layers:+ # Only the consumers' staged fills need host-side expert IDs.+ # Without them this sync + tolist is pure waste (fix 2).+ core.eval(inds_all, scores_all)+ flat = inds_all.reshape(len(metas), -1).tolist() # ONE tolistexpert_sets: dict[int, list[int]] = {}for i, owner in enumerate(metas):consumer = owner + 1- experts = sorted(set(int(v) for v in flat[i]))- expert_sets[consumer] = experts+ experts = (sorted({int(v) for v in flat[i]})+ if flat is not None else None)+ expert_sets[consumer] = experts or []self._store(consumer, logits[i], inds_all[i],scores_all[i], experts)self._note(owner, cur_ohs[i])- if self.records is not None:- for owner, consumer in zip(metas, metas):+ if self.records is not None and flat is not None:+ for owner in metas:self.records.append(- (self.cur_step, consumer, expert_sets[consumer]))+ (self.cur_step, owner + 1, expert_sets[owner + 1]))self.cur_step += 1return expert_sets@@ -169,13 +242,13 @@ class CrossTokenStager(PrerouterStager):return select_from_logits(logits, self.top_k)def _store(self, consumer: int, logits_i, inds_i, scores_i,- experts: list[int]):+ experts: list[int] | None):st = self.statest.pred_inds[consumer] = inds_ist.pred_scores[consumer] = scores_ist.logits[consumer - 1] = logits_iexp = self.stream_layers.get(consumer)- if exp is not None:+ if exp is not None and experts:exp.stage_experts(experts)def _note(self, owner: int, cur_oh) -> None:@@ -227,10 +300,10 @@ class LingPrerouterStager(PrerouterStager):routed_scaling=self.spec.routed_scaling or 1.0)def _store(self, consumer: int, logits_i, inds_i, scores_i,- experts: list[int]):+ experts: list[int] | None):self.pg_cache[consumer] = logits_iexp = self.stream_layers.get(consumer)- if exp is not None:+ if exp is not None and experts:exp.stage_experts(experts)def _note(self, owner: int, cur_oh) -> None:diff --git a/src/edge0/registry.py b/src/edge0/registry.pyindex 1585051..d4a54a7 100644--- a/src/edge0/registry.py+++ b/src/edge0/registry.py@@ -109,3 +109,20 @@ class AutoEngine:from edge0 import models # noqa: F401key = _resolve_name(model_dir, name)return MODEL_REGISTRY[key].build_engine(model_dir, **kwargs)+++def demo_kwargs(model_dir=None, name=None, **kwargs) -> dict:+ """``from_pretrained`` overrides for the demo entry points.++ A tier pins its showcase configuration with the class attribute+ ``demo_no_prerouter``: ``edge0-8b`` runs the gate-routed exact path in+ ``edge0 demo`` / ``examples/demo.py``, because its prerouter is opt-in+ there. Explicit keyword arguments (e.g. ``--no-prerouter``) always win.+ """+ from edge0 import models # noqa: F401+ out = dict(kwargs)+ if "prerouter" not in out:+ adapter = MODEL_REGISTRY[_resolve_name(model_dir, name)]+ if getattr(adapter.Config, "demo_no_prerouter", False):+ out["prerouter"] = None+ return outdiff --git a/src/edge0/streaming/layer.py b/src/edge0/streaming/layer.pyindex a82027e..e794842 100644--- a/src/edge0/streaming/layer.py+++ b/src/edge0/streaming/layer.py@@ -841,6 +841,24 @@ class StreamingSwitchGLU:self.last_used = uniqueself.stage_experts(unique)+ def prefetch_from_prefill(self):+ """After prefill: prefetch (do NOT stage) the last prefill token's+ actual top-k.++ Counterpart of ``stage_from_prefill`` for the case where the first+ decode step does not consume a prerouter prediction: that step routes+ with the router (pos-0 fallback, as in training), so filling the slots+ from the prefill token's top-k silently zeroes every routed expert+ outside that set. Warm the cache instead and let the step run the+ exact path with the bundles already resident."""+ topk = getattr(self, "_last_prefill_topk", None)+ if not topk:+ return+ unique = sorted(set(int(v) for v in topk))+ if unique:+ self.last_used = unique+ self.prefetch(unique)+def _refresh_hot_pins(self):if self.hot_per_layer <= 0:returndiff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.pyindex 34a27a3..ff71e1a 100644--- a/src/edge0/streaming/options.py+++ b/src/edge0/streaming/options.py@@ -40,7 +40,14 @@ class LayerOptions:hot_decay: decay factor for hot-expert usage counts.pin_bonus: extra score for prerouter-predicted experts whenselecting hot pins.- cache_slots: shared LRU capacity across all layers.+ cache_slots: shared LRU capacity across all layers, keyed by+ ``(layer_idx, expert)`` with a single global budget (see+ ``SharedExpertCache``). Size it for the WHOLE model, not per+ layer: a decode step touches ``top_k`` experts on every MoE layer+ (23 x 8 = 184 distinct keys on edge0-8b), so a budget smaller than+ that cannot hold even one token's working set and the hit rate+ collapses to ~0 (the released 64 slots = 2.8 slots/layer measured+ exactly 0 hits, rebuilding 184 bundles/step).prefetch_cap: prefetch buffer capacity.load_threads: threads for on-demand expert builds.prefetch_threads: threads for eager prefetch builds.@@ -110,20 +117,43 @@ class LayerOptions:@classmethoddef prod_k8(cls, **overrides) -> "LayerOptions":- """edge0-8b tier: deployment production profile.-- Mirrors the reference deployment profile: STAGED_DECODE=0- (staged decode is OFF — the deployment verified staged decode on- ling degrades output, see the comment in start_server.sh),- STAGED_SYNC=0, STAGED_SLOTS=8, EXPERT_CACHE_SLOTS=64,- MLX_COMPILE_MOE=1, PREFILL_FULL_LAYER=1. The hybrid prerouter- still stages each next token's expert set into the shared LRU- (engine.py: stage_all at the step boundary AND right after- prefill), and the vendored MoE blocks re-select from the cached- prerouter logits."""+ """edge0-8b tier: staged decode ON for the prerouter consumers.++ Only the layers whose ROUTE is a prerouter prediction keep staged+ slots (consumer ``li >= start_layer + 1``, i.e. 8-23): the stager+ submits exactly the set the consuming block re-selects from the same+ cached logits, so ``slots == routing`` and nothing is dropped+ (measured ``staged_dropped = 0``; output bit-identical to the exact+ path). The layers below ``start_layer`` route with their own gate and+ run the exact path (``ling.load_installed`` sets+ ``_staged_mode = False``; legacy history staging there dropped 17-26%+ of their experts and degraded output -- opt back in with+ ``history_slots=True`` / ``--history-slots``).++ Measured on M4 Pro (same-process paired A/B): slots cut expert loads+ from 184 to 56 per step (8-23 served entirely by the prediction), and+ the prerouter routes ~5% faster than gate routing with the same+ loading path.++ ``cache_slots`` stays at the low-memory default 64 (~81 MiB): this+ tier's contract is a small resident footprint and every cached bundle+ is 1.27 MiB of RAM. The 64 is NOT enough to reuse anything across+ tokens (2.8 slots/layer for a 184-key/step working set, measured 0+ hits): every predicted expert whose set changed since the previous+ token (44% of the 128 staged uses/step -- the other 56% come from the+ previous token's slots) plus all 56 gate-routed uses on L1-7 are+ re-sliced and re-packed every step. That is the price of the small+ footprint, and the staged slots are how this tier avoids paying it for+ the consumer layers. Enlarging the LRU buys speed with RAM (1024 =+ +1.3 GiB, peak 1.4 -> 2.6 GiB -> 40.02 -> 31.44 ms/step, paired 3/3+ rounds, output identical) -- do it only when the memory budget allows;+ at cache_slots=64 the scoped staged path is the fastest arm measured+ (37.99 vs 39.52 ms/step for gate routing + exact loads, 6/8 rounds) and+ under a load-bound regime (build latency throttled to 200 us) it wins+ 32-40% (4/4 rounds)."""return cls(- staged=False, staged_replace=False, staged_n=8,- staged_trigger=8, staged_sync=False, history_prefetch=False,+ staged=True, staged_replace=False, staged_n=8,+ staged_trigger=8, staged_sync=True, history_prefetch=False,hot_per_layer=0,cache_slots=64, prefetch_cap=48,load_threads=8, prefetch_threads=4, use_compile=True, top_k=8,diff --git a/tests/test_registry.py b/tests/test_registry.pyindex a50e2c2..8b8e321 100644--- a/tests/test_registry.py+++ b/tests/test_registry.py@@ -26,11 +26,13 @@ def test_auto_config_defaults(name):assert cfg.name == nameassert cfg.moe_spec.num_experts > 0assert cfg.moe_spec.top_k in (4, 8)- # qwen tier defaults to staged decode; the ling tier mirrors the- # deployment production profile (STAGED_DECODE=0 — staged decode on- # ling degrades output, see start_server.sh) with the prerouter- # still staging each next token's expert set.- assert cfg.options.staged is (name == "edge0-35b")+ # Both tiers default to staged decode. Staging is zero-drop only for+ # the layers whose ROUTE is a prerouter prediction (the scoped+ # consumers, li >= start_layer + 1); every layer below start_layer+ # runs the exact path, and the legacy history-filled staging for them+ # is opt-in via history_slots / --history-slots (see docs/prerouter.md).+ assert cfg.options.staged is True+ assert cfg.history_slots is Falseassert cfg.prerouter is not Noneassert cfg.prerouter.weights_file.endswith(".safetensors")assert cfg.prerouter_top_k == cfg.moe_spec.top_k@@ -57,6 +59,10 @@ def test_qwen35_profile():assert cfg.prerouter.hidden == 512assert cfg.prerouter.dtype == "fp16"assert cfg.prerouter.feature_topk == "executed"+ # only layers whose route is a prerouter prediction keep staged slots+ assert cfg.history_slots is False+ assert AutoConfig.from_pretrained(+ name="edge0-35b", history_slots=True).history_slots is Trueassert cfg.gen.temperature == 0.7assert 248046 in cfg.gen.eos_idsassert cfg.port == 8085@@ -89,6 +95,21 @@ def test_override_and_reject():AutoConfig.from_pretrained(name="edge0-8b", bogus_field=1)+def test_demo_defaults():+ """The demo entry points run each tier's showcase configuration."""+ from edge0.registry import demo_kwargs++ # edge0-8b demos default to the gate-routed exact path (prerouter off).+ assert demo_kwargs(name="edge0-8b")["prerouter"] is None+ assert demo_kwargs("edge0-8b")["prerouter"] is None # model_dir form+ # edge0-35b demos keep the tier config untouched.+ assert "prerouter" not in demo_kwargs(name="edge0-35b")+ # an explicit decision always wins.+ sentinel = object()+ assert demo_kwargs(name="edge0-8b",+ prerouter=sentinel)["prerouter"] is sentinel++def test_unknown_name_lists_registry():with pytest.raises(KeyError, match="edge0-35b"):AutoConfig.from_pretrained(name="edge0-99b")