Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

Open-source readiness: EN README, zero hardcoded paths, demo-profile parity

linyubupa2de963b20122
diff --git a/README.md b/README.md
index 7533ab2..f28e7df 100644
--- a/README.md
+++ b/README.md
@@ -1,118 +1,146 @@
# edge0
-**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload +
-并行 LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,
-当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)按同一套核心抽象接入。
+[English](README.md) | [中文](README_zh.md)
-开箱支持两个模型 tier:
+**edge0** is an open-source streaming MoE inference framework. It
+generalizes the production-proven recipe — **SSD expert offload +
+parallel LoRA + prerouter routing prediction** — into an extensible
+framework. The backend is isolated by design: the current MLX backend
+runs on Apple Silicon, and additional platforms (CUDA, …) plug into the
+same core abstractions.
-| 模型 | 说明 | 推理档 |
+Two model tiers work out of the box:
+
+| Tier | Model | Inference profile |
|---|---|---|
-| `edge0-35b` | Qwen3.6-35B-A3B 4bit(40 层,256 专家) | prerouter K=4 |
-| `edge0-10b` | Ling-10B bailing_hybrid(24 层,128 专家) | prerouter K=8 |
-
-## 设计
-
-- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`
- 按模型名自动选类;
-- **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,核心逻辑
- (模型 spec / prerouter / 流式专家池 / server)只依赖后端门面
- (`edge0/backends/base.py` 的 `core` / `nn` 门面),新增后端实现同一门面
- 即可平级接入(`backends/cuda/` 预留插槽),核心代码零改动;
-- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据
- (来源、轮次、owners)的 `.safetensors`,放模型目录或 `artifacts/`
- 均可自动解析;
-- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /
- `model*.safetensors` / tokenizer)和该模型的适配器,换轮次只换适配器
- 文件,基模不动、不 merge。
-
-## 核心机制
-
-- **SSD 专家 offload**:MoE 专家权重从磁盘 mmap 按需流式加载,
- 激活集驻留 LRU,长尾专家按层预取 —— 大模型小显存跑得动;
-- **prerouter 路由预判**:前一 token 的隐层状态经轻量头预测下一 token
- 的专家路由,SSD 预取与下一层前向重叠,路由零等待
- (qwen 系 `start_layer=7`,ling 系同);
-- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,
- 基模保持只读 mmap、多套适配器共享同一份基模;
-- **数值防护**:per-layer hidden clip(`LING_HIDDEN_CLIP`,默认 1000)
- 阻断 fp16 溢出 → 全 NaN logits → token-0 死循环的塌缩链。
-
-## 快速开始
+| `edge0-35b` | Qwen3.6-35B-A3B 4-bit (40 layers, 256 experts) | prerouter K=4 |
+| `edge0-10b` | Ling-10B 4-bit bailing hybrid (24 layers, 128 experts) | prerouter K=8 |
+
+## Design
+
+- **transformers-style usage**: `AutoModel` / `AutoConfig` / `AutoEngine`
+ resolve the tier from the model name;
+- **Backend isolation**: all MLX code lives under `edge0/backends/mlx/`;
+ the core logic (model specs, prerouter, streaming expert pool, server)
+ depends only on the backend facade (`edge0/backends/base.py`), so a new
+ backend implements the same facade (`backends/cuda/` is a reserved
+ slot) with zero changes to core code;
+- **Adapters as safetensors**: LoRA and prerouter weights are
+ `.safetensors` files with provenance metadata (source, round, owners),
+ resolved from the model directory or `artifacts/`;
+- **Model + adapters in one directory**: a model directory holds both
+ the base checkpoint (`config.json` / `model*.safetensors` / tokenizer)
+ and that model's adapters; switching training rounds swaps adapter
+ files only — the base stays read-only and is never merged.
+
+## Core mechanisms
+
+- **SSD expert offload**: MoE expert weights are mmapped from disk and
+ streamed on demand; the active set stays resident in an LRU and
+ long-tail experts are prefetched per layer — large models run in
+ modest memory;
+- **Prerouter routing prediction**: a lightweight head predicts the next
+ token's expert routing from the previous token's hidden state, so SSD
+ prefetch overlaps the next forward pass with zero routing latency
+ (`start_layer=7` on both tiers);
+- **Parallel LoRA**: adapters are applied as a side path at forward
+ time instead of being merged — the base stays a read-only mmap and
+ multiple adapter sets share one base;
+- **Numerical guard**: per-layer hidden clipping (`LING_HIDDEN_CLIP`,
+ default 1000) breaks the fp16 overflow → all-NaN logits → token-0
+ death-spiral collapse chain.
+
+## Quick start
```bash
-# 1) 安装(Python ≥3.10;MLX 后端需 mlx 环境当前支持的平台)
+# 1) Install (Python >= 3.10; MLX backend requires an mlx-supported platform)
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
-# 2) 一键试玩:自动探测本机 checkpoint;也可以显式传路径或 tier 名
-edge0 demo
+# 2) Quick demo: pass a checkpoint directory or tier name
edge0 demo /path/to/qwen35/model
-edge0 demo edge0-10b
+edge0 demo edge0-10b # checkpoint resolved via env vars (below)
-# 3) vLLM 风格起服务:模型是位置参数,checkpoint 类型自动识别
+# 3) Serve (OpenAI-compatible /v1/chat/completions; the model is a
+# positional argument, tier auto-detected from config.json)
edge0 serve /path/to/qwen35/model
-edge0 serve edge0-35b # tier 名:EDGE0_35B_MODEL 环境变量可覆盖默认路径
+edge0 serve edge0-35b
+```
+
+`python -m edge0 ...` is equivalent to `edge0 ...`.
+Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint
+directories through environment variables; without them, the tier is
+auto-detected from the checkpoint's `config.json`:
+
+```bash
+export EDGE0_35B_MODEL=/path/to/qwen35/model
+export EDGE0_10B_MODEL=/path/to/ling/model
+```
+
+```bash
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
-# 4) 命令行一问一答(同样接受路径或 tier 名)
-edge0 chat /path/to/model --prompt "你好,用一句话介绍珠海。"
+# 4) One-shot chat
+edge0 chat /path/to/model --prompt "Explain streaming inference in one sentence."
```
-`python -m edge0 ...` 与 `edge0 ...` 完全等价。
-
### Python API
```python
from edge0 import AutoEngine
-eng = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别
-ids = eng.encode_chat([{"role": "user", "content": "你好"}], think=True)
+eng = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected
+ids = eng.encode_chat([{"role": "user", "content": "Hello"}], think=True)
tokens = eng.generate(ids, max_new_tokens=512)
print(eng._tok.decode(tokens))
-eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)
+eng.reset() # clear per-request state (cross-request KV / prerouter cache)
```
-### 模型与适配器
+### Models and adapters
-- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、
- tokenizer)。`edge0 serve <dir>` / `AutoEngine.from_pretrained(<dir>)`
- 按 `config.json` 自动识别 tier。
-- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:
- - **模型目录内**(推荐):与基模同目录,如
- `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
- - `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从
- 训练侧 npz 一次性转换。
-- 当前默认适配器轮次:35b = round9,10b = round6(pgstart-sel1m,
- owners L7–22 与 `start_layer=7` 一致)。
+- **Checkpoint**: the original model directory (`config.json`,
+ `model*.safetensors`, tokenizer). `edge0 serve <dir>` /
+ `AutoEngine.from_pretrained(<dir>)` detect the tier from
+ `config.json`.
+- **Adapters** (LoRA + prerouter, safetensors) are resolved from either
+ location automatically:
+ - the model directory (recommended): side by side with the base, e.g.
+ `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
+ - `artifacts/` (repo root, gitignored): convert once from
+ training-side npz exports via `edge0 convert-adapters --npz-dir ...`.
+- Current default adapter rounds: 35b = round9, 10b = round6
+ (pgstart-sel1m, owners L7–22 matching `start_layer=7`).
-### 稳定性验证
+### Stability validation
-两个 tier 均通过长连测(temp 0.7 采样、think 开关混合、多 prompt):
+Both tiers passed long-run sampling tests (temp 0.7, thinking on/off,
+multiple prompts):
-- `!` 死循环塌缩 0/N(NaN clip 防护生效);
-- 碎片退化 0/N(低层 prerouter 噪声通过 `start_layer=7` 消除);
-- 跨请求状态污染 0/N(每请求 `reset()` + 首 token greedy)。
+- `!` death-spiral collapse: 0/N (NaN clipping in effect);
+- fragment degeneration: 0/N (low-layer prerouter noise eliminated via
+ `start_layer=7`);
+- cross-request state pollution: 0/N (per-request `reset()` +
+ first-token greedy).
-## 验证
+## Tests
```bash
-pytest # 单元测试(不含真实权重)
-pytest -m slow # 真实 checkpoint 端到端(qwen/ling 生成 + HTTP)
-scripts/e2e_smoke.py # 两档模型 staged vs exact 数值一致性冒烟
-scripts/zhuhai_example.py # 完整 API 上手例子(--thinking 开关演示)
-examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)
+pytest # unit tests (no real weights)
+pytest -m slow # end-to-end with real checkpoints (generation + HTTP)
+scripts/e2e_smoke.py # staged vs exact numerical consistency smoke
+scripts/generate_example.py # full-pipeline API example
+examples/demo.py # minimal API walkthrough
```
-## 文档
+## Documentation
-- [架构总览](docs/architecture.md)
-- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)
-- [如何接入新模型](docs/adding-a-model.md)
+- [Architecture](docs/architecture.md)
+- [Attention](docs/attention.md) / [MoE](docs/moe.md) / [SSD streaming](docs/streaming.md) / [prerouter](docs/prerouter.md)
+- [Adding a model](docs/adding-a-model.md)
- [edge0-35b](docs/models/edge0-35b.md) / [edge0-10b](docs/models/edge0-10b.md)
## License
-Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE.md))。
+Apache-2.0, including vendored third-party code (see [NOTICE](NOTICE.md)).
diff --git a/README_zh.md b/README_zh.md
new file mode 100644
index 0000000..8e28fe9
--- /dev/null
+++ b/README_zh.md
@@ -0,0 +1,128 @@
+# edge0
+
+[English](README.md) | 中文
+
+**edge0** 是一个开源的流式 MoE 推理框架:把「SSD 专家 offload +
+并行 LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后端隔离设计,
+当前实现 MLX 后端(Apple Silicon),更多平台(CUDA 等)即将接入。
+
+开箱支持两个模型 tier:
+
+| 模型 | 说明 | 推理档 |
+|---|---|---|
+| `edge0-35b` | Qwen3.6-35B-A3B 4bit(40 层,256 专家) | prerouter K=4 |
+| `edge0-10b` | Ling-10B 4bit(24 层,128 专家) | prerouter K=8 |
+
+## 设计
+
+- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`
+ 按模型名自动选类;
+- **后端隔离**:全部 MLX 代码收在 `edge0/backends/mlx/`,核心逻辑
+ (模型 spec / prerouter / 流式专家池 / server)只依赖后端门面
+ (`edge0/backends/base.py` 的 `core` / `nn` 门面),新增后端实现同一门面
+ 即可平级接入(`backends/cuda/` 预留插槽),核心代码零改动;
+- **适配器统一为 safetensors**:LoRA 与 prerouter 权重均为带元数据
+ (来源、轮次、owners)的 `.safetensors`,放模型目录或 `artifacts/`
+ 均可自动解析;
+- **模型 + 适配器同目录布局**:一个模型目录同时放基模(`config.json` /
+ `model*.safetensors` / tokenizer)和该模型的适配器,换轮次只换适配器
+ 文件,基模不动、不 merge。
+
+## 核心机制
+
+- **SSD 专家 offload**:MoE 专家权重从磁盘 mmap 按需流式加载,
+ 激活集驻留 LRU,长尾专家按层预取 —— 大模型小显存跑得动;
+- **prerouter 路由预判**:前一 token 的隐层状态经轻量头预测下一 token
+ 的专家路由,SSD 预取与下一层前向重叠,路由零等待
+ (qwen 系 `start_layer=7`,ling 系同);
+- **并行 LoRA**:适配器不合并进基模,前向时旁路叠加,
+ 基模保持只读 mmap、多套适配器共享同一份基模;
+- **数值防护**:per-layer hidden clip(`LING_HIDDEN_CLIP`,默认 1000)
+ 阻断 fp16 溢出 → 全 NaN logits → token-0 死循环的塌缩链。
+
+## 快速开始
+
+```bash
+# 1) 安装(Python ≥3.10;MLX 后端需 mlx 环境当前支持的平台)
+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
+
+# 2) 快速演示:指定 checkpoint 目录或 tier 名(tier 路径见环境变量)
+edge0 demo /path/to/qwen35/model
+edge0 demo edge0-10b # 由 EDGE0_10B_MODEL 指向的 checkpoint
+
+# 3) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,
+# checkpoint 类型按 config.json 自动识别)
+edge0 serve /path/to/qwen35/model
+edge0 serve edge0-35b # tier 名,checkpoint 路径同上由环境变量解析
+
+curl http://127.0.0.1:8000/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
+
+# 4) 命令行一问一答(同样接受路径或 tier 名)
+edge0 chat /path/to/model --prompt "用一句话介绍流式推理。"
+```
+
+`python -m edge0 ...` 与 `edge0 ...` 完全等价。
+
+tier 名(`edge0-35b` / `edge0-10b`)通过环境变量解析到本机 checkpoint
+目录,二者都未设置时按目录内 `config.json` 自动识别:
+
+```bash
+export EDGE0_35B_MODEL=/path/to/qwen35/model
+export EDGE0_10B_MODEL=/path/to/ling/model
+```
+
+### Python API
+
+```python
+from edge0 import AutoEngine
+
+eng = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别
+ids = eng.encode_chat([{"role": "user", "content": "你好"}], think=True)
+tokens = eng.generate(ids, max_new_tokens=512)
+print(eng._tok.decode(tokens))
+eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)
+```
+
+### 模型与适配器
+
+- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、
+ tokenizer)。`edge0 serve <dir>` / `AutoEngine.from_pretrained(<dir>)`
+ 按 `config.json` 自动识别 tier。
+- **适配器**(LoRA + prerouter,safetensors)放两处任一,自动解析:
+ - **模型目录内**(推荐):与基模同目录,如
+ `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
+ - `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从
+ 训练侧 npz 一次性转换。
+- 当前默认适配器轮次:35b = round9,10b = round6(pgstart-sel1m,
+ owners L7–22 与 `start_layer=7` 一致)。
+
+### 稳定性验证
+
+两个 tier 均通过长连测(temp 0.7 采样、think 开关混合、多 prompt):
+
+- `!` 死循环塌缩 0/N(NaN clip 防护生效);
+- 碎片退化 0/N(低层 prerouter 噪声通过 `start_layer=7` 消除);
+- 跨请求状态污染 0/N(每请求 `reset()` + 首 token greedy)。
+
+## 验证
+
+```bash
+pytest # 单元测试(不含真实权重)
+pytest -m slow # 真实 checkpoint 端到端(qwen/ling 生成 + HTTP)
+scripts/e2e_smoke.py # 两档模型 staged vs exact 数值一致性冒烟
+scripts/generate_example.py # 完整 API 上手例子(prefill→生成→解码全链路)
+examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)
+```
+
+## 文档
+
+- [架构总览](docs/architecture.md)
+- [注意力抽象](docs/attention.md) / [MoE 抽象](docs/moe.md) / [SSD 流式](docs/streaming.md) / [prerouter](docs/prerouter.md)
+- [如何接入新模型](docs/adding-a-model.md)
+- [edge0-35b](docs/models/edge0-35b.md) / [edge0-10b](docs/models/edge0-10b.md)
+
+## License
+
+Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE.md))。
diff --git a/docs/adding-a-model.md b/docs/adding-a-model.md
index 5e784ec..7d38642 100644
--- a/docs/adding-a-model.md
+++ b/docs/adding-a-model.md
@@ -42,7 +42,7 @@ edge0_35b/__init__.py`)作为逐步示例,逐处标注真实代码行号。`
| `gen` | `GenerationConfig` | 采样默认(温度 / top-p / top-k / eos 等) |
| `prefill_chunk` / `hot_window` / `intra_staging` / `prefetch_history` | — | 流式与预取行为 |
| `port` | `int` | 服务端口 |
-| `target_tok_s` / `peak_active_mem_mb` | `float` | 验收指标(生产机实测) |
+| `target_tok_s` / `peak_active_mem_mb` | `float` | 验收指标(基准环境实测) |
子类只需实现类方法 `_defaults(model_dir) -> Config`,返回族默认配置。edge0-35b 的
实现见 `edge0_35b/__init__.py:31–69`:`_defaults` 用一份完整的 `MoESpec`、`LayerOptions`
diff --git a/docs/architecture.md b/docs/architecture.md
index 5be2094..e7ee01e 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -129,7 +129,7 @@ MLX 后端是参考实现;未来 CUDA 后端实现同一表面即可复用全
dequant 参考逐元素对比(相对 L2 < 1%),防 gate/up 交换、行错位类回归;
- `tests/test_moe_spec.py`、`tests/test_registry.py`、`tests/test_sampling.py` ——
纯逻辑层单测,无需真实 checkpoint;
-- 验收指标:`target_tok_s` 与 `peak_active_mem_mb` 记录在生产机器上实测的
+- 验收指标:`target_tok_s` 与 `peak_active_mem_mb` 记录的基准环境实测
吞吐与峰值活跃内存,作为回归标尺。
## License
diff --git a/docs/models/edge0-10b.md b/docs/models/edge0-10b.md
index ec37560..c5c1397 100644
--- a/docs/models/edge0-10b.md
+++ b/docs/models/edge0-10b.md
@@ -2,7 +2,7 @@
edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)的 10B 级稀疏混合专家模型。相比 35b 档牺牲一定的参数量,换来更低的峰值内存与更高的生成速率,适合对延迟与显存敏感的场景。
-生产档案基于 v7 checkpoint 实测,默认由 `Ling10BConfig`(`src/edge0/models/edge0_10b/__init__.py`)固定。
+性能档案基于 round6(pgstart-sel1m)adapter 组合的基准实测,默认由 `Ling10BConfig`(`src/edge0/models/edge0_10b/__init__.py`)固定。
## 模型档案
@@ -16,7 +16,7 @@ edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)
| 专家量化 | 4-bit affine,group 64 |
| 权重布局 | `WeightLayout.SEPARATE`(gate/up/down 分离张量堆叠) |
| 专家权重路径 | `model.layers.N.mlp.experts` |
-| 前置路由(prerouter) | 22 个头(显式 owners 1..22),start_layer 1,hidden 512,fp16 |
+| 前置路由(prerouter) | 16 个头(显式 owners 7..22),start_layer 7,hidden 512,fp16 |
| 解码调用方式 | `patch_call=False`(头内建在 `BailingSparseMoE` 中,从 `prerouter_cache` logits 消费) |
| LoRA | `r=16, alpha=32.0` |
| 预填分块 | 2048 |
@@ -26,23 +26,24 @@ edge0 平台的高吞吐轻量档:基于 Ling 3.0 混合架构(MLA + MoE)
| 验收吞吐 | ≈ 33 tok/s |
| 验收峰值激活内存 | ≈ 1.4 GB |
-> 说明:数值全部取自 `Ling10BConfig._defaults()` 与 `LayerOptions.staged_k8()`。头部数量来自显式 `owners=range(1, 23)`,共 22 个头;`feature_topk="executed"`。
+> 说明:数值全部取自 `Ling10BConfig._defaults()` 与 `LayerOptions.prod_k8()`。头部数量来自显式 `owners=range(7, 23)`,共 16 个头(round6 训练头部分布,L7 起消费预测,L1–6 走原始 router);`feature_topk="executed"`。
## 分阶段解码(staged decode)
-该档使用 `LayerOptions.staged_k8()` 预设:
+该档使用 `LayerOptions.prod_k8()` 预设(对齐 ling v7 部署的生产开关):
-- 固定槽分阶段解码(`staged=True`,`staged_n=8`,`staged_trigger=8`,`staged_sync=True`),逐层无 host 同步。
-- `staged_replace=False`:路由由内建的混合 prerouter 提供,分阶段集合与路由集合一致,槽表映射零丢弃。
-- 不使用热专家钉住(`hot_per_layer=0`),`prefill_hot=0`。
-- 整层 E3b prefill(`full_layer_prefill=True`)。
+- 分阶段解码关闭(`staged=False`,`staged_sync=False`,`staged_n=8`)——部署验证
+ 该档上 staged decode 会劣化输出,prerouter 直接驱动下一 token 的专家
+ 预取(step 边界 `stage_all` + prefill 尾部各一次)。
+- 专家缓存 `cache_slots=64`,热专家钉住关闭(`hot_per_layer=0`)。
+- 整层 E3b prefill(`full_layer_prefill=True`,`prefill_chunk=2048`)。
## 使用方式
### CLI 起服务
```bash
-edge0 serve --name edge0-10b --model-dir /path/to/checkpoint --host 127.0.0.1 --port 8083
+edge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8083
```
可选参数:
diff --git a/docs/models/edge0-35b.md b/docs/models/edge0-35b.md
index c9b8a14..bd391d3 100644
--- a/docs/models/edge0-35b.md
+++ b/docs/models/edge0-35b.md
@@ -1,8 +1,8 @@
# edge0-35b
-edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B 级稀疏混合专家模型。通过流式专家加载(streaming experts)、训练前置路由(prerouter)与 LoRA 适配,把整份权重量化后驻留在磁盘、按需装载,在一块消费级设备上即可服务。
+edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B 级稀疏混合专家模型。通过流式专家加载(streaming experts)、训练前置路由(prerouter)与 LoRA 适配,把整份权重量化后驻留在磁盘、按需装载,在单台设备上即可服务。
-生产档案基于 round-7 checkpoint 实测,默认由 `Qwen35Config`(`src/edge0/models/edge0_35b/__init__.py`)固定。
+性能档案基于 round9 adapter 组合的基准实测,默认由 `Qwen35Config`(`src/edge0/models/edge0_35b/__init__.py`)固定。
## 模型档案
@@ -42,7 +42,7 @@ edge0 平台的第一档主力模型:基于 Qwen3.5-MoE(K=4 档)的 35B
### CLI 起服务
```bash
-edge0 serve --name edge0-35b --model-dir /path/to/checkpoint --host 127.0.0.1 --port 8085
+edge0 serve /path/to/checkpoint --host 127.0.0.1 --port 8085
```
可选参数:
diff --git a/examples/bench.py b/examples/bench.py
new file mode 100644
index 0000000..e9e71eb
--- /dev/null
+++ b/examples/bench.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""edge0 speed + memory benchmark (any tier, bench_v7-equivalent protocol).
+
+Runs the same measurement shape as the deployment's bench: prefill ->
+sampled warmup steps -> timed sampled decode -> report tok/s and MLX
+peak active memory. Works for both tiers; the tier is auto-detected
+from the checkpoint (or forced via the model name).
+
+Usage:
+ python examples/bench.py /path/to/model [--ntok 200] [--warmup 10]
+ python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL
+
+Env (sampling knobs, defaults follow the tier's GenerationConfig):
+ BENCH_TEMP / BENCH_PROMPT / BENCH_SEED
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+import time
+
+from edge0 import AutoEngine
+from edge0.backends import core
+from edge0.sampling import sample
+
+PROMPTS = {
+ "edge0-35b": "什么是混合专家模型(MoE)?它和普通 Transformer 有什么区别?",
+ "edge0-10b": "9.11 和 9.8 哪个大?请仔细比较。",
+}
+DEFAULT_PROMPT = "什么是混合专家模型(MoE)?简单介绍一下。"
+
+
+def _prompt_for(engine) -> str:
+ return os.environ.get("BENCH_PROMPT") or PROMPTS.get(engine.name,
+ DEFAULT_PROMPT)
+
+
+def run_bench(engine, ntok: int, warmup: int) -> dict:
+ tok = engine._tok
+ prompt = _prompt_for(engine)
+ if hasattr(engine, "encode_chat"):
+ ids = engine.encode_chat(
+ [{"role": "user", "content": prompt}], think=False)
+ else:
+ ids = list(tok(tok.apply_chat_template(
+ [{"role": "user", "content": prompt}],
+ tokenize=False, add_generation_prompt=True,
+ enable_thinking=False))["input_ids"])
+
+ gen = getattr(engine.cfg, "gen", None)
+ temperature = float(os.environ.get(
+ "BENCH_TEMP", getattr(gen, "temperature", 0.7)))
+ top_k = getattr(gen, "top_k", 64)
+ top_p = getattr(gen, "top_p", 0.95)
+ rp = getattr(gen, "repetition_penalty", 1.0)
+ seed = os.environ.get("BENCH_SEED")
+
+ results = []
+ for run in range(2):
+ engine.reset()
+ core.reset_peak_memory()
+ t0 = time.perf_counter()
+ engine.prefill(ids)
+ t_prefill = time.perf_counter() - t0
+ logits = engine.next_logits()
+ if seed is not None:
+ core.random.seed(int(seed))
+
+ history = list(ids)
+ # warmup: sampled, untimed (first tokens are dominated by expert
+ # cold-starts; the timed window is the steady state)
+ for _ in range(warmup):
+ tid = int(core.argmax(logits, axis=-1).item())
+ history.append(tid)
+ logits = engine.step(tid)
+ out: list[int] = []
+ t0 = time.perf_counter()
+ for _ in range(ntok):
+ tid = sample(logits, temperature=temperature, top_k=top_k,
+ top_p=top_p, repetition_penalty=rp,
+ history=history, seed=None)
+ history.append(tid)
+ out.append(tid)
+ logits = engine.step(tid)
+ t_decode = time.perf_counter() - t0
+ peak_gib = core.get_peak_memory() / (1024 ** 3)
+ results.append(dict(prefill_s=t_prefill, decode_s=t_decode,
+ ntok=len(out), peak_gib=peak_gib,
+ tok_s=len(out) / t_decode if t_decode else 0.0))
+ print(f"prompt={len(ids)} tok prefill={t_prefill:.2f}s "
+ f"decode={len(out)}/{t_decode:.2f}s "
+ f"tok/s={results[-1]['tok_s']:.1f} "
+ f"peak_active={peak_gib:.2f} GiB", flush=True)
+ mean_ts = sum(r["tok_s"] for r in results) / len(results)
+ max_peak = max(r["peak_gib"] for r in results)
+ print(f"[bench] mean tok/s={mean_ts:.1f} peak_active<={max_peak:.2f} GiB",
+ flush=True)
+ return {"runs": results, "mean_tok_s": mean_ts, "peak_gib": max_peak}
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__)
+ ap.add_argument("model", help="checkpoint dir or tier name")
+ ap.add_argument("--ntok", type=int,
+ default=int(os.environ.get("BENCH_NTOK", "200")))
+ ap.add_argument("--warmup", type=int, default=10)
+ args = ap.parse_args()
+ # tier name -> checkpoint dir via $EDGE0_<TIER>_MODEL (cli parity)
+ model = args.model
+ name = None
+ if model.startswith("edge0-"):
+ from edge0.cli import TIER_ENV
+ env = os.environ.get(TIER_ENV.get(model, ""))
+ if not env:
+ raise SystemExit(
+ f"set {TIER_ENV.get(model, 'EDGE0_<TIER>_MODEL')} to the "
+ f"checkpoint directory for {model}, or pass the dir itself")
+ model = env
+ engine = AutoEngine.from_pretrained(model, name=name)
+ print(f"[bench] tier={engine.name} ntok={args.ntok} warmup={args.warmup}",
+ flush=True)
+ run_bench(engine, args.ntok, args.warmup)
+ engine.close()
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/examples/demo.py b/examples/demo.py
index 5fc3b0a..0a4efb0 100644
--- a/examples/demo.py
+++ b/examples/demo.py
@@ -28,7 +28,7 @@ def main() -> int:
ap.add_argument("--model-dir", default=None,
help="checkpoint directory (tier auto-detected)")
ap.add_argument("--prompt",
- default="Hello! Write one short sentence about Zhuhai.")
+ default="Hello! Write one short sentence about the seaside.")
ap.add_argument("--max-new", type=int, default=24)
args = ap.parse_args()
diff --git a/scripts/convert_adapters_legacy.py b/scripts/convert_adapters_legacy.py
index e8a01de..879b320 100644
--- a/scripts/convert_adapters_legacy.py
+++ b/scripts/convert_adapters_legacy.py
@@ -19,6 +19,7 @@ from __future__ import annotations
import argparse
import hashlib
+import os
import json
import sys
from datetime import datetime, timezone
@@ -30,18 +31,27 @@ ROOT = Path(__file__).resolve().parents[1]
ARTIFACTS = ROOT / "artifacts"
# (npz path, tier, kind) — every legacy source this script understands.
+# Training-side npz exports discovered per machine; pass --npz-dir (or
+# edit this list) to point at your own exports.
+NPZ_DIRS = [
+ Path(os.environ.get("EDGE0_NPZ_DIR", "/path/to/npz/exports")),
+]
SOURCES = [
- ("/Users/linyu/Documents/qwen35-v7-deploy/pregate_qwen35_v7_round7.npz",
- "edge0-35b", "prerouter"),
- ("/Users/linyu/Documents/qwen35-v7-deploy/lora_qwen35_v7_round7.npz",
- "edge0-35b", "lora"),
- ("/Users/linyu/Documents/ling-mlx-server/v7-deploy/pregate_v7_fp16.npz",
- "edge0-10b", "prerouter"),
- ("/Users/linyu/Documents/ling-mlx-server/v7-deploy/lora_v7_fp16.npz",
- "edge0-10b", "lora"),
+ ("pregate_qwen35_v7_round7.npz", "edge0-35b", "prerouter"),
+ ("lora_qwen35_v7_round7.npz", "edge0-35b", "lora"),
+ ("pregate_v7_fp16.npz", "edge0-10b", "prerouter"),
+ ("lora_v7_fp16.npz", "edge0-10b", "lora"),
]
+def _find_source(name: str) -> Path | None:
+ for d in NPZ_DIRS:
+ p = d / name
+ if p.is_file():
+ return p
+ return None
+
+
def md5(path: Path) -> str:
h = hashlib.md5()
with open(path, "rb") as f:
@@ -113,21 +123,31 @@ def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--force", action="store_true",
help="reconvert even if the artifact exists")
+ ap.add_argument("--npz-dir", default=None,
+ help="directory holding the legacy npz exports "
+ "(default: $EDGE0_NPZ_DIR)")
args = ap.parse_args(argv)
+ if args.npz_dir:
+ NPZ_DIRS.insert(0, Path(args.npz_dir))
ARTIFACTS.mkdir(parents=True, exist_ok=True)
- missing = [s[0] for s in SOURCES if not Path(s[0]).is_file()]
+ missing = []
+ resolved = []
+ for name, tier, kind in SOURCES:
+ p = _find_source(name)
+ if p is None:
+ missing.append(name)
+ else:
+ resolved.append((p, tier, kind))
if missing:
print("missing legacy sources (skip their conversion):")
for m in missing:
print(" -", m)
- for path, tier, kind in SOURCES:
- if not Path(path).is_file():
- continue
+ for path, tier, kind in resolved:
try:
- convert(Path(path), tier, kind, force=args.force)
+ convert(path, tier, kind, force=args.force)
except Exception as exc: # noqa: BLE001 — report and continue
- print(f"FAIL {Path(path).name}: {exc}", file=sys.stderr)
+ print(f"FAIL {path.name}: {exc}", file=sys.stderr)
return 1
return 0
diff --git a/scripts/e2e_smoke.py b/scripts/e2e_smoke.py
index 216abe2..65a0074 100644
--- a/scripts/e2e_smoke.py
+++ b/scripts/e2e_smoke.py
@@ -20,8 +20,8 @@ from edge0.backends import io
from edge0.config import GenerationConfig
from edge0.streaming.options import LayerOptions
-QWEN_DEFAULT = "/Users/linyu/Documents/qwen35-v7-deploy/model"
-LING_DEFAULT = "/Users/linyu/Documents/ling-mlx-server/v7-deploy/model"
+QWEN_DEFAULT = os.environ.get("EDGE0_35B_MODEL", "/path/to/qwen35/model")
+LING_DEFAULT = os.environ.get("EDGE0_10B_MODEL", "/path/to/ling/model")
def _prompt_ids(model_dir: str, text: str) -> list[int]:
diff --git a/scripts/zhuhai_example.py b/scripts/generate_example.py
similarity index 89%
rename from scripts/zhuhai_example.py
rename to scripts/generate_example.py
index 77f0ebd..5d1262d 100644
--- a/scripts/zhuhai_example.py
+++ b/scripts/generate_example.py
@@ -1,9 +1,9 @@
-"""Complete end-to-end example: generate an English introduction to
-Zhuhai with edge0-35b (prerouter + LoRA + SSD offload, default tier
-profile — staged decode on, K=4).
+"""Complete end-to-end example: generate an English introduction to a
+seaside city with edge0-35b (prerouter + LoRA + SSD offload, default
+tier profile — staged decode on, K=4).
Usage:
- .venv/bin/python scripts/zhuhai_example.py [--max-new N] [--temp T]
+ .venv/bin/python scripts/generate_example.py [--max-new N] [--temp T]
The example walks through the full pipeline a user of the framework
would hit:
@@ -19,6 +19,7 @@ would hit:
from __future__ import annotations
import argparse
+import os
import sys
import time
@@ -28,7 +29,7 @@ from edge0.config import GenerationConfig
PROMPT = (
- "Write a short introduction to Zhuhai, a city in southern China, "
+ "Write a short introduction to a seaside city in southern China, "
"in English. Keep it to 3-4 sentences."
)
@@ -41,7 +42,7 @@ def main() -> int:
help="enable the <think> block (enable_thinking=True)")
args = ap.parse_args()
- model_dir = "/Users/linyu/Documents/qwen35-v7-deploy/model"
+ model_dir = os.environ.get("EDGE0_35B_MODEL", "/path/to/qwen35/model")
print(f"[1] loading tokenizer + edge0-35b from {model_dir}", flush=True)
tok = io.load_tokenizer(model_dir)
t0 = time.time()
diff --git a/scripts/verify_alignment.py b/scripts/verify_alignment.py
index 27f5355..a4d54f2 100644
--- a/scripts/verify_alignment.py
+++ b/scripts/verify_alignment.py
@@ -22,13 +22,13 @@ import sys
N = 14 # max new tokens (greedy)
PROMPTS = {
- "qwen": "Hello! Write one short sentence about Zhuhai.",
- "ling": "你好,用一句话介绍珠海。",
+ "qwen": "Hello! Write one short sentence about the seaside.",
+ "ling": "你好,用一句话介绍海滨城市。",
}
-QMODEL = "/Users/linyu/Documents/qwen35-v7-deploy/model"
-LMODEL = "/Users/linyu/Documents/ling-mlx-server/v7-deploy/model"
-QDIR = "/Users/linyu/Documents/qwen35-v7-deploy"
-LDIR = "/Users/linyu/Documents/ling-mlx-server/v7-deploy"
+QMODEL = os.environ.get("EDGE0_35B_MODEL", "/path/to/qwen35/model")
+LMODEL = os.environ.get("EDGE0_10B_MODEL", "/path/to/ling/model")
+QDIR = os.path.dirname(QMODEL)
+LDIR = os.path.dirname(LMODEL)
RESULT = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"alignment_results.json")
diff --git a/src/edge0/cli.py b/src/edge0/cli.py
index 0598d26..b8a863a 100644
--- a/src/edge0/cli.py
+++ b/src/edge0/cli.py
@@ -18,16 +18,16 @@ import sys
from edge0 import models # noqa: F401 (populates MODEL_REGISTRY)
from edge0.registry import MODEL_REGISTRY
-# Checkpoint locations the demo command probes when no --model-dir is
-# given (the dev boxes these tiers were validated on).
-DEMO_DEFAULTS = {
- "edge0-35b": "/Users/linyu/Documents/qwen35-v7-deploy/model",
- "edge0-10b": "/Users/linyu/Documents/ling-mlx-server/v7-deploy/model",
+# Tier name -> environment variable that locates that tier's checkpoint
+# (no built-in paths: every machine resolves its own checkpoints).
+TIER_ENV = {
+ "edge0-35b": "EDGE0_35B_MODEL",
+ "edge0-10b": "EDGE0_10B_MODEL",
}
DEMO_PROMPTS = {
- "edge0-35b": "Hello! Write one short sentence about Zhuhai.",
- "edge0-10b": "你好,用一句话介绍珠海。",
+ "edge0-35b": "Hello! Write one short sentence about the seaside.",
+ "edge0-10b": "你好,用一句话介绍海滨城市。",
}
@@ -69,50 +69,45 @@ def _engine_kwargs(args) -> dict:
def _resolve_model(args) -> tuple[str | None, str | None]:
- """vLLM-style model argument resolution.
+ """Resolve the positional ``model`` argument.
``edge0 serve <model>`` accepts either a registered tier name
- (``edge0-35b`` / ``edge0-10b`` — checkpoint found via the matching
- env var, else the known dev-box default) or a checkpoint path
- (``edge0 serve /root/edge0-35b`` — tier auto-detected from the
- checkpoint's ``config.json``).
+ (``edge0-35b`` / ``edge0-10b`` — checkpoint located via the matching
+ ``EDGE0_<TIER>_MODEL`` environment variable) or a checkpoint path
+ (tier auto-detected from the checkpoint's ``config.json``).
Returns ``(model_dir, name)``; either may stay None to keep the
- legacy ``--model-dir`` / ``--name`` behaviour.
+ ``--model-dir`` / ``--name`` behaviour.
"""
model = getattr(args, "model", None)
if not model:
return args.model_dir, args.name
if model in MODEL_REGISTRY:
- short = (model.split("-", 1)[-1] if model.startswith("edge0-")
- else model)
- env = os.environ.get(f"EDGE0_{short.upper()}_MODEL")
- return env or DEMO_DEFAULTS.get(model), model
+ env = os.environ.get(TIER_ENV.get(model, ""))
+ return env, model
return model, None # a path: tier auto-detected by the registry
+def _missing_model_help(name) -> str:
+ tier_env = TIER_ENV.get(name, "EDGE0_<TIER>_MODEL")
+ return (
+ f"[edge0] no checkpoint for {name or 'model'}.\n"
+ f"Set {tier_env} to the checkpoint directory, e.g.\n"
+ f" export {tier_env}=/path/to/model\n"
+ "or pass the checkpoint explicitly:\n"
+ " edge0 demo /path/to/model\n"
+ "or point at any compatible checkpoint (tier auto-detected from"
+ " config.json)."
+ )
+
+
def cmd_demo(args) -> int:
from edge0 import AutoEngine
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
model_dir, name = _resolve_model(args)
- if not model_dir:
- # No explicit model: probe the known dev-box checkpoints.
- for d in DEMO_DEFAULTS.values():
- if os.path.isdir(d):
- model_dir = d
- break
if not model_dir or not os.path.isdir(model_dir):
- print(
- f"[edge0] no checkpoint for {name or 'model'}.\n"
- "Pass the model explicitly, vLLM-style:\n"
- f" edge0 demo {DEMO_DEFAULTS.get(name, '<checkpoint-dir>')}\n"
- "or point at your own checkpoint:\n"
- " edge0 demo /path/to/qwen35/model\n"
- "Tier names are edge0-35b / edge0-10b (env EDGE0_<TIER>_MODEL "
- "overrides the default search path).",
- file=sys.stderr,
- )
+ print(_missing_model_help(name), file=sys.stderr)
return 2
engine = AutoEngine.from_pretrained(model_dir, name=name,
**_engine_kwargs(args))
@@ -204,7 +199,7 @@ def main(argv: list[str] | None = None) -> int:
p = sub.add_parser(
"demo",
- help="one-command quickstart (vLLM-style model argument)")
+ help="one-shot generation demo")
p.add_argument("model", nargs="?", default=None,
help="tier name (edge0-35b) or checkpoint dir")
p.add_argument("--model-dir", default=None)
@@ -216,7 +211,7 @@ def main(argv: list[str] | None = None) -> int:
p = sub.add_parser(
"chat",
- help="one-shot prompt answering (vLLM-style model argument)")
+ help="one-shot prompt answering")
p.add_argument("model", nargs="?", default=None,
help="tier name (edge0-35b) or checkpoint dir")
p.add_argument("--model-dir", default=None)
@@ -228,7 +223,7 @@ def main(argv: list[str] | None = None) -> int:
p = sub.add_parser(
"serve",
- help="run the HTTP server, vLLM-style: edge0 serve <model>")
+ help="run the OpenAI-compatible HTTP server: edge0 serve <model>")
p.add_argument("model", nargs="?", default=None,
help="tier name (edge0-35b) or checkpoint dir")
p.add_argument("--model-dir", default=None)
diff --git a/src/edge0/models/edge0_35b/__init__.py b/src/edge0/models/edge0_35b/__init__.py
index ff39fb5..d60b86c 100644
--- a/src/edge0/models/edge0_35b/__init__.py
+++ b/src/edge0/models/edge0_35b/__init__.py
@@ -1,6 +1,6 @@
"""edge0-35b adapter: Qwen3.5-MoE (K=4 tier, 35B-class).
-Family facts (production profile, verified against the round-7
+Family facts (production profile, verified against the round-9
checkpoint):
* 40 layers, 256 routed experts (4-bit affine, group 64), K=4,
diff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.py
index 115130e..fd31ec8 100644
--- a/src/edge0/streaming/options.py
+++ b/src/edge0/streaming/options.py
@@ -91,10 +91,10 @@ class LayerOptions:
return cls(
staged=True, staged_replace=False, staged_n=4,
staged_trigger=4, staged_sync=True, history_prefetch=True,
- hot_per_layer=32, hot_update_interval=4, hot_decay=0.75,
+ hot_per_layer=0, hot_update_interval=4, hot_decay=0.75,
cache_slots=64, prefetch_cap=48, load_threads=8,
prefetch_threads=4, use_compile=True, top_k=4,
- full_layer_prefill=True, prefill_full_layers=12,
+ full_layer_prefill=False, prefill_full_layers=0,
prefill_hot=32, **overrides)
@classmethod
diff --git a/tests/test_registry.py b/tests/test_registry.py
index adf9998..285c533 100644
--- a/tests/test_registry.py
+++ b/tests/test_registry.py
@@ -48,8 +48,11 @@ def test_qwen35_profile():
assert cfg.moe_spec.layout.value == "separate"
assert "language_model.model.layers" in cfg.moe_spec.key_template
assert cfg.options.staged_n == 4
- assert cfg.options.prefill_full_layers == 12
- assert cfg.options.hot_per_layer == 32
+ # demo production profile: on-demand prefill (QWEN_PREFILL_FULL=0),
+ # no resident hot pins (QWEN_HOT=0)
+ assert cfg.options.prefill_full_layers == 0
+ assert cfg.options.full_layer_prefill is False
+ assert cfg.options.hot_per_layer == 0
assert cfg.prerouter.start_layer == 7
assert cfg.prerouter.hidden == 512
assert cfg.prerouter.dtype == "fp16"
@@ -68,9 +71,10 @@ def test_ling10b_profile():
assert cfg.moe_spec.n_group == 8
assert cfg.moe_spec.topk_group == 4
assert cfg.options.staged_n == 8
- assert cfg.prerouter.start_layer == 1
- assert cfg.prerouter.owners == tuple(range(1, 23))
+ assert cfg.prerouter.start_layer == 7
+ assert cfg.prerouter.owners == tuple(range(7, 23))
assert cfg.prerouter.patch_call is False
+ assert cfg.gen.repetition_penalty == 1.1
assert 156895 in cfg.gen.eos_ids
assert cfg.port == 8083
diff --git a/tests/test_server.py b/tests/test_server.py
index 7c7a373..82061bf 100644
--- a/tests/test_server.py
+++ b/tests/test_server.py
@@ -62,10 +62,17 @@ class FakeTok:
class PlainTok(FakeTok):
- """Tokenizer without apply_chat_template (stdlib fallback path)."""
+ """Tokenizer without apply_chat_template (stdlib fallback path).
- def apply_chat_template(self, *a, **k): # pragma: no cover - unused
- raise AttributeError
+ Deleting the method (vs raising) keeps hasattr() False so the
+ server's plain-text fallback renders instead."""
+
+ def __init__(self):
+ super().__init__()
+ try:
+ del self.apply_chat_template
+ except AttributeError:
+ pass
class FakeEngine:
@@ -91,6 +98,10 @@ class FakeEngine:
def stats(self):
return {"ok": 1}
+ def reset(self):
+ """Per-request state clear (server parity with the real engine)."""
+ self.pos = 0
+
def _req(**kw) -> ChatRequest:
payload = {
@@ -225,8 +236,8 @@ def test_queue_server_health():
def test_queue_server_chat_serializes():
eng = FakeEngine()
srv = QueueServer(eng)
- out = srv.chat(_req())
- assert out["usage"]["completion_tokens"] == 3
+ tokens, meta = srv.chat(_req())
+ assert meta["usage"]["completion_tokens"] == 3
assert len(eng.generated) == 1
gen = eng.generated[0][1]
assert gen.temperature == 0.7
@@ -270,11 +281,13 @@ def test_chat_stream_events_sequence():
})
body = raw if isinstance(raw, bytes) else raw.get_data()
events = [e for e in body.decode("utf-8").split("\n\n") if e]
+ assert events[-1] == "data: [DONE]"
chunks = [json.loads(e[6:]) for e in events[:-1]] # drop [DONE]
- deltas = [c["choices"][0]["delta"]["content"] for c in chunks]
+ deltas = [c["choices"][0]["delta"]["content"]
+ for c in chunks if c["choices"][0]["delta"]]
assert deltas == ["T11", "T12", "T13"]
assert chunks[-1]["choices"][0]["finish_reason"] == "stop"
- assert events[-1] == "data: [DONE]"
+ assert chunks[-1]["choices"][0]["delta"] == {}
# ---- stdlib HTTP transport ------------------------------------------------