Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

Think-off default, temp 0.7, benchmark section, prewarm (opt-in)

linyubupafdbd06832617
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000..9c3a3ad
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,33 @@
+name: CI
+
+on:
+ push:
+ branches: [main]
+ pull_request:
+
+jobs:
+ test-macos:
+ name: Unit tests (macOS + MLX)
+ runs-on: macos-14
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.12"
+ - name: Install
+ run: pip install -e '.[dev]'
+ - name: Test
+ run: pytest -m 'not slow'
+
+ hygiene:
+ name: Repo hygiene (pure Python, no MLX)
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.12"
+ - name: Install pytest
+ run: pip install pytest
+ - name: Hygiene checks
+ run: pytest tests/test_repo_hygiene.py
diff --git a/README.md b/README.md
index f28e7df..67a046b 100644
--- a/README.md
+++ b/README.md
@@ -16,6 +16,18 @@ Two model tiers work out of the box:
| `edge0-35b` | Qwen3.6-35B-A3B 4-bit (40 layers, 256 experts) | prerouter K=4 |
| `edge0-10b` | Ling-10B 4-bit bailing hybrid (24 layers, 128 experts) | prerouter K=8 |
+## Requirements
+
+- **OS / hardware**: the MLX backend runs on macOS with Apple Silicon
+ (M1/M2/M3/M4). The CUDA backend is on the roadmap — no other
+ platforms are supported yet.
+- **Python**: 3.10+ (3.12 recommended).
+- **Memory**: ~3.3 GB peak active memory for `edge0-35b`, ~1.4 GB for
+ `edge0-10b` (measured). Add headroom for the OS and tokenizer.
+- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB
+ (`edge0-10b`); expert weights are mmapped and read on demand, they are
+ not loaded into RAM up front.
+
## Design
- **transformers-style usage**: `AutoModel` / `AutoConfig` / `AutoEngine`
@@ -53,51 +65,74 @@ Two model tiers work out of the box:
## Quick start
```bash
-# 1) Install (Python >= 3.10; MLX backend requires an mlx-supported platform)
-python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
-
-# 2) Quick demo: pass a checkpoint directory or tier name
+# 1) Install (Python >= 3.10; MLX backend requires macOS + Apple Silicon)
+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
+
+# 2) Get a model: base checkpoint + trained LoRA/prerouter adapters in ONE dir.
+# (Set EDGE0_35B_REPO / EDGE0_10B_REPO to the published Hugging Face
+# repo ids, then:)
+.venv/bin/python scripts/fetch_models.py --tier edge0-35b
+.venv/bin/python scripts/fetch_models.py --tier edge0-10b
+export EDGE0_35B_MODEL=$PWD/models/edge0-35b
+export EDGE0_10B_MODEL=$PWD/models/edge0-10b
+
+# 3) Quick demo: pass a tier name or a checkpoint directory
+edge0 demo edge0-35b
edge0 demo /path/to/qwen35/model
-edge0 demo edge0-10b # checkpoint resolved via env vars (below)
-# 3) Serve (OpenAI-compatible /v1/chat/completions; the model is a
+# 4) Serve (OpenAI-compatible /v1/chat/completions; the model is a
# positional argument, tier auto-detected from config.json)
-edge0 serve /path/to/qwen35/model
edge0 serve edge0-35b
```
+```bash
+curl http://127.0.0.1:8000/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
+
+# 5) One-shot chat (pass --max-new to cap length; add --show-thinking to
+# print the model's reasoning block too)
+edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."
+```
+
`python -m edge0 ...` is equivalent to `edge0 ...`.
-Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint
-directories through environment variables; without them, the tier is
+If you already have a checkpoint on disk, just point at it — the tier is
auto-detected from the checkpoint's `config.json`:
```bash
-export EDGE0_35B_MODEL=/path/to/qwen35/model
-export EDGE0_10B_MODEL=/path/to/ling/model
+edge0 demo /path/to/model
+edge0 serve /path/to/model
```
-```bash
-curl http://127.0.0.1:8000/v1/chat/completions \
- -H 'Content-Type: application/json' \
- -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
+Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint
+directories through environment variables:
-# 4) One-shot chat
-edge0 chat /path/to/model --prompt "Explain streaming inference in one sentence."
+```bash
+export EDGE0_35B_MODEL=/path/to/qwen35/model
+export EDGE0_10B_MODEL=/path/to/ling/model
```
### Python API
```python
from edge0 import AutoEngine
-
-eng = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected
-ids = eng.encode_chat([{"role": "user", "content": "Hello"}], think=True)
-tokens = eng.generate(ids, max_new_tokens=512)
-print(eng._tok.decode(tokens))
-eng.reset() # clear per-request state (cross-request KV / prerouter cache)
+from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
+
+engine = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected
+req = ChatRequest(
+ model=engine.name,
+ messages=[ChatMessage(role="user", content="Hello!")],
+ max_tokens=64,
+)
+tokens, meta = ChatSession(engine, req).run()
+print(engine._tok.decode(tokens))
+engine.close() # release mmaps / expert cache
```
+`examples/demo.py` is the same minimal walkthrough (`edge0 demo` runs
+this exact path).
+
### Models and adapters
- **Checkpoint**: the original model directory (`config.json`,
@@ -110,8 +145,12 @@ eng.reset() # clear per-request state (cross-request KV / prerouter cache)
`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
- `artifacts/` (repo root, gitignored): convert once from
training-side npz exports via `edge0 convert-adapters --npz-dir ...`.
-- Current default adapter rounds: 35b = round9, 10b = round6
- (pgstart-sel1m, owners L7–22 matching `start_layer=7`).
+- The published model repos bundle both the base checkpoint and the
+ current default adapters (35b = round9, 10b = round6), so
+ `scripts/fetch_models.py` produces a ready-to-run model directory.
+- Both adapters are required for the prerouter + LoRA pipeline; if a
+ file is missing, `edge0` fails with a clear message (or pass
+ `--no-prerouter` / `--no-lora` to run the plain base model).
### Stability validation
@@ -124,6 +163,27 @@ multiple prompts):
- cross-request state pollution: 0/N (per-request `reset()` +
first-token greedy).
+## Benchmark
+
+Measured with `examples/bench.py` (prefill → 10 sampled warmup steps → 200
+timed sampled decode tokens, 2 runs per tier):
+
+| Tier | Decode speed | Peak active memory | Test machine |
+|---|---|---|---|
+| `edge0-35b` | 14.9–17.7 tok/s | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-10b` | 25.3 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+
+*Peak active memory is the MLX allocator's peak (model weights + KV cache +
+expert working set), not RSS: expert weights stream from SSD via mmap and the
+OS page cache is not counted.*
+
+Reproduce:
+
+```bash
+python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL
+python examples/bench.py edge0-10b # via $EDGE0_10B_MODEL
+```
+
## Tests
```bash
@@ -143,4 +203,4 @@ examples/demo.py # minimal API walkthrough
## License
-Apache-2.0, including vendored third-party code (see [NOTICE](NOTICE.md)).
+Apache-2.0, including vendored third-party code (see [NOTICE](NOTICE)).
diff --git a/README_zh.md b/README_zh.md
index 8e28fe9..b4095d0 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -13,6 +13,16 @@
| `edge0-35b` | Qwen3.6-35B-A3B 4bit(40 层,256 专家) | prerouter K=4 |
| `edge0-10b` | Ling-10B 4bit(24 层,128 专家) | prerouter K=8 |
+## 环境要求
+
+- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS
+ (M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。
+- **Python**:3.10+(推荐 3.12)。
+- **内存**:`edge0-35b` 峰值激活内存约 3.3 GB,`edge0-10b` 约 1.4 GB
+ (实测);另需为系统与 tokenizer 预留余量。
+- **磁盘**:4bit checkpoint 约 23 GB(`edge0-35b`)/ 4.2 GB
+ (`edge0-10b`);专家权重 mmap 按需读取,不一次性载入内存。
+
## 设计
- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`
@@ -43,30 +53,45 @@
## 快速开始
```bash
-# 1) 安装(Python ≥3.10;MLX 后端需 mlx 环境当前支持的平台)
-python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'
-
-# 2) 快速演示:指定 checkpoint 目录或 tier 名(tier 路径见环境变量)
+# 1) 安装(Python ≥3.10;MLX 后端需 macOS + Apple Silicon)
+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'
+
+# 2) 下载模型:基模 + 训练好的 LoRA/prerouter 适配器在一个目录里
+# (把 EDGE0_35B_REPO / EDGE0_10B_REPO 设为发布的 Hugging Face 仓库 id)
+.venv/bin/python scripts/fetch_models.py --tier edge0-35b
+.venv/bin/python scripts/fetch_models.py --tier edge0-10b
+export EDGE0_35B_MODEL=$PWD/models/edge0-35b
+export EDGE0_10B_MODEL=$PWD/models/edge0-10b
+
+# 3) 快速演示:指定 tier 名或 checkpoint 目录
+edge0 demo edge0-35b
edge0 demo /path/to/qwen35/model
-edge0 demo edge0-10b # 由 EDGE0_10B_MODEL 指向的 checkpoint
-# 3) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,
+# 4) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,
# checkpoint 类型按 config.json 自动识别)
-edge0 serve /path/to/qwen35/model
-edge0 serve edge0-35b # tier 名,checkpoint 路径同上由环境变量解析
+edge0 serve edge0-35b
+```
+```bash
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'
-# 4) 命令行一问一答(同样接受路径或 tier 名)
-edge0 chat /path/to/model --prompt "用一句话介绍流式推理。"
+# 5) 命令行一问一答(用 --max-new 控制长度,加 --show-thinking 打印思考过程)
+edge0 chat edge0-35b --prompt "用一句话介绍流式推理。"
```
`python -m edge0 ...` 与 `edge0 ...` 完全等价。
+如果本机已有 checkpoint,直接传路径即可,tier 按 `config.json` 自动识别:
+
+```bash
+edge0 demo /path/to/model
+edge0 serve /path/to/model
+```
+
tier 名(`edge0-35b` / `edge0-10b`)通过环境变量解析到本机 checkpoint
-目录,二者都未设置时按目录内 `config.json` 自动识别:
+目录:
```bash
export EDGE0_35B_MODEL=/path/to/qwen35/model
@@ -77,14 +102,21 @@ export EDGE0_10B_MODEL=/path/to/ling/model
```python
from edge0 import AutoEngine
-
-eng = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别
-ids = eng.encode_chat([{"role": "user", "content": "你好"}], think=True)
-tokens = eng.generate(ids, max_new_tokens=512)
-print(eng._tok.decode(tokens))
-eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)
+from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
+
+engine = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别
+req = ChatRequest(
+ model=engine.name,
+ messages=[ChatMessage(role="user", content="你好!")],
+ max_tokens=64,
+)
+tokens, meta = ChatSession(engine, req).run()
+print(engine._tok.decode(tokens))
+engine.close() # 释放 mmap / 专家缓存
```
+`examples/demo.py` 就是这条最小路径(`edge0 demo` 内部等价运行)。
+
### 模型与适配器
- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、
@@ -95,8 +127,10 @@ eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)
`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;
- `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从
训练侧 npz 一次性转换。
-- 当前默认适配器轮次:35b = round9,10b = round6(pgstart-sel1m,
- owners L7–22 与 `start_layer=7` 一致)。
+- 发布的模型仓库同时包含基模与当前默认适配器(35b = round9,
+ 10b = round6),`scripts/fetch_models.py` 下载后即为可运行的模型目录。
+- prerouter + LoRA 两条适配器都必需;缺文件时 `edge0` 会给出明确报错
+ (也可加 `--no-prerouter` / `--no-lora` 直接跑裸基模)。
### 稳定性验证
@@ -106,6 +140,25 @@ eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)
- 碎片退化 0/N(低层 prerouter 噪声通过 `start_layer=7` 消除);
- 跨请求状态污染 0/N(每请求 `reset()` + 首 token greedy)。
+## 性能实测
+
+`examples/bench.py` 实测(prefill → 10 步采样 warmup → 200 token 计时段,每档 2 轮):
+
+| 档位 | 解码速度 | 峰值 active 内存 | 测试机器 |
+|---|---|---|---|
+| `edge0-35b` | 14.9–17.7 tok/s | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |
+| `edge0-10b` | 25.3 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |
+
+*峰值 active 内存为 MLX allocator 的峰值(权重 + KV cache + 专家工作集),
+不含 RSS:专家权重经 mmap 从 SSD 流式读取,OS 页缓存不计入。*
+
+复现:
+
+```bash
+python examples/bench.py edge0-35b # 经 $EDGE0_35B_MODEL
+python examples/bench.py edge0-10b # 经 $EDGE0_10B_MODEL
+```
+
## 验证
```bash
@@ -125,4 +178,4 @@ examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)
## License
-Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE.md))。
+Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE))。
diff --git a/pyproject.toml b/pyproject.toml
index 14b90f7..5186b0b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,6 +38,9 @@ dependencies = [
dev = [
"pytest>=8.0",
]
+fetch = [
+ "huggingface_hub>=0.23",
+]
[project.scripts]
edge0 = "edge0.cli:main"
diff --git a/scripts/fetch_models.py b/scripts/fetch_models.py
new file mode 100755
index 0000000..e8c35a4
--- /dev/null
+++ b/scripts/fetch_models.py
@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+"""Download a tier's model directory (base checkpoint + trained adapters).
+
+The published Hugging Face repos co-locate the base checkpoint and the
+trained LoRA / prerouter adapters in ONE directory, so a single snapshot
+download produces a ready-to-run model directory that ``edge0 demo`` /
+``edge0 serve`` accept directly.
+
+Requirements:
+ pip install 'edge0[fetch]' # or: pip install huggingface_hub
+
+Repo ids come from the environment (set them once, e.g. in ~/.zshrc):
+
+ export EDGE0_35B_REPO=<your-hf-org>/edge0-35b
+ export EDGE0_10B_REPO=<your-hf-org>/edge0-10b
+
+Usage:
+ python scripts/fetch_models.py --tier edge0-35b --target-dir models
+ python scripts/fetch_models.py --tier all --target-dir models
+
+Afterwards point the tier names at what you downloaded:
+
+ export EDGE0_35B_MODEL=$PWD/models/edge0-35b
+ edge0 demo edge0-35b
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+TIER_REPOS = {
+ "edge0-35b": "EDGE0_35B_REPO",
+ "edge0-10b": "EDGE0_10B_REPO",
+}
+
+
+def _repo_for(tier: str) -> str:
+ env = TIER_REPOS[tier]
+ repo = os.environ.get(env, "")
+ if not repo:
+ raise SystemExit(
+ f"[fetch] {tier} needs a repo id.\n"
+ f"Set {env} to your Hugging Face repo, e.g.\n"
+ f" export {env}=<your-hf-org>/{tier}\n"
+ f"then re-run this script.")
+ return repo
+
+
+def _download(tier: str, target_dir: Path) -> None:
+ try:
+ from huggingface_hub import snapshot_download
+ except ImportError:
+ raise SystemExit(
+ "[fetch] huggingface_hub is required: pip install 'edge0[fetch]' "
+ "or pip install huggingface_hub")
+ repo = _repo_for(tier)
+ dest = target_dir / tier
+ print(f"[fetch] {tier} <- {repo} (-> {dest})", flush=True)
+ snapshot_download(repo_id=repo, local_dir=str(dest))
+ print(f"[fetch] done: {dest}\n"
+ f" export {TIER_REPOS[tier].replace('_REPO','_MODEL')}"
+ f"={dest}")
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__)
+ ap.add_argument("--tier", required=True,
+ choices=list(TIER_REPOS) + ["all"])
+ ap.add_argument("--target-dir", default="models",
+ help="directory to write <tier>/ under (default: models)")
+ args = ap.parse_args()
+
+ target = Path(args.target_dir).expanduser()
+ target.mkdir(parents=True, exist_ok=True)
+ tiers = list(TIER_REPOS) if args.tier == "all" else [args.tier]
+ for tier in tiers:
+ _download(tier, target)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/src/edge0/adapters/lora.py b/src/edge0/adapters/lora.py
index 93b5beb..b13bf32 100644
--- a/src/edge0/adapters/lora.py
+++ b/src/edge0/adapters/lora.py
@@ -23,6 +23,8 @@ migration (see ``scripts/convert_adapters_legacy.py``).
from __future__ import annotations
+import os
+
from edge0.backends import core
from edge0.backends import nn
@@ -76,6 +78,13 @@ def install_lora(model: nn.Module, adapters_path: str,
Returns ``{applied: [...], not_found: [...], skipped: [...],
scale: float, dtype: str}``.
"""
+ if not os.path.isfile(adapters_path):
+ raise FileNotFoundError(
+ f"LoRA adapter file not found: {adapters_path}\n"
+ "edge0 tiers are shipped with trained LoRA + prerouter "
+ "adapters; download them for this tier (README -> 'Getting "
+ "the models & adapters') and place them in the model "
+ "directory, or disable LoRA with lora="" / --no-lora.")
lora = load_safetensors(adapters_path)
if not lora:
raise ValueError("adapters has no lora tensors")
diff --git a/src/edge0/cli.py b/src/edge0/cli.py
index b8a863a..580b223 100644
--- a/src/edge0/cli.py
+++ b/src/edge0/cli.py
@@ -13,6 +13,7 @@ from __future__ import annotations
import argparse
import os
+import re
import sys
from edge0 import models # noqa: F401 (populates MODEL_REGISTRY)
@@ -101,6 +102,24 @@ def _missing_model_help(name) -> str:
)
+def _strip_thinking(text: str) -> str:
+ """Strip a leading `` thinking... response`` reasoning block for display.
+
+ The qwen chat template defaults to thinking mode, so a raw generation
+ echoes the reasoning chain before the ``response`` marker. The demo
+ and chat commands show only the final answer unless ``--show-thinking``
+ is given; the engine/server output is never modified.
+ """
+ m = re.search(r"\n\s*response\b", text)
+ if m:
+ return text[m.end():].strip()
+ return text.strip()
+
+
+def _display_text(text: str, show_thinking: bool) -> str:
+ return text if show_thinking else _strip_thinking(text)
+
+
def cmd_demo(args) -> int:
from edge0 import AutoEngine
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
@@ -117,11 +136,12 @@ def cmd_demo(args) -> int:
raise SystemExit("model has no tokenizer; cannot demo")
prompt = args.prompt or DEMO_PROMPTS.get(engine.name, "Hello!")
req = ChatRequest(model=engine.name, messages=[
- ChatMessage(role="user", content=prompt)])
+ ChatMessage(role="user", content=prompt)],
+ max_tokens=args.max_new)
sess = ChatSession(engine, req)
tokens, meta = sess.run()
print(f"user : {prompt}")
- print(f"edge0: {tok.decode(tokens)}")
+ print(f"edge0: {_display_text(tok.decode(tokens), args.show_thinking)}")
print(f"# {len(tokens)} tokens in {meta['wall_s']}s",
file=sys.stderr)
engine.close()
@@ -150,10 +170,11 @@ def cmd_chat(args) -> int:
for p in prompts:
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession
req = ChatRequest(model=name, messages=[
- ChatMessage(role="user", content=p)])
+ ChatMessage(role="user", content=p)],
+ max_tokens=args.max_new)
sess = ChatSession(engine, req)
tokens, meta = sess.run()
- print(tok.decode(tokens))
+ print(_display_text(tok.decode(tokens), args.show_thinking))
print(f"# {len(tokens)} tokens in {meta['wall_s']}s", file=sys.stderr)
engine.close()
return 0
@@ -205,6 +226,10 @@ def main(argv: list[str] | None = None) -> int:
p.add_argument("--model-dir", default=None)
p.add_argument("--name", default=None)
p.add_argument("--prompt", default=None)
+ p.add_argument("--max-new", type=int, default=None,
+ help="max tokens to generate (default: tier config)")
+ p.add_argument("--show-thinking", action="store_true",
+ help="print the model's reasoning block too")
p.add_argument("--no-prerouter", action="store_true")
p.add_argument("--no-lora", action="store_true")
p.set_defaults(fn=cmd_demo)
@@ -217,6 +242,10 @@ def main(argv: list[str] | None = None) -> int:
p.add_argument("--model-dir", default=None)
p.add_argument("--name", default=None)
p.add_argument("--prompt", default=None)
+ p.add_argument("--max-new", type=int, default=None,
+ help="max tokens to generate (default: tier config)")
+ p.add_argument("--show-thinking", action="store_true",
+ help="print the model's reasoning block too")
p.add_argument("--no-prerouter", action="store_true")
p.add_argument("--no-lora", action="store_true")
p.set_defaults(fn=cmd_chat)
diff --git a/src/edge0/engine/ling.py b/src/edge0/engine/ling.py
index b0f3936..cb9c41c 100644
--- a/src/edge0/engine/ling.py
+++ b/src/edge0/engine/ling.py
@@ -129,6 +129,45 @@ class Ling10BEngine(Edge0Engine):
self._all_stream_layers, enabled=cfg.prefetch_history)
self.cache = self.model.make_cache()
+ # start_server.sh LING_PREWARM=1 parity (opt-in): warm the OS page
+ # cache over the whole checkpoint (madvise + sequential read) and
+ # run a tiny dummy prefill+step so the first real request runs at
+ # near-steady-state speed (kernel JIT + LRU + hot pins warm).
+ if os.environ.get("EDGE0_PREWARM", os.environ.get(
+ "LING_PREWARM", "0")) == "1":
+ self._prewarm()
+
+ # ---- startup warm-up --------------------------------------------------
+
+ def _prewarm(self):
+ """Page-cache + kernel warm-up (ling v7 ``_prewarm`` parity).
+
+ 1. madvise(WILLNEED) + a full sequential read over every shard —
+ removes per-expert page-fault cost from the first request.
+ 2. A dummy prefill + one decode step — materializes attention /
+ router / shared weights, warms the LRU and hot pins, and
+ pre-compiles the single-token decode kernels.
+ KV state is reset afterwards; only page-cache warmth remains.
+ """
+ import time as _t
+ t0 = _t.perf_counter()
+ try:
+ for shard in self.shards:
+ shard.advise_willneed() if hasattr(
+ shard, "advise_willneed") else None
+ shard.seq_read() if hasattr(shard, "seq_read") else None
+ except Exception: # noqa: BLE001 — advisory only
+ pass
+ dummy = [self._tok.bos_token_id or 0] if self._tok else [0]
+ try:
+ self.reset()
+ self.prefill(dummy * 4)
+ self.step(dummy[0])
+ self.reset()
+ except Exception: # noqa: BLE001 — advisory only
+ pass
+ print(f"[edge0-10b] prewarm done in "
+ f"{_t.perf_counter() - t0:.1f}s", flush=True)
# ---- forward ----------------------------------------------------------
diff --git a/src/edge0/engine/qwen.py b/src/edge0/engine/qwen.py
index 75286cd..78e9739 100644
--- a/src/edge0/engine/qwen.py
+++ b/src/edge0/engine/qwen.py
@@ -137,8 +137,13 @@ class Qwen35Engine(Edge0Engine):
logits = self._lm.lm_head(h[0, -1])
core.eval(logits)
# Step boundary: ONE stacked head batch -> ONE tolist -> fills.
- if (self._pg_stager is not None and not self._prefill_active
- and len(ids) == 1):
+ # Demo parity (engine_qwen._forward: `if (pregate or trained_state)
+ # and not full_layer`): stage_all+swap run after EVERY forward,
+ # including prefill chunks — so the last prefill token's
+ # predictions fill pred_inds and oh_prev, and the FIRST decode
+ # step consumes a real prediction at L7 instead of falling back
+ # to the router (a different, cleaner-but-divergent trajectory).
+ if (self._pg_stager is not None and not full_layer):
self._pg_stager.stage_all()
self._pg_state.swap()
return logits
diff --git a/src/edge0/models/edge0_35b/__init__.py b/src/edge0/models/edge0_35b/__init__.py
index d60b86c..130a839 100644
--- a/src/edge0/models/edge0_35b/__init__.py
+++ b/src/edge0/models/edge0_35b/__init__.py
@@ -57,7 +57,7 @@ class Qwen35Config(ModelConfig):
lora=artifact("lora_edge0_35b.safetensors", model_dir),
lora_r=16, lora_alpha=32.0,
gen=GenerationConfig(
- temperature=0.6, top_p=0.95, top_k=64,
+ temperature=0.7, top_p=0.95, top_k=64,
repetition_penalty=1.0, max_new_tokens=2048,
eos_ids=(248046, 248044)),
prefill_chunk=2048,
diff --git a/src/edge0/prerouter/install.py b/src/edge0/prerouter/install.py
index 13fa101..39976b6 100644
--- a/src/edge0/prerouter/install.py
+++ b/src/edge0/prerouter/install.py
@@ -73,6 +73,15 @@ def install_prerouter(
if not pspec.weights_file:
raise ValueError(
"install_prerouter needs weights or PrerouterSpec.weights_file")
+ import os
+ if not os.path.isfile(pspec.weights_file):
+ raise FileNotFoundError(
+ f"prerouter weights file not found: {pspec.weights_file}\n"
+ "edge0 tiers are shipped with trained LoRA + prerouter "
+ "adapters; download them for this tier (README -> 'Getting "
+ "the models & adapters') and place them in the model "
+ "directory, or disable the prerouter with "
+ "prerouter=None / --no-prerouter.")
weights = load_safetensors(pspec.weights_file)
dtype = core.float16 if pspec.dtype == "fp16" else core.float32
head_weights = _parse_weights(weights, dtype)
diff --git a/src/edge0/server/chat.py b/src/edge0/server/chat.py
index 3babc68..dc9e056 100644
--- a/src/edge0/server/chat.py
+++ b/src/edge0/server/chat.py
@@ -84,17 +84,25 @@ class ChatSession:
return encode(
[m.__dict__ for m in self.req.messages], think=bool(think))
if hasattr(tok, "apply_chat_template"):
- # Deployment parity with the correct demo (serve.py
- # encode_chat): the qwen35 template's default IS the thinking
- # mode (<think> opener). Passing enable_thinking=False on
- # this template renders an EMPTY <think>\n\n</think> block
- # instead of omitting thinking — a different prompt than the
- # demo, so it must not be passed.
- text = tok.apply_chat_template(
- [m.__dict__ for m in self.req.messages],
- tokenize=False, add_generation_prompt=True)
- else:
- text = self._chat_text()
+ # The qwen35 template supports ``enable_thinking``: False
+ # renders the canonical no-think prompt — an EMPTY think
+ # block closer, the model's direct-answer form. Default OFF
+ # (CLI demo parity).
+ think = self.req.enable_thinking
+ if think is None:
+ think = False
+ try:
+ text = tok.apply_chat_template(
+ [m.__dict__ for m in self.req.messages],
+ tokenize=False, add_generation_prompt=True,
+ enable_thinking=bool(think))
+ except TypeError:
+ # tokenizer template without the kwarg: plain render
+ text = tok.apply_chat_template(
+ [m.__dict__ for m in self.req.messages],
+ tokenize=False, add_generation_prompt=True)
+ else:
+ text = self._chat_text()
ids = tok.encode(text)
if not ids:
ids = [tok.bos_token_id or 0]
diff --git a/src/edge0/streaming/mmap.py b/src/edge0/streaming/mmap.py
index 3ea9441..86ce369 100644
--- a/src/edge0/streaming/mmap.py
+++ b/src/edge0/streaming/mmap.py
@@ -40,6 +40,25 @@ class SafetensorsMmap:
"shape": tuple(meta["shape"]),
}
+ def advise_willneed(self):
+ """OS readahead hint over the whole shard (advisory)."""
+ try:
+ mmap_madv = getattr(mmap, "MADV_WILLNEED", None)
+ if mmap_madv is not None:
+ self._mm.madvise(mmap_madv)
+ except Exception: # noqa: BLE001 — advisory only
+ pass
+
+ def seq_read(self, chunk: int = 1 << 24):
+ """Force every page resident: one sequential pass over the shard.
+
+ macOS madvise only warms roughly half the file, so a full read is
+ the reliable way to remove per-expert page-fault cost from the
+ first request (ling v7 ``_prewarm`` step 2 parity)."""
+ total = len(self._mm)
+ for off in range(0, total, chunk):
+ _ = self._mm[off:off + chunk]
+
def raw(self, name: str) -> np.ndarray:
e = self.entries[name]
return np.frombuffer(
diff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.py
index fd31ec8..59deba8 100644
--- a/src/edge0/streaming/options.py
+++ b/src/edge0/streaming/options.py
@@ -124,7 +124,8 @@ class LayerOptions:
return cls(
staged=False, staged_replace=False, staged_n=8,
staged_trigger=8, staged_sync=False, history_prefetch=False,
- hot_per_layer=0, cache_slots=64, prefetch_cap=48,
+ hot_per_layer=0,
+ cache_slots=64, prefetch_cap=48,
load_threads=8, prefetch_threads=4, use_compile=True, top_k=8,
full_layer_prefill=True, prefill_hot=0, **overrides)
diff --git a/tests/test_registry.py b/tests/test_registry.py
index 285c533..6cdcb47 100644
--- a/tests/test_registry.py
+++ b/tests/test_registry.py
@@ -57,7 +57,7 @@ def test_qwen35_profile():
assert cfg.prerouter.hidden == 512
assert cfg.prerouter.dtype == "fp16"
assert cfg.prerouter.feature_topk == "executed"
- assert cfg.gen.temperature == 0.6
+ assert cfg.gen.temperature == 0.7
assert 248046 in cfg.gen.eos_ids
assert cfg.port == 8085
diff --git a/tests/test_repo_hygiene.py b/tests/test_repo_hygiene.py
new file mode 100644
index 0000000..c8ef1d0
--- /dev/null
+++ b/tests/test_repo_hygiene.py
@@ -0,0 +1,116 @@
+"""Repository-hygiene guards for a clean open-source release.
+
+These tests import nothing from ``edge0`` (and nothing from ``mlx``), so
+they can run on any platform — including the Linux CI job where MLX has no
+wheels. They enforce the "no dependency on the author's machine" rule:
+
+* no personal / hardcoded absolute paths in shipped code, docs or scripts;
+* the MLX backend boundary is respected (``import mlx`` only inside
+ ``edge0/backends/mlx/``);
+* no obvious secrets (API keys / private keys) in the tree;
+* README/NOTICE references resolve.
+"""
+
+from __future__ import annotations
+
+import pathlib
+import re
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+
+_EXCLUDE_DIRS = {".git", ".venv", ".pytest_cache", "__pycache__", ".agents",
+ ".codex"}
+_EXCLUDE_PARTS = {"egg-info"}
+_EXCLUDE_SUFFIXES = {".pyc", ".so", ".dylib", ".bin", ".safetensors", ".npz",
+ ".png", ".jpg", ".jpeg", ".gif", ".ico"}
+
+# NOTE: patterns are assembled with + so this file never contains the
+# literal string it is looking for.
+_USR = "Users"
+_NAS = "nas"
+_HOME = "home"
+_ROOT = "root"
+_LOCAL_PATH_PATTERNS = [
+ re.compile(r"/" + _USR + r"/[\w.-]+/"),
+ re.compile(r"/" + _NAS + r"/"),
+ re.compile(r"/" + _HOME + r"/[\w.-]+/"),
+ re.compile(r"/" + _ROOT + r"(?:/|$)"),
+ re.compile(r"C:" + re.escape("\\") + _USR + re.escape("\\")),
+ re.compile(r"HPC-\d"),
+ re.compile(r"ali-pod\d"),
+]
+
+_MLX_IMPORT = re.compile(r"^\s*(?:from\s+mlx(?:\s|\.)|import\s+mlx(?:\s|\.))",
+ re.MULTILINE)
+
+_SECRET_PATTERNS = [
+ re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b"),
+ re.compile(r"\bghp_[A-Za-z0-9]{20,}\b"),
+ re.compile(r"AKIA[0-9A-Z]{16}"),
+ re.compile(r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----"),
+]
+
+
+def _iter_text_files():
+ for path in ROOT.rglob("*"):
+ rel = path.relative_to(ROOT)
+ if any(part in _EXCLUDE_DIRS for part in rel.parts):
+ continue
+ if any(part in _EXCLUDE_PARTS for part in rel.parts):
+ continue
+ if path.suffix in _EXCLUDE_SUFFIXES or path.is_dir():
+ continue
+ yield path
+
+
+def _read(path: pathlib.Path) -> str:
+ try:
+ return path.read_text(encoding="utf-8", errors="replace")
+ except OSError:
+ return ""
+
+
+def test_no_hardcoded_local_paths():
+ hits = []
+ for path in _iter_text_files():
+ text = _read(path)
+ if not text:
+ continue
+ for pat in _LOCAL_PATH_PATTERNS:
+ m = pat.search(text)
+ if m:
+ hits.append(f"{path.relative_to(ROOT)}: {m.group(0)!r}")
+ assert not hits, "hardcoded local paths found:\n" + "\n".join(hits)
+
+
+def test_mlx_imports_stay_inside_the_backend():
+ bad = []
+ for path in (ROOT / "src" / "edge0").rglob("*.py"):
+ rel = path.relative_to(ROOT)
+ text = path.read_text(encoding="utf-8")
+ if _MLX_IMPORT.search(text) and "backends/mlx" not in rel.as_posix():
+ bad.append(str(rel))
+ assert not bad, (
+ "MLX imports must live under edge0/backends/mlx/ only:\n"
+ + "\n".join(bad))
+
+
+def test_no_secrets():
+ hits = []
+ for path in _iter_text_files():
+ text = _read(path)
+ if not text:
+ continue
+ for pat in _SECRET_PATTERNS:
+ m = pat.search(text)
+ if m:
+ hits.append(f"{path.relative_to(ROOT)}: {m.group(0)!r}")
+ assert not hits, "possible secrets found:\n" + "\n".join(hits)
+
+
+def test_notice_link_resolves():
+ readme = (ROOT / "README.md").read_text(encoding="utf-8")
+ m = re.search(r"\[NOTICE\]\(([^)]+)\)", readme)
+ assert m, "README should link the NOTICE file"
+ target = ROOT / m.group(1)
+ assert target.is_file(), f"README NOTICE link target missing: {target}"