Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
Think-off default, temp 0.7, benchmark section, prewarm (opt-in)
linyubupafdbd06832617
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlnew file mode 100644index 0000000..9c3a3ad--- /dev/null+++ b/.github/workflows/ci.yml@@ -0,0 +1,33 @@+name: CI++on:+ push:+ branches: [main]+ pull_request:++jobs:+ test-macos:+ name: Unit tests (macOS + MLX)+ runs-on: macos-14+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: "3.12"+ - name: Install+ run: pip install -e '.[dev]'+ - name: Test+ run: pytest -m 'not slow'++ hygiene:+ name: Repo hygiene (pure Python, no MLX)+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: "3.12"+ - name: Install pytest+ run: pip install pytest+ - name: Hygiene checks+ run: pytest tests/test_repo_hygiene.pydiff --git a/README.md b/README.mdindex f28e7df..67a046b 100644--- a/README.md+++ b/README.md@@ -16,6 +16,18 @@ Two model tiers work out of the box:| `edge0-35b` | Qwen3.6-35B-A3B 4-bit (40 layers, 256 experts) | prerouter K=4 || `edge0-10b` | Ling-10B 4-bit bailing hybrid (24 layers, 128 experts) | prerouter K=8 |+## Requirements++- **OS / hardware**: the MLX backend runs on macOS with Apple Silicon+ (M1/M2/M3/M4). The CUDA backend is on the roadmap — no other+ platforms are supported yet.+- **Python**: 3.10+ (3.12 recommended).+- **Memory**: ~3.3 GB peak active memory for `edge0-35b`, ~1.4 GB for+ `edge0-10b` (measured). Add headroom for the OS and tokenizer.+- **Disk**: the 4-bit checkpoints are ~23 GB (`edge0-35b`) and ~4.2 GB+ (`edge0-10b`); expert weights are mmapped and read on demand, they are+ not loaded into RAM up front.+## Design- **transformers-style usage**: `AutoModel` / `AutoConfig` / `AutoEngine`@@ -53,51 +65,74 @@ Two model tiers work out of the box:## Quick start```bash-# 1) Install (Python >= 3.10; MLX backend requires an mlx-supported platform)-python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'--# 2) Quick demo: pass a checkpoint directory or tier name+# 1) Install (Python >= 3.10; MLX backend requires macOS + Apple Silicon)+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'++# 2) Get a model: base checkpoint + trained LoRA/prerouter adapters in ONE dir.+# (Set EDGE0_35B_REPO / EDGE0_10B_REPO to the published Hugging Face+# repo ids, then:)+.venv/bin/python scripts/fetch_models.py --tier edge0-35b+.venv/bin/python scripts/fetch_models.py --tier edge0-10b+export EDGE0_35B_MODEL=$PWD/models/edge0-35b+export EDGE0_10B_MODEL=$PWD/models/edge0-10b++# 3) Quick demo: pass a tier name or a checkpoint directory+edge0 demo edge0-35bedge0 demo /path/to/qwen35/model-edge0 demo edge0-10b # checkpoint resolved via env vars (below)-# 3) Serve (OpenAI-compatible /v1/chat/completions; the model is a+# 4) Serve (OpenAI-compatible /v1/chat/completions; the model is a# positional argument, tier auto-detected from config.json)-edge0 serve /path/to/qwen35/modeledge0 serve edge0-35b```+```bash+curl http://127.0.0.1:8000/v1/chat/completions \+ -H 'Content-Type: application/json' \+ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'++# 5) One-shot chat (pass --max-new to cap length; add --show-thinking to+# print the model's reasoning block too)+edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."+```+`python -m edge0 ...` is equivalent to `edge0 ...`.-Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint-directories through environment variables; without them, the tier is+If you already have a checkpoint on disk, just point at it — the tier isauto-detected from the checkpoint's `config.json`:```bash-export EDGE0_35B_MODEL=/path/to/qwen35/model-export EDGE0_10B_MODEL=/path/to/ling/model+edge0 demo /path/to/model+edge0 serve /path/to/model```-```bash-curl http://127.0.0.1:8000/v1/chat/completions \- -H 'Content-Type: application/json' \- -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'+Tier names (`edge0-35b` / `edge0-10b`) resolve to local checkpoint+directories through environment variables:-# 4) One-shot chat-edge0 chat /path/to/model --prompt "Explain streaming inference in one sentence."+```bash+export EDGE0_35B_MODEL=/path/to/qwen35/model+export EDGE0_10B_MODEL=/path/to/ling/model```### Python API```pythonfrom edge0 import AutoEngine--eng = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected-ids = eng.encode_chat([{"role": "user", "content": "Hello"}], think=True)-tokens = eng.generate(ids, max_new_tokens=512)-print(eng._tok.decode(tokens))-eng.reset() # clear per-request state (cross-request KV / prerouter cache)+from edge0.server.chat import ChatMessage, ChatRequest, ChatSession++engine = AutoEngine.from_pretrained("/path/to/model") # tier auto-detected+req = ChatRequest(+ model=engine.name,+ messages=[ChatMessage(role="user", content="Hello!")],+ max_tokens=64,+)+tokens, meta = ChatSession(engine, req).run()+print(engine._tok.decode(tokens))+engine.close() # release mmaps / expert cache```+`examples/demo.py` is the same minimal walkthrough (`edge0 demo` runs+this exact path).+### Models and adapters- **Checkpoint**: the original model directory (`config.json`,@@ -110,8 +145,12 @@ eng.reset() # clear per-request state (cross-request KV / prerouter cache)`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- `artifacts/` (repo root, gitignored): convert once fromtraining-side npz exports via `edge0 convert-adapters --npz-dir ...`.-- Current default adapter rounds: 35b = round9, 10b = round6- (pgstart-sel1m, owners L7–22 matching `start_layer=7`).+- The published model repos bundle both the base checkpoint and the+ current default adapters (35b = round9, 10b = round6), so+ `scripts/fetch_models.py` produces a ready-to-run model directory.+- Both adapters are required for the prerouter + LoRA pipeline; if a+ file is missing, `edge0` fails with a clear message (or pass+ `--no-prerouter` / `--no-lora` to run the plain base model).### Stability validation@@ -124,6 +163,27 @@ multiple prompts):- cross-request state pollution: 0/N (per-request `reset()` +first-token greedy).+## Benchmark++Measured with `examples/bench.py` (prefill → 10 sampled warmup steps → 200+timed sampled decode tokens, 2 runs per tier):++| Tier | Decode speed | Peak active memory | Test machine |+|---|---|---|---|+| `edge0-35b` | 14.9–17.7 tok/s | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 25.3 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |++*Peak active memory is the MLX allocator's peak (model weights + KV cache ++expert working set), not RSS: expert weights stream from SSD via mmap and the+OS page cache is not counted.*++Reproduce:++```bash+python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL+python examples/bench.py edge0-10b # via $EDGE0_10B_MODEL+```+## Tests```bash@@ -143,4 +203,4 @@ examples/demo.py # minimal API walkthrough## License-Apache-2.0, including vendored third-party code (see [NOTICE](NOTICE.md)).+Apache-2.0, including vendored third-party code (see [NOTICE](NOTICE)).diff --git a/README_zh.md b/README_zh.mdindex 8e28fe9..b4095d0 100644--- a/README_zh.md+++ b/README_zh.md@@ -13,6 +13,16 @@| `edge0-35b` | Qwen3.6-35B-A3B 4bit(40 层,256 专家) | prerouter K=4 || `edge0-10b` | Ling-10B 4bit(24 层,128 专家) | prerouter K=8 |+## 环境要求++- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS+ (M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。+- **Python**:3.10+(推荐 3.12)。+- **内存**:`edge0-35b` 峰值激活内存约 3.3 GB,`edge0-10b` 约 1.4 GB+ (实测);另需为系统与 tokenizer 预留余量。+- **磁盘**:4bit checkpoint 约 23 GB(`edge0-35b`)/ 4.2 GB+ (`edge0-10b`);专家权重 mmap 按需读取,不一次性载入内存。+## 设计- **像 transformers 一样使用**:`AutoModel` / `AutoConfig` / `AutoEngine`@@ -43,30 +53,45 @@## 快速开始```bash-# 1) 安装(Python ≥3.10;MLX 后端需 mlx 环境当前支持的平台)-python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev]'--# 2) 快速演示:指定 checkpoint 目录或 tier 名(tier 路径见环境变量)+# 1) 安装(Python ≥3.10;MLX 后端需 macOS + Apple Silicon)+python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'++# 2) 下载模型:基模 + 训练好的 LoRA/prerouter 适配器在一个目录里+# (把 EDGE0_35B_REPO / EDGE0_10B_REPO 设为发布的 Hugging Face 仓库 id)+.venv/bin/python scripts/fetch_models.py --tier edge0-35b+.venv/bin/python scripts/fetch_models.py --tier edge0-10b+export EDGE0_35B_MODEL=$PWD/models/edge0-35b+export EDGE0_10B_MODEL=$PWD/models/edge0-10b++# 3) 快速演示:指定 tier 名或 checkpoint 目录+edge0 demo edge0-35bedge0 demo /path/to/qwen35/model-edge0 demo edge0-10b # 由 EDGE0_10B_MODEL 指向的 checkpoint-# 3) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,+# 4) 启动推理服务(OpenAI 兼容 /v1/chat/completions;模型为位置参数,# checkpoint 类型按 config.json 自动识别)-edge0 serve /path/to/qwen35/model-edge0 serve edge0-35b # tier 名,checkpoint 路径同上由环境变量解析+edge0 serve edge0-35b+```+```bashcurl http://127.0.0.1:8000/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'-# 4) 命令行一问一答(同样接受路径或 tier 名)-edge0 chat /path/to/model --prompt "用一句话介绍流式推理。"+# 5) 命令行一问一答(用 --max-new 控制长度,加 --show-thinking 打印思考过程)+edge0 chat edge0-35b --prompt "用一句话介绍流式推理。"````python -m edge0 ...` 与 `edge0 ...` 完全等价。+如果本机已有 checkpoint,直接传路径即可,tier 按 `config.json` 自动识别:++```bash+edge0 demo /path/to/model+edge0 serve /path/to/model+```+tier 名(`edge0-35b` / `edge0-10b`)通过环境变量解析到本机 checkpoint-目录,二者都未设置时按目录内 `config.json` 自动识别:+目录:```bashexport EDGE0_35B_MODEL=/path/to/qwen35/model@@ -77,14 +102,21 @@ export EDGE0_10B_MODEL=/path/to/ling/model```pythonfrom edge0 import AutoEngine--eng = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别-ids = eng.encode_chat([{"role": "user", "content": "你好"}], think=True)-tokens = eng.generate(ids, max_new_tokens=512)-print(eng._tok.decode(tokens))-eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)+from edge0.server.chat import ChatMessage, ChatRequest, ChatSession++engine = AutoEngine.from_pretrained("/path/to/model") # tier 自动识别+req = ChatRequest(+ model=engine.name,+ messages=[ChatMessage(role="user", content="你好!")],+ max_tokens=64,+)+tokens, meta = ChatSession(engine, req).run()+print(engine._tok.decode(tokens))+engine.close() # 释放 mmap / 专家缓存```+`examples/demo.py` 就是这条最小路径(`edge0 demo` 内部等价运行)。+### 模型与适配器- **checkpoint**:原始模型目录(`config.json`、`model*.safetensors`、@@ -95,8 +127,10 @@ eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)`lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors`;- `artifacts/`(仓库根,gitignored):`edge0 convert-adapters` 从训练侧 npz 一次性转换。-- 当前默认适配器轮次:35b = round9,10b = round6(pgstart-sel1m,- owners L7–22 与 `start_layer=7` 一致)。+- 发布的模型仓库同时包含基模与当前默认适配器(35b = round9,+ 10b = round6),`scripts/fetch_models.py` 下载后即为可运行的模型目录。+- prerouter + LoRA 两条适配器都必需;缺文件时 `edge0` 会给出明确报错+ (也可加 `--no-prerouter` / `--no-lora` 直接跑裸基模)。### 稳定性验证@@ -106,6 +140,25 @@ eng.reset() # 每请求前清状态(跨请求 KV / prerouter 缓存)- 碎片退化 0/N(低层 prerouter 噪声通过 `start_layer=7` 消除);- 跨请求状态污染 0/N(每请求 `reset()` + 首 token greedy)。+## 性能实测++`examples/bench.py` 实测(prefill → 10 步采样 warmup → 200 token 计时段,每档 2 轮):++| 档位 | 解码速度 | 峰值 active 内存 | 测试机器 |+|---|---|---|---|+| `edge0-35b` | 14.9–17.7 tok/s | 2.9–3.1 GiB | Mac mini M4 Pro, 24 GB |+| `edge0-10b` | 25.3 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB |++*峰值 active 内存为 MLX allocator 的峰值(权重 + KV cache + 专家工作集),+不含 RSS:专家权重经 mmap 从 SSD 流式读取,OS 页缓存不计入。*++复现:++```bash+python examples/bench.py edge0-35b # 经 $EDGE0_35B_MODEL+python examples/bench.py edge0-10b # 经 $EDGE0_10B_MODEL+```+## 验证```bash@@ -125,4 +178,4 @@ examples/demo.py # 最小 API walkthrough(edge0 demo 的等价代码)## License-Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE.md))。+Apache-2.0,含 vendored 第三方代码(详见 [NOTICE](NOTICE))。diff --git a/pyproject.toml b/pyproject.tomlindex 14b90f7..5186b0b 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -38,6 +38,9 @@ dependencies = [dev = ["pytest>=8.0",]+fetch = [+ "huggingface_hub>=0.23",+][project.scripts]edge0 = "edge0.cli:main"diff --git a/scripts/fetch_models.py b/scripts/fetch_models.pynew file mode 100755index 0000000..e8c35a4--- /dev/null+++ b/scripts/fetch_models.py@@ -0,0 +1,85 @@+#!/usr/bin/env python3+"""Download a tier's model directory (base checkpoint + trained adapters).++The published Hugging Face repos co-locate the base checkpoint and the+trained LoRA / prerouter adapters in ONE directory, so a single snapshot+download produces a ready-to-run model directory that ``edge0 demo`` /+``edge0 serve`` accept directly.++Requirements:+ pip install 'edge0[fetch]' # or: pip install huggingface_hub++Repo ids come from the environment (set them once, e.g. in ~/.zshrc):++ export EDGE0_35B_REPO=<your-hf-org>/edge0-35b+ export EDGE0_10B_REPO=<your-hf-org>/edge0-10b++Usage:+ python scripts/fetch_models.py --tier edge0-35b --target-dir models+ python scripts/fetch_models.py --tier all --target-dir models++Afterwards point the tier names at what you downloaded:++ export EDGE0_35B_MODEL=$PWD/models/edge0-35b+ edge0 demo edge0-35b+"""++from __future__ import annotations++import argparse+import os+import sys+from pathlib import Path++TIER_REPOS = {+ "edge0-35b": "EDGE0_35B_REPO",+ "edge0-10b": "EDGE0_10B_REPO",+}+++def _repo_for(tier: str) -> str:+ env = TIER_REPOS[tier]+ repo = os.environ.get(env, "")+ if not repo:+ raise SystemExit(+ f"[fetch] {tier} needs a repo id.\n"+ f"Set {env} to your Hugging Face repo, e.g.\n"+ f" export {env}=<your-hf-org>/{tier}\n"+ f"then re-run this script.")+ return repo+++def _download(tier: str, target_dir: Path) -> None:+ try:+ from huggingface_hub import snapshot_download+ except ImportError:+ raise SystemExit(+ "[fetch] huggingface_hub is required: pip install 'edge0[fetch]' "+ "or pip install huggingface_hub")+ repo = _repo_for(tier)+ dest = target_dir / tier+ print(f"[fetch] {tier} <- {repo} (-> {dest})", flush=True)+ snapshot_download(repo_id=repo, local_dir=str(dest))+ print(f"[fetch] done: {dest}\n"+ f" export {TIER_REPOS[tier].replace('_REPO','_MODEL')}"+ f"={dest}")+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__)+ ap.add_argument("--tier", required=True,+ choices=list(TIER_REPOS) + ["all"])+ ap.add_argument("--target-dir", default="models",+ help="directory to write <tier>/ under (default: models)")+ args = ap.parse_args()++ target = Path(args.target_dir).expanduser()+ target.mkdir(parents=True, exist_ok=True)+ tiers = list(TIER_REPOS) if args.tier == "all" else [args.tier]+ for tier in tiers:+ _download(tier, target)+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/src/edge0/adapters/lora.py b/src/edge0/adapters/lora.pyindex 93b5beb..b13bf32 100644--- a/src/edge0/adapters/lora.py+++ b/src/edge0/adapters/lora.py@@ -23,6 +23,8 @@ migration (see ``scripts/convert_adapters_legacy.py``).from __future__ import annotations+import os+from edge0.backends import corefrom edge0.backends import nn@@ -76,6 +78,13 @@ def install_lora(model: nn.Module, adapters_path: str,Returns ``{applied: [...], not_found: [...], skipped: [...],scale: float, dtype: str}``."""+ if not os.path.isfile(adapters_path):+ raise FileNotFoundError(+ f"LoRA adapter file not found: {adapters_path}\n"+ "edge0 tiers are shipped with trained LoRA + prerouter "+ "adapters; download them for this tier (README -> 'Getting "+ "the models & adapters') and place them in the model "+ "directory, or disable LoRA with lora="" / --no-lora.")lora = load_safetensors(adapters_path)if not lora:raise ValueError("adapters has no lora tensors")diff --git a/src/edge0/cli.py b/src/edge0/cli.pyindex b8a863a..580b223 100644--- a/src/edge0/cli.py+++ b/src/edge0/cli.py@@ -13,6 +13,7 @@ from __future__ import annotationsimport argparseimport os+import reimport sysfrom edge0 import models # noqa: F401 (populates MODEL_REGISTRY)@@ -101,6 +102,24 @@ def _missing_model_help(name) -> str:)+def _strip_thinking(text: str) -> str:+ """Strip a leading `` thinking... response`` reasoning block for display.++ The qwen chat template defaults to thinking mode, so a raw generation+ echoes the reasoning chain before the ``response`` marker. The demo+ and chat commands show only the final answer unless ``--show-thinking``+ is given; the engine/server output is never modified.+ """+ m = re.search(r"\n\s*response\b", text)+ if m:+ return text[m.end():].strip()+ return text.strip()+++def _display_text(text: str, show_thinking: bool) -> str:+ return text if show_thinking else _strip_thinking(text)++def cmd_demo(args) -> int:from edge0 import AutoEnginefrom edge0.server.chat import ChatMessage, ChatRequest, ChatSession@@ -117,11 +136,12 @@ def cmd_demo(args) -> int:raise SystemExit("model has no tokenizer; cannot demo")prompt = args.prompt or DEMO_PROMPTS.get(engine.name, "Hello!")req = ChatRequest(model=engine.name, messages=[- ChatMessage(role="user", content=prompt)])+ ChatMessage(role="user", content=prompt)],+ max_tokens=args.max_new)sess = ChatSession(engine, req)tokens, meta = sess.run()print(f"user : {prompt}")- print(f"edge0: {tok.decode(tokens)}")+ print(f"edge0: {_display_text(tok.decode(tokens), args.show_thinking)}")print(f"# {len(tokens)} tokens in {meta['wall_s']}s",file=sys.stderr)engine.close()@@ -150,10 +170,11 @@ def cmd_chat(args) -> int:for p in prompts:from edge0.server.chat import ChatMessage, ChatRequest, ChatSessionreq = ChatRequest(model=name, messages=[- ChatMessage(role="user", content=p)])+ ChatMessage(role="user", content=p)],+ max_tokens=args.max_new)sess = ChatSession(engine, req)tokens, meta = sess.run()- print(tok.decode(tokens))+ print(_display_text(tok.decode(tokens), args.show_thinking))print(f"# {len(tokens)} tokens in {meta['wall_s']}s", file=sys.stderr)engine.close()return 0@@ -205,6 +226,10 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--model-dir", default=None)p.add_argument("--name", default=None)p.add_argument("--prompt", default=None)+ p.add_argument("--max-new", type=int, default=None,+ help="max tokens to generate (default: tier config)")+ p.add_argument("--show-thinking", action="store_true",+ help="print the model's reasoning block too")p.add_argument("--no-prerouter", action="store_true")p.add_argument("--no-lora", action="store_true")p.set_defaults(fn=cmd_demo)@@ -217,6 +242,10 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--model-dir", default=None)p.add_argument("--name", default=None)p.add_argument("--prompt", default=None)+ p.add_argument("--max-new", type=int, default=None,+ help="max tokens to generate (default: tier config)")+ p.add_argument("--show-thinking", action="store_true",+ help="print the model's reasoning block too")p.add_argument("--no-prerouter", action="store_true")p.add_argument("--no-lora", action="store_true")p.set_defaults(fn=cmd_chat)diff --git a/src/edge0/engine/ling.py b/src/edge0/engine/ling.pyindex b0f3936..cb9c41c 100644--- a/src/edge0/engine/ling.py+++ b/src/edge0/engine/ling.py@@ -129,6 +129,45 @@ class Ling10BEngine(Edge0Engine):self._all_stream_layers, enabled=cfg.prefetch_history)self.cache = self.model.make_cache()+ # start_server.sh LING_PREWARM=1 parity (opt-in): warm the OS page+ # cache over the whole checkpoint (madvise + sequential read) and+ # run a tiny dummy prefill+step so the first real request runs at+ # near-steady-state speed (kernel JIT + LRU + hot pins warm).+ if os.environ.get("EDGE0_PREWARM", os.environ.get(+ "LING_PREWARM", "0")) == "1":+ self._prewarm()++ # ---- startup warm-up --------------------------------------------------++ def _prewarm(self):+ """Page-cache + kernel warm-up (ling v7 ``_prewarm`` parity).++ 1. madvise(WILLNEED) + a full sequential read over every shard —+ removes per-expert page-fault cost from the first request.+ 2. A dummy prefill + one decode step — materializes attention /+ router / shared weights, warms the LRU and hot pins, and+ pre-compiles the single-token decode kernels.+ KV state is reset afterwards; only page-cache warmth remains.+ """+ import time as _t+ t0 = _t.perf_counter()+ try:+ for shard in self.shards:+ shard.advise_willneed() if hasattr(+ shard, "advise_willneed") else None+ shard.seq_read() if hasattr(shard, "seq_read") else None+ except Exception: # noqa: BLE001 — advisory only+ pass+ dummy = [self._tok.bos_token_id or 0] if self._tok else [0]+ try:+ self.reset()+ self.prefill(dummy * 4)+ self.step(dummy[0])+ self.reset()+ except Exception: # noqa: BLE001 — advisory only+ pass+ print(f"[edge0-10b] prewarm done in "+ f"{_t.perf_counter() - t0:.1f}s", flush=True)# ---- forward ----------------------------------------------------------diff --git a/src/edge0/engine/qwen.py b/src/edge0/engine/qwen.pyindex 75286cd..78e9739 100644--- a/src/edge0/engine/qwen.py+++ b/src/edge0/engine/qwen.py@@ -137,8 +137,13 @@ class Qwen35Engine(Edge0Engine):logits = self._lm.lm_head(h[0, -1])core.eval(logits)# Step boundary: ONE stacked head batch -> ONE tolist -> fills.- if (self._pg_stager is not None and not self._prefill_active- and len(ids) == 1):+ # Demo parity (engine_qwen._forward: `if (pregate or trained_state)+ # and not full_layer`): stage_all+swap run after EVERY forward,+ # including prefill chunks — so the last prefill token's+ # predictions fill pred_inds and oh_prev, and the FIRST decode+ # step consumes a real prediction at L7 instead of falling back+ # to the router (a different, cleaner-but-divergent trajectory).+ if (self._pg_stager is not None and not full_layer):self._pg_stager.stage_all()self._pg_state.swap()return logitsdiff --git a/src/edge0/models/edge0_35b/__init__.py b/src/edge0/models/edge0_35b/__init__.pyindex d60b86c..130a839 100644--- a/src/edge0/models/edge0_35b/__init__.py+++ b/src/edge0/models/edge0_35b/__init__.py@@ -57,7 +57,7 @@ class Qwen35Config(ModelConfig):lora=artifact("lora_edge0_35b.safetensors", model_dir),lora_r=16, lora_alpha=32.0,gen=GenerationConfig(- temperature=0.6, top_p=0.95, top_k=64,+ temperature=0.7, top_p=0.95, top_k=64,repetition_penalty=1.0, max_new_tokens=2048,eos_ids=(248046, 248044)),prefill_chunk=2048,diff --git a/src/edge0/prerouter/install.py b/src/edge0/prerouter/install.pyindex 13fa101..39976b6 100644--- a/src/edge0/prerouter/install.py+++ b/src/edge0/prerouter/install.py@@ -73,6 +73,15 @@ def install_prerouter(if not pspec.weights_file:raise ValueError("install_prerouter needs weights or PrerouterSpec.weights_file")+ import os+ if not os.path.isfile(pspec.weights_file):+ raise FileNotFoundError(+ f"prerouter weights file not found: {pspec.weights_file}\n"+ "edge0 tiers are shipped with trained LoRA + prerouter "+ "adapters; download them for this tier (README -> 'Getting "+ "the models & adapters') and place them in the model "+ "directory, or disable the prerouter with "+ "prerouter=None / --no-prerouter.")weights = load_safetensors(pspec.weights_file)dtype = core.float16 if pspec.dtype == "fp16" else core.float32head_weights = _parse_weights(weights, dtype)diff --git a/src/edge0/server/chat.py b/src/edge0/server/chat.pyindex 3babc68..dc9e056 100644--- a/src/edge0/server/chat.py+++ b/src/edge0/server/chat.py@@ -84,17 +84,25 @@ class ChatSession:return encode([m.__dict__ for m in self.req.messages], think=bool(think))if hasattr(tok, "apply_chat_template"):- # Deployment parity with the correct demo (serve.py- # encode_chat): the qwen35 template's default IS the thinking- # mode (<think> opener). Passing enable_thinking=False on- # this template renders an EMPTY <think>\n\n</think> block- # instead of omitting thinking — a different prompt than the- # demo, so it must not be passed.- text = tok.apply_chat_template(- [m.__dict__ for m in self.req.messages],- tokenize=False, add_generation_prompt=True)- else:- text = self._chat_text()+ # The qwen35 template supports ``enable_thinking``: False+ # renders the canonical no-think prompt — an EMPTY think+ # block closer, the model's direct-answer form. Default OFF+ # (CLI demo parity).+ think = self.req.enable_thinking+ if think is None:+ think = False+ try:+ text = tok.apply_chat_template(+ [m.__dict__ for m in self.req.messages],+ tokenize=False, add_generation_prompt=True,+ enable_thinking=bool(think))+ except TypeError:+ # tokenizer template without the kwarg: plain render+ text = tok.apply_chat_template(+ [m.__dict__ for m in self.req.messages],+ tokenize=False, add_generation_prompt=True)+ else:+ text = self._chat_text()ids = tok.encode(text)if not ids:ids = [tok.bos_token_id or 0]diff --git a/src/edge0/streaming/mmap.py b/src/edge0/streaming/mmap.pyindex 3ea9441..86ce369 100644--- a/src/edge0/streaming/mmap.py+++ b/src/edge0/streaming/mmap.py@@ -40,6 +40,25 @@ class SafetensorsMmap:"shape": tuple(meta["shape"]),}+ def advise_willneed(self):+ """OS readahead hint over the whole shard (advisory)."""+ try:+ mmap_madv = getattr(mmap, "MADV_WILLNEED", None)+ if mmap_madv is not None:+ self._mm.madvise(mmap_madv)+ except Exception: # noqa: BLE001 — advisory only+ pass++ def seq_read(self, chunk: int = 1 << 24):+ """Force every page resident: one sequential pass over the shard.++ macOS madvise only warms roughly half the file, so a full read is+ the reliable way to remove per-expert page-fault cost from the+ first request (ling v7 ``_prewarm`` step 2 parity)."""+ total = len(self._mm)+ for off in range(0, total, chunk):+ _ = self._mm[off:off + chunk]+def raw(self, name: str) -> np.ndarray:e = self.entries[name]return np.frombuffer(diff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.pyindex fd31ec8..59deba8 100644--- a/src/edge0/streaming/options.py+++ b/src/edge0/streaming/options.py@@ -124,7 +124,8 @@ class LayerOptions:return cls(staged=False, staged_replace=False, staged_n=8,staged_trigger=8, staged_sync=False, history_prefetch=False,- hot_per_layer=0, cache_slots=64, prefetch_cap=48,+ hot_per_layer=0,+ cache_slots=64, prefetch_cap=48,load_threads=8, prefetch_threads=4, use_compile=True, top_k=8,full_layer_prefill=True, prefill_hot=0, **overrides)diff --git a/tests/test_registry.py b/tests/test_registry.pyindex 285c533..6cdcb47 100644--- a/tests/test_registry.py+++ b/tests/test_registry.py@@ -57,7 +57,7 @@ def test_qwen35_profile():assert cfg.prerouter.hidden == 512assert cfg.prerouter.dtype == "fp16"assert cfg.prerouter.feature_topk == "executed"- assert cfg.gen.temperature == 0.6+ assert cfg.gen.temperature == 0.7assert 248046 in cfg.gen.eos_idsassert cfg.port == 8085diff --git a/tests/test_repo_hygiene.py b/tests/test_repo_hygiene.pynew file mode 100644index 0000000..c8ef1d0--- /dev/null+++ b/tests/test_repo_hygiene.py@@ -0,0 +1,116 @@+"""Repository-hygiene guards for a clean open-source release.++These tests import nothing from ``edge0`` (and nothing from ``mlx``), so+they can run on any platform — including the Linux CI job where MLX has no+wheels. They enforce the "no dependency on the author's machine" rule:++* no personal / hardcoded absolute paths in shipped code, docs or scripts;+* the MLX backend boundary is respected (``import mlx`` only inside+ ``edge0/backends/mlx/``);+* no obvious secrets (API keys / private keys) in the tree;+* README/NOTICE references resolve.+"""++from __future__ import annotations++import pathlib+import re++ROOT = pathlib.Path(__file__).resolve().parents[1]++_EXCLUDE_DIRS = {".git", ".venv", ".pytest_cache", "__pycache__", ".agents",+ ".codex"}+_EXCLUDE_PARTS = {"egg-info"}+_EXCLUDE_SUFFIXES = {".pyc", ".so", ".dylib", ".bin", ".safetensors", ".npz",+ ".png", ".jpg", ".jpeg", ".gif", ".ico"}++# NOTE: patterns are assembled with + so this file never contains the+# literal string it is looking for.+_USR = "Users"+_NAS = "nas"+_HOME = "home"+_ROOT = "root"+_LOCAL_PATH_PATTERNS = [+ re.compile(r"/" + _USR + r"/[\w.-]+/"),+ re.compile(r"/" + _NAS + r"/"),+ re.compile(r"/" + _HOME + r"/[\w.-]+/"),+ re.compile(r"/" + _ROOT + r"(?:/|$)"),+ re.compile(r"C:" + re.escape("\\") + _USR + re.escape("\\")),+ re.compile(r"HPC-\d"),+ re.compile(r"ali-pod\d"),+]++_MLX_IMPORT = re.compile(r"^\s*(?:from\s+mlx(?:\s|\.)|import\s+mlx(?:\s|\.))",+ re.MULTILINE)++_SECRET_PATTERNS = [+ re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b"),+ re.compile(r"\bghp_[A-Za-z0-9]{20,}\b"),+ re.compile(r"AKIA[0-9A-Z]{16}"),+ re.compile(r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----"),+]+++def _iter_text_files():+ for path in ROOT.rglob("*"):+ rel = path.relative_to(ROOT)+ if any(part in _EXCLUDE_DIRS for part in rel.parts):+ continue+ if any(part in _EXCLUDE_PARTS for part in rel.parts):+ continue+ if path.suffix in _EXCLUDE_SUFFIXES or path.is_dir():+ continue+ yield path+++def _read(path: pathlib.Path) -> str:+ try:+ return path.read_text(encoding="utf-8", errors="replace")+ except OSError:+ return ""+++def test_no_hardcoded_local_paths():+ hits = []+ for path in _iter_text_files():+ text = _read(path)+ if not text:+ continue+ for pat in _LOCAL_PATH_PATTERNS:+ m = pat.search(text)+ if m:+ hits.append(f"{path.relative_to(ROOT)}: {m.group(0)!r}")+ assert not hits, "hardcoded local paths found:\n" + "\n".join(hits)+++def test_mlx_imports_stay_inside_the_backend():+ bad = []+ for path in (ROOT / "src" / "edge0").rglob("*.py"):+ rel = path.relative_to(ROOT)+ text = path.read_text(encoding="utf-8")+ if _MLX_IMPORT.search(text) and "backends/mlx" not in rel.as_posix():+ bad.append(str(rel))+ assert not bad, (+ "MLX imports must live under edge0/backends/mlx/ only:\n"+ + "\n".join(bad))+++def test_no_secrets():+ hits = []+ for path in _iter_text_files():+ text = _read(path)+ if not text:+ continue+ for pat in _SECRET_PATTERNS:+ m = pat.search(text)+ if m:+ hits.append(f"{path.relative_to(ROOT)}: {m.group(0)!r}")+ assert not hits, "possible secrets found:\n" + "\n".join(hits)+++def test_notice_link_resolves():+ readme = (ROOT / "README.md").read_text(encoding="utf-8")+ m = re.search(r"\[NOTICE\]\(([^)]+)\)", readme)+ assert m, "README should link the NOTICE file"+ target = ROOT / m.group(1)+ assert target.is_file(), f"README NOTICE link target missing: {target}"