Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
fix(deps): pin mlx 0.30.6 and document the A18 NAX mis-gating
linyubupa0d52bf8c11c5
diff --git a/README.md b/README.mdindex 8234c6e..8e756a7 100644--- a/README.md+++ b/README.md@@ -43,6 +43,20 @@ are co-located with each checkpoint and load automatically, so(M1/M2/M3/M4). The CUDA backend is on the roadmap — no otherplatforms are supported yet.- **Python**: 3.10+ (3.12 recommended).+- **MLX runtime**: this release is built and tested against+ `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (pinned in+ `pyproject.toml`). If a checkpoint loads cleanly and generation runs to+ completion, but the text comes out as incoherent, mixed-language noise,+ upgrade the runtime first: `mlx<=0.30.4` mis-gates MLX's NAX+ (tensor-core) matmul kernels on phone-class GPUs (Apple A18 / A18 Pro)+ and returns silently wrong numbers — no error, no NaN, no crash+ ([edge0#8](https://github.com/Edge0-AI/Edge0/issues/8)). Run+ `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`; the fix is upstream+ ([#3083](https://github.com/ml-explore/mlx/pull/3083),+ [#3092](https://github.com/ml-explore/mlx/pull/3092), released in+ 0.30.5). Do not move to `mlx>=0.31.2` yet — it makes default streams+ thread-local and the streaming expert loader then fails with+ `There is no Stream(gpu, N) in current thread`.- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for`edge0-8b` (short contexts; see [Benchmark](#benchmark)). Addheadroom for the OS, tokenizer, and long-context KV growth.diff --git a/README_zh.md b/README_zh.mdindex 89aef98..b31404c 100644--- a/README_zh.md+++ b/README_zh.md@@ -37,6 +37,17 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS(M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。- **Python**:3.10+(推荐 3.12)。+- **MLX 运行时**:本版本基于 `mlx==0.30.6` / `mlx-metal==0.30.6` 与+ `mlx-lm==0.31.0` 构建并测试(pin 见 `pyproject.toml`)。若 checkpoint+ 加载正常、生成也能跑完,但输出是跨语言混杂的乱码,请先升级运行时:+ `mlx<=0.30.4` 会在手机级 GPU(Apple A18 / A18 Pro)上误判走 NAX+ (张量核)matmul kernel,静默算出错误数值——不报错、不出 NaN、不崩+ ([edge0#8](https://github.com/Edge0-AI/Edge0/issues/8))。执行+ `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'` 即可;修复来自上游+ ([#3083](https://github.com/ml-explore/mlx/pull/3083)、+ [#3092](https://github.com/ml-explore/mlx/pull/3092),0.30.5 起发布)。+ 暂不要升到 `mlx>=0.31.2`:它把默认 stream 改成 thread-local,专家流式+ 加载会报 `There is no Stream(gpu, N) in current thread`。- **内存**:短上下文下 `edge0-35b` ≈2.9 GB、`edge0-8b` ≈1.0 GB峰值激活内存(见[性能实测](#性能实测));另为系统、tokenizer 与长上下文 KV 增长预留余量。diff --git a/pyproject.toml b/pyproject.tomlindex 5186b0b..b6e3b19 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -22,11 +22,27 @@ classifiers = ["Topic :: Scientific/Engineering :: Artificial Intelligence",]dependencies = [- # NOTE: mlx-lm is pinned to 0.31.0 on purpose. Newer releases crash in- # decode with a `tolist()` on lazily-evaluated arrays. Do not upgrade- # without re-running the full test suite.- "mlx==0.30.4",- "mlx-metal==0.30.4; platform_system == 'Darwin'",+ # NOTE: mlx/mlx-metal are pinned to 0.30.6 on purpose — do not move them+ # on their own without re-running the full test suite (`pytest` plus+ # `pytest -m slow` with real checkpoints).+ #+ # * 0.30.4 and older mis-gate the NAX (tensor-core) matmul kernels on+ # phone-class GPUs: `can_use_nax &= get_architecture_gen() >= 17`+ # admitted Apple A18/A18 Pro, whose NAX path returns silently wrong+ # numbers — fluent-but-incoherent output with no error, no NaN and no+ # crash (github.com/Edge0-AI/Edge0/issues/8). Fixed in 0.30.5 by+ # ml-explore/mlx#3083 ("Fix nax condition for iphone", now requires+ # gen >= 18 for 'p' architectures) and ml-explore/mlx#3092+ # ("Fix for NAX overflow").+ # * Do not jump to mlx >= 0.31.2 yet: it makes default streams+ # thread-local (ml-explore/mlx#3281, ml-explore/mlx#3405) and the+ # streaming layer builds mx arrays on pool threads+ # (edge0/streaming/layer.py `_build`), which then fails with+ # "There is no Stream(gpu, N) in current thread".+ # * mlx-lm is pinned to 0.31.0 on purpose. Newer releases crash in+ # decode with a `tolist()` on lazily-evaluated arrays.+ "mlx==0.30.6",+ "mlx-metal==0.30.6; platform_system == 'Darwin'","mlx-lm==0.31.0","numpy>=1.24","safetensors>=0.4",