Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
fix(deps): pin mlx 0.30.6 and document the A18 NAX mis-gating
linyubupa2e168c5c7280
diff --git a/README.md b/README.mdindex 8234c6e..fb8168a 100644--- a/README.md+++ b/README.md@@ -43,6 +43,10 @@ are co-located with each checkpoint and load automatically, so(M1/M2/M3/M4). The CUDA backend is on the roadmap — no otherplatforms are supported yet.- **Python**: 3.10+ (3.12 recommended).+- **MLX**: `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (see+ `pyproject.toml`). Garbled, mixed-language output on Apple A18 / A18 Pro+ means an older `mlx`: `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`+ ([#8](https://github.com/Edge0-AI/Edge0/issues/8)).- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for`edge0-8b` (short contexts; see [Benchmark](#benchmark)). Addheadroom for the OS, tokenizer, and long-context KV growth.diff --git a/README_zh.md b/README_zh.mdindex 89aef98..649023d 100644--- a/README_zh.md+++ b/README_zh.md@@ -37,6 +37,10 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS(M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。- **Python**:3.10+(推荐 3.12)。+- **MLX**:`mlx==0.30.6` / `mlx-metal==0.30.6`(`mlx-lm==0.31.0`,见+ `pyproject.toml`)。Apple A18 / A18 Pro 上输出乱码 = mlx 版本旧:+ `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`+ ([#8](https://github.com/Edge0-AI/Edge0/issues/8))。- **内存**:短上下文下 `edge0-35b` ≈2.9 GB、`edge0-8b` ≈1.0 GB峰值激活内存(见[性能实测](#性能实测));另为系统、tokenizer 与长上下文 KV 增长预留余量。diff --git a/pyproject.toml b/pyproject.tomlindex 5186b0b..7d93e06 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -22,11 +22,15 @@ classifiers = ["Topic :: Scientific/Engineering :: Artificial Intelligence",]dependencies = [- # NOTE: mlx-lm is pinned to 0.31.0 on purpose. Newer releases crash in- # decode with a `tolist()` on lazily-evaluated arrays. Do not upgrade- # without re-running the full test suite.- "mlx==0.30.4",- "mlx-metal==0.30.4; platform_system == 'Darwin'",+ # NOTE: mlx/mlx-metal pinned to 0.30.6 on purpose. 0.30.4 and older+ # mis-gate the NAX matmul kernels on Apple A18/A18 Pro (silently wrong+ # numbers -> garbled output, issue #8); fixed upstream in 0.30.5+ # (ml-explore/mlx#3083, #3092). Do not go to mlx>=0.31.2 yet: default+ # streams become thread-local and the pool-thread builds in+ # edge0/streaming/layer.py fail with "no Stream(gpu, N) in current+ # thread". mlx-lm stays 0.31.0 (newer releases crash in decode).+ "mlx==0.30.6",+ "mlx-metal==0.30.6; platform_system == 'Darwin'","mlx-lm==0.31.0","numpy>=1.24","safetensors>=0.4",