Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

fix(deps): pin mlx 0.30.6 and document the A18 NAX mis-gating

linyubupa0d52bf8c11c5
diff --git a/README.md b/README.md
index 8234c6e..8e756a7 100644
--- a/README.md
+++ b/README.md
@@ -43,6 +43,20 @@ are co-located with each checkpoint and load automatically, so
(M1/M2/M3/M4). The CUDA backend is on the roadmap — no other
platforms are supported yet.
- **Python**: 3.10+ (3.12 recommended).
+- **MLX runtime**: this release is built and tested against
+ `mlx==0.30.6` / `mlx-metal==0.30.6` with `mlx-lm==0.31.0` (pinned in
+ `pyproject.toml`). If a checkpoint loads cleanly and generation runs to
+ completion, but the text comes out as incoherent, mixed-language noise,
+ upgrade the runtime first: `mlx<=0.30.4` mis-gates MLX's NAX
+ (tensor-core) matmul kernels on phone-class GPUs (Apple A18 / A18 Pro)
+ and returns silently wrong numbers — no error, no NaN, no crash
+ ([edge0#8](https://github.com/Edge0-AI/Edge0/issues/8)). Run
+ `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'`; the fix is upstream
+ ([#3083](https://github.com/ml-explore/mlx/pull/3083),
+ [#3092](https://github.com/ml-explore/mlx/pull/3092), released in
+ 0.30.5). Do not move to `mlx>=0.31.2` yet — it makes default streams
+ thread-local and the streaming expert loader then fails with
+ `There is no Stream(gpu, N) in current thread`.
- **Memory**: ~2.9 GB peak active memory for `edge0-35b`, ~1.0 GB for
`edge0-8b` (short contexts; see [Benchmark](#benchmark)). Add
headroom for the OS, tokenizer, and long-context KV growth.
diff --git a/README_zh.md b/README_zh.md
index 89aef98..b31404c 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -37,6 +37,17 @@ Recover-LoRA + prerouter 路由预判」抽象成可扩展的通用框架。后
- **系统 / 硬件**:MLX 后端目前仅支持 Apple Silicon 的 macOS
(M1/M2/M3/M4);CUDA 后端在路线图中,其余平台暂不支持。
- **Python**:3.10+(推荐 3.12)。
+- **MLX 运行时**:本版本基于 `mlx==0.30.6` / `mlx-metal==0.30.6` 与
+ `mlx-lm==0.31.0` 构建并测试(pin 见 `pyproject.toml`)。若 checkpoint
+ 加载正常、生成也能跑完,但输出是跨语言混杂的乱码,请先升级运行时:
+ `mlx<=0.30.4` 会在手机级 GPU(Apple A18 / A18 Pro)上误判走 NAX
+ (张量核)matmul kernel,静默算出错误数值——不报错、不出 NaN、不崩
+ ([edge0#8](https://github.com/Edge0-AI/Edge0/issues/8))。执行
+ `pip install 'mlx==0.30.6' 'mlx-metal==0.30.6'` 即可;修复来自上游
+ ([#3083](https://github.com/ml-explore/mlx/pull/3083)、
+ [#3092](https://github.com/ml-explore/mlx/pull/3092),0.30.5 起发布)。
+ 暂不要升到 `mlx>=0.31.2`:它把默认 stream 改成 thread-local,专家流式
+ 加载会报 `There is no Stream(gpu, N) in current thread`。
- **内存**:短上下文下 `edge0-35b` ≈2.9 GB、`edge0-8b` ≈1.0 GB
峰值激活内存(见[性能实测](#性能实测));另为系统、tokenizer 与
长上下文 KV 增长预留余量。
diff --git a/pyproject.toml b/pyproject.toml
index 5186b0b..b6e3b19 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -22,11 +22,27 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
- # NOTE: mlx-lm is pinned to 0.31.0 on purpose. Newer releases crash in
- # decode with a `tolist()` on lazily-evaluated arrays. Do not upgrade
- # without re-running the full test suite.
- "mlx==0.30.4",
- "mlx-metal==0.30.4; platform_system == 'Darwin'",
+ # NOTE: mlx/mlx-metal are pinned to 0.30.6 on purpose — do not move them
+ # on their own without re-running the full test suite (`pytest` plus
+ # `pytest -m slow` with real checkpoints).
+ #
+ # * 0.30.4 and older mis-gate the NAX (tensor-core) matmul kernels on
+ # phone-class GPUs: `can_use_nax &= get_architecture_gen() >= 17`
+ # admitted Apple A18/A18 Pro, whose NAX path returns silently wrong
+ # numbers — fluent-but-incoherent output with no error, no NaN and no
+ # crash (github.com/Edge0-AI/Edge0/issues/8). Fixed in 0.30.5 by
+ # ml-explore/mlx#3083 ("Fix nax condition for iphone", now requires
+ # gen >= 18 for 'p' architectures) and ml-explore/mlx#3092
+ # ("Fix for NAX overflow").
+ # * Do not jump to mlx >= 0.31.2 yet: it makes default streams
+ # thread-local (ml-explore/mlx#3281, ml-explore/mlx#3405) and the
+ # streaming layer builds mx arrays on pool threads
+ # (edge0/streaming/layer.py `_build`), which then fails with
+ # "There is no Stream(gpu, N) in current thread".
+ # * mlx-lm is pinned to 0.31.0 on purpose. Newer releases crash in
+ # decode with a `tolist()` on lazily-evaluated arrays.
+ "mlx==0.30.6",
+ "mlx-metal==0.30.6; platform_system == 'Darwin'",
"mlx-lm==0.31.0",
"numpy>=1.24",
"safetensors>=0.4",