Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: drop the long-context KV-cache figure for edge0-8b
linyubupa48bd22456a57
diff --git a/README.md b/README.mdindex 01c20ca..66bba35 100644--- a/README.md+++ b/README.md@@ -233,8 +233,7 @@ SSD); warm = subsequent requests (page cache resident). Prefill numbersare throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).***Peak active memory at short contexts (MLX allocator peak; expert weights-stream from SSD via mmap and are not resident). Long contexts add KV-cache: ~3.3 GiB on `edge0-8b` at 3.3k tokens.*+stream from SSD via mmap and are not resident).*Reproduce:diff --git a/README_zh.md b/README_zh.mdindex 46987e1..4413371 100644--- a/README_zh.md+++ b/README_zh.md@@ -212,7 +212,7 @@ engine.close() # 释放 mmap / 专家缓存*冷 = 进程启动后首请求(专家权重从 SSD 逐页换入);热 = 后续请求(页缓存常驻)。Prefill 为 ≈3.3k token 长 prompt 的吞吐(`BENCH_LONG=1`)。-**短上下文下的峰值 active 内存(MLX allocator 峰值;专家权重经 mmap 流式读取、不常驻内存)。长上下文增加 KV cache:3.3k token 下 `edge0-8b` ≈3.3 GiB。**+**短上下文下的峰值 active 内存(MLX allocator 峰值;专家权重经 mmap 流式读取、不常驻内存)。**复现:diff --git a/docs/models/edge0-8b.md b/docs/models/edge0-8b.mdindex 470e2cc..5b83a06 100644--- a/docs/models/edge0-8b.md+++ b/docs/models/edge0-8b.md@@ -24,7 +24,7 @@ The performance profile is based on benchmarks of the current release adapter ve| Streaming prefetch history | on (`prefetch_history=True`) || Serving port | 8083 || Measured throughput | 23.9–25.3 tok/s (M4 Pro) |-| Measured peak activation memory | ≈ 1.0 GiB (short context) / 3.3 GiB (3.3k-token context) |+| Measured peak activation memory | ≈ 1.0 GiB (short context) |> Note: all values are taken from `Ling8BConfig._defaults()` and `LayerOptions.prod_k8()`. The head count comes from the explicit `owners=range(7, 23)` — 16 heads in total (the current release head distribution: prediction is consumed from L7 onward, and L1–6 use the original router); `feature_topk="executed"`.