Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs(streaming): document warm_willneed as inert under prod_k8()

Demetrios Agourakis08fcb468b6e1
diff --git a/docs/streaming.md b/docs/streaming.md
index c049da0..8719194 100644
--- a/docs/streaming.md
+++ b/docs/streaming.md
@@ -73,7 +73,7 @@ With `use_compile`, the staged/exact paths are wrapped in `mx.compile`:
| `load_threads` / `prefetch_threads` | Build / prefetch thread counts | 8 / 4 |
| `full_layer_prefill` / `prefill_full_layers` | Whole-layer prefill loading / number of leading layers | False / 0 |
| `prefill_hot` | hot stack size during prefill | 0 |
-| `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch | False |
+| `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch. Only read by `prefetch()` and `stage_experts()` -- **inert whenever `staged=False` and `history_prefetch=False`**, which is `prod_k8()`'s (edge0-8b) default, so toggling it has no effect on that tier without also enabling one of those paths. | False |
| `use_compile` / `top_k` | compile wrapping / routing top-k override | True / None |
Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile — staged decode off, E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.
diff --git a/src/edge0/streaming/options.py b/src/edge0/streaming/options.py
index 8a293f8..44d005b 100644
--- a/src/edge0/streaming/options.py
+++ b/src/edge0/streaming/options.py
@@ -90,6 +90,11 @@ class LayerOptions:
#: a step otherwise degrades into "cold pages x per-fault latency", with
#: those faults serializing on the VM map lock. Retains no MLX arrays,
#: so it does not displace the page cache.
+ #: Only consulted by ``prefetch()`` and ``stage_experts()`` — INERT when
+ #: both ``staged`` and ``history_prefetch`` are False, since neither call
+ #: site then ever runs. ``prod_k8()`` (edge0-8b's production profile) is
+ #: exactly that combination, so toggling this flag on that tier has no
+ #: effect (see issue #110).
warm_willneed: bool = False
# ---- presets ----------------------------------------------------------