Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs: refresh warm_willneed guidance against current main
Demetrios Agourakise4a1eeb9afb5
diff --cc docs/streaming.mdindex 8719194,c049da0..4466cec--- a/docs/streaming.md+++ b/docs/streaming.md@@@ -73,10 -73,10 +73,17 @@@ With `use_compile`, the staged/exact pa| `load_threads` / `prefetch_threads` | Build / prefetch thread counts | 8 / 4 || `full_layer_prefill` / `prefill_full_layers` | Whole-layer prefill loading / number of leading layers | False / 0 || `prefill_hot` | hot stack size during prefill | 0 |- | `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch. Only read by `prefetch()` and `stage_experts()` -- **inert whenever `staged=False` and `history_prefetch=False`**, which is `prod_k8()`'s (edge0-8b) default, so toggling it has no effect on that tier without also enabling one of those paths. | False |-| `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch | False |++| `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch. Does not enable prefetch/staging or affect plain on-demand / whole-layer loading; see below. | False || `use_compile` / `top_k` | compile wrapping / routing top-k override | True / None |--Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile — staged decode off, E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.++Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile with staged decode for prerouter consumer layers and E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.++++`warm_willneed` is consulted only by `prefetch()` when there are missing experts++and by `stage_experts()` on staged layers. Turning it on does not enable either++path. Disabling history prefetch and staging removes those automatic decode++paths, but explicit prefetch calls (including `prefetch_from_prefill()`) can++still issue readahead. The flag does not warm plain on-demand or whole-layer++loads by itself, so it is not a general first-token-latency switch (see #110).The whole-layer prefill is the fastest path **when the checkpoint stays in the page cache** (warm 27-token prefill: 0.24 s vs 0.37 s on-demand on an M4 Pro), and the slowest one when it does not (cold: 5.3 s / 4.06 GiB read vs 0.6-1.1 s / 0.4-0.8 GiB; the on-demand figure varies with how many distinct experts the prompt routes to). `edge0 demo|chat|serve --prefill-ondemand` selects the on-demand path for machines in the second group.diff --cc python/src/edge0/streaming/options.pyindex 44d005b,8a293f8..25da43e--- a/python/src/edge0/streaming/options.py+++ b/python/src/edge0/streaming/options.py@@@ -90,11 -90,6 +90,11 @@@ class LayerOptions#: a step otherwise degrades into "cold pages x per-fault latency", with#: those faults serializing on the VM map lock. Retains no MLX arrays,#: so it does not displace the page cache.- #: Only consulted by ``prefetch()`` and ``stage_experts()`` — INERT when- #: both ``staged`` and ``history_prefetch`` are False, since neither call- #: site then ever runs. ``prod_k8()`` (edge0-8b's production profile) is- #: exactly that combination, so toggling this flag on that tier has no- #: effect (see issue #110).++ #: Only consulted by ``prefetch()`` (for missing experts) and++ #: ``stage_experts()`` (on staged layers). Does not enable either path++ #: or affect plain on-demand / whole-layer loading. Explicit prefetch++ #: calls, including ``prefetch_from_prefill()``, can still consult it++ #: when automatic history prefetch and staging are disabled.warm_willneed: bool = False# ---- presets ----------------------------------------------------------