Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs: refresh warm_willneed guidance against current main

Demetrios Agourakise4a1eeb9afb5
diff --cc docs/streaming.md
index 8719194,c049da0..4466cec
--- a/docs/streaming.md
+++ b/docs/streaming.md
@@@ -73,10 -73,10 +73,17 @@@ With `use_compile`, the staged/exact pa
| `load_threads` / `prefetch_threads` | Build / prefetch thread counts | 8 / 4 |
| `full_layer_prefill` / `prefill_full_layers` | Whole-layer prefill loading / number of leading layers | False / 0 |
| `prefill_hot` | hot stack size during prefill | 0 |
- | `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch. Only read by `prefetch()` and `stage_experts()` -- **inert whenever `staged=False` and `history_prefetch=False`**, which is `prod_k8()`'s (edge0-8b) default, so toggling it has no effect on that tier without also enabling one of those paths. | False |
-| `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch | False |
++| `warm_willneed` | Kernel bulk readahead (`madvise WILLNEED`) over the expert ranges a prefetch/stage is about to touch. Does not enable prefetch/staging or affect plain on-demand / whole-layer loading; see below. | False |
| `use_compile` / `top_k` | compile wrapping / routing top-k override | True / None |
--Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile — staged decode off, E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.
++Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack 32, on-demand prefill), `prod_k8()` (edge0-8b: the reference deployment profile with staged decode for prerouter consumer layers and E3b whole-layer prefill), and `staged_k8()` (the plain K=8 staged variant). Both tiers share `cache_slots=64`.
++
++`warm_willneed` is consulted only by `prefetch()` when there are missing experts
++and by `stage_experts()` on staged layers. Turning it on does not enable either
++path. Disabling history prefetch and staging removes those automatic decode
++paths, but explicit prefetch calls (including `prefetch_from_prefill()`) can
++still issue readahead. The flag does not warm plain on-demand or whole-layer
++loads by itself, so it is not a general first-token-latency switch (see #110).
The whole-layer prefill is the fastest path **when the checkpoint stays in the page cache** (warm 27-token prefill: 0.24 s vs 0.37 s on-demand on an M4 Pro), and the slowest one when it does not (cold: 5.3 s / 4.06 GiB read vs 0.6-1.1 s / 0.4-0.8 GiB; the on-demand figure varies with how many distinct experts the prompt routes to). `edge0 demo|chat|serve --prefill-ondemand` selects the on-demand path for machines in the second group.
diff --cc python/src/edge0/streaming/options.py
index 44d005b,8a293f8..25da43e
--- a/python/src/edge0/streaming/options.py
+++ b/python/src/edge0/streaming/options.py
@@@ -90,11 -90,6 +90,11 @@@ class LayerOptions
#: a step otherwise degrades into "cold pages x per-fault latency", with
#: those faults serializing on the VM map lock. Retains no MLX arrays,
#: so it does not displace the page cache.
- #: Only consulted by ``prefetch()`` and ``stage_experts()`` — INERT when
- #: both ``staged`` and ``history_prefetch`` are False, since neither call
- #: site then ever runs. ``prod_k8()`` (edge0-8b's production profile) is
- #: exactly that combination, so toggling this flag on that tier has no
- #: effect (see issue #110).
++ #: Only consulted by ``prefetch()`` (for missing experts) and
++ #: ``stage_experts()`` (on staged layers). Does not enable either path
++ #: or affect plain on-demand / whole-layer loading. Explicit prefetch
++ #: calls, including ``prefetch_from_prefill()``, can still consult it
++ #: when automatic history prefetch and staging are disabled.
warm_willneed: bool = False
# ---- presets ----------------------------------------------------------