Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs(streaming): qualify prefill readahead prerequisites

Demetrios Agourakis2c609e3b0e44
diff --git a/docs/streaming.md b/docs/streaming.md
index 4466cec..fcb4879 100644
--- a/docs/streaming.md
+++ b/docs/streaming.md
@@ -81,9 +81,12 @@ Presets: `staged_k4()` (edge0-35b: staged decode with 4 slots, prefill hot stack
`warm_willneed` is consulted only by `prefetch()` when there are missing experts
and by `stage_experts()` on staged layers. Turning it on does not enable either
path. Disabling history prefetch and staging removes those automatic decode
-paths, but explicit prefetch calls (including `prefetch_from_prefill()`) can
-still issue readahead. The flag does not warm plain on-demand or whole-layer
-loads by itself, so it is not a general first-token-latency switch (see #110).
+paths, but a direct `prefetch(experts)` call can still issue readahead for
+missing experts. `prefetch_from_prefill()` calls `prefetch()` only if a prefill
+expert set was captured while staging was enabled; with staging disabled from
+initialization, it is a no-op. The flag does not warm plain on-demand or
+whole-layer loads by itself, so it is not a general first-token-latency switch
+(see #110).
The whole-layer prefill is the fastest path **when the checkpoint stays in the page cache** (warm 27-token prefill: 0.24 s vs 0.37 s on-demand on an M4 Pro), and the slowest one when it does not (cold: 5.3 s / 4.06 GiB read vs 0.6-1.1 s / 0.4-0.8 GiB; the on-demand figure varies with how many distinct experts the prompt routes to). `edge0 demo|chat|serve --prefill-ondemand` selects the on-demand path for machines in the second group.
diff --git a/python/src/edge0/streaming/options.py b/python/src/edge0/streaming/options.py
index 25da43e..6b12fe5 100644
--- a/python/src/edge0/streaming/options.py
+++ b/python/src/edge0/streaming/options.py
@@ -92,9 +92,11 @@ class LayerOptions:
#: so it does not displace the page cache.
#: Only consulted by ``prefetch()`` (for missing experts) and
#: ``stage_experts()`` (on staged layers). Does not enable either path
- #: or affect plain on-demand / whole-layer loading. Explicit prefetch
- #: calls, including ``prefetch_from_prefill()``, can still consult it
- #: when automatic history prefetch and staging are disabled.
+ #: or affect plain on-demand / whole-layer loading. Direct
+ #: ``prefetch(experts)`` calls can still consult it when automatic
+ #: history prefetch and staging are disabled. ``prefetch_from_prefill()``
+ #: requires an expert set captured while staging was enabled; with
+ #: staging disabled from initialization, that helper is a no-op.
warm_willneed: bool = False
# ---- presets ----------------------------------------------------------