Skip to content

armgit / Edge0

Public Imported from GitHub
0

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Python

docs(streaming): distinguish paging from file-backed eviction

Demetrios Agourakis538b6695b709
diff --git a/python/src/edge0/streaming/layer.py b/python/src/edge0/streaming/layer.py
index b2b2d99..a17c247 100644
--- a/python/src/edge0/streaming/layer.py
+++ b/python/src/edge0/streaming/layer.py
@@ -993,10 +993,11 @@ class StreamingSwitchGLU:
Backing store = one NUMPY array per (proj, part) holding the
stacked rows of the top-n_hot experts. ``np.concatenate`` copies,
- so this is resident anonymous process memory, not page cache --
- it survives across requests (the layer keeps the reference) and
- counts ZERO toward MLX active, but it is not reclaimable by the
- OS the way mmap'd file-backed pages are. The mx window
+ so this is anonymous process memory, not file-backed page cache.
+ The layer retains it across requests; these host allocations are
+ outside MLX active memory. The OS may page them out, but cannot
+ discard and reload them from the checkpoint like clean mapped
+ file pages. The mx window
(materialize_hot) exposes a few layers at a time so peak MLX
memory stays at window_size * n_hot * ~1.2MB instead of
40 * n_hot * 1.2MB.
@@ -1037,8 +1038,7 @@ class StreamingSwitchGLU:
raw = self._shard_for(name).raw(name)
per = raw.size // self.num_experts
rows = [raw[e * per:(e + 1) * per] for e in hot]
- # concatenated numpy backing (anonymous process memory,
- # not GPU -- see load_hot_layer's docstring), plus
+ # copied numpy backing (anonymous process memory), plus
# one all-zero row: the prefill path sends misses to row
# n_hot ("overflow row") and needs it to contribute zero.
# Without it that gather reads past the end of the stack.