Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs(streaming): distinguish paging from file-backed eviction
Demetrios Agourakis538b6695b709
diff --git a/python/src/edge0/streaming/layer.py b/python/src/edge0/streaming/layer.pyindex b2b2d99..a17c247 100644--- a/python/src/edge0/streaming/layer.py+++ b/python/src/edge0/streaming/layer.py@@ -993,10 +993,11 @@ class StreamingSwitchGLU:Backing store = one NUMPY array per (proj, part) holding thestacked rows of the top-n_hot experts. ``np.concatenate`` copies,- so this is resident anonymous process memory, not page cache --- it survives across requests (the layer keeps the reference) and- counts ZERO toward MLX active, but it is not reclaimable by the- OS the way mmap'd file-backed pages are. The mx window+ so this is anonymous process memory, not file-backed page cache.+ The layer retains it across requests; these host allocations are+ outside MLX active memory. The OS may page them out, but cannot+ discard and reload them from the checkpoint like clean mapped+ file pages. The mx window(materialize_hot) exposes a few layers at a time so peak MLXmemory stays at window_size * n_hot * ~1.2MB instead of40 * n_hot * 1.2MB.@@ -1037,8 +1038,7 @@ class StreamingSwitchGLU:raw = self._shard_for(name).raw(name)per = raw.size // self.num_expertsrows = [raw[e * per:(e + 1) * per] for e in hot]- # concatenated numpy backing (anonymous process memory,- # not GPU -- see load_hot_layer's docstring), plus+ # copied numpy backing (anonymous process memory), plus# one all-zero row: the prefill path sends misses to row# n_hot ("overflow row") and needs it to contribute zero.# Without it that gather reads past the end of the stack.