Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
docs(streaming): correct load_hot_layer's "page cache" backing claim
Demetrios Agourakis58014c472c28
diff --git a/python/src/edge0/streaming/layer.py b/python/src/edge0/streaming/layer.pyindex ef1320b..b2b2d99 100644--- a/python/src/edge0/streaming/layer.py+++ b/python/src/edge0/streaming/layer.py@@ -992,8 +992,11 @@ class StreamingSwitchGLU:"""Load/refresh the layer's hot-expert weights.Backing store = one NUMPY array per (proj, part) holding the- stacked rows of the top-n_hot experts (page cache, survives across- requests, counts ZERO toward MLX active). The mx window+ stacked rows of the top-n_hot experts. ``np.concatenate`` copies,+ so this is resident anonymous process memory, not page cache --+ it survives across requests (the layer keeps the reference) and+ counts ZERO toward MLX active, but it is not reclaimable by the+ OS the way mmap'd file-backed pages are. The mx window(materialize_hot) exposes a few layers at a time so peak MLXmemory stays at window_size * n_hot * ~1.2MB instead of40 * n_hot * 1.2MB.@@ -1034,7 +1037,8 @@ class StreamingSwitchGLU:raw = self._shard_for(name).raw(name)per = raw.size // self.num_expertsrows = [raw[e * per:(e + 1) * per] for e in hot]- # concatenated numpy backing (page cache, not GPU), plus+ # concatenated numpy backing (anonymous process memory,+ # not GPU -- see load_hot_layer's docstring), plus# one all-zero row: the prefill path sends misses to row# n_hot ("overflow row") and needs it to contribute zero.# Without it that gather reads past the end of the stack.