Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
"""Backend abstraction.
The framework core (engine, streaming, prerouter, adapters, model
adapters, server) imports ONLY the backend namespaces re-exported here::
from edge0.backends import core, nn, io, quant
A backend is a directory under ``edge0/backends/<name>/`` exposing:
``backend.py``
``core`` — array ops namespace (matmul, softmax, argpartition, take,
put_along_axis, eval, compile, ...)
``nn`` — module factory namespace (Module, Linear, RMSNorm, silu, ...)
``io`` — model/tokenizer/tensor-store loading
``quant`` — quantized gather kernels (``gather_qmm``)
``BackendImpl`` — small object with ``name`` and metadata
The MLX backend is the reference implementation. A future CUDA backend
implements the same surface with eager semantics; engine / prerouter /
server / model-adapter code is shared verbatim. The op list below is the
contract the framework currently exercises (grow it as models are added).
Contract (documented, enforced by tests/grep in CI):
* The MLX package (``mlx`` / ``mlx_lm``) must never be imported
outside ``edge0/backends/mlx/`` — framework code reaches the backend
only through the namespaces re-exported here.
* ``core`` must provide (for the MLX backend these are mlx.core names):
array, zeros, eye, arange, full, expand_dims, squeeze, reshape,
transpose, concatenate, stack, split, matmul, softmax, silu (via nn),
sigmoid, erf, where, sum, cumsum, sort, topk, argpartition, take,
take_along_axis, put_along_axis, astype, item, tolist, eval, compile,
random.seed, random.categorical, and the dtypes float16 / float32 /
bfloat16 / int32 / uint32 / int64.
* ``nn`` must provide Module, Linear, RMSNorm, silu, gelu.
* ``io`` must provide load_model, load_tokenizer, open_tensor_store.
* ``quant`` must provide gather_qmm.
"""
import os
_BACKEND = os.environ.get("EDGE0_BACKEND", "mlx")
if _BACKEND == "mlx":
from edge0.backends.mlx.backend import ( # noqa: F401
BackendImpl,
core,
io,
nn,
quant,
)
elif _BACKEND == "cuda":
raise ImportError(
"the CUDA backend is not implemented yet; set EDGE0_BACKEND=mlx "
"(or unset it)")
else:
raise ImportError(
f"unknown backend {_BACKEND!r}; available: mlx")
backend = BackendImpl()
__all__ = ["backend", "core", "io", "nn", "quant"]