Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported
Python
chore: add issue forms and PR template
JiahaoWang87efe800e2c2
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.ymlnew file mode 100644index 0000000..deb83cb--- /dev/null+++ b/.github/ISSUE_TEMPLATE/bug_report.yml@@ -0,0 +1,133 @@+name: π Bug report+description: Something is broken, crashes, or produces wrong/garbled output.+labels: ["bug"]+body:+ - type: markdown+ attributes:+ value: |+ Thanks for reporting a bug! Please fill in the environment fields β most issues we see+ (garbled output, install errors, collapse-to-`!` loops) are pinned down by the platform,+ chip, mlx version, and whether the prerouter/LoRA adapters are loaded.++ **Before filing:** search [existing issues](https://github.com/Edge0-AI/Edge0/issues?q=is%3Aissue)+ and check the [known mlx version pin](https://github.com/Edge0-AI/Edge0/blob/main/python/pyproject.toml)+ (A18 / A18 Pro garbled output β `mlx==0.30.6`, see #8).++ - type: dropdown+ id: platform+ attributes:+ label: Platform / runtime+ description: Which runtime are you using?+ options:+ - Python (MLX, Apple Silicon macOS)+ - macOS app / CLI (Rust)+ - iOS app+ - Android app / engine+ - Windows app / engine+ validations:+ required: true++ - type: dropdown+ id: tier+ attributes:+ label: Model tier+ options:+ - edge0-35b+ - edge0-8b+ - both+ - other / custom checkpoint+ validations:+ required: true++ - type: input+ id: hardware+ attributes:+ label: Hardware / chip+ description: e.g. "Mac mini M4 Pro", "MacBook Air M1", "iPhone 16 Pro (A18 Pro)", "RTX 4090", "Pixel 10 Pro".+ placeholder: Apple M1 Pro+ validations:+ required: true++ - type: input+ id: ram+ attributes:+ label: RAM / unified memory+ description: Total memory of the device. Many "slow" / OOM reports come from low-memory machines.+ placeholder: 16 GB+ validations:+ required: true++ - type: input+ id: os+ attributes:+ label: OS version+ placeholder: macOS 15.1 / Windows 11 23H2 / Android 15 / iOS 18.1+ validations:+ required: true++ - type: input+ id: mlx_version+ attributes:+ label: mlx / mlx-metal / mlx-lm version (Python runtime only)+ description: "Output of: pip show mlx mlx-metal mlx-lm | grep -i version β pin is mlx==0.30.6, mlx-lm==0.31.0."+ placeholder: mlx 0.30.6, mlx-metal 0.30.6, mlx-lm 0.31.0+ validations:+ required: false++ - type: dropdown+ id: install_method+ attributes:+ label: How did you install / get the model?+ options:+ - pip install -e '.[dev,fetch]' from source+ - scripts/fetch_models.py+ - huggingface-cli download+ - ModelScope+ - Prebuilt app (macOS/iOS/Android/Windows)+ - Other (describe below)+ validations:+ required: false++ - type: dropdown+ id: adapters+ attributes:+ label: Adapters in use+ description: Does the bug still happen with the prerouter disabled? (`--no-prerouter`) This isolates a whole class of bugs (see #16).+ options:+ - Default (prerouter + LoRA enabled)+ - "--no-prerouter"+ - "--no-lora"+ - Both disabled (bare base model)+ validations:+ required: false++ - type: textarea+ id: what_happened+ attributes:+ label: What happened?+ description: What went wrong, and what did you expect instead?+ placeholder: After the 3rd request in the same process, every output collapses to "!!!!".+ validations:+ required: true++ - type: textarea+ id: repro+ attributes:+ label: Steps to reproduce+ description: Exact commands / code. Include the prompt if output is wrong.+ placeholder: |+ 1. edge0 serve edge0-35b+ 2. curl -X POST http://127.0.0.1:8000/v1/chat/completions -d '...'+ 3. Repeat 3 times+ render: shell+ validations:+ required: true++ - type: textarea+ id: logs+ attributes:+ label: Logs / stack trace / output+ description: Paste any error output or the garbled text. This will be rendered as code, no need for backticks.+ render: shell+ validations:+ required: falsediff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.ymlnew file mode 100644index 0000000..1f2c7a0--- /dev/null+++ b/.github/ISSUE_TEMPLATE/config.yml@@ -0,0 +1,11 @@+blank_issues_enabled: false+contact_links:+ - name: π¬ Questions & Discussions+ url: https://github.com/Edge0-AI/Edge0/discussions+ about: Usage questions, "how do I run it on X", show-and-tell, and general Q&A. Please ask here instead of opening an issue.+ - name: π Documentation+ url: https://github.com/Edge0-AI/Edge0#readme+ about: Setup, quickstart, benchmarks, and the per-platform READMEs (python/ Β· macos/ Β· ios/ Β· android/ Β· windows/).+ - name: π§ Security / private contact+ url: mailto:[email protected]+ about: Report security issues or anything you cannot share publicly.diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.ymlnew file mode 100644index 0000000..a037dcc--- /dev/null+++ b/.github/ISSUE_TEMPLATE/feature_request.yml@@ -0,0 +1,48 @@+name: π‘ Feature request+description: Suggest a capability, backend, platform, or API improvement.+labels: ["enhancement"]+body:+ - type: markdown+ attributes:+ value: |+ Before filing, please check the [roadmap in the README](https://github.com/Edge0-AI/Edge0#roadmap)+ and existing issues β several popular asks are already tracked (CUDA backend #107/#20/#19,+ non-macOS support #18, tool/function calling #115/#119).++ - type: dropdown+ id: area+ attributes:+ label: Area+ options:+ - New backend (CUDA / other accelerator)+ - New platform support+ - Server / OpenAI-compatible API+ - CLI / Python API+ - Quantization / adapters (LoRA / prerouter)+ - Documentation+ - Other+ validations:+ required: true++ - type: textarea+ id: problem+ attributes:+ label: What problem are you trying to solve?+ description: Describe the use case and who it helps. "I have an NVIDIA GPU and want to run edge0-8b for ..."+ validations:+ required: true++ - type: textarea+ id: proposal+ attributes:+ label: Proposed solution+ description: What would you like to see? If you're willing to contribute it, say so β we love PRs (see docs/adding-a-model.md for model work).+ validations:+ required: false++ - type: textarea+ id: alternatives+ attributes:+ label: Alternatives considered+ validations:+ required: falsediff --git a/.github/ISSUE_TEMPLATE/model_request.yml b/.github/ISSUE_TEMPLATE/model_request.ymlnew file mode 100644index 0000000..a17a204--- /dev/null+++ b/.github/ISSUE_TEMPLATE/model_request.yml@@ -0,0 +1,56 @@+name: π§© New model support request+description: Request (or offer to add) support for another MoE model / checkpoint.+labels: ["model-support"]+body:+ - type: markdown+ attributes:+ value: |+ edge0 is built to take new models through a documented path β see+ [`docs/adding-a-model.md`](https://github.com/Edge0-AI/Edge0/blob/main/docs/adding-a-model.md).+ The best model requests come with the config below. **Contributions welcome** β if you+ want to add it yourself, say so and we'll help review.++ - type: input+ id: model_name+ attributes:+ label: Model name+ placeholder: Qwen3.6-Next-80B-A3B+ validations:+ required: true++ - type: input+ id: hf_link+ attributes:+ label: Hugging Face / ModelScope link+ placeholder: https://huggingface.co/...+ validations:+ required: true++ - type: input+ id: model_type+ attributes:+ label: config.json `model_type`+ description: The architecture id from the checkpoint's config.json β edge0 resolves tiers via this (see TYPE_ALIASES).+ placeholder: qwen3_5_moe / bailing_hybrid / ...+ validations:+ required: false++ - type: textarea+ id: arch+ attributes:+ label: Architecture summary+ description: Sparse MoE? Number of layers / experts / top-k routing width? Attention type (MHA / MLA / linear)? Anything unusual.+ placeholder: Sparse MoE, 48 layers, 128 experts, top-8 routing, MLA attention.+ validations:+ required: true++ - type: dropdown+ id: contribute+ attributes:+ label: Are you willing to contribute this?+ options:+ - "Yes β I'd like to open a PR (happy to get guidance)"+ - "Maybe, with help"+ - "No, this is a request"+ validations:+ required: falsediff --git a/.github/ISSUE_TEMPLATE/performance_report.yml b/.github/ISSUE_TEMPLATE/performance_report.ymlnew file mode 100644index 0000000..8d5e30c--- /dev/null+++ b/.github/ISSUE_TEMPLATE/performance_report.yml@@ -0,0 +1,89 @@+name: π’ Performance report+description: Decode/prefill speed, time-to-first-token, or memory usage differs from the documented benchmarks.+labels: ["performance"]+body:+ - type: markdown+ attributes:+ value: |+ Use this for "slower than the README numbers" or "high memory / OOM" reports+ (e.g. #17, #110). The documented benchmarks were measured on a **Mac mini M4 Pro, 24 GB**.+ Low-memory machines (8β16 GB) can be dramatically slower because expert pages are+ continuously evicted and re-read from SSD β please include your total RAM.++ - type: dropdown+ id: tier+ attributes:+ label: Model tier+ options:+ - edge0-35b+ - edge0-8b+ validations:+ required: true++ - type: input+ id: hardware+ attributes:+ label: Hardware / chip+ placeholder: Apple M1 Pro+ validations:+ required: true++ - type: input+ id: ram+ attributes:+ label: Total RAM / unified memory+ placeholder: 16 GB+ validations:+ required: true++ - type: input+ id: disk+ attributes:+ label: Storage type+ description: Internal SSD vs external/USB matters a lot for streaming expert reads.+ placeholder: Internal NVMe SSD+ validations:+ required: false++ - type: input+ id: decode_observed+ attributes:+ label: Observed decode speed (tok/s)+ placeholder: "0.6 tok/s (doc says ~15)"+ validations:+ required: false++ - type: input+ id: ttft+ attributes:+ label: Observed time-to-first-token+ placeholder: 11 s+ validations:+ required: false++ - type: input+ id: memory+ attributes:+ label: Observed peak memory+ description: Note whether this is MLX allocator peak or process RSS (they differ β see #106).+ placeholder: 4.2 GB (process RSS)+ validations:+ required: false++ - type: textarea+ id: flags+ attributes:+ label: Flags / settings tried+ description: e.g. --prefill-ondemand, full_layer_prefill, warm_willneed, context length, max_tokens.+ placeholder: Tried warm_willneed and full_layer_prefill=False, no change.+ validations:+ required: false++ - type: textarea+ id: command+ attributes:+ label: Exact command used to measure+ description: Ideally the output of examples/bench.py so numbers are comparable.+ render: shell+ validations:+ required: truediff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.mdnew file mode 100644index 0000000..3de612b--- /dev/null+++ b/.github/PULL_REQUEST_TEMPLATE.md@@ -0,0 +1,28 @@+<!-- Thanks for contributing to edge0! Keep this short but complete. -->++## What & why++<!-- What does this PR change, and what problem does it solve? Link issues: "Fixes #NN". -->++Fixes #++## Area++- [ ] Python framework (`python/`)+- [ ] macOS (`macos/`)+- [ ] iOS (`ios/`)+- [ ] Android (`android/`)+- [ ] Windows (`windows/`)+- [ ] Docs / other++## How was it tested?++<!-- Commands you ran, hardware/chip, model tier. Paste bench.py numbers for perf changes. -->++## Checklist++- [ ] `pytest -m 'not slow'` passes locally (Python changes; CI runs this on macOS + MLX)+- [ ] Repo hygiene passes: `pytest tests/test_repo_hygiene.py` (no hardcoded paths, no secrets, **no MLX imports outside `backends/mlx/`**)+- [ ] Added/updated tests for new behavior+- [ ] Did **not** bump `mlx` past the pinned `0.30.6` (see the note in `python/pyproject.toml` / issue #8) unless this PR is specifically about that+- [ ] Updated relevant docs / READMEs (incl. translations if user-facing)