The 80B Qwen3-Coder-Next on a 16 GB Mac, about 13 tok/s, fully offline. A mixture-of-experts inference engine for Apple silicon.
Rust
docs: per-model measurements (speed, 8-task quality check) and the script to rerun them
1picassoai413f3b62461f
diff --git a/README.md b/README.mdindex 3a8bfa1..a93d5ef 100644--- a/README.md+++ b/README.md@@ -61,9 +61,11 @@ Coachwhip is an engine for MoE models, in GGUF format.| Qwen3-30B-A3B, other Qwen3 MoE | qwen3moe | Supported. || Mixtral, Qwen2-MoE, OLMoE, DeepSeek, gpt-oss, GLM | | Coming next. |+Speed and a quality check for each model file, measured on a 16 GB Mac: [docs/MODELS.md](docs/MODELS.md).+Try other MoE models and tell us what happens. If a model's architecture is not supported yet, Coachwhip says so, and an issue is the fastest way to move it up the list.-**Which GGUF file.** Coachwhip reads the standard GGUF quantisations (Q4_K, Q5_K, Q6_K, Q8_0, F16). Some newer conversions store a few tensors in MXFP4 or fuse the attention weights into one tensor; Coachwhip does not read those yet. For Qwen3-Coder-Next, [MaziyarPanahi's Q4_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) loads as is.+**Which GGUF file.** Coachwhip reads the standard GGUF quantisations (Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, F16). Some newer conversions store a few tensors in MXFP4 or fuse the attention weights into one tensor; Coachwhip does not read those yet. For Qwen3-Coder-Next, [MaziyarPanahi's Q4_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) loads as is.## What you needdiff --git a/bench/quality/run.py b/bench/quality/run.pynew file mode 100644index 0000000..1235a3f--- /dev/null+++ b/bench/quality/run.py@@ -0,0 +1,55 @@+"""Quality check: ask the running Coachwhip the 8 tasks in tasks.json, run each answer against tests.py.++Start Coachwhip with --temperature 0 --chat 8090 first, then: python3 bench/quality/run.py+Standard library only. Answers and their code go to ./coachwhip-quality/ in the current folder.+"""+import json, os, re, subprocess, sys, urllib.request++HERE = os.path.dirname(os.path.abspath(__file__))+OUT = os.path.abspath("coachwhip-quality")+URL = os.environ.get("COACHWHIP_URL", "http://127.0.0.1:8090/v1/chat/completions")++RUNNER = r'''+import sys+sys.path.insert(0, sys.argv[1])+import tests+ns = {}+exec(open(sys.argv[2], encoding="utf-8").read(), ns)+getattr(tests, "t_" + sys.argv[3])(ns)+print("PASS")+'''+++def ask(prompt):+ body = json.dumps({"model": "m", "max_tokens": 1200, "messages": [{"role": "user", "content": prompt}]}).encode()+ req = urllib.request.Request(URL, body, {"Content-Type": "application/json"})+ return json.load(urllib.request.urlopen(req, timeout=900))["choices"][0]["message"]["content"]+++def code_of(text):+ blocks = re.findall(r"```(?:python|py)?\s*\n(.*?)```", text, re.S)+ return "\n\n".join(blocks) if blocks else text+++def main():+ tasks = json.load(open(os.path.join(HERE, "tasks.json")))+ os.makedirs(OUT, exist_ok=True)+ answers, passed = {}, 0+ for t in tasks:+ answers[t["id"]] = ask(t["prompt"])+ path = os.path.join(OUT, f"answer_{t['id']}.py")+ open(path, "w", encoding="utf-8").write(code_of(answers[t["id"]]))+ try:+ p = subprocess.run([sys.executable, "-c", RUNNER, HERE, path, t["id"]], capture_output=True, text=True, timeout=10)+ ok = "PASS" in p.stdout+ why = "" if ok else (p.stderr.strip().splitlines() or ["?"])[-1][:100]+ except subprocess.TimeoutExpired:+ ok, why = False, "timeout"+ passed += ok+ print(f"{t['id']:12s} {'pass' if ok else 'FAIL'} {why}", flush=True)+ json.dump(answers, open(os.path.join(OUT, "answers.json"), "w"), indent=1)+ print(f"total {passed} / {len(tasks)}")+++if __name__ == "__main__":+ main()diff --git a/bench/quality/tasks.json b/bench/quality/tasks.jsonnew file mode 100644index 0000000..93223c1--- /dev/null+++ b/bench/quality/tasks.json@@ -0,0 +1,10 @@+[+ {"id": "intervals", "prompt": "Write a Python function merge_intervals(intervals) that takes a list of [start, end] pairs (in any order, possibly empty) and returns the merged list of non-overlapping intervals sorted by start. Intervals that touch, like [1,3] and [3,5], are merged. Return only the code in one python block."},+ {"id": "lru", "prompt": "Write a Python class LRUCache with __init__(self, capacity), get(self, key) returning the value or -1 if missing, and put(self, key, value). When putting a new key into a full cache, evict the least recently used key. Both get and put count as a use. Return only the code in one python block."},+ {"id": "duration", "prompt": "Write a Python function parse_duration(s) that turns strings like '2h', '45m', '90s', '1h30m' and '1h0m5s' into a total number of seconds (int). Units are h, m, s, each at most once, always in that order. Raise ValueError for an empty string or anything else invalid, such as '1x', 'h1', '1m1h' or '1h1h'. Return only the code in one python block."},+ {"id": "topk", "prompt": "Write a Python function top_k_frequent(words, k) that returns the k most frequent words, ordered by frequency (highest first) and, for equal frequency, alphabetically. Return only the code in one python block."},+ {"id": "rpn", "prompt": "Write a Python function eval_rpn(tokens) that evaluates a list of string tokens in reverse Polish notation with the operators +, -, * and /, where division truncates toward zero (so 7 / -2 is -3), and returns an int. Return only the code in one python block."},+ {"id": "palindrome", "prompt": "Write a Python function longest_palindrome(s) that returns the longest palindromic substring of s. If several have the same maximum length, return the one that starts first. For an empty string return ''. Return only the code in one python block."},+ {"id": "wrap", "prompt": "Write a Python function wrap(text, width) that splits text into words on whitespace and greedily packs them into lines of at most width characters, words separated by single spaces, returning the list of lines. A word longer than width goes on its own line unbroken. For text with no words return []. Return only the code in one python block."},+ {"id": "brackets", "prompt": "Write a Python function balanced(s) that returns True if every bracket in s, of the kinds (), [] and {}, is properly matched and nested, ignoring all other characters, and False otherwise. Return only the code in one python block."}+]diff --git a/bench/quality/tests.py b/bench/quality/tests.pynew file mode 100644index 0000000..15f48ec--- /dev/null+++ b/bench/quality/tests.py@@ -0,0 +1,97 @@+"""Tests for the quality check in tasks.json: one function per task id, each raising on a failure.+The model never sees these; run.py asks the tasks and runs each answer against them."""+++def t_intervals(ns):+ f = ns["merge_intervals"]+ assert f([]) == []+ assert f([[1, 3], [2, 6], [8, 10], [15, 18]]) == [[1, 6], [8, 10], [15, 18]]+ assert f([[1, 3], [3, 5]]) == [[1, 5]]+ assert f([[5, 6], [1, 2], [2, 4]]) == [[1, 4], [5, 6]]+ assert f([[1, 10], [2, 3], [4, 5]]) == [[1, 10]]+ assert [list(x) for x in f([[1, 4], [0, 0]])] == [[0, 0], [1, 4]]+++def t_lru(ns):+ C = ns["LRUCache"]+ c = C(2)+ c.put(1, 1); c.put(2, 2)+ assert c.get(1) == 1+ c.put(3, 3)+ assert c.get(2) == -1+ c.put(4, 4)+ assert c.get(1) == -1+ assert c.get(3) == 3 and c.get(4) == 4+ c = C(2)+ c.put(1, 1); c.put(2, 2); c.put(1, 10)+ c.put(3, 3)+ assert c.get(2) == -1 and c.get(1) == 10+ c = C(1)+ c.put(1, 1); c.put(2, 2)+ assert c.get(1) == -1 and c.get(2) == 2+++def t_duration(ns):+ f = ns["parse_duration"]+ assert f("2h") == 7200+ assert f("45m") == 2700+ assert f("90s") == 90+ assert f("1h30m") == 5400+ assert f("1h0m5s") == 3605+ assert f("10m5s") == 605+ for bad in ["", "1x", "h1", "1m1h", "1h1h", "abc"]:+ try:+ f(bad)+ except ValueError:+ continue+ raise AssertionError(f"no ValueError for {bad!r}")+++def t_topk(ns):+ f = ns["top_k_frequent"]+ assert f(["i", "love", "leetcode", "i", "love", "coding"], 2) == ["i", "love"]+ w = ["the", "day", "is", "sunny", "the", "the", "the", "sunny", "is", "is"]+ assert f(w, 4) == ["the", "is", "sunny", "day"]+ assert f(["b", "a", "c"], 2) == ["a", "b"]+++def t_rpn(ns):+ f = ns["eval_rpn"]+ assert f(["2", "1", "+", "3", "*"]) == 9+ assert f(["4", "13", "5", "/", "+"]) == 6+ assert f(["10", "6", "9", "3", "+", "-11", "*", "/", "*", "17", "+", "5", "+"]) == 22+ assert f(["7", "-2", "/"]) == -3+ assert f(["-7", "2", "/"]) == -3+ assert f(["3", "4", "-"]) == -1+++def t_palindrome(ns):+ f = ns["longest_palindrome"]+ assert f("") == ""+ assert f("a") == "a"+ assert f("babad") == "bab"+ assert f("cbbd") == "bb"+ assert f("forgeeksskeegfor") == "geeksskeeg"+ assert f("abcd") == "a"+++def t_wrap(ns):+ f = ns["wrap"]+ assert f("", 10) == []+ assert f(" ", 10) == []+ assert f("the quick brown fox", 10) == ["the quick", "brown fox"]+ assert f("a bb ccc", 4) == ["a bb", "ccc"]+ assert f("supercalifragilistic is long", 10) == ["supercalifragilistic", "is long"]+ assert f("one two\nthree", 7) == ["one two", "three"]+++def t_brackets(ns):+ f = ns["balanced"]+ assert f("") is True+ assert f("()[]{}") is True+ assert f("{[()]}") is True+ assert f("a(b[c]{d}e)f") is True+ assert f("(]") is False+ assert f("([)]") is False+ assert f("((") is False+ assert f(")(") is Falsediff --git a/docs/MODELS.md b/docs/MODELS.mdnew file mode 100644index 0000000..f09a1d8--- /dev/null+++ b/docs/MODELS.md@@ -0,0 +1,37 @@+# Models++What each model file does on a 16 GB Mac, measured the same way every time. These are measurements, not promises: your Mac, your prompt and your other apps will move them. Run the same checks yourself (below) and post what you get in an [issue](https://github.com/1picassoai/coachwhip/issues).++## How we measure++- **Machine:** Mac mini M4, 16 GB, macOS 26.+- **Engine:** Coachwhip 0.2.0, temperature 0.+- **Speed:** one answer to *"Write a Python function that merges overlapping intervals, with a short explanation."*, capped at 300 tokens. "Writing" is tokens written per second once the answer has started, as Coachwhip counts it: the same figure the chat shows under each answer.+- **Quality:** 8 small coding tasks in [`bench/quality/tasks.json`](../bench/quality/tasks.json). Each answer's code is run against tests the model never sees ([`bench/quality/tests.py`](../bench/quality/tests.py)); a task passes only if every test does. Quality depends on the model file, not on Coachwhip's settings: the bank and the prefetch change how fast an answer comes, never what it says.++## Qwen3-Coder-Next (80B)++| GGUF file | Size | `--bank` on 16 GB | Bank memory | Writing | Quality |+|---|---|---|---|---|---|+| [MaziyarPanahi Q4_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) | 48.4 GB | 56 | 5.5 GB | 3.4 tok/s | 7 / 8 |+| [MaziyarPanahi Q3_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) | 38.2 GB | 70 | 5.5 GB | 4.5 tok/s | 7 / 8 |++Speed measured 3 Oct 2026; quality measured 4 Oct 2026. At bank 56, Q3_K_M wrote 4.4 tok/s.++- **Why Q3_K_M is faster:** its experts are 1.6 MB instead of 2.0 MB, so each one read from the SSD arrives sooner, and the same bank memory holds 70 of them per layer instead of 56.+- **Quality:** both files failed one task, a different one each. Q4_K_M's word wrap counted a space before the first word of each line; Q3_K_M's duration parser accepted a repeated unit (`1h1h`). Eight tasks is a small sample: treat it as "no difference found", not "no difference".+- Both use the same [tokenizer.json](https://huggingface.co/Qwen/Qwen3-Coder-Next).++## Not measured yet++Qwen3-Coder-30B-A3B and Qwen3-Next-80B-A3B-Instruct run on Coachwhip (see the README) but have not been through these checks yet.++## Run it yourself++```sh+~/coachwhip/target/release/coachwhip --model /path/to/model.gguf --tokenizer /path/to/tokenizer.json \+ --bank 56 --temperature 0 --chat 8090+python3 bench/quality/run.py+```++`run.py` asks the 8 tasks through Coachwhip's API, runs each answer against the tests, and prints pass or fail per task. It needs only Python 3. The chat's speed line gives the writing speed for any prompt you like.