Skip to content

The 80B Qwen3-Coder-Next on a 16 GB Mac, about 13 tok/s, fully offline. A mixture-of-experts inference engine for Apple silicon.

Rust

0.5.1 docs: the tester's numbers (about 12 tok/s, 11.5 at 13K), disk numbers, split marked not yet measured

1picassoai37f2ac1172c8
diff --git a/README.md b/README.md
index 3375dd5..03b62d3 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@
# Coachwhip
-**The 80-billion-parameter Qwen3-Coder-Next on a 16 GB Mac, at about 13 tok/s. Fully offline. Nothing leaves your machine.**
+**The 80-billion-parameter Qwen3-Coder-Next on a 16 GB Mac, at about 12 tok/s. Fully offline. Nothing leaves your machine.**
```sh
curl -fsSL https://raw.githubusercontent.com/1picassoai/coachwhip/main/install.sh | bash
@@ -23,7 +23,7 @@ curl -fsSL https://raw.githubusercontent.com/1picassoai/coachwhip/main/install.s
~/coachwhip/coachwhip.sh ~/Downloads/Qwen3-Coder-Next.Q2X.gguf ~/Downloads/tokenizer.json 109 8090 --parallel --flash-reads --stand-ins
```
-That is fast mode, about 13 tok/s on a real answer; drop `--stand-ins` for every expert read at about 5. Measured on a Mac mini M4 with 16 GB; the numbers and the quality check are in the [release notes](docs/releases/0.5.1.md).
+That is fast mode: 13.4 tok/s on a short prompt, 11.5 tok/s with a 13,000-token prompt, three runs each, with 3 GB left for macOS at its tightest; drop `--stand-ins` for every expert read at about 5. Measured on a Mac mini M4 with 16 GB; the numbers and the quality check are in the [release notes](docs/releases/0.5.1.md).
**Also on a 16 GB Mac: OpenAI's gpt-oss-120b at about 10 tok/s.** It thinks before it answers. Four steps, because its official file is squeezed on your own Mac (106 GB free while it runs, 43 GB after):
diff --git a/docs/releases/0.5.1.md b/docs/releases/0.5.1.md
index 0f38777..01500c9 100644
--- a/docs/releases/0.5.1.md
+++ b/docs/releases/0.5.1.md
@@ -4,8 +4,9 @@ Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers. Th
## The 80B coder, ready-squeezed
-- **Qwen3-Coder-Next, 80 billion parameters, 3 billion active, on a 16 GB Mac at about 13 tok/s.** The machine cannot hold the 38 GB file; Coachwhip streams the experts it needs and keeps the rest asleep.
+- **Qwen3-Coder-Next, 80 billion parameters, 3 billion active, on a 16 GB Mac at about 12 tok/s.** The machine cannot hold the 38 GB file; Coachwhip streams the experts it needs and keeps the rest asleep.
- **Download the squeezed file instead of squeezing it yourself.** `Qwen3-Coder-Next.Q2X.gguf`, 29,335,336,000 bytes, sha256 `cdc0584c1e4d30134cbb83f18e4c7607af90bd1145a64940c7a357b02a0216af`, made by the shipped `coachwhip-prepare` from MaziyarPanahi's Q3_K_M (gate and up experts to Q2_K, down to Q3_K; everything else untouched). Download: [huggingface.co/vinnj/coachwhip-models](https://huggingface.co/vinnj/coachwhip-models), the one place for every squeezed file we host; each file's sha256 and source are listed there. Check the hash after the download: `shasum -a 256 ~/Downloads/Qwen3-Coder-Next.Q2X.gguf`.
+- **Disk.** About 30 GB free to download our file and keep it. To make it yourself from the 38 GB Q3_K_M with `coachwhip-prepare`, about 67 GB free, source and output together, for the 211 seconds it takes; then delete the source.
- **One line to start it**, bank 109, fast mode on: `~/coachwhip/coachwhip.sh ~/Downloads/Qwen3-Coder-Next.Q2X.gguf ~/Downloads/tokenizer.json 109 8090 --parallel --flash-reads --stand-ins`
## Measured, same prompt, same Mac
@@ -13,9 +14,11 @@ Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers. Th
| setup | writing | quality on the retry-helper prompt |
|---|---|---|
| plain Q3_K_M file, bank 70 | 5.4 tok/s | 5 of 5 |
-| squeezed file, stand-ins on, bank 109 | 13.5 tok/s | 4.5 of 5 |
+| squeezed file, stand-ins on, bank 109 | 12.8 to 13.4 tok/s short prompts; 11.5 tok/s at a 13,061-token prompt (read at 34 tok/s); three runs each, tester's hand | 4.5 of 5 |
-Raw 300-word runs: 6.2 plain, 10.4 with stand-ins on the plain file, 11.9 squeezed with stand-ins. Bytes read from the SSD per 300 words: 31 GB, 6 GB, 5 GB. Fast mode changes the answers: the half point was two faults in the usage example, one an undefined function; the loop and the error handling were right. The split between the squeeze's share and the stand-ins' share: (filled in from the tester's runs).
+Tester's runs on the squeezed file: a trapped LRU cache with TTL, code extracted and executed, 7 of 7 checks pass, no timers; a 13,061-token read of our own docs returned three accurate risks. Memory at the 13K prompt: 10.67 GB peak of the 12.71 GB GPU limit, 3.00 GB left for macOS at its tightest. The squeeze is reproducible: the file remade from the same source with the shipped `coachwhip-prepare` has the same sha256.
+
+Raw 300-word runs, builder's hand: 6.2 plain, 10.4 with stand-ins on the plain file, 11.9 squeezed with stand-ins. Bytes read from the SSD per 300 words: 31 GB, 6 GB, 5 GB. Fast mode changes the answers: the half point on the retry-helper prompt was two faults in the usage example, one of them a call to an undefined function, the "looks finished, does not run" kind; the loop and the error handling were right. Not yet measured: which of the two levers, the squeeze or the stand-ins, costs that half point.
## Changed words