The 80B Qwen3-Coder-Next on a 16 GB Mac, about 13 tok/s, fully offline. A mixture-of-experts inference engine for Apple silicon.
Rust
0.5.1 docs: the tester's numbers (about 12 tok/s, 11.5 at 13K), disk numbers, split marked not yet measured
1picassoai37f2ac1172c8
diff --git a/README.md b/README.mdindex 3375dd5..03b62d3 100644--- a/README.md+++ b/README.md@@ -7,7 +7,7 @@# Coachwhip-**The 80-billion-parameter Qwen3-Coder-Next on a 16 GB Mac, at about 13 tok/s. Fully offline. Nothing leaves your machine.**+**The 80-billion-parameter Qwen3-Coder-Next on a 16 GB Mac, at about 12 tok/s. Fully offline. Nothing leaves your machine.**```shcurl -fsSL https://raw.githubusercontent.com/1picassoai/coachwhip/main/install.sh | bash@@ -23,7 +23,7 @@ curl -fsSL https://raw.githubusercontent.com/1picassoai/coachwhip/main/install.s~/coachwhip/coachwhip.sh ~/Downloads/Qwen3-Coder-Next.Q2X.gguf ~/Downloads/tokenizer.json 109 8090 --parallel --flash-reads --stand-ins```-That is fast mode, about 13 tok/s on a real answer; drop `--stand-ins` for every expert read at about 5. Measured on a Mac mini M4 with 16 GB; the numbers and the quality check are in the [release notes](docs/releases/0.5.1.md).+That is fast mode: 13.4 tok/s on a short prompt, 11.5 tok/s with a 13,000-token prompt, three runs each, with 3 GB left for macOS at its tightest; drop `--stand-ins` for every expert read at about 5. Measured on a Mac mini M4 with 16 GB; the numbers and the quality check are in the [release notes](docs/releases/0.5.1.md).**Also on a 16 GB Mac: OpenAI's gpt-oss-120b at about 10 tok/s.** It thinks before it answers. Four steps, because its official file is squeezed on your own Mac (106 GB free while it runs, 43 GB after):diff --git a/docs/releases/0.5.1.md b/docs/releases/0.5.1.mdindex 0f38777..01500c9 100644--- a/docs/releases/0.5.1.md+++ b/docs/releases/0.5.1.md@@ -4,8 +4,9 @@ Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers. Th## The 80B coder, ready-squeezed-- **Qwen3-Coder-Next, 80 billion parameters, 3 billion active, on a 16 GB Mac at about 13 tok/s.** The machine cannot hold the 38 GB file; Coachwhip streams the experts it needs and keeps the rest asleep.+- **Qwen3-Coder-Next, 80 billion parameters, 3 billion active, on a 16 GB Mac at about 12 tok/s.** The machine cannot hold the 38 GB file; Coachwhip streams the experts it needs and keeps the rest asleep.- **Download the squeezed file instead of squeezing it yourself.** `Qwen3-Coder-Next.Q2X.gguf`, 29,335,336,000 bytes, sha256 `cdc0584c1e4d30134cbb83f18e4c7607af90bd1145a64940c7a357b02a0216af`, made by the shipped `coachwhip-prepare` from MaziyarPanahi's Q3_K_M (gate and up experts to Q2_K, down to Q3_K; everything else untouched). Download: [huggingface.co/vinnj/coachwhip-models](https://huggingface.co/vinnj/coachwhip-models), the one place for every squeezed file we host; each file's sha256 and source are listed there. Check the hash after the download: `shasum -a 256 ~/Downloads/Qwen3-Coder-Next.Q2X.gguf`.+- **Disk.** About 30 GB free to download our file and keep it. To make it yourself from the 38 GB Q3_K_M with `coachwhip-prepare`, about 67 GB free, source and output together, for the 211 seconds it takes; then delete the source.- **One line to start it**, bank 109, fast mode on: `~/coachwhip/coachwhip.sh ~/Downloads/Qwen3-Coder-Next.Q2X.gguf ~/Downloads/tokenizer.json 109 8090 --parallel --flash-reads --stand-ins`## Measured, same prompt, same Mac@@ -13,9 +14,11 @@ Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers. Th| setup | writing | quality on the retry-helper prompt ||---|---|---|| plain Q3_K_M file, bank 70 | 5.4 tok/s | 5 of 5 |-| squeezed file, stand-ins on, bank 109 | 13.5 tok/s | 4.5 of 5 |+| squeezed file, stand-ins on, bank 109 | 12.8 to 13.4 tok/s short prompts; 11.5 tok/s at a 13,061-token prompt (read at 34 tok/s); three runs each, tester's hand | 4.5 of 5 |-Raw 300-word runs: 6.2 plain, 10.4 with stand-ins on the plain file, 11.9 squeezed with stand-ins. Bytes read from the SSD per 300 words: 31 GB, 6 GB, 5 GB. Fast mode changes the answers: the half point was two faults in the usage example, one an undefined function; the loop and the error handling were right. The split between the squeeze's share and the stand-ins' share: (filled in from the tester's runs).+Tester's runs on the squeezed file: a trapped LRU cache with TTL, code extracted and executed, 7 of 7 checks pass, no timers; a 13,061-token read of our own docs returned three accurate risks. Memory at the 13K prompt: 10.67 GB peak of the 12.71 GB GPU limit, 3.00 GB left for macOS at its tightest. The squeeze is reproducible: the file remade from the same source with the shipped `coachwhip-prepare` has the same sha256.++Raw 300-word runs, builder's hand: 6.2 plain, 10.4 with stand-ins on the plain file, 11.9 squeezed with stand-ins. Bytes read from the SSD per 300 words: 31 GB, 6 GB, 5 GB. Fast mode changes the answers: the half point on the retry-helper prompt was two faults in the usage example, one of them a call to an undefined function, the "looks finished, does not run" kind; the loop and the error handling were right. Not yet measured: which of the two levers, the squeeze or the stand-ins, costs that half point.## Changed words