The 80B Qwen3-Coder-Next on a 16 GB Mac, about 13 tok/s, fully offline. A mixture-of-experts inference engine for Apple silicon.
Rust
Commits on main
docs: squeeze takes 5 to 9 minutes, needs about 105 GB free while both files exist, and is deterministic (tester, 9 Oct)
bd964c5 Merlin committed 1 month ago
docs: the 122B claims 12K, what the tester measured (Captain, 9 Oct); no 16K claim for it
73fb4ed Merlin committed 1 month ago
docs: bank 40, the two-row table (6K answer, 12K prompt: speed, memory used, left over) for the tester to fill, and Wrangler in the notes
d1d50f2 Merlin committed 1 month ago
Wrangler 2: the governor caps the bank so a full 16K chat fits under the GPU limit, and warns on macOS memory pressure
03fc380 Merlin committed 1 month ago
Wrangler 1: the word table stays packed in ordinary memory; only the rows a forward needs are unpacked
0d25ee1 Merlin committed 1 month ago
bank at its exact size: candle pool rounds buffers to a power of two (1.15 GB lost at bank 32 on the squeezed 122B); plus a GPU memory print
c3eb493 Merlin committed 1 month ago
parallel path: stamp each published pick with its job, so a stale mix can never be placed
e606dfe Merlin committed 1 month ago
cli: per-model guess defaults; the 122B runs prefetch 6, ahead 2 unless told otherwise
dfa1756 Merlin committed 1 month ago
0.4.0: the 122B after a local squeeze; two modes; README, models page and release notes
12a82c6 Picassoai committed 1 month ago
prepare: squeeze gate and up on the GPU (requant), down on the CPU to Q3_K; 45 GB in 5 min
e37c398 Picassoai committed 1 month ago
lab: heavy-reuse re-read, register repair, admission line, no-squeeze mechanics switch, prompt-phase bank fix — all off by default; dense shrink found to poison text, stays off
270810e Picassoai committed 1 month ago
lab: admission line on the use-sharp flow; fix: never make the banks during the prompt (the two-kinds change had moved the check); memory diagnostics
16bd78d Picassoai committed 1 month ago
lab: by-weight gate, use-sharp-then-squeeze flow (Captain), memory diagnostics — all off by default
6dce306 Picassoai committed 1 month ago
lab: GPU shrink (Q2_K kernels Q3K/Q4K/Q5K), two kinds of slot, dense shrink at load, probability-rule prefetch, shaped fit, measurement switches — all off by default, nothing public
3fed327 Picassoai committed 1 month ago
Installer pinned to the signed release tag; README opens with the measured numbers and shows Qwen3.6
be00834 1picassoai committed 1 month ago
MODELS.md: the 35B speed named as one person, one run, with the load, like the 80B's
c18ddc4 1picassoai committed 1 month ago
Qwen3.6-35B-A3B measured: 6.2 tok/s at bank 44, 8/8 quality; bank 44 recommended, with why
2b931c6 1picassoai committed 1 month ago
README: what's new in 0.3.0, the Qwen3.6 rows, the thinking switch; first release notes in docs/releases/0.3.0.md
69f0001 1picassoai committed 1 month ago
Qwen3.5/3.6: pair value heads with key heads the way the family does, and open the turn as its template says
84a5873 1picassoai committed 1 month ago
Qwen3.5 and 3.6 (qwen35moe): the Qwen3-Next family with split linear-attention tensors
a2b2eec 1picassoai committed 1 month ago
MODELS.md: the Q3 speed as three runs by two people, with the load each was measured under
ff5cc25 1picassoai committed 1 month ago
tests-mac: scripts are executable in git, so a fresh clone can run them
a0aaa86 1picassoai committed 1 month ago
MODELS.md: say two prompts finish before the cap; .gitattributes: shell scripts always LF
f86b4d1 1picassoai committed 1 month ago
0.3.0: measured on five prompts; Q3_K_M at bank 70 is the recommended file
05b2fd5 1picassoai committed 1 month ago
bench/speed: five prompts and a script that prints each answer's writing speed; MODELS.md measures with it
2d54749 1picassoai committed 1 month ago
Early guess: a later layer's router, run early, picks the experts to read ahead (v0.3.0)
bb0a181 1picassoai committed 1 month ago
tests-mac: same answers with every speed setting, and models load or are refused cleanly
85a3afc 1picassoai committed 1 month ago
Tests: router, bank, prefetch guess and API door; their decisions become small pure functions
fcb1cf6 1picassoai committed 1 month ago