Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docs: Gyro on the Mac (MAC.md, GYRO.md, MODELS.md, README, 0.2.2 release notes)
Niko1221ca40c88ef445
diff --git a/README.md b/README.mdindex 4b7b92e40..e476898dc 100644--- a/README.md+++ b/README.md@@ -159,7 +159,7 @@ Smaller sizes are faster. Larger sizes are a bit smarter.| **Gyro-S** | 27.6 GB | 58.5 GB | For PCs that fit nothing else. Quality is lower than IQ2_XS. || **Gyro-M** | 35 GB | 92.0 GB | About IQ3_XXS quality with about 20% less GPU memory. It writes answers 17-21% slower than Q2_0 on a 12 GB card and on an R9700. |-Gyro runs on NVIDIA and AMD cards (AMD is experimental), not on Intel Arc or the Mac. Details and measurements:+Gyro runs on NVIDIA and AMD cards (AMD is experimental) and, from 0.2.2, on Apple Silicon Macs (Gyro-S on a 32 GB Mac mini: 9 tok/s, streaming from the SSD; [docs/MAC.md](docs/MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts)), not on Intel Arc. Details and measurements:[docs/GYRO.md](docs/GYRO.md). Start it with `strata launch --model gyro-s`.### The other sizesdiff --git a/docs/GYRO.md b/docs/GYRO.mdindex c53a9ce7e..b8d6aca58 100644--- a/docs/GYRO.md+++ b/docs/GYRO.md@@ -280,3 +280,10 @@ Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, J- **Gyro-M:** 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache; `auto` streams the rest.- **Not validated end to end yet** (Strata runs them and logs a warning): an AMD GPU other than gfx1151, andprompt processing on a peer GPU over P2P.++## On a Mac++Apple Silicon Macs run Gyro-S and Gyro-M from 0.2.2 (Metal kernels in `metal/kernels`, the same published files, the same+Hadamard-folded pack). What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in+[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup: `./setup.sh --family gyro --model Gyro-S` (the app and+`strata launch` list it under Gyro models).diff --git a/docs/MAC.md b/docs/MAC.mdindex d21131e85..fb1fa2f67 100644--- a/docs/MAC.md+++ b/docs/MAC.md@@ -34,6 +34,8 @@ with 32 GB and an M1 Max with 34 GB, where the experts stream from the SSD (the| IQ3_S (recommended; the default from 96 GB) | 54.8 GB | 63 GB | 96 GB || UD-Q4_K_XL (Unsloth, recommended: higher quality) | 82.6 GB | 91 GB | 128 GB || UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | stream from the SSD on 128 GB | 128 GB, streamed |+| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the file's other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that (measured on 32 GB) |+| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that (not measured on a Mac) |- **The n-gram table:** each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at a time and does not count against the memory above.- **Disk:** the download is 66-84 GB, plus about 7 GB for the pack and the draft layer. A Mac that streams also needs room for the experts' copy (below): 34-40 GB.@@ -169,6 +171,40 @@ The arena was 11.5-14.7 GB. Tokens per second, one token per step (no MTP), medi`--spec 2` is the best default on mixed text. An explicit `--spec N` in the config's `args` is kept.+### Gyro-S and Gyro-M (agentionai's trellis experts)++From 0.2.2 the Mac engine runs [Gyro](GYRO.md): the routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a+Hadamard-rotated basis, read from the published file unchanged. The kernels are ours (`metal/kernels/quant.metal`, `moe.metal`,+`moe_mm.metal`); the codebook and the decoder are the CPU engine's, the rotation is applied to the gate/up and down inputs as on CUDA.+Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the default, and says+what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD like every other+pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident (29.7 GB of weights).++**Measured on the M4 Mac mini, 32 GB** (the experts stream from the SSD; no experts' copy, because the copy would need 26 GB more+disk than this Mac had; one token per step, no draft layer; the arena was 11-12 GB of the 25.8 GB of experts):++| | Gyro-S |+|---|---|+| Decode, story, 120 tokens (5 interleaved pairs, medians) | **9.14 tok/s** (9.07-9.15) |+| Decode, with the dense-weight tiles turned off (`STRATA_METAL_GYRO_TILES=0`) | 8.70 tok/s (8.69-8.73); same tokens in all 10 runs |+| Prompt, 4,065 tokens (3 runs) | **109 tok/s** (108-110) |+| Memory the engine held | 18.6 GB of the 32 GB |++Where the time goes: a decode window of two tokens takes 111 ms. The routed experts are about 40 ms of it, the dense weights+(Gyro keeps its attention, state-space and shared-expert matrices in Q5_K and Q6_K: 3.6 GB read a token)+about 66 ms before the tile change (a run with those kernels skipped). The kernel for one expert layer (ten experts, one token) takes+278 us on the M4, 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency and ALU, not by memory:+32-bit funnel shifts for the stream bits, fetching the next unit while decoding the current one, one lane a row with the unit index+folded away and the codebook in constant memory were all tried and none beat the plain kernel, so it is the plain kernel with the+tile that measured fastest. The prompt kernels (fp16 matrix units) are on par with IQ3_XXS: 68 ms against 62 ms a layer for+1,024 tokens.++**Checked against the CPU engine's decoder, on the real file:** the decoder equals the format's definition written out bit by bit+(614,400 sampled blocks a layer, none differ); the Hadamard rotation equals the CPU's bit for bit (f32 and f16); the expert+kernels reproduce a double-precision reference to 1e-7 relative on real experts of layers 0 and 47 of Gyro-S and of layers 0, 23+and 47 of Gyro-M (the 92 GB file was read by range requests, 16 experts a layer: it does not fit this Mac's disk). Run it with+`metal/tests/gyro_parity.sh`.+### Quality- **The Q8_0 hyper-connection read is the default on Mac.** It keeps a Q8_0 copy of the hyper-connection matrices instead of BF16. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read.@@ -208,7 +244,7 @@ The arena was 11.5-14.7 GB. Tokens per second, one token per step (no MTP), medi- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).- **Streaming:** measured at 32 GB only. The model's file must be on the internal SSD; an external disk is untested and likely much slower.-- **Families:** the original Qwen3.8-Flash-Next and Unsloth's UD-Q4/Q5/Q6_K_XL are measured (above). Setup repacks Unsloth's shards into one file for the GPU. The Swift and Coder releases are not set up on Mac yet, and Qwen3.6 does not run on the Mac engine.+- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured (above). Setup repacks Unsloth's shards into one file for the GPU; a Gyro file is used as it is. Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac here could hold it. The Swift and Coder releases are not set up on Mac yet, and Qwen3.6 does not run on the Mac engine.- **The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips.** The zips(for `START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it.diff --git a/docs/MODELS.md b/docs/MODELS.mdindex c51fae258..9c755098d 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -19,7 +19,7 @@ other size stays a choice. A PC with a card of 12 GB and 32 GB of RAM is in the| Your memory (RAM + VRAM) | Pre-selected | Notes || --- | --- | --- || **32 GB** | **Gyro-S**, or the **Coder** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code ([why](#coder)). Gyro-S has lower quality than the 2-3-bit sizes, and the 2-bit Q2_0 decodes faster |-| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 17-21% slower than Q2_0. Gyro runs on NVIDIA and AMD cards, not on Intel Arc or the Mac |+| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 17-21% slower than Q2_0. Gyro runs on NVIDIA and AMD cards and, from 0.2.2, on Macs (Gyro-S measured on a 32 GB Mac mini), not on Intel Arc || **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better || **64 GB** | **IQ3_S** (54.8 GB), the original. Swift 1.5's IQ3_S is an equal choice | The slowest, the best: it matches the full model on the published tests || **96 GB or more** | [Unsloth's UD-IQ4_XS to Q8_0](#unsloths-files-ud-iq4_xs-to-q8_0), 4 to 8 bit | UD-IQ4_XS keeps every expert in RAM from about 80 GB of RAM, Q8_0 from about 136 GB. Up to 192 GB, the 5-8-bit files fit in RAM. Setup still pre-selects IQ3_S here, except on a Strix Halo |@@ -183,7 +183,7 @@ experts in a rotated basis. They are the smallest options and never setup's defa| **Gyro-S** | 58.5 GB | 27.6 GB | the smallest option, for PCs with little memory; lower quality || **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory; decodes about 20% slower than Q2_0 on 12 GB cards |-NVIDIA, and AMD (experimental); not Intel Arc or the Mac. Gyro reads pictures (the same rule as above). It needs+NVIDIA, AMD (experimental) and, from 0.2.2, Apple Silicon Macs ([MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts): Gyro-S is measured on a 32 GB Mac mini, Gyro-M only tensor by tensor); not Intel Arc. Gyro reads pictures (the same rule as above). It needsengine 0.1.45.1 or newer; if the engine is older, the app says "This model needs a newer engine" and offers Updateengine. Gyro gives correct output on RTX 50 cards with engine 0.1.45.3. Details, measurements and credit:[GYRO.md](GYRO.md).diff --git a/docs/release-notes-0.2.2.md b/docs/release-notes-0.2.2.mdindex 15a55b096..828be5b09 100644--- a/docs/release-notes-0.2.2.md+++ b/docs/release-notes-0.2.2.md@@ -17,3 +17,4 @@- **Docker image build.** The terminal stage no longer needs git in the image (it failed with `FileNotFoundError`); the commit comes in as a build argument.- **The app and the Terminal share one models folder.** Each of the two settings files now remembers when the models folder was chosen; the one changed last wins and the other follows. Nothing is moved. A folder picked in the Terminal was not seen by the app before, so the same model could be downloaded twice.- **Docker.** An old (0.1.x) volume with `/data/models`, `/data/packs` and `/data/mtp` is moved into `/data/Strata-data` by rename instead of being downloaded and packed again (it used to start a 70 GB download). A card without `--gpus` stops at once with the `--gpus all` hint instead of after minutes of setup. A missing or empty `MODEL_PATH` gets a plain message. Settings that name another image's component folder (a CUDA 12 image on a CUDA 13 volume) are made again. Tested on one RTX 3060 PC; no AMD or Intel image and no Docker Desktop.+- **Gyro-S and Gyro-M run on Apple Silicon Macs.** The Metal engine now has kernels for agentionai's trellis experts (TQ2_T, TQK6, TQK7) and the Hadamard rotation they need, in the decode window and in both prompt paths, and reads the published files unchanged. Setup, the app and `strata launch` list Gyro on a Mac next to the other sizes, never as the default, and warn (never refuse) when the Mac's memory is short. Gyro-S is 58.5 GB on disk, of which 28.8 GB is the n-gram table that is read from the SSD; its weights need about 38 GB to stay in memory, so a 48 GB Mac holds it and a 32 GB Mac streams the experts. **Measured on a Mac mini M4 with 32 GB:** 9.1 tok/s decode (story, 120 tokens, 5 interleaved pairs) and 109 tok/s on a 4,065-token prompt. The Gyro-specific dense-weight tiles gave +5 % decode over the generic ones (5 of 5 pairs, same tokens). **Checked:** the decoder equals the format's definition bit for bit on the real tensors, the rotation equals the CPU's bit for bit, and the expert kernels match a double-precision reference to 1e-7 on real experts of Gyro-S and (read by range requests, since it does not fit this Mac) Gyro-M. KL against the CUDA engine: see `docs/MAC.md`. **Not measured:** Gyro-M end to end, a Mac with 48 GB or more (the resident path ran only against a sparse copy of the file), and the experts' streaming copy for Gyro (it needs 26 GB more disk than the test Mac had). It needs the 0.2.2 Mac engine; an older one refuses the file when it loads, and setup says so.