Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docs: Gyro-S on the Mac against the CUDA engine (KL at the CUDA noise level)
Niko1221bc3f293861c5
diff --git a/docs/MAC.md b/docs/MAC.mdindex fb1fa2f67..136bbc160 100644--- a/docs/MAC.md+++ b/docs/MAC.md@@ -205,6 +205,18 @@ kernels reproduce a double-precision reference to 1e-7 relative on real expertsand 47 of Gyro-M (the 92 GB file was read by range requests, 16 experts a layer: it does not fit this Mac's disk). Run it with`metal/tests/gyro_parity.sh`.+**Against the CUDA engine, same file** (the pinned Gyro-S; `tools/quant_quality.py`, teacher-forced over the 5 text segments of+`data/quant_quality/v1`, 4,995 positions; CUDA = an RTX 3060 with the engine of this release, run twice: experts on the GPU, experts on the CPU):++| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity of the text |+|---|---|---|---|---|+| CUDA GPU experts -> CUDA CPU experts (the noise floor: two numerics of one engine) | 95.80 % | 0.0186 | 0.195 | 4.462 / 4.477 |+| CUDA GPU experts -> Mac | 95.46 % | 0.0207 | 0.241 | 4.486 / 4.477 |+| CUDA CPU experts -> Mac | 95.38 % | 0.0187 | 0.272 | 4.486 / 4.462 |++The Mac is as far from either CUDA run as the two CUDA runs are from each other (the CUDA GPU path quantizes the activations to q8_1 and+reads an int8 copy of the codebook, the CPU path an int16 one; the Mac multiplies fp16 codebook values with the float activations).+### Quality- **The Q8_0 hyper-connection read is the default on Mac.** It keeps a Q8_0 copy of the hyper-connection matrices instead of BF16. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read.diff --git a/docs/release-notes-0.2.2.md b/docs/release-notes-0.2.2.mdindex 94b544743..ec5698761 100644--- a/docs/release-notes-0.2.2.md+++ b/docs/release-notes-0.2.2.md@@ -18,4 +18,4 @@- **Docker image build.** The terminal stage no longer needs git in the image (it failed with `FileNotFoundError`); the commit comes in as a build argument.- **The app and the Terminal share one models folder.** Each of the two settings files now remembers when the models folder was chosen; the one changed last wins and the other follows. Nothing is moved. A folder picked in the Terminal was not seen by the app before, so the same model could be downloaded twice.- **Docker.** An old (0.1.x) volume with `/data/models`, `/data/packs` and `/data/mtp` is moved into `/data/Strata-data` by rename instead of being downloaded and packed again (it used to start a 70 GB download). A card without `--gpus` stops at once with the `--gpus all` hint instead of after minutes of setup. A missing or empty `MODEL_PATH` gets a plain message. Settings that name another image's component folder (a CUDA 12 image on a CUDA 13 volume) are made again. Tested on one RTX 3060 PC; no AMD or Intel image and no Docker Desktop.-- **Gyro-S and Gyro-M run on Apple Silicon Macs.** The Metal engine now has kernels for agentionai's trellis experts (TQ2_T, TQK6, TQK7) and the Hadamard rotation they need, in the decode window and in both prompt paths, and reads the published files unchanged. Setup, the app and `strata launch` list Gyro on a Mac next to the other sizes, never as the default, and warn (never refuse) when the Mac's memory is short. Gyro-S is 58.5 GB on disk, of which 28.8 GB is the n-gram table that is read from the SSD; its weights need about 38 GB to stay in memory, so a 48 GB Mac holds it and a 32 GB Mac streams the experts. **Measured on a Mac mini M4 with 32 GB:** 9.1 tok/s decode (story, 120 tokens, 5 interleaved pairs) and 109 tok/s on a 4,065-token prompt. The Gyro-specific dense-weight tiles gave +5 % decode over the generic ones (5 of 5 pairs, same tokens). **Checked:** the decoder equals the format's definition bit for bit on the real tensors, the rotation equals the CPU's bit for bit, and the expert kernels match a double-precision reference to 1e-7 on real experts of Gyro-S and (read by range requests, since it does not fit this Mac) Gyro-M. KL against the CUDA engine: see `docs/MAC.md`. **Not measured:** Gyro-M end to end, a Mac with 48 GB or more (the resident path ran only against a sparse copy of the file), and the experts' streaming copy for Gyro (it needs 26 GB more disk than the test Mac had). It needs the 0.2.2 Mac engine; an older one refuses the file when it loads, and setup says so.+- **Gyro-S and Gyro-M run on Apple Silicon Macs.** The Metal engine now has kernels for agentionai's trellis experts (TQ2_T, TQK6, TQK7) and the Hadamard rotation they need, in the decode window and in both prompt paths, and reads the published files unchanged. Setup, the app and `strata launch` list Gyro on a Mac next to the other sizes, never as the default, and warn (never refuse) when the Mac's memory is short. Gyro-S is 58.5 GB on disk, of which 28.8 GB is the n-gram table that is read from the SSD; its weights need about 38 GB to stay in memory, so a 48 GB Mac holds it and a 32 GB Mac streams the experts. **Measured on a Mac mini M4 with 32 GB:** 9.1 tok/s decode (story, 120 tokens, 5 interleaved pairs) and 109 tok/s on a 4,065-token prompt. The Gyro-specific dense-weight tiles gave +5 % decode over the generic ones (5 of 5 pairs, same tokens). **Checked:** the decoder equals the format's definition bit for bit on the real tensors, the rotation equals the CPU's bit for bit, and the expert kernels match a double-precision reference to 1e-7 on real experts of Gyro-S and (read by range requests, since it does not fit this Mac) Gyro-M. **KL against the CUDA engine** on the same file (4,995 positions of the quant-quality set, mean KL / top-1): Mac against CUDA GPU experts 0.0207 / 95.5 %, Mac against CUDA CPU experts 0.0187 / 95.4 %, and the two CUDA runs against each other 0.0186 / 95.8 %: the Mac sits at the CUDA engine's own noise level. **Not measured:** Gyro-M end to end, a Mac with 48 GB or more (the resident path ran only against a sparse copy of the file), and the experts' streaming copy for Gyro (it needs 26 GB more disk than the test Mac had). It needs the 0.2.2 Mac engine; an older one refuses the file when it loads, and setup says so.