Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Merge f-022-pascaltq: Gyro trellis kernels about 30% faster on Pascal (sm_60), other cards unchanged

Niko122188942998c54f
diff --cc docs/release-notes-0.2.2.md
index ec5698761,fb5fce8fd..33b5a89ab
--- a/docs/release-notes-0.2.2.md
+++ b/docs/release-notes-0.2.2.md
@@@ -18,4 -17,4 +18,5 @@@
- **Docker image build.** The terminal stage no longer needs git in the image (it failed with `FileNotFoundError`); the commit comes in as a build argument.
- **The app and the Terminal share one models folder.** Each of the two settings files now remembers when the models folder was chosen; the one changed last wins and the other follows. Nothing is moved. A folder picked in the Terminal was not seen by the app before, so the same model could be downloaded twice.
- **Docker.** An old (0.1.x) volume with `/data/models`, `/data/packs` and `/data/mtp` is moved into `/data/Strata-data` by rename instead of being downloaded and packed again (it used to start a 70 GB download). A card without `--gpus` stops at once with the `--gpus all` hint instead of after minutes of setup. A missing or empty `MODEL_PATH` gets a plain message. Settings that name another image's component folder (a CUDA 12 image on a CUDA 13 volume) are made again. Tested on one RTX 3060 PC; no AMD or Intel image and no Docker Desktop.
+- **Gyro-S and Gyro-M run on Apple Silicon Macs.** The Metal engine now has kernels for agentionai's trellis experts (TQ2_T, TQK6, TQK7) and the Hadamard rotation they need, in the decode window and in both prompt paths, and reads the published files unchanged. Setup, the app and `strata launch` list Gyro on a Mac next to the other sizes, never as the default, and warn (never refuse) when the Mac's memory is short. Gyro-S is 58.5 GB on disk, of which 28.8 GB is the n-gram table that is read from the SSD; its weights need about 38 GB to stay in memory, so a 48 GB Mac holds it and a 32 GB Mac streams the experts. **Measured on a Mac mini M4 with 32 GB:** 9.1 tok/s decode (story, 120 tokens, 5 interleaved pairs) and 109 tok/s on a 4,065-token prompt. The Gyro-specific dense-weight tiles gave +5 % decode over the generic ones (5 of 5 pairs, same tokens). **Checked:** the decoder equals the format's definition bit for bit on the real tensors, the rotation equals the CPU's bit for bit, and the expert kernels match a double-precision reference to 1e-7 on real experts of Gyro-S and (read by range requests, since it does not fit this Mac) Gyro-M. **KL against the CUDA engine** on the same file (4,995 positions of the quant-quality set, mean KL / top-1): Mac against CUDA GPU experts 0.0207 / 95.5 %, Mac against CUDA CPU experts 0.0187 / 95.4 %, and the two CUDA runs against each other 0.0186 / 95.8 %: the Mac sits at the CUDA engine's own noise level. **Not measured:** Gyro-M end to end, a Mac with 48 GB or more (the resident path ran only against a sparse copy of the file), and the experts' streaming copy for Gyro (it needs 26 GB more disk than the test Mac had). It needs the 0.2.2 Mac engine; an older one refuses the file when it loads, and setup says so.
+ - **Gyro decodes about 30 % faster on a Tesla P100 (source build for compute capability 6.0 only; the ready-made CUDA 13 packages have no code for that card).** On a P100, Gyro-S ran at about 70 % of the speed of the Q2_0 file of similar size: 30 / 36 / 33 tok/s against 41 / 51 / 48 (story / code / facts, median of 5 interleaved pairs, default flags). On compute capability 6.0 the engine now keeps the 4 KB trellis codebook in shared memory, reads the down rows as 8-lane rows, lets each block work on several row chunks, and copies the activations of an expert group to shared memory once per block instead of reading them from global memory for every row. Gyro-S now decodes 38.5 / 45.6 / 42.7 tok/s (+30 %; every one of 10 reps and 5 of 5 pairs faster), 0.96 / 0.90 / 0.91 of Q2_0. Prompt speed is unchanged: 4096 tokens take 14.0 s and 20000 tokens 62 s, 3 to 5 % slower than Q2_0. Answers are the same: the identity gate (10 prompts, 4096- and 20000-token prompt dumps) is byte-identical on the P100 and on an RTX 3060, and Q2_0 on the P100 did not change (0.999 / 0.998 / 1.001). Other cards run the old kernels, because the new path is picked by compute capability 6.0 at run time; `STRATA_TQ_STAGE=0` turns it off. Two attempts to speed up the prompt (a faster dp4a in llama.cpp's MMQ kernels, which this card lacks, and a register-tiled MMQ dot) were slower and are not included.