Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Gyro: decode speed on NVIDIA (docs, 0.2.2 notes), Gyro-M's slowdown against Q2_0 16-21%
Niko1221aae92ceb264f
diff --git a/README.md b/README.mdindex d0afcab5a..9b3b8c7a6 100644--- a/README.md+++ b/README.md@@ -157,7 +157,7 @@ Smaller sizes are faster. Larger sizes are a bit smarter.| Size | Memory it needs (RAM + VRAM) | Download | What it is || --- | ---: | ---: | --- || **Gyro-S** | 27.6 GB | 58.5 GB | For PCs that fit nothing else. Quality is lower than IQ2_XS. |-| **Gyro-M** | 35 GB | 92.0 GB | About IQ3_XXS quality with about 20% less GPU memory. It writes answers 17-21% slower than Q2_0 on a 12 GB card and on an R9700. |+| **Gyro-M** | 35 GB | 92.0 GB | About IQ3_XXS quality with about 20% less GPU memory. It writes answers 16% (12 GB RTX 3060) to 21% (R9700) slower than Q2_0. |Gyro runs on NVIDIA and AMD cards (AMD is experimental) on Intel Arc cards on Linux (tested on an Arc A750) and, from 0.2.2, on Apple Silicon Macs (Gyro-S on a 32 GB Mac mini: 9 tok/s, streaming from the SSD; [docs/MAC.md](docs/MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts)). Details and measurements:[docs/GYRO.md](docs/GYRO.md). Start it with `strata launch --model gyro-s`.@@ -167,7 +167,7 @@ Gyro runs on NVIDIA and AMD cards (AMD is experimental) on Intel Arc cards on Li| Your memory (RAM + VRAM) | Take | Notes || --- | --- | --- || **32 GB** | **Gyro-S**, or the **[Coder](docs/MODELS.md#coder)** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code |-| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39.2 GB), or **Gyro-M** (35 GB) | Swift thinks for a shorter time before it answers. Gyro-M is close to IQ3_XXS in quality, but it writes answers 17-21% slower than Q2_0 |+| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39.2 GB), or **Gyro-M** (35 GB) | Swift thinks for a shorter time before it answers. Gyro-M is close to IQ3_XXS in quality, but it writes answers 16-21% slower than Q2_0 || **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better || **64 GB** | **IQ3_S** (54.8 GB), the original or Swift 1.5 | The slowest, the best: it matches the full model on the published tests || **96 GB or more** | Unsloth's **UD-IQ4_XS** (about 72 GB of RAM) to **Q8_0** (about 136 GB), 4 to 8 bit | Closer to the full model. With less RAM, Strata reads part of them from the SSD, which is much slower |diff --git a/docs/GYRO.md b/docs/GYRO.mdindex c7b3610d8..453bc2a85 100644--- a/docs/GYRO.md+++ b/docs/GYRO.md@@ -12,14 +12,14 @@ is anybody's default.| **Download** (the n-gram table is inside the file) | **58.5 GB** | **92.0 GB** || **Memory it needs** (VRAM + RAM together) | **27.6 GB** | **35 GB** || Experts (trellis) | TQK6 gate/up, TQK7 down | TQ2_T |-| In plain words | the smallest option, for PCs with little memory; lower quality than the 2-3-bit sizes | about IQ3_XXS quality with about 20% less GPU memory; decodes about 20% slower than Q2_0 on 12 GB cards |+| In plain words | the smallest option, for PCs with little memory; lower quality than the 2-3-bit sizes | about IQ3_XXS quality with about 20% less GPU memory; decodes about 16% slower than Q2_0 on a 12 GB RTX 3060 (23% slower before engine 0.2.2) |About half of each file is the n-gram (PLE) table: it stays on the SSD and is read row by row for every token, so keepthe file on a fast local drive. The memory figure is what the weights take once loaded.Not defaults, by measurement (our evaluation, 2026-10): against Q2_0 at the same GPU bytes Gyro-M halves the KLdivergence (0.159 against 0.312 on 128 chunks) and is level with IQ3_XXS (0.157), but decodes 17-21% slower on a 12 GB-card (RTX 3060) and an R9700, and its reasoning set scored lower than Q2_0's (49/60 against 57/60). Gyro-S is 13% worse+card (RTX 3060, measured before engine 0.2.2: 16% since) and an R9700, and its reasoning set scored lower than Q2_0's (49/60 against 57/60). Gyro-S is 13% worsein KL than IQ2_XS (0.268 against 0.236) with 24% fewer bytes. So: pick Gyro-S where nothing bigger fits, Gyro-M whenthe 20% memory saving over IQ3_XXS matters. Setup and the MCP server mention them only where the other sizes do notfit the PC; they warn, never refuse. Gyro-L (agentionai's 50 GiB file, 2026-10-12) is not supported: it uses another@@ -118,6 +118,31 @@ short of one cache slot and the engine stops with that sentence. It starts withput `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used about 24 GB of RAM for theexpert arena. Keep the file on a fast SSD.+## Decode speed on NVIDIA (engine 0.2.2)++The trellis decode kernels read the codebook (2048 two-byte entries, 8 random lookups for every 16 weights) from the global+table. On an RTX 3060 that made the gate/up and down kernels wait on the L1 cache, not on memory: 18 L1 sectors for every+lookup request, the card's L1 at 64% busy while the DRAM was 27% busy. The kernels now keep the 4 KB codebook in shared memory+(`STRATA_IQ_STAGE_GRID=0` reads it in place again) and take four rows per warp, 32 rows per thread block, so the copy is shared+and no lane idles in the 160-dot gate/up rows or the 40-dot down rows. The sums are added in the same order as before, so the+answers are bit for bit the old ones: the identity gate (10 prompts, 4096 and 20000 token prompts) is identical on Gyro-S,+Gyro-M, Q2_0 and IQ3_XXS, and the Q2_0 and IQ3_XXS kernels did not change.++RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), 5 interleaved pairs of whole engine+starts, medians of the story, code and facts prompts (200 tokens each), engine 0.2.1 against 0.2.2. Gyro files of the first+release; the pinned revision is checked in the next block.++| | decode before, tok/s | after | gain | pairs faster | 4K / 20K prompt |+| --- | ---: | ---: | ---: | --- | --- |+| Gyro-S | 53.6 | 62.8 | +17.2% | 4 of 5 (8-9 of 10 requests per prompt) | unchanged (1.008, 1.000) |+| Gyro-M | 50.6 | 56.1 | +10.9% | 5 of 5 | unchanged (1.013, 0.997) |+| Q2_0 (not changed) | 66.9 | 66.1 | 0.99 (noise) | 2 of 5 | unchanged |++Against Q2_0 on the same card, before: Gyro-S 53.8 against 65.7 tok/s (18% slower), Gyro-M 51.7 against 67.6 (24% slower).+After: Gyro-S about 6% slower, Gyro-M about 16% slower. What is left of Gyro-M's gap is mostly its dense weights: in a+decode profile its Q8_0 matrix-vector kernels take 26% of the GPU time, and they read as many bytes as the card's memory+delivers. The trellis kernels themselves are limited by the shared-memory bank conflicts of the random codebook lookups now. Prompt speed is not changed by this: Gyro prompts stay 8% (Gyro-S) and 11-19% (Gyro-M) slower than Q2_0's.+## Credit and licensesThe trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijkdiff --git a/docs/MODELS.md b/docs/MODELS.mdindex 7662a6d1b..f3b03ba3c 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -19,7 +19,7 @@ other size stays a choice. A PC with a card of 12 GB and 32 GB of RAM is in the| Your memory (RAM + VRAM) | Pre-selected | Notes || --- | --- | --- || **32 GB** | **Gyro-S**, or the **Coder** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code ([why](#coder)). Gyro-S has lower quality than the 2-3-bit sizes, and the 2-bit Q2_0 decodes faster |-| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 17-21% slower than Q2_0. Gyro runs on NVIDIA, AMD and Intel Arc cards and, from 0.2.2, on Macs (Gyro-S measured on a 32 GB Mac mini) |+| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 16-21% slower than Q2_0. Gyro runs on NVIDIA, AMD and Intel Arc cards and, from 0.2.2, on Macs (Gyro-S measured on a 32 GB Mac mini) || **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better || **64 GB** | **IQ3_S** (54.8 GB), the original. Swift 1.5's IQ3_S is an equal choice | The slowest, the best: it matches the full model on the published tests || **96 GB or more** | [Unsloth's UD-IQ4_XS to Q8_0](#unsloths-files-ud-iq4_xs-to-q8_0), 4 to 8 bit | UD-IQ4_XS keeps every expert in RAM from about 80 GB of RAM, Q8_0 from about 136 GB. Up to 192 GB, the 5-8-bit files fit in RAM. Setup still pre-selects IQ3_S here, except on a Strix Halo |@@ -181,7 +181,7 @@ experts in a rotated basis. They are the smallest options and never setup's defa| Size | Download | Memory it needs (VRAM + RAM) | In plain words || --- | ---: | ---: | --- || **Gyro-S** | 58.5 GB | 27.6 GB | the smallest option, for PCs with little memory; lower quality |-| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory; decodes about 20% slower than Q2_0 on 12 GB cards |+| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory; decodes 16-21% slower than Q2_0 (RTX 3060 12 GB, R9700) |Gyro runs on NVIDIA and AMD (experimental) cards and on Intel Arc cards on Linux (tested on an Arc A750; the Arc engine is built from this release's source, see [INTEL.md](INTEL.md)) and, from 0.2.2, on Apple Silicon Macs ([MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts): Gyro-S is measured on a 32 GB Mac mini, Gyro-M only tensor by tensor). Gyro reads pictures (the same rule as above). On NVIDIA and AMD it needsengine 0.1.45.1 or newer; if the engine is older, the app says "This model needs a newer engine" and offers Updatediff --git a/docs/release-notes-0.2.2.md b/docs/release-notes-0.2.2.mdindex ec5698761..2d416d474 100644--- a/docs/release-notes-0.2.2.md+++ b/docs/release-notes-0.2.2.md@@ -3,6 +3,7 @@## What is new- **Gyro runs on Intel Arc (Linux, tested on an Arc A750 only).** The SYCL engine has the trellis kernels (TQ2_T, TQK6, TQK7) and the Hadamard rotation, so Gyro-S and Gyro-M load and answer on an Arc card. Setup lists them like for any card, warns and never refuses when VRAM and RAM together are short, and keeps Gyro's n-gram table on the SSD. On the A750 (8 GB, Ryzen 5 5600X, 62 GB of RAM) Gyro-S reads a 1,400-token prompt at 82 tok/s (40 before the matrix-unit launch took the trellis types), a 20,000-token prompt at 195 tok/s, and writes 15-16 tok/s with the MTP draft. The card holds 410 of 24,576 experts, so the CPU and the PCIe link carry the decode; for these packs the engine gives the link a larger share (0.48 instead of 0.29, +16% decode). On real tensors of both files the kernels equal a host replay of their own arithmetic in every bit, and 7 of 10 test prompts give the same 48 tokens as the CUDA engine (the other three agree for 6 to 9 tokens and then differ at a near-tie; first-token KL 1e-4 to 1e-2). Gyro-M needs `--max-context 8192 --prefill 1024` on an 8 GB card. Not tested: B-series cards and two Arc cards (the second card is refused with a message). The CUDA comparison used the first-release Gyro-S file; the pinned file passes the kernel checks and answers correctly. Gyro still does not run on the Mac. Details: [GYRO.md](GYRO.md#intel-arc-sycl).+- **Gyro decodes faster on NVIDIA cards.** The gate/up and down kernels for agentionai's trellis experts now keep the codebook in shared memory and take four rows per warp. On an RTX 3060 (12 GB) Gyro-S went from 53.6 to 62.8 tokens per second (+17%) and Gyro-M from 50.6 to 56.1 (+11%), 5 interleaved pairs each; prompts are unchanged. The answers are bit for bit the old ones (identity gate on Gyro-S, Gyro-M, Q2_0 and IQ3_XXS), and Q2_0 and IQ3_XXS are not touched. Gyro-S is now about 6% and Gyro-M about 16% behind Q2_0 on that card (18% and 24% before); `STRATA_IQ_STAGE_GRID=0` goes back to the old lookup. Cards of other generations were not measured.- **An engine package for the RX 6000 series (built, not tested on a real RX 6000 card).** Linux has `rx6000` and Windows has `rdna2`, each with GPU code for gfx1030 and gfx1031 (RX 6800, 6900 and 6950 series, RX 6700 XT). They are offered to a PC whose card reports one of those two and to no other: the RX 7000, RX 9000 and Strix Halo packages are unchanged, and a PC with an RX 7000 or 9000 card next to an RX 6000 card keeps the package it had. The engine runs plain kernels on these cards (RDNA2 has no matrix cores, so none of the RX 7000 / RX 9000 or Radeon 780M prompt paths apply) and plain hipBLAS (hipBLASLt ships no kernels for gfx1030, so there is no tuning table). Setup and the app say once that this package was built and not tested on a real RX 6000 card. What was checked: the Linux package builds, every program in it has code for exactly gfx1030 and gfx1031, the rocBLAS kernel files for both are in it, every library it imports is in the package or belongs to the system, and the whole engine (all targets, tests included) compiles for gfx1030 and gfx1031. What was not: a model on an RX 6000 card, on either system, and the Windows package itself (the Windows build scripts take the new architecture list, but no Windows `rdna2` package has been built). Earlier reports on these cards came from people who built the engine from source ([docs/AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030)). If something goes wrong, open an issue with the end of `strata-<model>.log`.- **The CPU expert pool splits per NUMA node on a machine with two or more NUMA nodes (built from the report in #3201, not tested on a two-socket machine).** Until now the pool kept its workers on the host thread's socket, because workers on the other socket read the experts across the link and were slower. The split gives each socket its own slice of the pool with its own task counter, and places each socket's half of every CPU-side expert in that socket's memory, so a slice reads local memory only. The reporter measured +10 to +14 % decode on a dual Xeon Gold 6226 (59.3 against 54.05 tok/s with the IOMMU in passthrough); a second reporter's dual Xeon E5-2670 v2 preferred more workers across both sockets, which the split also uses. It starts by itself when the machine has two or more NUMA nodes with CPUs and runs only when it can (Linux, two nodes, pinned threads, enough free memory per node); otherwise the pool is exactly what it was and the log says why. **A machine with one NUMA node runs the same code path as 0.2.1:** the identity gate (10 prompts, exact_pp 4K and 20K) is byte-identical and the single-request speed A/B is not slower on the one-node test boxes (numbers in the notes of the branch). `STRATA_NUMA_POOL=0` turns the split off; `STRATA_NUMA_SPLIT=1` still asks for it on any machine. The start-up log says what it did: `NUMA pool on: 2 NUMA nodes found ...` or `NUMA pool off: <reason>`. On Windows the node count is read but the split itself is Linux-only for now, so a two-node Windows PC logs the reason and runs as before. The logic runs in unit tests on made-up two-, three- and four-node machines; we have no two-socket machine, so the placement and the speed on real ones are the reporters' numbers, not ours. Do not wrap the server in `numactl --cpunodebind` or `--membind`: it pulls both halves onto one node. Thanks to baptisterajaut for the branch this is built on, and to xrip and Tworks25 for their numbers. Please send the start-up lines and a decode run with and without `STRATA_NUMA_POOL=0` from a two-socket machine.- **The API key is optional.** The server, `strata launch`, the Terminal and the app no longer refuse to share without a key. Say so (`--no-api-key`, or `"api_key": "none"` in the server config) and it starts, with one plain warning that anyone who can reach the port can use the model and its tools; a public Cloudflare address gets a stronger one. The Terminal and the app still offer to make a key and do by default. In the app, Remove key now also works while local sharing or the tunnel runs. Fixes the refusals reported in #3198 and #3401.diff --git a/setup.py b/setup.pyindex 90850afe6..a5527ebc9 100644--- a/setup.py+++ b/setup.py@@ -462,7 +462,7 @@ MODELS = {"download_gb": 58.5, "ram_gb": 27.6, "arena_gb": 25.8, "families": ("gyro",), "mem_total": True,"file": "Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf", "shards": 1, "engine": GYRO_ENGINE, "vision": True},"Gyro-M": {"about": "about IQ3_XXS quality with about 20% less GPU memory (35 GB of VRAM + RAM together); decodes "- "about 20% slower than Q2_0 on 12 GB cards",+ "about 16-21% slower than Q2_0 on 12 GB cards","download_gb": 92.0, "ram_gb": 35.0, "arena_gb": 32.1, "families": ("gyro",), "mem_total": True,"file": "Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf", "shards": 1, "engine": GYRO_ENGINE, "vision": True},"NVFP4": {"about": "best quality of the NVFP4 versions (GPTQ, sergqwer's strata-nvfp4 pack); RTX 50 has FP4 tensor cores, "diff --git a/tools/strata_mcp.py b/tools/strata_mcp.pyindex eab649158..424bb94e5 100644--- a/tools/strata_mcp.py+++ b/tools/strata_mcp.py@@ -83,7 +83,7 @@ FALLBACK_MODELS = {"download_gb": 58.5, "ram_gb": 27.6, "arena_gb": 25.8, "families": ("gyro",), "mem_total": True,"vision": True},"Gyro-M": {"about": "about IQ3_XXS quality with about 20% less GPU memory (35 GB of VRAM + RAM together); decodes "- "about 20% slower than Q2_0 on 12 GB cards",+ "about 16-21% slower than Q2_0 on 12 GB cards","engine": (0, 1, 45, 1),"download_gb": 92.0, "ram_gb": 35.0, "arena_gb": 32.1, "families": ("gyro",), "mem_total": True,"vision": True},