Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs/GYRO.md: setup is the install path, our measurements, credit and licenses; docs/MODELS.md lists Gyro-S and Gyro-M; tools/gyro_setup.sh (rc2's script) is replaced by setup
Niko1221eaf874569753
diff --git a/docs/GYRO.md b/docs/GYRO.mdindex a12c2005e..ca8d890b0 100644--- a/docs/GYRO.md+++ b/docs/GYRO.md@@ -1,46 +1,72 @@# Gyro (agentionai rotor quants) on Strata-Validated on Linux with an RTX 5090 (32 GB, CUDA 13.0, sm_120; 2026-10-02 and 2026-10-03), an RTX 3090 (24 GB,-sm_86; 2026-10-03) and two RTX 4090s (24 GB each, sm_89; 2026-10-04, single and dual GPU). A short end-to-end run on AMD (Strix Halo, gfx1151, ROCm) passed on 2026-10-03.+Gyro-S and Gyro-M are Qwen3.8-Flash-Next quantizations by agentionai+([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)): the routed+experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata reads the+published files unchanged. **Setup lists them as a model family of their own** ("Qwen3.8-Flash-Next (Gyro)"); neither+is anybody's default.++| | Gyro-S | Gyro-M |+| --- | --- | --- |+| File | `Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf` | `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` |+| **Download** (the n-gram table is inside the file) | **58.5 GB** | **92.0 GB** |+| **Memory it needs** (VRAM + RAM together) | **27.6 GB** | **35 GB** |+| Experts (trellis) | TQK6 gate/up, TQK7 down | TQ2_T |+| In plain words | the smallest option, for PCs with little memory; lower quality than the 2-3-bit sizes | about IQ3_XXS quality with about 20% less GPU memory; decodes about 20% slower than Q2_0 on 12 GB cards |++About half of each file is the n-gram (PLE) table: it stays on the SSD and is read row by row for every token, so keep+the file on a fast local drive. The memory figure is what the weights take once loaded.++Not defaults, by measurement (our evaluation, 2026-10): against Q2_0 at the same GPU bytes Gyro-M halves the KL+divergence (0.159 against 0.312 on 128 chunks) and is level with IQ3_XXS (0.157), but decodes 17-21% slower on a 12 GB+card (RTX 3060) and an R9700, and its reasoning set scored lower than Q2_0's (49/60 against 57/60). Gyro-S is 13% worse+in KL than IQ2_XS (0.268 against 0.236) with 24% fewer bytes. So: pick Gyro-S where nothing bigger fits, Gyro-M when+the 20% memory saving over IQ3_XXS matters. Setup and the MCP server mention them only where the other sizes do not+fit the PC; they warn, never refuse. Gyro-L (agentionai's 50 GiB file, 2026-10-12) is not supported: it uses another+trellis family (types 147-149) that the engine does not have.++## Install-The targets are the published files of `agentionai/Qwen3.8-Flash-Next-Gyro-GGUF`, used unchanged:--| File | Experts | GPU memory for the experts | Status |-|---|---|---:|---|-| `Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf` (58.5 GB) | TQK6 gate/up, TQK7 down, block-128 Hadamard | 24.0 GiB | validated end to end (RTX 5090, RTX 3090) |-| `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` (92.0 GB) | TQ2_T, block-128 Hadamard | 29.9 GiB | validated end to end (RTX 5090, RTX 3090); the experts do not all fit a 32 GB card, `auto` streams the rest |--Both files carry their own experts in the trellis types TQ2_T/TQK6/TQK7 and record the activation rotation in-`prism.hadamard.*` metadata. Strata reads both natively. About half of each file is the-n-gram (PLE) table, which stays on disk and is read row by row; it is not loaded into RAM or VRAM.+```sh+./setup.sh --setup --family gyro --model Gyro-S # or Gyro-M; Windows: START-HERE.bat --setup ...+```-The draft head is Strata's own MTP runtime, prepared from the original model's MTP weights as for the other-Qwen3.8-Flash-Next models. It is not the GGUF draft file from the Gyro repository.+Setup downloads the pinned revision of the file (size and SHA-256 checked), prepares the pack (`tools/iq_pack.py+--compat-bf16`: the Hadamard table and the dense weights, 1.5 GB), the draft head (Strata's own MTP head, prepared from+the original model as for every Qwen3.8-Flash-Next file) and the start script. Needs engine 0.1.46 or newer; a+ready-made engine that is older is replaced by one compiled from this source.-## Quick setup+**Supported:** NVIDIA (CUDA, Linux and Windows) and AMD (HIP, experimental: the authors validated gfx1151; we ran+gfx1201 with identical output to rc2). **Not supported:** Intel Arc (SYCL) and Apple Metal: the engines there have no+trellis kernels; the Metal engine refuses the file at load with the reason. Images are off for these files.-Gyro support lives on the `rc2` branch of the agentionai Strata fork: Strata v0.1.40.3 plus the trellis types-(TQ2_T / TQK6 / TQK7), the Hadamard rotation and the CPU trellis kernels.+**Prompt speed and the ggml build.** The trellis prompt kernels (MMQ) come from agentionai's llama.cpp fork+(MIT, `STRATA_GGML_DIR`). A build against upstream ggml works and gives the same answers, but reads prompts through the+FP16 path, which is slower. For the fast prompt path compile with the fork's ggml:```sh-git clone -b rc2 https://github.com/agentionai/Strata-cd Strata+git clone https://github.com/agentionai/llama.cpp ../agention-llama.cpp+cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \+ -DCMAKE_CUDA_ARCHITECTURES=86 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp+# AMD: -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_GGML_DIR=...```-`tools/gyro_setup.sh` does every step below on Linux:+## Credit and licenses-```sh-tools/gyro_setup.sh --model S --backend cuda --cuda-arch 120 # RTX 50 series-tools/gyro_setup.sh --model S --backend cuda --cuda-arch 89 # RTX 40 series-tools/gyro_setup.sh --model S --backend cuda --cuda-arch 86 # RTX 30 series / RTX A6000-tools/gyro_setup.sh --model S --backend hip --hip-arch gfx1201 # Radeon RX 9070 / AI PRO R9700 (experimental)-```+The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk+(agentionai) in agentionai/Strata `rc2` (9ff02c9, MIT, Strata v0.1.40.3 plus Gyro) and ported here commit by commit with+his authorship kept; `third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the+ggml authors). The weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0+(<https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF/blob/main/LICENSE>).++## How it runs (by hand)-It prints the command that starts the server. The steps, by hand:+The rest of this page is the authors' own guide for a manual install and their measurements on their machines (RTX 5090,+3090, 4090 and a Strix Halo, with Strata rc2). The numbers are theirs; ours are in the next section.## 1. Build-From the `rc2` checkout above. Strata's GGML dependency must be the agentionai llama.cpp fork, which has the trellis types:+Setup does these steps for you; by hand, from this checkout. For the fast prompt path Strata's GGML dependency is the agentionai llama.cpp fork, which has the trellis types:```shgit clone --depth 1 https://github.com/agentionai/llama.cpp ../agention-llama.cpp@@ -61,7 +87,7 @@ python3 -m venv .venv && .venv/bin/pip install -r requirements.txt.venv/bin/pip install huggingface_hub hf_transferHF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next-Gyro-GGUF \Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf --local-dir models-STRATA_GGUF_PY=../agention-llama.cpp/gguf-py .venv/bin/python tools/iq_pack.py \+.venv/bin/python tools/iq_pack.py \--gguf models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf --out packs/gyro-s --compat-bf16build/native_expert_parity --load-check packs/gyro-s models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf```@@ -111,7 +137,34 @@ The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0128k context it holds 23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the restfrom RAM; nothing else changes.-## Measured+## Measured here (the port, 2026-10-15/16)++One AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, exclusive box lock, `--prefill auto --spec 4 --spec-min-p 0.5+--mtp --kv int8`, 22K context, interleaved pairs (the arm that goes first alternates), medians, the files of the first+Gyro release (2026-10-07). A = agentionai/Strata rc2 9ff02c9, B = this port.++| B, ggml of | file | decode tok/s A / B (B/A) | pairs B faster | prompt 4K ms A / B | prompt 20K ms A / B |+| --- | --- | ---: | --- | ---: | ---: |+| agentionai fork | Gyro-S | 85.4 / 89.7 (1.051) | 5/5 | 5418 / 5313 (0.98) | 14358 / 13859 (0.97) |+| agentionai fork | Gyro-M | 72.7 / 82.7 (1.136) | 5/5 | 5183 / 5520 (1.07) | 13421 / 14164 (1.06) |+| upstream | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5425 / 6939 (1.28) | 14364 / 21677 (1.51) |++Decode is faster than rc2's because the port sits on the newer Strata (0.1.45); the same engine change makes IQ3_XXS 7% faster+(rc2 86.9, port 93.1 tok/s, 2 pairs). Prompts of the ISTA files gained 11-16% from it too (IQ3_XXS 4K 6184 to 5480 ms, 20K+16587 to 13864 ms); Gyro-M's prompts did not and are 5-7% slower than rc2's, Gyro-S's are 2-3% faster. The cause for Gyro-M is+not isolated (`STRATA_GFX12_DEFAULTS=0` does not give it back). With **upstream ggml** (the ready-made engine) the trellis+prompt kernels are missing and prompts go through the FP16 path: 28% (4K) to 51% (20K) slower than rc2, and a different+rounding (the answer after a 4K prompt can differ). Decode is unaffected. That is why a source build against the fork's ggml is+the fast configuration.++Identity (the fixed gate flags, 10 prompts, exact_pp 4096 and 20000 tokens, same card for both arms):++- Q2_0 and IQ3_XXS, f-split against the port (upstream ggml): 10/10 identical, residual dumps and outputs IDENTICAL.+- Gyro-S and Gyro-M, rc2 against the port (fork ggml): 10/10 identical; exact_pp 4096 / 20000 residual dumps and outputs+ IDENTICAL when run alone (a run beside other engines on the box once showed a different dump: not reproducible alone).+- The pack step (`tools/iq_pack.py --compat-bf16`) writes byte-identical files to rc2's tools for Gyro-S.++## Measured (the authors', rc2)All runs: greedy or the stated sampling, same prompt tokens as llama.cpp, MTP draft `--spec 4 --spec-min-p 0.5`,int8 KV. "llama.cpp" is the agentionai fork with the CUDA trellis fixes (in its `main` since 2026-10-04).@@ -124,7 +177,7 @@ Decode tokens/s, single prompts of 34-177 tokens, 384 new tokens, `--max-context|---|---:|---:|---:|| Strata | 139.9 | 221.3 | 209.0 |-Server, `tools/gyro_setup.sh` config (128k context, 23,463 experts cached), thinking off, temperature 0.7,+Server, rc2's `gyro_setup.sh` config (128k context, 23,463 experts cached), thinking off, temperature 0.7,top-k 20, top-p 0.95, min-p 0.05, 512 new tokens, 3 repetitions:| | prose | JSON | code | copy |@@ -200,6 +253,12 @@ Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, J## Known limits+- **Not in this port** (rc2 has them; see the list at the end of this page): the prompt helper on a second GPU for a Gyro+ pack. Setup of two cards still runs the layer split and the decode peer; the prompt peer says it is not used for a+ Hadamard-folded pack and the prompt stays on the first card. Gyro-L, Intel Arc (SYCL) and Apple Metal are not supported.+- **Source builds:** the SYCL tree builds with these sources (P1006, 0 errors) but has no trellis GPU kernels; the Metal+ engine builds and refuses the file at load (`native_expert_supported`).+- **Not measured yet:** token parity against llama.cpp.- **Benchmark mode only:** `strata --tokens-file ... --max-new N` decodes N tokens past the end of the answer;compare speeds only up to the answer's end.diff --git a/tools/gyro_setup.sh b/tools/gyro_setup.shdeleted file mode 100755index de1e71132..000000000--- a/tools/gyro_setup.sh+++ /dev/null@@ -1,85 +0,0 @@-#!/usr/bin/env bash-# Set up agentionai Gyro (Qwen3.8-Flash-Next rotor quants) on Strata: build, download, pack, draft head, config.-# Linux only. See docs/GYRO.md for what each step does.-# tools/gyro_setup.sh --model S|M --backend cuda|hip [--cuda-arch 120] [--hip-arch gfx1201]-# [--models DIR] [--llama DIR] [--ctx 131072] [--port 8080]-set -euo pipefail-cd "$(dirname "$0")/.."-MODEL=S BACKEND=cuda CUDA_ARCH=120 HIP_ARCH=gfx1201 MODELS=models LLAMA=../agention-llama.cpp CTX=131072 PORT=8080-while [ $# -gt 0 ]; do case "$1" in- --model) MODEL=$2; shift 2;; --backend) BACKEND=$2; shift 2;; --cuda-arch) CUDA_ARCH=$2; shift 2;;- --hip-arch) HIP_ARCH=$2; shift 2;; --models) MODELS=$2; shift 2;; --llama) LLAMA=$2; shift 2;;- --ctx) CTX=$2; shift 2;; --port) PORT=$2; shift 2;; -h|--help) sed -n 2,5p "$0"; exit 0;;- *) echo "unknown option $1"; exit 2;; esac; done-case "$MODEL" in S) F=Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf; P=gyro-s;; M) F=Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf; P=gyro-m;;- *) echo "--model must be S or M"; exit 2;; esac-step() { printf '\n== %s\n' "$*"; }-for t in git python3; do command -v $t >/dev/null || { echo "missing: $t"; exit 1; }; done--step "python environment (also provides cmake >= 3.24 and ninja)"-[ -x .venv/bin/python ] || python3 -m venv .venv-.venv/bin/pip install -q -r requirements.txt huggingface_hub hf_transfer-export PATH="$PWD/.venv/bin:$PATH"--step "agentionai llama.cpp fork (GGML with the trellis types) -> $LLAMA"-[ -d "$LLAMA/ggml" ] || git clone --depth 1 https://github.com/agentionai/llama.cpp "$LLAMA"-LLAMA=$(cd "$LLAMA" && pwd)-export STRATA_GGUF_PY="$LLAMA/gguf-py" # iq_pack.py and mtp_pack.py read GGUF through it--step "build Strata ($BACKEND)"-if [ "$BACKEND" = cuda ]; then- NV=$(nvcc --version 2>/dev/null | sed -n 's/.*release \([0-9]*\)\..*/\1/p' || true)- if [ "${CUDA_ARCH}" -ge 120 ] && [ "${NV:-0}" -lt 13 ]; then- echo "warning: RTX 50 (sm_${CUDA_ARCH}) with CUDA ${NV:-?}: use CUDA 13 (12.8 crashes the prompt path, issues #220/#224)"; fi- cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \- -DCMAKE_CUDA_ARCHITECTURES="$CUDA_ARCH" -DSTRATA_GGML_DIR="$LLAMA"-elif [ "$BACKEND" = hip ]; then- echo "note: AMD is experimental for Gyro (kernel parity and an end-to-end run validated on gfx1151 only)"- cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \- -DCMAKE_HIP_ARCHITECTURES="$HIP_ARCH" -DSTRATA_GGML_DIR="$LLAMA" \- -DCMAKE_PREFIX_PATH="${ROCM_PATH:-/opt/rocm}"-else echo "--backend must be cuda or hip"; exit 2; fi-cmake --build build -j--step "download $F -> $MODELS (keep it on a fast local SSD: the n-gram table is read per token)"-mkdir -p "$MODELS"-if [ -s "$MODELS/$F" ]; then echo "already there: $MODELS/$F (delete it to download again)"-else HF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next-Gyro-GGUF "$F" --local-dir "$MODELS"; fi-G=$(cd "$MODELS" && pwd)/$F--step "pack -> packs/$P"-.venv/bin/python tools/iq_pack.py --gguf "$G" --out "packs/$P" --compat-bf16-LC=$(build/native_expert_parity --load-check "packs/$P" "$G" 2>&1) || true; echo "$LC"-echo "$LC" | grep -q " 0 failures" || { echo "load check failed: see above"; exit 1; }--step "draft head (Strata's MTP runtime)"-if [ ! -f mtp/rt/draft_vocab.bin ]; then- .venv/bin/python tools/mtp_fetch.py fetch --out mtp- .venv/bin/python tools/mtp_pack.py --src mtp --experts q2_0 --out mtp/mtp-q2_0.gguf- .venv/bin/python tools/mtp_rt.py --gguf mtp/mtp-q2_0.gguf --out mtp/rt- cp data/draft_vocab.bin mtp/rt/draft_vocab.bin-fi--step "config strata-$P.json"-cat > "strata-$P.json" <<JSON-{- "exe": "build/strata",- "args": [- "--pack", "packs/$P",- "--native", "$G",- "--expert-profile", "data/expert-profile.bin", "--expert-cache", "auto",- "--prefill", "auto", "--spec", "4", "--spec-min-p", "0.5",- "--mtp", "mtp/rt", "--max-context", "$CTX", "--kv", "int8"- ],- "cwd": ".",- "tokenizer": "packs/$P/tokenizer",- "model_name": "$P-strata",- "log": "strata-$P.log",- "host": "127.0.0.1",- "port": $PORT-}-JSON--step "done. Start the server with:"-echo " .venv/bin/python -m serve.server --engine strata --config strata-$P.json --port $PORT"-echo " web: http://127.0.0.1:$PORT API: http://127.0.0.1:$PORT/v1"