Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Gyro: setup guide and script for the published agentionai Gyro-S/Gyro-M files
Laurent Zuijdwijk0ca3752ea6df
diff --git a/docs/GYRO.md b/docs/GYRO.mdnew file mode 100644index 000000000..5d7a42551--- /dev/null+++ b/docs/GYRO.md@@ -0,0 +1,128 @@+# Gyro (agentionai rotor quants) on Strata++Validated on Linux with an RTX 5090 (32 GB), 16 CPU cores and 30 GB RAM on 2026-10-02 (CUDA 13.0, sm_120).++The targets are the published files of `agentionai/Qwen3.8-Flash-Next-Gyro-GGUF`, used unchanged:++| File | Experts | GPU memory for the experts | Status |+|---|---|---:|---|+| `Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf` (54.5 GB) | TQK6 gate/up, TQK7 down, block-128 Hadamard | 24.0 GiB | validated end to end |+| `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` (92.0 GB) | TQ2_T, block-128 Hadamard | 29.9 GiB | packs and passes `--load-check`; not run end to end |++Both files carry their own experts in the trellis types TQ2_T/TQK6/TQK7 and record the activation rotation in+`prism.hadamard.*` metadata. Strata reads both natively (set `STRATA_APR=1`). About half of each file is the+n-gram (PLE) table, which stays on disk and is read row by row; it is not loaded into RAM or VRAM.++The draft head is Strata's own MTP runtime, prepared from the original model's MTP weights as for the other+Qwen3.8-Flash-Next models. It is not the GGUF draft file from the Gyro repository.++## Quick setup++`tools/gyro_setup.sh` does every step below on Linux:++```sh+tools/gyro_setup.sh --model S --backend cuda --cuda-arch 120 # RTX 50 series+tools/gyro_setup.sh --model S --backend cuda --cuda-arch 89 # RTX 40 series+tools/gyro_setup.sh --model S --backend hip --hip-arch gfx1201 # Radeon RX 9070 / AI PRO R9700 (experimental)+```++It prints the command that starts the server. The steps, by hand:++## 1. Build++Strata's GGML dependency must be the agentionai llama.cpp fork, which has the trellis types:++```sh+git clone --depth 1 https://github.com/agentionai/llama.cpp ../agention-llama.cpp+# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120: issues #220, #224)+cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \+ -DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp+# AMD (experimental: kernel parity validated on gfx1151, not run end to end)+cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \+ -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp+cmake --build build -j+python3 -m venv .venv && .venv/bin/pip install -r requirements.txt+```++## 2. Download and pack++```sh+.venv/bin/pip install huggingface_hub hf_transfer+HF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next-Gyro-GGUF \+ Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf --local-dir models+STRATA_GGUF_PY=../agention-llama.cpp/gguf-py .venv/bin/python tools/iq_pack.py \+ --gguf models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf --out packs/gyro-s --compat-bf16+build/native_expert_parity --load-check packs/gyro-s models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf+```++The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables); the experts are read from the+GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for every token.+`--load-check` must end with `0 failures`.++## 3. Draft head++```sh+.venv/bin/python tools/mtp_fetch.py fetch --out mtp+.venv/bin/python tools/mtp_pack.py --src mtp --experts q2_0 --out mtp/mtp-q2_0.gguf+.venv/bin/python tools/mtp_rt.py --gguf mtp/mtp-q2_0.gguf --out mtp/rt+cp data/draft_vocab.bin mtp/rt/draft_vocab.bin+```++## 4. Serve++Save as `strata-gyro-s.json` at the repository root:++```json+{+ "exe": "build/strata",+ "args": [+ "--pack", "packs/gyro-s",+ "--native", "models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf",+ "--expert-profile", "data/expert-profile.bin", "--expert-cache", "auto",+ "--prefill", "512", "--spec", "4", "--spec-min-p", "0.5",+ "--mtp", "mtp/rt", "--max-context", "131072", "--kv", "int8"+ ],+ "cwd": ".",+ "tokenizer": "packs/gyro-s/tokenizer",+ "model_name": "gyro-s-strata",+ "log": "strata-gyro-s.log",+ "host": "127.0.0.1",+ "port": 8080+}+```++```sh+STRATA_APR=1 .venv/bin/python -m serve.server --engine strata --config strata-gyro-s.json --port 8080+```++The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1` (OpenAI) or+`/v1/messages` (Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM: on a 32 GB card at+128k context it holds 23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest+from RAM; nothing else changes.++## Measured (RTX 5090, CUDA 13.0, greedy, same prompt tokens as llama.cpp)++Decode tokens/s, single prompts of 34-177 tokens, 384 new tokens, `--max-context 16384`:++| Expert cache | prose | JSON | code |+|---|---:|---:|---:|+| 24,576 slots (all experts) | 116.4 | 194.3 | 173.5 |+| 22,000 slots | 111.4 | 175.7 | 166.1 |+| 9,000 slots (about a 16 GB card's room) | 81.4 | 74.0 | 96.8 |+| 3,000 slots (about an 8 GB card's room) | 48.0 | 52.4 | 57.9 |+| llama.cpp (agentionai main), best of no draft / cost-aware MTP | 106.0 | 172.9 | 142.2 |++The smaller caches were emulated on the 5090 (fast GPU, PCIe 5.0 x16 at ~51 GB/s). A real 8 or 16 GB card is a+slower GPU, often on a slower link: read those rows as "it stays usable", not as a prediction for a specific card.+The decoded prose output was read and is fluent and correct; token-level parity against llama.cpp has not been+checked. A real chat at 128k context worked through the web interface.++## Known limits++- **Not measured yet:** long-prompt prefill, token parity against llama.cpp, Gyro-M end to end.+- **Benchmark mode only:** `strata --tokens-file ... --max-new N` decodes N tokens past the end of the answer;+ compare speeds only up to the answer's end.+- **Agent harnesses:** one tool-calling agent client did not work with the Strata server while the web interface+ did; not yet diagnosed. llama.cpp's server is the tested path for agents.+- **AMD:** the trellis kernels pass parity on gfx1151 (ROCm 7.2.1); no end-to-end run on AMD yet.+- **Gyro-M:** 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache; `auto` streams the rest.diff --git a/tools/gyro_setup.sh b/tools/gyro_setup.shnew file mode 100755index 000000000..2705222d8--- /dev/null+++ b/tools/gyro_setup.sh@@ -0,0 +1,81 @@+#!/usr/bin/env bash+# Set up agentionai Gyro (Qwen3.8-Flash-Next rotor quants) on Strata: build, download, pack, draft head, config.+# Linux only. See docs/GYRO.md for what each step does.+# tools/gyro_setup.sh --model S|M --backend cuda|hip [--cuda-arch 120] [--hip-arch gfx1201]+# [--models DIR] [--llama DIR] [--ctx 131072] [--port 8080]+set -euo pipefail+cd "$(dirname "$0")/.."+MODEL=S BACKEND=cuda CUDA_ARCH=120 HIP_ARCH=gfx1201 MODELS=models LLAMA=../agention-llama.cpp CTX=131072 PORT=8080+while [ $# -gt 0 ]; do case "$1" in+ --model) MODEL=$2; shift 2;; --backend) BACKEND=$2; shift 2;; --cuda-arch) CUDA_ARCH=$2; shift 2;;+ --hip-arch) HIP_ARCH=$2; shift 2;; --models) MODELS=$2; shift 2;; --llama) LLAMA=$2; shift 2;;+ --ctx) CTX=$2; shift 2;; --port) PORT=$2; shift 2;; -h|--help) sed -n 2,5p "$0"; exit 0;;+ *) echo "unknown option $1"; exit 2;; esac; done+case "$MODEL" in S) F=Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf; P=gyro-s;; M) F=Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf; P=gyro-m;;+ *) echo "--model must be S or M"; exit 2;; esac+step() { printf '\n== %s\n' "$*"; }+for t in git cmake ninja python3; do command -v $t >/dev/null || { echo "missing: $t"; exit 1; }; done++step "agentionai llama.cpp fork (GGML with the trellis types) -> $LLAMA"+[ -d "$LLAMA/ggml" ] || git clone --depth 1 https://github.com/agentionai/llama.cpp "$LLAMA"+LLAMA=$(cd "$LLAMA" && pwd)++step "build Strata ($BACKEND)"+if [ "$BACKEND" = cuda ]; then+ NV=$(nvcc --version 2>/dev/null | sed -n 's/.*release \([0-9]*\)\..*/\1/p' || true)+ if [ "${CUDA_ARCH}" -ge 120 ] && [ "${NV:-0}" -lt 13 ]; then+ echo "warning: RTX 50 (sm_${CUDA_ARCH}) with CUDA ${NV:-?}: use CUDA 13 (12.8 crashes the prompt path, issues #220/#224)"; fi+ cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \+ -DCMAKE_CUDA_ARCHITECTURES="$CUDA_ARCH" -DSTRATA_GGML_DIR="$LLAMA"+elif [ "$BACKEND" = hip ]; then+ echo "note: AMD is experimental for Gyro (kernel parity validated on gfx1151, no end-to-end run yet)"+ cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \+ -DCMAKE_HIP_ARCHITECTURES="$HIP_ARCH" -DSTRATA_GGML_DIR="$LLAMA"+else echo "--backend must be cuda or hip"; exit 2; fi+cmake --build build -j++step "python environment"+[ -x .venv/bin/python ] || python3 -m venv .venv+.venv/bin/pip install -q -r requirements.txt huggingface_hub hf_transfer++step "download $F -> $MODELS (keep it on a fast local SSD: the n-gram table is read per token)"+mkdir -p "$MODELS"+HF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next-Gyro-GGUF "$F" --local-dir "$MODELS"+G=$(cd "$MODELS" && pwd)/$F++step "pack -> packs/$P"+STRATA_GGUF_PY="$LLAMA/gguf-py" .venv/bin/python tools/iq_pack.py --gguf "$G" --out "packs/$P" --compat-bf16+LC=$(build/native_expert_parity --load-check "packs/$P" "$G" 2>&1) || true; echo "$LC"+echo "$LC" | grep -q " 0 failures" || { echo "load check failed: see above"; exit 1; }++step "draft head (Strata's MTP runtime)"+if [ ! -f mtp/rt/draft_vocab.bin ]; then+ .venv/bin/python tools/mtp_fetch.py fetch --out mtp+ .venv/bin/python tools/mtp_pack.py --src mtp --experts q2_0 --out mtp/mtp-q2_0.gguf+ .venv/bin/python tools/mtp_rt.py --gguf mtp/mtp-q2_0.gguf --out mtp/rt+ cp data/draft_vocab.bin mtp/rt/draft_vocab.bin+fi++step "config strata-$P.json"+cat > "strata-$P.json" <<JSON+{+ "exe": "build/strata",+ "args": [+ "--pack", "packs/$P",+ "--native", "$G",+ "--expert-profile", "data/expert-profile.bin", "--expert-cache", "auto",+ "--prefill", "512", "--spec", "4", "--spec-min-p", "0.5",+ "--mtp", "mtp/rt", "--max-context", "$CTX", "--kv", "int8"+ ],+ "cwd": ".",+ "tokenizer": "packs/$P/tokenizer",+ "model_name": "$P-strata",+ "log": "strata-$P.log",+ "host": "127.0.0.1",+ "port": $PORT+}+JSON++step "done. Start the server with:"+echo " STRATA_APR=1 .venv/bin/python -m serve.server --engine strata --config strata-$P.json --port $PORT"+echo " web: http://127.0.0.1:$PORT API: http://127.0.0.1:$PORT/v1"