Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: Gyro on Intel Arc (GYRO.md, INTEL.md, release notes 0.2.2)
Niko1221f35308710485
diff --git a/docs/GYRO.md b/docs/GYRO.mdindex e8238a5c1..b1ad92389 100644--- a/docs/GYRO.md+++ b/docs/GYRO.md@@ -38,8 +38,8 @@ APR port, and the released 0.1.45 stops at v4. A ready-made engine that is older(setup), or the app says the model needs a newer engine and offers Update engine.**Supported:** NVIDIA (CUDA, Linux and Windows) and AMD (HIP, experimental: the authors validated gfx1151; we ran-gfx1201 with identical output to rc2). **Not supported:** Intel Arc (SYCL) and Apple Metal: the engines there have no-trellis kernels; the Metal engine refuses the file at load with the reason.+gfx1201 with identical output to rc2). **Intel Arc** (SYCL, Linux, built from source) runs both files: see [Intel Arc](#intel-arc-sycl). **Not supported:** Apple Metal: the engine there has no+trellis kernels and refuses the file at load with the reason.**Pictures.** Gyro reads pictures with the original model's picture reader file (the same one every Qwen3.8-Flash-Nextmodel uses; setup downloads it with the model). The reader opens only the vocabulary and the header of the Gyro file, so@@ -62,6 +62,60 @@ prompts through it as fast as rc2 does, and bit for bit the same. If the pinnedno longer matches, the overlay switches itself off with a message and prompts take the slower FP16 path. A build against thefork (`-DSTRATA_GGML_DIR=<agentionai/llama.cpp>`, as rc2 does) uses the fork's own instances instead.+## Intel Arc (SYCL)++Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from this source ([INTEL.md](INTEL.md)).+Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM together are short. Tested on one+card: an Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM, on the old-release files of Gyro-S and+Gyro-M (the same size as the pinned ones, a different SHA-256; the pinned Gyro-S was still downloading when this was written).+Not tested: a B-series card, two Arc cards (the second-card expert helper refuses a Gyro pack and says so), an Arc on the xe driver.++The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook, q8_1 activations),+the Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt takes the grouped XMX path,+which now decodes the trellis types into its tiles. Experts in the card's cache run on the card; the other misses go to the CPU pool or+over the PCIe link, as with every pack.++**Checks.** `tq_sycl_parity <pack> <model.gguf> [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and `--bench GU/DOWN`) takes real+experts of the file (layers 0-3, 10, 20, 30 and 47, two experts each) and compares the card with a float reference and with the CPU path:++- the decode kernels equal a host replay of their own arithmetic **in every bit** (0 of 3,840 gate/up values and 0 of 7,680 down values+ differ, Gyro-S and Gyro-M, all 16 cases each); the prompt dequantizers equal the fp16 codebook in every bit (0 of 4.9 million values);+ the card's Hadamard rotation equals the CPU's bit for bit (the fp16 form within 3e-4);+- against the float reference the card is within 1.7-2.0e-2 relative L1 error (the int8 codebook; 3e-2 is the bound every expert type+ gets), the CPU path within 0.9e-2. The card and the CPU path therefore agree within tolerance, not bit for bit: the CPU path uses an+ int16 codebook and Q8_0 activations, the card the int8 codebook and q8_1, as on CUDA.++**Against CUDA.** The same Gyro-S file on an RTX 3060 (the f-022-int engine) and on the A750, both with 64 cached experts, no PCIe share and+no CPU prompt share, ten prompts of 26-100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens. The other three (prompts 5, 7 and 8) agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and the top token is always the same. The two SYCL prompt paths (oneMKL and XMX) also flip prompt 7 against each other, so the prompt-path rounding alone moves it.++**Measured** (A750, setup's config: 32K context, 300 MiB VRAM reserve, MTP draft of 4, PCIe probe 10.4 GB/s, median of 5 interleaved pairs):++| | before | now |+| --- | ---: | ---: |+| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | **82.0 tok/s** (5/5 pairs) |+| Gyro-S prompt, 2,377 tokens | | 101.9 tok/s |+| Gyro-S prompt, 20,000 tokens | | 195 tok/s |+| Gyro-S decode | 13.7 tok/s | **15.8 tok/s** (5/5 pairs) |+| Gyro-M decode (8K context) | | 14.2 tok/s (12.6-14.4), prompt 62 tok/s at 1,406 tokens |++The decode speed is the CPU's and the link's: with 8 GB the card holds 410 of the 24,576 experts (the dense weights, the draft layer,+the KV cache and the prompt buffers take the rest), so about 15 experts per layer and token come from the CPU pool+(5 workers, AVX2, 8 GB/s: the trellis decode is compute bound, 48 cycles a block) and about 5 over the link. Both move about 10 experts a+millisecond, so the link takes a bigger share than for the i-quants: for packs with trellis experts the engine scales the share with a 12 GB/s+reference instead of 20 (0.48 instead of 0.29 here; fixed 0.45 and 0.60 measured the same; 0.40 gained only 7%). The card's kernels are not the+limit: 20 microseconds an expert (53-68 GB/s of weights) against 18 for IQ3_S, 15 experts take 0.3 ms.++Smaller prompt chunks leave room for more cached experts and cost prompt speed: `--prefill 1024` gives 1,090 slots and +13% decode on a+1,406-token prompt but -24% prompt speed, `--prefill 512` 1,429 slots, -36%. The default stays `--prefill auto`.++**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9): with setup's 32K context the card is 750 MiB+short of one cache slot and the engine stops with that sentence. It starts with `--max-context 8192 --vram-reserve-mib 100 --prefill 1024`+(412 slots). A card with 12 GB or more does not need that.++**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together; the n-gram table (28 GB) stays on the SSD, so on an Arc setup does not+put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used about 24 GB of RAM for the+expert arena. Keep the file on a fast SSD.+## Credit and licensesThe trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk@@ -269,8 +323,8 @@ Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, J- **Not in this port** (rc2 has them; see the list at the end of this page): the prompt helper on a second GPU for a Gyropack. Setup of two cards still runs the layer split and the decode peer; the prompt peer says it is not used for a- Hadamard-folded pack and the prompt stays on the first card. Gyro-L, Intel Arc (SYCL) and Apple Metal are not supported.-- **Source builds:** the SYCL tree builds with these sources (P1006, 0 errors) but has no trellis GPU kernels; the Metal+ Hadamard-folded pack and the prompt stays on the first card. Gyro-L and Apple Metal are not supported.+- **Source builds:** the SYCL tree has the trellis kernels (Intel Arc, above); the Metalengine builds and refuses the file at load (`native_expert_supported`).- **Not measured yet:** token parity against llama.cpp.diff --git a/docs/INTEL.md b/docs/INTEL.mdindex 788157ee3..1aa957a30 100644--- a/docs/INTEL.md+++ b/docs/INTEL.md@@ -829,6 +829,26 @@ by the SYCL reader 10 of 10; two pictures in one request, a picture kept in thepictures all answered; a text request after every picture answered; 47 picture requests, outside and inside the container (inside, oneanswer was cut off by the 200-token limit of the test), no GPU fault in the kernel log, no 503.+## Gyro on an Arc (2026-10-20, branch f-022-arcgyro)++Gyro-S and Gyro-M (trellis-coded experts, [GYRO.md](GYRO.md#intel-arc-sycl)) run on the SYCL engine. Tested on the Arc A750 only (i915, 8 GB,+Ryzen 5 5600X, 62 GB of RAM). Setup writes the usual A750 config for them (32K context, 300 MiB VRAM reserve), without `--ple-io ram`: the+28 GB n-gram table stays on the SSD, because the expert arena already takes 24 GB (Gyro-S) of the RAM.++| A750, Gyro-S | |+| --- | ---: |+| prompt, 1,406 tokens | 82 tok/s (40 with dequantize + oneMKL; the grouped XMX launch now takes the trellis types, 5/5 pairs) |+| prompt, 2,377 / 20,000 tokens | 102 / 195 tok/s |+| decode, MTP draft of 4 | 15-16 tok/s (13.7 with the old link share, 5/5 pairs) |+| experts in the card's cache | 410 of 24,576 |++Decode is bound by the CPU pool (5 workers, 8 GB/s) and the PCIe link (10.4 GB/s measured), not by the card's kernels (20 microseconds an+expert). For trellis packs the engine takes the link share against a 12 GB/s reference instead of 20, which gives 0.48 on this card instead+of 0.29. A smaller prompt chunk makes room for more cached experts and slows the prompt: `--prefill 1024` +13% decode, -24% prompt speed;+the default stays `--prefill auto`. Gyro-M needs `--max-context 8192 --vram-reserve-mib 100 --prefill 1024` on 8 GB (14 tok/s decode,+62 tok/s prompt). A second Arc card does not take Gyro's experts (the engine refuses and says so). The Battlemage AOT build (`bmg-g31`)+compiles with the trellis kernels; it was not run, there is no B-series card here.+## Arc Pro B70 with a model that does not fit: Flash-Next IQ3_S (2026-10-07, branch f-141-intel)The earlier B70 rows are for models whose experts all fit in VRAM. IQ3_S has 24,576 experts of 1.97 MB (46 GB): the 32 GBdiff --git a/docs/release-notes-0.2.2.md b/docs/release-notes-0.2.2.mdindex 15a55b096..565972eaf 100644--- a/docs/release-notes-0.2.2.md+++ b/docs/release-notes-0.2.2.md@@ -2,6 +2,7 @@## What is new+- **Gyro runs on Intel Arc (Linux, tested on an Arc A750 only).** The SYCL engine has the trellis kernels (TQ2_T, TQK6, TQK7) and the Hadamard rotation, so Gyro-S and Gyro-M load and answer on an Arc card. Setup lists them like for any card, warns and never refuses when VRAM and RAM together are short, and keeps Gyro's n-gram table on the SSD. On the A750 (8 GB, Ryzen 5 5600X, 62 GB of RAM) Gyro-S reads a 1,400-token prompt at 82 tok/s (40 before the matrix-unit launch took the trellis types), a 20,000-token prompt at 195 tok/s, and writes 15-16 tok/s with the MTP draft. The card holds 410 of 24,576 experts, so the CPU and the PCIe link carry the decode; for these packs the engine gives the link a larger share (0.48 instead of 0.29, +16% decode). On real tensors of both files the kernels equal a host replay of their own arithmetic in every bit, and 7 of 10 test prompts give the same 48 tokens as the CUDA engine (the other three agree for 6 to 9 tokens and then differ at a near-tie; first-token KL 1e-4 to 1e-2). Gyro-M needs `--max-context 8192 --prefill 1024` on an 8 GB card. Not tested: B-series cards, two Arc cards (refused with a message), the pinned Gyro-S file (the first-release file of the same size was used). Gyro still does not run on the Mac. Details: [GYRO.md](GYRO.md#intel-arc-sycl).- **An engine package for the RX 6000 series (built, not tested on a real RX 6000 card).** Linux has `rx6000` and Windows has `rdna2`, each with GPU code for gfx1030 and gfx1031 (RX 6800, 6900 and 6950 series, RX 6700 XT). They are offered to a PC whose card reports one of those two and to no other: the RX 7000, RX 9000 and Strix Halo packages are unchanged, and a PC with an RX 7000 or 9000 card next to an RX 6000 card keeps the package it had. The engine runs plain kernels on these cards (RDNA2 has no matrix cores, so none of the RX 7000 / RX 9000 or Radeon 780M prompt paths apply) and plain hipBLAS (hipBLASLt ships no kernels for gfx1030, so there is no tuning table). Setup and the app say once that this package was built and not tested on a real RX 6000 card. What was checked: the Linux package builds, every program in it has code for exactly gfx1030 and gfx1031, the rocBLAS kernel files for both are in it, every library it imports is in the package or belongs to the system, and the whole engine (all targets, tests included) compiles for gfx1030 and gfx1031. What was not: a model on an RX 6000 card, on either system, and the Windows package itself (the Windows build scripts take the new architecture list, but no Windows `rdna2` package has been built). Earlier reports on these cards came from people who built the engine from source ([docs/AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030)). If something goes wrong, open an issue with the end of `strata-<model>.log`.- **The CPU expert pool splits per NUMA node on a machine with two or more NUMA nodes (built from the report in #3201, not tested on a two-socket machine).** Until now the pool kept its workers on the host thread's socket, because workers on the other socket read the experts across the link and were slower. The split gives each socket its own slice of the pool with its own task counter, and places each socket's half of every CPU-side expert in that socket's memory, so a slice reads local memory only. The reporter measured +10 to +14 % decode on a dual Xeon Gold 6226 (59.3 against 54.05 tok/s with the IOMMU in passthrough); a second reporter's dual Xeon E5-2670 v2 preferred more workers across both sockets, which the split also uses. It starts by itself when the machine has two or more NUMA nodes with CPUs and runs only when it can (Linux, two nodes, pinned threads, enough free memory per node); otherwise the pool is exactly what it was and the log says why. **A machine with one NUMA node runs the same code path as 0.2.1:** the identity gate (10 prompts, exact_pp 4K and 20K) is byte-identical and the single-request speed A/B is not slower on the one-node test boxes (numbers in the notes of the branch). `STRATA_NUMA_POOL=0` turns the split off; `STRATA_NUMA_SPLIT=1` still asks for it on any machine. The start-up log says what it did: `NUMA pool on: 2 NUMA nodes found ...` or `NUMA pool off: <reason>`. On Windows the node count is read but the split itself is Linux-only for now, so a two-node Windows PC logs the reason and runs as before. The logic runs in unit tests on made-up two-, three- and four-node machines; we have no two-socket machine, so the placement and the speed on real ones are the reporters' numbers, not ours. Do not wrap the server in `numactl --cpunodebind` or `--membind`: it pulls both halves onto one node. Thanks to baptisterajaut for the branch this is built on, and to xrip and Tworks25 for their numbers. Please send the start-up lines and a decode run with and without `STRATA_NUMA_POOL=0` from a two-socket machine.- **The API key is optional.** The server, `strata launch`, the Terminal and the app no longer refuse to share without a key. Say so (`--no-api-key`, or `"api_key": "none"` in the server config) and it starts, with one plain warning that anyone who can reach the port can use the model and its tools; a public Cloudflare address gets a stronger one. The Terminal and the app still offer to make a key and do by default. In the app, Remove key now also works while local sharing or the tunnel runs. Fixes the refusals reported in #3198 and #3401.