Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Release notes 0.2.1: final two-card batch A/B result, drop TODOs
Niko122190767252f420
diff --git a/docs/release-notes-0.2.1.md b/docs/release-notes-0.2.1.mdindex 635b9a865..1f11d1d2d 100644--- a/docs/release-notes-0.2.1.md+++ b/docs/release-notes-0.2.1.md@@ -40,7 +40,7 @@ Speed numbers are medians of interleaved pairs on the same box, a fresh engine p- **Mini PCs with a Radeon 780M (gfx1103), Linux: the matrix-core prompt set is on.** Measured on a Ryzen 7 8745HS with a Radeon 780M, 60 GB DDR5, Linux kernel 7.0.0-38, Qwen3.8-Flash-Next Q2_0 with MTP, 6 interleaved pairs: a 1K prompt 98.5 to 157.3 tok/s (x1.60, 6 of 6 pairs), a 4K prompt 115.1 to 233.3 (x2.03, 6 of 6), decode unchanged (x0.999 and x1.000). The greedy token ids equal the old default's in every pair (6 of 6, and 5 of 5 in the final build). Same answers, as far as the ids show. `STRATA_IGPU_PROMPT_SET=0` turns it off. It applies to the 780M and 760M on Linux only; 880M, 890M, Strix Halo and Windows stay off. The 780M's GPU can still reset on its own firmware (see Known limits); with the set on we saw no more resets than with it off, in small samples.- **Strix Halo (gfx1151): the hipBLASLt tuning table is on.** The packaged Strix Halo engines (Linux `strixhalo` and the Windows `rdna3` package) use the table for their card when their hipBLASLt version matches it. The measurement is a community A/B from `bench/results/2026-10-05-community-gfx1151` (Linux, ROCm 7.17, hipBLASLt 1.5.0): prompts 250.9 to 528.9 tok/s (2.11x). We have no Strix Halo and did not repeat it. The table changes which matrix-product algorithm reads the prompt, so the answers are not bit-identical to plain hipBLAS (on an R9700 the first-token KL was 0.016, against a run-to-run difference of 0.00004; top-1 the same on 24 of 24 prompts). `STRATA_HIPBLASLT_TABLE=off` turns it off. On every other card the table is opt-in (`STRATA_HIPBLASLT_TABLE=on`): on the R9700 a table calibrated for the packaged hipBLASLt gave about +1% prompt and -2% decode, so none is shipped as a default. The tables are packaged per card (`tuning/` inside the engine package): Windows `rdna3` gfx1100 and gfx1151, Windows `hip` (RX 9000, R9700) gfx1201, Linux `rx7900` gfx1100, Linux `strixhalo` gfx1151; the Linux RX 9000 / R9700 package carries none, because no table matches its hipBLASLt. Only the gfx1151 table is used by default.-- **Two NVIDIA cards, several requests at once: one batch group.** With a layer split over exactly two NVIDIA cards and `--batch N`, the engine now runs one serial group instead of two pipelined groups. Measured on two RTX A4000 16 GB with Qwen3.8-Flash-Next IQ3_XXS, 5 interleaved pairs, aggregate tokens per second: 2 requests +15.9% (5 of 5 pairs), 4 requests +12.0% (5 of 5), 8 requests +14.7% (3 of 3). Every slot's tokens equal the solo run's, and the old default's, under the fixed identity flags (2 and 4 slots, 64 greedy tokens). A single request is not changed. It is not faster everywhere: when most hot experts sit in the CPU, as under the identity gate's small cache, the pipelined groups were about 10% faster (24.4 against 22.0 tok/s), so a card with a very small cache can lose here. `STRATA_BATCH_GROUPS_NV_SERIAL=0` restores the old default; `--batch-groups auto` or a number asks for the pipeline. AMD cards, three or more cards and the single-request paths are unchanged. TODO(gate): the 5-pair A/B of the final engine at 4 slots (2 of 5 pairs done: new default faster 2 of 2) and the batch identity check on the final build.+- **Two NVIDIA cards, several requests at once: one batch group.** With a layer split over exactly two NVIDIA cards and `--batch N`, the engine now runs one serial group instead of two pipelined groups. Measured on two RTX A4000 16 GB with Qwen3.8-Flash-Next IQ3_XXS, 5 interleaved pairs, aggregate tokens per second: 2 requests +15.9% (5 of 5 pairs), 4 requests +12.0% (5 of 5), 8 requests +14.7% (3 of 3). Every slot's tokens equal the solo run's, and the old default's, under the fixed identity flags (2 and 4 slots, 64 greedy tokens). A single request is not changed. It is not faster everywhere: when most hot experts sit in the CPU, as under the identity gate's small cache, the pipelined groups were about 10% faster (24.4 against 22.0 tok/s), so a card with a very small cache can lose here. `STRATA_BATCH_GROUPS_NV_SERIAL=0` restores the old default; `--batch-groups auto` or a number asks for the pipeline. AMD cards, three or more cards and the single-request paths are unchanged. On the final engine, 4 requests, 5 pairs: 78.06 against 65.97 tok/s (+18.3%, the old default faster in 0 of 5), and every slot's tokens equal the solo run's.- **A card that draws no display keeps less in reserve.** The reserve of graphics memory the engine leaves free is now 512 MiB (700 MiB with pictures) on a card the probe says drives no screen (a compute-only card), instead of 1536 MiB. A card that drives a display keeps the 1536 / 2048 MiB safety default, and a number you set yourself (0 included) is always used as it is. Only the amount of free memory changes, so the answers are identical and the expert cache gets room for more experts. We did not time this ourselves: the report behind it (#3215) measured 80 against 50 tok/s with the forced reserve. Where the probe cannot tell (AMD and Intel on Linux) the safety default stays.- **Packed models (`experts.bin`) on Linux: I/O prefetch and the decode gate are on when the page cache has room for at least 55% of the experts.** Measured on an RTX 3060 12 GB (Core Ultra 7 265), Qwen3.8-Flash-Next IQ3_XXS 43 GB pack with `--mmap-experts`, RAM limited to 32 GB by cgroup, cold page cache, 5 interleaved pairs: decode story / code / document 26.0 / 28.3 / 18.5 to 35.2 / 34.4 / 24.7 tok/s (x1.38 / x1.21 / x1.33, 5 of 5 pairs each), a 4K prompt x1.04 (4 of 5), a 20K prompt x1.06 (5 of 5). Same answers: the fixed-flag identity gate (10 prompts, 4K and 20K exact prompt checks) is identical with no RAM limit and with the 32 GB limit. With 16 GB of RAM the page cache holds 38% of the experts, so the default stays off: with it on, a cold 4K prompt read 4 to 6% slower. `STRATA_IO_PREFETCH=0` turns it off and `=1` forces it on. Linux only; only the IQ3_XXS pack was tested (not Q2_0, IQ3_S or Coder packs).- **AMD RX 9000 / R9700 on Linux: the engine opens its GPU queues in a fixed order at start.** Before, about half of the fresh engines with MTP read prompts 8 to 14% slower, depending on which compute pipe the card's firmware gave the engine's two busiest queues (both on one pipe was slow). The engine now opens two queues on its main thread before anything else starts. Measured on two Radeon AI PRO R9700 boxes (Linux, ROCm 7.10, Q2_0 with MTP): without the fix 10 of 32 fresh starts read the 4K prompt under 2,000 tok/s (about 2,290 for a fast start); with it 0 of 32. Decode against the fix switched off, 20 interleaved rounds: 1.000 / 1.001 / 1.000 (story / code / facts). Opening more queues cost decode (five queues: 2 to 3.6% slower), so two is the default. Same answers: the identity gate (10 prompts, 4K and 20K exact checks) is identical for Q2_0 and IQ3_XXS, also with the 0.2.0 prompt defaults on. `STRATA_HIP_QUEUE_PRIME=0` turns it off. One card was measured; Windows and RX 7000 cards are not changed.@@ -92,7 +92,6 @@ Speed numbers are medians of interleaved pairs on the same box, a fresh engine p- **A100 (sm_80), #3252:** a report of decode -10% and prompts -20% since 0.1.45 was not reproduced. On a Tesla P100 (sm_60) and two RTX A4000 (sm_86) the 0.1.45 build against 0.1.44.1 measured decode x1.015 and x0.992, prompts x1.004 and x1.022 (5 interleaved rounds each), and the 0.2.1 tip is 7 to 10% faster in decode than 0.1.45 on both. Reading the commits found no sm_80-specific change. We have no A100; the reporter's numbers are needed.- **Fast drives, #3237:** the drive probe above is proposed and untested on a drive over 4 GB/s.- **NUMA-split CPU pool (#3201):** not part of 0.2.1; we have no dual-socket box to measure it.-- TODO(release): anything the gates find.Thanks to H3buss, who found that the Gyro import was missing `--compat-bf16` and where to put it (#3193); to bext1998, whose trace of the MTP download and the orphaned process (#3257) led to the progress and cancel fixes; and to Gazoo, who reported the card-selection, dropdown and re-download bugs (#3195, #3221, #3199). Thanks also to everyone who tested 0.2.0-beta and sent reports.