Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Release notes 0.2.1: the R9700 queue fix with the two-queue numbers
Niko1221b42ec662c768
diff --git a/docs/release-notes-0.2.1.md b/docs/release-notes-0.2.1.mdindex ba3b05afc..bca31d9e3 100644--- a/docs/release-notes-0.2.1.md+++ b/docs/release-notes-0.2.1.md@@ -45,7 +45,7 @@ Speed numbers are medians of interleaved pairs on the same box, a fresh engine p- **Two NVIDIA cards, several requests at once: one batch group.** With a layer split over exactly two NVIDIA cards and `--batch N`, the engine now runs one serial group instead of two pipelined groups. Measured on two RTX A4000 16 GB with Qwen3.8-Flash-Next IQ3_XXS, 5 interleaved pairs, aggregate tokens per second: 2 requests +15.9% (5 of 5 pairs), 4 requests +12.0% (5 of 5), 8 requests +14.7% (3 of 3). Every slot's tokens equal the solo run's, and the old default's, under the fixed identity flags (2 and 4 slots, 64 greedy tokens). A single request is not changed. It is not faster everywhere: when most hot experts sit in the CPU, as under the identity gate's small cache, the pipelined groups were about 10% faster (24.4 against 22.0 tok/s), so a card with a very small cache can lose here. `STRATA_BATCH_GROUPS_NV_SERIAL=0` restores the old default; `--batch-groups auto` or a number asks for the pipeline. AMD cards, three or more cards and the single-request paths are unchanged. TODO(gate): the 5-pair A/B of the final engine at 4 slots (2 of 5 pairs done: new default faster 2 of 2) and the batch identity check on the final build.- **A card that draws no display keeps less in reserve.** The reserve of graphics memory the engine leaves free is now 512 MiB (700 MiB with pictures) on a card the probe says drives no screen (a compute-only card), instead of 1536 MiB. A card that drives a display keeps the 1536 / 2048 MiB safety default, and a number you set yourself (0 included) is always used as it is. Only the amount of free memory changes, so the answers are identical and the expert cache gets room for more experts. We did not time this ourselves: the report behind it (#3215) measured 80 against 50 tok/s with the forced reserve. Where the probe cannot tell (AMD and Intel on Linux) the safety default stays.- **Packed models (`experts.bin`) on Linux: I/O prefetch and the decode gate are on when the page cache has room for at least 55% of the experts.** Measured on an RTX 3060 12 GB (Core Ultra 7 265), Qwen3.8-Flash-Next IQ3_XXS 43 GB pack with `--mmap-experts`, RAM limited to 32 GB by cgroup, cold page cache, 5 interleaved pairs: decode story / code / document 26.0 / 28.3 / 18.5 to 35.2 / 34.4 / 24.7 tok/s (x1.38 / x1.21 / x1.33, 5 of 5 pairs each), a 4K prompt x1.04 (4 of 5), a 20K prompt x1.06 (5 of 5). Same answers: the fixed-flag identity gate (10 prompts, 4K and 20K exact prompt checks) is identical with no RAM limit and with the 32 GB limit. With 16 GB of RAM the page cache holds 38% of the experts, so the default stays off: with it on, a cold 4K prompt read 4 to 6% slower. `STRATA_IO_PREFETCH=0` turns it off and `=1` forces it on. Linux only; only the IQ3_XXS pack was tested (not Q2_0, IQ3_S or Coder packs).-- **AMD RX 9000 / R9700 on Linux: the engine opens its GPU queues in a fixed order at start.** Before, about half of the fresh engines with MTP read prompts 8 to 14% slower, depending on which compute pipe the card's firmware gave the engine's two busiest queues (both on one pipe was slow). Measured on a Radeon AI PRO R9700 (Linux, ROCm 7.10, Q2_0 with MTP), 12 interleaved pairs: the old engine had 4 slow or cold starts of 12 (4K prompt 1,973 to 2,094 tok/s against 2,287 for a fast start); the new one had 12 of 12 fast (4K 2,281 to 2,291, 20K 2,759 to 2,773). Over 108 starts with the queue order set 2 were slow (about 2%), against about 45% before. The same fix helps without MTP (1 to 4 of 12 slow before, 12 of 12 fast after). Decode is unchanged. Same answers: the identity gate (10 prompts, 4K and 20K exact checks) is identical for Q2_0 and IQ3_XXS, also with the 0.2.0 prompt defaults on. `STRATA_HIP_QUEUE_PRIME=0` turns it off. One card was measured; Windows and RX 7000 cards are not changed.+- **AMD RX 9000 / R9700 on Linux: the engine opens its GPU queues in a fixed order at start.** Before, about half of the fresh engines with MTP read prompts 8 to 14% slower, depending on which compute pipe the card's firmware gave the engine's two busiest queues (both on one pipe was slow). The engine now opens two queues on its main thread before anything else starts. Measured on two Radeon AI PRO R9700 boxes (Linux, ROCm 7.10, Q2_0 with MTP): without the fix 10 of 32 fresh starts read the 4K prompt under 2,000 tok/s (about 2,290 for a fast start); with it 0 of 32. Decode against the fix switched off, 20 interleaved rounds: 1.000 / 1.001 / 1.000 (story / code / facts). Opening more queues cost decode (five queues: 2 to 3.6% slower), so two is the default. Same answers: the identity gate (10 prompts, 4K and 20K exact checks) is identical for Q2_0 and IQ3_XXS, also with the 0.2.0 prompt defaults on. `STRATA_HIP_QUEUE_PRIME=0` turns it off. One card was measured; Windows and RX 7000 cards are not changed.- **Linux hybrid CPUs with low-power E cores (Core Ultra "Meteor Lake" / "Arrow Lake-H", #3262).** The engine and setup now count the cores listed in `cpu_lowpower` as efficiency cores. A Core Ultra 7 255H used to look like 8 P and 8 E cores, so the hybrid plan did not apply and 15 workers ran; it now plans 10. The reporter measured the 15-worker default about 35% slower than 9. We have no such CPU and ran unit tests only (the plan and a fake `/sys`); only the number of workers changes, not the answers.- **Two GPUs of different architectures (HIP): no start-up warm-up (#3204).** The warm-up product that runs on every card at start made prompts about 25% slower on a split of gfx1151 and gfx1201 for the reporter (`STRATA_PREFILL_WARM=0` restored them). It now runs only when every visible card has the same architecture. We have no mixed pair; on two R9700 (same architecture) the warm-up costs nothing (3,855 against 3,853 tok/s at a 32K prompt, 5 pairs), so the change leaves them as they were. The first request on a mixed pair pays the 1 to 2.5 s library load again, as before 0.2.0. `STRATA_PREFILL_WARM=1` / `0` overrides.- **Linux, models read in place: a drive probe decides between the file cache and unbuffered reads (#3237). Pending owner OK; not final for the release.** The room rule picks the file cache when RAM seems to have room, which was 2x slower in prompts and 3 to 4x slower in decode for a reporter with a PCIe 5 drive (about 8 GB/s). When the rule says "cache" and the cache would hold under half of the reads, the engine now reads 96 random 1 MiB pieces both ways at start (under 2 s) and goes unbuffered only if the drive reads at least 4 GB/s and is faster that way. Every drive we have (1.6 to 3.1 GB/s) keeps the file cache; on a 2 GB/s NVMe the cache was better for decode (up to x1.6) and unbuffered reads better for prompts, the opposite of the report. Untested on a fast drive; the answers cannot change (only where reads go), and no identity gate was run. `STRATA_FILE_TIER_PROBE=0` turns it off.