Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

docs: route prior 1.5, final verdict of the 80-task check on the RTX 3060 (no copy or code errors; agent loops may finish slightly less often, within the run-to-run spread), speed and the off switch

Niko1221c41bd44de7f3
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index ce2aee9e8..ed2d07573 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -957,7 +957,7 @@ them on your own card.
| `STRATA_PREFILL_EQUAL=1` | CUDA | A prompt segment is read in chunks of equal size, not full chunks and a short last one. Changes rounding. | |
| `STRATA_OWNED_PRICE=exact` | CUDA | The cache sizing prices the prompt path's own buffers by their real allocation. | |
| `STRATA_IQ_MT_MIN=1`, `STRATA_IQ3S_MT1=1` | CPU | See [Draft layer](#draft-layer) (reproducible output; the second gives IQ3_S the multi-token kernel for one token on Alder Lake and later without AVX-512). | |
-| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended, except that 0.2.7 turns `STRATA_ROUTE_PRIOR=1.5` on by itself on a discrete NVIDIA card of 13 GiB or less (the owner approved the quality trade for 12 GB cards; the first 0.2.7 candidates used 2.0, see the 12 GB default in the release notes for why it is 1.5); `STRATA_ROUTE_PRIOR=0` turns it off and an explicit value wins. It is never a default on a Radeon, an integrated GPU or a bigger card. The 1.5 default was checked with 61 tasks on an RTX 3060 12 GB (IQ3_XXS): no copy, code-constant, JSON-spelling, retrieval or unanswerable defects (retrieval 12/12, JSON 3/3, unanswerable 9/9, extraction equal to off), and two long-winded answers (a rambling agent paragraph, a code answer cut off in a comment); the runs with the prior off show two or three such cases each. Speed against off: story +15%, code +37%. |
+| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended, except that 0.2.7 turns `STRATA_ROUTE_PRIOR=1.5` on by itself on a discrete NVIDIA card of 13 GiB or less (the owner approved the quality trade for 12 GB cards; the first 0.2.7 candidates used 2.0, see the 12 GB default in the release notes for why it is 1.5); `STRATA_ROUTE_PRIOR=0` turns it off and an explicit value wins. It is never a default on a Radeon, an integrated GPU or a bigger card. The 1.5 default was checked on an RTX 3060 12 GB (IQ3_XXS) with 80 tasks: no copy or code errors seen; agent loops may finish slightly less often, within the run-to-run spread (out of 10 agent tasks 4.5 passed on average over 4 runs at 1.5, against 5.6 over 7 runs with the prior off, where the off runs alone ranged from 4 to 6). Speed against off: story +15%, code +37%. Off: the Expert-settings row "Faster answers on 12 GB NVIDIA cards" or `STRATA_ROUTE_PRIOR=0`. |
| `STRATA_HOST_CORE=last\|sibling` (or `--host-core`) | Windows | `last` puts the host thread on the last physical core, away from the core that gets the GPU's interrupts. `sibling` keeps it on the first core's other hardware thread, so the workers keep every core. The default `first` leaves a hybrid CPU as it is. | One layer-split box went from a ±6% swing to steady and 8% faster. |
| `--pool-affinity measure` | Hybrid CPUs, native packs | Times the `all` and `p-cores` layouts at start (about a second) and takes the faster. The measurement is taken once; where the E-cores' speed changes while the engine runs, `--pool-affinity p-cores` is the steady choice. | |
| `STRATA_KEEP_EMPTY_TURNS=1`, `STRATA_TOPK_STREAM=0` | | See [TROUBLESHOOTING.md](TROUBLESHOOTING.md). | |
@@ -1137,7 +1137,7 @@ On 12.0 cards (sm_120) these also apply:
prompt time and start-up time, which is why smaller PCs keep the SSD reads.
A prompt chunk above 8192 (`--prefill auto:16384` or `auto:32768`) is not bit-exact on CUDA, so the default stays 8192 on
-NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR` raises KL, so below 0.2.7 it was opt-in; 0.2.7 sets 1.5 by itself only on a discrete NVIDIA card of 13 GiB or less. A 61-task check of 1.5 on an RTX 3060 found no copy, constant, spelling or retrieval defects and two long-winded answers (the prior off has two or three per run); story +15%, code +37%.
+NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR` raises KL, so below 0.2.7 it was opt-in; 0.2.7 sets 1.5 by itself only on a discrete NVIDIA card of 13 GiB or less. An 80-task check of 1.5 on an RTX 3060 saw no copy or code errors; agent loops may finish slightly less often, within the run-to-run spread; story +15%, code +37%.
---
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.md
index 0d09c9641..39fb9e7b6 100644
--- a/docs/release-notes-0.2.7.md
+++ b/docs/release-notes-0.2.7.md
@@ -6,7 +6,7 @@ DRAFT of 27 October. Text in square brackets that starts with "pending" or "TO C
## What you will notice
-- **NVIDIA cards with 12 GB or less answer faster.** The router now prefers experts that are already on the card, which saves page-ins from RAM. On an RTX 3060 12 GB (Qwen3.8-Flash-Next IQ3_XXS, five paired runs against the same build with the setting off), a story answer went from 46.2 to 53.1 tok/s (+15%) and a code answer from 45.7 to 62.5 tok/s (+37%). It picks a slightly different set of experts, so the wording of answers changes. We checked the quality of the answers with 61 test tasks on that card. Retrieval (12 of 12), JSON (3 of 3) and unanswerable questions (9 of 9) were all right, and extraction found the same facts as with the setting off; we found no copy, code-constant or spelling defect. It did give two long-winded answers: one rambling paragraph in an agent task and one code answer that was cut off in the middle of a comment. Runs with the setting off show two or three such cases each too. The automatic grader passed 49 of the 61 tasks against 54 with the setting off; we read all five tasks it lost: three were grader flags (a name in a summary, a product name in a summary and a non-breaking hyphen in an agent task) and two were the long-winded answers above. A stronger setting (2.0) did show copy and code defects, so the default is 1.5. It applies to a discrete NVIDIA card of 13 GiB or less; larger cards, Radeon and Strix Halo are not changed. Off: "Faster answers on 12 GB NVIDIA cards" in the model's settings, or `STRATA_ROUTE_PRIOR=0`.
+- **NVIDIA cards with 12 GB or less answer faster.** The router now prefers experts that are already on the card, which saves page-ins from RAM. On an RTX 3060 12 GB (Qwen3.8-Flash-Next IQ3_XXS, five paired runs against the same build with the setting off), a story answer went from 46.2 to 53.1 tok/s (+15%) and a code answer from 45.7 to 62.5 tok/s (+37%). It picks a slightly different set of experts, so the wording of answers changes. We checked the answers on that card (IQ3_XXS) with 80 tasks and saw no copy or code errors. Agent loops may finish slightly less often, within the run-to-run spread: out of 10 agent tasks, 4.5 passed on average over 4 runs at 1.5, against 5.6 over 7 runs with the setting off, where the off runs alone ranged from 4 to 6. A stronger setting (2.0) did show copy and code defects, so the default is 1.5. It applies to a discrete NVIDIA card of 13 GiB or less; larger cards, Radeon and Strix Halo are not changed. Off: the row "Faster answers on 12 GB NVIDIA cards" in the Expert settings (Models, Native engine settings, Speed), or `STRATA_ROUTE_PRIOR=0`.
- **Radeon RX 9000 and AI PRO R9700: faster Qwen.** Two changes: the large per-token lookup table stays in RAM when the PC has room for it, and the decode split between card and processor starts at a better value. One R9700 (32 GB) with 125 GB of RAM on Linux, Qwen IQ3_S, medians of three runs:
| Context | Prompt before | Prompt now | Decode before | Decode now |