Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Route prior: the 0.2.7 default on a discrete NVIDIA card of 13 GiB or less drops from 2.0 to 1.5 (owner decision 2026-10-26). On the RTX 3060 12 GB (IQ3_XXS) the copy and code check found real defects at 2.0 and none in 19 tasks at 1.5 or 1.0; 1.5 keeps story +15% and code +37% against off. The value lives in strata::kRoutePriorSmallCard (sm120_series.hpp); scope (CUDA discrete <= 13 GiB), STRATA_ROUTE_PRIOR=0, an explicit value and STRATA_DEFAULTS=026 are unchanged. arch_defaults_test, verify.cpp comments, DETAILS.md and the 0.2.7 release notes follow.
Niko12210afaaa399edd
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex e0b9a7d12..eb9451325 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -957,7 +957,7 @@ them on your own card.| `STRATA_PREFILL_EQUAL=1` | CUDA | A prompt segment is read in chunks of equal size, not full chunks and a short last one. Changes rounding. | || `STRATA_OWNED_PRICE=exact` | CUDA | The cache sizing prices the prompt path's own buffers by their real allocation. | || `STRATA_IQ_MT_MIN=1`, `STRATA_IQ3S_MT1=1` | CPU | See [Draft layer](#draft-layer) (reproducible output; the second gives IQ3_S the multi-token kernel for one token on Alder Lake and later without AVX-512). | |-| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended, except that 0.2.7 turns `STRATA_ROUTE_PRIOR=2.0` on by itself on a discrete NVIDIA card of 13 GiB or less (the owner approved the quality trade for 12 GB cards); `STRATA_ROUTE_PRIOR=0` turns it off. It is never a default on a Radeon, an integrated GPU or a bigger card. |+| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended, except that 0.2.7 turns `STRATA_ROUTE_PRIOR=1.5` on by itself on a discrete NVIDIA card of 13 GiB or less (the owner approved the quality trade for 12 GB cards; the first 0.2.7 candidates used 2.0, see the 12 GB default in the release notes for why it is 1.5); `STRATA_ROUTE_PRIOR=0` turns it off and an explicit value wins. It is never a default on a Radeon, an integrated GPU or a bigger card. || `STRATA_HOST_CORE=last\|sibling` (or `--host-core`) | Windows | `last` puts the host thread on the last physical core, away from the core that gets the GPU's interrupts. `sibling` keeps it on the first core's other hardware thread, so the workers keep every core. The default `first` leaves a hybrid CPU as it is. | One layer-split box went from a ±6% swing to steady and 8% faster. || `--pool-affinity measure` | Hybrid CPUs, native packs | Times the `all` and `p-cores` layouts at start (about a second) and takes the faster. The measurement is taken once; where the E-cores' speed changes while the engine runs, `--pool-affinity p-cores` is the steady choice. | || `STRATA_KEEP_EMPTY_TURNS=1`, `STRATA_TOPK_STREAM=0` | | See [TROUBLESHOOTING.md](TROUBLESHOOTING.md). | |@@ -1137,7 +1137,7 @@ On 12.0 cards (sm_120) these also apply:prompt time and start-up time, which is why smaller PCs keep the SSD reads.A prompt chunk above 8192 (`--prefill auto:16384` or `auto:32768`) is not bit-exact on CUDA, so the default stays 8192 on-NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR` raises KL, so below 0.2.7 it was opt-in; 0.2.7 sets 2.0 by itself only on a discrete NVIDIA card of 13 GiB or less.+NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR` raises KL, so below 0.2.7 it was opt-in; 0.2.7 sets 1.5 by itself only on a discrete NVIDIA card of 13 GiB or less.---diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex a37622782..8c0081cf4 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -74,10 +74,14 @@ These need no engine change; they are settings. Each was measured on the same bo## The 12 GB default: residency-biased routing-**On by default at 2.0 on a discrete NVIDIA card with 13 GiB of memory or less** (`STRATA_ROUTE_PRIOR=0` turns it off;+**On by default at 1.5 on a discrete NVIDIA card with 13 GiB of memory or less** (`STRATA_ROUTE_PRIOR=0` turns it off, an explicit value wins;not a default on a Radeon, a Strix Halo or any integrated GPU, where it was not measured). The router prefers-experts that are already resident, so a window stops waiting on a page-in that the next window would need again. Measured-on the RTX 3060 12 GB (IQ3_XXS), five interleaved pairs of fresh engines:+experts that are already resident, so a window stops waiting on a page-in that the next window would need again. The first+0.2.7 candidates used a bonus of 2.0, and the table below was measured at 2.0 on the RTX 3060 12 GB (IQ3_XXS), five interleaved+pairs of fresh engines. The default was lowered to 1.5 by the owner on 26 October after the copy and code check on that card+(IQ3_XXS) found real defects in the answers at 2.0 and **no defect in 19 tasks at 1.5** (or at 1.0). 1.5 keeps most of the speed:+story **+15%** and code **+37%** against the routing off. That is 19 tasks, not a proof; a longer 61-task check of 0, 1.5 and 2.0+is still running, and if it shows defects at 1.5 the default falls back to 1.0. The 2.0 measurements:| axis | before | after ||---|---|---|@@ -94,8 +98,8 @@ document prompt), and the whole recipe - routing, a workload-fitted expert profi896` and the English draft vocabulary - gives **+26.5%** there and **+80%** on code in a sustained 80-request session.It works by choosing a slightly different set of experts, so replies change: the owner approved that trade for 12 GB cards.-**Do not raise the bonus beyond 2.0.** 3.0 measures +17.7% / +24.0% on the test prompts but **-4.5% on held-out content**,-and 4.0 breaks outright - 2.0 is the setting that generalises.+**Do not raise the bonus to 2.0 or more.** 2.0 showed defects in the copy and code check; 3.0 measures +17.7% / +24.0% on the test+prompts but **-4.5% on held-out content**, and 4.0 breaks outright. `STRATA_ROUTE_PRIOR=2` still works if you want the old value.## What was tried and did not make itdiff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hppindex 8a6932a65..d21d5b051 100644--- a/include/strata/core/sm120_series.hpp+++ b/include/strata/core/sm120_series.hpp@@ -49,10 +49,16 @@ inline bool sm120_default(int major, int minor) { return is_sm120(major, minor)// `total_bytes`: the card's memory (cudaDeviceProp::totalGlobalMem); `integrated`: a GPU that shares the PC's memory.constexpr double kSmallCardGiB = 13.0; // "a 12 GB card": 12 GB cards report 11.6 - 12.0 GiB, 16 GB ones 15.6 - 16.0-/// STRATA_ROUTE_PRIOR when the user set nothing: 2.0 (residency-biased routing, changes which experts run; owner-approved for 12 GB cards) on a-/// discrete CUDA card of 13 GiB or less, 0 everywhere else. Measured on an RTX 3060 12 GB only.+/// The residency-bias bonus a discrete CUDA card of 13 GiB or less gets when the user set nothing: THE one place the value lives. 1.5 since the+/// owner decision of 2026-10-26 (it was 2.0 in the first 0.2.7 candidates): on the RTX 3060 12 GB with IQ3_XXS the copy and code check found real+/// defects at 2.0 and none in 19 tasks at 1.5 or 1.0, and 1.5 keeps the speed (story +15%, code +37% against off). A longer 61-task check of+/// 0 / 1.5 / 2.0 is still running; if it shows defects at 1.5 this falls back to 1.0.+constexpr float kRoutePriorSmallCard = 1.5f;++/// STRATA_ROUTE_PRIOR when the user set nothing: kRoutePriorSmallCard (residency-biased routing, changes which experts run; owner-approved for 12 GB+/// cards) on a discrete CUDA card of 13 GiB or less, 0 everywhere else. Measured on an RTX 3060 12 GB only.inline float route_prior_default(bool hip, bool integrated, double total_bytes, bool v026 = false) {- return !v026 && !hip && !integrated && total_bytes > 0.0 && total_bytes / 1073741824.0 <= kSmallCardGiB ? 2.0f : 0.0f;+ return !v026 && !hip && !integrated && total_bytes > 0.0 && total_bytes / 1073741824.0 <= kSmallCardGiB ? kRoutePriorSmallCard : 0.0f;}/// STRATA_GR_V3 (the stream-split hyper-connection read, another summation order) when the user set nothing: ON for a discrete compute capabilitydiff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cppindex 51f70493b..ac80d4dd3 100644--- a/src/core/arch_defaults_test.cpp+++ b/src/core/arch_defaults_test.cpp@@ -293,8 +293,8 @@ int main() {using namespace strata;CHECK(is_026_token("026") && is_026_token("0.2.6") && is_026_token("v0.2.6"));CHECK(!is_026_token(nullptr) && !is_026_token("") && !is_026_token("0") && !is_026_token("027"));- // residency-biased routing: 0 under 026 on the card that gets 2.0 otherwise- CHECK(route_prior_default(false, false, 12.0 * GiB, false) == 2.0f);+ // residency-biased routing: 0 under 026 on the card that gets 1.5 otherwise+ CHECK(route_prior_default(false, false, 12.0 * GiB, false) == 1.5f);CHECK(route_prior_default(false, false, 12.0 * GiB, true) == 0.0f);CHECK(route_prior_default(false, false, 8.0 * GiB, true) == 0.0f);// the wide HC up read: on for a 12.0 card on CUDA, off under 026, off in HIP, off on sm_86 / sm_121@@ -340,11 +340,12 @@ int main() {{const double GiB = 1073741824.0;using namespace strata;- // residency-biased routing: 2.0 on a discrete CUDA card of 13 GiB or less, nothing else- CHECK(route_prior_default(false, false, 11.9 * GiB) == 2.0f); // RTX 3060 / 5070 12 GB- CHECK(route_prior_default(false, false, 12.0 * GiB) == 2.0f);- CHECK(route_prior_default(false, false, 8.0 * GiB) == 2.0f); // an 8 GB card- CHECK(route_prior_default(false, false, 13.0 * GiB) == 2.0f); // the boundary is inclusive+ // residency-biased routing: 1.5 on a discrete CUDA card of 13 GiB or less, nothing else+ CHECK(route_prior_default(false, false, 11.9 * GiB) == 1.5f); // RTX 3060 / 5070 12 GB+ CHECK(route_prior_default(false, false, 12.0 * GiB) == 1.5f);+ CHECK(route_prior_default(false, false, 8.0 * GiB) == 1.5f); // an 8 GB card+ CHECK(route_prior_default(false, false, 13.0 * GiB) == 1.5f); // the boundary is inclusive+ CHECK(route_prior_default(false, false, 12.0 * GiB) == strata::kRoutePriorSmallCard); // the one place the value lives (1.5 by the owner decision of 2026-10-26; 2.0 before)CHECK(route_prior_default(false, false, 13.5 * GiB) == 0.0f);CHECK(route_prior_default(false, false, 15.7 * GiB) == 0.0f); // 16 GB card: z440's A4000, P100CHECK(route_prior_default(false, false, 72.0 * GiB) == 0.0f); // RTX PRO 5000diff --git a/src/core/verify.cpp b/src/core/verify.cppindex cdb20bef1..4d6519470 100644--- a/src/core/verify.cpp+++ b/src/core/verify.cpp@@ -206,8 +206,10 @@ inline int g_side_mixer() {#endif}// STRATA_ROUTE_PRIOR=lambda: residency-biased routing (the top-10 picked again with a logit bonus for the resident-// experts; route_prior.cu). Unset: 2.0 on a DISCRETE NVIDIA card of 13 GiB or less (CUDA builds only), 0 elsewhere.-// Measured on an RTX 3060 12 GB only (IQ3_XXS, 5 interleaved pairs: +37.6% / +35.2% decode); the owner approved the+// experts; route_prior.cu). Unset: 1.5 (strata::kRoutePriorSmallCard, sm120_series.hpp; it was 2.0 until the owner decision of+// 2026-10-26: defects in the copy and code check at 2.0, none in 19 tasks at 1.5) on a DISCRETE NVIDIA card of 13 GiB or less+// (CUDA builds only), 0 elsewhere.+// Measured on an RTX 3060 12 GB only (IQ3_XXS, 5 interleaved pairs at 2.0: +37.6% / +35.2% decode; at 1.5 story +15%, code +37%); the owner approved the// quality trade for 12 GB cards (it changes which experts run). Not a default in a HIP build (a 12 GB Radeon, or a// Strix Halo whose reported VRAM is a small carve-out of unified memory: unmeasured there) and not on an integrated GPU.// STRATA_ROUTE_PRIOR=0 turns it off, any other value wins everywhere. The default is read from the device properties@@ -2002,8 +2004,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false;}// STRATA_ROUTE_PRIOR=lambda (changes the output): the top-10 picked again with a logit bonus for the resident- // experts; the weights stay the router's (route_prior.cu). The default is g_route_prior() above: 2.0 on a- // discrete CUDA card of 13 GiB or less, off elsewhere (measured on an RTX 3060 12 GB: story +37.6%, code +35.2%;+ // experts; the weights stay the router's (route_prior.cu). The default is g_route_prior() above: 1.5 on a+ // discrete CUDA card of 13 GiB or less, off elsewhere (measured on an RTX 3060 12 GB at 2.0: story +37.6%, code +35.2%;// on an RTX PRO 5000 72 GB -1.9% / -0.5%, on 2x RTX A4000 -4.2% / +11.0%, hence not a default on bigger cards).const float route_prior = g_route_prior();if (route_prior > 0.0f && NE == 512 && K == 10 && hits_.d_res != nullptr)