Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Residency-biased routing default is CUDA-discrete-only: read from the device properties once in Verifier::init (never first inside a graph capture), off in HIP builds and on integrated GPUs, a line in the log when it is on; the wide HC up read never defaults on in a HIP build (gfx12 reports compute capability 12.0 too)

Niko12217c4c20799c2c
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 43538ddca..afd8290a7 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -899,7 +899,7 @@ them on your own card.
| `STRATA_PREFILL_EQUAL=1` | CUDA | A prompt segment is read in chunks of equal size, not full chunks and a short last one. Changes rounding. | |
| `STRATA_OWNED_PRICE=exact` | CUDA | The cache sizing prices the prompt path's own buffers by their real allocation. | |
| `STRATA_IQ_MT_MIN=1`, `STRATA_IQ3S_MT1=1` | CPU | See [Draft layer](#draft-layer) (reproducible output; the second gives IQ3_S the multi-token kernel for one token on Alder Lake and later without AVX-512). | |
-| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended. |
+| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended, except that 0.2.7 turns `STRATA_ROUTE_PRIOR=2.0` on by itself on a discrete NVIDIA card of 13 GiB or less (the owner approved the quality trade for 12 GB cards); `STRATA_ROUTE_PRIOR=0` turns it off. It is never a default on a Radeon, an integrated GPU or a bigger card. |
| `STRATA_HOST_CORE=last\|sibling` (or `--host-core`) | Windows | `last` puts the host thread on the last physical core, away from the core that gets the GPU's interrupts. `sibling` keeps it on the first core's other hardware thread, so the workers keep every core. The default `first` leaves a hybrid CPU as it is. | One layer-split box went from a ±6% swing to steady and 8% faster. |
| `--pool-affinity measure` | Hybrid CPUs, native packs | Times the `all` and `p-cores` layouts at start (about a second) and takes the faster. The measurement is taken once; where the E-cores' speed changes while the engine runs, `--pool-affinity p-cores` is the steady choice. | |
| `STRATA_KEEP_EMPTY_TURNS=1`, `STRATA_TOPK_STREAM=0` | | See [TROUBLESHOOTING.md](TROUBLESHOOTING.md). | |
@@ -1078,7 +1078,7 @@ On 12.0 cards (sm_120) these also apply:
prompt time and start-up time, which is why smaller PCs keep the SSD reads.
A prompt chunk above 8192 (`--prefill auto:16384` or `auto:32768`) is not bit-exact on CUDA, so the default stays 8192 on
-NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR=0.5` stays opt-in because it raises KL.
+NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR` raises KL, so below 0.2.7 it was opt-in; 0.2.7 sets 2.0 by itself only on a discrete NVIDIA card of 13 GiB or less.
---
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.md
index 8c10493d3..8166d70db 100644
--- a/docs/release-notes-0.2.7.md
+++ b/docs/release-notes-0.2.7.md
@@ -42,7 +42,8 @@ These need no engine change; they are settings. Each was measured on the same bo
## The 12 GB default: residency-biased routing
-**On by default at 2.0 on a card with 13 GiB of memory or less** (`STRATA_ROUTE_PRIOR=0` turns it off). The router prefers
+**On by default at 2.0 on a discrete NVIDIA card with 13 GiB of memory or less** (`STRATA_ROUTE_PRIOR=0` turns it off;
+not a default on a Radeon, a Strix Halo or any integrated GPU, where it was not measured). The router prefers
experts that are already resident, so a window stops waiting on a page-in that the next window would need again. Measured
on the RTX 3060 12 GB (IQ3_XXS), five interleaved pairs of fresh engines:
diff --git a/src/core/verify.cpp b/src/core/verify.cpp
index 6c7d1a1b0..b9d4eeded 100644
--- a/src/core/verify.cpp
+++ b/src/core/verify.cpp
@@ -204,6 +204,32 @@ inline int g_side_mixer() {
return per_dev[dev] - 1;
#endif
}
+// STRATA_ROUTE_PRIOR=lambda: residency-biased routing (the top-10 picked again with a logit bonus for the resident
+// experts; route_prior.cu). Unset: 2.0 on a DISCRETE NVIDIA card of 13 GiB or less (CUDA builds only), 0 elsewhere.
+// Measured on an RTX 3060 12 GB only (IQ3_XXS, 5 interleaved pairs: +37.6% / +35.2% decode); the owner approved the
+// quality trade for 12 GB cards (it changes which experts run). Not a default in a HIP build (a 12 GB Radeon, or a
+// Strix Halo whose reported VRAM is a small carve-out of unified memory: unmeasured there) and not on an integrated GPU.
+// STRATA_ROUTE_PRIOR=0 turns it off, any other value wins everywhere. The default is read from the device properties
+// (no allocation, no stream work), and Verifier::init asks once per card, so the first ask is never inside a graph capture.
+inline float g_route_prior() {
+ static const char* const env_s = std::getenv("STRATA_ROUTE_PRIOR");
+ static const float env = env_s ? (float) std::atof(env_s) : 0.0f;
+ if (env_s != nullptr) return env;
+#if defined(STRATA_USE_HIP) || defined(STRATA_GPU_METAL)
+ return 0.0f;
+#else
+ static float per_dev[64]; // 0 unknown, else prior + 1
+ int dev = 0;
+ if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) { cudaGetLastError(); return 0.0f; }
+ if (per_dev[dev] == 0.0f) {
+ cudaDeviceProp p;
+ const bool ok = cudaGetDeviceProperties(&p, dev) == cudaSuccess;
+ if (!ok) cudaGetLastError();
+ per_dev[dev] = 1.0f + (ok && !p.integrated && (double) p.totalGlobalMem / 1073741824.0 <= 13.0 ? 2.0f : 0.0f);
+ }
+ return per_dev[dev] - 1.0f;
+#endif
+}
inline bool g_qfuse() { static const bool on = [] { const char* v = std::getenv("STRATA_QFUSE"); return v != nullptr && std::atoi(v) != 0; }(); return on; }
#define VDBG(...) do { if (g_dbg) { std::fprintf(stderr, "verify dbg: " __VA_ARGS__); std::fflush(stderr); } } while (0)
@@ -703,6 +729,12 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState&
release_gpu_fn().store(&release_live_verifiers);
cudaGetDevice(&device_); // a layer split's stage on another GPU: its streams, graphs and buffers live there
rr_stats_ = route_resident_cfg().stats_on(device_); // STRATA_ROUTE_RESIDENT: this GPU's counters, outside graph capture
+ { // once per card, here: record_window (graph capture) only reads the cached value
+ const float rp = g_route_prior();
+ if (rp > 0.0f && std::getenv("STRATA_ROUTE_PRIOR") == nullptr)
+ std::fprintf(stderr, "strata verify: residency-biased routing is on by default on this card (STRATA_ROUTE_PRIOR=%.1f: a card of 13 GiB "
+ "or less; it changes which experts run; STRATA_ROUTE_PRIOR=0 turns it off)\n", (double) rp);
+ }
strata::kernels::fused_gr_check(); // once per card: which bitwise-equal hyper-connection read runs there
strata::kernels::fused_gr_persist_init(); // opt-in STRATA_HC_PERSIST=1 (HIP)
wt_ = &wt;
@@ -1966,23 +1998,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
mb.logits = logits_ + t * NE; mb.ids = ids_ + t * K; mb.weights = w_ + t * K;
if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false;
}
- // STRATA_ROUTE_PRIOR=lambda: the top-10 picked again with a logit bonus for the resident experts; the weights
- // stay the router's (route_prior.cu). **On by default at 2.0 on a card of 13 GiB or less**, where the expert
- // cache cannot hold the model: measured on an RTX 3060 12 GB (IQ3_XXS), 5 interleaved pairs, story 44.2 ->
- // 60.9 tok/s (+37.6%) and code 47.3 -> 64.0 (+35.2%); at 1.0 it is +22.6% / +24.6%. The owner approved the
- // quality trade for 12 GB cards (it changes which experts run). It is NOT a default where the cache holds
- // the model: on an RTX PRO 5000 72 GB it measured -1.9% / -0.5% and on 2x RTX A4000 -4.2% / +11.0%, so
- // STRATA_ROUTE_PRIOR=0 turns it off and any explicit value wins.
- static const float route_prior = [] {
- const char* v = std::getenv("STRATA_ROUTE_PRIOR");
- if (v != nullptr) return (float) std::atof(v);
- int dev = 0;
- if (cudaGetDevice(&dev) != cudaSuccess) return 0.0f;
- size_t free_b = 0, total_b = 0;
- if (cudaMemGetInfo(&free_b, &total_b) != cudaSuccess) return 0.0f;
- cudaGetLastError(); // drop what a query on a device without a context leaves behind
- return (double) total_b / 1073741824.0 <= 13.0 ? 2.0f : 0.0f;
- }();
+ // STRATA_ROUTE_PRIOR=lambda (changes the output): the top-10 picked again with a logit bonus for the resident
+ // experts; the weights stay the router's (route_prior.cu). The default is g_route_prior() above: 2.0 on a
+ // discrete CUDA card of 13 GiB or less, off elsewhere (measured on an RTX 3060 12 GB: story +37.6%, code +35.2%;
+ // on an RTX PRO 5000 72 GB -1.9% / -0.5%, on 2x RTX A4000 -4.2% / +11.0%, hence not a default on bigger cards).
+ const float route_prior = g_route_prior();
if (route_prior > 0.0f && NE == 512 && K == 10 && hits_.d_res != nullptr)
route_prior_top10(logits_ + tb * NE, hits_.d_res + l * g.n_expert, route_prior, ids_ + tb * K, w_ + tb * K, n, cs);
if (route_resident_cfg().margin > 0.0f && NE == 512 && K == 10 && hits_.d_res != nullptr) {
diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu
index a70b7abe6..73a9089b7 100644
--- a/src/kernels/cuda/fused_gr.cu
+++ b/src/kernels/cuda/fused_gr.cu
@@ -444,6 +444,11 @@ constexpr int UPW_BLOCKS = N / UPW_COLS;
/// resident, 5 interleaved pairs of fresh engines): story 197.6 -> 201.3 tok/s (+1.9%), code 243.5 -> 245.1 (+0.7%),
/// the produced token ids identical in 6/6 runs; the isolated read (fused_gr_bench) is bitwise the old kernel's.
bool gr_up_wide_on() {
+#if defined(__HIPCC__)
+ // A HIP build never takes it: gfx1200 / gfx1201 report compute capability 12.0 too, and sm120_default() would turn the
+ // CUDA-measured kernel on there (unmeasured on AMD). The persistent HC read is the HIP card's own opt-in.
+ return false;
+#else
static const int env = [] {
const char* v = std::getenv("STRATA_GR_UP_WIDE");
return v != nullptr ? (std::atoi(v) != 0 ? 1 : 0) : -1;
@@ -460,6 +465,7 @@ bool gr_up_wide_on() {
cudaGetLastError(); // drop what an attribute query on a CPU-only box leaves behind
}
return per_dev[dev] == 2;
+#endif
}
template <int MAX_T = kFusedGrMaxT, bool EXACT_T = false>