Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Residency-biased routing default is CUDA-discrete-only: read from the device properties once in Verifier::init (never first inside a graph capture), off in HIP builds and on integrated GPUs, a line in the log when it is on; the wide HC up read never defaults on in a HIP build (gfx12 reports compute capability 12.0 too)
Niko12217c4c20799c2c
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 43538ddca..afd8290a7 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -899,7 +899,7 @@ them on your own card.| `STRATA_PREFILL_EQUAL=1` | CUDA | A prompt segment is read in chunks of equal size, not full chunks and a short last one. Changes rounding. | || `STRATA_OWNED_PRICE=exact` | CUDA | The cache sizing prices the prompt path's own buffers by their real allocation. | || `STRATA_IQ_MT_MIN=1`, `STRATA_IQ3S_MT1=1` | CPU | See [Draft layer](#draft-layer) (reproducible output; the second gives IQ3_S the multi-token kernel for one token on Alder Lake and later without AVX-512). | |-| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended. |+| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended, except that 0.2.7 turns `STRATA_ROUTE_PRIOR=2.0` on by itself on a discrete NVIDIA card of 13 GiB or less (the owner approved the quality trade for 12 GB cards); `STRATA_ROUTE_PRIOR=0` turns it off. It is never a default on a Radeon, an integrated GPU or a bigger card. || `STRATA_HOST_CORE=last\|sibling` (or `--host-core`) | Windows | `last` puts the host thread on the last physical core, away from the core that gets the GPU's interrupts. `sibling` keeps it on the first core's other hardware thread, so the workers keep every core. The default `first` leaves a hybrid CPU as it is. | One layer-split box went from a ±6% swing to steady and 8% faster. || `--pool-affinity measure` | Hybrid CPUs, native packs | Times the `all` and `p-cores` layouts at start (about a second) and takes the faster. The measurement is taken once; where the E-cores' speed changes while the engine runs, `--pool-affinity p-cores` is the steady choice. | || `STRATA_KEEP_EMPTY_TURNS=1`, `STRATA_TOPK_STREAM=0` | | See [TROUBLESHOOTING.md](TROUBLESHOOTING.md). | |@@ -1078,7 +1078,7 @@ On 12.0 cards (sm_120) these also apply:prompt time and start-up time, which is why smaller PCs keep the SSD reads.A prompt chunk above 8192 (`--prefill auto:16384` or `auto:32768`) is not bit-exact on CUDA, so the default stays 8192 on-NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR=0.5` stays opt-in because it raises KL.+NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR` raises KL, so below 0.2.7 it was opt-in; 0.2.7 sets 2.0 by itself only on a discrete NVIDIA card of 13 GiB or less.---diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex 8c10493d3..8166d70db 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -42,7 +42,8 @@ These need no engine change; they are settings. Each was measured on the same bo## The 12 GB default: residency-biased routing-**On by default at 2.0 on a card with 13 GiB of memory or less** (`STRATA_ROUTE_PRIOR=0` turns it off). The router prefers+**On by default at 2.0 on a discrete NVIDIA card with 13 GiB of memory or less** (`STRATA_ROUTE_PRIOR=0` turns it off;+not a default on a Radeon, a Strix Halo or any integrated GPU, where it was not measured). The router prefersexperts that are already resident, so a window stops waiting on a page-in that the next window would need again. Measuredon the RTX 3060 12 GB (IQ3_XXS), five interleaved pairs of fresh engines:diff --git a/src/core/verify.cpp b/src/core/verify.cppindex 6c7d1a1b0..b9d4eeded 100644--- a/src/core/verify.cpp+++ b/src/core/verify.cpp@@ -204,6 +204,32 @@ inline int g_side_mixer() {return per_dev[dev] - 1;#endif}+// STRATA_ROUTE_PRIOR=lambda: residency-biased routing (the top-10 picked again with a logit bonus for the resident+// experts; route_prior.cu). Unset: 2.0 on a DISCRETE NVIDIA card of 13 GiB or less (CUDA builds only), 0 elsewhere.+// Measured on an RTX 3060 12 GB only (IQ3_XXS, 5 interleaved pairs: +37.6% / +35.2% decode); the owner approved the+// quality trade for 12 GB cards (it changes which experts run). Not a default in a HIP build (a 12 GB Radeon, or a+// Strix Halo whose reported VRAM is a small carve-out of unified memory: unmeasured there) and not on an integrated GPU.+// STRATA_ROUTE_PRIOR=0 turns it off, any other value wins everywhere. The default is read from the device properties+// (no allocation, no stream work), and Verifier::init asks once per card, so the first ask is never inside a graph capture.+inline float g_route_prior() {+ static const char* const env_s = std::getenv("STRATA_ROUTE_PRIOR");+ static const float env = env_s ? (float) std::atof(env_s) : 0.0f;+ if (env_s != nullptr) return env;+#if defined(STRATA_USE_HIP) || defined(STRATA_GPU_METAL)+ return 0.0f;+#else+ static float per_dev[64]; // 0 unknown, else prior + 1+ int dev = 0;+ if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) { cudaGetLastError(); return 0.0f; }+ if (per_dev[dev] == 0.0f) {+ cudaDeviceProp p;+ const bool ok = cudaGetDeviceProperties(&p, dev) == cudaSuccess;+ if (!ok) cudaGetLastError();+ per_dev[dev] = 1.0f + (ok && !p.integrated && (double) p.totalGlobalMem / 1073741824.0 <= 13.0 ? 2.0f : 0.0f);+ }+ return per_dev[dev] - 1.0f;+#endif+}inline bool g_qfuse() { static const bool on = [] { const char* v = std::getenv("STRATA_QFUSE"); return v != nullptr && std::atoi(v) != 0; }(); return on; }#define VDBG(...) do { if (g_dbg) { std::fprintf(stderr, "verify dbg: " __VA_ARGS__); std::fflush(stderr); } } while (0)@@ -703,6 +729,12 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState&release_gpu_fn().store(&release_live_verifiers);cudaGetDevice(&device_); // a layer split's stage on another GPU: its streams, graphs and buffers live thererr_stats_ = route_resident_cfg().stats_on(device_); // STRATA_ROUTE_RESIDENT: this GPU's counters, outside graph capture+ { // once per card, here: record_window (graph capture) only reads the cached value+ const float rp = g_route_prior();+ if (rp > 0.0f && std::getenv("STRATA_ROUTE_PRIOR") == nullptr)+ std::fprintf(stderr, "strata verify: residency-biased routing is on by default on this card (STRATA_ROUTE_PRIOR=%.1f: a card of 13 GiB "+ "or less; it changes which experts run; STRATA_ROUTE_PRIOR=0 turns it off)\n", (double) rp);+ }strata::kernels::fused_gr_check(); // once per card: which bitwise-equal hyper-connection read runs therestrata::kernels::fused_gr_persist_init(); // opt-in STRATA_HC_PERSIST=1 (HIP)wt_ = &wt;@@ -1966,23 +1998,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {mb.logits = logits_ + t * NE; mb.ids = ids_ + t * K; mb.weights = w_ + t * K;if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false;}- // STRATA_ROUTE_PRIOR=lambda: the top-10 picked again with a logit bonus for the resident experts; the weights- // stay the router's (route_prior.cu). **On by default at 2.0 on a card of 13 GiB or less**, where the expert- // cache cannot hold the model: measured on an RTX 3060 12 GB (IQ3_XXS), 5 interleaved pairs, story 44.2 ->- // 60.9 tok/s (+37.6%) and code 47.3 -> 64.0 (+35.2%); at 1.0 it is +22.6% / +24.6%. The owner approved the- // quality trade for 12 GB cards (it changes which experts run). It is NOT a default where the cache holds- // the model: on an RTX PRO 5000 72 GB it measured -1.9% / -0.5% and on 2x RTX A4000 -4.2% / +11.0%, so- // STRATA_ROUTE_PRIOR=0 turns it off and any explicit value wins.- static const float route_prior = [] {- const char* v = std::getenv("STRATA_ROUTE_PRIOR");- if (v != nullptr) return (float) std::atof(v);- int dev = 0;- if (cudaGetDevice(&dev) != cudaSuccess) return 0.0f;- size_t free_b = 0, total_b = 0;- if (cudaMemGetInfo(&free_b, &total_b) != cudaSuccess) return 0.0f;- cudaGetLastError(); // drop what a query on a device without a context leaves behind- return (double) total_b / 1073741824.0 <= 13.0 ? 2.0f : 0.0f;- }();+ // STRATA_ROUTE_PRIOR=lambda (changes the output): the top-10 picked again with a logit bonus for the resident+ // experts; the weights stay the router's (route_prior.cu). The default is g_route_prior() above: 2.0 on a+ // discrete CUDA card of 13 GiB or less, off elsewhere (measured on an RTX 3060 12 GB: story +37.6%, code +35.2%;+ // on an RTX PRO 5000 72 GB -1.9% / -0.5%, on 2x RTX A4000 -4.2% / +11.0%, hence not a default on bigger cards).+ const float route_prior = g_route_prior();if (route_prior > 0.0f && NE == 512 && K == 10 && hits_.d_res != nullptr)route_prior_top10(logits_ + tb * NE, hits_.d_res + l * g.n_expert, route_prior, ids_ + tb * K, w_ + tb * K, n, cs);if (route_resident_cfg().margin > 0.0f && NE == 512 && K == 10 && hits_.d_res != nullptr) {diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cuindex a70b7abe6..73a9089b7 100644--- a/src/kernels/cuda/fused_gr.cu+++ b/src/kernels/cuda/fused_gr.cu@@ -444,6 +444,11 @@ constexpr int UPW_BLOCKS = N / UPW_COLS;/// resident, 5 interleaved pairs of fresh engines): story 197.6 -> 201.3 tok/s (+1.9%), code 243.5 -> 245.1 (+0.7%),/// the produced token ids identical in 6/6 runs; the isolated read (fused_gr_bench) is bitwise the old kernel's.bool gr_up_wide_on() {+#if defined(__HIPCC__)+ // A HIP build never takes it: gfx1200 / gfx1201 report compute capability 12.0 too, and sm120_default() would turn the+ // CUDA-measured kernel on there (unmeasured on AMD). The persistent HC read is the HIP card's own opt-in.+ return false;+#elsestatic const int env = [] {const char* v = std::getenv("STRATA_GR_UP_WIDE");return v != nullptr ? (std::atoi(v) != 0 ? 1 : 0) : -1;@@ -460,6 +465,7 @@ bool gr_up_wide_on() {cudaGetLastError(); // drop what an attribute query on a CPU-only box leaves behind}return per_dev[dev] == 2;+#endif}template <int MAX_T = kFusedGrMaxT, bool EXACT_T = false>