Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
sm_120: PF_FUSED default and per-token HC down blocks only on 80+ SM cards
Niko1221b1e0fdaa4b2f
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 9951af43c..48035e6bc 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -2233,14 +2233,16 @@ level because of the fused prompt experts (below). What changed:- On a 12.0 card these bitwise switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graphbranches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=1` (the- hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token),+ hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token; token+ pairs, `=2`, below 80 SMs),`STRATA_GU_SWIGLU=1` (new: the routed experts' gate/up, SwiGLU and q8_1 in one launch). A fixed 4-token verify round went13.70 -> 11.80 ms with them. `STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and a5-token window cost 2 ms more than a 4-token one), with a 12.0 table for 5-8 tokens and, on 12.0 cards with 80+ SMs, its own2-4 token table. Bitwise (`mmvq_il_parity`, 1-8 tokens). `STRATA_MMVQ_IL_WIDE=0` / `STRATA_MMVQ_IL_120=0` turn them off.-- The native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`, the rule gfx12 has) for IQ2_S, IQ3_XXS,- IQ3_S and IQ4_XS gate/up; IQ2_XXS / IQ2_XS keep MMQ until measured. Quality on IQ3_S: teacher-forced KL against MMQ 0.0002+- On 12.0 cards with 80 or more SMs the native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`, the rule+ gfx12 has) for IQ2_S, IQ3_XXS, IQ3_S and IQ4_XS gate/up; IQ2_XXS / IQ2_XS keep MMQ until measured. An RTX 5070 (48 SMs, IQ2_XS+ pack, experts partly in RAM) was 4.7% slower on 20K prompts with them (0/6 pairs), so smaller 12.0 cards keep MMQ. Quality on IQ3_S: teacher-forced KL against MMQ 0.0002(5998 positions; MMQ against itself 0.0000) and 0.0010 after ~6K-token documents (1760 positions); first-token KL againstthe FP16 prompt path 0.01181 vs MMQ's 0.01236 (48 prompts of 1.5K-24K tokens); HumanEval 156/164, reasoning 26/30, agent6/6, needles 6/6, CJK 8/8, tool calls 4/5, the same as MMQ. `STRATA_PF_FUSED=0` keeps MMQ.diff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hppindex 66b2c7940..7dad43d72 100644--- a/include/strata/core/sm120_series.hpp+++ b/include/strata/core/sm120_series.hpp@@ -7,10 +7,12 @@// STRATA_SIDE_MIXER=2 the mixer's input-only work on branches of the verify graph// STRATA_ATTN_DAILY=1 the Daily's decode attention (attn_chunk_kernel_pre75 + merge v2)// STRATA_GR_UP75=1 the hyper-connection up projection with its rows' loads hoisted-// STRATA_GR_DOWN_TG=1 the hyper-connection down projection with one token a block row (T times the blocks)+// STRATA_GR_DOWN_TG=1 the hyper-connection down projection with one token a block row (T times the blocks); =2 (token+// pairs) on a 12.0 card with fewer than 80 SMs// STRATA_GU_SWIGLU=1 gate/up, SwiGLU and the q8_1 blocks of the routed experts in one launch// and one that rounds differently (KL-checked, the AMD gfx12 rule): STRATA_PF_FUSED (the native packs' prompt experts on the-// fused int8 kernels, IQ2_S / IQ3_XXS / IQ3_S / IQ4_XS gate/up only; =0 keeps MMQ, =1 takes every covered format).+// fused int8 kernels, IQ2_S / IQ3_XXS / IQ3_S / IQ4_XS gate/up only, 12.0 cards with 80+ SMs only; =0 keeps MMQ, =1 takes every+// covered format on any card).//// STRATA_SM120_PERF=0 turns the whole set off on a 12.0 card (the defaults every other card has).// Each switch's own variable still wins in both directions (=0 off, =1 / =2 on), on any card, as before.diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cuindex 51ef969a0..576e5d671 100644--- a/src/kernels/cuda/fused_gr.cu+++ b/src/kernels/cuda/fused_gr.cu@@ -1221,18 +1221,21 @@ int down_tg(int ct) {if (env >= 0) return env > 0 && env <= 3 && ct % env == 0 ? env : 0;// fused_gr_bench cold on an RTX 2080 Ti (down incl. a stamp): T 4 29.1 -> 27.3 us, T 6 38.5 -> 32.2 with pairs;// one token per block is slower at every T (the weights are read from L2 once per group)- static int per_dev[64]; // 0 unknown, 1 no, 2 sm_75, 3 a 12.0 card+ static int per_dev[64]; // 0 unknown, 1 no, 2 sm_75, 3 a 12.0 card with 80+ SMs, 4 a smaller 12.0 cardint dev = 0;if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) return 0;if (!per_dev[dev]) {- int major = 0, minor = 0;+ int major = 0, minor = 0, sms = 0;cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev);cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, dev);- per_dev[dev] = strata::sm75_default(major, minor) ? 2 : strata::sm120_default(major, minor) ? 3 : 1; // Turing, STRATA_SM75_PERF=1+ cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev);+ per_dev[dev] = strata::sm75_default(major, minor) ? 2 : strata::sm120_default(major, minor) ? (sms >= 80 ? 3 : 4) : 1; // Turing, STRATA_SM75_PERF=1}- // a 12.0 card (sm120_series.hpp): one token a block row for every launch of 2+ tokens (RTX PRO 5000, fixed rounds against- // token pairs: 2 tokens -2.1%, 3 -2.3%, 4 -0.2%, 5 -2.5%; pairs against no split at 4 tokens -2.4%)+ // a 12.0 card (sm120_series.hpp) with 80+ SMs: one token a block row for every launch of 2+ tokens (RTX PRO 5000, fixed+ // rounds against token pairs: 2 tokens -2.1%, 3 -2.3%, 4 -0.2%, 5 -2.5%; pairs against no split at 4 tokens -2.4%); a+ // smaller one (RTX 5070, 48 SMs: 1 and 2 could not be told apart, an earlier run leaned to 2) token pairs at every even launchif (per_dev[dev] == 3) return ct >= 2 ? 1 : 0;+ if (per_dev[dev] == 4) return ct % 2 == 0 ? 2 : 0;return per_dev[dev] == 2 && (ct == 4 || ct == 6) ? 2 : 0;#endif}diff --git a/src/prefill/moe_fused_iq.cu b/src/prefill/moe_fused_iq.cuindex 2c1ae1cf6..4e3dc24e7 100644--- a/src/prefill/moe_fused_iq.cu+++ b/src/prefill/moe_fused_iq.cu@@ -1401,7 +1401,7 @@ bool native_supported(int gu_type, int d_type) {}();if (off || env == 0 || !gu_covered(gu_type)) return false;#if !defined(__HIPCC__)- // CUDA: unset, on by default on a compute capability 12.0 card (sm120_series.hpp, STRATA_SM120_PERF=0 off) for the gate/up+ // CUDA: unset, on by default on a compute capability 12.0 card with 80+ SMs (sm120_series.hpp, STRATA_SM120_PERF=0 off) for the gate/up// formats measured there - the IQ3_S pack's IQ2_S, IQ3_XXS, IQ3_S and IQ4_XS layers with IQ4_NL / Q2_0 down (RTX PRO 5000:// prompt +7.0% 4K, +5.2% 20K, +3.4% 98K; teacher-forced KL against MMQ 0.0002 over 5998 positions, 0.0010 over 1760 long// ones; first-token KL against the FP16 prompt path 0.01181 vs MMQ's 0.01236 over 48 prompts; HumanEval, reasoning,@@ -1410,7 +1410,9 @@ bool native_supported(int gu_type, int d_type) {if (env < 0) {if (gu_type != T_IQ2_S && gu_type != T_IQ3_XXS && gu_type != T_IQ3_S && gu_type != T_IQ4_XS) return false;const DevInfo& d = dev_info();- return d.ok && d.s120 && !d.t75 && d_covered_on(d_type, d);+ // 12.0 cards with 80+ SMs only (RTX PRO 5000 / 6000, RTX 5080 / 5090): on an RTX 5070 (48 SMs, IQ2_XS pack, experts+ // partly in RAM) the fused kernels were 4.7% slower on 20K prompts (0/6 pairs) - it keeps MMQ unless STRATA_PF_FUSED=1+ return d.ok && d.s120 && d.sms >= 80 && !d.t75 && d_covered_on(d_type, d);}#endifif (env < 0 && gu_type != T_IQ3_S && gu_type != T_IQ3_XXS) return false; // default: only these two