Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

sm_120: PF_FUSED default and per-token HC down blocks only on 80+ SM cards

Niko1221b1e0fdaa4b2f
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 9951af43c..48035e6bc 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -2233,14 +2233,16 @@ level because of the fused prompt experts (below). What changed:
- On a 12.0 card these bitwise switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph
branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=1` (the
- hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token),
+ hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token; token
+ pairs, `=2`, below 80 SMs),
`STRATA_GU_SWIGLU=1` (new: the routed experts' gate/up, SwiGLU and q8_1 in one launch). A fixed 4-token verify round went
13.70 -> 11.80 ms with them. `STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.
- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and a
5-token window cost 2 ms more than a 4-token one), with a 12.0 table for 5-8 tokens and, on 12.0 cards with 80+ SMs, its own
2-4 token table. Bitwise (`mmvq_il_parity`, 1-8 tokens). `STRATA_MMVQ_IL_WIDE=0` / `STRATA_MMVQ_IL_120=0` turn them off.
-- The native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`, the rule gfx12 has) for IQ2_S, IQ3_XXS,
- IQ3_S and IQ4_XS gate/up; IQ2_XXS / IQ2_XS keep MMQ until measured. Quality on IQ3_S: teacher-forced KL against MMQ 0.0002
+- On 12.0 cards with 80 or more SMs the native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`, the rule
+ gfx12 has) for IQ2_S, IQ3_XXS, IQ3_S and IQ4_XS gate/up; IQ2_XXS / IQ2_XS keep MMQ until measured. An RTX 5070 (48 SMs, IQ2_XS
+ pack, experts partly in RAM) was 4.7% slower on 20K prompts with them (0/6 pairs), so smaller 12.0 cards keep MMQ. Quality on IQ3_S: teacher-forced KL against MMQ 0.0002
(5998 positions; MMQ against itself 0.0000) and 0.0010 after ~6K-token documents (1760 positions); first-token KL against
the FP16 prompt path 0.01181 vs MMQ's 0.01236 (48 prompts of 1.5K-24K tokens); HumanEval 156/164, reasoning 26/30, agent
6/6, needles 6/6, CJK 8/8, tool calls 4/5, the same as MMQ. `STRATA_PF_FUSED=0` keeps MMQ.
diff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hpp
index 66b2c7940..7dad43d72 100644
--- a/include/strata/core/sm120_series.hpp
+++ b/include/strata/core/sm120_series.hpp
@@ -7,10 +7,12 @@
// STRATA_SIDE_MIXER=2 the mixer's input-only work on branches of the verify graph
// STRATA_ATTN_DAILY=1 the Daily's decode attention (attn_chunk_kernel_pre75 + merge v2)
// STRATA_GR_UP75=1 the hyper-connection up projection with its rows' loads hoisted
-// STRATA_GR_DOWN_TG=1 the hyper-connection down projection with one token a block row (T times the blocks)
+// STRATA_GR_DOWN_TG=1 the hyper-connection down projection with one token a block row (T times the blocks); =2 (token
+// pairs) on a 12.0 card with fewer than 80 SMs
// STRATA_GU_SWIGLU=1 gate/up, SwiGLU and the q8_1 blocks of the routed experts in one launch
// and one that rounds differently (KL-checked, the AMD gfx12 rule): STRATA_PF_FUSED (the native packs' prompt experts on the
-// fused int8 kernels, IQ2_S / IQ3_XXS / IQ3_S / IQ4_XS gate/up only; =0 keeps MMQ, =1 takes every covered format).
+// fused int8 kernels, IQ2_S / IQ3_XXS / IQ3_S / IQ4_XS gate/up only, 12.0 cards with 80+ SMs only; =0 keeps MMQ, =1 takes every
+// covered format on any card).
//
// STRATA_SM120_PERF=0 turns the whole set off on a 12.0 card (the defaults every other card has).
// Each switch's own variable still wins in both directions (=0 off, =1 / =2 on), on any card, as before.
diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu
index 51ef969a0..576e5d671 100644
--- a/src/kernels/cuda/fused_gr.cu
+++ b/src/kernels/cuda/fused_gr.cu
@@ -1221,18 +1221,21 @@ int down_tg(int ct) {
if (env >= 0) return env > 0 && env <= 3 && ct % env == 0 ? env : 0;
// fused_gr_bench cold on an RTX 2080 Ti (down incl. a stamp): T 4 29.1 -> 27.3 us, T 6 38.5 -> 32.2 with pairs;
// one token per block is slower at every T (the weights are read from L2 once per group)
- static int per_dev[64]; // 0 unknown, 1 no, 2 sm_75, 3 a 12.0 card
+ static int per_dev[64]; // 0 unknown, 1 no, 2 sm_75, 3 a 12.0 card with 80+ SMs, 4 a smaller 12.0 card
int dev = 0;
if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) return 0;
if (!per_dev[dev]) {
- int major = 0, minor = 0;
+ int major = 0, minor = 0, sms = 0;
cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev);
cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, dev);
- per_dev[dev] = strata::sm75_default(major, minor) ? 2 : strata::sm120_default(major, minor) ? 3 : 1; // Turing, STRATA_SM75_PERF=1
+ cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev);
+ per_dev[dev] = strata::sm75_default(major, minor) ? 2 : strata::sm120_default(major, minor) ? (sms >= 80 ? 3 : 4) : 1; // Turing, STRATA_SM75_PERF=1
}
- // a 12.0 card (sm120_series.hpp): one token a block row for every launch of 2+ tokens (RTX PRO 5000, fixed rounds against
- // token pairs: 2 tokens -2.1%, 3 -2.3%, 4 -0.2%, 5 -2.5%; pairs against no split at 4 tokens -2.4%)
+ // a 12.0 card (sm120_series.hpp) with 80+ SMs: one token a block row for every launch of 2+ tokens (RTX PRO 5000, fixed
+ // rounds against token pairs: 2 tokens -2.1%, 3 -2.3%, 4 -0.2%, 5 -2.5%; pairs against no split at 4 tokens -2.4%); a
+ // smaller one (RTX 5070, 48 SMs: 1 and 2 could not be told apart, an earlier run leaned to 2) token pairs at every even launch
if (per_dev[dev] == 3) return ct >= 2 ? 1 : 0;
+ if (per_dev[dev] == 4) return ct % 2 == 0 ? 2 : 0;
return per_dev[dev] == 2 && (ct == 4 || ct == 6) ? 2 : 0;
#endif
}
diff --git a/src/prefill/moe_fused_iq.cu b/src/prefill/moe_fused_iq.cu
index 2c1ae1cf6..4e3dc24e7 100644
--- a/src/prefill/moe_fused_iq.cu
+++ b/src/prefill/moe_fused_iq.cu
@@ -1401,7 +1401,7 @@ bool native_supported(int gu_type, int d_type) {
}();
if (off || env == 0 || !gu_covered(gu_type)) return false;
#if !defined(__HIPCC__)
- // CUDA: unset, on by default on a compute capability 12.0 card (sm120_series.hpp, STRATA_SM120_PERF=0 off) for the gate/up
+ // CUDA: unset, on by default on a compute capability 12.0 card with 80+ SMs (sm120_series.hpp, STRATA_SM120_PERF=0 off) for the gate/up
// formats measured there - the IQ3_S pack's IQ2_S, IQ3_XXS, IQ3_S and IQ4_XS layers with IQ4_NL / Q2_0 down (RTX PRO 5000:
// prompt +7.0% 4K, +5.2% 20K, +3.4% 98K; teacher-forced KL against MMQ 0.0002 over 5998 positions, 0.0010 over 1760 long
// ones; first-token KL against the FP16 prompt path 0.01181 vs MMQ's 0.01236 over 48 prompts; HumanEval, reasoning,
@@ -1410,7 +1410,9 @@ bool native_supported(int gu_type, int d_type) {
if (env < 0) {
if (gu_type != T_IQ2_S && gu_type != T_IQ3_XXS && gu_type != T_IQ3_S && gu_type != T_IQ4_XS) return false;
const DevInfo& d = dev_info();
- return d.ok && d.s120 && !d.t75 && d_covered_on(d_type, d);
+ // 12.0 cards with 80+ SMs only (RTX PRO 5000 / 6000, RTX 5080 / 5090): on an RTX 5070 (48 SMs, IQ2_XS pack, experts
+ // partly in RAM) the fused kernels were 4.7% slower on 20K prompts (0/6 pairs) - it keeps MMQ unless STRATA_PF_FUSED=1
+ return d.ok && d.s120 && d.sms >= 80 && !d.t75 && d_covered_on(d_type, d);
}
#endif
if (env < 0 && gu_type != T_IQ3_S && gu_type != T_IQ3_XXS) return false; // default: only these two