Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
sm_120: the hyper-connection down projection with one token a block row (bitwise)
Niko1221941041768b7d
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 831137849..ea9607b0b 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -2231,8 +2231,8 @@ The answers are the same tokens (every case and pair), and the identity gate pasdumps, with the fixed small cache and with every expert resident). What changed:- On a 12.0 card these existing switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph- branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=2` (the- hyper-connection read's up projection with its loads hoisted, and its down projection as token pairs). Each is bitwise equal+ branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=1` (the+ hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token). Each is bitwise equalto the kernel it replaces. A fixed 4-token verify round went 13.70 -> 12.23 ms with the four together.`STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and adiff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hppindex 6777cdbf3..31c031ca7 100644--- a/include/strata/core/sm120_series.hpp+++ b/include/strata/core/sm120_series.hpp@@ -7,7 +7,7 @@// STRATA_SIDE_MIXER=2 the mixer's input-only work on branches of the verify graph// STRATA_ATTN_DAILY=1 the Daily's decode attention (attn_chunk_kernel_pre75 + merge v2)// STRATA_GR_UP75=1 the hyper-connection up projection with its rows' loads hoisted-// STRATA_GR_DOWN_TG=2 the hyper-connection down projection as two token groups (twice the blocks)+// STRATA_GR_DOWN_TG=1 the hyper-connection down projection with one token a block row (T times the blocks)//// STRATA_SM120_PERF=0 turns the whole set off on a 12.0 card (the defaults every other card has).// Each switch's own variable still wins in both directions (=0 off, =1 / =2 on), on any card, as before.diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cuindex 690ea955c..51ef969a0 100644--- a/src/kernels/cuda/fused_gr.cu+++ b/src/kernels/cuda/fused_gr.cu@@ -1209,7 +1209,7 @@ bool up75_on() {return per_dev[dev] == 2;#endif}-// sm_75: the staged down kernel's tokens per block (gridDim.y = ct / tg), 0 = one block row for all of them.+// sm_75 / a 12.0 card: the staged down kernel's tokens per block (gridDim.y = ct / tg), 0 = one block row for all of them.// STRATA_GR_DOWN_TG=n forces n where it divides the launch (0 = off).int down_tg(int ct) {#if defined(__HIPCC__)@@ -1230,8 +1230,9 @@ int down_tg(int ct) {cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, dev);per_dev[dev] = strata::sm75_default(major, minor) ? 2 : strata::sm120_default(major, minor) ? 3 : 1; // Turing, STRATA_SM75_PERF=1}- // a 12.0 card (sm120_series.hpp): token pairs at every even launch (RTX PRO 5000, a 4-token round -2.4%)- if (per_dev[dev] == 3) return ct % 2 == 0 ? 2 : 0;+ // a 12.0 card (sm120_series.hpp): one token a block row for every launch of 2+ tokens (RTX PRO 5000, fixed rounds against+ // token pairs: 2 tokens -2.1%, 3 -2.3%, 4 -0.2%, 5 -2.5%; pairs against no split at 4 tokens -2.4%)+ if (per_dev[dev] == 3) return ct >= 2 ? 1 : 0;return per_dev[dev] == 2 && (ct == 4 || ct == 6) ? 2 : 0;#endif}