Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

mmvq_il: the sm_89 (Ada) decode tables are on by default on compute capability 8.9 only, STRATA_MMVQ_IL_89=0 and STRATA_DEFAULTS=026 turn them off (#3453)

Niko1221663ec80abf9e
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index cb863f48d..082ad0e6d 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -1186,6 +1186,8 @@ On 12.0 cards (sm_120) these also apply:
- The interleaved small-batch matvec (`native_mmvq_il`, [MMVQ_IL_TABLE.md](MMVQ_IL_TABLE.md)) runs 5 to 8 tokens, with a 12.0 table
and, on cards with 80 or more SMs, its own 2-4 token table. `STRATA_MMVQ_IL_WIDE=0` and `STRATA_MMVQ_IL_120=0` turn them off.
+ RTX 40 cards (compute capability 8.9) have their own 2-4 token tables and a one-token Q3_K row, on by default in 0.2.7 and not yet measured
+ on an RTX 40 card by us (same output bit for bit): `STRATA_MMVQ_IL_89=0` or `STRATA_DEFAULTS=026` turns them off.
- With 80 or more SMs the native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`) for IQ2_S, IQ3_XXS,
IQ3_S and IQ4_XS gate/up. IQ2_XXS and IQ2_XS keep MMQ. Smaller 12.0 cards (an RTX 5070 with 48 SMs) keep MMQ because the fused
kernels were 4.7% slower on 20K prompts there. `STRATA_PF_FUSED=0` keeps MMQ.
diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.md
index 8974c3be9..26c637b19 100644
--- a/docs/MMVQ_IL_TABLE.md
+++ b/docs/MMVQ_IL_TABLE.md
@@ -15,7 +15,7 @@ Which table a card gets:
|---|---|
| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |
| sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |
-| sm_89 (Ada) | the shared table (`kIlRowsShared`) unless `STRATA_MMVQ_IL_89=1`, which switches on `kIlRows_89`: the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. It is opt-in because no sm_89 card has run `mmvq_il_parity` on this code, and the issue saw no end-to-end decode change on a 4090 or a 4080 (issue 3453), so the gain is a microbenchmark one. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card. To flip the default, run `mmvq_il_parity` on an Ada card (it must end with `OK`) and an interleaved decode A/B with the switch on and off. |
+| sm_89 (Ada) | `kIlRows_89`, ON by default on compute capability 8.9 cards only (0.2.7): the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. `STRATA_MMVQ_IL_89=0` or `STRATA_DEFAULTS=026` gives the shared table (`kIlRowsShared`) again, as in 0.2.6; an explicit `STRATA_MMVQ_IL_89=1` wins over the master switch. The output is bitwise the same either way (a table only changes how many rows a warp takes). Not measured on an Ada card by us: no sm_89 card has run `mmvq_il_parity` on this code, and the issue saw no end-to-end decode change on a 4090 or a 4080 (issue 3453), so the gain may be a microbenchmark one. The reporter tests 0.2.7 and reports back. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card. To check a card: run `mmvq_il_parity` on it (it must end with `OK`) and an interleaved decode A/B with the variable at 1 and 0. |
| every other sm_80+ card | `kIlRowsShared` |
| sm_75 and older | the path is off; the plain kernels run |
| a card listed in `kIlArch` | its own table |
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.md
index 6e806adef..6a7416008 100644
--- a/docs/release-notes-0.2.7.md
+++ b/docs/release-notes-0.2.7.md
@@ -23,6 +23,10 @@
- The hyper-connection read split over two kernels (a discrete RTX 50 / RTX PRO Blackwell card with more than 13 GiB): **+1.2%
decode on the RTX PRO 5000 with identical tokens**, about +1.5% together with the switch above. On 2x RTX A4000 and the
RTX 3060 it was slower, so those cards keep the old read. Off switch: `STRATA_GR_V3=0`.
+ - RTX 40 cards (Ada, compute capability 8.9): new decode row tables for the interleaved matrix kernels, on by default. The output is the
+ same bit for bit; only how many rows a warp takes changes. **We have not measured them on an RTX 40 card**: the tables come from two
+ community runs (an RTX 4070 and an RTX 4080), and the report with the 4090 saw no change in decode speed. If you have an RTX 40
+ card, please try 0.2.7 and post your decode numbers on issue 3453. Off switch: `STRATA_MMVQ_IL_89=0`.
## Going back to 0.2.6
@@ -34,7 +38,7 @@ Two switches bring back every default that 0.2.7 changed, in one step, for a tes
the RAM store's spare slots and put and its settle after a prompt, the dense dequantizer, the F16 gate/up operand, the drafter's
markov head and the head's column count, and the RDNA3 dense GEMM all go back to what 0.2.6 did.
- **`STRATA_DEFAULTS=026`** (Qwen): the residency-biased routing, the wide and the stream-split hyper-connection reads, the int8
- prompt attention, the four gfx12 decode switches, the PLE residency policy and the gfx12 decode split of 0.15 (back to 0.55).
+ prompt attention, the RTX 40 (sm_89) decode tables, the four gfx12 decode switches, the PLE residency policy and the gfx12 decode split of 0.15 (back to 0.55).
Long prompts on the approximate read (`STRATA_KVA`) are off by default and stay as `STRATA_KVA` says.
A variable or flag you set yourself (`DS4_ATTN_SLAB=N`, `STRATA_ROUTE_PRIOR=2`, `--experts 5`, `--prefill-chunk N`, `--pcie-frac`,
diff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hpp
index 8a6932a65..2674b28c4 100644
--- a/include/strata/core/sm120_series.hpp
+++ b/include/strata/core/sm120_series.hpp
@@ -67,6 +67,19 @@ inline bool gr_v3_default(bool hip, int major, int minor, bool integrated, doubl
/// resolves it (an emulated or forced Turing counts as 75). Measured on sm_120 (RTX PRO 5000) and sm_86 (2x RTX A4000): +1.1 .. +1.8% prompt.
inline bool prompt_attn_imma_default(bool hip, int cc, bool v026 = false) { return !v026 && !hip && cc >= 80; }
+/// STRATA_MMVQ_IL_89 (#3453: the decode kernel row tables of an Ada card, the cells where an RTX 4070 and an RTX 4080 agreed; bitwise the same
+/// output, only the rows-a-warp split differs) when the user set nothing: ON for compute capability 8.9 only (owner decision, 26 October: not yet
+/// measured on an Ada card by us), off under STRATA_DEFAULTS=026 and in HIP builds. `cc` = 10 * major + minor as the engine resolves it.
+inline bool mmvq_il_89_default(bool hip, int cc, bool v026 = false) { return !v026 && !hip && cc == 89; }
+
+/// The answer for STRATA_MMVQ_IL_89 given the variable's text (nullptr = unset): "1" on and "0" off win over everything, anything else (unset,
+/// empty, other text) is the rule above.
+inline bool mmvq_il_89_resolve(const char* env, bool hip, int cc, bool v026 = false) {
+ if (env != nullptr && env[0] == '1') return true;
+ if (env != nullptr && env[0] == '0') return false;
+ return mmvq_il_89_default(hip, cc, v026);
+}
+
/// STRATA_GR_UP_WIDE (the hyper-connection up projection, one warp a row; bitwise the old kernel) when the user set nothing: ON for a compute
/// capability 12.0 card (STRATA_SM120_PERF=0 turns the set off), CUDA builds only.
inline bool gr_up_wide_default(bool hip, int major, int minor, bool v026 = false) { return !v026 && !hip && sm120_default(major, minor); }
diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cpp
index db193a7f7..3195bbba7 100644
--- a/src/core/arch_defaults_test.cpp
+++ b/src/core/arch_defaults_test.cpp
@@ -243,6 +243,20 @@ int main() {
CHECK(!gr_up_wide_default(true, 12, 0));
CHECK(!gr_up_wide_default(false, 8, 6));
CHECK(!gr_up_wide_default(false, 12, 1));
+ // #3453: the sm_89 MMVQ-IL tables: on by default for compute capability 8.9 only, off under 026 and in HIP; "1" and "0" win
+ CHECK(mmvq_il_89_default(false, 89));
+ CHECK(!mmvq_il_89_default(false, 86) && !mmvq_il_89_default(false, 80) && !mmvq_il_89_default(false, 90) &&
+ !mmvq_il_89_default(false, 120) && !mmvq_il_89_default(false, 75) && !mmvq_il_89_default(false, 70) &&
+ !mmvq_il_89_default(false, 60) && !mmvq_il_89_default(false, 0));
+ CHECK(!mmvq_il_89_default(true, 89));
+ CHECK(!mmvq_il_89_default(false, 89, true)); // STRATA_DEFAULTS=026
+ CHECK(mmvq_il_89_resolve(nullptr, false, 89, false)); // unset: the rule
+ CHECK(!mmvq_il_89_resolve(nullptr, false, 89, true)); // unset + 026: off
+ CHECK(!mmvq_il_89_resolve(nullptr, false, 86, false)); // another card: off
+ CHECK(!mmvq_il_89_resolve("0", false, 89, false)); // STRATA_MMVQ_IL_89=0 turns it off
+ CHECK(mmvq_il_89_resolve("1", false, 89, true)); // an explicit =1 wins over the master switch
+ CHECK(mmvq_il_89_resolve("1", false, 86, false)); // (the tables are only looked up for cc 89 anyway)
+ CHECK(mmvq_il_89_resolve("", false, 89, false) && !mmvq_il_89_resolve("", false, 89, true)); // empty text: the rule
// GR_V3 and IMMA
CHECK(gr_v3_default(false, 12, 0, false, 72.0 * GiB, false));
CHECK(!gr_v3_default(false, 12, 0, false, 72.0 * GiB, true));
diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu
index f0c3913ef..ef6758826 100644
--- a/src/kernels/cuda/native_mmvq.cu
+++ b/src/kernels/cuda/native_mmvq.cu
@@ -29,6 +29,8 @@
#include "strata/kernels/iq_kernels.hpp"
#include "strata/kernels/gfx_arch.hpp"
#include "strata/core/sm75_series.hpp"
+#include "strata/core/sm120_series.hpp"
+#include "strata/core/defaults_026.hpp"
#include "s26_tsum.cuh"
#if !defined(__HIPCC__)
#include "q8_1_il.cuh"
@@ -3158,10 +3160,14 @@ bool il_env_on(const char* name) {
}
bool il_t1_on() { static const bool on = il_env_on("STRATA_MMVQ_IL_T1"); return on; }
bool il_q3k_on() { static const bool on = il_env_on("STRATA_MMVQ_IL_Q3K"); return on; }
-// #3453: sm_89's own tables (kIlRows_89 / kIl1Rows_89) are opt-in with STRATA_MMVQ_IL_89=1 until an sm_89 card has run
-// mmvq_il_parity and an end-to-end A/B (docs/MMVQ_IL_TABLE.md). Unset, an Ada card answers what it answered before: the
-// shared table, no one-column row. Cached once; the lookup runs on the host once a launch.
-bool il_89_opt_in() { static const bool on = [] { const char* v = std::getenv("STRATA_MMVQ_IL_89"); return v != nullptr && v[0] == '1'; }(); return on; }
+// #3453: sm_89's own tables (kIlRows_89 / kIl1Rows_89) are ON by default on compute capability 8.9 only (owner decision, 0.2.7; no sm_89 card
+// has run mmvq_il_parity or an end-to-end A/B for us, the reporter tests and reports back). STRATA_MMVQ_IL_89=0 gives an Ada card what it
+// answered before (the shared table, no one-column row) and so does STRATA_DEFAULTS=026; STRATA_MMVQ_IL_89=1 wins over that master switch
+// (sm120_series.hpp mmvq_il_89_resolve). Cached once; the lookup runs on the host once a launch.
+bool il_89_opt_in() {
+ static const bool on = strata::mmvq_il_89_resolve(std::getenv("STRATA_MMVQ_IL_89"), false, 89, strata::qwen_defaults_026());
+ return on;
+}
struct IlArch { int cc; const IlRows* t; size_t n; };
constexpr IlArch kIlArch[] = {
{60, kIlRows_60, sizeof(kIlRows_60) / sizeof(IlRows)}, // Pascal GP100: measured on a Tesla P100
diff --git a/src/kernels/mmvq_il_parity.cpp b/src/kernels/mmvq_il_parity.cpp
index 9e9cd1654..00ad704b8 100644
--- a/src/kernels/mmvq_il_parity.cpp
+++ b/src/kernels/mmvq_il_parity.cpp
@@ -10,6 +10,8 @@
// the block scales rewritten as normal fp16 values, so every output is finite (a NaN would compare equal whatever
// produced it). The bench cycles through enough copies of the weights to miss the L2 cache, as a layer's call does.
#include "strata/kernels/iq_kernels.hpp"
+#include "strata/core/defaults_026.hpp"
+#include "strata/core/sm120_series.hpp"
#include "strata/kernels/native_mmvq.hpp"
#include <cuda_runtime.h>
@@ -273,7 +275,8 @@ const int kAda4080[5][3][5] = {
int check_tables() {
int bad = 0;
const char* e89 = std::getenv("STRATA_MMVQ_IL_89");
- const bool opt89 = e89 != nullptr && e89[0] == '1'; // sm_89's own tables are opt-in (#3453)
+ // sm_89's own tables are ON by default (#3453, 0.2.7) unless STRATA_MMVQ_IL_89=0 or STRATA_DEFAULTS=026
+ const bool opt89 = strata::mmvq_il_89_resolve(e89, false, 89, strata::qwen_defaults_026());
for (int type : {23, 12, 13, 14, 11})
for (int nc = 2; nc <= 4; ++nc)
for (int n_out : {512, 2560, 6144, 10240, 12288, 248320}) {
@@ -283,7 +286,7 @@ int check_tables() {
const int cls = n_out < 2048 ? 0 : n_out < 4096 ? 1 : n_out < 8192 ? 2 : n_out < 12288 ? 3 : 4;
const int ti = type == 23 ? 0 : type == 12 ? 1 : type == 13 ? 2 : type == 14 ? 3 : 4;
const int b = native_mmvq_il_rows_for(89, type, nc, n_out);
- // STRATA_MMVQ_IL_89=1: the agreement; unset: what sm_89 answered before #3453 (the shared table, which is
+ // tables on (the default): the agreement; off (=0 or STRATA_DEFAULTS=026): what sm_89 answered before #3453 (the shared table, which is
// sm_86's except Q3_K, a type the shared table has no row for)
const int want = opt89 ? (kAda4070[ti][nc - 2][cls] == kAda4080[ti][nc - 2][cls] ? kAda4070[ti][nc - 2][cls] : 0)
: (type == 11 ? 0 : a);
@@ -303,7 +306,7 @@ int check_tables() {
++bad;
std::printf("FAIL table: the sm_60 table is not the P100's\n");
}
- // sm_89 one column: only Q3_K with STRATA_MMVQ_IL_89=1 (the 4070 and the 4080 agree), else none; wider than 4: the plain kernels
+ // sm_89 one column: only Q3_K with the tables on (the 4070 and the 4080 agree), else none; wider than 4: the plain kernels
if (native_mmvq_il_rows_for(89, 11, 1, 2560) != 0 || native_mmvq_il_rows_for(89, 11, 1, 6144) != (opt89 ? 1 : 0) ||
native_mmvq_il_rows_for(89, 12, 1, 6144) != 0) {
++bad;