Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
mmvq_il: the sm_89 (Ada) decode tables are on by default on compute capability 8.9 only, STRATA_MMVQ_IL_89=0 and STRATA_DEFAULTS=026 turn them off (#3453)
Niko1221663ec80abf9e
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex cb863f48d..082ad0e6d 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -1186,6 +1186,8 @@ On 12.0 cards (sm_120) these also apply:- The interleaved small-batch matvec (`native_mmvq_il`, [MMVQ_IL_TABLE.md](MMVQ_IL_TABLE.md)) runs 5 to 8 tokens, with a 12.0 tableand, on cards with 80 or more SMs, its own 2-4 token table. `STRATA_MMVQ_IL_WIDE=0` and `STRATA_MMVQ_IL_120=0` turn them off.+ RTX 40 cards (compute capability 8.9) have their own 2-4 token tables and a one-token Q3_K row, on by default in 0.2.7 and not yet measured+ on an RTX 40 card by us (same output bit for bit): `STRATA_MMVQ_IL_89=0` or `STRATA_DEFAULTS=026` turns them off.- With 80 or more SMs the native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`) for IQ2_S, IQ3_XXS,IQ3_S and IQ4_XS gate/up. IQ2_XXS and IQ2_XS keep MMQ. Smaller 12.0 cards (an RTX 5070 with 48 SMs) keep MMQ because the fusedkernels were 4.7% slower on 20K prompts there. `STRATA_PF_FUSED=0` keeps MMQ.diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.mdindex 8974c3be9..26c637b19 100644--- a/docs/MMVQ_IL_TABLE.md+++ b/docs/MMVQ_IL_TABLE.md@@ -15,7 +15,7 @@ Which table a card gets:|---|---|| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) || sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |-| sm_89 (Ada) | the shared table (`kIlRowsShared`) unless `STRATA_MMVQ_IL_89=1`, which switches on `kIlRows_89`: the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. It is opt-in because no sm_89 card has run `mmvq_il_parity` on this code, and the issue saw no end-to-end decode change on a 4090 or a 4080 (issue 3453), so the gain is a microbenchmark one. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card. To flip the default, run `mmvq_il_parity` on an Ada card (it must end with `OK`) and an interleaved decode A/B with the switch on and off. |+| sm_89 (Ada) | `kIlRows_89`, ON by default on compute capability 8.9 cards only (0.2.7): the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. `STRATA_MMVQ_IL_89=0` or `STRATA_DEFAULTS=026` gives the shared table (`kIlRowsShared`) again, as in 0.2.6; an explicit `STRATA_MMVQ_IL_89=1` wins over the master switch. The output is bitwise the same either way (a table only changes how many rows a warp takes). Not measured on an Ada card by us: no sm_89 card has run `mmvq_il_parity` on this code, and the issue saw no end-to-end decode change on a 4090 or a 4080 (issue 3453), so the gain may be a microbenchmark one. The reporter tests 0.2.7 and reports back. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card. To check a card: run `mmvq_il_parity` on it (it must end with `OK`) and an interleaved decode A/B with the variable at 1 and 0. || every other sm_80+ card | `kIlRowsShared` || sm_75 and older | the path is off; the plain kernels run || a card listed in `kIlArch` | its own table |diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex 6e806adef..6a7416008 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -23,6 +23,10 @@- The hyper-connection read split over two kernels (a discrete RTX 50 / RTX PRO Blackwell card with more than 13 GiB): **+1.2%decode on the RTX PRO 5000 with identical tokens**, about +1.5% together with the switch above. On 2x RTX A4000 and theRTX 3060 it was slower, so those cards keep the old read. Off switch: `STRATA_GR_V3=0`.+ - RTX 40 cards (Ada, compute capability 8.9): new decode row tables for the interleaved matrix kernels, on by default. The output is the+ same bit for bit; only how many rows a warp takes changes. **We have not measured them on an RTX 40 card**: the tables come from two+ community runs (an RTX 4070 and an RTX 4080), and the report with the 4090 saw no change in decode speed. If you have an RTX 40+ card, please try 0.2.7 and post your decode numbers on issue 3453. Off switch: `STRATA_MMVQ_IL_89=0`.## Going back to 0.2.6@@ -34,7 +38,7 @@ Two switches bring back every default that 0.2.7 changed, in one step, for a testhe RAM store's spare slots and put and its settle after a prompt, the dense dequantizer, the F16 gate/up operand, the drafter'smarkov head and the head's column count, and the RDNA3 dense GEMM all go back to what 0.2.6 did.- **`STRATA_DEFAULTS=026`** (Qwen): the residency-biased routing, the wide and the stream-split hyper-connection reads, the int8- prompt attention, the four gfx12 decode switches, the PLE residency policy and the gfx12 decode split of 0.15 (back to 0.55).+ prompt attention, the RTX 40 (sm_89) decode tables, the four gfx12 decode switches, the PLE residency policy and the gfx12 decode split of 0.15 (back to 0.55).Long prompts on the approximate read (`STRATA_KVA`) are off by default and stay as `STRATA_KVA` says.A variable or flag you set yourself (`DS4_ATTN_SLAB=N`, `STRATA_ROUTE_PRIOR=2`, `--experts 5`, `--prefill-chunk N`, `--pcie-frac`,diff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hppindex 8a6932a65..2674b28c4 100644--- a/include/strata/core/sm120_series.hpp+++ b/include/strata/core/sm120_series.hpp@@ -67,6 +67,19 @@ inline bool gr_v3_default(bool hip, int major, int minor, bool integrated, doubl/// resolves it (an emulated or forced Turing counts as 75). Measured on sm_120 (RTX PRO 5000) and sm_86 (2x RTX A4000): +1.1 .. +1.8% prompt.inline bool prompt_attn_imma_default(bool hip, int cc, bool v026 = false) { return !v026 && !hip && cc >= 80; }+/// STRATA_MMVQ_IL_89 (#3453: the decode kernel row tables of an Ada card, the cells where an RTX 4070 and an RTX 4080 agreed; bitwise the same+/// output, only the rows-a-warp split differs) when the user set nothing: ON for compute capability 8.9 only (owner decision, 26 October: not yet+/// measured on an Ada card by us), off under STRATA_DEFAULTS=026 and in HIP builds. `cc` = 10 * major + minor as the engine resolves it.+inline bool mmvq_il_89_default(bool hip, int cc, bool v026 = false) { return !v026 && !hip && cc == 89; }++/// The answer for STRATA_MMVQ_IL_89 given the variable's text (nullptr = unset): "1" on and "0" off win over everything, anything else (unset,+/// empty, other text) is the rule above.+inline bool mmvq_il_89_resolve(const char* env, bool hip, int cc, bool v026 = false) {+ if (env != nullptr && env[0] == '1') return true;+ if (env != nullptr && env[0] == '0') return false;+ return mmvq_il_89_default(hip, cc, v026);+}+/// STRATA_GR_UP_WIDE (the hyper-connection up projection, one warp a row; bitwise the old kernel) when the user set nothing: ON for a compute/// capability 12.0 card (STRATA_SM120_PERF=0 turns the set off), CUDA builds only.inline bool gr_up_wide_default(bool hip, int major, int minor, bool v026 = false) { return !v026 && !hip && sm120_default(major, minor); }diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cppindex db193a7f7..3195bbba7 100644--- a/src/core/arch_defaults_test.cpp+++ b/src/core/arch_defaults_test.cpp@@ -243,6 +243,20 @@ int main() {CHECK(!gr_up_wide_default(true, 12, 0));CHECK(!gr_up_wide_default(false, 8, 6));CHECK(!gr_up_wide_default(false, 12, 1));+ // #3453: the sm_89 MMVQ-IL tables: on by default for compute capability 8.9 only, off under 026 and in HIP; "1" and "0" win+ CHECK(mmvq_il_89_default(false, 89));+ CHECK(!mmvq_il_89_default(false, 86) && !mmvq_il_89_default(false, 80) && !mmvq_il_89_default(false, 90) &&+ !mmvq_il_89_default(false, 120) && !mmvq_il_89_default(false, 75) && !mmvq_il_89_default(false, 70) &&+ !mmvq_il_89_default(false, 60) && !mmvq_il_89_default(false, 0));+ CHECK(!mmvq_il_89_default(true, 89));+ CHECK(!mmvq_il_89_default(false, 89, true)); // STRATA_DEFAULTS=026+ CHECK(mmvq_il_89_resolve(nullptr, false, 89, false)); // unset: the rule+ CHECK(!mmvq_il_89_resolve(nullptr, false, 89, true)); // unset + 026: off+ CHECK(!mmvq_il_89_resolve(nullptr, false, 86, false)); // another card: off+ CHECK(!mmvq_il_89_resolve("0", false, 89, false)); // STRATA_MMVQ_IL_89=0 turns it off+ CHECK(mmvq_il_89_resolve("1", false, 89, true)); // an explicit =1 wins over the master switch+ CHECK(mmvq_il_89_resolve("1", false, 86, false)); // (the tables are only looked up for cc 89 anyway)+ CHECK(mmvq_il_89_resolve("", false, 89, false) && !mmvq_il_89_resolve("", false, 89, true)); // empty text: the rule// GR_V3 and IMMACHECK(gr_v3_default(false, 12, 0, false, 72.0 * GiB, false));CHECK(!gr_v3_default(false, 12, 0, false, 72.0 * GiB, true));diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cuindex f0c3913ef..ef6758826 100644--- a/src/kernels/cuda/native_mmvq.cu+++ b/src/kernels/cuda/native_mmvq.cu@@ -29,6 +29,8 @@#include "strata/kernels/iq_kernels.hpp"#include "strata/kernels/gfx_arch.hpp"#include "strata/core/sm75_series.hpp"+#include "strata/core/sm120_series.hpp"+#include "strata/core/defaults_026.hpp"#include "s26_tsum.cuh"#if !defined(__HIPCC__)#include "q8_1_il.cuh"@@ -3158,10 +3160,14 @@ bool il_env_on(const char* name) {}bool il_t1_on() { static const bool on = il_env_on("STRATA_MMVQ_IL_T1"); return on; }bool il_q3k_on() { static const bool on = il_env_on("STRATA_MMVQ_IL_Q3K"); return on; }-// #3453: sm_89's own tables (kIlRows_89 / kIl1Rows_89) are opt-in with STRATA_MMVQ_IL_89=1 until an sm_89 card has run-// mmvq_il_parity and an end-to-end A/B (docs/MMVQ_IL_TABLE.md). Unset, an Ada card answers what it answered before: the-// shared table, no one-column row. Cached once; the lookup runs on the host once a launch.-bool il_89_opt_in() { static const bool on = [] { const char* v = std::getenv("STRATA_MMVQ_IL_89"); return v != nullptr && v[0] == '1'; }(); return on; }+// #3453: sm_89's own tables (kIlRows_89 / kIl1Rows_89) are ON by default on compute capability 8.9 only (owner decision, 0.2.7; no sm_89 card+// has run mmvq_il_parity or an end-to-end A/B for us, the reporter tests and reports back). STRATA_MMVQ_IL_89=0 gives an Ada card what it+// answered before (the shared table, no one-column row) and so does STRATA_DEFAULTS=026; STRATA_MMVQ_IL_89=1 wins over that master switch+// (sm120_series.hpp mmvq_il_89_resolve). Cached once; the lookup runs on the host once a launch.+bool il_89_opt_in() {+ static const bool on = strata::mmvq_il_89_resolve(std::getenv("STRATA_MMVQ_IL_89"), false, 89, strata::qwen_defaults_026());+ return on;+}struct IlArch { int cc; const IlRows* t; size_t n; };constexpr IlArch kIlArch[] = {{60, kIlRows_60, sizeof(kIlRows_60) / sizeof(IlRows)}, // Pascal GP100: measured on a Tesla P100diff --git a/src/kernels/mmvq_il_parity.cpp b/src/kernels/mmvq_il_parity.cppindex 9e9cd1654..00ad704b8 100644--- a/src/kernels/mmvq_il_parity.cpp+++ b/src/kernels/mmvq_il_parity.cpp@@ -10,6 +10,8 @@// the block scales rewritten as normal fp16 values, so every output is finite (a NaN would compare equal whatever// produced it). The bench cycles through enough copies of the weights to miss the L2 cache, as a layer's call does.#include "strata/kernels/iq_kernels.hpp"+#include "strata/core/defaults_026.hpp"+#include "strata/core/sm120_series.hpp"#include "strata/kernels/native_mmvq.hpp"#include <cuda_runtime.h>@@ -273,7 +275,8 @@ const int kAda4080[5][3][5] = {int check_tables() {int bad = 0;const char* e89 = std::getenv("STRATA_MMVQ_IL_89");- const bool opt89 = e89 != nullptr && e89[0] == '1'; // sm_89's own tables are opt-in (#3453)+ // sm_89's own tables are ON by default (#3453, 0.2.7) unless STRATA_MMVQ_IL_89=0 or STRATA_DEFAULTS=026+ const bool opt89 = strata::mmvq_il_89_resolve(e89, false, 89, strata::qwen_defaults_026());for (int type : {23, 12, 13, 14, 11})for (int nc = 2; nc <= 4; ++nc)for (int n_out : {512, 2560, 6144, 10240, 12288, 248320}) {@@ -283,7 +286,7 @@ int check_tables() {const int cls = n_out < 2048 ? 0 : n_out < 4096 ? 1 : n_out < 8192 ? 2 : n_out < 12288 ? 3 : 4;const int ti = type == 23 ? 0 : type == 12 ? 1 : type == 13 ? 2 : type == 14 ? 3 : 4;const int b = native_mmvq_il_rows_for(89, type, nc, n_out);- // STRATA_MMVQ_IL_89=1: the agreement; unset: what sm_89 answered before #3453 (the shared table, which is+ // tables on (the default): the agreement; off (=0 or STRATA_DEFAULTS=026): what sm_89 answered before #3453 (the shared table, which is// sm_86's except Q3_K, a type the shared table has no row for)const int want = opt89 ? (kAda4070[ti][nc - 2][cls] == kAda4080[ti][nc - 2][cls] ? kAda4070[ti][nc - 2][cls] : 0): (type == 11 ? 0 : a);@@ -303,7 +306,7 @@ int check_tables() {++bad;std::printf("FAIL table: the sm_60 table is not the P100's\n");}- // sm_89 one column: only Q3_K with STRATA_MMVQ_IL_89=1 (the 4070 and the 4080 agree), else none; wider than 4: the plain kernels+ // sm_89 one column: only Q3_K with the tables on (the 4070 and the 4080 agree), else none; wider than 4: the plain kernelsif (native_mmvq_il_rows_for(89, 11, 1, 2560) != 0 || native_mmvq_il_rows_for(89, 11, 1, 6144) != (opt89 ? 1 : 0) ||native_mmvq_il_rows_for(89, 12, 1, 6144) != 0) {++bad;