Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
native_mmvq: sm_89 (Ada) MMVQ-IL table from the cells two cards agree on (#3453)
rodi4egbdecbffc6bef
diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.mdindex 1f3bae102..c49296258 100644--- a/docs/MMVQ_IL_TABLE.md+++ b/docs/MMVQ_IL_TABLE.md@@ -15,18 +15,19 @@ Which table a card gets:|---|---|| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) || sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |-| sm_89 (Ada) and every other sm_80+ card | `kIlRowsShared` too. Nobody has measured sm_89 yet (we have no such card). |+| sm_89 (Ada) | `kIlRows_89`: the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card, so it is not taken until it reproduces. No end-to-end decode change was seen on a 4090 or a 4080 (issue 3453), so this is a microbenchmark gain only. |+| every other sm_80+ card | `kIlRowsShared` || sm_75 and older | the path is off; the plain kernels run || a card listed in `kIlArch` | its own table |-## Measuring sm_89 (or any other card)+## Measuring a card that has no table yet1. Build with tests: `cmake -S . -B build -DSTRATA_BUILD_TESTS=ON` and `cmake --build build --target mmvq_il_parity`.2. Close everything else on the GPU, then run `build/mmvq_il_parity --bench --emit-table` (a few minutes).It first checks bit-for-bit parity of every shape and every rows choice (it must end with `mmvq_il_parity: OK`),prints a `BENCH` line per shape and token count, and last a ready-to-paste block, one line per type.-3. Post the whole output on the issue. A maintainer pastes the block as `kIlRows_89[]` and adds- `{89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)}` to `kIlArch`.+3. Post the whole output on the issue. A maintainer pastes the block as `kIlRows_<cc>[]` and adds+ `{<cc>, kIlRows_<cc>, sizeof(kIlRows_<cc>) / sizeof(IlRows)}` to `kIlArch`.4. To try a table without rebuilding, set `STRATA_MMVQ_IL_ROWS="type:ncols:r0,r1,r2,r3,r4;..."`, for example`STRATA_MMVQ_IL_ROWS="23:3:0,0,2,4,4;12:2:0,1,1,1,1"` (the five numbers are the row classes below 2048, 4096, 8192,12288 and above; each is 0, 1, 2 or 4). Then A/B the decode: same flags, `STRATA_MMVQ_IL_ROWS` set or unset, 10diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cuindex afaec6cb5..c4b119425 100644--- a/src/kernels/cuda/native_mmvq.cu+++ b/src/kernels/cuda/native_mmvq.cu@@ -3037,10 +3037,9 @@ constexpr IlRows kIlRowsShared[] = { // sm_86 + sm_120, and the fallback for e{14, {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 2}, {0, 0, 1, 1, 1}}}, // Q6_K};// Per-architecture tables (P10). A card whose compute capability (major * 10 + minor) is listed here uses its own table;-// every other sm_80+ card uses kIlRowsShared. sm_89 (Ada) has none yet: nobody measured it (we have no such card), so-// it takes the shared table, which sits between the measured sm_86 and sm_120. To add one, run+// every other sm_80+ card uses kIlRowsShared. sm_89 (Ada) has its own table (kIlRows_89, below). To add one, run// `mmvq_il_parity --bench --emit-table` on the card (docs/MMVQ_IL_TABLE.md), paste the printed block as a-// `constexpr IlRows kIlRows_89[]` and add {89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)} below. Each choice+// `constexpr IlRows kIlRows_<cc>[]` and add {<cc>, kIlRows_<cc>, sizeof(kIlRows_<cc>) / sizeof(IlRows)} below. Each choice// is bitwise the same output, so a table is only speed. Without a rebuild: STRATA_MMVQ_IL_ROWS (see il_env_rows).// Volta (sm_70): read off mmvq_il_parity --bench on a V100-SXM2-32GB (two runs, mean), the same rule - per cell the// rows count whose every measured shape takes at least 3% off native_mmvq's time (e.g. the Q5_K head at 3 columns@@ -3091,6 +3090,19 @@ constexpr IlRows kIlRows_120[] = {{14, {{0, 0, 2, 1, 1}, {0, 0, 4, 2, 4}, {0, 2, 2, 2, 2}}}, // Q6_K{11, {{0, 1, 2, 2, 2}, {0, 2, 4, 2, 2}, {0, 2, 2, 2, 2}}}, // Q3_K};+// Ada (sm_89): the cells where two cards agree. mmvq_il_parity --bench --emit-table was run on an RTX 4070 (46 SMs,+// #3453) and on an RTX 4080 (48 SMs, same issue); a cell is kept only where both runs give the same rows count, else 0+// (the plain kernel), so a card that disagrees never gets a split it did not measure. The 5-8 column table of that+// issue is not taken: the 4080 reproduced the 2-4 column table byte for byte in four runs, and the 5-8 column table+// changed in 10-14 of 20 entries between runs. Bitwise equal to native_mmvq, so the table is only speed.+// mmvq_il_parity's check_tables rebuilds the table from the two source runs and compares.+constexpr IlRows kIlRows_89[] = {+ {23, {{0, 0, 0, 0, 0}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}}}, // IQ4_XS+ {12, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}}}, // Q4_K+ {13, {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 0}, {0, 0, 1, 0, 0}}}, // Q5_K+ {14, {{0, 0, 0, 0, 0}, {0, 0, 1, 0, 0}, {0, 0, 0, 1, 1}}}, // Q6_K+ {11, {{0, 0, 1, 0, 1}, {0, 1, 1, 1, 1}, {0, 1, 1, 1, 1}}}, // Q3_K+};// Turing (sm_75): read off mmvq_il_parity --bench --emit-table on an RTX 2080 Ti 22 GB (SM 1650 MHz), the same 3% rule.// The plain multi-column kernels are bound by dp4a throughput on Turing from 2 columns on (IQ4_XS 2560x10240 at 4// columns 81.1 -> 37.4 us, the Q5_K head 1671 -> 1045 us); bitwise equal to native_mmvq. No one-column table here:@@ -3124,6 +3136,10 @@ struct Il1Rows { int type; uint8_t r[5]; };constexpr Il1Rows kIl1Rows_86[] = {{11, {0, 1, 1, 1, 1}}, // Q3_K: 2560x10240 60 -> 43 us, 6144x2560 30 -> 26 us on the A4000};+// Ada (sm_89): the RTX 4070 and the RTX 4080 runs gave the same one-column row (only Q3_K takes the interleaved kernel)+constexpr Il1Rows kIl1Rows_89[] = {+ {11, {0, 0, 1, 1, 1}}, // Q3_K+};// Tesla P40 (sm_61), the same three runs: IQ4_XS 2560x10240 102 -> 58 us, the Q5_K head 1754 -> 1507 usconstexpr Il1Rows kIl1Rows_61[] = {{23, {2, 2, 4, 4, 4}}, // IQ4_XS@@ -3134,6 +3150,7 @@ struct Il1Arch { int cc; const Il1Rows* t; size_t n; };constexpr Il1Arch kIl1Arch[] = {{61, kIl1Rows_61, sizeof(kIl1Rows_61) / sizeof(Il1Rows)},{86, kIl1Rows_86, sizeof(kIl1Rows_86) / sizeof(Il1Rows)},+ {89, kIl1Rows_89, sizeof(kIl1Rows_89) / sizeof(Il1Rows)}, // Ada: the RTX 4070 and 4080 agree};bool il_env_on(const char* name) {const char* v = std::getenv(name);@@ -3148,6 +3165,7 @@ constexpr IlArch kIlArch[] = {{75, kIlRows_75, sizeof(kIlRows_75) / sizeof(IlRows)}, // Turing: measured on an RTX 2080 Ti{70, kIlRows_70, sizeof(kIlRows_70) / sizeof(IlRows)}, // Volta: PR 1401, measured on a V100-SXM2{86, kIlRows_86, sizeof(kIlRows_86) / sizeof(IlRows)}, // 143-dense: RTX A4000 with the real shapes (Q3_K added)+ {89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)}, // Ada: the cells where an RTX 4070 and an RTX 4080 agree// 147-verify: RTX PRO 5000 Blackwell (110 SMs). Only 12.0 cards with 80+ SMs (il_cc's 1201) take it: the shared table// was measured on an RTX 5070 (48 SMs) and stays that card's{1201, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)},diff --git a/src/kernels/mmvq_il_parity.cpp b/src/kernels/mmvq_il_parity.cppindex 9e78015a2..05225d9f2 100644--- a/src/kernels/mmvq_il_parity.cpp+++ b/src/kernels/mmvq_il_parity.cpp@@ -260,21 +260,33 @@ void emit_table() {}// the per-architecture lookup, no GPU needed for its answers+// The two Ada runs sm_89's table was merged from (mmvq_il_parity --bench --emit-table, issue 3453): by type IQ4_XS, Q4_K,+// Q5_K, Q6_K, Q3_K, then 2/3/4 columns, then the five n_out classes.+const int kAda4070[5][3][5] = {+ {{0, 0, 0, 1, 0}, {1, 1, 1, 1, 1}, {2, 2, 1, 1, 1}}, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 1, 1, 1, 1}},+ {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {1, 2, 1, 2, 2}}, {{0, 0, 0, 0, 0}, {0, 0, 1, 2, 2}, {0, 1, 4, 1, 1}},+ {{1, 1, 1, 2, 1}, {1, 1, 1, 1, 1}, {2, 1, 1, 1, 1}}};+const int kAda4080[5][3][5] = {+ {{0, 0, 0, 0, 0}, {0, 0, 1, 1, 1}, {0, 1, 1, 1, 1}}, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}},+ {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 0}, {0, 0, 1, 1, 1}}, {{0, 0, 0, 1, 0}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}},+ {{0, 0, 1, 1, 1}, {0, 1, 1, 1, 1}, {0, 1, 1, 1, 1}}};int check_tables() {int bad = 0;for (int type : {23, 12, 13, 14, 11})for (int nc = 2; nc <= 4; ++nc)for (int n_out : {512, 2560, 6144, 10240, 12288, 248320}) {const int a = native_mmvq_il_rows_for(86, type, nc, n_out);- // sm_89 has no table of its own: it answers the shared table (what sm_86 answered before its own- // 143-dense table, which differs only in Q3_K, a type the shared table does not use); sm_120 has its own- // since 147-verify+ if (a != 0 && a != 1 && a != 2 && a != 4) { ++bad; std::printf("FAIL table: rows %d\n", a); }+ // sm_89 (#3453): the cells where an RTX 4070 and an RTX 4080 gave the same rows count, else 0+ const int cls = n_out < 2048 ? 0 : n_out < 4096 ? 1 : n_out < 8192 ? 2 : n_out < 12288 ? 3 : 4;+ const int ti = type == 23 ? 0 : type == 12 ? 1 : type == 13 ? 2 : type == 14 ? 3 : 4;const int b = native_mmvq_il_rows_for(89, type, nc, n_out);- if ((type != 11 && b != a) || (type == 11 && b != 0)) {+ const int want = kAda4070[ti][nc - 2][cls] == kAda4080[ti][nc - 2][cls] ? kAda4070[ti][nc - 2][cls] : 0;+ if (b != want) {++bad;- std::printf("FAIL table: sm_89 differs from sm_86 at type %d ncols %d n_out %d\n", type, nc, n_out);+ std::printf("FAIL table: sm_89 is %d, the 4070/4080 agreement says %d at type %d ncols %d n_out %d\n", b, want,+ type, nc, n_out);}- if (a != 0 && a != 1 && a != 2 && a != 4) { ++bad; std::printf("FAIL table: rows %d\n", a); }}// Volta has its own table (PR 1401): Q6_K at 2 columns, 2560 rows -> 1 row a warp; the Ampere table says 0 thereif (native_mmvq_il_rows_for(70, 14, 2, 2560) != 1 || native_mmvq_il_rows_for(86, 14, 2, 2560) != 0) {@@ -286,6 +298,11 @@ int check_tables() {++bad;std::printf("FAIL table: the sm_60 table is not the P100's\n");}+ // sm_89 one column: only Q3_K (the 4070 and the 4080 agree); wider than 4: the plain kernels+ if (native_mmvq_il_rows_for(89, 11, 1, 2560) != 0 || native_mmvq_il_rows_for(89, 11, 1, 6144) != 1 || native_mmvq_il_rows_for(89, 12, 1, 6144) != 0) {+ ++bad;+ std::printf("FAIL table: the sm_89 one-column row is not Q3_K only\n");+ }if (native_mmvq_il_rows_for(89, 99, 2, 2560) != 0 || native_mmvq_il_rows_for(89, 12, 5, 2560) != 0 ||native_mmvq_il_rows_for(86, 12, 5, 2560) != 0 || native_mmvq_il_rows_for(120, 12, 9, 2560) != 0) {++bad;