Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

native_mmvq: sm_89 (Ada) MMVQ-IL table from the cells two cards agree on (#3453)

rodi4egbdecbffc6bef
diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.md
index 1f3bae102..c49296258 100644
--- a/docs/MMVQ_IL_TABLE.md
+++ b/docs/MMVQ_IL_TABLE.md
@@ -15,18 +15,19 @@ Which table a card gets:
|---|---|
| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |
| sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |
-| sm_89 (Ada) and every other sm_80+ card | `kIlRowsShared` too. Nobody has measured sm_89 yet (we have no such card). |
+| sm_89 (Ada) | `kIlRows_89`: the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card, so it is not taken until it reproduces. No end-to-end decode change was seen on a 4090 or a 4080 (issue 3453), so this is a microbenchmark gain only. |
+| every other sm_80+ card | `kIlRowsShared` |
| sm_75 and older | the path is off; the plain kernels run |
| a card listed in `kIlArch` | its own table |
-## Measuring sm_89 (or any other card)
+## Measuring a card that has no table yet
1. Build with tests: `cmake -S . -B build -DSTRATA_BUILD_TESTS=ON` and `cmake --build build --target mmvq_il_parity`.
2. Close everything else on the GPU, then run `build/mmvq_il_parity --bench --emit-table` (a few minutes).
It first checks bit-for-bit parity of every shape and every rows choice (it must end with `mmvq_il_parity: OK`),
prints a `BENCH` line per shape and token count, and last a ready-to-paste block, one line per type.
-3. Post the whole output on the issue. A maintainer pastes the block as `kIlRows_89[]` and adds
- `{89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)}` to `kIlArch`.
+3. Post the whole output on the issue. A maintainer pastes the block as `kIlRows_<cc>[]` and adds
+ `{<cc>, kIlRows_<cc>, sizeof(kIlRows_<cc>) / sizeof(IlRows)}` to `kIlArch`.
4. To try a table without rebuilding, set `STRATA_MMVQ_IL_ROWS="type:ncols:r0,r1,r2,r3,r4;..."`, for example
`STRATA_MMVQ_IL_ROWS="23:3:0,0,2,4,4;12:2:0,1,1,1,1"` (the five numbers are the row classes below 2048, 4096, 8192,
12288 and above; each is 0, 1, 2 or 4). Then A/B the decode: same flags, `STRATA_MMVQ_IL_ROWS` set or unset, 10
diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu
index afaec6cb5..c4b119425 100644
--- a/src/kernels/cuda/native_mmvq.cu
+++ b/src/kernels/cuda/native_mmvq.cu
@@ -3037,10 +3037,9 @@ constexpr IlRows kIlRowsShared[] = { // sm_86 + sm_120, and the fallback for e
{14, {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 2}, {0, 0, 1, 1, 1}}}, // Q6_K
};
// Per-architecture tables (P10). A card whose compute capability (major * 10 + minor) is listed here uses its own table;
-// every other sm_80+ card uses kIlRowsShared. sm_89 (Ada) has none yet: nobody measured it (we have no such card), so
-// it takes the shared table, which sits between the measured sm_86 and sm_120. To add one, run
+// every other sm_80+ card uses kIlRowsShared. sm_89 (Ada) has its own table (kIlRows_89, below). To add one, run
// `mmvq_il_parity --bench --emit-table` on the card (docs/MMVQ_IL_TABLE.md), paste the printed block as a
-// `constexpr IlRows kIlRows_89[]` and add {89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)} below. Each choice
+// `constexpr IlRows kIlRows_<cc>[]` and add {<cc>, kIlRows_<cc>, sizeof(kIlRows_<cc>) / sizeof(IlRows)} below. Each choice
// is bitwise the same output, so a table is only speed. Without a rebuild: STRATA_MMVQ_IL_ROWS (see il_env_rows).
// Volta (sm_70): read off mmvq_il_parity --bench on a V100-SXM2-32GB (two runs, mean), the same rule - per cell the
// rows count whose every measured shape takes at least 3% off native_mmvq's time (e.g. the Q5_K head at 3 columns
@@ -3091,6 +3090,19 @@ constexpr IlRows kIlRows_120[] = {
{14, {{0, 0, 2, 1, 1}, {0, 0, 4, 2, 4}, {0, 2, 2, 2, 2}}}, // Q6_K
{11, {{0, 1, 2, 2, 2}, {0, 2, 4, 2, 2}, {0, 2, 2, 2, 2}}}, // Q3_K
};
+// Ada (sm_89): the cells where two cards agree. mmvq_il_parity --bench --emit-table was run on an RTX 4070 (46 SMs,
+// #3453) and on an RTX 4080 (48 SMs, same issue); a cell is kept only where both runs give the same rows count, else 0
+// (the plain kernel), so a card that disagrees never gets a split it did not measure. The 5-8 column table of that
+// issue is not taken: the 4080 reproduced the 2-4 column table byte for byte in four runs, and the 5-8 column table
+// changed in 10-14 of 20 entries between runs. Bitwise equal to native_mmvq, so the table is only speed.
+// mmvq_il_parity's check_tables rebuilds the table from the two source runs and compares.
+constexpr IlRows kIlRows_89[] = {
+ {23, {{0, 0, 0, 0, 0}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}}}, // IQ4_XS
+ {12, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}}}, // Q4_K
+ {13, {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 0}, {0, 0, 1, 0, 0}}}, // Q5_K
+ {14, {{0, 0, 0, 0, 0}, {0, 0, 1, 0, 0}, {0, 0, 0, 1, 1}}}, // Q6_K
+ {11, {{0, 0, 1, 0, 1}, {0, 1, 1, 1, 1}, {0, 1, 1, 1, 1}}}, // Q3_K
+};
// Turing (sm_75): read off mmvq_il_parity --bench --emit-table on an RTX 2080 Ti 22 GB (SM 1650 MHz), the same 3% rule.
// The plain multi-column kernels are bound by dp4a throughput on Turing from 2 columns on (IQ4_XS 2560x10240 at 4
// columns 81.1 -> 37.4 us, the Q5_K head 1671 -> 1045 us); bitwise equal to native_mmvq. No one-column table here:
@@ -3124,6 +3136,10 @@ struct Il1Rows { int type; uint8_t r[5]; };
constexpr Il1Rows kIl1Rows_86[] = {
{11, {0, 1, 1, 1, 1}}, // Q3_K: 2560x10240 60 -> 43 us, 6144x2560 30 -> 26 us on the A4000
};
+// Ada (sm_89): the RTX 4070 and the RTX 4080 runs gave the same one-column row (only Q3_K takes the interleaved kernel)
+constexpr Il1Rows kIl1Rows_89[] = {
+ {11, {0, 0, 1, 1, 1}}, // Q3_K
+};
// Tesla P40 (sm_61), the same three runs: IQ4_XS 2560x10240 102 -> 58 us, the Q5_K head 1754 -> 1507 us
constexpr Il1Rows kIl1Rows_61[] = {
{23, {2, 2, 4, 4, 4}}, // IQ4_XS
@@ -3134,6 +3150,7 @@ struct Il1Arch { int cc; const Il1Rows* t; size_t n; };
constexpr Il1Arch kIl1Arch[] = {
{61, kIl1Rows_61, sizeof(kIl1Rows_61) / sizeof(Il1Rows)},
{86, kIl1Rows_86, sizeof(kIl1Rows_86) / sizeof(Il1Rows)},
+ {89, kIl1Rows_89, sizeof(kIl1Rows_89) / sizeof(Il1Rows)}, // Ada: the RTX 4070 and 4080 agree
};
bool il_env_on(const char* name) {
const char* v = std::getenv(name);
@@ -3148,6 +3165,7 @@ constexpr IlArch kIlArch[] = {
{75, kIlRows_75, sizeof(kIlRows_75) / sizeof(IlRows)}, // Turing: measured on an RTX 2080 Ti
{70, kIlRows_70, sizeof(kIlRows_70) / sizeof(IlRows)}, // Volta: PR 1401, measured on a V100-SXM2
{86, kIlRows_86, sizeof(kIlRows_86) / sizeof(IlRows)}, // 143-dense: RTX A4000 with the real shapes (Q3_K added)
+ {89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)}, // Ada: the cells where an RTX 4070 and an RTX 4080 agree
// 147-verify: RTX PRO 5000 Blackwell (110 SMs). Only 12.0 cards with 80+ SMs (il_cc's 1201) take it: the shared table
// was measured on an RTX 5070 (48 SMs) and stays that card's
{1201, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)},
diff --git a/src/kernels/mmvq_il_parity.cpp b/src/kernels/mmvq_il_parity.cpp
index 9e78015a2..05225d9f2 100644
--- a/src/kernels/mmvq_il_parity.cpp
+++ b/src/kernels/mmvq_il_parity.cpp
@@ -260,21 +260,33 @@ void emit_table() {
}
// the per-architecture lookup, no GPU needed for its answers
+// The two Ada runs sm_89's table was merged from (mmvq_il_parity --bench --emit-table, issue 3453): by type IQ4_XS, Q4_K,
+// Q5_K, Q6_K, Q3_K, then 2/3/4 columns, then the five n_out classes.
+const int kAda4070[5][3][5] = {
+ {{0, 0, 0, 1, 0}, {1, 1, 1, 1, 1}, {2, 2, 1, 1, 1}}, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 1, 1, 1, 1}},
+ {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {1, 2, 1, 2, 2}}, {{0, 0, 0, 0, 0}, {0, 0, 1, 2, 2}, {0, 1, 4, 1, 1}},
+ {{1, 1, 1, 2, 1}, {1, 1, 1, 1, 1}, {2, 1, 1, 1, 1}}};
+const int kAda4080[5][3][5] = {
+ {{0, 0, 0, 0, 0}, {0, 0, 1, 1, 1}, {0, 1, 1, 1, 1}}, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}},
+ {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 0}, {0, 0, 1, 1, 1}}, {{0, 0, 0, 1, 0}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}},
+ {{0, 0, 1, 1, 1}, {0, 1, 1, 1, 1}, {0, 1, 1, 1, 1}}};
int check_tables() {
int bad = 0;
for (int type : {23, 12, 13, 14, 11})
for (int nc = 2; nc <= 4; ++nc)
for (int n_out : {512, 2560, 6144, 10240, 12288, 248320}) {
const int a = native_mmvq_il_rows_for(86, type, nc, n_out);
- // sm_89 has no table of its own: it answers the shared table (what sm_86 answered before its own
- // 143-dense table, which differs only in Q3_K, a type the shared table does not use); sm_120 has its own
- // since 147-verify
+ if (a != 0 && a != 1 && a != 2 && a != 4) { ++bad; std::printf("FAIL table: rows %d\n", a); }
+ // sm_89 (#3453): the cells where an RTX 4070 and an RTX 4080 gave the same rows count, else 0
+ const int cls = n_out < 2048 ? 0 : n_out < 4096 ? 1 : n_out < 8192 ? 2 : n_out < 12288 ? 3 : 4;
+ const int ti = type == 23 ? 0 : type == 12 ? 1 : type == 13 ? 2 : type == 14 ? 3 : 4;
const int b = native_mmvq_il_rows_for(89, type, nc, n_out);
- if ((type != 11 && b != a) || (type == 11 && b != 0)) {
+ const int want = kAda4070[ti][nc - 2][cls] == kAda4080[ti][nc - 2][cls] ? kAda4070[ti][nc - 2][cls] : 0;
+ if (b != want) {
++bad;
- std::printf("FAIL table: sm_89 differs from sm_86 at type %d ncols %d n_out %d\n", type, nc, n_out);
+ std::printf("FAIL table: sm_89 is %d, the 4070/4080 agreement says %d at type %d ncols %d n_out %d\n", b, want,
+ type, nc, n_out);
}
- if (a != 0 && a != 1 && a != 2 && a != 4) { ++bad; std::printf("FAIL table: rows %d\n", a); }
}
// Volta has its own table (PR 1401): Q6_K at 2 columns, 2560 rows -> 1 row a warp; the Ampere table says 0 there
if (native_mmvq_il_rows_for(70, 14, 2, 2560) != 1 || native_mmvq_il_rows_for(86, 14, 2, 2560) != 0) {
@@ -286,6 +298,11 @@ int check_tables() {
++bad;
std::printf("FAIL table: the sm_60 table is not the P100's\n");
}
+ // sm_89 one column: only Q3_K (the 4070 and the 4080 agree); wider than 4: the plain kernels
+ if (native_mmvq_il_rows_for(89, 11, 1, 2560) != 0 || native_mmvq_il_rows_for(89, 11, 1, 6144) != 1 || native_mmvq_il_rows_for(89, 12, 1, 6144) != 0) {
+ ++bad;
+ std::printf("FAIL table: the sm_89 one-column row is not Q3_K only\n");
+ }
if (native_mmvq_il_rows_for(89, 99, 2, 2560) != 0 || native_mmvq_il_rows_for(89, 12, 5, 2560) != 0 ||
native_mmvq_il_rows_for(86, 12, 5, 2560) != 0 || native_mmvq_il_rows_for(120, 12, 9, 2560) != 0) {
++bad;