Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

mmvq_il: the sm_120 2-4 column table only on 80+ SM cards; docs

Niko122144997124ffae
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 5c5724f12..831137849 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -2217,6 +2217,33 @@ signal, early cleanup and `--errors` error-propagation checks. Its `--synchronou
mode is the negative control under memcheck: an eight-second watchdog releases
the original launch/service deadlock and the test fails.
+## Decode on compute capability 12.0 cards: four bitwise switches on by default, interleaved matvec up to 8 tokens
+
+Measured on an RTX PRO 5000 Blackwell 72 GB (110 SMs, sm_120, Linux, CUDA 13.2), IQ3_S with every expert in VRAM, setup's
+flags (`--spec 4 --spec-min-p 0.5`), 6 interleaved pairs of fresh engines, 300-token greedy answers, medians:
+
+| | story | code | 8K document | 4K / 20K / 98K prompt |
+|---|---|---|---|---|
+| before (tok/s) | 168.8 | 224.5 | 186.8 | 3827 / 4607 / 4767 |
+| now | 189.0 (+11.9%, 6/6 pairs) | 256.2 (+14.0%, 6/6) | 211.2 (+13.1%, 6/6) | unchanged |
+
+The answers are the same tokens (every case and pair), and the identity gate passes (10 prompts plus the 4K and 20K prompt
+dumps, with the fixed small cache and with every expert resident). What changed:
+
+- On a 12.0 card these existing switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph
+ branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=2` (the
+ hyper-connection read's up projection with its loads hoisted, and its down projection as token pairs). Each is bitwise equal
+ to the kernel it replaces. A fixed 4-token verify round went 13.70 -> 12.23 ms with the four together.
+ `STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.
+- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and a
+ 5-token window cost 2 ms more than a 4-token one), with a 12.0 table for 5-8 tokens and, on 12.0 cards with 80+ SMs, its own
+ 2-4 token table. Bitwise (`mmvq_il_parity`, 1-8 tokens). `STRATA_MMVQ_IL_WIDE=0` / `STRATA_MMVQ_IL_120=0` turn them off.
+- With the cheaper rows, deeper speculation was measured again (3 pairs): `--spec 3` 1.145 / 1.020 / 1.123 of before,
+ `--spec 4` 1.122 / 1.141 / 1.132, `--spec 5` 1.079 / 1.101 / 1.074, `--spec 6` 1.060 / 1.177 / 1.097, `--spec 8` 1.027 /
+ 1.105 / 1.069 (story / code / document). `--spec 4 --spec-min-p 0.5` stays the default.
+
+Not measured: other 12.0 cards (an RTX 5070 has not been run with these defaults), Windows, and partly-resident caches.
+
## CUDA opt-in switches measured for 0.1.45 (RTX 3060 and RTX A4000, sm_86; all off unless you set them)
Ported from open pull requests and A/B'd as whole engine runs against the default (a fresh engine per side, IQ3_XXS, 5 pairs, medians;
diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.md
index 93cf1b53b..1f3bae102 100644
--- a/docs/MMVQ_IL_TABLE.md
+++ b/docs/MMVQ_IL_TABLE.md
@@ -1,16 +1,20 @@
# The interleaved small-batch matvec (STRATA_MMVQ_IL): how to measure a table for your card
-The decode and verify windows run their dense projections (2 to 4 tokens at a time) through `native_mmvq_il`, which reads an
+The decode and verify windows run their dense projections (2 to 8 tokens at a time) through `native_mmvq_il`, which reads an
interleaved copy of the q8_1 activations. Every choice it makes gives the same bits as the plain kernels
(`STRATA_MMVQ_IL=0` turns the whole path off); only the speed differs. How many matrix rows one warp takes
(1, 2 or 4, or 0 = use the plain kernel) is a table in `src/kernels/cuda/native_mmvq.cu`, indexed by the
-quantisation type (IQ4_XS, Q4_K, Q5_K, Q6_K), the token count (2-4) and the matrix's row count (five classes).
+quantisation type (IQ4_XS, Q4_K, Q5_K, Q6_K, Q3_K), the token count (2-4) and the matrix's row count (five classes).
+Windows of 5 to 8 tokens have a second table (`kIlRows58_<cc>`, 1 or 2 rows a warp); only sm_120 has one so far (measured on
+the RTX PRO 5000: e.g. the Q5_K head at 5 tokens 899 -> 491 us), every other card keeps the plain kernels there.
+`STRATA_MMVQ_IL_WIDE=0` turns the 5-8 token path off. `--emit-table` prints both tables.
Which table a card gets:
| card | table |
|---|---|
-| sm_86 (Ampere consumer) and sm_120 (Blackwell consumer) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |
+| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |
+| sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |
| sm_89 (Ada) and every other sm_80+ card | `kIlRowsShared` too. Nobody has measured sm_89 yet (we have no such card). |
| sm_75 and older | the path is off; the plain kernels run |
| a card listed in `kIlArch` | its own table |
diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu
index ec0da97bb..57e1c377b 100644
--- a/src/kernels/cuda/native_mmvq.cu
+++ b/src/kernels/cuda/native_mmvq.cu
@@ -3069,6 +3069,7 @@ constexpr IlRows58 kIlRows58_120[] = { // 4 = 2 rows a warp here (launch_il_ro
struct Il58Arch { int cc; const IlRows58* t; size_t n; };
constexpr Il58Arch kIl58Arch[] = {
{120, kIlRows58_120, sizeof(kIlRows58_120) / sizeof(IlRows58)},
+ {1201, kIlRows58_120, sizeof(kIlRows58_120) / sizeof(IlRows58)}, // a 12.0 card with 80+ SMs (il_cc)
};
// One column: rows a warp by n_out class, 0 = native_mmvq's single-column kernels (STRATA_MMVQ_IL_T1=0 turns this off).
struct Il1Rows { int type; uint8_t r[5]; };
@@ -3099,7 +3100,9 @@ constexpr IlArch kIlArch[] = {
{75, kIlRows_75, sizeof(kIlRows_75) / sizeof(IlRows)}, // Turing: measured on an RTX 2080 Ti
{70, kIlRows_70, sizeof(kIlRows_70) / sizeof(IlRows)}, // Volta: PR 1401, measured on a V100-SXM2
{86, kIlRows_86, sizeof(kIlRows_86) / sizeof(IlRows)}, // 143-dense: RTX A4000 with the real shapes (Q3_K added)
- {120, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)}, // 147-verify: RTX PRO 5000 Blackwell
+ // 147-verify: RTX PRO 5000 Blackwell (110 SMs). Only 12.0 cards with 80+ SMs (il_cc's 1201) take it: the shared table
+ // was measured on an RTX 5070 (48 SMs) and stays that card's
+ {1201, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)},
};
// STRATA_MMVQ_IL_ROWS="type:ncols:r0,r1,r2,r3,r4;..." overrides single rows of the table for every card (type = ggml
// type 23/12/13/14, ncols 2-4, r* = rows a warp 0/1/2/4 for the five n_out classes). Parsed once.
@@ -3161,9 +3164,9 @@ int native_mmvq_il_rows_for(int cc, int type, int ncols, int n_out) {
if (type == 11 && !il_q3k_on()) return 0; // STRATA_MMVQ_IL_Q3K=0: Q3_K keeps native_mmvq's kernels
const IlRows* t = kIlRowsShared;
size_t n = sizeof(kIlRowsShared) / sizeof(IlRows);
- static const bool own120 = il_env_on("STRATA_MMVQ_IL_120"); // 147-verify: =0 gives sm_120 the shared table again
+ static const bool own120 = il_env_on("STRATA_MMVQ_IL_120"); // 147-verify: =0 gives a big 12.0 card the shared table again
for (const IlArch& a : kIlArch)
- if (a.cc == cc && a.t != nullptr && (cc != 120 || own120)) { t = a.t; n = a.n; }
+ if (a.cc == cc && a.t != nullptr && (cc != 1201 || own120)) { t = a.t; n = a.n; }
for (size_t i = 0; i < n; ++i)
if (t[i].type == type) return t[i].r[ncols - 2][cls];
return 0;
@@ -3178,6 +3181,12 @@ int il_cc() {
if (cudaDeviceGetAttribute(&maj, cudaDevAttrComputeCapabilityMajor, dev) != cudaSuccess ||
cudaDeviceGetAttribute(&mnr, cudaDevAttrComputeCapabilityMinor, dev) != cudaSuccess) { cudaGetLastError(); return 0; }
cc[dev] = maj * 10 + mnr;
+ // 147-verify: a 12.0 card with 80+ SMs (RTX PRO 5000 / 6000, RTX 5080 / 5090) answers 1201, its own table's key;
+ // a smaller one (RTX 5070: 48 SMs) keeps 120 and the shared table it was measured into
+ int sms = 0;
+ if (cc[dev] == 120 && cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev) == cudaSuccess && sms >= 80)
+ cc[dev] = 1201;
+ cudaGetLastError();
}
return cc[dev];
}