Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
mmvq_il: the sm_120 2-4 column table only on 80+ SM cards; docs
Niko122144997124ffae
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 5c5724f12..831137849 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -2217,6 +2217,33 @@ signal, early cleanup and `--errors` error-propagation checks. Its `--synchronoumode is the negative control under memcheck: an eight-second watchdog releasesthe original launch/service deadlock and the test fails.+## Decode on compute capability 12.0 cards: four bitwise switches on by default, interleaved matvec up to 8 tokens++Measured on an RTX PRO 5000 Blackwell 72 GB (110 SMs, sm_120, Linux, CUDA 13.2), IQ3_S with every expert in VRAM, setup's+flags (`--spec 4 --spec-min-p 0.5`), 6 interleaved pairs of fresh engines, 300-token greedy answers, medians:++| | story | code | 8K document | 4K / 20K / 98K prompt |+|---|---|---|---|---|+| before (tok/s) | 168.8 | 224.5 | 186.8 | 3827 / 4607 / 4767 |+| now | 189.0 (+11.9%, 6/6 pairs) | 256.2 (+14.0%, 6/6) | 211.2 (+13.1%, 6/6) | unchanged |++The answers are the same tokens (every case and pair), and the identity gate passes (10 prompts plus the 4K and 20K prompt+dumps, with the fixed small cache and with every expert resident). What changed:++- On a 12.0 card these existing switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph+ branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=2` (the+ hyper-connection read's up projection with its loads hoisted, and its down projection as token pairs). Each is bitwise equal+ to the kernel it replaces. A fixed 4-token verify round went 13.70 -> 12.23 ms with the four together.+ `STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.+- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and a+ 5-token window cost 2 ms more than a 4-token one), with a 12.0 table for 5-8 tokens and, on 12.0 cards with 80+ SMs, its own+ 2-4 token table. Bitwise (`mmvq_il_parity`, 1-8 tokens). `STRATA_MMVQ_IL_WIDE=0` / `STRATA_MMVQ_IL_120=0` turn them off.+- With the cheaper rows, deeper speculation was measured again (3 pairs): `--spec 3` 1.145 / 1.020 / 1.123 of before,+ `--spec 4` 1.122 / 1.141 / 1.132, `--spec 5` 1.079 / 1.101 / 1.074, `--spec 6` 1.060 / 1.177 / 1.097, `--spec 8` 1.027 /+ 1.105 / 1.069 (story / code / document). `--spec 4 --spec-min-p 0.5` stays the default.++Not measured: other 12.0 cards (an RTX 5070 has not been run with these defaults), Windows, and partly-resident caches.+## CUDA opt-in switches measured for 0.1.45 (RTX 3060 and RTX A4000, sm_86; all off unless you set them)Ported from open pull requests and A/B'd as whole engine runs against the default (a fresh engine per side, IQ3_XXS, 5 pairs, medians;diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.mdindex 93cf1b53b..1f3bae102 100644--- a/docs/MMVQ_IL_TABLE.md+++ b/docs/MMVQ_IL_TABLE.md@@ -1,16 +1,20 @@# The interleaved small-batch matvec (STRATA_MMVQ_IL): how to measure a table for your card-The decode and verify windows run their dense projections (2 to 4 tokens at a time) through `native_mmvq_il`, which reads an+The decode and verify windows run their dense projections (2 to 8 tokens at a time) through `native_mmvq_il`, which reads aninterleaved copy of the q8_1 activations. Every choice it makes gives the same bits as the plain kernels(`STRATA_MMVQ_IL=0` turns the whole path off); only the speed differs. How many matrix rows one warp takes(1, 2 or 4, or 0 = use the plain kernel) is a table in `src/kernels/cuda/native_mmvq.cu`, indexed by the-quantisation type (IQ4_XS, Q4_K, Q5_K, Q6_K), the token count (2-4) and the matrix's row count (five classes).+quantisation type (IQ4_XS, Q4_K, Q5_K, Q6_K, Q3_K), the token count (2-4) and the matrix's row count (five classes).+Windows of 5 to 8 tokens have a second table (`kIlRows58_<cc>`, 1 or 2 rows a warp); only sm_120 has one so far (measured on+the RTX PRO 5000: e.g. the Q5_K head at 5 tokens 899 -> 491 us), every other card keeps the plain kernels there.+`STRATA_MMVQ_IL_WIDE=0` turns the 5-8 token path off. `--emit-table` prints both tables.Which table a card gets:| card | table ||---|---|-| sm_86 (Ampere consumer) and sm_120 (Blackwell consumer) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |+| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |+| sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table || sm_89 (Ada) and every other sm_80+ card | `kIlRowsShared` too. Nobody has measured sm_89 yet (we have no such card). || sm_75 and older | the path is off; the plain kernels run || a card listed in `kIlArch` | its own table |diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cuindex ec0da97bb..57e1c377b 100644--- a/src/kernels/cuda/native_mmvq.cu+++ b/src/kernels/cuda/native_mmvq.cu@@ -3069,6 +3069,7 @@ constexpr IlRows58 kIlRows58_120[] = { // 4 = 2 rows a warp here (launch_il_rostruct Il58Arch { int cc; const IlRows58* t; size_t n; };constexpr Il58Arch kIl58Arch[] = {{120, kIlRows58_120, sizeof(kIlRows58_120) / sizeof(IlRows58)},+ {1201, kIlRows58_120, sizeof(kIlRows58_120) / sizeof(IlRows58)}, // a 12.0 card with 80+ SMs (il_cc)};// One column: rows a warp by n_out class, 0 = native_mmvq's single-column kernels (STRATA_MMVQ_IL_T1=0 turns this off).struct Il1Rows { int type; uint8_t r[5]; };@@ -3099,7 +3100,9 @@ constexpr IlArch kIlArch[] = {{75, kIlRows_75, sizeof(kIlRows_75) / sizeof(IlRows)}, // Turing: measured on an RTX 2080 Ti{70, kIlRows_70, sizeof(kIlRows_70) / sizeof(IlRows)}, // Volta: PR 1401, measured on a V100-SXM2{86, kIlRows_86, sizeof(kIlRows_86) / sizeof(IlRows)}, // 143-dense: RTX A4000 with the real shapes (Q3_K added)- {120, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)}, // 147-verify: RTX PRO 5000 Blackwell+ // 147-verify: RTX PRO 5000 Blackwell (110 SMs). Only 12.0 cards with 80+ SMs (il_cc's 1201) take it: the shared table+ // was measured on an RTX 5070 (48 SMs) and stays that card's+ {1201, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)},};// STRATA_MMVQ_IL_ROWS="type:ncols:r0,r1,r2,r3,r4;..." overrides single rows of the table for every card (type = ggml// type 23/12/13/14, ncols 2-4, r* = rows a warp 0/1/2/4 for the five n_out classes). Parsed once.@@ -3161,9 +3164,9 @@ int native_mmvq_il_rows_for(int cc, int type, int ncols, int n_out) {if (type == 11 && !il_q3k_on()) return 0; // STRATA_MMVQ_IL_Q3K=0: Q3_K keeps native_mmvq's kernelsconst IlRows* t = kIlRowsShared;size_t n = sizeof(kIlRowsShared) / sizeof(IlRows);- static const bool own120 = il_env_on("STRATA_MMVQ_IL_120"); // 147-verify: =0 gives sm_120 the shared table again+ static const bool own120 = il_env_on("STRATA_MMVQ_IL_120"); // 147-verify: =0 gives a big 12.0 card the shared table againfor (const IlArch& a : kIlArch)- if (a.cc == cc && a.t != nullptr && (cc != 120 || own120)) { t = a.t; n = a.n; }+ if (a.cc == cc && a.t != nullptr && (cc != 1201 || own120)) { t = a.t; n = a.n; }for (size_t i = 0; i < n; ++i)if (t[i].type == type) return t[i].r[ncols - 2][cls];return 0;@@ -3178,6 +3181,12 @@ int il_cc() {if (cudaDeviceGetAttribute(&maj, cudaDevAttrComputeCapabilityMajor, dev) != cudaSuccess ||cudaDeviceGetAttribute(&mnr, cudaDevAttrComputeCapabilityMinor, dev) != cudaSuccess) { cudaGetLastError(); return 0; }cc[dev] = maj * 10 + mnr;+ // 147-verify: a 12.0 card with 80+ SMs (RTX PRO 5000 / 6000, RTX 5080 / 5090) answers 1201, its own table's key;+ // a smaller one (RTX 5070: 48 SMs) keeps 120 and the shared table it was measured into+ int sms = 0;+ if (cc[dev] == 120 && cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev) == cudaSuccess && sms >= 80)+ cc[dev] = 1201;+ cudaGetLastError();}return cc[dev];}