Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

sm_80 defaults of #3779 (wide dense dequant, MMVQ-IL sm_80 tables) follow STRATA_DEFAULTS=026: back to the 0.2.6 kernels and the shared table; explicit =1 and =0 still win; arch_defaults_test checks the rule; release-notes line

Niko1221257963513842
diff --git a/docs/MMVQ_IL_TABLE.md b/docs/MMVQ_IL_TABLE.md
index 5a2251d73..0d56273ad 100644
--- a/docs/MMVQ_IL_TABLE.md
+++ b/docs/MMVQ_IL_TABLE.md
@@ -16,7 +16,7 @@ Which table a card gets:
| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) |
| sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |
| sm_89 (Ada) | `kIlRows_89`, ON by default on compute capability 8.9 cards only (0.2.7): the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. `STRATA_MMVQ_IL_89=0` or `STRATA_DEFAULTS=026` gives the shared table (`kIlRowsShared`) again, as in 0.2.6; an explicit `STRATA_MMVQ_IL_89=1` wins over the master switch. The output is bitwise the same either way (a table only changes how many rows a warp takes). Not measured on an Ada card by us: no sm_89 card has run `mmvq_il_parity` on this code, and the issue saw no end-to-end decode change on a 4090 or a 4080 (issue 3453), so the gain may be a microbenchmark one. The reporter tests 0.2.7 and reports back. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card. To check a card: run `mmvq_il_parity` on it (it must end with `OK`) and an interleaved decode A/B with the variable at 1 and 0. |
-| sm_80 (A100, A30, CMP 170HX) | `kIlRows_80`, `kIlRows58_80` and `kIl1Rows_80`, read off `--emit-table` by the author of PR #3779 on one CMP 170HX (not an A100: re-measure there); `STRATA_MMVQ_IL_SM80=0` gives the card the shared table again |
+| sm_80 (A100, A30, CMP 170HX) | `kIlRows_80`, `kIlRows58_80` and `kIl1Rows_80`, read off `--emit-table` by the author of PR #3779 on one CMP 170HX (not an A100: re-measure there); `STRATA_MMVQ_IL_SM80=0` or `STRATA_DEFAULTS=026` gives the card the shared table again |
| every other sm_80+ card | `kIlRowsShared` |
| sm_75 and older | the path is off; the plain kernels run |
| a card listed in `kIlArch` | its own table |
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.md
index 6a7416008..9837b5989 100644
--- a/docs/release-notes-0.2.7.md
+++ b/docs/release-notes-0.2.7.md
@@ -27,6 +27,8 @@
same bit for bit; only how many rows a warp takes changes. **We have not measured them on an RTX 40 card**: the tables come from two
community runs (an RTX 4070 and an RTX 4080), and the report with the 4090 saw no change in decode speed. If you have an RTX 40
card, please try 0.2.7 and post your decode numbers on issue 3453. Off switch: `STRATA_MMVQ_IL_89=0`.
+ - NVIDIA A100 / A30 / CMP 170HX class cards (compute capability 8.0): the wide dense dequantizer and decode row tables of PR 3779, on by default,
+ same output bit for bit, measured only on a CMP 170HX by the contributor. Off switches: `STRATA_DEQUANT_WIDE=0`, `STRATA_MMVQ_IL_SM80=0`.
## Going back to 0.2.6
@@ -38,7 +40,7 @@ Two switches bring back every default that 0.2.7 changed, in one step, for a tes
the RAM store's spare slots and put and its settle after a prompt, the dense dequantizer, the F16 gate/up operand, the drafter's
markov head and the head's column count, and the RDNA3 dense GEMM all go back to what 0.2.6 did.
- **`STRATA_DEFAULTS=026`** (Qwen): the residency-biased routing, the wide and the stream-split hyper-connection reads, the int8
- prompt attention, the RTX 40 (sm_89) decode tables, the four gfx12 decode switches, the PLE residency policy and the gfx12 decode split of 0.15 (back to 0.55).
+ prompt attention, the RTX 40 (sm_89) and sm_80 decode tables and sm_80 dense dequantizer, the four gfx12 decode switches, the PLE residency policy and the gfx12 decode split of 0.15 (back to 0.55).
Long prompts on the approximate read (`STRATA_KVA`) are off by default and stay as `STRATA_KVA` says.
A variable or flag you set yourself (`DS4_ATTN_SLAB=N`, `STRATA_ROUTE_PRIOR=2`, `--experts 5`, `--prefill-chunk N`, `--pcie-frac`,
diff --git a/include/strata/core/sm120_series.hpp b/include/strata/core/sm120_series.hpp
index 2674b28c4..bff1823b5 100644
--- a/include/strata/core/sm120_series.hpp
+++ b/include/strata/core/sm120_series.hpp
@@ -72,6 +72,14 @@ inline bool prompt_attn_imma_default(bool hip, int cc, bool v026 = false) { retu
/// measured on an Ada card by us), off under STRATA_DEFAULTS=026 and in HIP builds. `cc` = 10 * major + minor as the engine resolves it.
inline bool mmvq_il_89_default(bool hip, int cc, bool v026 = false) { return !v026 && !hip && cc == 89; }
+/// The two sm_80-only defaults of 0.2.7 (#3779: the wide dense dequant and the MMVQ-IL sm_80 tables, both bitwise the old output, measured on a
+/// CMP 170HX only) given the text of their variable (nullptr = unset): "1" on and "0" off win, unset is on unless STRATA_DEFAULTS=026.
+inline bool sm80_default_on(const char* env, bool v026 = false) {
+ if (env != nullptr && env[0] == '1') return true;
+ if (env != nullptr && env[0] == '0') return false;
+ return !v026;
+}
+
/// The answer for STRATA_MMVQ_IL_89 given the variable's text (nullptr = unset): "1" on and "0" off win over everything, anything else (unset,
/// empty, other text) is the rule above.
inline bool mmvq_il_89_resolve(const char* env, bool hip, int cc, bool v026 = false) {
diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cpp
index 3195bbba7..dfe22283e 100644
--- a/src/core/arch_defaults_test.cpp
+++ b/src/core/arch_defaults_test.cpp
@@ -257,6 +257,10 @@ int main() {
CHECK(mmvq_il_89_resolve("1", false, 89, true)); // an explicit =1 wins over the master switch
CHECK(mmvq_il_89_resolve("1", false, 86, false)); // (the tables are only looked up for cc 89 anyway)
CHECK(mmvq_il_89_resolve("", false, 89, false) && !mmvq_il_89_resolve("", false, 89, true)); // empty text: the rule
+ // #3779: the two sm_80-only defaults (wide dequant, MMVQ-IL sm_80 tables): on unless STRATA_DEFAULTS=026; "1" and "0" win
+ CHECK(sm80_default_on(nullptr, false) && !sm80_default_on(nullptr, true));
+ CHECK(!sm80_default_on("0", false) && sm80_default_on("1", true));
+ CHECK(sm80_default_on("", false) && !sm80_default_on("", true));
// GR_V3 and IMMA
CHECK(gr_v3_default(false, 12, 0, false, 72.0 * GiB, false));
CHECK(!gr_v3_default(false, 12, 0, false, 72.0 * GiB, true));
diff --git a/src/kernels/cuda/dequant_bf16.cu b/src/kernels/cuda/dequant_bf16.cu
index 00e1dd51e..9764fe9b6 100644
--- a/src/kernels/cuda/dequant_bf16.cu
+++ b/src/kernels/cuda/dequant_bf16.cu
@@ -5,6 +5,8 @@
#include "strata/kernels/dequant_bf16.hpp"
#include "strata/core/tc_sm75.hpp"
#include "strata/kernels/iq_kernels.hpp"
+#include "strata/core/defaults_026.hpp"
+#include "strata/core/sm120_series.hpp"
#include <cuda_fp16.h>
#include <cuda_runtime.h>
@@ -294,7 +296,9 @@ bool dequant_wide_on() {
int major = 0, minor = 0;
if (cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev) != cudaSuccess ||
cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, dev) != cudaSuccess) { (void) cudaGetLastError(); return false; }
- per_dev[dev] = (major == 6 && minor == 1) || (major == 8 && (minor == 6 || minor == 0)) ? 2 : 1;
+ // sm_80 is the 0.2.7 addition (#3779): STRATA_DEFAULTS=026 gives it the per-group kernels again, as 0.2.6 had
+ per_dev[dev] = (major == 6 && minor == 1) || (major == 8 && minor == 6) ||
+ (major == 8 && minor == 0 && strata::sm80_default_on(nullptr, strata::qwen_defaults_026())) ? 2 : 1;
}
return per_dev[dev] == 2;
}
diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu
index 62bc3c033..c806e1630 100644
--- a/src/kernels/cuda/native_mmvq.cu
+++ b/src/kernels/cuda/native_mmvq.cu
@@ -3810,7 +3810,7 @@ int il_cc() {
if (cc[dev] == 120 && cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev) == cudaSuccess && sms >= 80)
cc[dev] = 1201;
// STRATA_MMVQ_IL_SM80=0: an sm_80 card takes the shared table as before kIlRows_80 (A/B)
- if (cc[dev] == 80) { const char* v = std::getenv("STRATA_MMVQ_IL_SM80"); if (v != nullptr && v[0] == '0') cc[dev] = 81; }
+ if (cc[dev] == 80 && !strata::sm80_default_on(std::getenv("STRATA_MMVQ_IL_SM80"), strata::qwen_defaults_026())) cc[dev] = 81; // =0 or STRATA_DEFAULTS=026
cudaGetLastError();
}
return cc[dev];