Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge f-027-gh-eng-n (#3779 sm_80 tuning: wide dequant, MMVQ-IL sm_80 tables and the opt-ins)
Niko12211e278a280df0
diff --cc CMakeLists.txtindex 237a1119d,250a22f1f..ff7fb5919--- a/CMakeLists.txt+++ b/CMakeLists.txt@@@ -1150,9 -1132,10 +1158,13 @@@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_target_include_directories(mmvq_il_parity PRIVATE ${_strata_gpu_include_directories})target_link_libraries(mmvq_il_parity PRIVATE strata_kernels ${_strata_gpu_runtime_target})add_test(NAME mmvq_il_parity COMMAND mmvq_il_parity)+ # sm_89's opt-in tables (#3453, STRATA_MMVQ_IL_89=1): the table check only (no case matches MMVQ_IL_ONLY)+ add_test(NAME mmvq_il_parity_89table COMMAND mmvq_il_parity)+ set_tests_properties(mmvq_il_parity_89table PROPERTIES ENVIRONMENT "STRATA_MMVQ_IL_89=1;MMVQ_IL_ONLY=no-such-case")+ # the opt-in STRATA_MMVQ_IL_PF kernels (hoisted group loads, exact integer forms): every variant bit-checked against+ # native_mmvq for every type, shape, T and rows choice+ add_test(NAME mmvq_il_parity_pf COMMAND mmvq_il_parity --pf 0,2,3,4,5,6)+ set_tests_properties(mmvq_il_parity_pf PROPERTIES TIMEOUT 900)# the wide dequant kernels (a thread per 8 values; sm_61's default), bitwise vs the per-group ones (GPU, synthetic,# no model; --bench adds old/new timings)add_executable(dequant_wide_parity src/kernels/dequant_wide_parity.cpp)diff --cc docs/MMVQ_IL_TABLE.mdindex 26c637b19,c8e5a7fad..5a2251d73--- a/docs/MMVQ_IL_TABLE.md+++ b/docs/MMVQ_IL_TABLE.md@@@ -15,8 -15,8 +15,9 @@@ Which table a card gets|---|---|| sm_86 (Ampere consumer) and sm_120 cards with fewer than 80 SMs (RTX 5070 and smaller) | `kIlRowsShared`, measured on an RTX 3060 and an RTX 5070 together (a cell is non-zero only if the worse of the two cards still gains 3%) || sm_120 cards with 80 or more SMs (RTX PRO 5000 / 6000 Blackwell, RTX 5080 / 5090) | `kIlRows_120`, measured on an RTX PRO 5000 Blackwell (110 SMs); `STRATA_MMVQ_IL_120=0` gives them the shared table |+| sm_89 (Ada) | `kIlRows_89`, ON by default on compute capability 8.9 cards only (0.2.7): the cells where an RTX 4070 (46 SMs) and an RTX 4080 (48 SMs) community run gave the same rows count, else 0 (the plain kernel); one column: Q3_K only. `STRATA_MMVQ_IL_89=0` or `STRATA_DEFAULTS=026` gives the shared table (`kIlRowsShared`) again, as in 0.2.6; an explicit `STRATA_MMVQ_IL_89=1` wins over the master switch. The output is bitwise the same either way (a table only changes how many rows a warp takes). Not measured on an Ada card by us: no sm_89 card has run `mmvq_il_parity` on this code, and the issue saw no end-to-end decode change on a 4090 or a 4080 (issue 3453), so the gain may be a microbenchmark one. The reporter tests 0.2.7 and reports back. No 5-8 column table: the 4080's changed in 10-14 of 20 entries between runs of the same card. To check a card: run `mmvq_il_parity` on it (it must end with `OK`) and an interleaved decode A/B with the variable at 1 and 0. |+ | sm_80 (A100, A30, CMP 170HX) | `kIlRows_80`, `kIlRows58_80` and `kIl1Rows_80`, read off `--emit-table` by the author of PR #3779 on one CMP 170HX (not an A100: re-measure there); `STRATA_MMVQ_IL_SM80=0` gives the card the shared table again |-| sm_89 (Ada) and every other sm_80+ card | `kIlRowsShared` too. Nobody has measured sm_89 yet (we have no such card). |+| every other sm_80+ card | `kIlRowsShared` || sm_75 and older | the path is off; the plain kernels run || a card listed in `kIlArch` | its own table |diff --cc metal/src/engine/stubs_0145.cppindex 8aa89f3d5,da99868cf..a429dab1d--- a/metal/src/engine/stubs_0145.cpp+++ b/metal/src/engine/stubs_0145.cpp@@@ -147,12 -147,10 +147,17 @@@ namespace strata::kernels { bool gemm_i// STRATA_Q8_SM60 (Pascal GP100, opt-in): a no-op on Metal; NativeHead::load()/unload() call these for a Q8_0 head, so they must exist and answer "nothing packed".namespace strata::kernels { bool native_q8_0_sm60_pack(void const* p0, int p1, int p2, char const* p3) { return false; } }namespace strata::kernels { void native_q8_0_sm60_release(void const* p0) { } }+// the MTP drafter's STRATA_MTP_Q8_SM60 packing (mtp.cpp): nothing packed on Metal+namespace strata::kernels { bool native_q8_0_sm60_pack_quiet(void const* p0, int p1, int p2, char const* p3) { return false; } }+namespace strata::kernels { bool native_q8_0_sm60_pack_in_place(void* p0, int p1, int p2, char const* p3) { return false; } }+// STRATA_SELECT_LONG (verify.cpp's long-context decode select): CUDA only; never reached on Metal (no scratch asked for)+namespace strata::kernels { long long qsa_topk_multi_scratch_ints(long long p0) { return 0; } }+namespace strata::kernels { void qsa_block_topk_long(float const* p0, int const* p1, long long p2, long long p3, long long p4, strata::kernels::QsaShapes const& p5, int* p6, void* p7, long long p8, int* p9) { strata::metal_engine::missing("strata::kernels::qsa_block_topk_long"); } }+namespace strata::kernels { void qsa_block_scores_long(float const* p0, float const* p1, float const* p2, int const* p3, long long p4, long long p5, strata::kernels::QsaShapes const& p6, float* p7, void* p8, long long p9) { strata::metal_engine::missing("strata::kernels::qsa_block_scores_long"); } }// only reached with pf_wmma (off on Metal)namespace strata::prefill { void swiglu_split_f16(float const* p0, unsigned short* p1, long long p2, int p3, bool p4, void* p5) { } }+ // STRATA_PLE_COMPACT (opt-in, CUDA / HIP prompt path): prefill.cpp references it; the Metal prompt runs through its window and never reaches it.+ namespace strata::kernels { void ple_iq4nl_dequant(const uint8_t* p0, int64_t p1, float* p2, void* p3) { strata::metal_engine::missing("strata::kernels::ple_iq4nl_dequant"); } }+ // STRATA_DEQUANT_SIDE (opt-in, CUDA only): off on Metal; the prompt path takes no second scratch and set_dequant_side(nullptr) keeps the serial form.+ namespace strata::prefill { bool gemm_dequant_side() { return false; } }+ auto strata::prefill::Gemm::set_dequant_side(uint16_t* p0, std::string& p1) -> bool { if (p0 == nullptr) return true; strata::metal_engine::missing("strata::prefill::Gemm::set_dequant_side"); return false; }diff --cc src/kernels/cuda/native_mmvq.cuindex ef6758826,4867ec7f6..62bc3c033--- a/src/kernels/cuda/native_mmvq.cu+++ b/src/kernels/cuda/native_mmvq.cu@@@ -3092,19 -3553,29 +3600,42 @@@ constexpr IlRows kIlRows_120[] ={14, {{0, 0, 2, 1, 1}, {0, 0, 4, 2, 4}, {0, 2, 2, 2, 2}}}, // Q6_K{11, {{0, 1, 2, 2, 2}, {0, 2, 4, 2, 2}, {0, 2, 2, 2, 2}}}, // Q3_K};+// Ada (sm_89): the cells where two cards agree. mmvq_il_parity --bench --emit-table was run on an RTX 4070 (46 SMs,+// #3453) and on an RTX 4080 (48 SMs, same issue); a cell is kept only where both runs give the same rows count, else 0+// (the plain kernel), so a card that disagrees never gets a split it did not measure. The 5-8 column table of that+// issue is not taken: the 4080 reproduced the 2-4 column table byte for byte in four runs, and the 5-8 column table+// changed in 10-14 of 20 entries between runs. Bitwise equal to native_mmvq, so the table is only speed.+// mmvq_il_parity's check_tables rebuilds the table from the two source runs and compares.+constexpr IlRows kIlRows_89[] = {+ {23, {{0, 0, 0, 0, 0}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}}}, // IQ4_XS+ {12, {{0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}, {0, 0, 1, 1, 1}}}, // Q4_K+ {13, {{0, 0, 1, 1, 0}, {0, 0, 1, 1, 0}, {0, 0, 1, 0, 0}}}, // Q5_K+ {14, {{0, 0, 0, 0, 0}, {0, 0, 1, 0, 0}, {0, 0, 0, 1, 1}}}, // Q6_K+ {11, {{0, 0, 1, 0, 1}, {0, 1, 1, 1, 1}, {0, 1, 1, 1, 1}}}, // Q3_K+};+ // sm_80: read off mmvq_il_parity --bench --emit-table on a CMP 170HX (GA100, 70 SMs, HBM2e 1.5 TB/s), the same 3% rule;+ // bitwise equal to native_mmvq (the tool checks every case).+ constexpr IlRows kIlRows_80[] = {+ {23, {{0, 2, 2, 2, 2}, {0, 2, 2, 2, 2}, {0, 2, 1, 1, 1}}}, // IQ4_XS+ {12, {{0, 0, 4, 4, 4}, {0, 0, 2, 2, 2}, {0, 0, 2, 2, 2}}}, // Q4_K+ {13, {{0, 0, 2, 2, 4}, {0, 0, 2, 2, 2}, {0, 0, 2, 2, 2}}}, // Q5_K+ {14, {{0, 0, 2, 2, 2}, {0, 2, 2, 2, 2}, {0, 2, 4, 2, 2}}}, // Q6_K+ {11, {{1, 2, 2, 2, 2}, {1, 1, 2, 2, 2}, {1, 2, 2, 2, 2}}}, // Q3_K+ };+ // sm_80 P1/P2: the hoisted-load / exact-integer kernel variant and its rows a warp per cell, pf << 4 | rows+ // (0x32 = variant 3 at 2 rows a warp; 0 = today's kernel with kIlRows_80), read off mmvq_il_parity --bench --emit-table+ // --pf 0,3,4 on the CMP 170HX with the same 3% rule against native_mmvq; STRATA_MMVQ_IL_PF=1 takes it. Every variant is+ // bitwise native_mmvq (the tool checks every case, T 1-8, rows 1/2/4, each variant).+ struct IlPf { int type; uint8_t v[3][5]; };+ struct IlPf58 { int type; uint8_t v[4][5]; };+ // Only Q6_K: the same variants are bitwise and measured for Q4_K / Q5_K / IQ4_XS too, but there they tie today's kernel+ // within the run-to-run noise (their emitted cells flip between 0 and a variant), so those types keep kIlRows_80.+ constexpr IlPf kIlPf_80[] = {+ {14, {{0x31, 0x42, 0x32, 0x42, 0x42}, {0x31, 0x41, 0x32, 0x32, 0x41}, {0x31, 0x41, 0x32, 0x32, 0x32}}}, // Q6_K+ };+ constexpr IlPf58 kIlPf58_80[] = { // (rows 4 runs as 2 at 5+ columns, so the emitted 0x34 cells are written 0x32)+ {14, {{0x31, 0x42, 0x32, 0x32, 0x32}, {0x31, 0x31, 0x32, 0x32, 0x32}, {0x31, 0x31, 0x02, 0x32, 0x02}, {0x31, 0x31, 0x31, 0x31, 0x31}}}, // Q6_K+ };// Turing (sm_75): read off mmvq_il_parity --bench --emit-table on an RTX 2080 Ti 22 GB (SM 1650 MHz), the same 3% rule.// The plain multi-column kernels are bound by dp4a throughput on Turing from 2 columns on (IQ4_XS 2560x10240 at 4// columns 81.1 -> 37.4 us, the Q5_K head 1671 -> 1045 us); bitwise equal to native_mmvq. No one-column table here:@@@ -3148,11 -3623,17 +3687,18 @@@ constexpr Il1Rows kIl1Rows_61[] ={13, {0, 0, 0, 2, 2}}, // Q5_K{11, {0, 0, 2, 2, 2}}, // Q3_K};+ constexpr Il1Rows kIl1Rows_80[] = { // CMP 170HX, as kIlRows_80+ {23, {0, 0, 1, 4, 1}}, // IQ4_XS+ {12, {0, 0, 2, 2, 2}}, // Q4_K+ {13, {0, 0, 0, 4, 1}}, // Q5_K+ {11, {0, 0, 2, 2, 2}}, // Q3_K+ };struct Il1Arch { int cc; const Il1Rows* t; size_t n; };constexpr Il1Arch kIl1Arch[] = {+ {80, kIl1Rows_80, sizeof(kIl1Rows_80) / sizeof(Il1Rows)},{61, kIl1Rows_61, sizeof(kIl1Rows_61) / sizeof(Il1Rows)},{86, kIl1Rows_86, sizeof(kIl1Rows_86) / sizeof(Il1Rows)},+ {89, kIl1Rows_89, sizeof(kIl1Rows_89) / sizeof(Il1Rows)}, // Ada: the RTX 4070 and 4080 agree};bool il_env_on(const char* name) {const char* v = std::getenv(name);@@@ -3174,8 -3647,8 +3720,9 @@@ constexpr IlArch kIlArch[] ={61, kIlRows_61, sizeof(kIlRows_61) / sizeof(IlRows)}, // Pascal with dp4a: measured on a Tesla P40{75, kIlRows_75, sizeof(kIlRows_75) / sizeof(IlRows)}, // Turing: measured on an RTX 2080 Ti{70, kIlRows_70, sizeof(kIlRows_70) / sizeof(IlRows)}, // Volta: PR 1401, measured on a V100-SXM2+ {80, kIlRows_80, sizeof(kIlRows_80) / sizeof(IlRows)}, // sm_80: measured on a CMP 170HX{86, kIlRows_86, sizeof(kIlRows_86) / sizeof(IlRows)}, // 143-dense: RTX A4000 with the real shapes (Q3_K added)+ {89, kIlRows_89, sizeof(kIlRows_89) / sizeof(IlRows)}, // Ada: the cells where an RTX 4070 and an RTX 4080 agree// 147-verify: RTX PRO 5000 Blackwell (110 SMs). Only 12.0 cards with 80+ SMs (il_cc's 1201) take it: the shared table// was measured on an RTX 5070 (48 SMs) and stays that card's{1201, kIlRows_120, sizeof(kIlRows_120) / sizeof(IlRows)},diff --cc src/kernels/mmvq_il_parity.cppindex 00ad704b8,e55d6d013..ef65fec1f--- a/src/kernels/mmvq_il_parity.cpp+++ b/src/kernels/mmvq_il_parity.cpp@@@ -306,12 -412,26 +432,32 @@@ int check_tables()++bad;std::printf("FAIL table: the sm_60 table is not the P100's\n");}+ // sm_89 one column: only Q3_K with the tables on (the 4070 and the 4080 agree), else none; wider than 4: the plain kernels+ if (native_mmvq_il_rows_for(89, 11, 1, 2560) != 0 || native_mmvq_il_rows_for(89, 11, 1, 6144) != (opt89 ? 1 : 0) ||+ native_mmvq_il_rows_for(89, 12, 1, 6144) != 0) {+ ++bad;+ std::printf("FAIL table: the sm_89 one-column row is not Q3_K only\n");+ }+ // sm_80 has its own tables (kIlRows_80 / kIlRows58_80 / kIl1Rows_80: a CMP 170HX's); the key 81 is what STRATA_MMVQ_IL_SM80=0+ // gives an sm_80 card: no table of its own, so the shared one (and no one-column table). Every answer is 0, 1, 2 or 4.+ {+ bool differs = false;+ for (int type : {23, 12, 13, 14, 11})+ for (int nc = 1; nc <= 8; ++nc)+ for (int n_out : {512, 2560, 6144, 10240, 12288, 248320}) {+ const int a = native_mmvq_il_rows_for(80, type, nc, n_out), b = native_mmvq_il_rows_for(81, type, nc, n_out);+ if (a != 0 && a != 1 && a != 2 && a != 4) { ++bad; std::printf("FAIL table: sm_80 rows %d\n", a); }+ differs = differs || a != b;+ }+ if (!differs) { ++bad; std::printf("FAIL table: sm_80 answers like the shared table\n"); }+ // spot checks read off the source tables: Q6_K at 3 columns and 2560 rows (class 1) is {0, 2, 2, 2, 2}[1] = 2; one-column+ // IQ4_XS at 6144 rows (class 2) is {0, 0, 1, 4, 1}[2] = 1 on sm_80 and 0 (no one-column table) on the key 81+ if (native_mmvq_il_rows_for(80, 14, 3, 2560) != 2 || native_mmvq_il_rows_for(80, 23, 1, 6144) != 1 ||+ native_mmvq_il_rows_for(81, 23, 1, 6144) != 0) {+ ++bad;+ std::printf("FAIL table: the sm_80 table is not the CMP 170HX's\n");+ }+ }if (native_mmvq_il_rows_for(89, 99, 2, 2560) != 0 || native_mmvq_il_rows_for(89, 12, 5, 2560) != 0 ||native_mmvq_il_rows_for(86, 12, 5, 2560) != 0 || native_mmvq_il_rows_for(120, 12, 9, 2560) != 0) {++bad;diff --cc src/prefill/prefill.cppindex 92d0efbac,0d19ef90f..9ceec4606--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@@ -120,19 -116,11 +121,22 @@@ bool native_supported(int, int) { returvoid quantize_act_native(const float*, int64_t, int64_t, void*, void*) {}void experts_native(const Batch&, const NativeGeom&, int, int64_t, const void*, const void*, const int32_t*, void*,float*, void*) {}+ bool native_wide() { return false; }+ void experts_native(const WideBatch&, const NativeGeom&, int, int64_t, const void*, const void*, const int32_t*, void*,+ float*, void*) {}} // namespace strata::prefill::fused#endif+#ifndef STRATA_PREFILL_WS+// the weight-stationary experts (moe_ws.hpp) are in the CUDA STRATA_EXPERIMENTAL_SM60 build only+namespace strata::prefill::ws {+int mode() { return 0; }+bool supported(int, int) { return false; }+bool faster(int, double) { return false; }+size_t chunks(const int32_t*, size_t, const int32_t*, const int32_t*, Chunk*) { return 0; }+bool gate_up(const Layer&, void*) { return false; }+void down(const Layer&, void*) {}+} // namespace strata::prefill::ws+#endif#if defined(STRATA_GPU_METAL)// The Metal backend reads a prompt through its decode window (metal/src/engine/window.cpp): init keeps the chunk, run