Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

gfx12: the WMMA prompt attention and the Q2_0 fused prompt experts are default-on on Linux only; Windows HIP builds keep them opt-in until tested there

Niko122197b680e13e4f
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.md
index eccdb1ee3..c4fb19435 100644
--- a/docs/AMD_HIP.md
+++ b/docs/AMD_HIP.md
@@ -434,7 +434,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39
1,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode
89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);
its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.
- **Default on gfx12 (148-amd).** Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical):
+ **Default on gfx12 on Linux (148-amd).** Windows: opt-in until tested on Windows (`STRATA_HIP_WMMA=1` turns it on; the engine says so in its log at the first prompt). Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical):
teacher-forced KL against the old kernel through the real serve decode path (STRATA_FOLLOW, 28 prompts, 6,024 positions) 0.0003 (p99 0.006, top-1
99.8%, perplexity of the forced text x1.0003), and 0.0014 over 1,760 positions of eight 6K-token documents (top-1 98.9%, x1.0009); HumanEval 157/164
(default 156 and 156), reasoning 25/30 (24 and 26), agent 6/6, needles at 32K and 125K x three depths 6/6, CJK 8/8, tool calls 5/5. Prompt speed,
@@ -478,7 +478,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39
prompt's top token was the same one, so that check says little); 4 of 4 long-context retrieval checks passed.
0.1.42 re-check on an R9700 (Linux, ROCm 7.14), 12 prompts of 1.5K-24K tokens, first-token KL against the FP16 prompt path (MMQ / fused): IQ3_S 0.00332 / 0.00284, top-1 12 of 12 for both; IQ3_XXS 0.00335 / 0.00452 on one sample and 0.00344 / 0.00323 on a second (24 prompts pooled: 0.00340 / 0.00388, top-1 24 of 24 for both); the Q2_0 pack does not take these kernels on gfx12 (output bit-identical to MMQ). Because the IQ3_XXS pool is not at or below MMQ's, the flag stays opt-in. Measured on Linux only.
gfx1200 builds the same kernels and was not run; Windows and K-quant packs (`STRATA_PF_FUSED_KQ=1`) were not measured.
- **Q2_0 gate/up (148-amd).** The native Q2_0 pack's gate/up (64-value blocks) take the same kernels and are in the gfx12 default set with IQ3_S and
+ **Q2_0 gate/up (148-amd).** Windows: opt-in until tested on Windows (`STRATA_PF_FUSED=1`); the default below is Linux only. The native Q2_0 pack's gate/up (64-value blocks) take the same kernels and are in the gfx12 default set with IQ3_S and
IQ3_XXS (`STRATA_PF_FUSED=0` keeps MMQ). One layer, 2,048 tokens x top 10 over 512 experts: MMQ 24.5 ms, fused 4.4 ms; 8,192 tokens 31.7 vs 13.2 ms
(`hip_prefill_fused_iq`, fused against MMQ rel. RMS 1.6e-5). Engine, R9700 Q2_0 32 GB, 5 interleaved pairs: prompt 4K +16.7%, 20K +19.0% (noise arm 1.001 / 0.999).
Quality: teacher-forced KL against MMQ 0.0004 (6,022 positions, top-1 99.5%) and 0.0016 (1,760 long-document positions, top-1 98.2%), the same tasks as
diff --git a/include/strata/core/gfx12_platform_defaults.hpp b/include/strata/core/gfx12_platform_defaults.hpp
new file mode 100644
index 000000000..5124ed8d9
--- /dev/null
+++ b/include/strata/core/gfx12_platform_defaults.hpp
@@ -0,0 +1,18 @@
+// include/strata/core/gfx12_platform_defaults.hpp - which platforms take the two approved gfx12 prompt defaults.
+//
+// 148-amd measured and the owner approved STRATA_HIP_WMMA (matrix-core prompt attention) and the native Q2_0 pack's fused
+// int8 prompt experts (STRATA_PF_FUSED) as defaults on gfx12. Both were measured on Linux only: on a Windows HIP build
+// they stay off by default until they are tested there, and STRATA_HIP_WMMA=1 / STRATA_PF_FUSED=1 still switch them on.
+#pragma once
+
+namespace strata::core {
+
+constexpr bool gfx12_prompt_defaults_on(bool windows) noexcept { return !windows; }
+
+#if defined(_WIN32)
+inline constexpr bool kGfx12PromptDefaultsOn = gfx12_prompt_defaults_on(true);
+#else
+inline constexpr bool kGfx12PromptDefaultsOn = gfx12_prompt_defaults_on(false);
+#endif
+
+} // namespace strata::core
diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cpp
index 737d4a230..87f6246b4 100644
--- a/src/core/arch_defaults_test.cpp
+++ b/src/core/arch_defaults_test.cpp
@@ -1,5 +1,6 @@
// src/core/arch_defaults_test.cpp - the per-architecture default table (no GPU needed).
#include "strata/core/arch_defaults.hpp"
+#include "strata/core/gfx12_platform_defaults.hpp"
#include <cstdio>
#include <cstdlib>
@@ -70,6 +71,14 @@ int main() {
put("STRATA_GFX1151_DEFAULTS", "0");
CHECK(arch_default_env("gfx1151").empty());
CHECK(apply_arch_defaults("gfx1151").empty());
+ // the two approved gfx12 prompt defaults (WMMA attention, Q2_0 fused experts): on for Linux, off for Windows
+ CHECK(gfx12_prompt_defaults_on(false));
+ CHECK(!gfx12_prompt_defaults_on(true));
+#if defined(_WIN32)
+ CHECK(!kGfx12PromptDefaultsOn);
+#else
+ CHECK(kGfx12PromptDefaultsOn);
+#endif
std::printf(fails ? "arch_defaults_test: %d FAILED\n" : "arch_defaults_test: OK\n", fails);
return fails ? 1 : 0;
}
diff --git a/src/kernels/cuda/qsa_prompt_attn.cu b/src/kernels/cuda/qsa_prompt_attn.cu
index e1b3efa18..3bdc4ffc9 100644
--- a/src/kernels/cuda/qsa_prompt_attn.cu
+++ b/src/kernels/cuda/qsa_prompt_attn.cu
@@ -2,6 +2,7 @@
#include "strata/core/emulate.hpp"
#include "strata/core/sm75_series.hpp"
#include "strata/core/geometry.hpp"
+#include "strata/core/gfx12_platform_defaults.hpp"
#include <cstdlib>
#include <cstring>
#include "strata/kernels/qsa_prompt_attn.hpp"
@@ -2176,7 +2177,7 @@ __global__ void __launch_bounds__(THREADS) prompt_attn_wmma_kernel(const float*
#endif
}
-// gfx12 (RDNA4): on by default (148-amd; STRATA_HIP_WMMA=0 turns it off). gfx11 (RDNA3 / RDNA3.5): only on request
+// gfx12 (RDNA4): on by default on Linux (148-amd; STRATA_HIP_WMMA=0 turns it off; Windows: opt-in with =1 until tested there). gfx11 (RDNA3 / RDNA3.5): only on request
// (STRATA_HIP_WMMA=1). STRATA_HIP_WMMA=auto (151-amd) takes the matrix-core kernel on a gfx12 card for a batch of
// STRATA_HIP_WMMA_MIN_Q (64) or more queries and keeps the FP32 kernel for smaller calls. The output differs from the
// old kernel's in its last bits (FP32-level accuracy, another summation order): teacher-forced KL against the old kernel
@@ -2206,10 +2207,14 @@ int hip_wmma_arch(int dev) {
static bool told = false;
if (!told && (mode >= 1 || arch[dev] == 2)) {
told = true;
- const bool on = arch[dev] == 2 || (arch[dev] == 1 && mode >= 1);
- std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", mode >= 1 ? "STRATA_HIP_WMMA: " : "",
- !on ? "unavailable" : (mode == 2 ? "chosen per call" : "on"), prop.gcnArchName,
- mode >= 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");
+ const bool on = (arch[dev] == 2 && (mode >= 1 || strata::core::kGfx12PromptDefaultsOn)) || (arch[dev] == 1 && mode >= 1);
+ if (mode >= 1 || on)
+ std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", mode >= 1 ? "STRATA_HIP_WMMA: " : "",
+ !on ? "unavailable" : (mode == 2 ? "chosen per call" : "on"), prop.gcnArchName,
+ mode >= 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");
+ else
+ std::fprintf(stderr, "strata: the prompt attention on matrix cores is off by default on Windows (opt-in until tested on Windows); "
+ "STRATA_HIP_WMMA=1 turns it on (%s)\n", prop.gcnArchName);
}
}
return arch[dev];
@@ -2231,7 +2236,7 @@ bool hip_wmma_usable(int64_t n_q, int64_t cap) {
if (arch == 3) return false;
if (mode == 1) return true;
if (mode == 2) return hip_wmma_auto_pick(arch, n_q, cap);
- return arch == 2; // unset: the gfx12 default
+ return arch == 2 && strata::core::kGfx12PromptDefaultsOn; // unset: the gfx12 default (Linux; Windows is opt-in until tested there)
}
bool launch_wmma(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,
@@ -2322,7 +2327,7 @@ bool qsa_prompt_attn_matrix_cores_requested() {
int dev = 0;
if (hipGetDevice(&dev) != hipSuccess) { (void) hipGetLastError(); return false; }
const int mode = hip_wmma_mode_env(), arch = hip_wmma_arch(dev);
- return mode != 0 && arch != 3 && !g_wmma_failed.load(std::memory_order_relaxed) && (mode >= 1 || arch == 2);
+ return mode != 0 && arch != 3 && !g_wmma_failed.load(std::memory_order_relaxed) && (mode >= 1 || (arch == 2 && strata::core::kGfx12PromptDefaultsOn));
#else
return false;
#endif
diff --git a/src/prefill/moe_fused_iq.cu b/src/prefill/moe_fused_iq.cu
index 6eeb21db6..de2726815 100644
--- a/src/prefill/moe_fused_iq.cu
+++ b/src/prefill/moe_fused_iq.cu
@@ -23,6 +23,7 @@
#include "strata/kernels/gfx_arch.hpp"
#include "strata/core/emulate.hpp"
#include "strata/core/sm120_series.hpp"
+#include "strata/core/gfx12_platform_defaults.hpp"
#if defined(__HIPCC__)
#include "moe_f16_common.cuh"
@@ -1417,7 +1418,8 @@ bool native_supported(int gu_type, int d_type) {
return d.ok && d.s120 && d.sms >= 80 && !d.t75 && d_covered_on(d_type, d);
}
#endif
- if (env < 0 && gu_type != T_IQ3_S && gu_type != T_IQ3_XXS && gu_type != T_Q2_0) return false; // default: only these three
+ // default: only these three; the Q2_0 one (148-amd) is on Linux only: on Windows it is opt-in (STRATA_PF_FUSED=1) until tested there
+ if (env < 0 && gu_type != T_IQ3_S && gu_type != T_IQ3_XXS && !(gu_type == T_Q2_0 && strata::core::kGfx12PromptDefaultsOn)) return false;
const DevInfo& d = dev_info();
if (!d.ok || !d_covered_on(d_type, d)) return false;
if (d.t75 && (gu_type != T_Q2_0 || d_type != T_Q2_0)) return false;