Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
gfx12: the WMMA prompt attention and the Q2_0 fused prompt experts are default-on on Linux only; Windows HIP builds keep them opt-in until tested there
Niko122197b680e13e4f
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex eccdb1ee3..c4fb19435 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -434,7 +434,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 391,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.- **Default on gfx12 (148-amd).** Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical):+ **Default on gfx12 on Linux (148-amd).** Windows: opt-in until tested on Windows (`STRATA_HIP_WMMA=1` turns it on; the engine says so in its log at the first prompt). Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical):teacher-forced KL against the old kernel through the real serve decode path (STRATA_FOLLOW, 28 prompts, 6,024 positions) 0.0003 (p99 0.006, top-199.8%, perplexity of the forced text x1.0003), and 0.0014 over 1,760 positions of eight 6K-token documents (top-1 98.9%, x1.0009); HumanEval 157/164(default 156 and 156), reasoning 25/30 (24 and 26), agent 6/6, needles at 32K and 125K x three depths 6/6, CJK 8/8, tool calls 5/5. Prompt speed,@@ -478,7 +478,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39prompt's top token was the same one, so that check says little); 4 of 4 long-context retrieval checks passed.0.1.42 re-check on an R9700 (Linux, ROCm 7.14), 12 prompts of 1.5K-24K tokens, first-token KL against the FP16 prompt path (MMQ / fused): IQ3_S 0.00332 / 0.00284, top-1 12 of 12 for both; IQ3_XXS 0.00335 / 0.00452 on one sample and 0.00344 / 0.00323 on a second (24 prompts pooled: 0.00340 / 0.00388, top-1 24 of 24 for both); the Q2_0 pack does not take these kernels on gfx12 (output bit-identical to MMQ). Because the IQ3_XXS pool is not at or below MMQ's, the flag stays opt-in. Measured on Linux only.gfx1200 builds the same kernels and was not run; Windows and K-quant packs (`STRATA_PF_FUSED_KQ=1`) were not measured.- **Q2_0 gate/up (148-amd).** The native Q2_0 pack's gate/up (64-value blocks) take the same kernels and are in the gfx12 default set with IQ3_S and+ **Q2_0 gate/up (148-amd).** Windows: opt-in until tested on Windows (`STRATA_PF_FUSED=1`); the default below is Linux only. The native Q2_0 pack's gate/up (64-value blocks) take the same kernels and are in the gfx12 default set with IQ3_S andIQ3_XXS (`STRATA_PF_FUSED=0` keeps MMQ). One layer, 2,048 tokens x top 10 over 512 experts: MMQ 24.5 ms, fused 4.4 ms; 8,192 tokens 31.7 vs 13.2 ms(`hip_prefill_fused_iq`, fused against MMQ rel. RMS 1.6e-5). Engine, R9700 Q2_0 32 GB, 5 interleaved pairs: prompt 4K +16.7%, 20K +19.0% (noise arm 1.001 / 0.999).Quality: teacher-forced KL against MMQ 0.0004 (6,022 positions, top-1 99.5%) and 0.0016 (1,760 long-document positions, top-1 98.2%), the same tasks asdiff --git a/include/strata/core/gfx12_platform_defaults.hpp b/include/strata/core/gfx12_platform_defaults.hppnew file mode 100644index 000000000..5124ed8d9--- /dev/null+++ b/include/strata/core/gfx12_platform_defaults.hpp@@ -0,0 +1,18 @@+// include/strata/core/gfx12_platform_defaults.hpp - which platforms take the two approved gfx12 prompt defaults.+//+// 148-amd measured and the owner approved STRATA_HIP_WMMA (matrix-core prompt attention) and the native Q2_0 pack's fused+// int8 prompt experts (STRATA_PF_FUSED) as defaults on gfx12. Both were measured on Linux only: on a Windows HIP build+// they stay off by default until they are tested there, and STRATA_HIP_WMMA=1 / STRATA_PF_FUSED=1 still switch them on.+#pragma once++namespace strata::core {++constexpr bool gfx12_prompt_defaults_on(bool windows) noexcept { return !windows; }++#if defined(_WIN32)+inline constexpr bool kGfx12PromptDefaultsOn = gfx12_prompt_defaults_on(true);+#else+inline constexpr bool kGfx12PromptDefaultsOn = gfx12_prompt_defaults_on(false);+#endif++} // namespace strata::corediff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cppindex 737d4a230..87f6246b4 100644--- a/src/core/arch_defaults_test.cpp+++ b/src/core/arch_defaults_test.cpp@@ -1,5 +1,6 @@// src/core/arch_defaults_test.cpp - the per-architecture default table (no GPU needed).#include "strata/core/arch_defaults.hpp"+#include "strata/core/gfx12_platform_defaults.hpp"#include <cstdio>#include <cstdlib>@@ -70,6 +71,14 @@ int main() {put("STRATA_GFX1151_DEFAULTS", "0");CHECK(arch_default_env("gfx1151").empty());CHECK(apply_arch_defaults("gfx1151").empty());+ // the two approved gfx12 prompt defaults (WMMA attention, Q2_0 fused experts): on for Linux, off for Windows+ CHECK(gfx12_prompt_defaults_on(false));+ CHECK(!gfx12_prompt_defaults_on(true));+#if defined(_WIN32)+ CHECK(!kGfx12PromptDefaultsOn);+#else+ CHECK(kGfx12PromptDefaultsOn);+#endifstd::printf(fails ? "arch_defaults_test: %d FAILED\n" : "arch_defaults_test: OK\n", fails);return fails ? 1 : 0;}diff --git a/src/kernels/cuda/qsa_prompt_attn.cu b/src/kernels/cuda/qsa_prompt_attn.cuindex e1b3efa18..3bdc4ffc9 100644--- a/src/kernels/cuda/qsa_prompt_attn.cu+++ b/src/kernels/cuda/qsa_prompt_attn.cu@@ -2,6 +2,7 @@#include "strata/core/emulate.hpp"#include "strata/core/sm75_series.hpp"#include "strata/core/geometry.hpp"+#include "strata/core/gfx12_platform_defaults.hpp"#include <cstdlib>#include <cstring>#include "strata/kernels/qsa_prompt_attn.hpp"@@ -2176,7 +2177,7 @@ __global__ void __launch_bounds__(THREADS) prompt_attn_wmma_kernel(const float*#endif}-// gfx12 (RDNA4): on by default (148-amd; STRATA_HIP_WMMA=0 turns it off). gfx11 (RDNA3 / RDNA3.5): only on request+// gfx12 (RDNA4): on by default on Linux (148-amd; STRATA_HIP_WMMA=0 turns it off; Windows: opt-in with =1 until tested there). gfx11 (RDNA3 / RDNA3.5): only on request// (STRATA_HIP_WMMA=1). STRATA_HIP_WMMA=auto (151-amd) takes the matrix-core kernel on a gfx12 card for a batch of// STRATA_HIP_WMMA_MIN_Q (64) or more queries and keeps the FP32 kernel for smaller calls. The output differs from the// old kernel's in its last bits (FP32-level accuracy, another summation order): teacher-forced KL against the old kernel@@ -2206,10 +2207,14 @@ int hip_wmma_arch(int dev) {static bool told = false;if (!told && (mode >= 1 || arch[dev] == 2)) {told = true;- const bool on = arch[dev] == 2 || (arch[dev] == 1 && mode >= 1);- std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", mode >= 1 ? "STRATA_HIP_WMMA: " : "",- !on ? "unavailable" : (mode == 2 ? "chosen per call" : "on"), prop.gcnArchName,- mode >= 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");+ const bool on = (arch[dev] == 2 && (mode >= 1 || strata::core::kGfx12PromptDefaultsOn)) || (arch[dev] == 1 && mode >= 1);+ if (mode >= 1 || on)+ std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", mode >= 1 ? "STRATA_HIP_WMMA: " : "",+ !on ? "unavailable" : (mode == 2 ? "chosen per call" : "on"), prop.gcnArchName,+ mode >= 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");+ else+ std::fprintf(stderr, "strata: the prompt attention on matrix cores is off by default on Windows (opt-in until tested on Windows); "+ "STRATA_HIP_WMMA=1 turns it on (%s)\n", prop.gcnArchName);}}return arch[dev];@@ -2231,7 +2236,7 @@ bool hip_wmma_usable(int64_t n_q, int64_t cap) {if (arch == 3) return false;if (mode == 1) return true;if (mode == 2) return hip_wmma_auto_pick(arch, n_q, cap);- return arch == 2; // unset: the gfx12 default+ return arch == 2 && strata::core::kGfx12PromptDefaultsOn; // unset: the gfx12 default (Linux; Windows is opt-in until tested there)}bool launch_wmma(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,@@ -2322,7 +2327,7 @@ bool qsa_prompt_attn_matrix_cores_requested() {int dev = 0;if (hipGetDevice(&dev) != hipSuccess) { (void) hipGetLastError(); return false; }const int mode = hip_wmma_mode_env(), arch = hip_wmma_arch(dev);- return mode != 0 && arch != 3 && !g_wmma_failed.load(std::memory_order_relaxed) && (mode >= 1 || arch == 2);+ return mode != 0 && arch != 3 && !g_wmma_failed.load(std::memory_order_relaxed) && (mode >= 1 || (arch == 2 && strata::core::kGfx12PromptDefaultsOn));#elsereturn false;#endifdiff --git a/src/prefill/moe_fused_iq.cu b/src/prefill/moe_fused_iq.cuindex 6eeb21db6..de2726815 100644--- a/src/prefill/moe_fused_iq.cu+++ b/src/prefill/moe_fused_iq.cu@@ -23,6 +23,7 @@#include "strata/kernels/gfx_arch.hpp"#include "strata/core/emulate.hpp"#include "strata/core/sm120_series.hpp"+#include "strata/core/gfx12_platform_defaults.hpp"#if defined(__HIPCC__)#include "moe_f16_common.cuh"@@ -1417,7 +1418,8 @@ bool native_supported(int gu_type, int d_type) {return d.ok && d.s120 && d.sms >= 80 && !d.t75 && d_covered_on(d_type, d);}#endif- if (env < 0 && gu_type != T_IQ3_S && gu_type != T_IQ3_XXS && gu_type != T_Q2_0) return false; // default: only these three+ // default: only these three; the Q2_0 one (148-amd) is on Linux only: on Windows it is opt-in (STRATA_PF_FUSED=1) until tested there+ if (env < 0 && gu_type != T_IQ3_S && gu_type != T_IQ3_XXS && !(gu_type == T_Q2_0 && strata::core::kGfx12PromptDefaultsOn)) return false;const DevInfo& d = dev_info();if (!d.ok || !d_covered_on(d_type, d)) return false;if (d.t75 && (gu_type != T_Q2_0 || d_type != T_Q2_0)) return false;