Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Merge f-151-amd (d9070189c) into f-split-int2

Niko122158f81f20e367
diff --cc docs/AMD_HIP.md
index 11448a3dc,f5df3cef1..eccdb1ee3
--- a/docs/AMD_HIP.md
+++ b/docs/AMD_HIP.md
@@@ -434,12 -434,26 +434,32 @@@ an RX 9070 XT 16 GB and a Radeon AI PR
1,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode
89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);
its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.
+ **Default on gfx12 (148-amd).** Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical):
+ teacher-forced KL against the old kernel through the real serve decode path (STRATA_FOLLOW, 28 prompts, 6,024 positions) 0.0003 (p99 0.006, top-1
+ 99.8%, perplexity of the forced text x1.0003), and 0.0014 over 1,760 positions of eight 6K-token documents (top-1 98.9%, x1.0009); HumanEval 157/164
+ (default 156 and 156), reasoning 25/30 (24 and 26), agent 6/6, needles at 32K and 125K x three depths 6/6, CJK 8/8, tool calls 5/5. Prompt speed,
+ interleaved pairs on a quiet R9700, Q2_0 32 GB: see the 148-amd table below. The first-token KL against the FP16 prompt path cannot rank it (0.0296 against 0.0280 for the old
+ kernel, a paired difference of +0.0016 +- 0.0026): on a 2-bit pack any change of the prompt numerics, even `--prefill 4096`, moves the first token by about 0.025.
+ - `STRATA_HIP_WMMA=auto`: the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of
+ `STRATA_HIP_WMMA_MIN_Q` (64) or more queries it runs the matrix-core kernel, everything else the FP32 one (`=1` forces it
- on every call, `=0` or unset keeps the FP32 kernel; the default has not moved). A launch error switches the matrix-core
++ on every call and on gfx11, `=0` keeps the FP32 kernel; unset is the gfx12 default above: the matrix-core kernel on every call). A launch error switches the matrix-core
+ kernel off for the rest of the process and the FP32 kernel runs (before, the engine exited). Kernel timing on an R9700
+ (`hip_prompt_attn_wmma`, context 1.5K-32K, 64-2,048 queries): 6.9-8.4x the FP32 kernel at every shape, error against FP64
+ 3e-6 of the output scale (FP32: 2e-6). Engine, Q2_0 pack, 6 interleaved pairs, equal slots, fixed flags
+ (`--pcie-frac 0 --adapt-every 0`, `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`): 20K prompt x1.46 (`=1`) /
+ x1.54 (`auto`), 4K x1.33 / x1.34 (6/6 pairs; the noise arm of that run read +6.4-6.9%, so read them as +40-50% and
+ +30%); the 8-token answers were the default's in every run.
+ - The first prompt after a start pays two one-time hipBLAS library loads (the first BF16 product 1.2 s, the first FP16
+ product 1.0 s on an R9700 with the gfx120X wheel's hipBLASLt 1.2.0; `STRATA_COLD_TRACE=1` times the first 80 dense
+ projections to show them). The engine now runs one tiny BF16 and one tiny FP16 product on a thread of its own while the
+ expert arena loads, so the first request does not carry them (`STRATA_PREFILL_WARM=0` turns it off; the outputs go to
+ scratch buffers, nothing changes; start-up time is the same, the arena load takes longer than the thread). First
+ 721-token prompt after the 33-token warm request, 5 interleaved pairs: 154 -> 673 tok/s. The next 4K prompt reads
+ ~1,090 tok/s either way (the "cold" first-prompt numbers above are about this).
+ - Ship status of `tools/hip/gfx1201-hipblaslt-*.txt`: the gfx120X wheel's hipBLASLt is 1.2.0 (version number 100200); the
+ tables here are 100202 and 100500, so the engine says "version mismatch ... using hipBLASEx" and runs plain hipBLAS. A
+ table tuned on the R9700 for 100200 (32 rows, `tune_hipblaslt`) measured within noise end to end (20K prompt x1.035 vs
+ a noise arm x1.031), so none is shipped: the dense projections are ~10% of a 20K prompt here.
- The prompt path's QSA top-k picks its kernel by the blocks a query actually has, not the cache's capacity (#337,
bsorensen110): the same ids, on by default with AMD (NVIDIA keeps its capacity rule: there the 64K prompts read
1-3% slower with it). `STRATA_SELECT_WMMA=1` (opt-in, gfx12) adds #337's
diff --cc src/kernels/cuda/qsa_prompt_attn.cu
index f5397e650,66e09ecd0..e1b3efa18
--- a/src/kernels/cuda/qsa_prompt_attn.cu
+++ b/src/kernels/cuda/qsa_prompt_attn.cu
@@@ -2175,33 -2176,58 +2176,62 @@@ __global__ void __launch_bounds__(THREA
#endif
}
-// gfx12 (RDNA4) and gfx11 (RDNA3 / RDNA3.5), only on request: the output differs from the default kernel's in its
-// last bits. STRATA_HIP_WMMA=1 forces it on every call; =auto takes it where it was measured to beat the FP32 kernel
-// (hip_wmma_auto_pick: a gfx12 card and a batch of STRATA_HIP_WMMA_MIN_Q queries or more; other calls keep the FP32
-// kernel); unset or =0 keeps the FP32 kernel. A launch that fails switches the path off for the rest of the process
-// (the caller then runs the FP32 kernel) instead of ending it.
-int hip_wmma_mode_env() {
+// gfx12 (RDNA4): on by default (148-amd; STRATA_HIP_WMMA=0 turns it off). gfx11 (RDNA3 / RDNA3.5): only on request
- // (STRATA_HIP_WMMA=1). The output differs from the old kernel's in its last bits (FP32-level accuracy, another summation
- // order): teacher-forced KL against the old kernel 0.0003 (28 prompts) / 0.0014 (8 long documents) on an R9700.
- bool hip_wmma_usable() {
- static const int env = [] { // -1 unset, 0 off, 1 on
++// (STRATA_HIP_WMMA=1). STRATA_HIP_WMMA=auto (151-amd) takes the matrix-core kernel on a gfx12 card for a batch of
++// STRATA_HIP_WMMA_MIN_Q (64) or more queries and keeps the FP32 kernel for smaller calls. The output differs from the
++// old kernel's in its last bits (FP32-level accuracy, another summation order): teacher-forced KL against the old kernel
++// 0.0003 (28 prompts) / 0.0014 (8 long documents) on an R9700. A launch that fails switches the path off for the rest of
++// the process (the caller then runs the FP32 kernel) instead of ending it.
++int hip_wmma_mode_env() { // -1 unset (the arch default), 0 off, 1 forced, 2 auto
+ static const int mode = [] {
const char* e = std::getenv("STRATA_HIP_WMMA");
- return e == nullptr || e[0] == '\0' ? -1 : (e[0] == '1' ? 1 : 0);
- if (e == nullptr) return 0;
++ if (e == nullptr || e[0] == '\0') return -1;
+ if (e[0] == '1') return 1;
+ if (!std::strcmp(e, "auto")) return 2;
+ return 0;
}();
- if (env == 0) return false;
- static int arch[64] = {}; // 0 unknown, 1 gfx12, 2 other, 3 gfx11 matrix cores
- int dev = 0;
- if (hipGetDevice(&dev) != hipSuccess || dev < 0 || dev >= 64) { (void) hipGetLastError(); return false; }
+ return mode;
+ }
+ std::atomic<bool> g_wmma_failed{false};
+ // the device's class: 1 gfx11 matrix-core part, 2 gfx12 (RDNA4), 3 anything else
+ int hip_wmma_arch(int dev) {
+ static int arch[64] = {};
+ if (dev < 0 || dev >= 64) return 3;
if (arch[dev] == 0) {
hipDeviceProp_t prop{};
- if (hipGetDeviceProperties(&prop, dev) != hipSuccess) { (void) hipGetLastError(); return false; }
- arch[dev] = std::strncmp(prop.gcnArchName, "gfx12", 5) == 0 ? 1
- : strata::kernels::gfx_arch_is_gfx11_wmma(prop.gcnArchName) ? 3 : 2;
+ if (hipGetDeviceProperties(&prop, dev) != hipSuccess) { (void) hipGetLastError(); return 3; }
+ arch[dev] = std::strncmp(prop.gcnArchName, "gfx12", 5) == 0 ? 2
+ : strata::kernels::gfx_arch_is_gfx11_wmma(prop.gcnArchName) ? 1 : 3;
++ const int mode = hip_wmma_mode_env();
static bool told = false;
- if (!told && (env == 1 || arch[dev] == 1)) {
- if (!told) {
++ if (!told && (mode >= 1 || arch[dev] == 2)) {
told = true;
- const bool on = arch[dev] == 1 || (arch[dev] == 3 && env == 1);
- std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", env == 1 ? "STRATA_HIP_WMMA: " : "",
- on ? "on" : "unavailable", prop.gcnArchName,
- env == 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");
- const int mode = hip_wmma_mode_env();
- std::fprintf(stderr, "strata: STRATA_HIP_WMMA=%s: the prompt attention on matrix cores %s (%s)\n",
- mode == 2 ? "auto" : "1", arch[dev] == 3 ? "unavailable" : (mode == 2 ? "chosen per call" : "on"),
- prop.gcnArchName);
++ const bool on = arch[dev] == 2 || (arch[dev] == 1 && mode >= 1);
++ std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", mode >= 1 ? "STRATA_HIP_WMMA: " : "",
++ !on ? "unavailable" : (mode == 2 ? "chosen per call" : "on"), prop.gcnArchName,
++ mode >= 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");
}
}
- return arch[dev] == 1 || (arch[dev] == 3 && env == 1);
+ return arch[dev];
+ }
+ // auto: where the matrix-core kernel beats the FP32 one. Measured on the R9700 (gfx1201) only; the gfx11 parts take
+ // it only when forced with =1.
+ bool hip_wmma_auto_pick(int arch, int64_t n_q, int64_t cap) {
+ if (arch != 2) return false;
+ static const int64_t min_q = [] { const char* e = std::getenv("STRATA_HIP_WMMA_MIN_Q"); return e ? (int64_t) std::atoll(e) : (int64_t) 64; }();
+ static const int64_t min_cap = [] { const char* e = std::getenv("STRATA_HIP_WMMA_MIN_CAP"); return e ? (int64_t) std::atoll(e) : (int64_t) 0; }();
+ return n_q >= min_q && cap >= min_cap;
+ }
+ bool hip_wmma_usable(int64_t n_q, int64_t cap) {
+ const int mode = hip_wmma_mode_env();
+ if (mode == 0 || g_wmma_failed.load(std::memory_order_relaxed)) return false;
+ int dev = 0;
+ if (hipGetDevice(&dev) != hipSuccess) { (void) hipGetLastError(); return false; }
+ const int arch = hip_wmma_arch(dev);
+ if (arch == 3) return false;
- return mode == 1 || hip_wmma_auto_pick(arch, n_q, cap);
++ if (mode == 1) return true;
++ if (mode == 2) return hip_wmma_auto_pick(arch, n_q, cap);
++ return arch == 2; // unset: the gfx12 default
}
bool launch_wmma(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,
@@@ -2286,15 -2313,16 +2317,19 @@@ bool qsa_decode_attn_split(const float
#endif
}
void qsa_prompt_attn_set_imma(int on) { g_pa_imma = on < 0 ? -1 : (on ? 1 : 0); }
-
- bool qsa_hip_wmma_active() {
+ bool qsa_prompt_attn_matrix_cores_requested() {
#if defined(__HIPCC__)
- return hip_wmma_usable();
+ int dev = 0;
+ if (hipGetDevice(&dev) != hipSuccess) { (void) hipGetLastError(); return false; }
- return hip_wmma_mode_env() != 0 && hip_wmma_arch(dev) != 3 && !g_wmma_failed.load(std::memory_order_relaxed);
++ const int mode = hip_wmma_mode_env(), arch = hip_wmma_arch(dev);
++ return mode != 0 && arch != 3 && !g_wmma_failed.load(std::memory_order_relaxed) && (mode >= 1 || arch == 2);
#else
return false;
#endif
}
++bool qsa_hip_wmma_active() { return qsa_prompt_attn_matrix_cores_requested(); }
++
bool qsa_prompt_attn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps,
int64_t cap, const QsaShapes& s, float* attn, int64_t n_q, void* stream) {
if (n_q <= 0) return true;
diff --cc src/prefill/prefill.cpp
index f145736fc,df1909c91..5319731dc
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@@ -308,9 -308,10 +308,10 @@@ inline int ring_budget_slots()
inline int ring_slots(size_t T) {
const char* v = std::getenv("STRATA_PREFILL_RING");
#if defined(STRATA_USE_HIP)
- // S6: with the RDNA4 matrix-core attention (the gfx12 default since 148-amd; STRATA_HIP_WMMA=1 elsewhere) a 96-slot
- // ring: measured with it, 9070 XT 4K prompts 718 -> 1,211 tok/s (16K 1,949 -> 2,032), R9700 4K 2,426 -> 2,483 (16K the same)
- if (!v && g_ring_override <= 0 && strata::kernels::qsa_hip_wmma_active()) return (int64_t) T >= stream_all_min() ? 96 : STAGE;
- // S6: with the matrix-core attention (STRATA_HIP_WMMA=1 or auto, on a card that has it) a 96-slot ring: measured with it, 9070 XT
- // 4K prompts 718 -> 1,211 tok/s (16K 1,949 -> 2,032), R9700 4K 2,426 -> 2,483 (16K the same)
++ // S6: with the matrix-core attention (the gfx12 default since 148-amd; STRATA_HIP_WMMA=1 or auto elsewhere / forced, on a card that
++ // has it) a 96-slot ring: measured with it, 9070 XT 4K prompts 718 -> 1,211 tok/s (16K 1,949 -> 2,032), R9700 4K 2,426 -> 2,483 (16K the same)
+ static const bool wmma = strata::kernels::qsa_prompt_attn_matrix_cores_requested();
+ if (!v && g_ring_override <= 0 && wmma) return (int64_t) T >= stream_all_min() ? 96 : STAGE;
#endif
// the byte budget as this pack's slots: 1024 fused / 384 not on Q2_0, fewer on a pack with bigger blobs. The
// unpinned arm stays a slot count (96): it was measured where the host copies are the limit, and there the ring
diff --cc src/program/generate.cpp
index b9d601623,531b85725..780015c67
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@@ -5373,8 -5358,15 +5377,10 @@@ int main(int argc, char** argv)
const char* name = named && p.name[0] ? p.name : "(an unnamed GPU)";
#if defined(STRATA_USE_HIP)
std::fprintf(stderr, "strata generate: GPU %d: %s (%s)\n", dev, name, named ? p.gcnArchName : "?");
+ // the first prompt's one-time hipBLAS library loads (about 2.5 s), done on a thread while the expert arena loads
+ if (named) g_warm_thr = std::thread([dev] { strata::prefill::warm_gemm_libraries(dev); });
- // A hint only, never an action: on RDNA4 with an int8 KV the opt-in matrix-core prompt attention reads prompts
- // about 30% faster (R9700, PR #1318), but its output differs from the default kernel's in the last bits, so it
- // stays the user's choice. Silent when the variable is set to anything (STRATA_HIP_WMMA=0 included).
- if (named && o.kv == "int8" && std::strncmp(p.gcnArchName, "gfx12", 5) == 0 && std::getenv("STRATA_HIP_WMMA") == nullptr)
- std::fprintf(stderr, "strata generate: hint: STRATA_HIP_WMMA=1 reads prompts about 30%% faster on this card with "
- "--kv int8 (prompt attention on the matrix cores); it is off by default because the output "
- "bits change (last-place rounding). Nothing was enabled.\n");
+ // (RDNA4's matrix-core prompt attention is the default since 148-amd: the engine says so once, at its first prompt;
+ // STRATA_HIP_WMMA=0 turns it off.)
#if defined(_WIN32)
// #468 #461: which HIP runtime was loaded - the bundled one beside the exe, or an AMD driver's System32 copy
if (HMODULE h = GetModuleHandleA("amdhip64_7.dll")) {