Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge f-151-amd (d9070189c) into f-split-int2
Niko122158f81f20e367
diff --cc docs/AMD_HIP.mdindex 11448a3dc,f5df3cef1..eccdb1ee3--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@@ -434,12 -434,26 +434,32 @@@ an RX 9070 XT 16 GB and a Radeon AI PR1,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.+ **Default on gfx12 (148-amd).** Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical):+ teacher-forced KL against the old kernel through the real serve decode path (STRATA_FOLLOW, 28 prompts, 6,024 positions) 0.0003 (p99 0.006, top-1+ 99.8%, perplexity of the forced text x1.0003), and 0.0014 over 1,760 positions of eight 6K-token documents (top-1 98.9%, x1.0009); HumanEval 157/164+ (default 156 and 156), reasoning 25/30 (24 and 26), agent 6/6, needles at 32K and 125K x three depths 6/6, CJK 8/8, tool calls 5/5. Prompt speed,+ interleaved pairs on a quiet R9700, Q2_0 32 GB: see the 148-amd table below. The first-token KL against the FP16 prompt path cannot rank it (0.0296 against 0.0280 for the old+ kernel, a paired difference of +0.0016 +- 0.0026): on a 2-bit pack any change of the prompt numerics, even `--prefill 4096`, moves the first token by about 0.025.+ - `STRATA_HIP_WMMA=auto`: the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of+ `STRATA_HIP_WMMA_MIN_Q` (64) or more queries it runs the matrix-core kernel, everything else the FP32 one (`=1` forces it- on every call, `=0` or unset keeps the FP32 kernel; the default has not moved). A launch error switches the matrix-core++ on every call and on gfx11, `=0` keeps the FP32 kernel; unset is the gfx12 default above: the matrix-core kernel on every call). A launch error switches the matrix-core+ kernel off for the rest of the process and the FP32 kernel runs (before, the engine exited). Kernel timing on an R9700+ (`hip_prompt_attn_wmma`, context 1.5K-32K, 64-2,048 queries): 6.9-8.4x the FP32 kernel at every shape, error against FP64+ 3e-6 of the output scale (FP32: 2e-6). Engine, Q2_0 pack, 6 interleaved pairs, equal slots, fixed flags+ (`--pcie-frac 0 --adapt-every 0`, `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`): 20K prompt x1.46 (`=1`) /+ x1.54 (`auto`), 4K x1.33 / x1.34 (6/6 pairs; the noise arm of that run read +6.4-6.9%, so read them as +40-50% and+ +30%); the 8-token answers were the default's in every run.+ - The first prompt after a start pays two one-time hipBLAS library loads (the first BF16 product 1.2 s, the first FP16+ product 1.0 s on an R9700 with the gfx120X wheel's hipBLASLt 1.2.0; `STRATA_COLD_TRACE=1` times the first 80 dense+ projections to show them). The engine now runs one tiny BF16 and one tiny FP16 product on a thread of its own while the+ expert arena loads, so the first request does not carry them (`STRATA_PREFILL_WARM=0` turns it off; the outputs go to+ scratch buffers, nothing changes; start-up time is the same, the arena load takes longer than the thread). First+ 721-token prompt after the 33-token warm request, 5 interleaved pairs: 154 -> 673 tok/s. The next 4K prompt reads+ ~1,090 tok/s either way (the "cold" first-prompt numbers above are about this).+ - Ship status of `tools/hip/gfx1201-hipblaslt-*.txt`: the gfx120X wheel's hipBLASLt is 1.2.0 (version number 100200); the+ tables here are 100202 and 100500, so the engine says "version mismatch ... using hipBLASEx" and runs plain hipBLAS. A+ table tuned on the R9700 for 100200 (32 rows, `tune_hipblaslt`) measured within noise end to end (20K prompt x1.035 vs+ a noise arm x1.031), so none is shipped: the dense projections are ~10% of a 20K prompt here.- The prompt path's QSA top-k picks its kernel by the blocks a query actually has, not the cache's capacity (#337,bsorensen110): the same ids, on by default with AMD (NVIDIA keeps its capacity rule: there the 64K prompts read1-3% slower with it). `STRATA_SELECT_WMMA=1` (opt-in, gfx12) adds #337'sdiff --cc src/kernels/cuda/qsa_prompt_attn.cuindex f5397e650,66e09ecd0..e1b3efa18--- a/src/kernels/cuda/qsa_prompt_attn.cu+++ b/src/kernels/cuda/qsa_prompt_attn.cu@@@ -2175,33 -2176,58 +2176,62 @@@ __global__ void __launch_bounds__(THREA#endif}-// gfx12 (RDNA4) and gfx11 (RDNA3 / RDNA3.5), only on request: the output differs from the default kernel's in its-// last bits. STRATA_HIP_WMMA=1 forces it on every call; =auto takes it where it was measured to beat the FP32 kernel-// (hip_wmma_auto_pick: a gfx12 card and a batch of STRATA_HIP_WMMA_MIN_Q queries or more; other calls keep the FP32-// kernel); unset or =0 keeps the FP32 kernel. A launch that fails switches the path off for the rest of the process-// (the caller then runs the FP32 kernel) instead of ending it.-int hip_wmma_mode_env() {+// gfx12 (RDNA4): on by default (148-amd; STRATA_HIP_WMMA=0 turns it off). gfx11 (RDNA3 / RDNA3.5): only on request- // (STRATA_HIP_WMMA=1). The output differs from the old kernel's in its last bits (FP32-level accuracy, another summation- // order): teacher-forced KL against the old kernel 0.0003 (28 prompts) / 0.0014 (8 long documents) on an R9700.- bool hip_wmma_usable() {- static const int env = [] { // -1 unset, 0 off, 1 on++// (STRATA_HIP_WMMA=1). STRATA_HIP_WMMA=auto (151-amd) takes the matrix-core kernel on a gfx12 card for a batch of++// STRATA_HIP_WMMA_MIN_Q (64) or more queries and keeps the FP32 kernel for smaller calls. The output differs from the++// old kernel's in its last bits (FP32-level accuracy, another summation order): teacher-forced KL against the old kernel++// 0.0003 (28 prompts) / 0.0014 (8 long documents) on an R9700. A launch that fails switches the path off for the rest of++// the process (the caller then runs the FP32 kernel) instead of ending it.++int hip_wmma_mode_env() { // -1 unset (the arch default), 0 off, 1 forced, 2 auto+ static const int mode = [] {const char* e = std::getenv("STRATA_HIP_WMMA");- return e == nullptr || e[0] == '\0' ? -1 : (e[0] == '1' ? 1 : 0);- if (e == nullptr) return 0;++ if (e == nullptr || e[0] == '\0') return -1;+ if (e[0] == '1') return 1;+ if (!std::strcmp(e, "auto")) return 2;+ return 0;}();- if (env == 0) return false;- static int arch[64] = {}; // 0 unknown, 1 gfx12, 2 other, 3 gfx11 matrix cores- int dev = 0;- if (hipGetDevice(&dev) != hipSuccess || dev < 0 || dev >= 64) { (void) hipGetLastError(); return false; }+ return mode;+ }+ std::atomic<bool> g_wmma_failed{false};+ // the device's class: 1 gfx11 matrix-core part, 2 gfx12 (RDNA4), 3 anything else+ int hip_wmma_arch(int dev) {+ static int arch[64] = {};+ if (dev < 0 || dev >= 64) return 3;if (arch[dev] == 0) {hipDeviceProp_t prop{};- if (hipGetDeviceProperties(&prop, dev) != hipSuccess) { (void) hipGetLastError(); return false; }- arch[dev] = std::strncmp(prop.gcnArchName, "gfx12", 5) == 0 ? 1- : strata::kernels::gfx_arch_is_gfx11_wmma(prop.gcnArchName) ? 3 : 2;+ if (hipGetDeviceProperties(&prop, dev) != hipSuccess) { (void) hipGetLastError(); return 3; }+ arch[dev] = std::strncmp(prop.gcnArchName, "gfx12", 5) == 0 ? 2+ : strata::kernels::gfx_arch_is_gfx11_wmma(prop.gcnArchName) ? 1 : 3;++ const int mode = hip_wmma_mode_env();static bool told = false;- if (!told && (env == 1 || arch[dev] == 1)) {- if (!told) {++ if (!told && (mode >= 1 || arch[dev] == 2)) {told = true;- const bool on = arch[dev] == 1 || (arch[dev] == 3 && env == 1);- std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", env == 1 ? "STRATA_HIP_WMMA: " : "",- on ? "on" : "unavailable", prop.gcnArchName,- env == 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");- const int mode = hip_wmma_mode_env();- std::fprintf(stderr, "strata: STRATA_HIP_WMMA=%s: the prompt attention on matrix cores %s (%s)\n",- mode == 2 ? "auto" : "1", arch[dev] == 3 ? "unavailable" : (mode == 2 ? "chosen per call" : "on"),- prop.gcnArchName);++ const bool on = arch[dev] == 2 || (arch[dev] == 1 && mode >= 1);++ std::fprintf(stderr, "strata: %sthe prompt attention on matrix cores %s (%s)%s\n", mode >= 1 ? "STRATA_HIP_WMMA: " : "",++ !on ? "unavailable" : (mode == 2 ? "chosen per call" : "on"), prop.gcnArchName,++ mode >= 1 ? "" : " by default; STRATA_HIP_WMMA=0 turns it off");}}- return arch[dev] == 1 || (arch[dev] == 3 && env == 1);+ return arch[dev];+ }+ // auto: where the matrix-core kernel beats the FP32 one. Measured on the R9700 (gfx1201) only; the gfx11 parts take+ // it only when forced with =1.+ bool hip_wmma_auto_pick(int arch, int64_t n_q, int64_t cap) {+ if (arch != 2) return false;+ static const int64_t min_q = [] { const char* e = std::getenv("STRATA_HIP_WMMA_MIN_Q"); return e ? (int64_t) std::atoll(e) : (int64_t) 64; }();+ static const int64_t min_cap = [] { const char* e = std::getenv("STRATA_HIP_WMMA_MIN_CAP"); return e ? (int64_t) std::atoll(e) : (int64_t) 0; }();+ return n_q >= min_q && cap >= min_cap;+ }+ bool hip_wmma_usable(int64_t n_q, int64_t cap) {+ const int mode = hip_wmma_mode_env();+ if (mode == 0 || g_wmma_failed.load(std::memory_order_relaxed)) return false;+ int dev = 0;+ if (hipGetDevice(&dev) != hipSuccess) { (void) hipGetLastError(); return false; }+ const int arch = hip_wmma_arch(dev);+ if (arch == 3) return false;- return mode == 1 || hip_wmma_auto_pick(arch, n_q, cap);++ if (mode == 1) return true;++ if (mode == 2) return hip_wmma_auto_pick(arch, n_q, cap);++ return arch == 2; // unset: the gfx12 default}bool launch_wmma(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,@@@ -2286,15 -2313,16 +2317,19 @@@ bool qsa_decode_attn_split(const float#endif}void qsa_prompt_attn_set_imma(int on) { g_pa_imma = on < 0 ? -1 : (on ? 1 : 0); }-- bool qsa_hip_wmma_active() {+ bool qsa_prompt_attn_matrix_cores_requested() {#if defined(__HIPCC__)- return hip_wmma_usable();+ int dev = 0;+ if (hipGetDevice(&dev) != hipSuccess) { (void) hipGetLastError(); return false; }- return hip_wmma_mode_env() != 0 && hip_wmma_arch(dev) != 3 && !g_wmma_failed.load(std::memory_order_relaxed);++ const int mode = hip_wmma_mode_env(), arch = hip_wmma_arch(dev);++ return mode != 0 && arch != 3 && !g_wmma_failed.load(std::memory_order_relaxed) && (mode >= 1 || arch == 2);#elsereturn false;#endif}++bool qsa_hip_wmma_active() { return qsa_prompt_attn_matrix_cores_requested(); }++bool qsa_prompt_attn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps,int64_t cap, const QsaShapes& s, float* attn, int64_t n_q, void* stream) {if (n_q <= 0) return true;diff --cc src/prefill/prefill.cppindex f145736fc,df1909c91..5319731dc--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@@ -308,9 -308,10 +308,10 @@@ inline int ring_budget_slots()inline int ring_slots(size_t T) {const char* v = std::getenv("STRATA_PREFILL_RING");#if defined(STRATA_USE_HIP)- // S6: with the RDNA4 matrix-core attention (the gfx12 default since 148-amd; STRATA_HIP_WMMA=1 elsewhere) a 96-slot- // ring: measured with it, 9070 XT 4K prompts 718 -> 1,211 tok/s (16K 1,949 -> 2,032), R9700 4K 2,426 -> 2,483 (16K the same)- if (!v && g_ring_override <= 0 && strata::kernels::qsa_hip_wmma_active()) return (int64_t) T >= stream_all_min() ? 96 : STAGE;- // S6: with the matrix-core attention (STRATA_HIP_WMMA=1 or auto, on a card that has it) a 96-slot ring: measured with it, 9070 XT- // 4K prompts 718 -> 1,211 tok/s (16K 1,949 -> 2,032), R9700 4K 2,426 -> 2,483 (16K the same)++ // S6: with the matrix-core attention (the gfx12 default since 148-amd; STRATA_HIP_WMMA=1 or auto elsewhere / forced, on a card that++ // has it) a 96-slot ring: measured with it, 9070 XT 4K prompts 718 -> 1,211 tok/s (16K 1,949 -> 2,032), R9700 4K 2,426 -> 2,483 (16K the same)+ static const bool wmma = strata::kernels::qsa_prompt_attn_matrix_cores_requested();+ if (!v && g_ring_override <= 0 && wmma) return (int64_t) T >= stream_all_min() ? 96 : STAGE;#endif// the byte budget as this pack's slots: 1024 fused / 384 not on Q2_0, fewer on a pack with bigger blobs. The// unpinned arm stays a slot count (96): it was measured where the host copies are the limit, and there the ringdiff --cc src/program/generate.cppindex b9d601623,531b85725..780015c67--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@@ -5373,8 -5358,15 +5377,10 @@@ int main(int argc, char** argv)const char* name = named && p.name[0] ? p.name : "(an unnamed GPU)";#if defined(STRATA_USE_HIP)std::fprintf(stderr, "strata generate: GPU %d: %s (%s)\n", dev, name, named ? p.gcnArchName : "?");+ // the first prompt's one-time hipBLAS library loads (about 2.5 s), done on a thread while the expert arena loads+ if (named) g_warm_thr = std::thread([dev] { strata::prefill::warm_gemm_libraries(dev); });- // A hint only, never an action: on RDNA4 with an int8 KV the opt-in matrix-core prompt attention reads prompts- // about 30% faster (R9700, PR #1318), but its output differs from the default kernel's in the last bits, so it- // stays the user's choice. Silent when the variable is set to anything (STRATA_HIP_WMMA=0 included).- if (named && o.kv == "int8" && std::strncmp(p.gcnArchName, "gfx12", 5) == 0 && std::getenv("STRATA_HIP_WMMA") == nullptr)- std::fprintf(stderr, "strata generate: hint: STRATA_HIP_WMMA=1 reads prompts about 30%% faster on this card with "- "--kv int8 (prompt attention on the matrix cores); it is off by default because the output "- "bits change (last-place rounding). Nothing was enabled.\n");+ // (RDNA4's matrix-core prompt attention is the default since 148-amd: the engine says so once, at its first prompt;+ // STRATA_HIP_WMMA=0 turns it off.)#if defined(_WIN32)// #468 #461: which HIP runtime was loaded - the bundled one beside the exe, or an AMD driver's System32 copyif (HMODULE h = GetModuleHandleA("amdhip64_7.dll")) {