Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Radeon 780M: the matrix-core prompt set is the default on gfx1103 (integrated, Linux): prompts x1.6 at 1K and x2.0 at 4K, decode unchanged, same greedy ids; STRATA_IGPU_PROMPT_SET=0 turns it off; gfx1150/1151 and Windows stay off; docs note the MES reset issue; per-arch tests
Niko1221eb7eac250b75
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex 84a97bdd1..5d12089c0 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -546,8 +546,8 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39## Integrated GPUs (mini PCs)On a Radeon that shares the computer's memory (the Ryzen mini PCs: Radeon 780M / 760M `gfx1103`, 890M / 880M `gfx1150`,-Strix Halo `gfx1151`) the matrix-core prompt set below is opt-in (`STRATA_IGPU_PROMPT_SET=1`): it changes bits, so it waits for the maintainer's-decision before it becomes the default. With the opt-in the engine picks the set from the device's own "integrated" flag and+Strix Halo `gfx1151`) the matrix-core prompt set below is on by default on the Radeon 780M (`gfx1103`, Linux) and opt-in+(`STRATA_IGPU_PROMPT_SET=1`) on the others: it changes bits, and only the 780M was measured. The engine picks the set from the device's own "integrated" flag andthe architecture, not from the name: a discrete card (an RX 7900 XTX `gfx1100`, an R9700 `gfx1201`, any other) never gets it,and neither does a computer that also holds a discrete card next to the iGPU. The engine prints one line at start-up that nameswhat it set.@@ -560,7 +560,7 @@ what it set.| `STRATA_SELECT_WMMA=1` | the block scorer on the matrix cores || `STRATA_HC_UPMIX=1` | the hyper-connection up projection with its epilogue |-**Why.** The dense projections are 63% of a prompt's GPU time on a 12-CU iGPU (the experts only 16%), and they are what these+**Why.** (148-amd first measurement; 021-minipc repeated it with 6 interleaved pairs: prompt x1.60 at 1K, x2.03 at 4K, decode x0.999, identical greedy ids.) The dense projections are 63% of a prompt's GPU time on a 12-CU iGPU (the experts only 16%), and they are what theseswitches speed up. Radeon 780M (P2006, Ryzen 7 255, Q2_0, `--kv int8`, 4K prompt, one run each; 148-amd): 109.5 tok/s withnone of them, 214.7 with the first four (+96%), 233 with `STRATA_HC_UPMIX` as well (+111%); decode is the same (16.6 against 17-19tok/s, inside the run-to-run noise). The prompt switches do not touch the decode kernels.@@ -569,7 +569,7 @@ tok/s, inside the run-to-run noise). The prompt switches do not touch the decodediffer in the last bits. Measured on an R9700 with the same kernels (Q2_0): teacher-forced KL 0.0003 (28 prompts) / 0.0014 (eight~6K-token documents), and the task sets inside their run-to-run noise.-**Turning it on and off.** `STRATA_IGPU_PROMPT_SET=1` turns the whole set on; each switch keeps `=0` as its off switch+**Turning it on and off.** `STRATA_IGPU_PROMPT_SET=0` turns the whole set off, `=1` turns it on where it is not the default; each switch keeps `=0` as its off switch(`STRATA_HC_UPMIX=0`); `STRATA_IGPU_DEFAULTS=0` turns the set off whatever else is set. A switch you set yourself, to anything, is never overridden. Put them in the server config's `env`, or in the shell.**Windows.** The same code applies, but it has not been tested on Windows (there was no Windows iGPU machine to run it on).diff --git a/docs/RADEON_780M.md b/docs/RADEON_780M.mdindex c74f8fb22..a8cc89a89 100644--- a/docs/RADEON_780M.md+++ b/docs/RADEON_780M.md@@ -79,10 +79,19 @@ greedy tokens, 3 of 3 pairs), with the greedy token ids unchanged. They are the| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | decode layouts and kernels (packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores) || `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE`, `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH`, `STRATA_SH_STREAM` | grouped decode experts, butterfly warp sums, fewer launches, the GDN step over four blocks per head, producer-written activation images, batched PLE projections, the shared expert on its own stream |-## 5. The matrix-core prompt set (opt-in on an integrated GPU: `STRATA_IGPU_PROMPT_SET=1`; it changes bits)+## 5. The matrix-core prompt set (on by default on the 780M; it changes bits)-Opt-in (AMD_HIP.md, "Integrated GPUs"); with `STRATA_IGPU_PROMPT_SET=1` the engine sets `STRATA_HIP_WMMA`, `STRATA_PF_GEMM`, `STRATA_PA_FAST`,-`STRATA_SELECT_WMMA` and `STRATA_HC_UPMIX` itself on an integrated Radeon; `NAME=0` turns one off. `STRATA_PF_FUSED` stays opt-in.+On by default on the Radeon 780M (`gfx1103`, integrated, Linux; AMD_HIP.md, "Integrated GPUs"): the engine sets `STRATA_HIP_WMMA`, `STRATA_PF_GEMM`,+`STRATA_PA_FAST`, `STRATA_SELECT_WMMA` and `STRATA_HC_UPMIX` itself. `STRATA_IGPU_PROMPT_SET=0` turns the whole set off, `NAME=0` one switch. The 880M / 890M+(`gfx1150`) and Windows keep it off (no hardware / not tested); `STRATA_IGPU_PROMPT_SET=1` turns it on there. `STRATA_PF_FUSED` stays opt-in.++Measured (021-minipc, P2006, Q2_0, fresh engine per run, 10 interleaved rounds, 6 complete pairs): prompt 1K 98.5 -> 157.3 tok/s (x1.60, 6/6 pairs faster),+4K 115.1 -> 233.3 (x2.03, 6/6), decode 22.02 -> 21.99 (code) and 15.09 -> 15.05 (story) (x0.999), greedy token ids identical in 6 of 6 pairs.++**GPU resets.** The 780M's current Ubuntu firmware (gc 11.0.1 MES) can still reset the GPU now and then, with or without this set: when the kernel moves memory that the+GPU has registered, the process's queues are evicted, and a kernel that sits in a barrier at that moment never answers. The engine recovers+("device wedged, but recovered through reset") but the running request fails. In the same runs the default engine crashed 4 of 10 times and the set 0 of 10, so it did+not add resets. Until AMD fixes the firmware a mini PC may still see them; the engine does not touch the firmware. A newer `linux-firmware` is the thing to try.The prompt levers need the table to pay: with the table, the WMMA setread 248 tok/s at 4K against 186 without it, and at 32K 272 against 178. No KL was measured on this chip.diff --git a/docs/STRIX_HALO.md b/docs/STRIX_HALO.mdindex 9228d15f1..8ab2d222c 100644--- a/docs/STRIX_HALO.md+++ b/docs/STRIX_HALO.md@@ -98,7 +98,7 @@ they change speed, not answers. Nothing here applies to CUDA or to another AMD a## 5. What is not on by default (it changes bits)-**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only.** `STRATA_PF_FUSED`, `STRATA_HC_Q8`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.+**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_HC_Q8`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.These are faster on gfx1151 and **round differently** from the default path (a different summation order or a half-precisionintermediate), so they are opt-in and the maintainers gate them with KL against the default (`STRATA_PF_SWITCH_MIN_T=4096` keepsdiff --git a/include/strata/core/arch_defaults.hpp b/include/strata/core/arch_defaults.hppindex b836867a8..71d05df39 100644--- a/include/strata/core/arch_defaults.hpp+++ b/include/strata/core/arch_defaults.hpp@@ -25,11 +25,12 @@ std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gc/// matrix-core prompt set is offered (opt-in; 021-minipc). Pure architecture-name test; the caller adds the UMA test.bool igpu_prompt_defaults_arch(const char* gcn_arch);-/// 021-minipc: the matrix-core prompt set changes bits, so it is OFF by default until the owner approves it. Flip this constant-/// (one line) to make it the default of the integrated Radeons; STRATA_IGPU_PROMPT_SET=1 / =0 switches it per run either way,-/// and STRATA_IGPU_DEFAULTS=0 turns it off whatever the constant says.-inline constexpr bool kIgpuPromptSetDefaultOn = false;-bool igpu_prompt_set_enabled();+/// 021-minipc: the matrix-core prompt set changes bits. The owner approved it as the DEFAULT of the Radeon 780M only (gfx1103,+/// integrated, Linux): prompts 1.60x (1K) / 2.03x (4K), decode within 0.1%, greedy ids identical in 6 of 6 pairs (notes-021-minipc.md).+/// gfx1150 / gfx1151 (no hardware) and Windows keep it off by default. STRATA_IGPU_PROMPT_SET=1 turns it on for any integrated+/// arch of igpu_prompt_defaults_arch, STRATA_IGPU_PROMPT_SET=0 (or STRATA_IGPU_DEFAULTS=0) turns it off.+inline constexpr bool kIgpuPromptSetDefaultOn = true;+bool igpu_prompt_set_enabled(const char* gcn_arch, bool windows);/// 021-minipc: UMA mode (the expert arena stays unregistered). Exact (no bit changes) but NOT a default: measured on the 780M it was/// 1-4% slower and did not stop the resets (4 of 20 runs against 1 of 10), so kIgpuUmaArenaDefaultOn is false. When true it appliesdiff --git a/src/core/arch_defaults.cpp b/src/core/arch_defaults.cppindex fe534c65b..fcdb65010 100644--- a/src/core/arch_defaults.cpp+++ b/src/core/arch_defaults.cpp@@ -64,13 +64,13 @@ bool igpu_prompt_defaults_arch(const char* gcn_arch) {strata::kernels::gfx_arch_is(gcn_arch, "gfx1151");}-bool igpu_prompt_set_enabled() {+bool igpu_prompt_set_enabled(const char* gcn_arch, bool windows) {const char* off = std::getenv("STRATA_IGPU_DEFAULTS");if (off != nullptr && off[0] == '0') return false;const char* e = std::getenv("STRATA_IGPU_PROMPT_SET");if (e != nullptr && e[0] == '1') return true;if (e != nullptr && e[0] == '0') return false;- return kIgpuPromptSetDefaultOn;+ return kIgpuPromptSetDefaultOn && !windows && strata::kernels::gfx_arch_is(gcn_arch, "gfx1103");}bool igpu_uma_arena_default(const char* gcn_arch, bool integrated, bool windows, bool on) {@@ -82,7 +82,12 @@ bool igpu_uma_arena_default(const char* gcn_arch, bool integrated, bool windows,std::vector<std::pair<std::string, std::string>> igpu_default_env(const char* gcn_arch, bool integrated) {std::vector<std::pair<std::string, std::string>> t;if (!integrated || !igpu_prompt_defaults_arch(gcn_arch)) return t;- if (!igpu_prompt_set_enabled()) return t;+#if defined(_WIN32)+ const bool win = true;+#else+ const bool win = false;+#endif+ if (!igpu_prompt_set_enabled(gcn_arch, win)) return t;// Measured on a Radeon 780M (gfx1103, P2006, Q2_0, notes-153-igpu.md / notes-148-amd.md): prompts 109.5 -> 214.7 tok/s// with the first four, 233 with HC_UPMIX, decode unchanged. Dense projections are 63% of a prompt's GPU time there.t = {@@ -126,7 +131,7 @@ std::vector<std::string> apply_arch_defaults(const char* gcn_arch, bool integratigpu.push_back(kv.first);}if (!igpu.empty()) {- std::fprintf(stderr, "strata: integrated GPU (%s): the matrix-core prompt set is on (about 2x faster prompts on a "+ std::fprintf(stderr, "strata: integrated GPU (%s): the matrix-core prompt set is on (1.6x to 2x faster prompts on a ""Radeon 780M, the answers differ in the last bits): ", gcn_arch);for (size_t i = 0; i < igpu.size(); ++i) std::fprintf(stderr, "%s%s", i ? " " : "", igpu[i].c_str() + 7);std::fprintf(stderr, ". Turn one off with NAME=0 (for example STRATA_HC_UPMIX=0), all of them with STRATA_IGPU_PROMPT_SET=0");diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cppindex dfea2c4fa..c471bbf47 100644--- a/src/core/arch_defaults_test.cpp+++ b/src/core/arch_defaults_test.cpp@@ -101,18 +101,41 @@ int main() {};unset("STRATA_IGPU_DEFAULTS");unset("STRATA_IGPU_PROMPT_SET");- // 021-minipc: the set changes bits, so with nothing set it is OFF everywhere (the owner flips kIgpuPromptSetDefaultOn)- CHECK(!kIgpuPromptSetDefaultOn);- CHECK(!igpu_prompt_set_enabled());- for (const Mock& m : mocks) CHECK(igpu_default_env(m.gcn, m.integrated).empty());+ // 021-minipc: with nothing set the prompt set is the default of the Radeon 780M only (gfx1103, integrated, not Windows)+ CHECK(kIgpuPromptSetDefaultOn);+ CHECK(igpu_prompt_set_enabled("gfx1103", false));+ CHECK(igpu_prompt_set_enabled("gfx1103:sramecc-:xnack-", false));+ CHECK(!igpu_prompt_set_enabled("gfx1103", true)); // Windows: not tested+ for (const char* g : {"gfx1150", "gfx1151", "gfx1100", "gfx1201", "gfx1152", "", "gfx11030"}) CHECK(!igpu_prompt_set_enabled(g, false));+ CHECK(!igpu_prompt_set_enabled(nullptr, false));+ for (const Mock& m : mocks) {+ const bool want = m.expect_on && std::string(m.gcn).rfind("gfx1103", 0) == 0;+#if defined(_WIN32)+ CHECK(igpu_default_env(m.gcn, m.integrated).empty());+#else+ CHECK(igpu_default_env(m.gcn, m.integrated).size() == (want ? 5u : 0u));+#endif+ }+#if !defined(_WIN32)+ for (const char* k : {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_SELECT_WMMA", "STRATA_HC_UPMIX"}) unset(k);+ CHECK(apply_arch_defaults("gfx1103", true).size() == 5); // a 780M gets the set with nothing set+ for (const char* k : {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_SELECT_WMMA", "STRATA_HC_UPMIX"}) unset(k);+ CHECK(apply_arch_defaults("gfx1150", true).empty()); // 880M / 890M: no hardware, stays as it wasfor (const char* k : {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_SELECT_WMMA", "STRATA_HC_UPMIX"}) unset(k);- CHECK(apply_arch_defaults("gfx1103", true).empty());- CHECK(std::getenv("STRATA_HIP_WMMA") == nullptr);- // opt-in per run: STRATA_IGPU_PROMPT_SET=1; the rest of the checks below run with it on+#endif+ // the off switches+ put("STRATA_IGPU_PROMPT_SET", "0");+ CHECK(!igpu_prompt_set_enabled("gfx1103", false));+ CHECK(igpu_default_env("gfx1103", true).empty());+ unset("STRATA_IGPU_PROMPT_SET");+ put("STRATA_IGPU_DEFAULTS", "0");+ CHECK(!igpu_prompt_set_enabled("gfx1103", false));+ unset("STRATA_IGPU_DEFAULTS");+ // opt-in per run for the other arches: STRATA_IGPU_PROMPT_SET=1; the checks below run with it on (every mock arch of the list)put("STRATA_IGPU_PROMPT_SET", "1");- CHECK(igpu_prompt_set_enabled());+ CHECK(igpu_prompt_set_enabled("gfx1150", false));put("STRATA_IGPU_DEFAULTS", "0");- CHECK(!igpu_prompt_set_enabled()); // the whole-set off switch wins+ CHECK(!igpu_prompt_set_enabled("gfx1150", false)); // the whole-set off switch winsunset("STRATA_IGPU_DEFAULTS");for (const Mock& m : mocks) {const auto t = igpu_default_env(m.gcn, m.integrated);