Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Radeon 780M: the matrix-core prompt set is the default on gfx1103 (integrated, Linux): prompts x1.6 at 1K and x2.0 at 4K, decode unchanged, same greedy ids; STRATA_IGPU_PROMPT_SET=0 turns it off; gfx1150/1151 and Windows stay off; docs note the MES reset issue; per-arch tests

Niko1221eb7eac250b75
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.md
index 84a97bdd1..5d12089c0 100644
--- a/docs/AMD_HIP.md
+++ b/docs/AMD_HIP.md
@@ -546,8 +546,8 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39
## Integrated GPUs (mini PCs)
On a Radeon that shares the computer's memory (the Ryzen mini PCs: Radeon 780M / 760M `gfx1103`, 890M / 880M `gfx1150`,
-Strix Halo `gfx1151`) the matrix-core prompt set below is opt-in (`STRATA_IGPU_PROMPT_SET=1`): it changes bits, so it waits for the maintainer's
-decision before it becomes the default. With the opt-in the engine picks the set from the device's own "integrated" flag and
+Strix Halo `gfx1151`) the matrix-core prompt set below is on by default on the Radeon 780M (`gfx1103`, Linux) and opt-in
+(`STRATA_IGPU_PROMPT_SET=1`) on the others: it changes bits, and only the 780M was measured. The engine picks the set from the device's own "integrated" flag and
the architecture, not from the name: a discrete card (an RX 7900 XTX `gfx1100`, an R9700 `gfx1201`, any other) never gets it,
and neither does a computer that also holds a discrete card next to the iGPU. The engine prints one line at start-up that names
what it set.
@@ -560,7 +560,7 @@ what it set.
| `STRATA_SELECT_WMMA=1` | the block scorer on the matrix cores |
| `STRATA_HC_UPMIX=1` | the hyper-connection up projection with its epilogue |
-**Why.** The dense projections are 63% of a prompt's GPU time on a 12-CU iGPU (the experts only 16%), and they are what these
+**Why.** (148-amd first measurement; 021-minipc repeated it with 6 interleaved pairs: prompt x1.60 at 1K, x2.03 at 4K, decode x0.999, identical greedy ids.) The dense projections are 63% of a prompt's GPU time on a 12-CU iGPU (the experts only 16%), and they are what these
switches speed up. Radeon 780M (P2006, Ryzen 7 255, Q2_0, `--kv int8`, 4K prompt, one run each; 148-amd): 109.5 tok/s with
none of them, 214.7 with the first four (+96%), 233 with `STRATA_HC_UPMIX` as well (+111%); decode is the same (16.6 against 17-19
tok/s, inside the run-to-run noise). The prompt switches do not touch the decode kernels.
@@ -569,7 +569,7 @@ tok/s, inside the run-to-run noise). The prompt switches do not touch the decode
differ in the last bits. Measured on an R9700 with the same kernels (Q2_0): teacher-forced KL 0.0003 (28 prompts) / 0.0014 (eight
~6K-token documents), and the task sets inside their run-to-run noise.
-**Turning it on and off.** `STRATA_IGPU_PROMPT_SET=1` turns the whole set on; each switch keeps `=0` as its off switch
+**Turning it on and off.** `STRATA_IGPU_PROMPT_SET=0` turns the whole set off, `=1` turns it on where it is not the default; each switch keeps `=0` as its off switch
(`STRATA_HC_UPMIX=0`); `STRATA_IGPU_DEFAULTS=0` turns the set off whatever else is set. A switch you set yourself, to anything, is never overridden. Put them in the server config's `env`, or in the shell.
**Windows.** The same code applies, but it has not been tested on Windows (there was no Windows iGPU machine to run it on).
diff --git a/docs/RADEON_780M.md b/docs/RADEON_780M.md
index c74f8fb22..a8cc89a89 100644
--- a/docs/RADEON_780M.md
+++ b/docs/RADEON_780M.md
@@ -79,10 +79,19 @@ greedy tokens, 3 of 3 pairs), with the greedy token ids unchanged. They are the
| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | decode layouts and kernels (packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores) |
| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE`, `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH`, `STRATA_SH_STREAM` | grouped decode experts, butterfly warp sums, fewer launches, the GDN step over four blocks per head, producer-written activation images, batched PLE projections, the shared expert on its own stream |
-## 5. The matrix-core prompt set (opt-in on an integrated GPU: `STRATA_IGPU_PROMPT_SET=1`; it changes bits)
+## 5. The matrix-core prompt set (on by default on the 780M; it changes bits)
-Opt-in (AMD_HIP.md, "Integrated GPUs"); with `STRATA_IGPU_PROMPT_SET=1` the engine sets `STRATA_HIP_WMMA`, `STRATA_PF_GEMM`, `STRATA_PA_FAST`,
-`STRATA_SELECT_WMMA` and `STRATA_HC_UPMIX` itself on an integrated Radeon; `NAME=0` turns one off. `STRATA_PF_FUSED` stays opt-in.
+On by default on the Radeon 780M (`gfx1103`, integrated, Linux; AMD_HIP.md, "Integrated GPUs"): the engine sets `STRATA_HIP_WMMA`, `STRATA_PF_GEMM`,
+`STRATA_PA_FAST`, `STRATA_SELECT_WMMA` and `STRATA_HC_UPMIX` itself. `STRATA_IGPU_PROMPT_SET=0` turns the whole set off, `NAME=0` one switch. The 880M / 890M
+(`gfx1150`) and Windows keep it off (no hardware / not tested); `STRATA_IGPU_PROMPT_SET=1` turns it on there. `STRATA_PF_FUSED` stays opt-in.
+
+Measured (021-minipc, P2006, Q2_0, fresh engine per run, 10 interleaved rounds, 6 complete pairs): prompt 1K 98.5 -> 157.3 tok/s (x1.60, 6/6 pairs faster),
+4K 115.1 -> 233.3 (x2.03, 6/6), decode 22.02 -> 21.99 (code) and 15.09 -> 15.05 (story) (x0.999), greedy token ids identical in 6 of 6 pairs.
+
+**GPU resets.** The 780M's current Ubuntu firmware (gc 11.0.1 MES) can still reset the GPU now and then, with or without this set: when the kernel moves memory that the
+GPU has registered, the process's queues are evicted, and a kernel that sits in a barrier at that moment never answers. The engine recovers
+("device wedged, but recovered through reset") but the running request fails. In the same runs the default engine crashed 4 of 10 times and the set 0 of 10, so it did
+not add resets. Until AMD fixes the firmware a mini PC may still see them; the engine does not touch the firmware. A newer `linux-firmware` is the thing to try.
The prompt levers need the table to pay: with the table, the WMMA set
read 248 tok/s at 4K against 186 without it, and at 32K 272 against 178. No KL was measured on this chip.
diff --git a/docs/STRIX_HALO.md b/docs/STRIX_HALO.md
index 9228d15f1..8ab2d222c 100644
--- a/docs/STRIX_HALO.md
+++ b/docs/STRIX_HALO.md
@@ -98,7 +98,7 @@ they change speed, not answers. Nothing here applies to CUDA or to another AMD a
## 5. What is not on by default (it changes bits)
-**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only.** `STRATA_PF_FUSED`, `STRATA_HC_Q8`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.
+**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_HC_Q8`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.
These are faster on gfx1151 and **round differently** from the default path (a different summation order or a half-precision
intermediate), so they are opt-in and the maintainers gate them with KL against the default (`STRATA_PF_SWITCH_MIN_T=4096` keeps
diff --git a/include/strata/core/arch_defaults.hpp b/include/strata/core/arch_defaults.hpp
index b836867a8..71d05df39 100644
--- a/include/strata/core/arch_defaults.hpp
+++ b/include/strata/core/arch_defaults.hpp
@@ -25,11 +25,12 @@ std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gc
/// matrix-core prompt set is offered (opt-in; 021-minipc). Pure architecture-name test; the caller adds the UMA test.
bool igpu_prompt_defaults_arch(const char* gcn_arch);
-/// 021-minipc: the matrix-core prompt set changes bits, so it is OFF by default until the owner approves it. Flip this constant
-/// (one line) to make it the default of the integrated Radeons; STRATA_IGPU_PROMPT_SET=1 / =0 switches it per run either way,
-/// and STRATA_IGPU_DEFAULTS=0 turns it off whatever the constant says.
-inline constexpr bool kIgpuPromptSetDefaultOn = false;
-bool igpu_prompt_set_enabled();
+/// 021-minipc: the matrix-core prompt set changes bits. The owner approved it as the DEFAULT of the Radeon 780M only (gfx1103,
+/// integrated, Linux): prompts 1.60x (1K) / 2.03x (4K), decode within 0.1%, greedy ids identical in 6 of 6 pairs (notes-021-minipc.md).
+/// gfx1150 / gfx1151 (no hardware) and Windows keep it off by default. STRATA_IGPU_PROMPT_SET=1 turns it on for any integrated
+/// arch of igpu_prompt_defaults_arch, STRATA_IGPU_PROMPT_SET=0 (or STRATA_IGPU_DEFAULTS=0) turns it off.
+inline constexpr bool kIgpuPromptSetDefaultOn = true;
+bool igpu_prompt_set_enabled(const char* gcn_arch, bool windows);
/// 021-minipc: UMA mode (the expert arena stays unregistered). Exact (no bit changes) but NOT a default: measured on the 780M it was
/// 1-4% slower and did not stop the resets (4 of 20 runs against 1 of 10), so kIgpuUmaArenaDefaultOn is false. When true it applies
diff --git a/src/core/arch_defaults.cpp b/src/core/arch_defaults.cpp
index fe534c65b..fcdb65010 100644
--- a/src/core/arch_defaults.cpp
+++ b/src/core/arch_defaults.cpp
@@ -64,13 +64,13 @@ bool igpu_prompt_defaults_arch(const char* gcn_arch) {
strata::kernels::gfx_arch_is(gcn_arch, "gfx1151");
}
-bool igpu_prompt_set_enabled() {
+bool igpu_prompt_set_enabled(const char* gcn_arch, bool windows) {
const char* off = std::getenv("STRATA_IGPU_DEFAULTS");
if (off != nullptr && off[0] == '0') return false;
const char* e = std::getenv("STRATA_IGPU_PROMPT_SET");
if (e != nullptr && e[0] == '1') return true;
if (e != nullptr && e[0] == '0') return false;
- return kIgpuPromptSetDefaultOn;
+ return kIgpuPromptSetDefaultOn && !windows && strata::kernels::gfx_arch_is(gcn_arch, "gfx1103");
}
bool igpu_uma_arena_default(const char* gcn_arch, bool integrated, bool windows, bool on) {
@@ -82,7 +82,12 @@ bool igpu_uma_arena_default(const char* gcn_arch, bool integrated, bool windows,
std::vector<std::pair<std::string, std::string>> igpu_default_env(const char* gcn_arch, bool integrated) {
std::vector<std::pair<std::string, std::string>> t;
if (!integrated || !igpu_prompt_defaults_arch(gcn_arch)) return t;
- if (!igpu_prompt_set_enabled()) return t;
+#if defined(_WIN32)
+ const bool win = true;
+#else
+ const bool win = false;
+#endif
+ if (!igpu_prompt_set_enabled(gcn_arch, win)) return t;
// Measured on a Radeon 780M (gfx1103, P2006, Q2_0, notes-153-igpu.md / notes-148-amd.md): prompts 109.5 -> 214.7 tok/s
// with the first four, 233 with HC_UPMIX, decode unchanged. Dense projections are 63% of a prompt's GPU time there.
t = {
@@ -126,7 +131,7 @@ std::vector<std::string> apply_arch_defaults(const char* gcn_arch, bool integrat
igpu.push_back(kv.first);
}
if (!igpu.empty()) {
- std::fprintf(stderr, "strata: integrated GPU (%s): the matrix-core prompt set is on (about 2x faster prompts on a "
+ std::fprintf(stderr, "strata: integrated GPU (%s): the matrix-core prompt set is on (1.6x to 2x faster prompts on a "
"Radeon 780M, the answers differ in the last bits): ", gcn_arch);
for (size_t i = 0; i < igpu.size(); ++i) std::fprintf(stderr, "%s%s", i ? " " : "", igpu[i].c_str() + 7);
std::fprintf(stderr, ". Turn one off with NAME=0 (for example STRATA_HC_UPMIX=0), all of them with STRATA_IGPU_PROMPT_SET=0");
diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cpp
index dfea2c4fa..c471bbf47 100644
--- a/src/core/arch_defaults_test.cpp
+++ b/src/core/arch_defaults_test.cpp
@@ -101,18 +101,41 @@ int main() {
};
unset("STRATA_IGPU_DEFAULTS");
unset("STRATA_IGPU_PROMPT_SET");
- // 021-minipc: the set changes bits, so with nothing set it is OFF everywhere (the owner flips kIgpuPromptSetDefaultOn)
- CHECK(!kIgpuPromptSetDefaultOn);
- CHECK(!igpu_prompt_set_enabled());
- for (const Mock& m : mocks) CHECK(igpu_default_env(m.gcn, m.integrated).empty());
+ // 021-minipc: with nothing set the prompt set is the default of the Radeon 780M only (gfx1103, integrated, not Windows)
+ CHECK(kIgpuPromptSetDefaultOn);
+ CHECK(igpu_prompt_set_enabled("gfx1103", false));
+ CHECK(igpu_prompt_set_enabled("gfx1103:sramecc-:xnack-", false));
+ CHECK(!igpu_prompt_set_enabled("gfx1103", true)); // Windows: not tested
+ for (const char* g : {"gfx1150", "gfx1151", "gfx1100", "gfx1201", "gfx1152", "", "gfx11030"}) CHECK(!igpu_prompt_set_enabled(g, false));
+ CHECK(!igpu_prompt_set_enabled(nullptr, false));
+ for (const Mock& m : mocks) {
+ const bool want = m.expect_on && std::string(m.gcn).rfind("gfx1103", 0) == 0;
+#if defined(_WIN32)
+ CHECK(igpu_default_env(m.gcn, m.integrated).empty());
+#else
+ CHECK(igpu_default_env(m.gcn, m.integrated).size() == (want ? 5u : 0u));
+#endif
+ }
+#if !defined(_WIN32)
+ for (const char* k : {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_SELECT_WMMA", "STRATA_HC_UPMIX"}) unset(k);
+ CHECK(apply_arch_defaults("gfx1103", true).size() == 5); // a 780M gets the set with nothing set
+ for (const char* k : {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_SELECT_WMMA", "STRATA_HC_UPMIX"}) unset(k);
+ CHECK(apply_arch_defaults("gfx1150", true).empty()); // 880M / 890M: no hardware, stays as it was
for (const char* k : {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_SELECT_WMMA", "STRATA_HC_UPMIX"}) unset(k);
- CHECK(apply_arch_defaults("gfx1103", true).empty());
- CHECK(std::getenv("STRATA_HIP_WMMA") == nullptr);
- // opt-in per run: STRATA_IGPU_PROMPT_SET=1; the rest of the checks below run with it on
+#endif
+ // the off switches
+ put("STRATA_IGPU_PROMPT_SET", "0");
+ CHECK(!igpu_prompt_set_enabled("gfx1103", false));
+ CHECK(igpu_default_env("gfx1103", true).empty());
+ unset("STRATA_IGPU_PROMPT_SET");
+ put("STRATA_IGPU_DEFAULTS", "0");
+ CHECK(!igpu_prompt_set_enabled("gfx1103", false));
+ unset("STRATA_IGPU_DEFAULTS");
+ // opt-in per run for the other arches: STRATA_IGPU_PROMPT_SET=1; the checks below run with it on (every mock arch of the list)
put("STRATA_IGPU_PROMPT_SET", "1");
- CHECK(igpu_prompt_set_enabled());
+ CHECK(igpu_prompt_set_enabled("gfx1150", false));
put("STRATA_IGPU_DEFAULTS", "0");
- CHECK(!igpu_prompt_set_enabled()); // the whole-set off switch wins
+ CHECK(!igpu_prompt_set_enabled("gfx1150", false)); // the whole-set off switch wins
unset("STRATA_IGPU_DEFAULTS");
for (const Mock& m : mocks) {
const auto t = igpu_default_env(m.gcn, m.integrated);