Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

gfx12: the four bit-identical decode switches back on (ATTN_LANECELL, ATTN_MERGE_V2, TSUM, EXPDEC), measured on p620 R9700 IQ3_S 8K: +3.5% decode, 3/3 pairs, answer md5 identical; docs: the PLE residency policy and its numbers; ple_reader_test: table_bytes

Niko12213fd5a748c786
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.md
index f806f4238..16f152d95 100644
--- a/docs/AMD_HIP.md
+++ b/docs/AMD_HIP.md
@@ -274,6 +274,7 @@ them in the server config's `env`, or in the shell.
| `STRATA_HIP_ROUTER_OLD=1` | all HIP | Off | Run the portable MoE router kernel instead of the fast one |
| `STRATA_PREFILL_WARM=0` | all | On | Turn off the warm-up BF16 and FP16 products run while the expert arena loads |
| `STRATA_HIP_QUEUE_PRIME=0` | gfx1200, gfx1201, Linux | On | Turn off the ordered opening of two GPU queues at start (it avoids slow starts) |
+| `STRATA_PLE_AUTO_RAM=0` | all | On where the RAM fits | Keep the n-gram table in RAM when `--ple-io` was not given (RDNA4: 8K prompt +22.5%, see [RDNA4](#rdna4-gfx1201)); `=0` keeps the direct reads |
| `STRATA_TOPK_OLD=1`, `STRATA_TOPK_STREAM=0` | HIP | Off | Run the original top-k kernel, or the register kernel in the 90K-135K band |
### Opt-in chunked prompt GDN on gfx11
@@ -372,6 +373,15 @@ signed dot4 instruction and a 100 MHz wall clock. Switch defaults for gfx12 are
arena loads so the first request does not carry them (`STRATA_COLD_TRACE=1` times them).
- **Pictures:** the card's reader (`strata-vision-hip`) keeps about 1.4 GiB of VRAM. It has not run on an RDNA3 card
(gfx1100-gfx1102) or on Windows with an AMD card.
+- **PLE table residency (0.2.7):** when `--ple-io` is not given and the machine has room for the n-gram table plus
+ 8 GiB, the engine keeps the table in RAM (`--ple-io ram`: map, pre-touch, mlock attempt; `STRATA_PLE_AUTO_RAM=0`
+ turns the policy off, an explicit `--ple-io` always wins). The rows are random reads in a 26.8 GiB table, so this
+ is the one prompt lever that paid on this card. Measured on p620 (R9700 32 GB, IQ3_S, engine 0.1.51, 24576 expert
+ pairs of which 13207 resident): 8K prompt 2045 tok/s with direct reads, 2276 with `mmap`, **2506 with the table in
+ RAM (+22.5%)**; 32K prompt 2522 against 2754 (+9.2%). Decode unchanged, answer md5 identical; the project's
+ `ple_reader_selftest` proves the mapped rows are the same bytes as the direct ones. The table is 26.8 GiB
+ (320,001,536 rows of 90 B), so a start reads it once: about 3 s when it is in the file cache, up to a minute
+ from a cold disk, and it needs the RAM for it while the engine runs.
- **Not validated:** answer-quality benchmarks and contexts beyond 16K on the 9070 XT.
> **Measured on:** Radeon AI PRO R9700 32 GB and RX 9070 XT 16 GB (gfx1201), Ryzen 9 3900X, 47 GB RAM, Ubuntu 24.04
diff --git a/src/core/arch_defaults.cpp b/src/core/arch_defaults.cpp
index abe40e5c9..efacbdad0 100644
--- a/src/core/arch_defaults.cpp
+++ b/src/core/arch_defaults.cpp
@@ -12,8 +12,12 @@ namespace strata::core {
std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gcn_arch) {
std::vector<std::pair<std::string, std::string>> t;
if (strata::kernels::gfx_arch_is_gfx12(gcn_arch)) {
- // gfx1200 / gfx1201 (RDNA4): only the PROMPT half of the gfx1151 table. Reported exact (identical answers) on an R9700
- // in #1478 and re-checked bitwise here (notes-142-amd.md); the decode half measured slower on that card, so it stays out.
+ // gfx1200 / gfx1201 (RDNA4): the PROMPT half of the gfx1151 table (reported exact on an R9700 in #1478, re-checked
+ // bitwise in notes-142-amd.md) plus four decode switches measured one at a time on an R9700 in 0.2.7 (p620, IQ3_S,
+ // engine 0.1.51, 8K prompt, rotated pairs): ATTN_LANECELL + ATTN_MERGE_V2 +1.9%, TSUM +1.9%, EXPDEC +0.6%, GDN_SPLIT
+ // 0.0%; the four together +3.5% (3/3 pairs) with the answer md5 identical to the default. The rest of the gfx1151
+ // decode half stays out (PF_GEMM/PF_FUSED are bit-changing; Q8_PACKED, SH_STREAM and QFUSE were measured on other
+ // cards only, and the whole half measured slower here when it was tried as one block).
// STRATA_HCD_EXACT is left out: it needs the gfx1151 hipBLASLt tuning table, finds none on gfx12, says so on every start
// and does nothing (measured: with and without it identical, 0.0 to +0.1%).
// STRATA_GDN_CONVL2 is AMD-only on purpose: it is not bitwise on CUDA 13.3 sm_120 (#1522), and this table is never
@@ -26,6 +30,11 @@ std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gc
{"STRATA_GDN_CONVL2", "1"},
{"STRATA_GDN_NOY", "1"},
{"STRATA_CVEC_FUSE", "1"},
+ // decode (bit-identical on the R9700 measurement above)
+ {"STRATA_ATTN_LANECELL", "1"},
+ {"STRATA_ATTN_MERGE_V2", "1"},
+ {"STRATA_TSUM", "1"},
+ {"STRATA_EXPDEC", "1"},
};
return t;
}
diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cpp
index 5313f429b..25a84ff26 100644
--- a/src/core/arch_defaults_test.cpp
+++ b/src/core/arch_defaults_test.cpp
@@ -45,11 +45,19 @@ int main() {
CHECK(kv.first != bits);
}
}
- // gfx12: the five prompt switches only, no decode switch, no bit-changing one
+ // gfx12: the five prompt switches and four bit-identical decode switches, no bit-changing one
unset("STRATA_GFX12_DEFAULTS");
for (const char* a : {"gfx1200", "gfx1201", "gfx1201:sramecc-:xnack-"}) {
const auto t = arch_default_env(a);
- CHECK(t.size() == 5);
+ CHECK(t.size() == 9);
+ bool attn_lane = false, attn_merge = false, tsum = false, expdec = false;
+ for (const auto& kv : t) {
+ if (kv.first == "STRATA_ATTN_LANECELL") attn_lane = kv.second == "1";
+ if (kv.first == "STRATA_ATTN_MERGE_V2") attn_merge = kv.second == "1";
+ if (kv.first == "STRATA_TSUM") tsum = kv.second == "1";
+ if (kv.first == "STRATA_EXPDEC") expdec = kv.second == "1";
+ }
+ CHECK(attn_lane && attn_merge && tsum && expdec); // the 0.2.7 R9700 decode measurement, bit-identical
for (const auto& kv : t) CHECK(kv.first != "STRATA_HCD_EXACT");
for (const auto& kv : t) {
CHECK(kv.first != "STRATA_Q8_PACKED" && kv.first != "STRATA_SH_STREAM" && kv.first != "STRATA_PF_GEMM" &&
diff --git a/src/ngram/ple_reader_test.cpp b/src/ngram/ple_reader_test.cpp
index 4123d77df..f49422c6a 100644
--- a/src/ngram/ple_reader_test.cpp
+++ b/src/ngram/ple_reader_test.cpp
@@ -168,6 +168,10 @@ void format_round_trip(const std::string& dir, const k::PleFormatInfo& f, uint32
if (g_fail) break;
CHECK(std::strcmp(t.format(), f.name) == 0, "%s: format() says \"%s\"", f.name, t.format());
CHECK(t.rows() == rows, "%s: rows %llu, not %u", f.name, (unsigned long long) t.rows(), rows);
+ // The residency policy in generate.cpp sizes the table with table_bytes(); it must be the file's own
+ // row count x row bytes in both modes, and 0 once the table is closed.
+ CHECK(t.table_bytes() == (uint64_t) rows * f.row_bytes, "%s: %s: table_bytes %llu, not %u x %u", f.name, name,
+ (unsigned long long) t.table_bytes(), rows, f.row_bytes);
std::vector<uint8_t> raw(f.row_bytes);
float got[k::PLE_HEAD_DIM], ref[k::PLE_HEAD_DIM];
for (uint32_t r : want) {