Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
gfx12: the four bit-identical decode switches back on (ATTN_LANECELL, ATTN_MERGE_V2, TSUM, EXPDEC), measured on p620 R9700 IQ3_S 8K: +3.5% decode, 3/3 pairs, answer md5 identical; docs: the PLE residency policy and its numbers; ple_reader_test: table_bytes
Niko12213fd5a748c786
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex f806f4238..16f152d95 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -274,6 +274,7 @@ them in the server config's `env`, or in the shell.| `STRATA_HIP_ROUTER_OLD=1` | all HIP | Off | Run the portable MoE router kernel instead of the fast one || `STRATA_PREFILL_WARM=0` | all | On | Turn off the warm-up BF16 and FP16 products run while the expert arena loads || `STRATA_HIP_QUEUE_PRIME=0` | gfx1200, gfx1201, Linux | On | Turn off the ordered opening of two GPU queues at start (it avoids slow starts) |+| `STRATA_PLE_AUTO_RAM=0` | all | On where the RAM fits | Keep the n-gram table in RAM when `--ple-io` was not given (RDNA4: 8K prompt +22.5%, see [RDNA4](#rdna4-gfx1201)); `=0` keeps the direct reads || `STRATA_TOPK_OLD=1`, `STRATA_TOPK_STREAM=0` | HIP | Off | Run the original top-k kernel, or the register kernel in the 90K-135K band |### Opt-in chunked prompt GDN on gfx11@@ -372,6 +373,15 @@ signed dot4 instruction and a 100 MHz wall clock. Switch defaults for gfx12 arearena loads so the first request does not carry them (`STRATA_COLD_TRACE=1` times them).- **Pictures:** the card's reader (`strata-vision-hip`) keeps about 1.4 GiB of VRAM. It has not run on an RDNA3 card(gfx1100-gfx1102) or on Windows with an AMD card.+- **PLE table residency (0.2.7):** when `--ple-io` is not given and the machine has room for the n-gram table plus+ 8 GiB, the engine keeps the table in RAM (`--ple-io ram`: map, pre-touch, mlock attempt; `STRATA_PLE_AUTO_RAM=0`+ turns the policy off, an explicit `--ple-io` always wins). The rows are random reads in a 26.8 GiB table, so this+ is the one prompt lever that paid on this card. Measured on p620 (R9700 32 GB, IQ3_S, engine 0.1.51, 24576 expert+ pairs of which 13207 resident): 8K prompt 2045 tok/s with direct reads, 2276 with `mmap`, **2506 with the table in+ RAM (+22.5%)**; 32K prompt 2522 against 2754 (+9.2%). Decode unchanged, answer md5 identical; the project's+ `ple_reader_selftest` proves the mapped rows are the same bytes as the direct ones. The table is 26.8 GiB+ (320,001,536 rows of 90 B), so a start reads it once: about 3 s when it is in the file cache, up to a minute+ from a cold disk, and it needs the RAM for it while the engine runs.- **Not validated:** answer-quality benchmarks and contexts beyond 16K on the 9070 XT.> **Measured on:** Radeon AI PRO R9700 32 GB and RX 9070 XT 16 GB (gfx1201), Ryzen 9 3900X, 47 GB RAM, Ubuntu 24.04diff --git a/src/core/arch_defaults.cpp b/src/core/arch_defaults.cppindex abe40e5c9..efacbdad0 100644--- a/src/core/arch_defaults.cpp+++ b/src/core/arch_defaults.cpp@@ -12,8 +12,12 @@ namespace strata::core {std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gcn_arch) {std::vector<std::pair<std::string, std::string>> t;if (strata::kernels::gfx_arch_is_gfx12(gcn_arch)) {- // gfx1200 / gfx1201 (RDNA4): only the PROMPT half of the gfx1151 table. Reported exact (identical answers) on an R9700- // in #1478 and re-checked bitwise here (notes-142-amd.md); the decode half measured slower on that card, so it stays out.+ // gfx1200 / gfx1201 (RDNA4): the PROMPT half of the gfx1151 table (reported exact on an R9700 in #1478, re-checked+ // bitwise in notes-142-amd.md) plus four decode switches measured one at a time on an R9700 in 0.2.7 (p620, IQ3_S,+ // engine 0.1.51, 8K prompt, rotated pairs): ATTN_LANECELL + ATTN_MERGE_V2 +1.9%, TSUM +1.9%, EXPDEC +0.6%, GDN_SPLIT+ // 0.0%; the four together +3.5% (3/3 pairs) with the answer md5 identical to the default. The rest of the gfx1151+ // decode half stays out (PF_GEMM/PF_FUSED are bit-changing; Q8_PACKED, SH_STREAM and QFUSE were measured on other+ // cards only, and the whole half measured slower here when it was tried as one block).// STRATA_HCD_EXACT is left out: it needs the gfx1151 hipBLASLt tuning table, finds none on gfx12, says so on every start// and does nothing (measured: with and without it identical, 0.0 to +0.1%).// STRATA_GDN_CONVL2 is AMD-only on purpose: it is not bitwise on CUDA 13.3 sm_120 (#1522), and this table is never@@ -26,6 +30,11 @@ std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gc{"STRATA_GDN_CONVL2", "1"},{"STRATA_GDN_NOY", "1"},{"STRATA_CVEC_FUSE", "1"},+ // decode (bit-identical on the R9700 measurement above)+ {"STRATA_ATTN_LANECELL", "1"},+ {"STRATA_ATTN_MERGE_V2", "1"},+ {"STRATA_TSUM", "1"},+ {"STRATA_EXPDEC", "1"},};return t;}diff --git a/src/core/arch_defaults_test.cpp b/src/core/arch_defaults_test.cppindex 5313f429b..25a84ff26 100644--- a/src/core/arch_defaults_test.cpp+++ b/src/core/arch_defaults_test.cpp@@ -45,11 +45,19 @@ int main() {CHECK(kv.first != bits);}}- // gfx12: the five prompt switches only, no decode switch, no bit-changing one+ // gfx12: the five prompt switches and four bit-identical decode switches, no bit-changing oneunset("STRATA_GFX12_DEFAULTS");for (const char* a : {"gfx1200", "gfx1201", "gfx1201:sramecc-:xnack-"}) {const auto t = arch_default_env(a);- CHECK(t.size() == 5);+ CHECK(t.size() == 9);+ bool attn_lane = false, attn_merge = false, tsum = false, expdec = false;+ for (const auto& kv : t) {+ if (kv.first == "STRATA_ATTN_LANECELL") attn_lane = kv.second == "1";+ if (kv.first == "STRATA_ATTN_MERGE_V2") attn_merge = kv.second == "1";+ if (kv.first == "STRATA_TSUM") tsum = kv.second == "1";+ if (kv.first == "STRATA_EXPDEC") expdec = kv.second == "1";+ }+ CHECK(attn_lane && attn_merge && tsum && expdec); // the 0.2.7 R9700 decode measurement, bit-identicalfor (const auto& kv : t) CHECK(kv.first != "STRATA_HCD_EXACT");for (const auto& kv : t) {CHECK(kv.first != "STRATA_Q8_PACKED" && kv.first != "STRATA_SH_STREAM" && kv.first != "STRATA_PF_GEMM" &&diff --git a/src/ngram/ple_reader_test.cpp b/src/ngram/ple_reader_test.cppindex 4123d77df..f49422c6a 100644--- a/src/ngram/ple_reader_test.cpp+++ b/src/ngram/ple_reader_test.cpp@@ -168,6 +168,10 @@ void format_round_trip(const std::string& dir, const k::PleFormatInfo& f, uint32if (g_fail) break;CHECK(std::strcmp(t.format(), f.name) == 0, "%s: format() says \"%s\"", f.name, t.format());CHECK(t.rows() == rows, "%s: rows %llu, not %u", f.name, (unsigned long long) t.rows(), rows);+ // The residency policy in generate.cpp sizes the table with table_bytes(); it must be the file's own+ // row count x row bytes in both modes, and 0 once the table is closed.+ CHECK(t.table_bytes() == (uint64_t) rows * f.row_bytes, "%s: %s: table_bytes %llu, not %u x %u", f.name, name,+ (unsigned long long) t.table_bytes(), rows, f.row_bytes);std::vector<uint8_t> raw(f.row_bytes);float got[k::PLE_HEAD_DIM], ref[k::PLE_HEAD_DIM];for (uint32_t r : want) {