Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
amd: open the GPU queues in a fixed order at start (R9700 fast/slow prompt starts)
Niko12213f1fa67f145e
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex cf9e27ba4..59c37b7fd 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -501,6 +501,17 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39scratch buffers, nothing changes; start-up time is the same, the arena load takes longer than the thread). First721-token prompt after the 33-token warm request, 5 interleaved pairs: 154 -> 673 tok/s. The next 4K prompt reads~1,090 tok/s either way (the "cold" first-prompt numbers above are about this).+ - **Fast and slow starts (the "bimodal" prompt speed, found in 0.2.1).** On an R9700 / RX 9000 under Linux about half of+ the fresh starts with MTP read prompts 10-23% slower than the others (p620, Q2_0: 4K 1,980 vs 2,290 tok/s, 20K 2,480 vs+ 2,770; decode was never affected). The cause is where the kernel's scheduler puts the process's GPU queues. The prompt+ path has two busy queues, the compute stream and the stream that copies experts to the card (about 31,000 copies and+ event hand-offs per 4K prompt). The runtime opens its queues in the order the loader threads reach them, which changes+ every start; when both busy queues land on the same compute pipe of the card (the card has two for this process), every+ hand-off waits for a queue switch. The engine now opens five queues (four at normal priority, one at the highest) one+ after the other on the main thread before any other thread starts (`STRATA_HIP_QUEUE_PRIME=0` turns it off; any other+ value is a list of `n` / `h` letters for tests). gfx1200 and gfx1201 on Linux only; nothing else changes (the answers+ are identical). Not clocks, not memory placement (read speed over the expert cache was 637-640 GB/s in both kinds of+ start), not CPU placement, not the MTP stream itself.- Ship status of `tools/hip/gfx1201-hipblaslt-*.txt`: the gfx120X wheel's hipBLASLt is 1.2.0 (version number 100200); thetables here are 100202 and 100500, so the engine says "version mismatch ... using hipBLASEx" and runs plain hipBLAS. Atable tuned on the R9700 for 100200 (32 rows, `tune_hipblaslt`) measured within noise end to end (20K prompt x1.035 vsdiff --git a/src/program/generate.cpp b/src/program/generate.cppindex 01b99f06c..97c6cb48a 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -2211,6 +2211,59 @@ static double ple_direct_rows_per_s(strata::kernels::PleTable& t) {return el > 0 ? (double) done / el : -1.0;}+#if defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906)+// The R9700 / RX 9000 "bimodal prompt speed" (0.2.0 known limit). The runtime opens the process's hardware queues+// lazily, as the first stream of each kind is used, and the kernel's scheduler (MES) puts a queue on one of the+// card's compute pipes when the queue first gets work. The loader threads reach their streams in a different order+// every start, so in about half of the starts the prompt path's two busy queues (the compute stream and the expert+// staging stream) landed on the SAME pipe: every one of the ~30,000 hand-offs of a prompt then waits for the pipe to+// switch queues, and a 4K prompt reads 13% slower (p620, 1,980 vs 2,290 tok/s). Opening the queues here, one after+// the other on the main thread and before any other thread exists, makes the placement the same in every start (and+// it is the fast one). Streams are destroyed again; the runtime keeps the queues.+// STRATA_HIP_QUEUE_PRIME unset: gfx1200 / gfx1201 on Linux only, "nnnnh"; "0": off; any other string is a spec+// ('n' = a stream at the default priority, 'h' = at the highest) for experiments on other cards.+static void prime_hw_queues() {+ const char* env = std::getenv("STRATA_HIP_QUEUE_PRIME");+ if (env != nullptr && std::strcmp(env, "0") == 0) return;+ const char* pv = std::getenv("STRATA_PRIMARY_DEVICE");+ const int dev = pv != nullptr && std::atoi(pv) > 0 ? std::atoi(pv) : 0;+ int n_dev = 0;+ if (cudaGetDeviceCount(&n_dev) != cudaSuccess || dev >= n_dev) { (void) cudaGetLastError(); return; }+ const char* spec = env != nullptr && env[0] != 0 ? env : nullptr;+ if (spec == nullptr) {+#if defined(_WIN32)+ return;+#else+ cudaDeviceProp prop{};+ if (cudaGetDeviceProperties(&prop, dev) != cudaSuccess || !strata::kernels::gfx_arch_is_gfx12(prop.gcnArchName)) {+ (void) cudaGetLastError();+ return;+ }+ spec = "nnnnh";+#endif+ }+ if (dev != 0 && cudaSetDevice(dev) != cudaSuccess) { (void) cudaGetLastError(); return; }+ void* buf = nullptr;+ if (cudaMalloc(&buf, 256) != cudaSuccess) { (void) cudaGetLastError(); return; }+ int lo = 0, hi = 0;+ const bool have_prio = hipDeviceGetStreamPriorityRange(&lo, &hi) == hipSuccess;+ std::vector<cudaStream_t> ss;+ for (const char* c = spec; *c != 0; ++c) {+ cudaStream_t s = nullptr;+ const bool ok = (*c == 'h' && have_prio) ? hipStreamCreateWithPriority(&s, hipStreamNonBlocking, hi) == hipSuccess+ : cudaStreamCreateWithFlags(&s, cudaStreamNonBlocking) == cudaSuccess;+ if (!ok) continue;+ cudaMemsetAsync(buf, 0, 256, s);+ cudaStreamSynchronize(s);+ ss.push_back(s);+ }+ for (cudaStream_t s : ss) cudaStreamDestroy(s);+ cudaFree(buf);+ (void) cudaGetLastError();+ std::fprintf(stderr, "strata generate: opened %zu GPU queues in a fixed order (STRATA_HIP_QUEUE_PRIME=0 turns it off)\n", ss.size());+}+#endif+int main(int argc, char** argv) {// **UNBUFFERED, BECAUSE THE INTERESTING OUTPUT IS THE OUTPUT BEFORE A CRASH.** `stdout` redirected to a// pipe or a file is block-buffered, so a program that dies loses every line it had already printed - which@@ -2294,6 +2347,9 @@ int main(int argc, char** argv) {if (qwen35) return strata::qwen35::run(argc, argv);break;}+#endif+#if defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906)+ prime_hw_queues();#endifOptions o;bool have_tokens = false;