Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

amd: open the GPU queues in a fixed order at start (R9700 fast/slow prompt starts)

Niko12213f1fa67f145e
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.md
index cf9e27ba4..59c37b7fd 100644
--- a/docs/AMD_HIP.md
+++ b/docs/AMD_HIP.md
@@ -501,6 +501,17 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39
scratch buffers, nothing changes; start-up time is the same, the arena load takes longer than the thread). First
721-token prompt after the 33-token warm request, 5 interleaved pairs: 154 -> 673 tok/s. The next 4K prompt reads
~1,090 tok/s either way (the "cold" first-prompt numbers above are about this).
+ - **Fast and slow starts (the "bimodal" prompt speed, found in 0.2.1).** On an R9700 / RX 9000 under Linux about half of
+ the fresh starts with MTP read prompts 10-23% slower than the others (p620, Q2_0: 4K 1,980 vs 2,290 tok/s, 20K 2,480 vs
+ 2,770; decode was never affected). The cause is where the kernel's scheduler puts the process's GPU queues. The prompt
+ path has two busy queues, the compute stream and the stream that copies experts to the card (about 31,000 copies and
+ event hand-offs per 4K prompt). The runtime opens its queues in the order the loader threads reach them, which changes
+ every start; when both busy queues land on the same compute pipe of the card (the card has two for this process), every
+ hand-off waits for a queue switch. The engine now opens five queues (four at normal priority, one at the highest) one
+ after the other on the main thread before any other thread starts (`STRATA_HIP_QUEUE_PRIME=0` turns it off; any other
+ value is a list of `n` / `h` letters for tests). gfx1200 and gfx1201 on Linux only; nothing else changes (the answers
+ are identical). Not clocks, not memory placement (read speed over the expert cache was 637-640 GB/s in both kinds of
+ start), not CPU placement, not the MTP stream itself.
- Ship status of `tools/hip/gfx1201-hipblaslt-*.txt`: the gfx120X wheel's hipBLASLt is 1.2.0 (version number 100200); the
tables here are 100202 and 100500, so the engine says "version mismatch ... using hipBLASEx" and runs plain hipBLAS. A
table tuned on the R9700 for 100200 (32 rows, `tune_hipblaslt`) measured within noise end to end (20K prompt x1.035 vs
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 01b99f06c..97c6cb48a 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -2211,6 +2211,59 @@ static double ple_direct_rows_per_s(strata::kernels::PleTable& t) {
return el > 0 ? (double) done / el : -1.0;
}
+#if defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906)
+// The R9700 / RX 9000 "bimodal prompt speed" (0.2.0 known limit). The runtime opens the process's hardware queues
+// lazily, as the first stream of each kind is used, and the kernel's scheduler (MES) puts a queue on one of the
+// card's compute pipes when the queue first gets work. The loader threads reach their streams in a different order
+// every start, so in about half of the starts the prompt path's two busy queues (the compute stream and the expert
+// staging stream) landed on the SAME pipe: every one of the ~30,000 hand-offs of a prompt then waits for the pipe to
+// switch queues, and a 4K prompt reads 13% slower (p620, 1,980 vs 2,290 tok/s). Opening the queues here, one after
+// the other on the main thread and before any other thread exists, makes the placement the same in every start (and
+// it is the fast one). Streams are destroyed again; the runtime keeps the queues.
+// STRATA_HIP_QUEUE_PRIME unset: gfx1200 / gfx1201 on Linux only, "nnnnh"; "0": off; any other string is a spec
+// ('n' = a stream at the default priority, 'h' = at the highest) for experiments on other cards.
+static void prime_hw_queues() {
+ const char* env = std::getenv("STRATA_HIP_QUEUE_PRIME");
+ if (env != nullptr && std::strcmp(env, "0") == 0) return;
+ const char* pv = std::getenv("STRATA_PRIMARY_DEVICE");
+ const int dev = pv != nullptr && std::atoi(pv) > 0 ? std::atoi(pv) : 0;
+ int n_dev = 0;
+ if (cudaGetDeviceCount(&n_dev) != cudaSuccess || dev >= n_dev) { (void) cudaGetLastError(); return; }
+ const char* spec = env != nullptr && env[0] != 0 ? env : nullptr;
+ if (spec == nullptr) {
+#if defined(_WIN32)
+ return;
+#else
+ cudaDeviceProp prop{};
+ if (cudaGetDeviceProperties(&prop, dev) != cudaSuccess || !strata::kernels::gfx_arch_is_gfx12(prop.gcnArchName)) {
+ (void) cudaGetLastError();
+ return;
+ }
+ spec = "nnnnh";
+#endif
+ }
+ if (dev != 0 && cudaSetDevice(dev) != cudaSuccess) { (void) cudaGetLastError(); return; }
+ void* buf = nullptr;
+ if (cudaMalloc(&buf, 256) != cudaSuccess) { (void) cudaGetLastError(); return; }
+ int lo = 0, hi = 0;
+ const bool have_prio = hipDeviceGetStreamPriorityRange(&lo, &hi) == hipSuccess;
+ std::vector<cudaStream_t> ss;
+ for (const char* c = spec; *c != 0; ++c) {
+ cudaStream_t s = nullptr;
+ const bool ok = (*c == 'h' && have_prio) ? hipStreamCreateWithPriority(&s, hipStreamNonBlocking, hi) == hipSuccess
+ : cudaStreamCreateWithFlags(&s, cudaStreamNonBlocking) == cudaSuccess;
+ if (!ok) continue;
+ cudaMemsetAsync(buf, 0, 256, s);
+ cudaStreamSynchronize(s);
+ ss.push_back(s);
+ }
+ for (cudaStream_t s : ss) cudaStreamDestroy(s);
+ cudaFree(buf);
+ (void) cudaGetLastError();
+ std::fprintf(stderr, "strata generate: opened %zu GPU queues in a fixed order (STRATA_HIP_QUEUE_PRIME=0 turns it off)\n", ss.size());
+}
+#endif
+
int main(int argc, char** argv) {
// **UNBUFFERED, BECAUSE THE INTERESTING OUTPUT IS THE OUTPUT BEFORE A CRASH.** `stdout` redirected to a
// pipe or a file is block-buffered, so a program that dies loses every line it had already printed - which
@@ -2294,6 +2347,9 @@ int main(int argc, char** argv) {
if (qwen35) return strata::qwen35::run(argc, argv);
break;
}
+#endif
+#if defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906)
+ prime_hw_queues();
#endif
Options o;
bool have_tokens = false;