Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

prompt: the RAM store's post-prompt preload is off by default - it costs a 4K prompt 4.7 s and the decode that follows gains nothing

Niko1221259532bf2e74
diff --git a/src/dsv41/ds4_model.cu b/src/dsv41/ds4_model.cu
index 6d151af60..48d6d54f9 100644
--- a/src/dsv41/ds4_model.cu
+++ b/src/dsv41/ds4_model.cu
@@ -6282,7 +6282,10 @@ struct Model::Impl {
if (sw_inflight == 0) break; // (nothing queued: every layer's best CPU-lane expert is not hotter than its coldest VRAM one)
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
- if (store && env_f("DS4_SETTLE_STORE", 1.0f) != 0.0f) { // the RAM store too: a window prompt fills it through the CPU
+ // The RAM store's preload after a chunked prompt is off by default: measured on Strix Halo (code fixture, prompt +
+ // 256 tokens, 3 runs a side) it costs the prompt 4.7 s and the decode that follows gains nothing (13.9/14.4 s with it
+ // against 14.2 s without). The swaps above are what the decode actually uses; DS4_SETTLE_STORE=1 restores it.
+ if (store && env_f("DS4_SETTLE_STORE", 0.0f) != 0.0f) { // the RAM store too: a window prompt fills it through the CPU
// lane's picks, a chunked one streams past it - the hottest CPU-lane experts of the prompt into its free slots
std::vector<std::pair<float, std::pair<int, int>>> hot;
for (const Layer& l : L) {