Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

DeepSeek engine: the post-prompt RAM-store preload skips the experts the CPU lane already holds as a RAM copy (copy_mode). In copy-all mode every routed expert is in that copy, so the preload read a second copy of the same bytes: 4152 experts, 40 GB at 6.4 GB/s, 6.2 s of every 8K prompt, with 0.0 store hits in the decode after it; with --mmap-experts off and the experts not all fitting, the in-place ones are still preloaded. R9700 12 GB cell, 2.25 pack, 8K: prompt 27918 -> 21567 ms (293 -> 380 ppts; a repeated prompt 526.6), decode 16.61 -> 16.82 tok/s, ids identical (5705e1164a). With DS4_COPY_PIN=1 the same run is 21347 ms (383.7 ppts, repeat 535.6).

Niko122100a129253157
diff --git a/src/dsv41/ds4_model.cu b/src/dsv41/ds4_model.cu
index ad2cac0be..bcb15a8a1 100644
--- a/src/dsv41/ds4_model.cu
+++ b/src/dsv41/ds4_model.cu
@@ -6263,11 +6263,15 @@ struct Model::Impl {
}
if (store && env_f("DS4_SETTLE_STORE", 1.0f) != 0.0f) { // the RAM store too: a window prompt fills it through the CPU
// lane's picks, a chunked one streams past it - the hottest CPU-lane experts of the prompt into its free slots
+ // (copy_mode: an expert already in the CPU lane's RAM copy gains nothing from a second copy of the same bytes -
+ // measured on the R9700 12 GB cell: 4152 such experts, 40 GB read at 6.4 GB/s, 6.2 s of an 8K prompt, 0.0 store
+ // hits in the decode after it; the VRAM swaps above already read them from the copy)
std::vector<std::pair<float, std::pair<int, int>>> hot;
for (const Layer& l : L) {
if (l.mtp) continue;
for (int e = 0; e < l.n_exp; ++e)
- if (l.slot[(size_t) e] < 0 && l.heat[(size_t) e] >= 4.0f) hot.push_back({l.heat[(size_t) e], {l.il, e}});
+ if (l.slot[(size_t) e] < 0 && !(copy_mode && !l.ram.empty() && l.ram[(size_t) e] >= 0) &&
+ l.heat[(size_t) e] >= 4.0f) hot.push_back({l.heat[(size_t) e], {l.il, e}});
}
std::sort(hot.begin(), hot.end(), [](const auto& a, const auto& b) { return a.first > b.first; });
std::vector<std::pair<int, int>> list;