Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

DeepSeek engine (AMD): the whole chunk's attention slab where the cards hold most of the experts

Niko12210f66c316eea3
diff --git a/src/dsv41/ds4_model.cu b/src/dsv41/ds4_model.cu
index 94641b4e0..ddafb5567 100644
--- a/src/dsv41/ds4_model.cu
+++ b/src/dsv41/ds4_model.cu
@@ -2946,11 +2946,17 @@ struct Model::Impl {
split128(g.ff, nm, 0, ff0_, FF); // (the widest member's share: member 0 takes the odd blocks)
const size_t C = (size_t) chunk;
c.C = chunk;
- c.A = std::max(64, (int) (((size_t) 8 << 20) / (size_t) HQ) / 64 * 64);
// DS4_ATTN_SLAB=N: the attention slab's rows. The dense projections (q_b, wo_a/b) are dequantized and GEMM'd
// once a slab, so a slab of 256 rows dequantizes the same weights 32 times for an 8K prompt (measured: the
// dense dequant 916 ms of the 8K prompt, rocBLAS 867 ms); a larger slab costs VRAM for c.q/c.o and the oa
// buffers (A * HQ x 4 bytes each).
+ const int a_rule = std::max(64, (int) (((size_t) 8 << 20) / (size_t) HQ) / 64 * 64); // 256 rows here
+ c.A = a_rule;
+ // Where the cards hold most of the experts, the whole chunk's slab is worth its VRAM: measured on 4x R9700
+ // (2.25 bpw, share 0.77, chunk 1024) the 256-row rule gives 8K warm 1204 / 1236 / 1250 ppts and a slab of
+ // the chunk's 1024 rows 1324 / 1353 / 1363 (+9%) with the decode unchanged (77.7 against 78.1 tok/s); the
+ // 1024 rows cost 268 MB a device, 1% of the card. Elsewhere the rule stays as it was.
+ if (gpu_expert_share() >= 0.5) c.A = std::max(a_rule, std::min(chunk, 1024) / 64 * 64);
if (const char* v = std::getenv("DS4_ATTN_SLAB")) {
const int a = std::atoi(v);
if (a > 0) c.A = std::max(64, std::min(chunk, a) / 64 * 64);