Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
DeepSeek engine: DS4_ATTN_SLAB=N sets the chunked prompt's attention slab rows (default unchanged: 8 MB / HQ rounded to 64). The slab loop runs the dense projections (q_b, wo_a, wo_b) per slab, and each call dequantizes the whole dense matrix: a 256-row slab dequantized the same weights 32 times for an 8K prompt (the dense dequant 916 ms and rocBLAS 867 ms of that prompt). R9700 12 GB cell, 2.25 pack, 8K warm: slab 256 -> 591.9 ppts (with the earlier fixes), slab 2048 + DS4_RING_MB=1600 -> 668.0/666.2 ppts (12.26/12.30 s), the first prompt 429.9 (from 143-402), ids identical (5705e1164a); q_b+rope 3.97 -> 1.15 s. Opt-in: the default slab is unchanged.
Niko1221457d8cc7913c
diff --git a/src/dsv41/ds4_model.cu b/src/dsv41/ds4_model.cuindex bcb15a8a1..d49aee4d0 100644--- a/src/dsv41/ds4_model.cu+++ b/src/dsv41/ds4_model.cu@@ -2930,6 +2930,14 @@ struct Model::Impl {const size_t C = (size_t) chunk;c.C = chunk;c.A = std::max(64, (int) (((size_t) 8 << 20) / (size_t) HQ) / 64 * 64);+ // DS4_ATTN_SLAB=N: the attention slab's rows. The dense projections (q_b, wo_a/b) are dequantized and GEMM'd+ // once a slab, so a slab of 256 rows dequantizes the same weights 32 times for an 8K prompt (measured: the+ // dense dequant 916 ms of the 8K prompt, rocBLAS 867 ms); a larger slab costs VRAM for c.q/c.o and the oa+ // buffers (A * HQ x 4 bytes each).+ if (const char* v = std::getenv("DS4_ATTN_SLAB")) {+ const int a = std::atoi(v);+ if (a > 0) c.A = std::max(64, std::min(chunk, a) / 64 * 64);+ }// (the indexer's slab: its scores rows are the context wide - about 64 MB of them)c.AI = std::max(32, std::min(128, (int) (((size_t) 64 << 20) / (((size_t) o.max_context + 64) * 4)) / 32 * 32));c.w16_elems = (size_t) 8 << 20;