Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

DeepSeek engine: 4096 chunk rule applies to a GGUF pack only when it is copied into RAM (--mmap-experts off) and to an explicit --tp 1 single card; the experts-read-in-place default stays 8192; other packs keep 2048

Niko1221829913ea5942
diff --git a/src/dsv41/ds4_experts_rule.hpp b/src/dsv41/ds4_experts_rule.hpp
index 6b0ee4ce1..25f7c13a8 100644
--- a/src/dsv41/ds4_experts_rule.hpp
+++ b/src/dsv41/ds4_experts_rule.hpp
@@ -29,7 +29,7 @@ inline int experts_per_word(int requested, uint64_t vram_bytes, int model_k) {
// (RTX 3060: x0.995 code, x1.002 agent) and made 8K prompts 20% faster (RTX 3060 175 -> 210, RTX 5070 186 -> 223, 32 GB class 285 -> 353 tok/s);
// 8192 returns up to 4% of agent decode. Cards under 10 GiB keep 1024 (their tier is the first thing a bigger buffer would eat).
constexpr uint64_t kChunk4096MinVram = 10ull << 30;
-// fallback: the value the case had before (1024: an explicit --tp 1 single card; 2048: the experts copied into RAM; the experts read in place
+// fallback: the value the case had before (1024: an explicit --tp 1 single card; 2048: a GGUF pack with --mmap-experts off, the experts copied into RAM; the experts
// keep 8192, which is not decided here). hip: a HIP build keeps the fallback until a one-card Radeon sweep shows 4096 is no slower.
// The result does not depend on --max-context: the chunk's buffers and the KV are allocated whole at load for the full context, and a
// chunk the card cannot hold is halved there (4096 -> 2048 -> 1024 -> ...), so a session never runs out later.
diff --git a/src/dsv41/ds4_model.cu b/src/dsv41/ds4_model.cu
index 614991840..89e5dd576 100644
--- a/src/dsv41/ds4_model.cu
+++ b/src/dsv41/ds4_model.cu
@@ -1969,8 +1969,11 @@ struct Model::Impl {
const bool own_default = !(par || o.tp > 1); // (the single-card default run the chunked path newly takes: chunks of 1024 - 8K prompt 370 tok/s on 4 x R9700, 214 at 8192; the threads / tp runs keep theirs)
// the chunk of the cases that are not the experts-read-in-place one (8192, measured best there): an explicit --tp 1 single card
// (was 1024) and the experts copied into RAM (was 2048) take 4096 from 10 GiB of VRAM (ds4_experts_rule.hpp)
+ // (a GGUF pack always has file_experts set: it is copied into RAM only by an explicit --mmap-experts off, which is the case
+ // measured: H100 8K, 32 GiB, experts copied: 2048 502, 4096 538, 8192 495 tok/s)
+ const bool gguf_copy = pk.gguf && o.mmap_experts == 0;
int own_chunk = 1024, copy_chunk = 2048;
- if (!devs.empty() && (own_default || !file_experts)) {
+ if (!devs.empty() && (own_default || gguf_copy)) {
uint64_t vmin = ~0ull;
for (const Dev& dv : devs) vmin = std::min<uint64_t>(vmin, (uint64_t) dv.total);
#if defined(__HIP_PLATFORM_AMD__) || defined(__HIPCC__)
@@ -1979,13 +1982,13 @@ struct Model::Impl {
const bool hip_build = false;
#endif
own_chunk = prefill_chunk_default(vmin, hip_build, 1024);
- copy_chunk = prefill_chunk_default(vmin, hip_build, 2048);
+ if (gguf_copy) copy_chunk = prefill_chunk_default(vmin, hip_build, 2048);
const int shown = own_default ? own_chunk : copy_chunk;
if (prompt_par && o.prefill_chunk < 0 && shown == 4096)
std::fprintf(stderr, "ds4: prompts in chunks of %d tokens (%.0f GiB of graphics memory: bigger chunks read the experts fewer times; "
"--prefill-chunk N changes it)\n", shown, (double) vmin / (double) (1ull << 30));
}
- chunk = prompt_par ? (o.prefill_chunk >= 0 ? o.prefill_chunk : own_default ? own_chunk : file_experts ? kVlRows : copy_chunk) : 0;
+ chunk = prompt_par ? (o.prefill_chunk >= 0 ? o.prefill_chunk : own_default ? own_chunk : file_experts && !gguf_copy ? kVlRows : copy_chunk) : 0;
const char* dc = std::getenv("DS4_CHUNK"); // (tests: down to 16, a short prompt in several chunks)
if (dc != nullptr) chunk = prompt_par ? std::atoi(dc) : 0;
if (const char* v = std::getenv("DS4_CHUNK_MIN")) chunk_min = std::max(1, std::atoi(v));