Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Merge f-027-marlin (DeepSeek on 4x R9700: staging ring from the room, copy pin auto, chunk 1024 where the cards hold most experts, whole-chunk attention slab, F16 gate/up activations, drafter cumulative cut on HIP, diagnostics). Conflicts resolved: attention slab = one rule (attn_slab_rows: Marlin's share >= 0.5 branch plus the automatic growth, unified memory and DS4_ATTN_SLAB=0/N untouched); staging ring = one function (staging_ring_mb: 16 MB where the cards hold the experts, Marlin's small per-device ring at 0.5..0.82, main's streamed 400..1200 MB on unified memory and under half); DS4_COPY_PIN auto = huge-page arena OR cards hold >= half; DS4_FAST_ATTN stays gfx12-only; chunk 1024 for multi-card tp runs too, but not for --threads on or one card on the default tp

Niko1221c8596ec39105
diff --cc src/dsv41/ds4_experts_rule.hpp
index d5085a32f,63c4c49ab..50e23d98f
--- a/src/dsv41/ds4_experts_rule.hpp
+++ b/src/dsv41/ds4_experts_rule.hpp
@@@ -36,12 -35,17 +36,20 @@@ inline int experts_per_word(int request
// Alloc failure halves the chunk at load (4096 -> 2048 -> 1024 ...), KV and score buffers are allocated whole at the full --max-context.
constexpr uint64_t kChunk4096MinVram = 10ull << 30;
constexpr uint64_t kChunkInPlace4096Below = 16ull << 30;
--// own_default: a single-card run (no --threads, tp 1); gpu_share: the share of the routed experts the cards can hold (1 = all);
++// own_default: a run with tp 1 and no --threads (`strata launch` writes --tp 1 for 4 or more AMD cards); multi_tp: two or more cards, tp above 1
++// and no `--threads on` (the engine's own default tp is 2, and `strata launch` writes --tp 2 for 2-3 cards); gpu_share: the share of the
++// routed experts the cards can hold (1 = all);
// gguf_copy: a GGUF pack with --mmap-experts off; file_experts: the experts stay in the files; hip: a HIP build; min_vram_bytes: the
// smallest card; big: the chunk of the streamed case (the engine's kVlRows, 8192)
inline int prefill_chunk_default(bool own_default, double gpu_share, bool gguf_copy, bool file_experts, bool hip, uint64_t min_vram_bytes,
-- int big = 8192) {
- if (own_default && gpu_share >= 0.5) return 1024;
- // The cards hold most of the experts: the small chunk is the measured one, and it is not a single-card matter. The
- // 4 x R9700 datapoint above (1024: 370 against 214 tok/s at 8192) is a multi-card run, and on that box the rule used
- // to fall through to 2048/8192: measured here again (2.25 bpw, 4 cards, --dspark auto, ids 5705e1164a) 8K warm
- // prompts 1024 -> 1247 ppts against 690 at 8192, with the decode unchanged, so the share alone decides it now.
- (void) own_default;
- if (gpu_share >= 0.5) return 1024;
++ int big = 8192, bool multi_tp = false) {
++ // The cards hold most of the experts: the small chunk is the measured one. Two datapoints, both on 4 x R9700: with `--tp 1` (what
++ // `strata launch` writes for 4 or more AMD cards), share 0.86: 1024 370 against 214 tok/s at 8192; and with the engine's own default tp
++ // (2, pairs of cards, nothing written), where the rule used to fall through to 2048/8192: 2.25 bpw, --dspark auto, ids 5705e1164a, 8K
++ // warm prompts 1024 -> 1247 ppts against 690 at 8192, decode unchanged. So a multi-card tp run takes it too. One card with the default
++ // tp, and `--threads on` (it stalled at the first 8K prompt in the 4-card runs), keep their old defaults: not measured with 1024.
++ // An explicit --prefill-chunk N wins over all of this (the call site takes the rule only when none was given).
++ if ((own_default || multi_tp) && gpu_share >= 0.5) return 1024;
if (gguf_copy && !hip && min_vram_bytes >= kChunk4096MinVram) return 4096;
// the experts read in place on a card under 16 GiB (the 12 GB PCs; not measured on HIP): 4096 keeps ~120 more experts resident than 8192 (RTX 5070
// 287 against 165, RTX 3060 386 against 277) - 5 interleaved pairs, 4096 against 8192, decode: 5070 agent x1.020 (5/5) code x1.004 (3/5), 3060
@@@ -60,24 -64,6 +68,55 @@@ inline bool copy_all_fits(double lane_b
return lane_bytes + ram_margin <= (double) ram_avail && lane_bytes <= commit_room;
}
+// The chunked prompt's attention slab (rows a pass: the dense projections q_b / wo_a / wo_b are dequantized and multiplied once a slab, so
+// a 256-row slab dequantizes them 32 times for an 8K prompt). `base`: the 0.2.6 rule (8 MB of rows / heads width, 256 on this model).
+// DS4_ATTN_SLAB=N (`env` >= 0): 0 gives the 0.2.6 rule back, N > 0 asks for that many rows (64..chunk, a multiple of 64). Unset (`env` < 0):
+// automatic - up to 2048 rows, but only where the bigger buffers take VRAM no expert would have used: `spare_bytes` is the card's room
+// beyond what its expert tier can use (huge when the tier is off, e.g. --vram-experts 0), `bytes_per_row` the buffers' cost of one row.
+// The answers do not change with the slab (a row's arithmetic is its own), but the experts a card holds do: measured 2026-10-26 on the RTX
+// 5070 (12 GB, 2.25 bpw, 8K prompt): 1024 rows +2.0% prompt, 2048 rows +2.2%, at 29 / 83 fewer experts in VRAM, a -3% decode and different
+// greedy ids (placement); so a card whose room is all experts keeps the 0.2.6 slab, and the automatic size is for cards with VRAM to spare.
- inline int attn_slab_rows(int env, int chunk, int base, double spare_bytes, double bytes_per_row) {
++// `expert_share` (the share of the routed experts the cards hold, 0 when unknown): the second measured case. Where the cards hold at least
++// half the experts the whole small chunk's slab is worth its VRAM: 4 x R9700 (2.25 bpw, share 0.77, chunk 1024), 8K warm 1204 / 1236 / 1250
++// ppts with the 256-row rule against 1324 / 1353 / 1363 with a 1024-row slab (+9%), decode 77.7 against 78.1 tok/s, 268 MB a device (1% of
++// the card). That branch asks for min(chunk, 1024) rows whatever the spare room (the spare of such a box is negative: its room is all
++// experts), and never less than the base; the automatic growth on top of it still applies where the room is there. A unified-memory GPU
++// (the caller passes env 0 there) and DS4_ATTN_SLAB=0 never take either branch.
++inline int attn_slab_rows(int env, int chunk, int base, double spare_bytes, double bytes_per_row, double expert_share = 0.0) {
+ auto clampn = [&](int a) { return std::max(64, std::min(chunk, a) / 64 * 64); };
+ if (env == 0) return base;
+ if (env > 0) return clampn(env);
++ int rows = base;
++ if (expert_share >= 0.5) rows = std::max(rows, clampn(1024));
+ const int target = clampn(2048);
- if (target <= base) return base;
- const double extra = (double) (target - base) * bytes_per_row;
- return spare_bytes >= extra * 1.25 + (256.0 * 1048576.0) ? target : base;
++ if (target > rows) {
++ const double extra = (double) (target - base) * bytes_per_row;
++ if (spare_bytes >= extra * 1.25 + (256.0 * 1048576.0)) rows = target;
++ }
++ return rows;
++}
++
++// The prompt path's expert staging ring (MB). The prompt streams the experts the cards do not hold through it. Three measured cases and
++// the rest left as it was:
++// - the cards hold the experts (share >= 0.82): 16 MB. 4 x R9700, share 0.86 (the 2.00 pack): a bigger ring costs decode after a prompt
++// (-1.8% at 16 MB against none, -4.4% at 64, -4.6% at 400); everything fitting needs no ring at all;
++// - they hold most of them (0.5 <= share < 0.82, discrete cards): a small ring from each DEVICE's room, 8 MB a GB of it, 128..512 MB
++// (4 x R9700, 2.25 bpw, share 0.72-0.77, chunk 1024, 8K prompt + six 128-token decodes a load: 2000 MB 1243 ppts / 73.2 tok/s, 1000 MB
++// 1287 / 77.5, 500 MB 1314 / 78.5, 256 MB 1324 / 78.4, 128 MB 1338 / 78.3; the room per device there is 31 GB -> 248 MB);
++// - a unified-memory GPU (Strix Halo) that holds under 1/1.3 of them, and every discrete card that holds under half: a share of the
++// summed room, 64 MB a GB, 400..1200 MB. Measured on Strix Halo only (400 -> 1200: the queue 3,200 -> 8,087 experts, 4K prompt 27.3
++// -> 21.3 s, decode unchanged); a discrete card in this band is not measured and takes the same, VRAM-bounded, rule.
++// A unified-memory GPU never takes the small ring (its datapoint is the 1200 MB one): it keeps the 16 MB "fit" rule at 1/1.3.
++inline int staging_ring_mb(double share, double room_bytes, int n_dev, bool integrated) {
++ const double GB = 1e9;
++ const double per_dev = n_dev > 0 ? room_bytes / (double) n_dev : 0.0;
++ const double streamed = std::min(1200.0, std::max(400.0, room_bytes / GB * 64.0));
++ if (integrated) return share >= 1.0 / 1.3 ? 16 : (int) streamed;
++ if (share >= 0.82) return 16;
++ if (share >= 0.5) return (int) std::min(512.0, std::max(128.0, per_dev / GB * 8.0));
++ return (int) streamed;
+}
+
// DS4_TOPK=N (a test, not for users): over the flag and the rule; returns the count to use
inline int experts_test_override(int chosen, int model_k) {
if (const char* v = std::getenv("DS4_TOPK")) {
diff --cc src/dsv41/ds4_kernels.cu
index 96d2474aa,a0b7c38a6..a05091f01
--- a/src/dsv41/ds4_kernels.cu
+++ b/src/dsv41/ds4_kernels.cu
@@@ -2026,40 -1998,17 +2026,42 @@@ bool split_enabled()
}();
return on;
}
-// The chunked prompt's packed-KV attention on the matrix cores (sparse_attn_wmma_kernel). ON by default: measured on
-// 4x R9700, 2.25 bpw, 8K code prompts, warm medians of three a load -- 1180 ppts without it, 1354 with it (+14.7%),
-// the same ids (5705e1164a). The QK/PV sums are the matrix cores' order, so the last places differ from the scalar
-// kernel's and the owner approved that trade for the prefill; the decode path stays on the scalar kernels.
-// DS4_FAST_ATTN=0 turns it off.
+// DS4_FAST_ATTN: the chunked prompt's packed-KV attention on the matrix cores (sparse_attn_wmma_kernel). The
+// QK/PV sums are the matrix cores' order, so the outputs differ from the scalar kernel's in the last places (the owner
+// approved the trade for the prefill; the decode path stays on the scalar kernels).
+// 0.2.7: ON by default on gfx1200 / gfx1201 (RDNA4: the R9700, RX 9000), where it was measured and gated (KL 0.0003 against the
+// 0.006 gate, ids identical); gfx11 and gfx1151 (Strix Halo) are untested and keep the scalar kernels. DS4_FAST_ATTN=0 is the
+// off switch (exactly the 0.2.6 arithmetic), DS4_FAST_ATTN=1 turns it on elsewhere. Decided per device (the first call there).
++// The Marlin stream measured it ON on 4x R9700 (gfx1201), 2.25 bpw, 8K code prompts: 1180 -> 1354 ppts (+14.7%), the same ids,
++// KL 0.10833 against 0.10925 for the scalar path. It stays gfx12-only until Strix Halo (gfx1151) is tested.
+#if defined(STRATA_USE_HIP)
+bool device_is_gfx12() {
+ static std::atomic<int> cache[64]; // 0 unknown, 1 no, 2 yes
+ int dev = 0;
+ cudaGetDevice(&dev);
+ if (dev < 0 || dev >= 64) return false;
+ int c = cache[dev].load();
+ if (c == 0) {
+ cudaDeviceProp prop{};
+ const bool yes = cudaGetDeviceProperties(&prop, dev) == cudaSuccess &&
+ (std::strncmp(prop.gcnArchName, "gfx1200", 7) == 0 || std::strncmp(prop.gcnArchName, "gfx1201", 7) == 0);
+ c = yes ? 2 : 1;
+ cache[dev].store(c);
+ }
+ return c == 2;
+}
+#endif
bool fast_attn_enabled() {
- static const bool on = [] {
+ static const int env = [] {
const char* e = std::getenv("DS4_FAST_ATTN");
- return e != nullptr ? std::atoi(e) != 0 : true;
+ return e == nullptr ? -1 : (std::atoi(e) != 0 ? 1 : 0);
}();
- return on;
+ if (env >= 0) return env == 1;
+#if defined(STRATA_USE_HIP)
+ return device_is_gfx12();
+#else
+ return false;
+#endif
}
attnsplit::AttnSplitScratch split_scratch(cudaStream_t s, int heads) {
static std::mutex mu;
diff --cc src/dsv41/ds4_model.cu
index deee99ab7,ddafb5567..8c3052ae8
--- a/src/dsv41/ds4_model.cu
+++ b/src/dsv41/ds4_model.cu
@@@ -2027,7 -1983,7 +2030,8 @@@ struct Model::Impl
// 1024 200, 2048 277, 4096 353, 8192 431 tok/s; RTX 5070 / 3060 12 GB in place: 8192 244 / 228 against 1024 186 / 175); where they hold
// most (4 x R9700: 8192 214, 1024 370) the small one wins; with the experts copied into RAM (--mmap-experts off) 4096 beats 8192 by 7-9%
// (H100 32 GiB: 2048 502, 4096 538, 8192 495). A GGUF pack always has file_experts set (copied only by --mmap-experts off).
-- const bool own_default = !(par || o.tp > 1); // (a single-card run: no --threads, tp 1 - a launch with several cards passes --tp 1; one card passes neither)
++ const bool own_default = !(par || o.tp > 1); // (a run with tp 1 and no --threads: `strata launch` writes --tp 1 for 4+ AMD cards, --tp 2 for 2-3 and nothing for one card = the engine's tp 2)
++ const bool multi_tp = !own_default && !o.threads && devs.size() >= 2; // (the 4-card default-tp run the 1024 chunk was also measured on; an explicit --prefill-chunk below wins over the rule)
const bool gguf_copy = pk.gguf && (o.mmap_experts == 0 || (store_gb() <= 0.0 && auto_copy_all));
uint64_t vmin_bytes = ~0ull;
for (const Dev& dv : devs) vmin_bytes = std::min<uint64_t>(vmin_bytes, (uint64_t) dv.total);
@@@ -2037,7 -1993,7 +2041,7 @@@
#else
const bool hip_build = false;
#endif
-- const int rule_chunk = prefill_chunk_default(own_default, gpu_expert_share(), gguf_copy, file_experts, hip_build, vmin_bytes, kVlRows);
++ const int rule_chunk = prefill_chunk_default(own_default, gpu_expert_share(), gguf_copy, file_experts, hip_build, vmin_bytes, kVlRows, multi_tp);
chunk = prompt_par ? (o.prefill_chunk >= 0 ? o.prefill_chunk : rule_chunk) : 0;
if (prompt_par && o.prefill_chunk < 0 && rule_chunk == 4096 && gguf_copy)
std::fprintf(stderr, "ds4: prompts in chunks of 4096 tokens (the experts are copied into RAM, %.0f GiB of graphics memory; --prefill-chunk N changes it)\n",
@@@ -2962,21 -2911,27 +2966,23 @@@
size_t ring_mb_auto() const {
double exp_b = 0, other_b = 0, room = 0;
const double share = gpu_expert_share(&exp_b, &room, &other_b);
- const bool fit = exp_b > 0 && share >= 1.0 / 1.3; // (R9700 x4: 0.86 -> ring 16: decode -1.8% after a prompt, 64: -4.4%, 400: -4.6%)
- // A box whose experts do not fit (a unified-memory APU) streams them from the disk, and the ring is what lets the
- // prompt queue its likely ones early: 400 -> 1200 MB on Strix Halo took the queue 3,200 -> 8,087 experts, the disk
- // 1.49 -> 2.44 GB/s and the compute's exposed read wait 11.4 -> 3.5 s (4K prompt 27.3 -> 21.3 s, 148.98 -> 190.98
- // tok/s) with decode unchanged (17.54/16.39 against 17.61 tok/s interleaved). A sixteenth of the VRAM room, so a
- // small pool still loads. DS4_RING_MB overrides.
- const size_t want = (size_t) std::max<double>(400.0, std::min<double>(1200.0, room / 1e9 * 64.0));
- // The prompt path streams the experts the cards cannot hold. When every expert fits, the ring is worth
- // nothing (16 MB): the VRAM is better as slots for the decode after the prompt (measured: decode -1.8 to -4.6%
- // at 64-400 MB, unchanged at 16). When they do not fit, the stream IS the prompt's expert stage, and its depth
- // is what decides how long the tiles wait: on the 4x R9700 8K prompt a 16 MB ring left 1.7 s of DMA waits in a
- // 5.2 s expert stage, and 1.2-3 GB of ring measured 632 -> 678-684 ppts (+7%) for about -2% decode (off a
- // fifth more CPU-lane experts). So: a share of each device's room, bounded, rather than the fixed 400 MB.
- const double per_dev = devs.empty() ? 0.0 : room / (double) devs.size();
- const size_t deep = (size_t) std::min(2000.0, std::max(400.0, per_dev / 1e9 * 0.08 * 1000.0)); // (MB)
- // ... but with the 1024-token prompt chunk (the default now when the cards hold most of the experts) the
- // ring's balance moved: every MB it takes is a tenth of an expert the decode keeps. Measured on the same
- // box, 8K prompt and six 128-token decodes a load, chunk 1024: 2000 MB 1243 ppts / 73.2 tok/s, 1000 MB
- // 1287 / 77.5, 500 MB 1314 / 78.5, 256 MB 1324 / 78.4. So a small bounded ring where the cards hold the
- // most of the experts, and the old depth where they do not (the streaming cases it was measured for).
- const size_t small = (size_t) std::min(512.0, std::max(128.0, per_dev / 1e9 * 0.008 * 1000.0)); // (MB)
- const size_t mb = share >= 0.98 ? 16 : (share >= 0.5 ? small : deep);
++ // One rule for the staging ring (ds4_experts_rule.hpp: staging_ring_mb): 16 MB where the cards hold the experts, a small
++ // ring from each device's room where they hold most of them (4 x R9700), the larger streamed ring from the summed room on a
++ // unified-memory GPU (Strix Halo: 400 -> 1200 MB) and where the cards hold under half. DS4_RING_MB overrides.
++ bool unified = false;
++ for (const Dev& d : devs) {
++ cudaDeviceProp ip{};
++ if (cudaGetDeviceProperties(&ip, d.id) == cudaSuccess && ip.integrated != 0) unified = true;
++ }
++ const size_t mb = (size_t) strata::ds4::staging_ring_mb(exp_b > 0 ? share : 0.0, room, (int) devs.size(), unified);
if (o.verbose)
std::fprintf(stderr, "ds4: staging ring: experts %.1f GB, VRAM room %.1f GB, dense %.1f GB -> %d MB\n", exp_b / 1e9,
- room / 1e9, other_b / 1e9, fit ? 16 : (int) want);
- return fit ? 16 : want;
+ room / 1e9, other_b / 1e9, (int) mb);
+ return mb;
}
+ bool slab_auto_grew_ = false, slab_force_base_ = false; // the automatic attention slab: grew this try / must not grow (it did not fit)
bool alloc_chunks(std::string& err) {
+ slab_auto_grew_ = false;
chunks.resize(devs.size());
const int D = g.dim, H = g.hc, HD = g.hd;
const int cols = (pk.eg_ngram - 1) * pk.eg_heads, ehd = pk.eg_hd;
@@@ -3010,28 -2949,17 +3016,36 @@@
// DS4_ATTN_SLAB=N: the attention slab's rows. The dense projections (q_b, wo_a/b) are dequantized and GEMM'd
// once a slab, so a slab of 256 rows dequantizes the same weights 32 times for an 8K prompt (measured: the
// dense dequant 916 ms of the 8K prompt, rocBLAS 867 ms); a larger slab costs VRAM for c.q/c.o and the oa
- // buffers (A * HQ x 4 bytes each).
- const int a_rule = std::max(64, (int) (((size_t) 8 << 20) / (size_t) HQ) / 64 * 64); // 256 rows here
- c.A = a_rule;
- // Where the cards hold most of the experts, the whole chunk's slab is worth its VRAM: measured on 4x R9700
- // (2.25 bpw, share 0.77, chunk 1024) the 256-row rule gives 8K warm 1204 / 1236 / 1250 ppts and a slab of
- // the chunk's 1024 rows 1324 / 1353 / 1363 (+9%) with the decode unchanged (77.7 against 78.1 tok/s); the
- // 1024 rows cost 268 MB a device, 1% of the card. Elsewhere the rule stays as it was.
- if (gpu_expert_share() >= 0.5) c.A = std::max(a_rule, std::min(chunk, 1024) / 64 * 64);
- if (const char* v = std::getenv("DS4_ATTN_SLAB")) {
- const int a = std::atoi(v);
- if (a > 0) c.A = std::max(64, std::min(chunk, a) / 64 * 64);
+ // buffers (A * HQ x 4 bytes each). 0.2.7: unset = automatic (attn_slab_rows: up to 2048 rows where the buffers
+ // take VRAM no expert would have used), 0 = the 0.2.6 rule above, N = that many rows.
+ {
+ int env = -1;
+ if (const char* v = std::getenv("DS4_ATTN_SLAB")) env = std::max(0, std::atoi(v));
+ if ((slab_force_base_ || unified || a_mb_env != 0) && env < 0) env = 0; // (no automatic growth on a unified-memory GPU, or when STRATA_DS4_A_MB chose the base)
+ // per row: c.q + c.o (float), c.o16 (half), c.oa (float), c.oa16 (half)
+ const double bpr = (double) HQ * 10.0 + (double) OG * (double) g.o_lora * 6.0;
+ double spare = 1e18; // the expert tier off (--vram-experts 0): the buffers cost no expert
++ double share = 0.0; // the share of the routed experts the cards hold (0 with the tier off)
+ if (o.vram_experts > 0.0) {
+ double exp_b = 0, room = 0, other_b = 0;
- gpu_expert_share(&exp_b, &room, &other_b);
++ share = gpu_expert_share(&exp_b, &room, &other_b);
+ spare = (room - other_b - exp_b) / (double) std::max<size_t>(1, devs.size());
+ }
+ const int base = c.A;
- c.A = strata::ds4::attn_slab_rows(env, chunk, base, spare, bpr);
++ // one rule (ds4_experts_rule.hpp: attn_slab_rows): the chunk's whole slab (<= 1024 rows) where the cards hold half the
++ // experts or more (4 x R9700, +9% prompt), up to 2048 rows where the VRAM is spare, the base otherwise
++ c.A = strata::ds4::attn_slab_rows(env, chunk, base, spare, bpr, share);
+ if (env < 0 && c.A > base) {
+ slab_auto_grew_ = true;
- if (di == 0 && o.verbose)
- std::fprintf(stderr, "ds4: attention slab %d rows (automatic: %.1f GB of VRAM to spare beyond the experts; DS4_ATTN_SLAB=0 keeps %d)\n",
- c.A, spare / 1e9, base);
++ if (di == 0 && o.verbose) {
++ if (share >= 0.5)
++ std::fprintf(stderr, "ds4: attention slab %d rows (automatic: the cards hold %.0f%% of the experts; DS4_ATTN_SLAB=0 keeps %d)\n",
++ c.A, share * 100.0, base);
++ else
++ std::fprintf(stderr, "ds4: attention slab %d rows (automatic: %.1f GB of VRAM to spare beyond the experts; DS4_ATTN_SLAB=0 keeps %d)\n",
++ c.A, spare / 1e9, base);
++ }
+ }
}
// (the indexer's slab: its scores rows are the context wide - about 64 MB of them)
c.AI = std::max(32, std::min(128, (int) (((size_t) 64 << 20) / (((size_t) o.max_context + 64) * 4)) / 32 * 32));
@@@ -4556,16 -4465,18 +4579,22 @@@
std::fprintf(stderr, "ds4: %zu of the CPU lane's %zu experts copied into RAM (%.1f GB, %d NUMA part%s) in %.1f s%s\n",
jobs.size(), hot.size(), cb / 1e9, NH, NH > 1 ? "s" : "", (now_ms() - t0) / 1000.0,
jobs.size() < hot.size() ? "; the others read in place" : "");
- // DS4_COPY_PIN=1: pin the copies, so the prompt path DMAs each expert from where it lies instead of copying it
- // to the pinned staging first (measured on the R9700 box: the 8K pass streams 122 GB at 6.8 GB/s through the
- // staging, where the RAM-to-staging and DMA probes are 24 and 28 GB/s). The CPU lane reads the same memory, and
- // pinned 4 KB pages can cost its TLB (an earlier registration measured -15% decode), so this stays opt-in.
+ // DS4_COPY_PIN: pin the copies, so the prompt path DMAs each expert from where it lies instead of copying it to
+ // the pinned staging first (the pass streams 122 GB at 6.8 GB/s through the staging, where the RAM-to-staging
+ // and DMA probes are 24 and 28 GB/s). The CPU lane reads the same memory, and pinned 4 KB pages can cost its
+ // TLB (an earlier registration measured -15% decode on a 12 GB card, where the CPU lane does nearly all of the
+ // decode) - so AUTO (unset) pins when the cards hold at least half the routed experts: the CPU lane is then a
+ // minority of the decode and the prompt stream is what hurts. Measured on 4x R9700, 2.25 bpw (share 0.72),
+ // three interleaved pairs: the second 8K prompt of a process 12.0/11.9/11.9 s against 28.8/13.8/12.3 s unpinned,
+ // decode 35.42/35.50/34.9-36.0 against 35.43/35.45/34.9-36.0 tok/s (no cost). DS4_COPY_PIN=0/1 forces it.
// DS4_COPY_PIN_RO=1: cudaHostRegisterReadOnly (the GPU only reads it).
- // 0.2.7: DS4_COPY_PIN unset = ON only when the whole arena is on explicit 2 MB pages (a pin of a THP arena registers 4 KB
- // pages and the DMA rate was erratic: 11.9 s one prompt, 30.4 s the next); =0 is the off switch, =1 pins any arena.
++ // 0.2.7: DS4_COPY_PIN unset = ON when EITHER condition measured reliable holds: (a) the whole arena is on explicit 2 MB pages (a pin of a
++ // THP arena registers 4 KB pages and the DMA rate was erratic on the 12 GB cell: 11.9 s one prompt, 30.4 s the next), or (b) the
++ // cards hold at least half the routed experts (4 x R9700: the pin was reliable in 4/4 runs there, see above). =0 is the off
++ // switch, =1 pins any arena.
const char* cp = std::getenv("DS4_COPY_PIN");
- const bool pin_copies = cp != nullptr ? std::atoi(cp) != 0 : (arena_all_huge && !ram_chunks.empty());
- const bool pin_on = cp != nullptr ? std::atoi(cp) != 0 : gpu_expert_share() >= 0.5;
- if (pin_on) {
++ const bool pin_copies = cp != nullptr ? std::atoi(cp) != 0 : ((arena_all_huge || gpu_expert_share() >= 0.5) && !ram_chunks.empty());
+ if (pin_copies) {
const bool ro = std::getenv("DS4_COPY_PIN_RO") != nullptr && std::atoi(std::getenv("DS4_COPY_PIN_RO")) != 0;
const double tr = now_ms();
bool ok = true;
diff --cc tests/dsv41/experts_rule_test.cpp
index ffc5078d4,f0fad2dc5..fa1691bba
--- a/tests/dsv41/experts_rule_test.cpp
+++ b/tests/dsv41/experts_rule_test.cpp
@@@ -8,7 -8,6 +8,8 @@@
using strata::ds4::experts_per_word;
using strata::ds4::experts_test_override;
using strata::ds4::copy_all_fits;
+using strata::ds4::attn_slab_rows;
++using strata::ds4::staging_ring_mb;
static int fails = 0;
#define CHECK_EQ(a, b) \
@@@ -71,26 -70,7 +72,70 @@@ int main()
unsetenv("DS4_TOPK");
#endif
CHECK_EQ(experts_test_override(5, 6), 5);
- // prompt chunk: prefill_chunk_default(own_default, gpu_share, gguf_copy, file_experts, hip, min_vram, big=8192). What `strata launch` runs on one card
+ // attention slab rows: attn_slab_rows(env, chunk, base, spare_bytes, bytes_per_row). bpr of this model ~ 32768 * 10 + 8 * 1024 * 6 = 377 KB.
+ {
+ const double bpr = 377000.0;
+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 1e18, bpr), 2048); // unset, the expert tier is off (--vram-experts 0): automatic 2048
+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 4.0e9, bpr), 2048); // 4 GB of VRAM to spare: automatic
+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, -5.0e9, bpr), 256); // the 5070: every spare byte is an expert's: the 0.2.6 slab
+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 0.0, bpr), 256);
+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 0.9e9, bpr), 256); // 1792 rows x 377 KB = 676 MB; needs 1.25 x + 256 MB = 1.1 GB
+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 1.2e9, bpr), 2048);
+ CHECK_EQ(attn_slab_rows(-1, 1024, 256, 1e18, bpr), 1024); // a 1024 chunk caps the slab
+ CHECK_EQ(attn_slab_rows(-1, 4096, 256, 1e18, bpr), 2048);
+ CHECK_EQ(attn_slab_rows(-1, 128, 64, 1e18, bpr), 128);
+ CHECK_EQ(attn_slab_rows(-1, 8192, 2560, 1e18, bpr), 2560); // a base already above the automatic size is kept
+ CHECK_EQ(attn_slab_rows(0, 8192, 256, 1e18, bpr), 256); // DS4_ATTN_SLAB=0: the 0.2.6 rule, whatever the spare
+ CHECK_EQ(attn_slab_rows(1024, 8192, 256, -5.0e9, bpr), 1024); // N: that many rows, the spare is the user's call
+ CHECK_EQ(attn_slab_rows(100, 8192, 256, 1e18, bpr), 64); // rounded down to 64, at least 64
+ CHECK_EQ(attn_slab_rows(99999, 8192, 256, 1e18, bpr), 8192); // at most the chunk
+ CHECK_EQ(attn_slab_rows(300, 8192, 256, 1e18, bpr), 256);
++ // the cards hold half the experts or more (expert_share, the 6th argument): the whole small chunk's slab, min(chunk, 1024) rows, whatever
++ // the spare (the 4 x R9700 cell: chunk 1024, share 0.77, spare negative: 256 -> 1024 rows)
++ CHECK_EQ(attn_slab_rows(-1, 1024, 256, -5.0e9, bpr, 0.77), 1024);
++ CHECK_EQ(attn_slab_rows(-1, 1024, 256, -5.0e9, bpr, 0.50), 1024);
++ CHECK_EQ(attn_slab_rows(-1, 1024, 256, -5.0e9, bpr, 0.49), 256); // under half: the base, as before
++ CHECK_EQ(attn_slab_rows(-1, 512, 256, -5.0e9, bpr, 0.77), 512); // a chunk under 1024 caps it
++ CHECK_EQ(attn_slab_rows(-1, 8192, 256, -5.0e9, bpr, 0.77), 1024); // a big chunk: still 1024 rows without the room for more
++ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 4.0e9, bpr, 0.77), 2048); // the room is there too: the automatic growth wins
++ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 1e18, bpr, 1.0), 2048); // everything fits (a big card): 2048 as without the share
++ CHECK_EQ(attn_slab_rows(-1, 8192, 2560, -5.0e9, bpr, 0.77), 2560); // never below the base
++ CHECK_EQ(attn_slab_rows(0, 1024, 256, 1e18, bpr, 0.77), 256); // DS4_ATTN_SLAB=0: the 0.2.6 rule whatever the share
++ CHECK_EQ(attn_slab_rows(512, 1024, 256, 1e18, bpr, 0.77), 512); // DS4_ATTN_SLAB=N: that many rows whatever the share
++ CHECK_EQ(attn_slab_rows(-1, 1024, 1024, -5.0e9, bpr, 0.77), 1024); // a unified-memory base (32 MB of q = 1024 rows) is kept
+ }
- // prompt chunk: prefill_chunk_default(own_default, gpu_share, gguf_copy, file_experts, hip, min_vram, big=8192). What `strata launch` runs on one card
++ // the staging ring (MB): staging_ring_mb(share, room_bytes, n_dev, integrated)
++ {
++ const double GB = 1e9;
++ // the cards hold the experts: 16 MB, discrete (4 x R9700 at 0.86 and a card that holds everything)
++ CHECK_EQ(staging_ring_mb(0.86, 4 * 30 * GB, 4, false), 16);
++ CHECK_EQ(staging_ring_mb(1.00, 70 * GB, 1, false), 16);
++ CHECK_EQ(staging_ring_mb(0.82, 4 * 30 * GB, 4, false), 16);
++ // they hold most of them (4 x R9700, share 0.72-0.77, 31 GB of room a device): 8 MB a GB of one device's room = 248 MB
++ CHECK_EQ(staging_ring_mb(0.77, 4 * 31 * GB, 4, false), 248);
++ CHECK_EQ(staging_ring_mb(0.72, 4 * 31 * GB, 4, false), 248);
++ CHECK_EQ(staging_ring_mb(0.81, 4 * 31 * GB, 4, false), 248);
++ CHECK_EQ(staging_ring_mb(0.77, 4 * 10 * GB, 4, false), 128); // small devices: at least 128
++ CHECK_EQ(staging_ring_mb(0.60, 3 * 90 * GB, 3, false), 512); // big devices: at most 512
++ CHECK_EQ(staging_ring_mb(0.50, 2 * 31 * GB, 2, false), 248); // the per-device room, not the summed one (1 card: 31 GB -> 248 too)
++ CHECK_EQ(staging_ring_mb(0.60, 31 * GB, 1, false), 248);
++ // they hold under half (discrete): 64 MB a GB of the summed room, 400..1200 (streamed; not measured on a discrete card)
++ CHECK_EQ(staging_ring_mb(0.19, 31 * GB, 1, false), 1200);
++ CHECK_EQ(staging_ring_mb(0.06, 10 * GB, 1, false), 640); // a 12 GB card
++ CHECK_EQ(staging_ring_mb(0.06, 4 * GB, 1, false), 400); // a small pool still gets 400
++ CHECK_EQ(staging_ring_mb(0.40, 2 * 31 * GB, 2, false), 1200); // capped
++ CHECK_EQ(staging_ring_mb(0.49, 4 * 31 * GB, 4, false), 1200);
++ // a unified-memory GPU (Strix Halo): never the small ring; the streamed 400..1200 MB under 1/1.3, 16 above
++ CHECK_EQ(staging_ring_mb(0.60, 40 * GB, 1, true), 1200); // share 0.6 would be the small ring on a discrete card
++ CHECK_EQ(staging_ring_mb(0.30, 18.75 * GB, 1, true), 1200);
++ CHECK_EQ(staging_ring_mb(0.30, 12 * GB, 1, true), 768);
++ CHECK_EQ(staging_ring_mb(0.30, 5 * GB, 1, true), 400);
++ CHECK_EQ(staging_ring_mb(0.77, 40 * GB, 1, true), 16); // main's "fit" threshold, 1/1.3 = 0.769
++ CHECK_EQ(staging_ring_mb(0.76, 40 * GB, 1, true), 1200);
++ // nothing known: no devices
++ CHECK_EQ(staging_ring_mb(0.0, 0.0, 0, false), 400);
++ }
++ // prompt chunk: prefill_chunk_default(own_default, gpu_share, gguf_copy, file_experts, hip, min_vram, big=8192, multi_tp=false). What `strata launch` runs on one card
// passes neither --tp nor --threads, but the engine's default tp is 2, so own_default is false there; a launch with several cards passes --tp 1.
using strata::ds4::prefill_chunk_default;
// RTX 5070 / RTX 3060 (12 GB), launch default: experts in place on a card under 16 GiB -> 4096 (their logs said 8192 before this rule)
@@@ -123,8 -103,8 +168,17 @@@
CHECK_EQ(prefill_chunk_default(true, 0.86, false, true, true, 32 * GiB), 1024);
CHECK_EQ(prefill_chunk_default(true, 0.50, false, true, true, 32 * GiB), 1024);
CHECK_EQ(prefill_chunk_default(true, 0.49, false, true, true, 32 * GiB), 8192);
-- // threads / tp runs (own_default false) keep their defaults whatever the share: 8192 streamed, 2048 for a pack that is not a GGUF
++ // a multi-card tp run (own_default false, multi_tp: 2+ cards, no --threads on; the engine's default tp 2, or --tp 2 from the launcher) takes
++ // the small chunk too: the 4 x R9700 default-tp datapoint (8K warm 1247 ppts at 1024 against 690)
++ CHECK_EQ(prefill_chunk_default(false, 0.86, false, true, true, 32 * GiB, 8192, true), 1024);
++ CHECK_EQ(prefill_chunk_default(false, 0.50, false, true, true, 32 * GiB, 8192, true), 1024);
++ CHECK_EQ(prefill_chunk_default(false, 0.49, false, true, true, 32 * GiB, 8192, true), 8192);
++ // one card with the default tp, and `--threads on` (neither own_default nor multi_tp) keep their defaults whatever the share: 8192
++ // streamed, 2048 for a pack that is not a GGUF, 4096 copied into RAM on a CUDA card
CHECK_EQ(prefill_chunk_default(false, 0.86, false, true, true, 32 * GiB), 8192);
++ CHECK_EQ(prefill_chunk_default(false, 1.00, false, true, true, 80 * GiB), 8192);
++ CHECK_EQ(prefill_chunk_default(false, 1.00, true, true, false, 80 * GiB), 4096);
++ CHECK_EQ(prefill_chunk_default(false, 0.86, false, false, true, 32 * GiB), 2048);
CHECK_EQ(prefill_chunk_default(false, 0.30, false, false, false, 24 * GiB), 2048);
// the rule takes no --max-context: a 128K session on a 12 GB card gets the same chunk, because the chunk's buffers and the KV are
// allocated whole at load for that context and a chunk that does not fit there is halved before the first prompt (4096 -> 2048 -> 1024 ...)