Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge f-027-marlin (DeepSeek on 4x R9700: staging ring from the room, copy pin auto, chunk 1024 where the cards hold most experts, whole-chunk attention slab, F16 gate/up activations, drafter cumulative cut on HIP, diagnostics). Conflicts resolved: attention slab = one rule (attn_slab_rows: Marlin's share >= 0.5 branch plus the automatic growth, unified memory and DS4_ATTN_SLAB=0/N untouched); staging ring = one function (staging_ring_mb: 16 MB where the cards hold the experts, Marlin's small per-device ring at 0.5..0.82, main's streamed 400..1200 MB on unified memory and under half); DS4_COPY_PIN auto = huge-page arena OR cards hold >= half; DS4_FAST_ATTN stays gfx12-only; chunk 1024 for multi-card tp runs too, but not for --threads on or one card on the default tp
Niko1221c8596ec39105
diff --cc src/dsv41/ds4_experts_rule.hppindex d5085a32f,63c4c49ab..50e23d98f--- a/src/dsv41/ds4_experts_rule.hpp+++ b/src/dsv41/ds4_experts_rule.hpp@@@ -36,12 -35,17 +36,20 @@@ inline int experts_per_word(int request// Alloc failure halves the chunk at load (4096 -> 2048 -> 1024 ...), KV and score buffers are allocated whole at the full --max-context.constexpr uint64_t kChunk4096MinVram = 10ull << 30;constexpr uint64_t kChunkInPlace4096Below = 16ull << 30;--// own_default: a single-card run (no --threads, tp 1); gpu_share: the share of the routed experts the cards can hold (1 = all);++// own_default: a run with tp 1 and no --threads (`strata launch` writes --tp 1 for 4 or more AMD cards); multi_tp: two or more cards, tp above 1++// and no `--threads on` (the engine's own default tp is 2, and `strata launch` writes --tp 2 for 2-3 cards); gpu_share: the share of the++// routed experts the cards can hold (1 = all);// gguf_copy: a GGUF pack with --mmap-experts off; file_experts: the experts stay in the files; hip: a HIP build; min_vram_bytes: the// smallest card; big: the chunk of the streamed case (the engine's kVlRows, 8192)inline int prefill_chunk_default(bool own_default, double gpu_share, bool gguf_copy, bool file_experts, bool hip, uint64_t min_vram_bytes,-- int big = 8192) {- if (own_default && gpu_share >= 0.5) return 1024;- // The cards hold most of the experts: the small chunk is the measured one, and it is not a single-card matter. The- // 4 x R9700 datapoint above (1024: 370 against 214 tok/s at 8192) is a multi-card run, and on that box the rule used- // to fall through to 2048/8192: measured here again (2.25 bpw, 4 cards, --dspark auto, ids 5705e1164a) 8K warm- // prompts 1024 -> 1247 ppts against 690 at 8192, with the decode unchanged, so the share alone decides it now.- (void) own_default;- if (gpu_share >= 0.5) return 1024;++ int big = 8192, bool multi_tp = false) {++ // The cards hold most of the experts: the small chunk is the measured one. Two datapoints, both on 4 x R9700: with `--tp 1` (what++ // `strata launch` writes for 4 or more AMD cards), share 0.86: 1024 370 against 214 tok/s at 8192; and with the engine's own default tp++ // (2, pairs of cards, nothing written), where the rule used to fall through to 2048/8192: 2.25 bpw, --dspark auto, ids 5705e1164a, 8K++ // warm prompts 1024 -> 1247 ppts against 690 at 8192, decode unchanged. So a multi-card tp run takes it too. One card with the default++ // tp, and `--threads on` (it stalled at the first 8K prompt in the 4-card runs), keep their old defaults: not measured with 1024.++ // An explicit --prefill-chunk N wins over all of this (the call site takes the rule only when none was given).++ if ((own_default || multi_tp) && gpu_share >= 0.5) return 1024;if (gguf_copy && !hip && min_vram_bytes >= kChunk4096MinVram) return 4096;// the experts read in place on a card under 16 GiB (the 12 GB PCs; not measured on HIP): 4096 keeps ~120 more experts resident than 8192 (RTX 5070// 287 against 165, RTX 3060 386 against 277) - 5 interleaved pairs, 4096 against 8192, decode: 5070 agent x1.020 (5/5) code x1.004 (3/5), 3060@@@ -60,24 -64,6 +68,55 @@@ inline bool copy_all_fits(double lane_breturn lane_bytes + ram_margin <= (double) ram_avail && lane_bytes <= commit_room;}+// The chunked prompt's attention slab (rows a pass: the dense projections q_b / wo_a / wo_b are dequantized and multiplied once a slab, so+// a 256-row slab dequantizes them 32 times for an 8K prompt). `base`: the 0.2.6 rule (8 MB of rows / heads width, 256 on this model).+// DS4_ATTN_SLAB=N (`env` >= 0): 0 gives the 0.2.6 rule back, N > 0 asks for that many rows (64..chunk, a multiple of 64). Unset (`env` < 0):+// automatic - up to 2048 rows, but only where the bigger buffers take VRAM no expert would have used: `spare_bytes` is the card's room+// beyond what its expert tier can use (huge when the tier is off, e.g. --vram-experts 0), `bytes_per_row` the buffers' cost of one row.+// The answers do not change with the slab (a row's arithmetic is its own), but the experts a card holds do: measured 2026-10-26 on the RTX+// 5070 (12 GB, 2.25 bpw, 8K prompt): 1024 rows +2.0% prompt, 2048 rows +2.2%, at 29 / 83 fewer experts in VRAM, a -3% decode and different+// greedy ids (placement); so a card whose room is all experts keeps the 0.2.6 slab, and the automatic size is for cards with VRAM to spare.- inline int attn_slab_rows(int env, int chunk, int base, double spare_bytes, double bytes_per_row) {++// `expert_share` (the share of the routed experts the cards hold, 0 when unknown): the second measured case. Where the cards hold at least++// half the experts the whole small chunk's slab is worth its VRAM: 4 x R9700 (2.25 bpw, share 0.77, chunk 1024), 8K warm 1204 / 1236 / 1250++// ppts with the 256-row rule against 1324 / 1353 / 1363 with a 1024-row slab (+9%), decode 77.7 against 78.1 tok/s, 268 MB a device (1% of++// the card). That branch asks for min(chunk, 1024) rows whatever the spare room (the spare of such a box is negative: its room is all++// experts), and never less than the base; the automatic growth on top of it still applies where the room is there. A unified-memory GPU++// (the caller passes env 0 there) and DS4_ATTN_SLAB=0 never take either branch.++inline int attn_slab_rows(int env, int chunk, int base, double spare_bytes, double bytes_per_row, double expert_share = 0.0) {+ auto clampn = [&](int a) { return std::max(64, std::min(chunk, a) / 64 * 64); };+ if (env == 0) return base;+ if (env > 0) return clampn(env);++ int rows = base;++ if (expert_share >= 0.5) rows = std::max(rows, clampn(1024));+ const int target = clampn(2048);- if (target <= base) return base;- const double extra = (double) (target - base) * bytes_per_row;- return spare_bytes >= extra * 1.25 + (256.0 * 1048576.0) ? target : base;++ if (target > rows) {++ const double extra = (double) (target - base) * bytes_per_row;++ if (spare_bytes >= extra * 1.25 + (256.0 * 1048576.0)) rows = target;++ }++ return rows;++}++++// The prompt path's expert staging ring (MB). The prompt streams the experts the cards do not hold through it. Three measured cases and++// the rest left as it was:++// - the cards hold the experts (share >= 0.82): 16 MB. 4 x R9700, share 0.86 (the 2.00 pack): a bigger ring costs decode after a prompt++// (-1.8% at 16 MB against none, -4.4% at 64, -4.6% at 400); everything fitting needs no ring at all;++// - they hold most of them (0.5 <= share < 0.82, discrete cards): a small ring from each DEVICE's room, 8 MB a GB of it, 128..512 MB++// (4 x R9700, 2.25 bpw, share 0.72-0.77, chunk 1024, 8K prompt + six 128-token decodes a load: 2000 MB 1243 ppts / 73.2 tok/s, 1000 MB++// 1287 / 77.5, 500 MB 1314 / 78.5, 256 MB 1324 / 78.4, 128 MB 1338 / 78.3; the room per device there is 31 GB -> 248 MB);++// - a unified-memory GPU (Strix Halo) that holds under 1/1.3 of them, and every discrete card that holds under half: a share of the++// summed room, 64 MB a GB, 400..1200 MB. Measured on Strix Halo only (400 -> 1200: the queue 3,200 -> 8,087 experts, 4K prompt 27.3++// -> 21.3 s, decode unchanged); a discrete card in this band is not measured and takes the same, VRAM-bounded, rule.++// A unified-memory GPU never takes the small ring (its datapoint is the 1200 MB one): it keeps the 16 MB "fit" rule at 1/1.3.++inline int staging_ring_mb(double share, double room_bytes, int n_dev, bool integrated) {++ const double GB = 1e9;++ const double per_dev = n_dev > 0 ? room_bytes / (double) n_dev : 0.0;++ const double streamed = std::min(1200.0, std::max(400.0, room_bytes / GB * 64.0));++ if (integrated) return share >= 1.0 / 1.3 ? 16 : (int) streamed;++ if (share >= 0.82) return 16;++ if (share >= 0.5) return (int) std::min(512.0, std::max(128.0, per_dev / GB * 8.0));++ return (int) streamed;+}+// DS4_TOPK=N (a test, not for users): over the flag and the rule; returns the count to useinline int experts_test_override(int chosen, int model_k) {if (const char* v = std::getenv("DS4_TOPK")) {diff --cc src/dsv41/ds4_kernels.cuindex 96d2474aa,a0b7c38a6..a05091f01--- a/src/dsv41/ds4_kernels.cu+++ b/src/dsv41/ds4_kernels.cu@@@ -2026,40 -1998,17 +2026,42 @@@ bool split_enabled()}();return on;}-// The chunked prompt's packed-KV attention on the matrix cores (sparse_attn_wmma_kernel). ON by default: measured on-// 4x R9700, 2.25 bpw, 8K code prompts, warm medians of three a load -- 1180 ppts without it, 1354 with it (+14.7%),-// the same ids (5705e1164a). The QK/PV sums are the matrix cores' order, so the last places differ from the scalar-// kernel's and the owner approved that trade for the prefill; the decode path stays on the scalar kernels.-// DS4_FAST_ATTN=0 turns it off.+// DS4_FAST_ATTN: the chunked prompt's packed-KV attention on the matrix cores (sparse_attn_wmma_kernel). The+// QK/PV sums are the matrix cores' order, so the outputs differ from the scalar kernel's in the last places (the owner+// approved the trade for the prefill; the decode path stays on the scalar kernels).+// 0.2.7: ON by default on gfx1200 / gfx1201 (RDNA4: the R9700, RX 9000), where it was measured and gated (KL 0.0003 against the+// 0.006 gate, ids identical); gfx11 and gfx1151 (Strix Halo) are untested and keep the scalar kernels. DS4_FAST_ATTN=0 is the+// off switch (exactly the 0.2.6 arithmetic), DS4_FAST_ATTN=1 turns it on elsewhere. Decided per device (the first call there).++// The Marlin stream measured it ON on 4x R9700 (gfx1201), 2.25 bpw, 8K code prompts: 1180 -> 1354 ppts (+14.7%), the same ids,++// KL 0.10833 against 0.10925 for the scalar path. It stays gfx12-only until Strix Halo (gfx1151) is tested.+#if defined(STRATA_USE_HIP)+bool device_is_gfx12() {+ static std::atomic<int> cache[64]; // 0 unknown, 1 no, 2 yes+ int dev = 0;+ cudaGetDevice(&dev);+ if (dev < 0 || dev >= 64) return false;+ int c = cache[dev].load();+ if (c == 0) {+ cudaDeviceProp prop{};+ const bool yes = cudaGetDeviceProperties(&prop, dev) == cudaSuccess &&+ (std::strncmp(prop.gcnArchName, "gfx1200", 7) == 0 || std::strncmp(prop.gcnArchName, "gfx1201", 7) == 0);+ c = yes ? 2 : 1;+ cache[dev].store(c);+ }+ return c == 2;+}+#endifbool fast_attn_enabled() {- static const bool on = [] {+ static const int env = [] {const char* e = std::getenv("DS4_FAST_ATTN");- return e != nullptr ? std::atoi(e) != 0 : true;+ return e == nullptr ? -1 : (std::atoi(e) != 0 ? 1 : 0);}();- return on;+ if (env >= 0) return env == 1;+#if defined(STRATA_USE_HIP)+ return device_is_gfx12();+#else+ return false;+#endif}attnsplit::AttnSplitScratch split_scratch(cudaStream_t s, int heads) {static std::mutex mu;diff --cc src/dsv41/ds4_model.cuindex deee99ab7,ddafb5567..8c3052ae8--- a/src/dsv41/ds4_model.cu+++ b/src/dsv41/ds4_model.cu@@@ -2027,7 -1983,7 +2030,8 @@@ struct Model::Impl// 1024 200, 2048 277, 4096 353, 8192 431 tok/s; RTX 5070 / 3060 12 GB in place: 8192 244 / 228 against 1024 186 / 175); where they hold// most (4 x R9700: 8192 214, 1024 370) the small one wins; with the experts copied into RAM (--mmap-experts off) 4096 beats 8192 by 7-9%// (H100 32 GiB: 2048 502, 4096 538, 8192 495). A GGUF pack always has file_experts set (copied only by --mmap-experts off).-- const bool own_default = !(par || o.tp > 1); // (a single-card run: no --threads, tp 1 - a launch with several cards passes --tp 1; one card passes neither)++ const bool own_default = !(par || o.tp > 1); // (a run with tp 1 and no --threads: `strata launch` writes --tp 1 for 4+ AMD cards, --tp 2 for 2-3 and nothing for one card = the engine's tp 2)++ const bool multi_tp = !own_default && !o.threads && devs.size() >= 2; // (the 4-card default-tp run the 1024 chunk was also measured on; an explicit --prefill-chunk below wins over the rule)const bool gguf_copy = pk.gguf && (o.mmap_experts == 0 || (store_gb() <= 0.0 && auto_copy_all));uint64_t vmin_bytes = ~0ull;for (const Dev& dv : devs) vmin_bytes = std::min<uint64_t>(vmin_bytes, (uint64_t) dv.total);@@@ -2037,7 -1993,7 +2041,7 @@@#elseconst bool hip_build = false;#endif-- const int rule_chunk = prefill_chunk_default(own_default, gpu_expert_share(), gguf_copy, file_experts, hip_build, vmin_bytes, kVlRows);++ const int rule_chunk = prefill_chunk_default(own_default, gpu_expert_share(), gguf_copy, file_experts, hip_build, vmin_bytes, kVlRows, multi_tp);chunk = prompt_par ? (o.prefill_chunk >= 0 ? o.prefill_chunk : rule_chunk) : 0;if (prompt_par && o.prefill_chunk < 0 && rule_chunk == 4096 && gguf_copy)std::fprintf(stderr, "ds4: prompts in chunks of 4096 tokens (the experts are copied into RAM, %.0f GiB of graphics memory; --prefill-chunk N changes it)\n",@@@ -2962,21 -2911,27 +2966,23 @@@size_t ring_mb_auto() const {double exp_b = 0, other_b = 0, room = 0;const double share = gpu_expert_share(&exp_b, &room, &other_b);- const bool fit = exp_b > 0 && share >= 1.0 / 1.3; // (R9700 x4: 0.86 -> ring 16: decode -1.8% after a prompt, 64: -4.4%, 400: -4.6%)- // A box whose experts do not fit (a unified-memory APU) streams them from the disk, and the ring is what lets the- // prompt queue its likely ones early: 400 -> 1200 MB on Strix Halo took the queue 3,200 -> 8,087 experts, the disk- // 1.49 -> 2.44 GB/s and the compute's exposed read wait 11.4 -> 3.5 s (4K prompt 27.3 -> 21.3 s, 148.98 -> 190.98- // tok/s) with decode unchanged (17.54/16.39 against 17.61 tok/s interleaved). A sixteenth of the VRAM room, so a- // small pool still loads. DS4_RING_MB overrides.- const size_t want = (size_t) std::max<double>(400.0, std::min<double>(1200.0, room / 1e9 * 64.0));- // The prompt path streams the experts the cards cannot hold. When every expert fits, the ring is worth- // nothing (16 MB): the VRAM is better as slots for the decode after the prompt (measured: decode -1.8 to -4.6%- // at 64-400 MB, unchanged at 16). When they do not fit, the stream IS the prompt's expert stage, and its depth- // is what decides how long the tiles wait: on the 4x R9700 8K prompt a 16 MB ring left 1.7 s of DMA waits in a- // 5.2 s expert stage, and 1.2-3 GB of ring measured 632 -> 678-684 ppts (+7%) for about -2% decode (off a- // fifth more CPU-lane experts). So: a share of each device's room, bounded, rather than the fixed 400 MB.- const double per_dev = devs.empty() ? 0.0 : room / (double) devs.size();- const size_t deep = (size_t) std::min(2000.0, std::max(400.0, per_dev / 1e9 * 0.08 * 1000.0)); // (MB)- // ... but with the 1024-token prompt chunk (the default now when the cards hold most of the experts) the- // ring's balance moved: every MB it takes is a tenth of an expert the decode keeps. Measured on the same- // box, 8K prompt and six 128-token decodes a load, chunk 1024: 2000 MB 1243 ppts / 73.2 tok/s, 1000 MB- // 1287 / 77.5, 500 MB 1314 / 78.5, 256 MB 1324 / 78.4. So a small bounded ring where the cards hold the- // most of the experts, and the old depth where they do not (the streaming cases it was measured for).- const size_t small = (size_t) std::min(512.0, std::max(128.0, per_dev / 1e9 * 0.008 * 1000.0)); // (MB)- const size_t mb = share >= 0.98 ? 16 : (share >= 0.5 ? small : deep);++ // One rule for the staging ring (ds4_experts_rule.hpp: staging_ring_mb): 16 MB where the cards hold the experts, a small++ // ring from each device's room where they hold most of them (4 x R9700), the larger streamed ring from the summed room on a++ // unified-memory GPU (Strix Halo: 400 -> 1200 MB) and where the cards hold under half. DS4_RING_MB overrides.++ bool unified = false;++ for (const Dev& d : devs) {++ cudaDeviceProp ip{};++ if (cudaGetDeviceProperties(&ip, d.id) == cudaSuccess && ip.integrated != 0) unified = true;++ }++ const size_t mb = (size_t) strata::ds4::staging_ring_mb(exp_b > 0 ? share : 0.0, room, (int) devs.size(), unified);if (o.verbose)std::fprintf(stderr, "ds4: staging ring: experts %.1f GB, VRAM room %.1f GB, dense %.1f GB -> %d MB\n", exp_b / 1e9,- room / 1e9, other_b / 1e9, fit ? 16 : (int) want);- return fit ? 16 : want;+ room / 1e9, other_b / 1e9, (int) mb);+ return mb;}+ bool slab_auto_grew_ = false, slab_force_base_ = false; // the automatic attention slab: grew this try / must not grow (it did not fit)bool alloc_chunks(std::string& err) {+ slab_auto_grew_ = false;chunks.resize(devs.size());const int D = g.dim, H = g.hc, HD = g.hd;const int cols = (pk.eg_ngram - 1) * pk.eg_heads, ehd = pk.eg_hd;@@@ -3010,28 -2949,17 +3016,36 @@@// DS4_ATTN_SLAB=N: the attention slab's rows. The dense projections (q_b, wo_a/b) are dequantized and GEMM'd// once a slab, so a slab of 256 rows dequantizes the same weights 32 times for an 8K prompt (measured: the// dense dequant 916 ms of the 8K prompt, rocBLAS 867 ms); a larger slab costs VRAM for c.q/c.o and the oa- // buffers (A * HQ x 4 bytes each).- const int a_rule = std::max(64, (int) (((size_t) 8 << 20) / (size_t) HQ) / 64 * 64); // 256 rows here- c.A = a_rule;- // Where the cards hold most of the experts, the whole chunk's slab is worth its VRAM: measured on 4x R9700- // (2.25 bpw, share 0.77, chunk 1024) the 256-row rule gives 8K warm 1204 / 1236 / 1250 ppts and a slab of- // the chunk's 1024 rows 1324 / 1353 / 1363 (+9%) with the decode unchanged (77.7 against 78.1 tok/s); the- // 1024 rows cost 268 MB a device, 1% of the card. Elsewhere the rule stays as it was.- if (gpu_expert_share() >= 0.5) c.A = std::max(a_rule, std::min(chunk, 1024) / 64 * 64);- if (const char* v = std::getenv("DS4_ATTN_SLAB")) {- const int a = std::atoi(v);- if (a > 0) c.A = std::max(64, std::min(chunk, a) / 64 * 64);+ // buffers (A * HQ x 4 bytes each). 0.2.7: unset = automatic (attn_slab_rows: up to 2048 rows where the buffers+ // take VRAM no expert would have used), 0 = the 0.2.6 rule above, N = that many rows.+ {+ int env = -1;+ if (const char* v = std::getenv("DS4_ATTN_SLAB")) env = std::max(0, std::atoi(v));+ if ((slab_force_base_ || unified || a_mb_env != 0) && env < 0) env = 0; // (no automatic growth on a unified-memory GPU, or when STRATA_DS4_A_MB chose the base)+ // per row: c.q + c.o (float), c.o16 (half), c.oa (float), c.oa16 (half)+ const double bpr = (double) HQ * 10.0 + (double) OG * (double) g.o_lora * 6.0;+ double spare = 1e18; // the expert tier off (--vram-experts 0): the buffers cost no expert++ double share = 0.0; // the share of the routed experts the cards hold (0 with the tier off)+ if (o.vram_experts > 0.0) {+ double exp_b = 0, room = 0, other_b = 0;- gpu_expert_share(&exp_b, &room, &other_b);++ share = gpu_expert_share(&exp_b, &room, &other_b);+ spare = (room - other_b - exp_b) / (double) std::max<size_t>(1, devs.size());+ }+ const int base = c.A;- c.A = strata::ds4::attn_slab_rows(env, chunk, base, spare, bpr);++ // one rule (ds4_experts_rule.hpp: attn_slab_rows): the chunk's whole slab (<= 1024 rows) where the cards hold half the++ // experts or more (4 x R9700, +9% prompt), up to 2048 rows where the VRAM is spare, the base otherwise++ c.A = strata::ds4::attn_slab_rows(env, chunk, base, spare, bpr, share);+ if (env < 0 && c.A > base) {+ slab_auto_grew_ = true;- if (di == 0 && o.verbose)- std::fprintf(stderr, "ds4: attention slab %d rows (automatic: %.1f GB of VRAM to spare beyond the experts; DS4_ATTN_SLAB=0 keeps %d)\n",- c.A, spare / 1e9, base);++ if (di == 0 && o.verbose) {++ if (share >= 0.5)++ std::fprintf(stderr, "ds4: attention slab %d rows (automatic: the cards hold %.0f%% of the experts; DS4_ATTN_SLAB=0 keeps %d)\n",++ c.A, share * 100.0, base);++ else++ std::fprintf(stderr, "ds4: attention slab %d rows (automatic: %.1f GB of VRAM to spare beyond the experts; DS4_ATTN_SLAB=0 keeps %d)\n",++ c.A, spare / 1e9, base);++ }+ }}// (the indexer's slab: its scores rows are the context wide - about 64 MB of them)c.AI = std::max(32, std::min(128, (int) (((size_t) 64 << 20) / (((size_t) o.max_context + 64) * 4)) / 32 * 32));@@@ -4556,16 -4465,18 +4579,22 @@@std::fprintf(stderr, "ds4: %zu of the CPU lane's %zu experts copied into RAM (%.1f GB, %d NUMA part%s) in %.1f s%s\n",jobs.size(), hot.size(), cb / 1e9, NH, NH > 1 ? "s" : "", (now_ms() - t0) / 1000.0,jobs.size() < hot.size() ? "; the others read in place" : "");- // DS4_COPY_PIN=1: pin the copies, so the prompt path DMAs each expert from where it lies instead of copying it- // to the pinned staging first (measured on the R9700 box: the 8K pass streams 122 GB at 6.8 GB/s through the- // staging, where the RAM-to-staging and DMA probes are 24 and 28 GB/s). The CPU lane reads the same memory, and- // pinned 4 KB pages can cost its TLB (an earlier registration measured -15% decode), so this stays opt-in.+ // DS4_COPY_PIN: pin the copies, so the prompt path DMAs each expert from where it lies instead of copying it to+ // the pinned staging first (the pass streams 122 GB at 6.8 GB/s through the staging, where the RAM-to-staging+ // and DMA probes are 24 and 28 GB/s). The CPU lane reads the same memory, and pinned 4 KB pages can cost its+ // TLB (an earlier registration measured -15% decode on a 12 GB card, where the CPU lane does nearly all of the+ // decode) - so AUTO (unset) pins when the cards hold at least half the routed experts: the CPU lane is then a+ // minority of the decode and the prompt stream is what hurts. Measured on 4x R9700, 2.25 bpw (share 0.72),+ // three interleaved pairs: the second 8K prompt of a process 12.0/11.9/11.9 s against 28.8/13.8/12.3 s unpinned,+ // decode 35.42/35.50/34.9-36.0 against 35.43/35.45/34.9-36.0 tok/s (no cost). DS4_COPY_PIN=0/1 forces it.// DS4_COPY_PIN_RO=1: cudaHostRegisterReadOnly (the GPU only reads it).- // 0.2.7: DS4_COPY_PIN unset = ON only when the whole arena is on explicit 2 MB pages (a pin of a THP arena registers 4 KB- // pages and the DMA rate was erratic: 11.9 s one prompt, 30.4 s the next); =0 is the off switch, =1 pins any arena.++ // 0.2.7: DS4_COPY_PIN unset = ON when EITHER condition measured reliable holds: (a) the whole arena is on explicit 2 MB pages (a pin of a++ // THP arena registers 4 KB pages and the DMA rate was erratic on the 12 GB cell: 11.9 s one prompt, 30.4 s the next), or (b) the++ // cards hold at least half the routed experts (4 x R9700: the pin was reliable in 4/4 runs there, see above). =0 is the off++ // switch, =1 pins any arena.const char* cp = std::getenv("DS4_COPY_PIN");- const bool pin_copies = cp != nullptr ? std::atoi(cp) != 0 : (arena_all_huge && !ram_chunks.empty());- const bool pin_on = cp != nullptr ? std::atoi(cp) != 0 : gpu_expert_share() >= 0.5;- if (pin_on) {++ const bool pin_copies = cp != nullptr ? std::atoi(cp) != 0 : ((arena_all_huge || gpu_expert_share() >= 0.5) && !ram_chunks.empty());+ if (pin_copies) {const bool ro = std::getenv("DS4_COPY_PIN_RO") != nullptr && std::atoi(std::getenv("DS4_COPY_PIN_RO")) != 0;const double tr = now_ms();bool ok = true;diff --cc tests/dsv41/experts_rule_test.cppindex ffc5078d4,f0fad2dc5..fa1691bba--- a/tests/dsv41/experts_rule_test.cpp+++ b/tests/dsv41/experts_rule_test.cpp@@@ -8,7 -8,6 +8,8 @@@using strata::ds4::experts_per_word;using strata::ds4::experts_test_override;using strata::ds4::copy_all_fits;+using strata::ds4::attn_slab_rows;++using strata::ds4::staging_ring_mb;static int fails = 0;#define CHECK_EQ(a, b) \@@@ -71,26 -70,7 +72,70 @@@ int main()unsetenv("DS4_TOPK");#endifCHECK_EQ(experts_test_override(5, 6), 5);- // prompt chunk: prefill_chunk_default(own_default, gpu_share, gguf_copy, file_experts, hip, min_vram, big=8192). What `strata launch` runs on one card+ // attention slab rows: attn_slab_rows(env, chunk, base, spare_bytes, bytes_per_row). bpr of this model ~ 32768 * 10 + 8 * 1024 * 6 = 377 KB.+ {+ const double bpr = 377000.0;+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 1e18, bpr), 2048); // unset, the expert tier is off (--vram-experts 0): automatic 2048+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 4.0e9, bpr), 2048); // 4 GB of VRAM to spare: automatic+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, -5.0e9, bpr), 256); // the 5070: every spare byte is an expert's: the 0.2.6 slab+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 0.0, bpr), 256);+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 0.9e9, bpr), 256); // 1792 rows x 377 KB = 676 MB; needs 1.25 x + 256 MB = 1.1 GB+ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 1.2e9, bpr), 2048);+ CHECK_EQ(attn_slab_rows(-1, 1024, 256, 1e18, bpr), 1024); // a 1024 chunk caps the slab+ CHECK_EQ(attn_slab_rows(-1, 4096, 256, 1e18, bpr), 2048);+ CHECK_EQ(attn_slab_rows(-1, 128, 64, 1e18, bpr), 128);+ CHECK_EQ(attn_slab_rows(-1, 8192, 2560, 1e18, bpr), 2560); // a base already above the automatic size is kept+ CHECK_EQ(attn_slab_rows(0, 8192, 256, 1e18, bpr), 256); // DS4_ATTN_SLAB=0: the 0.2.6 rule, whatever the spare+ CHECK_EQ(attn_slab_rows(1024, 8192, 256, -5.0e9, bpr), 1024); // N: that many rows, the spare is the user's call+ CHECK_EQ(attn_slab_rows(100, 8192, 256, 1e18, bpr), 64); // rounded down to 64, at least 64+ CHECK_EQ(attn_slab_rows(99999, 8192, 256, 1e18, bpr), 8192); // at most the chunk+ CHECK_EQ(attn_slab_rows(300, 8192, 256, 1e18, bpr), 256);++ // the cards hold half the experts or more (expert_share, the 6th argument): the whole small chunk's slab, min(chunk, 1024) rows, whatever++ // the spare (the 4 x R9700 cell: chunk 1024, share 0.77, spare negative: 256 -> 1024 rows)++ CHECK_EQ(attn_slab_rows(-1, 1024, 256, -5.0e9, bpr, 0.77), 1024);++ CHECK_EQ(attn_slab_rows(-1, 1024, 256, -5.0e9, bpr, 0.50), 1024);++ CHECK_EQ(attn_slab_rows(-1, 1024, 256, -5.0e9, bpr, 0.49), 256); // under half: the base, as before++ CHECK_EQ(attn_slab_rows(-1, 512, 256, -5.0e9, bpr, 0.77), 512); // a chunk under 1024 caps it++ CHECK_EQ(attn_slab_rows(-1, 8192, 256, -5.0e9, bpr, 0.77), 1024); // a big chunk: still 1024 rows without the room for more++ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 4.0e9, bpr, 0.77), 2048); // the room is there too: the automatic growth wins++ CHECK_EQ(attn_slab_rows(-1, 8192, 256, 1e18, bpr, 1.0), 2048); // everything fits (a big card): 2048 as without the share++ CHECK_EQ(attn_slab_rows(-1, 8192, 2560, -5.0e9, bpr, 0.77), 2560); // never below the base++ CHECK_EQ(attn_slab_rows(0, 1024, 256, 1e18, bpr, 0.77), 256); // DS4_ATTN_SLAB=0: the 0.2.6 rule whatever the share++ CHECK_EQ(attn_slab_rows(512, 1024, 256, 1e18, bpr, 0.77), 512); // DS4_ATTN_SLAB=N: that many rows whatever the share++ CHECK_EQ(attn_slab_rows(-1, 1024, 1024, -5.0e9, bpr, 0.77), 1024); // a unified-memory base (32 MB of q = 1024 rows) is kept+ }- // prompt chunk: prefill_chunk_default(own_default, gpu_share, gguf_copy, file_experts, hip, min_vram, big=8192). What `strata launch` runs on one card++ // the staging ring (MB): staging_ring_mb(share, room_bytes, n_dev, integrated)++ {++ const double GB = 1e9;++ // the cards hold the experts: 16 MB, discrete (4 x R9700 at 0.86 and a card that holds everything)++ CHECK_EQ(staging_ring_mb(0.86, 4 * 30 * GB, 4, false), 16);++ CHECK_EQ(staging_ring_mb(1.00, 70 * GB, 1, false), 16);++ CHECK_EQ(staging_ring_mb(0.82, 4 * 30 * GB, 4, false), 16);++ // they hold most of them (4 x R9700, share 0.72-0.77, 31 GB of room a device): 8 MB a GB of one device's room = 248 MB++ CHECK_EQ(staging_ring_mb(0.77, 4 * 31 * GB, 4, false), 248);++ CHECK_EQ(staging_ring_mb(0.72, 4 * 31 * GB, 4, false), 248);++ CHECK_EQ(staging_ring_mb(0.81, 4 * 31 * GB, 4, false), 248);++ CHECK_EQ(staging_ring_mb(0.77, 4 * 10 * GB, 4, false), 128); // small devices: at least 128++ CHECK_EQ(staging_ring_mb(0.60, 3 * 90 * GB, 3, false), 512); // big devices: at most 512++ CHECK_EQ(staging_ring_mb(0.50, 2 * 31 * GB, 2, false), 248); // the per-device room, not the summed one (1 card: 31 GB -> 248 too)++ CHECK_EQ(staging_ring_mb(0.60, 31 * GB, 1, false), 248);++ // they hold under half (discrete): 64 MB a GB of the summed room, 400..1200 (streamed; not measured on a discrete card)++ CHECK_EQ(staging_ring_mb(0.19, 31 * GB, 1, false), 1200);++ CHECK_EQ(staging_ring_mb(0.06, 10 * GB, 1, false), 640); // a 12 GB card++ CHECK_EQ(staging_ring_mb(0.06, 4 * GB, 1, false), 400); // a small pool still gets 400++ CHECK_EQ(staging_ring_mb(0.40, 2 * 31 * GB, 2, false), 1200); // capped++ CHECK_EQ(staging_ring_mb(0.49, 4 * 31 * GB, 4, false), 1200);++ // a unified-memory GPU (Strix Halo): never the small ring; the streamed 400..1200 MB under 1/1.3, 16 above++ CHECK_EQ(staging_ring_mb(0.60, 40 * GB, 1, true), 1200); // share 0.6 would be the small ring on a discrete card++ CHECK_EQ(staging_ring_mb(0.30, 18.75 * GB, 1, true), 1200);++ CHECK_EQ(staging_ring_mb(0.30, 12 * GB, 1, true), 768);++ CHECK_EQ(staging_ring_mb(0.30, 5 * GB, 1, true), 400);++ CHECK_EQ(staging_ring_mb(0.77, 40 * GB, 1, true), 16); // main's "fit" threshold, 1/1.3 = 0.769++ CHECK_EQ(staging_ring_mb(0.76, 40 * GB, 1, true), 1200);++ // nothing known: no devices++ CHECK_EQ(staging_ring_mb(0.0, 0.0, 0, false), 400);++ }++ // prompt chunk: prefill_chunk_default(own_default, gpu_share, gguf_copy, file_experts, hip, min_vram, big=8192, multi_tp=false). What `strata launch` runs on one card// passes neither --tp nor --threads, but the engine's default tp is 2, so own_default is false there; a launch with several cards passes --tp 1.using strata::ds4::prefill_chunk_default;// RTX 5070 / RTX 3060 (12 GB), launch default: experts in place on a card under 16 GiB -> 4096 (their logs said 8192 before this rule)@@@ -123,8 -103,8 +168,17 @@@CHECK_EQ(prefill_chunk_default(true, 0.86, false, true, true, 32 * GiB), 1024);CHECK_EQ(prefill_chunk_default(true, 0.50, false, true, true, 32 * GiB), 1024);CHECK_EQ(prefill_chunk_default(true, 0.49, false, true, true, 32 * GiB), 8192);-- // threads / tp runs (own_default false) keep their defaults whatever the share: 8192 streamed, 2048 for a pack that is not a GGUF++ // a multi-card tp run (own_default false, multi_tp: 2+ cards, no --threads on; the engine's default tp 2, or --tp 2 from the launcher) takes++ // the small chunk too: the 4 x R9700 default-tp datapoint (8K warm 1247 ppts at 1024 against 690)++ CHECK_EQ(prefill_chunk_default(false, 0.86, false, true, true, 32 * GiB, 8192, true), 1024);++ CHECK_EQ(prefill_chunk_default(false, 0.50, false, true, true, 32 * GiB, 8192, true), 1024);++ CHECK_EQ(prefill_chunk_default(false, 0.49, false, true, true, 32 * GiB, 8192, true), 8192);++ // one card with the default tp, and `--threads on` (neither own_default nor multi_tp) keep their defaults whatever the share: 8192++ // streamed, 2048 for a pack that is not a GGUF, 4096 copied into RAM on a CUDA cardCHECK_EQ(prefill_chunk_default(false, 0.86, false, true, true, 32 * GiB), 8192);++ CHECK_EQ(prefill_chunk_default(false, 1.00, false, true, true, 80 * GiB), 8192);++ CHECK_EQ(prefill_chunk_default(false, 1.00, true, true, false, 80 * GiB), 4096);++ CHECK_EQ(prefill_chunk_default(false, 0.86, false, false, true, 32 * GiB), 2048);CHECK_EQ(prefill_chunk_default(false, 0.30, false, false, false, 24 * GiB), 2048);// the rule takes no --max-context: a 128K session on a 12 GB card gets the same chunk, because the chunk's buffers and the KV are// allocated whole at load for that context and a chunk that does not fit there is halved before the first prompt (4096 -> 2048 -> 1024 ...)