Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
DeepSeek engine (AMD): a small staging ring where the cards hold most of the experts
Niko12214aa645c0075d
diff --git a/src/dsv41/ds4_model.cu b/src/dsv41/ds4_model.cuindex 5df38cf88..94641b4e0 100644--- a/src/dsv41/ds4_model.cu+++ b/src/dsv41/ds4_model.cu@@ -2919,7 +2919,13 @@ struct Model::Impl {// fifth more CPU-lane experts). So: a share of each device's room, bounded, rather than the fixed 400 MB.const double per_dev = devs.empty() ? 0.0 : room / (double) devs.size();const size_t deep = (size_t) std::min(2000.0, std::max(400.0, per_dev / 1e9 * 0.08 * 1000.0)); // (MB)- const size_t mb = share >= 0.98 ? 16 : deep;+ // ... but with the 1024-token prompt chunk (the default now when the cards hold most of the experts) the+ // ring's balance moved: every MB it takes is a tenth of an expert the decode keeps. Measured on the same+ // box, 8K prompt and six 128-token decodes a load, chunk 1024: 2000 MB 1243 ppts / 73.2 tok/s, 1000 MB+ // 1287 / 77.5, 500 MB 1314 / 78.5, 256 MB 1324 / 78.4. So a small bounded ring where the cards hold the+ // most of the experts, and the old depth where they do not (the streaming cases it was measured for).+ const size_t small = (size_t) std::min(512.0, std::max(128.0, per_dev / 1e9 * 0.008 * 1000.0)); // (MB)+ const size_t mb = share >= 0.98 ? 16 : (share >= 0.5 ? small : deep);if (o.verbose)std::fprintf(stderr, "ds4: staging ring: experts %.1f GB, VRAM room %.1f GB, dense %.1f GB -> %d MB\n", exp_b / 1e9,room / 1e9, other_b / 1e9, (int) mb);