Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
expert source: STRATA_RAM_ADAPT copies the incoming expert out of its stage buffer when it is still there, and moves up to 4 pairs a layer (#3060)
SanghunYun2b13a44933e2
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex a53ede74f..a6e35c0b3 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -238,8 +238,12 @@ When the RAM cannot hold them beside the system (the experts plus about 10 GB),- The adaptive tier (`--adapt-every`) keeps following the conversation in the resident mode. `--adapt-async 1` (opt-in,`--serve`) moves the swaps between decode windows on a helper thread. With `--batch` slots it works between theirwindows too, but not with `--batch-groups` or `--peer-device`. It is not bit-exact from run to run.-- `STRATA_RAM_ADAPT=<n>` (opt-in, `--serve`, one GPU, needs `--adapt-every`): the RAM copy also follows the conversation,- with up to `n` experts swapped per round.+- `STRATA_RAM_ADAPT=<n>` (opt-in, `--serve`, one GPU, needs `--adapt-every`): the RAM copy also follows the conversation.+ A layer swaps up to 4 pairs per window: the experts the windows keep routing to (decayed count 2 or more) take the+ RAM places of unused experts of the same layer (count below 0.5), largest difference first. The incoming expert was+ usually read for the window before; when its blob still sits in that read's stage buffer it is copied from there and+ no file read is made. `n` caps only the pairs that need a file read (`=1` is enough in practice). The log counts how+ many experts moved and how many came out of a stage buffer.- Several GPUs: setup recommends one GPU in the low-RAM mode and asks. `--gpus 0,1` shares the model across both cards([MULTI_GPU.md](MULTI_GPU.md#using-it)). `--yes` keeps one GPU.- With a page-locked copy the GPU also takes its share of the misses over PCIe (`--pcie-frac`). `--pcie-frac 0` (or@@ -1230,6 +1234,7 @@ which about doubled prompt speed on the measured card.| 0.2.0 | RTX PRO 5000 emulating a 12 GB card (`--expert-cache 4448`, Q2_0 GGUF in place, cgroup RAM limit), 3 interleaved pairs | Zero-copy, io prefetch and the decode gate vs 0.1.4x: 32 GB RAM decode 1.70 / 2.56 / 2.88x (story / code / doc8k; 34.0 / 28.0 / 21.7 -> 57.6 / 71.7 / 62.6 tok/s), prompts 1.54 / 1.91x; 16 GB decode 1.16 / 1.25 / 1.25x (the gate stays closed), prompts 1.63 / 1.68x. On a real RTX 3060 12 GB (32 GB cgroup) zero-copy alone gave 1.14 / 1.44 / 1.47x decode || 0.2.1 | RTX 3060 12 GB, IQ3_XXS `experts.bin` pack (43 GB), `--mmap-experts`, cgroup RAM limit, 5 pairs | io prefetch and the gate: 32 GB decode 1.38x / 1.21x / 1.33x (story / code / doc8k), prompts 4K 1.04x, 20K 1.06x. At 16 GB decode +3..8% but the cold 4K prompt 4-6% slower, so it stays off (room 38%) || 0.1.43 | RTX 3060 12 GB, 32 GB, Linux, IQ2_XS, 20.5 GiB experts in RAM, v0.1.43 as baseline, 47 frozen Korean requests replayed in fresh engines | `STRATA_RAM_ADAPT`: decode 43.26 -> 50.40 tok/s (+16.5%); file-tier demand reads per generated token 14.15 -> 2.67. One machine, one quantization |+| 0.2.1 (#3060) | RTX 3060 12 GB, 32 GB RAM, Linux NVMe, IQ2_XS and IQ3_XXS read in place, 20.5 GiB of experts in RAM, `--adapt-async 1`, `--spec 3`, fresh engine and dropped page cache per run, 3 interleaved runs | `STRATA_RAM_ADAPT` with the stage-buffer copy and `=1` against `=8` before it, 120 ShareGPT requests: IQ2_XS decode 48.71 -> 51.40 tok/s, prompt 137.4 -> 141.5, storage reads 408 -> 308 GB; IQ3_XXS decode 33.07 -> 38.71, prompt 97.8 -> 104.2, reads 1105 -> 717 GB (off: 37.70 and 23.24 tok/s). `standard-v1`: within noise. One machine; the author's numbers, not yet repeated here || 0.1.45.3 | RTX 3060 12 GB, IQ2_XS, 20 GiB of experts in RAM, cgroup 34 GB, SATA emulated at 450 MB/s, 5 pairs | Prompt-chunk planner `8192 -> 3840`: 4,101 tokens 138.1 vs 138.2 tok/s (one chunk either way); 20,400 tokens 334.0 -> 822.3 tok/s (2.46x, drive read 26.1 -> 2.7 GB, decode after 25.4 -> 42.6); 28,887 tokens 418.8 -> 910.4 (2.17x). With a real NVMe and a 32 GB cap it picks 8192 again || 2026-10-11 reports | RTX 3090 + Ryzen 5 5600X | `--adapt-async 1` on the arena: x1.07-1.11 decode, 12 of 12 prompts faster ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-async/README.md)). `STRATA_PIN_BY_NEED=1`: prompt read +5-7% at 56K-117K tokens ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need/README.md)) || 2026-10-11 report | RTX 5090 / Ryzen 9 9950X, paired 4K three-prompt test | `STRATA_ADAPT_CALLBACK_WARMUP`: 142.82 -> 207.75 tok/s ([report](../bench/results/2026-10-11-rtx-5090-callback-affinity/README.md)) |diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hppindex 3ab77dc2a..2ca367841 100644--- a/include/strata/core/expert_source.hpp+++ b/include/strata/core/expert_source.hpp@@ -819,7 +819,11 @@ public:/// windows, with no adaptive round in flight. A failed read leaves the copy as it was. Not with/// STRATA_EXCHANGE_ROTATE (false, nothing changes).bool replace_resident(int64_t layer, int64_t in, int64_t out);+ /// STRATA_RAM_ADAPT: the same move as replace_resident, but `in` is copied out of the finished stage buffer+ /// a demand read left it in (no file read). False, nothing changed, when `in` is not staged or still filling.+ bool replace_resident_staged(int64_t layer, int64_t in, int64_t out);int64_t ram_adapts() const { return ram_adapts_; }+ int64_t ram_staged() const { return ram_staged_; } ///< of ram_adapts(): copied out of a stage buffer// ---- 149-prefetch (STRATA_RAM_ADAPT2): the same promotion, staged. `ram_stage` queues the read of `in` on an I/O thread// (false: already queued, the queue is full, or no compact copy); `ram_install` - called by the decode thread at a window// boundary, when no window reads the RAM copy - puts every read that has landed into `out`'s place, unless the GPU tier@@ -1106,7 +1110,8 @@ private:std::unordered_map<int64_t, size_t> stage_of_;uint64_t stage_blob_ = 0;uint64_t stage_seq_ = 0;- int64_t ram_adapts_ = 0; ///< replace_resident: experts moved from the file into the compact copy+ int64_t ram_adapts_ = 0; ///< replace_resident(_staged): experts moved into the compact copy+ int64_t ram_staged_ = 0; ///< replace_resident_staged: of those, copied out of a stage bufferstd::vector<uint8_t> ram_adapt_buf_; ///< ...one blob, read here before it replaces another// 149-prefetch: the staged promotions (ram_stage / ram_install)struct RamJob { int64_t layer, in, out; std::vector<uint8_t> buf; int state; }; ///< state 0 queued, 1 reading, 2 landed, 3 faileddiff --git a/src/core/expert_source.cpp b/src/core/expert_source.cppindex b8ad8fb42..d0ce8bbbd 100644--- a/src/core/expert_source.cpp+++ b/src/core/expert_source.cpp@@ -4341,6 +4341,33 @@ bool FileExpertSource::replace_resident(int64_t layer, int64_t in, int64_t out)return install_resident(layer, in, out, ram_adapt_buf_.data());}+bool FileExpertSource::replace_resident_staged(int64_t layer, int64_t in, int64_t out) {+ if (!complement_ready_ || complement_host_ == nullptr || exchange_storage_.active() || !staged_.empty() ||+ layer < 0 || layer >= n_layers_ || in < 0 || out < 0 || in >= n_expert_ || out >= n_expert_ || in == out)+ return false;+ const size_t i_in = (size_t) layer * (size_t) n_expert_ + (size_t) in;+ const size_t i_out = (size_t) layer * (size_t) n_expert_ + (size_t) out;+ if (i_in >= complement_offsets_.size() || i_out >= complement_offsets_.size()) return false;+ if (!override_.empty() && (override_[i_in] != nullptr || override_[i_out] != nullptr)) return false;+ const uint64_t at = complement_offsets_[i_out], bytes = layer_blob_bytes_[(size_t) layer];+ if (at == kNoComplement || complement_offsets_[i_in] != kNoComplement || at > complement_bytes_ ||+ bytes > complement_bytes_ - at || bytes > stage_blob_) return false;+ ForesightSwap* fs = fs_swap_.load(std::memory_order_acquire);+ bool moved = false;+ {+ // held across the copy: claim_stage cannot hand the buffer to another blob meanwhile+ std::lock_guard<std::mutex> lk(stage_mu_);+ const auto it = stage_of_.find((int64_t) i_in);+ if (it == stage_of_.end() || stage_busy_[it->second] || stage_key_[it->second] != (int64_t) i_in) return false;+ if (fs != nullptr) fs->hold(layer, (int32_t) out);+ std::memcpy((uint8_t*) complement_host_ + (size_t) at, stage_buf_[it->second].get(), (size_t) bytes);+ moved = detail::exchange_cache_complement(complement_offsets_, i_out, i_in);+ if (fs != nullptr) fs->unhold(layer, (int32_t) out);+ }+ if (moved) { ++ram_adapts_; ++ram_staged_; }+ return moved;+}+// `in`'s bytes (src) into `out`'s place of the compact copy, and the two offsets swapped - the checks again, as the caller// may have read `src` a while ago (149-prefetch: ram_install)bool FileExpertSource::install_resident(int64_t layer, int64_t in, int64_t out, const uint8_t* src) {diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 4ce6a3dc0..a57ea8717 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -15344,25 +15344,40 @@ int main(int argc, char** argv) {if (ram_adapt_n > 0 && !first_window && !drive.d.usage.empty() && pending.empty() &&(!ajob || astate == AState::Idle) && src.complement_ready() && !multi_gpu && !peer.valid() &&!remote_opt && o.batch == 0) {+ // up to kRamAdaptPerLayer pairs a layer: the layer's most routed experts outside the copy against its least+ // routed ones in it, paired in that order while the outgoing one is still unusedconstexpr float kRamAdaptRouted = 2.0f, kRamAdaptUnused = 0.5f;+ constexpr size_t kRamAdaptPerLayer = 4;struct RamSwap { float gain; int32_t layer, in, out; };std::vector<RamSwap> swaps;+ std::vector<int32_t> ins, outs;for (int64_t l = 0; l < g.n_layers; ++l) {const float* u = drive.d.usage.data() + l * g.n_expert;const int32_t* r = host_res.data() + l * g.n_expert;- int32_t in = -1, out = -1;+ ins.clear();+ outs.clear();for (int32_t e = 0; e < (int32_t) g.n_expert; ++e) {if (r[e] >= 0) continue; // in VRAM: the RAM copy does not hold it- if (src.has_resident(l, e)) { if (out < 0 || u[e] < u[out]) out = e; }- else if (u[e] >= kRamAdaptRouted && (in < 0 || u[e] > u[in])) in = e;+ if (src.has_resident(l, e)) outs.push_back(e);+ else if (u[e] >= kRamAdaptRouted) ins.push_back(e);}- if (in >= 0 && out >= 0 && u[out] < kRamAdaptUnused)- swaps.push_back({u[in] - u[out], (int32_t) l, in, out});+ const size_t ni = std::min(ins.size(), kRamAdaptPerLayer), no = std::min(outs.size(), kRamAdaptPerLayer);+ std::partial_sort(ins.begin(), ins.begin() + (std::ptrdiff_t) ni, ins.end(),+ [&](int32_t a, int32_t b) { return u[a] > u[b]; });+ std::partial_sort(outs.begin(), outs.begin() + (std::ptrdiff_t) no, outs.end(),+ [&](int32_t a, int32_t b) { return u[a] < u[b]; });+ for (size_t i = 0; i < ni && i < no && u[outs[i]] < kRamAdaptUnused; ++i)+ swaps.push_back({u[ins[i]] - u[outs[i]], (int32_t) l, ins[i], outs[i]});}std::sort(swaps.begin(), swaps.end(),[](const RamSwap& a, const RamSwap& b) { return a.gain > b.gain; });- if ((int) swaps.size() > ram_adapt_n) swaps.resize((size_t) ram_adapt_n);- for (const RamSwap& x : swaps) src.replace_resident(x.layer, x.in, x.out);+ // the incoming expert was usually just read for the window before: a pair whose `in` still sits in that+ // read's stage buffer moves with a copy out of it (no file read); n caps only the pairs that need the file read+ int file_left = ram_adapt_n;+ for (const RamSwap& x : swaps) {+ if (src.replace_resident_staged(x.layer, x.in, x.out)) continue;+ if (file_left > 0) { --file_left; src.replace_resident(x.layer, x.in, x.out); }+ }}if (!adapt_tick(false)) { // --adapt-async: the round in flight moves on a step when it canstd::printf("ERR an adaptive refill failed\n");@@ -15881,8 +15896,9 @@ int main(int argc, char** argv) {(long long) src.ram_staged_reads(),(long long) ram_adapt2.installed, (long long) (src.ram_install_skips() + src.ram_install_gpu()), (long long) src.ram_install_gpu());if (src.ram_adapts() > 0) // STRATA_RAM_ADAPT, cumulative- std::fprintf(stderr, "strata serve: resident RAM: %lld experts moved in from the file in place of "- "unused ones (STRATA_RAM_ADAPT)\n", (long long) src.ram_adapts());+ std::fprintf(stderr, "strata serve: resident RAM: %lld experts moved in in place of unused ones, %lld of "+ "them out of a stage buffer (STRATA_RAM_ADAPT)\n", (long long) src.ram_adapts(),+ (long long) src.ram_staged());if (ajob) // --adapt-async, cumulative (a round may still be in flight: it lands before the next request)std::fprintf(stderr, "strata serve: asynchronous adaptive tier: %lld rounds, %lld experts swapped in, ""%.1f ms per round (start to flip, between windows)",diff --git a/tests/core/file_expert_source_test.cpp b/tests/core/file_expert_source_test.cppindex e162664d1..d6a274f1a 100644--- a/tests/core/file_expert_source_test.cpp+++ b/tests/core/file_expert_source_test.cpp@@ -979,6 +979,28 @@ void test_replace_resident(bool pin) {require(src.commit_exchanges() == 1 && src.has_resident(0, 3) && holds(3) && !src.has_resident(0, 1),"exchange after replace wrong");require(src.ram_adapts() == 2, "accounting wrong");+ // a staged incoming blob: RAM {2,3}, VRAM {0,1}, 4 on the file tier+ require(!src.replace_resident_staged(0, 4, 3) && src.has_resident(0, 3) && src.ram_adapts() == 2,+ "an unstaged expert moved");+ std::string why;+ set_env("STRATA_UNBUFFERED_LOAD", "1");+ const bool unbuffered = src.set_unbuffered(0, why);+ set_env("STRATA_UNBUFFERED_LOAD", nullptr);+ if (unbuffered && src.stages_blobs()) {+ require(!src.replace_resident_staged(0, 4, 3) && holds(3), "moved before any demand read");+ require(!src.has_resident(0, 4) && holds(4), "demand read wrong"); // leaves 4 in a stage buffer+ require(!src.replace_resident_staged(0, 4, 0) && !src.replace_resident_staged(0, 2, 3),+ "invalid staged pair accepted");+ require(src.replace_resident_staged(0, 4, 3) && src.has_resident(0, 4) && !src.has_resident(0, 3),+ "staged replacement residency wrong");+ require(src.resident_blob(0, 4) != nullptr && !std::memcmp(src.resident_blob(0, 4), truth[4].data(), bytes),+ "staged replacement bytes wrong");+ require(holds(2) && holds(3) && src.ram_adapts() == 3 && src.ram_staged() == 1,+ "staged replacement changed another expert");+ std::cout << "replace_resident_staged: pinned=" << pin << " exact_bytes=PASS\n";+ } else {+ std::cout << "replace_resident_staged: pinned=" << pin << " unbuffered unavailable here (" << why << "), positive case SKIPPED\n";+ }std::cout << "replace_resident: pinned=" << pin << " exact_bytes=PASS\n";}