Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

expert source: STRATA_RAM_ADAPT copies the incoming expert out of its stage buffer when it is still there, and moves up to 4 pairs a layer (#3060)

SanghunYun2b13a44933e2
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index a53ede74f..a6e35c0b3 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -238,8 +238,12 @@ When the RAM cannot hold them beside the system (the experts plus about 10 GB),
- The adaptive tier (`--adapt-every`) keeps following the conversation in the resident mode. `--adapt-async 1` (opt-in,
`--serve`) moves the swaps between decode windows on a helper thread. With `--batch` slots it works between their
windows too, but not with `--batch-groups` or `--peer-device`. It is not bit-exact from run to run.
-- `STRATA_RAM_ADAPT=<n>` (opt-in, `--serve`, one GPU, needs `--adapt-every`): the RAM copy also follows the conversation,
- with up to `n` experts swapped per round.
+- `STRATA_RAM_ADAPT=<n>` (opt-in, `--serve`, one GPU, needs `--adapt-every`): the RAM copy also follows the conversation.
+ A layer swaps up to 4 pairs per window: the experts the windows keep routing to (decayed count 2 or more) take the
+ RAM places of unused experts of the same layer (count below 0.5), largest difference first. The incoming expert was
+ usually read for the window before; when its blob still sits in that read's stage buffer it is copied from there and
+ no file read is made. `n` caps only the pairs that need a file read (`=1` is enough in practice). The log counts how
+ many experts moved and how many came out of a stage buffer.
- Several GPUs: setup recommends one GPU in the low-RAM mode and asks. `--gpus 0,1` shares the model across both cards
([MULTI_GPU.md](MULTI_GPU.md#using-it)). `--yes` keeps one GPU.
- With a page-locked copy the GPU also takes its share of the misses over PCIe (`--pcie-frac`). `--pcie-frac 0` (or
@@ -1230,6 +1234,7 @@ which about doubled prompt speed on the measured card.
| 0.2.0 | RTX PRO 5000 emulating a 12 GB card (`--expert-cache 4448`, Q2_0 GGUF in place, cgroup RAM limit), 3 interleaved pairs | Zero-copy, io prefetch and the decode gate vs 0.1.4x: 32 GB RAM decode 1.70 / 2.56 / 2.88x (story / code / doc8k; 34.0 / 28.0 / 21.7 -> 57.6 / 71.7 / 62.6 tok/s), prompts 1.54 / 1.91x; 16 GB decode 1.16 / 1.25 / 1.25x (the gate stays closed), prompts 1.63 / 1.68x. On a real RTX 3060 12 GB (32 GB cgroup) zero-copy alone gave 1.14 / 1.44 / 1.47x decode |
| 0.2.1 | RTX 3060 12 GB, IQ3_XXS `experts.bin` pack (43 GB), `--mmap-experts`, cgroup RAM limit, 5 pairs | io prefetch and the gate: 32 GB decode 1.38x / 1.21x / 1.33x (story / code / doc8k), prompts 4K 1.04x, 20K 1.06x. At 16 GB decode +3..8% but the cold 4K prompt 4-6% slower, so it stays off (room 38%) |
| 0.1.43 | RTX 3060 12 GB, 32 GB, Linux, IQ2_XS, 20.5 GiB experts in RAM, v0.1.43 as baseline, 47 frozen Korean requests replayed in fresh engines | `STRATA_RAM_ADAPT`: decode 43.26 -> 50.40 tok/s (+16.5%); file-tier demand reads per generated token 14.15 -> 2.67. One machine, one quantization |
+| 0.2.1 (#3060) | RTX 3060 12 GB, 32 GB RAM, Linux NVMe, IQ2_XS and IQ3_XXS read in place, 20.5 GiB of experts in RAM, `--adapt-async 1`, `--spec 3`, fresh engine and dropped page cache per run, 3 interleaved runs | `STRATA_RAM_ADAPT` with the stage-buffer copy and `=1` against `=8` before it, 120 ShareGPT requests: IQ2_XS decode 48.71 -> 51.40 tok/s, prompt 137.4 -> 141.5, storage reads 408 -> 308 GB; IQ3_XXS decode 33.07 -> 38.71, prompt 97.8 -> 104.2, reads 1105 -> 717 GB (off: 37.70 and 23.24 tok/s). `standard-v1`: within noise. One machine; the author's numbers, not yet repeated here |
| 0.1.45.3 | RTX 3060 12 GB, IQ2_XS, 20 GiB of experts in RAM, cgroup 34 GB, SATA emulated at 450 MB/s, 5 pairs | Prompt-chunk planner `8192 -> 3840`: 4,101 tokens 138.1 vs 138.2 tok/s (one chunk either way); 20,400 tokens 334.0 -> 822.3 tok/s (2.46x, drive read 26.1 -> 2.7 GB, decode after 25.4 -> 42.6); 28,887 tokens 418.8 -> 910.4 (2.17x). With a real NVMe and a 32 GB cap it picks 8192 again |
| 2026-10-11 reports | RTX 3090 + Ryzen 5 5600X | `--adapt-async 1` on the arena: x1.07-1.11 decode, 12 of 12 prompts faster ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-async/README.md)). `STRATA_PIN_BY_NEED=1`: prompt read +5-7% at 56K-117K tokens ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need/README.md)) |
| 2026-10-11 report | RTX 5090 / Ryzen 9 9950X, paired 4K three-prompt test | `STRATA_ADAPT_CALLBACK_WARMUP`: 142.82 -> 207.75 tok/s ([report](../bench/results/2026-10-11-rtx-5090-callback-affinity/README.md)) |
diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp
index 3ab77dc2a..2ca367841 100644
--- a/include/strata/core/expert_source.hpp
+++ b/include/strata/core/expert_source.hpp
@@ -819,7 +819,11 @@ public:
/// windows, with no adaptive round in flight. A failed read leaves the copy as it was. Not with
/// STRATA_EXCHANGE_ROTATE (false, nothing changes).
bool replace_resident(int64_t layer, int64_t in, int64_t out);
+ /// STRATA_RAM_ADAPT: the same move as replace_resident, but `in` is copied out of the finished stage buffer
+ /// a demand read left it in (no file read). False, nothing changed, when `in` is not staged or still filling.
+ bool replace_resident_staged(int64_t layer, int64_t in, int64_t out);
int64_t ram_adapts() const { return ram_adapts_; }
+ int64_t ram_staged() const { return ram_staged_; } ///< of ram_adapts(): copied out of a stage buffer
// ---- 149-prefetch (STRATA_RAM_ADAPT2): the same promotion, staged. `ram_stage` queues the read of `in` on an I/O thread
// (false: already queued, the queue is full, or no compact copy); `ram_install` - called by the decode thread at a window
// boundary, when no window reads the RAM copy - puts every read that has landed into `out`'s place, unless the GPU tier
@@ -1106,7 +1110,8 @@ private:
std::unordered_map<int64_t, size_t> stage_of_;
uint64_t stage_blob_ = 0;
uint64_t stage_seq_ = 0;
- int64_t ram_adapts_ = 0; ///< replace_resident: experts moved from the file into the compact copy
+ int64_t ram_adapts_ = 0; ///< replace_resident(_staged): experts moved into the compact copy
+ int64_t ram_staged_ = 0; ///< replace_resident_staged: of those, copied out of a stage buffer
std::vector<uint8_t> ram_adapt_buf_; ///< ...one blob, read here before it replaces another
// 149-prefetch: the staged promotions (ram_stage / ram_install)
struct RamJob { int64_t layer, in, out; std::vector<uint8_t> buf; int state; }; ///< state 0 queued, 1 reading, 2 landed, 3 failed
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
index b8ad8fb42..d0ce8bbbd 100644
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -4341,6 +4341,33 @@ bool FileExpertSource::replace_resident(int64_t layer, int64_t in, int64_t out)
return install_resident(layer, in, out, ram_adapt_buf_.data());
}
+bool FileExpertSource::replace_resident_staged(int64_t layer, int64_t in, int64_t out) {
+ if (!complement_ready_ || complement_host_ == nullptr || exchange_storage_.active() || !staged_.empty() ||
+ layer < 0 || layer >= n_layers_ || in < 0 || out < 0 || in >= n_expert_ || out >= n_expert_ || in == out)
+ return false;
+ const size_t i_in = (size_t) layer * (size_t) n_expert_ + (size_t) in;
+ const size_t i_out = (size_t) layer * (size_t) n_expert_ + (size_t) out;
+ if (i_in >= complement_offsets_.size() || i_out >= complement_offsets_.size()) return false;
+ if (!override_.empty() && (override_[i_in] != nullptr || override_[i_out] != nullptr)) return false;
+ const uint64_t at = complement_offsets_[i_out], bytes = layer_blob_bytes_[(size_t) layer];
+ if (at == kNoComplement || complement_offsets_[i_in] != kNoComplement || at > complement_bytes_ ||
+ bytes > complement_bytes_ - at || bytes > stage_blob_) return false;
+ ForesightSwap* fs = fs_swap_.load(std::memory_order_acquire);
+ bool moved = false;
+ {
+ // held across the copy: claim_stage cannot hand the buffer to another blob meanwhile
+ std::lock_guard<std::mutex> lk(stage_mu_);
+ const auto it = stage_of_.find((int64_t) i_in);
+ if (it == stage_of_.end() || stage_busy_[it->second] || stage_key_[it->second] != (int64_t) i_in) return false;
+ if (fs != nullptr) fs->hold(layer, (int32_t) out);
+ std::memcpy((uint8_t*) complement_host_ + (size_t) at, stage_buf_[it->second].get(), (size_t) bytes);
+ moved = detail::exchange_cache_complement(complement_offsets_, i_out, i_in);
+ if (fs != nullptr) fs->unhold(layer, (int32_t) out);
+ }
+ if (moved) { ++ram_adapts_; ++ram_staged_; }
+ return moved;
+}
+
// `in`'s bytes (src) into `out`'s place of the compact copy, and the two offsets swapped - the checks again, as the caller
// may have read `src` a while ago (149-prefetch: ram_install)
bool FileExpertSource::install_resident(int64_t layer, int64_t in, int64_t out, const uint8_t* src) {
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 4ce6a3dc0..a57ea8717 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -15344,25 +15344,40 @@ int main(int argc, char** argv) {
if (ram_adapt_n > 0 && !first_window && !drive.d.usage.empty() && pending.empty() &&
(!ajob || astate == AState::Idle) && src.complement_ready() && !multi_gpu && !peer.valid() &&
!remote_opt && o.batch == 0) {
+ // up to kRamAdaptPerLayer pairs a layer: the layer's most routed experts outside the copy against its least
+ // routed ones in it, paired in that order while the outgoing one is still unused
constexpr float kRamAdaptRouted = 2.0f, kRamAdaptUnused = 0.5f;
+ constexpr size_t kRamAdaptPerLayer = 4;
struct RamSwap { float gain; int32_t layer, in, out; };
std::vector<RamSwap> swaps;
+ std::vector<int32_t> ins, outs;
for (int64_t l = 0; l < g.n_layers; ++l) {
const float* u = drive.d.usage.data() + l * g.n_expert;
const int32_t* r = host_res.data() + l * g.n_expert;
- int32_t in = -1, out = -1;
+ ins.clear();
+ outs.clear();
for (int32_t e = 0; e < (int32_t) g.n_expert; ++e) {
if (r[e] >= 0) continue; // in VRAM: the RAM copy does not hold it
- if (src.has_resident(l, e)) { if (out < 0 || u[e] < u[out]) out = e; }
- else if (u[e] >= kRamAdaptRouted && (in < 0 || u[e] > u[in])) in = e;
+ if (src.has_resident(l, e)) outs.push_back(e);
+ else if (u[e] >= kRamAdaptRouted) ins.push_back(e);
}
- if (in >= 0 && out >= 0 && u[out] < kRamAdaptUnused)
- swaps.push_back({u[in] - u[out], (int32_t) l, in, out});
+ const size_t ni = std::min(ins.size(), kRamAdaptPerLayer), no = std::min(outs.size(), kRamAdaptPerLayer);
+ std::partial_sort(ins.begin(), ins.begin() + (std::ptrdiff_t) ni, ins.end(),
+ [&](int32_t a, int32_t b) { return u[a] > u[b]; });
+ std::partial_sort(outs.begin(), outs.begin() + (std::ptrdiff_t) no, outs.end(),
+ [&](int32_t a, int32_t b) { return u[a] < u[b]; });
+ for (size_t i = 0; i < ni && i < no && u[outs[i]] < kRamAdaptUnused; ++i)
+ swaps.push_back({u[ins[i]] - u[outs[i]], (int32_t) l, ins[i], outs[i]});
}
std::sort(swaps.begin(), swaps.end(),
[](const RamSwap& a, const RamSwap& b) { return a.gain > b.gain; });
- if ((int) swaps.size() > ram_adapt_n) swaps.resize((size_t) ram_adapt_n);
- for (const RamSwap& x : swaps) src.replace_resident(x.layer, x.in, x.out);
+ // the incoming expert was usually just read for the window before: a pair whose `in` still sits in that
+ // read's stage buffer moves with a copy out of it (no file read); n caps only the pairs that need the file read
+ int file_left = ram_adapt_n;
+ for (const RamSwap& x : swaps) {
+ if (src.replace_resident_staged(x.layer, x.in, x.out)) continue;
+ if (file_left > 0) { --file_left; src.replace_resident(x.layer, x.in, x.out); }
+ }
}
if (!adapt_tick(false)) { // --adapt-async: the round in flight moves on a step when it can
std::printf("ERR an adaptive refill failed\n");
@@ -15881,8 +15896,9 @@ int main(int argc, char** argv) {
(long long) src.ram_staged_reads(),
(long long) ram_adapt2.installed, (long long) (src.ram_install_skips() + src.ram_install_gpu()), (long long) src.ram_install_gpu());
if (src.ram_adapts() > 0) // STRATA_RAM_ADAPT, cumulative
- std::fprintf(stderr, "strata serve: resident RAM: %lld experts moved in from the file in place of "
- "unused ones (STRATA_RAM_ADAPT)\n", (long long) src.ram_adapts());
+ std::fprintf(stderr, "strata serve: resident RAM: %lld experts moved in in place of unused ones, %lld of "
+ "them out of a stage buffer (STRATA_RAM_ADAPT)\n", (long long) src.ram_adapts(),
+ (long long) src.ram_staged());
if (ajob) // --adapt-async, cumulative (a round may still be in flight: it lands before the next request)
std::fprintf(stderr, "strata serve: asynchronous adaptive tier: %lld rounds, %lld experts swapped in, "
"%.1f ms per round (start to flip, between windows)",
diff --git a/tests/core/file_expert_source_test.cpp b/tests/core/file_expert_source_test.cpp
index e162664d1..d6a274f1a 100644
--- a/tests/core/file_expert_source_test.cpp
+++ b/tests/core/file_expert_source_test.cpp
@@ -979,6 +979,28 @@ void test_replace_resident(bool pin) {
require(src.commit_exchanges() == 1 && src.has_resident(0, 3) && holds(3) && !src.has_resident(0, 1),
"exchange after replace wrong");
require(src.ram_adapts() == 2, "accounting wrong");
+ // a staged incoming blob: RAM {2,3}, VRAM {0,1}, 4 on the file tier
+ require(!src.replace_resident_staged(0, 4, 3) && src.has_resident(0, 3) && src.ram_adapts() == 2,
+ "an unstaged expert moved");
+ std::string why;
+ set_env("STRATA_UNBUFFERED_LOAD", "1");
+ const bool unbuffered = src.set_unbuffered(0, why);
+ set_env("STRATA_UNBUFFERED_LOAD", nullptr);
+ if (unbuffered && src.stages_blobs()) {
+ require(!src.replace_resident_staged(0, 4, 3) && holds(3), "moved before any demand read");
+ require(!src.has_resident(0, 4) && holds(4), "demand read wrong"); // leaves 4 in a stage buffer
+ require(!src.replace_resident_staged(0, 4, 0) && !src.replace_resident_staged(0, 2, 3),
+ "invalid staged pair accepted");
+ require(src.replace_resident_staged(0, 4, 3) && src.has_resident(0, 4) && !src.has_resident(0, 3),
+ "staged replacement residency wrong");
+ require(src.resident_blob(0, 4) != nullptr && !std::memcmp(src.resident_blob(0, 4), truth[4].data(), bytes),
+ "staged replacement bytes wrong");
+ require(holds(2) && holds(3) && src.ram_adapts() == 3 && src.ram_staged() == 1,
+ "staged replacement changed another expert");
+ std::cout << "replace_resident_staged: pinned=" << pin << " exact_bytes=PASS\n";
+ } else {
+ std::cout << "replace_resident_staged: pinned=" << pin << " unbuffered unavailable here (" << why << "), positive case SKIPPED\n";
+ }
std::cout << "replace_resident: pinned=" << pin << " exact_bytes=PASS\n";
}