Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

149: Linux defaults - zero-copy page-cached experts, io prefetch and its decode gate; cgroup RAM check counts the clean page cache

Niko122182c80ac4d76a
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 9951af43c..e0861c7cf 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -261,7 +261,28 @@ from the GGUF gave identical tokens and logits. An expert read from the GGUF is
engine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) with one batched page request
(Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet.
-**Linux file-tier I/O path (opt in, `STRATA_IO_PREFETCH=1`; 0.1.40.2):** the mapped file tier reads experts through
+**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read
+from the files (a GGUF read in place, `--mmap-experts`, or an `experts.bin` file tier), Linux now runs three things by default.
+(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the page
+cache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer
+(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates
+IDENTICAL). (2) **io prefetch** (`STRATA_IO_PREFETCH=1`, the path described below): whole-blob `pread`s and the router-
+predicted experts of the next layers read ahead. (3) **the decode gate** (`STRATA_IO_PF_GATE=1`): every 16 decode windows the
+decode's read-ahead is judged - it runs only while the page cache has room for the experts (the memory available, the
+tightest cgroup limit included, over the experts' bytes: off below 0.55, on above 0.65, `STRATA_IO_PF_GATE_LO` / `_HI`) and
+while it pays (at least 128 MB read ahead in 16 windows and 10% of it used, `STRATA_IO_PF_GATE_USEFUL`; a probe every 256
+windows when it closed for that). Closed, the decode reads as it did without io prefetch (the look-ahead predicts one layer
+and warms with an madvise hint); the prompt path keeps its preads either way. Measured on an RTX PRO 5000 emulating a 12 GB
+card (`--expert-cache 4448`, Q2_0 GGUF in place, page cache dropped before each engine, RAM limited by a cgroup), 5 pairs,
+story / code / doc8k decode, 4K / 20K prompts, against the 0.1.4x defaults: 32 GB of RAM 1.8 / 2.8 / 3.1x decode, 1.5 / 2.0x
+prompts; 16 GB 1.2 / 1.4 / 1.4x decode, 1.6 / 1.8x prompts (the gate stays closed there); an 18 GiB RAM tier B32_RESULT. On a real RTX 3060 12 GB (32 GB cgroup, 3 pairs) zero-copy alone gave 1.14 / 1.44 / 1.47x decode, 1.06 / 1.02x
+prompts. With it the container RAM check counts a cgroup's clean page cache (active and inactive) as available, as
+MemAvailable does on a bare host (before, a 32 GiB container whose own reads left 26 GiB of cache was judged short of RAM
+and its file tier switched to unbuffered reads). Off switches: `STRATA_COLD_OVERLAP=0`, `STRATA_IO_PREFETCH=0`,
+`STRATA_IO_PF_GATE=0`; the startup log names what is on. Windows and macOS are unchanged. Measured detail:
+phaseA-tests/notes-149-l620.md.
+
+**Linux file-tier I/O path (`STRATA_IO_PREFETCH=1`; 0.1.40.2, on by default on Linux since 0.2.0 - see above):** the mapped file tier reads experts through
page faults. With this on, the layer's uncached experts and the router-predicted ones of the next layer (`STRATA_LOOKAHEAD_K`,
`STRATA_IO_PREFETCH_DEPTH`, default 2 when on) are read by I/O threads (`STRATA_IO_PF_THREADS`, default 8) with whole-blob
`pread`s into the page cache, and the CPU kernels read the mapping as before. The bytes are the same, so the output is the
diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp
index 74cfdd955..ab7ae4a4a 100644
--- a/include/strata/core/expert_source.hpp
+++ b/include/strata/core/expert_source.hpp
@@ -65,9 +65,14 @@ struct CgroupMemoryStat {
uint64_t file_dirty = 0;
uint64_t file_writeback = 0;
bool valid = false;
+ uint64_t file = 0; ///< 149: memory.stat "file" (active + inactive page cache); 0 when not reported
};
-/// Calculate additional bytes under a finite cgroup limit after reclaiming only clean inactive file cache.
+/// Calculate additional bytes under a finite cgroup limit after reclaiming the group's clean page cache, as
+/// MemAvailable counts it on a bare host: every clean file page (active and inactive: memory.stat "file", else
+/// inactive_file), less a low-watermark share the kernel keeps (min(half the clean cache, limit / 64)). A process
+/// whose own reads (a GPU cache fill, a read-ahead) left gigabytes of active cache in its group was judged short of
+/// RAM before (149: 1.2 GiB "available" of a 32 GiB group, the file tier then switched to unbuffered reads).
/// Returns false when the required memory.stat counters were unavailable.
bool cgroup_available_bytes(uint64_t limit, const CgroupMemoryStat& stat, uint64_t& bytes);
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
index 15bc367ad..38b8be0be 100644
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -75,10 +75,12 @@ bool cgroup_available_bytes(uint64_t limit, const CgroupMemoryStat& stat, uint64
bytes = 0;
if (!stat.valid) return false;
- // memory.stat's inactive_file can race memory.current, so bound it to charged usage first.
- uint64_t reclaimable = std::min(stat.inactive_file, stat.current);
+ // memory.stat's counters can race memory.current, so bound them to charged usage first. The whole clean page
+ // cache is reclaimable (149: MemAvailable counts active file pages too); the kernel keeps a low-watermark share.
+ uint64_t reclaimable = std::min(std::max(stat.file, stat.inactive_file), stat.current);
reclaimable = stat.file_dirty >= reclaimable ? 0 : reclaimable - stat.file_dirty;
reclaimable = stat.file_writeback >= reclaimable ? 0 : reclaimable - stat.file_writeback;
+ reclaimable -= std::min(reclaimable / 2, limit / 64);
// Reclaiming clean file pages reduces usage; saturating subtraction also handles a transient over-limit read.
const uint64_t usage_after_reclaim = stat.current - reclaimable;
@@ -291,7 +293,7 @@ bool read_cgroup_memory_stat(const std::filesystem::path& path, uint64_t current
std::ifstream input(path / "memory.stat");
if (!input) return false;
- bool inactive_file = false, file_dirty = false, file_writeback = false;
+ bool inactive_file = false, file_dirty = false, file_writeback = false, file = false;
std::string line;
while (std::getline(input, line)) {
std::istringstream fields(line);
@@ -313,6 +315,10 @@ bool read_cgroup_memory_stat(const std::filesystem::path& path, uint64_t current
if (file_writeback) return false;
file_writeback = true;
stat.file_writeback = value;
+ } else if (key == "file") { // 149: optional (every v2 kernel has it); 0 = inactive_file alone
+ if (file) return false;
+ file = true;
+ stat.file = value;
}
}
if (!input.eof() || !inactive_file || !file_dirty || !file_writeback) return false;
@@ -771,7 +777,8 @@ void FileExpertSource::set_io_prefetch(bool on, int threads) {
if (const char* m = std::getenv("STRATA_IO_PF_BUDGET_MB")) io_budget_fixed_ = (uint64_t) std::max(0.0, std::atof(m) * 1048576.0);
if (const char* m = std::getenv("STRATA_IO_PF_BUDGET_FRAC")) io_budget_frac_ = std::clamp(std::atof(m), 0.01, 4.0);
if (const char* m = std::getenv("STRATA_IO_PF_TARGET")) io_target_ = std::clamp(std::atof(m), 0.0, 1.0);
- if (const char* m = std::getenv("STRATA_IO_PF_GATE")) gate_on_ = std::atoi(m) != 0;
+ gate_on_ = true; // 0.2.0: the decode gate comes with io prefetch; STRATA_IO_PF_GATE=0 turns it off
+ if (const char* m = std::getenv("STRATA_IO_PF_GATE"); m != nullptr && m[0] != '\0') gate_on_ = std::atoi(m) != 0;
if (const char* m = std::getenv("STRATA_IO_PF_GATE_LO")) gate_lo_ = std::clamp(std::atof(m), 0.0, 1.0);
if (const char* m = std::getenv("STRATA_IO_PF_GATE_HI")) gate_hi_ = std::clamp(std::atof(m), gate_lo_, 1.0);
if (const char* m = std::getenv("STRATA_IO_PF_GATE_USEFUL")) gate_min_useful_ = std::clamp(std::atof(m), 0.0, 1.0);
@@ -1815,7 +1822,14 @@ int64_t FileExpertSource::prefetch_begin(int64_t layer, const int64_t* experts,
prefetch_end();
async_todo_.clear();
async_fill_us_ = 0;
- static const int mode = [] { const char* v = std::getenv("STRATA_COLD_OVERLAP"); return v != nullptr ? std::atoi(v) : 0; }();
+ static const int mode = [] { // as the dispatch reads it (0.2.0: 4 by default on Linux)
+ const char* v = std::getenv("STRATA_COLD_OVERLAP");
+#if defined(__linux__)
+ return v == nullptr || v[0] == '\0' ? 4 : std::atoi(v);
+#else
+ return v == nullptr || v[0] == '\0' ? 0 : std::atoi(v);
+#endif
+ }();
claim_todo(layer, experts, n, async_todo_, pending, mode >= 3);
if (mode == 4) { // zero-copy only: the blobs that must be read are read here, as `prefetch` reads them
const auto t0 = std::chrono::steady_clock::now();
@@ -4370,9 +4384,14 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
const auto c0 = std::chrono::steady_clock::now();
const FileExpertSource::DecodeScope decode_scope; // 149 STRATA_IO_PF_GATE: this thread reads for the decode
// 149-cold STRATA_COLD_OVERLAP: 0 off (default), 1 overlap, 2 measure only, 3 overlap + zero-copy, 4 zero-copy only
+ // 0.2.0: 4 by default on Linux (zero-copy needs the page cache's residency query); elsewhere 0
static const int cold_env = [] {
const char* v = std::getenv("STRATA_COLD_OVERLAP");
- return v == nullptr ? 0 : std::clamp(std::atoi(v), 0, 4);
+#if defined(__linux__)
+ return v == nullptr || v[0] == '\0' ? 4 : std::clamp(std::atoi(v), 0, 4);
+#else
+ return v == nullptr || v[0] == '\0' ? 0 : std::clamp(std::atoi(v), 0, 4);
+#endif
}();
d.cold_mode = cold_env;
pt("begin");
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 47235e070..95975f1ad 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -5445,19 +5445,35 @@ int main(int argc, char** argv) {
"(%s)\n", why.c_str());
}
#endif
- // Linux I/O path (opt in): whole-blob preads instead of page faults, and the predicted experts of the next
- // layers read ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch)
- if (const char* v = std::getenv("STRATA_IO_PREFETCH"); v != nullptr && std::atoi(v) != 0) {
+ // Linux I/O path: whole-blob preads instead of page faults, and the predicted experts of the next layers read
+ // ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch). 0.2.0: on by
+ // default on Linux, with the decode gate (STRATA_IO_PF_GATE) and zero-copy page-cached experts
+ // (STRATA_COLD_OVERLAP=4); STRATA_IO_PREFETCH=0 / STRATA_IO_PF_GATE=0 / STRATA_COLD_OVERLAP=0 turn them off.
+ const char* io_pf_env = std::getenv("STRATA_IO_PREFETCH");
+#if defined(__linux__)
+ const bool io_pf_default = io_pf_env == nullptr || io_pf_env[0] == '\0';
+#else
+ const bool io_pf_default = false;
+#endif
+ if (io_pf_default || (io_pf_env != nullptr && std::atoi(io_pf_env) != 0)) {
const char* th = std::getenv("STRATA_IO_PF_THREADS");
src.set_io_prefetch(true, th != nullptr ? std::atoi(th) : 0);
- if (src.io_prefetch())
+ if (src.io_prefetch() && io_pf_default) {
+ const char* cv = std::getenv("STRATA_COLD_OVERLAP");
+ std::fprintf(stderr, "strata generate: file tier (Linux defaults): page-cached experts computed from the "
+ "mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, "
+ "the decode's read-ahead %s; off: STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, "
+ "STRATA_IO_PF_GATE=0\n", cv != nullptr && cv[0] != '\0' ? cv : "4", th != nullptr ? th : "8",
+ src.gate_on() ? "only while the page cache has room for it and it pays (STRATA_IO_PF_GATE=1)"
+ : "always (STRATA_IO_PF_GATE=0)");
+ } else if (src.io_prefetch())
std::fprintf(stderr, "strata generate: file tier io prefetch ON (STRATA_IO_PREFETCH=1): whole-blob preads, "
- "the layer's uncached experts and the predicted ones read by %s I/O threads (STRATA_IO_PF_THREADS, default 8)\n",
- th != nullptr ? th : "8");
+ "the layer's uncached experts and the predicted ones read by %s I/O threads (STRATA_IO_PF_THREADS, default 8)%s\n",
+ th != nullptr ? th : "8", src.gate_on() ? "; the decode's read-ahead gated (STRATA_IO_PF_GATE=0: always)" : "");
if (src.io_prefetch() && src.io_sched())
std::fprintf(stderr, "strata generate: io prefetch scheduling ON (STRATA_IO_PF_SCHED=1): a deadline per read, "
"confidence admission (STRATA_IO_PF_MINCONF), a per-window byte budget (STRATA_IO_PF_BUDGET_MB / _FRAC)\n");
- if (!src.io_prefetch())
+ if (!src.io_prefetch() && !io_pf_default)
std::fprintf(stderr, "strata generate: STRATA_IO_PREFETCH=1 is not available here (Linux mapped file tier "
"only, not with unbuffered reads); the file tier reads as before\n");
if (src.io_prefetch()) { // recommend, never force: the engine runs as asked
@@ -14954,7 +14970,8 @@ int main(int argc, char** argv) {
fb_prev = fb;
}
}
- if (drive.d.cold_mode != 0) { // 149-cold STRATA_COLD_OVERLAP: this request's layers that had a blob to read
+ static const bool cold_log = std::getenv("STRATA_COLD_OVERLAP") != nullptr || std::getenv("STRATA_IO_STATS") != nullptr;
+ if (drive.d.cold_mode != 0 && cold_log) { // 149-cold STRATA_COLD_OVERLAP: this request's layers that had a blob to read
static strata::core::ExpertDispatch cp;
const strata::core::ExpertDispatch& c = drive.d;
const double nl = (double) std::max<int64_t>(1, c.cold_layers - cp.cold_layers);
diff --git a/tests/core/file_expert_source_test.cpp b/tests/core/file_expert_source_test.cpp
index c4586c871..83ced92e3 100644
--- a/tests/core/file_expert_source_test.cpp
+++ b/tests/core/file_expert_source_test.cpp
@@ -433,18 +433,41 @@ void test_cgroup_memory_budget() {
constexpr uint64_t GiB = 1ull << 30;
uint64_t bytes = 0;
+ // the kernel keeps min(half the clean cache, limit / 64) of it: 56 GiB / 64 = 0.875 GiB here
+ const uint64_t keep56 = 56 * GiB / 64;
CgroupMemoryStat clean_cache{40 * GiB, 32 * GiB, 0, 0, true};
require(cgroup_available_bytes(56 * GiB, clean_cache, bytes), "valid cgroup memory.stat was rejected");
- require(bytes == 48 * GiB, "clean inactive file cache was not credited against the cgroup cap");
+ require(bytes == 48 * GiB - keep56, "clean inactive file cache was not credited against the cgroup cap");
CgroupMemoryStat dirty_cache{40 * GiB, 32 * GiB, 4 * GiB, 2 * GiB, true};
require(cgroup_available_bytes(56 * GiB, dirty_cache, bytes), "valid dirty-cache counters were rejected");
- require(bytes == 42 * GiB, "dirty/writeback pages were incorrectly counted as reclaimable");
+ require(bytes == 42 * GiB - keep56, "dirty/writeback pages were incorrectly counted as reclaimable");
CgroupMemoryStat oversized_inactive{10 * GiB, 20 * GiB, 0, 0, true};
require(cgroup_available_bytes(12 * GiB, oversized_inactive, bytes),
"valid oversized inactive-file counter was rejected");
- require(bytes == 12 * GiB, "inactive-file accounting exceeded charged current usage");
+ require(bytes == 12 * GiB - 12 * GiB / 64, "inactive-file accounting exceeded charged current usage");
+
+ // 149: the ACTIVE page cache is reclaimable too (memory.stat "file" = active + inactive), as MemAvailable counts
+ // it: a 32 GiB group whose own reads left 26 GiB of cache, 25 GiB of it active, and 4.5 GiB of anonymous memory
+ CgroupMemoryStat active_cache{30 * GiB + GiB / 2, 1 * GiB, 0, 0, true, 26 * GiB};
+ require(cgroup_available_bytes(32 * GiB, active_cache, bytes), "a memory.stat with file was rejected");
+ require(bytes == 32 * GiB - (4 * GiB + GiB / 2) - 32 * GiB / 64, "the active page cache was not credited");
+ // without "file" (an old kernel's stat): inactive_file alone, as before
+ CgroupMemoryStat no_file{30 * GiB + GiB / 2, 1 * GiB, 0, 0, true};
+ require(cgroup_available_bytes(32 * GiB, no_file, bytes) && bytes == 32 * GiB - (29 * GiB + GiB / 2) - GiB / 2,
+ "a stat without file did not fall back to inactive_file");
+ // dirty and writeback pages of the active cache stay charged; a small cache keeps half of itself
+ CgroupMemoryStat dirty_active{20 * GiB, 1 * GiB, 2 * GiB, 1 * GiB, true, 10 * GiB};
+ require(cgroup_available_bytes(32 * GiB, dirty_active, bytes) && bytes == 32 * GiB - (20 * GiB - 7 * GiB) - GiB / 2,
+ "dirty / writeback pages of the active cache were counted as reclaimable");
+ CgroupMemoryStat tiny_cache{8 * GiB, 0, 0, 0, true, 256ull << 20};
+ require(cgroup_available_bytes(32 * GiB, tiny_cache, bytes) && bytes == 32 * GiB - 8 * GiB + (128ull << 20),
+ "a small clean cache was not kept at half");
+ // "file" larger than what is charged (a racy read) is bounded by memory.current
+ CgroupMemoryStat racy{4 * GiB, 1 * GiB, 0, 0, true, 9 * GiB};
+ require(cgroup_available_bytes(32 * GiB, racy, bytes) && bytes == 32 * GiB - GiB / 2,
+ "a file counter above memory.current escaped the bound");
const uint64_t max = std::numeric_limits<uint64_t>::max();
CgroupMemoryStat large_counters{max, max, max - 1, max, true};
@@ -562,7 +585,13 @@ void test_host_memory() {
put("fs/box/memory.max", std::to_string(48 * GiB) + "\n");
put("fs/box/memory.current", std::to_string(16 * GiB) + "\n");
put("fs/box/memory.stat", "inactive_file " + std::to_string(4 * GiB) + "\nfile_dirty 0\nfile_writeback 0\n");
- require(probe("0::/box\n") && m.available == 36 * GiB && m.cgroup_limit == 48 * GiB, "v2 limit");
+ require(probe("0::/box\n") && m.available == 36 * GiB - 48 * GiB / 64 && m.cgroup_limit == 48 * GiB, "v2 limit");
+ // 149: with "file" (active + inactive cache) the active part counts too: 12 GiB of cache, 4 GiB anonymous
+ put("fs/box/memory.stat", "file " + std::to_string(12 * GiB) + "\ninactive_file " + std::to_string(4 * GiB) +
+ "\nfile_dirty 0\nfile_writeback 0\n");
+ require(probe("0::/box\n") && m.available == 44 * GiB - 48 * GiB / 64 && m.cgroup_limit == 48 * GiB,
+ "v2 limit, the active page cache credited");
+ put("fs/box/memory.stat", "inactive_file " + std::to_string(4 * GiB) + "\nfile_dirty 0\nfile_writeback 0\n");
put("fs/box/memory.max", "max\n");
require(probe("0::/box\n") && m.available == 100 * GiB && m.cgroup_limit == ~0ull, "v2 max");
put("fs/box/memory.max", "48G\n");