Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
149: Linux defaults - zero-copy page-cached experts, io prefetch and its decode gate; cgroup RAM check counts the clean page cache
Niko122182c80ac4d76a
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 9951af43c..e0861c7cf 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -261,7 +261,28 @@ from the GGUF gave identical tokens and logits. An expert read from the GGUF isengine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) with one batched page request(Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet.-**Linux file-tier I/O path (opt in, `STRATA_IO_PREFETCH=1`; 0.1.40.2):** the mapped file tier reads experts through+**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read+from the files (a GGUF read in place, `--mmap-experts`, or an `experts.bin` file tier), Linux now runs three things by default.+(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the page+cache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer+(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates+IDENTICAL). (2) **io prefetch** (`STRATA_IO_PREFETCH=1`, the path described below): whole-blob `pread`s and the router-+predicted experts of the next layers read ahead. (3) **the decode gate** (`STRATA_IO_PF_GATE=1`): every 16 decode windows the+decode's read-ahead is judged - it runs only while the page cache has room for the experts (the memory available, the+tightest cgroup limit included, over the experts' bytes: off below 0.55, on above 0.65, `STRATA_IO_PF_GATE_LO` / `_HI`) and+while it pays (at least 128 MB read ahead in 16 windows and 10% of it used, `STRATA_IO_PF_GATE_USEFUL`; a probe every 256+windows when it closed for that). Closed, the decode reads as it did without io prefetch (the look-ahead predicts one layer+and warms with an madvise hint); the prompt path keeps its preads either way. Measured on an RTX PRO 5000 emulating a 12 GB+card (`--expert-cache 4448`, Q2_0 GGUF in place, page cache dropped before each engine, RAM limited by a cgroup), 5 pairs,+story / code / doc8k decode, 4K / 20K prompts, against the 0.1.4x defaults: 32 GB of RAM 1.8 / 2.8 / 3.1x decode, 1.5 / 2.0x+prompts; 16 GB 1.2 / 1.4 / 1.4x decode, 1.6 / 1.8x prompts (the gate stays closed there); an 18 GiB RAM tier B32_RESULT. On a real RTX 3060 12 GB (32 GB cgroup, 3 pairs) zero-copy alone gave 1.14 / 1.44 / 1.47x decode, 1.06 / 1.02x+prompts. With it the container RAM check counts a cgroup's clean page cache (active and inactive) as available, as+MemAvailable does on a bare host (before, a 32 GiB container whose own reads left 26 GiB of cache was judged short of RAM+and its file tier switched to unbuffered reads). Off switches: `STRATA_COLD_OVERLAP=0`, `STRATA_IO_PREFETCH=0`,+`STRATA_IO_PF_GATE=0`; the startup log names what is on. Windows and macOS are unchanged. Measured detail:+phaseA-tests/notes-149-l620.md.++**Linux file-tier I/O path (`STRATA_IO_PREFETCH=1`; 0.1.40.2, on by default on Linux since 0.2.0 - see above):** the mapped file tier reads experts throughpage faults. With this on, the layer's uncached experts and the router-predicted ones of the next layer (`STRATA_LOOKAHEAD_K`,`STRATA_IO_PREFETCH_DEPTH`, default 2 when on) are read by I/O threads (`STRATA_IO_PF_THREADS`, default 8) with whole-blob`pread`s into the page cache, and the CPU kernels read the mapping as before. The bytes are the same, so the output is thediff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hppindex 74cfdd955..ab7ae4a4a 100644--- a/include/strata/core/expert_source.hpp+++ b/include/strata/core/expert_source.hpp@@ -65,9 +65,14 @@ struct CgroupMemoryStat {uint64_t file_dirty = 0;uint64_t file_writeback = 0;bool valid = false;+ uint64_t file = 0; ///< 149: memory.stat "file" (active + inactive page cache); 0 when not reported};-/// Calculate additional bytes under a finite cgroup limit after reclaiming only clean inactive file cache.+/// Calculate additional bytes under a finite cgroup limit after reclaiming the group's clean page cache, as+/// MemAvailable counts it on a bare host: every clean file page (active and inactive: memory.stat "file", else+/// inactive_file), less a low-watermark share the kernel keeps (min(half the clean cache, limit / 64)). A process+/// whose own reads (a GPU cache fill, a read-ahead) left gigabytes of active cache in its group was judged short of+/// RAM before (149: 1.2 GiB "available" of a 32 GiB group, the file tier then switched to unbuffered reads)./// Returns false when the required memory.stat counters were unavailable.bool cgroup_available_bytes(uint64_t limit, const CgroupMemoryStat& stat, uint64_t& bytes);diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cppindex 15bc367ad..38b8be0be 100644--- a/src/core/expert_source.cpp+++ b/src/core/expert_source.cpp@@ -75,10 +75,12 @@ bool cgroup_available_bytes(uint64_t limit, const CgroupMemoryStat& stat, uint64bytes = 0;if (!stat.valid) return false;- // memory.stat's inactive_file can race memory.current, so bound it to charged usage first.- uint64_t reclaimable = std::min(stat.inactive_file, stat.current);+ // memory.stat's counters can race memory.current, so bound them to charged usage first. The whole clean page+ // cache is reclaimable (149: MemAvailable counts active file pages too); the kernel keeps a low-watermark share.+ uint64_t reclaimable = std::min(std::max(stat.file, stat.inactive_file), stat.current);reclaimable = stat.file_dirty >= reclaimable ? 0 : reclaimable - stat.file_dirty;reclaimable = stat.file_writeback >= reclaimable ? 0 : reclaimable - stat.file_writeback;+ reclaimable -= std::min(reclaimable / 2, limit / 64);// Reclaiming clean file pages reduces usage; saturating subtraction also handles a transient over-limit read.const uint64_t usage_after_reclaim = stat.current - reclaimable;@@ -291,7 +293,7 @@ bool read_cgroup_memory_stat(const std::filesystem::path& path, uint64_t currentstd::ifstream input(path / "memory.stat");if (!input) return false;- bool inactive_file = false, file_dirty = false, file_writeback = false;+ bool inactive_file = false, file_dirty = false, file_writeback = false, file = false;std::string line;while (std::getline(input, line)) {std::istringstream fields(line);@@ -313,6 +315,10 @@ bool read_cgroup_memory_stat(const std::filesystem::path& path, uint64_t currentif (file_writeback) return false;file_writeback = true;stat.file_writeback = value;+ } else if (key == "file") { // 149: optional (every v2 kernel has it); 0 = inactive_file alone+ if (file) return false;+ file = true;+ stat.file = value;}}if (!input.eof() || !inactive_file || !file_dirty || !file_writeback) return false;@@ -771,7 +777,8 @@ void FileExpertSource::set_io_prefetch(bool on, int threads) {if (const char* m = std::getenv("STRATA_IO_PF_BUDGET_MB")) io_budget_fixed_ = (uint64_t) std::max(0.0, std::atof(m) * 1048576.0);if (const char* m = std::getenv("STRATA_IO_PF_BUDGET_FRAC")) io_budget_frac_ = std::clamp(std::atof(m), 0.01, 4.0);if (const char* m = std::getenv("STRATA_IO_PF_TARGET")) io_target_ = std::clamp(std::atof(m), 0.0, 1.0);- if (const char* m = std::getenv("STRATA_IO_PF_GATE")) gate_on_ = std::atoi(m) != 0;+ gate_on_ = true; // 0.2.0: the decode gate comes with io prefetch; STRATA_IO_PF_GATE=0 turns it off+ if (const char* m = std::getenv("STRATA_IO_PF_GATE"); m != nullptr && m[0] != '\0') gate_on_ = std::atoi(m) != 0;if (const char* m = std::getenv("STRATA_IO_PF_GATE_LO")) gate_lo_ = std::clamp(std::atof(m), 0.0, 1.0);if (const char* m = std::getenv("STRATA_IO_PF_GATE_HI")) gate_hi_ = std::clamp(std::atof(m), gate_lo_, 1.0);if (const char* m = std::getenv("STRATA_IO_PF_GATE_USEFUL")) gate_min_useful_ = std::clamp(std::atof(m), 0.0, 1.0);@@ -1815,7 +1822,14 @@ int64_t FileExpertSource::prefetch_begin(int64_t layer, const int64_t* experts,prefetch_end();async_todo_.clear();async_fill_us_ = 0;- static const int mode = [] { const char* v = std::getenv("STRATA_COLD_OVERLAP"); return v != nullptr ? std::atoi(v) : 0; }();+ static const int mode = [] { // as the dispatch reads it (0.2.0: 4 by default on Linux)+ const char* v = std::getenv("STRATA_COLD_OVERLAP");+#if defined(__linux__)+ return v == nullptr || v[0] == '\0' ? 4 : std::atoi(v);+#else+ return v == nullptr || v[0] == '\0' ? 0 : std::atoi(v);+#endif+ }();claim_todo(layer, experts, n, async_todo_, pending, mode >= 3);if (mode == 4) { // zero-copy only: the blobs that must be read are read here, as `prefetch` reads themconst auto t0 = std::chrono::steady_clock::now();@@ -4370,9 +4384,14 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32const auto c0 = std::chrono::steady_clock::now();const FileExpertSource::DecodeScope decode_scope; // 149 STRATA_IO_PF_GATE: this thread reads for the decode// 149-cold STRATA_COLD_OVERLAP: 0 off (default), 1 overlap, 2 measure only, 3 overlap + zero-copy, 4 zero-copy only+ // 0.2.0: 4 by default on Linux (zero-copy needs the page cache's residency query); elsewhere 0static const int cold_env = [] {const char* v = std::getenv("STRATA_COLD_OVERLAP");- return v == nullptr ? 0 : std::clamp(std::atoi(v), 0, 4);+#if defined(__linux__)+ return v == nullptr || v[0] == '\0' ? 4 : std::clamp(std::atoi(v), 0, 4);+#else+ return v == nullptr || v[0] == '\0' ? 0 : std::clamp(std::atoi(v), 0, 4);+#endif}();d.cold_mode = cold_env;pt("begin");diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 47235e070..95975f1ad 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -5445,19 +5445,35 @@ int main(int argc, char** argv) {"(%s)\n", why.c_str());}#endif- // Linux I/O path (opt in): whole-blob preads instead of page faults, and the predicted experts of the next- // layers read ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch)- if (const char* v = std::getenv("STRATA_IO_PREFETCH"); v != nullptr && std::atoi(v) != 0) {+ // Linux I/O path: whole-blob preads instead of page faults, and the predicted experts of the next layers read+ // ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch). 0.2.0: on by+ // default on Linux, with the decode gate (STRATA_IO_PF_GATE) and zero-copy page-cached experts+ // (STRATA_COLD_OVERLAP=4); STRATA_IO_PREFETCH=0 / STRATA_IO_PF_GATE=0 / STRATA_COLD_OVERLAP=0 turn them off.+ const char* io_pf_env = std::getenv("STRATA_IO_PREFETCH");+#if defined(__linux__)+ const bool io_pf_default = io_pf_env == nullptr || io_pf_env[0] == '\0';+#else+ const bool io_pf_default = false;+#endif+ if (io_pf_default || (io_pf_env != nullptr && std::atoi(io_pf_env) != 0)) {const char* th = std::getenv("STRATA_IO_PF_THREADS");src.set_io_prefetch(true, th != nullptr ? std::atoi(th) : 0);- if (src.io_prefetch())+ if (src.io_prefetch() && io_pf_default) {+ const char* cv = std::getenv("STRATA_COLD_OVERLAP");+ std::fprintf(stderr, "strata generate: file tier (Linux defaults): page-cached experts computed from the "+ "mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, "+ "the decode's read-ahead %s; off: STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, "+ "STRATA_IO_PF_GATE=0\n", cv != nullptr && cv[0] != '\0' ? cv : "4", th != nullptr ? th : "8",+ src.gate_on() ? "only while the page cache has room for it and it pays (STRATA_IO_PF_GATE=1)"+ : "always (STRATA_IO_PF_GATE=0)");+ } else if (src.io_prefetch())std::fprintf(stderr, "strata generate: file tier io prefetch ON (STRATA_IO_PREFETCH=1): whole-blob preads, "- "the layer's uncached experts and the predicted ones read by %s I/O threads (STRATA_IO_PF_THREADS, default 8)\n",- th != nullptr ? th : "8");+ "the layer's uncached experts and the predicted ones read by %s I/O threads (STRATA_IO_PF_THREADS, default 8)%s\n",+ th != nullptr ? th : "8", src.gate_on() ? "; the decode's read-ahead gated (STRATA_IO_PF_GATE=0: always)" : "");if (src.io_prefetch() && src.io_sched())std::fprintf(stderr, "strata generate: io prefetch scheduling ON (STRATA_IO_PF_SCHED=1): a deadline per read, ""confidence admission (STRATA_IO_PF_MINCONF), a per-window byte budget (STRATA_IO_PF_BUDGET_MB / _FRAC)\n");- if (!src.io_prefetch())+ if (!src.io_prefetch() && !io_pf_default)std::fprintf(stderr, "strata generate: STRATA_IO_PREFETCH=1 is not available here (Linux mapped file tier ""only, not with unbuffered reads); the file tier reads as before\n");if (src.io_prefetch()) { // recommend, never force: the engine runs as asked@@ -14954,7 +14970,8 @@ int main(int argc, char** argv) {fb_prev = fb;}}- if (drive.d.cold_mode != 0) { // 149-cold STRATA_COLD_OVERLAP: this request's layers that had a blob to read+ static const bool cold_log = std::getenv("STRATA_COLD_OVERLAP") != nullptr || std::getenv("STRATA_IO_STATS") != nullptr;+ if (drive.d.cold_mode != 0 && cold_log) { // 149-cold STRATA_COLD_OVERLAP: this request's layers that had a blob to readstatic strata::core::ExpertDispatch cp;const strata::core::ExpertDispatch& c = drive.d;const double nl = (double) std::max<int64_t>(1, c.cold_layers - cp.cold_layers);diff --git a/tests/core/file_expert_source_test.cpp b/tests/core/file_expert_source_test.cppindex c4586c871..83ced92e3 100644--- a/tests/core/file_expert_source_test.cpp+++ b/tests/core/file_expert_source_test.cpp@@ -433,18 +433,41 @@ void test_cgroup_memory_budget() {constexpr uint64_t GiB = 1ull << 30;uint64_t bytes = 0;+ // the kernel keeps min(half the clean cache, limit / 64) of it: 56 GiB / 64 = 0.875 GiB here+ const uint64_t keep56 = 56 * GiB / 64;CgroupMemoryStat clean_cache{40 * GiB, 32 * GiB, 0, 0, true};require(cgroup_available_bytes(56 * GiB, clean_cache, bytes), "valid cgroup memory.stat was rejected");- require(bytes == 48 * GiB, "clean inactive file cache was not credited against the cgroup cap");+ require(bytes == 48 * GiB - keep56, "clean inactive file cache was not credited against the cgroup cap");CgroupMemoryStat dirty_cache{40 * GiB, 32 * GiB, 4 * GiB, 2 * GiB, true};require(cgroup_available_bytes(56 * GiB, dirty_cache, bytes), "valid dirty-cache counters were rejected");- require(bytes == 42 * GiB, "dirty/writeback pages were incorrectly counted as reclaimable");+ require(bytes == 42 * GiB - keep56, "dirty/writeback pages were incorrectly counted as reclaimable");CgroupMemoryStat oversized_inactive{10 * GiB, 20 * GiB, 0, 0, true};require(cgroup_available_bytes(12 * GiB, oversized_inactive, bytes),"valid oversized inactive-file counter was rejected");- require(bytes == 12 * GiB, "inactive-file accounting exceeded charged current usage");+ require(bytes == 12 * GiB - 12 * GiB / 64, "inactive-file accounting exceeded charged current usage");++ // 149: the ACTIVE page cache is reclaimable too (memory.stat "file" = active + inactive), as MemAvailable counts+ // it: a 32 GiB group whose own reads left 26 GiB of cache, 25 GiB of it active, and 4.5 GiB of anonymous memory+ CgroupMemoryStat active_cache{30 * GiB + GiB / 2, 1 * GiB, 0, 0, true, 26 * GiB};+ require(cgroup_available_bytes(32 * GiB, active_cache, bytes), "a memory.stat with file was rejected");+ require(bytes == 32 * GiB - (4 * GiB + GiB / 2) - 32 * GiB / 64, "the active page cache was not credited");+ // without "file" (an old kernel's stat): inactive_file alone, as before+ CgroupMemoryStat no_file{30 * GiB + GiB / 2, 1 * GiB, 0, 0, true};+ require(cgroup_available_bytes(32 * GiB, no_file, bytes) && bytes == 32 * GiB - (29 * GiB + GiB / 2) - GiB / 2,+ "a stat without file did not fall back to inactive_file");+ // dirty and writeback pages of the active cache stay charged; a small cache keeps half of itself+ CgroupMemoryStat dirty_active{20 * GiB, 1 * GiB, 2 * GiB, 1 * GiB, true, 10 * GiB};+ require(cgroup_available_bytes(32 * GiB, dirty_active, bytes) && bytes == 32 * GiB - (20 * GiB - 7 * GiB) - GiB / 2,+ "dirty / writeback pages of the active cache were counted as reclaimable");+ CgroupMemoryStat tiny_cache{8 * GiB, 0, 0, 0, true, 256ull << 20};+ require(cgroup_available_bytes(32 * GiB, tiny_cache, bytes) && bytes == 32 * GiB - 8 * GiB + (128ull << 20),+ "a small clean cache was not kept at half");+ // "file" larger than what is charged (a racy read) is bounded by memory.current+ CgroupMemoryStat racy{4 * GiB, 1 * GiB, 0, 0, true, 9 * GiB};+ require(cgroup_available_bytes(32 * GiB, racy, bytes) && bytes == 32 * GiB - GiB / 2,+ "a file counter above memory.current escaped the bound");const uint64_t max = std::numeric_limits<uint64_t>::max();CgroupMemoryStat large_counters{max, max, max - 1, max, true};@@ -562,7 +585,13 @@ void test_host_memory() {put("fs/box/memory.max", std::to_string(48 * GiB) + "\n");put("fs/box/memory.current", std::to_string(16 * GiB) + "\n");put("fs/box/memory.stat", "inactive_file " + std::to_string(4 * GiB) + "\nfile_dirty 0\nfile_writeback 0\n");- require(probe("0::/box\n") && m.available == 36 * GiB && m.cgroup_limit == 48 * GiB, "v2 limit");+ require(probe("0::/box\n") && m.available == 36 * GiB - 48 * GiB / 64 && m.cgroup_limit == 48 * GiB, "v2 limit");+ // 149: with "file" (active + inactive cache) the active part counts too: 12 GiB of cache, 4 GiB anonymous+ put("fs/box/memory.stat", "file " + std::to_string(12 * GiB) + "\ninactive_file " + std::to_string(4 * GiB) ++ "\nfile_dirty 0\nfile_writeback 0\n");+ require(probe("0::/box\n") && m.available == 44 * GiB - 48 * GiB / 64 && m.cgroup_limit == 48 * GiB,+ "v2 limit, the active page cache credited");+ put("fs/box/memory.stat", "inactive_file " + std::to_string(4 * GiB) + "\nfile_dirty 0\nfile_writeback 0\n");put("fs/box/memory.max", "max\n");require(probe("0::/box\n") && m.available == 100 * GiB && m.cgroup_limit == ~0ull, "v2 max");put("fs/box/memory.max", "48G\n");