Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Linux: io prefetch + decode gate by default for experts.bin packs when the page cache has room for 55% of the experts
Niko1221ecf2bdcb7f05
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 07c9b3e68..0a574939c 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -270,8 +270,9 @@ engine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) w(Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet.**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read-from a GGUF read in place (the native packs, also with `--mmap-experts`), Linux now runs three things by default (an-`experts.bin` pack, such as the NVFP4 ones, keeps them off unless you set them: not measured there yet).+from a GGUF read in place (the native packs, also with `--mmap-experts`), Linux now runs three things by default+(0.2.1: an `experts.bin` pack gets io prefetch and the decode gate when the page cache has room for at least 55% of the experts, see below; zero-copy+does not apply to it, its blob is one range the kernels already read from the mapping).(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the pagecache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates@@ -293,6 +294,9 @@ host: before, the 18 GiB RAM tier in a 32 GiB cgroup re-checked "1.2 GiB availabtheir cache in the group, and its file tier switched to unbuffered reads (which also left zero-copy off). Off switches: `STRATA_COLD_OVERLAP=0`, `STRATA_IO_PREFETCH=0`,`STRATA_IO_PF_GATE=0`; the startup log names what is on. Windows and macOS are unchanged.+**Linux, `experts.bin` packs (0.2.1):** io prefetch and the decode gate are on by default when the page cache has room for at least 55% of the experts+(the tightest cgroup limit or MemAvailable over the pack's expert bytes, asked once at start; `STRATA_IO_PF_GATE_LO`). Below that the pack reads as before and the log says so; `STRATA_IO_PREFETCH=1` forces it, `0` turns it off. Measured on an RTX 3060 12 GB, IQ3_XXS pack (43 GB `experts.bin`), `--mmap-experts`, cgroup RAM limit, page cache dropped before each engine, 5 interleaved pairs: 32 GB decode 1.38x / 1.21x / 1.33x (story / code / doc8k, 5/5 pairs faster each), prompts 4K 1.04x, 20K 1.06x. At 16 GB io prefetch gave decode +3..8% but the cold 4K prompt 4-6% slower, so it stays off there (room 38%). Identity gates (fixed flags, no RAM limit and a 32 GB cgroup): 10/10 texts, exact_pp 4K / 20K residual dumps IDENTICAL.+**Linux file-tier I/O path (`STRATA_IO_PREFETCH=1`; 0.1.40.2, on by default on Linux since 0.2.0 - see above):** the mapped file tier reads experts throughpage faults. With this on, the layer's uncached experts and the router-predicted ones of the next layer (`STRATA_LOOKAHEAD_K`,`STRATA_IO_PREFETCH_DEPTH`, default 2 when on) are read by I/O threads (`STRATA_IO_PF_THREADS`, default 8) with whole-blobdiff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hppindex fb5e64281..fe8bf184c 100644--- a/include/strata/core/expert_source.hpp+++ b/include/strata/core/expert_source.hpp@@ -997,6 +997,20 @@ public:#if defined(__linux__)bool linux_file_defaults() const override { return !role_ptr_.empty(); } // the GGUF in place only (measured)#endif+ /// 0.2.1: io prefetch + the decode gate by default also for an experts.bin pack (zero-copy is moot there: its blob is one+ /// contiguous range the kernels already read from the mapping; the COLD_OVERLAP default stays with the GGUF in place).+ /// Measured (RTX 3060, 43 GB pack): 32 GB of RAM decode +20..+34%, 16 GB decode +3..+8% but the cold 4K prompt -4..-6%, so+ /// below the gate's room test the pack keeps reading as before+ bool linux_io_defaults() const {+#if defined(__linux__)+ return role_ptr_.empty() ? cache_room_ratio() >= 0.55 : true; // experts.bin: only while the page cache has room for it+#else+ return false;+#endif+ }+ /// the memory the page cache can use (the tightest cgroup limit less what is not page cache, else MemAvailable) over the+ /// experts' bytes; 0 when unknown. The decode gate's room test (STRATA_IO_PF_GATE_LO), asked once at start.+ double cache_room_ratio() const;bool gate_on() const { return gate_on_; }double gate_held() const { return gate_ewma_; }int64_t gate_closed_windows() const { return gate_closed_w_; }diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cppindex 5cd037da2..7be407d29 100644--- a/src/core/expert_source.cpp+++ b/src/core/expert_source.cpp@@ -1596,6 +1596,11 @@ uint64_t gate_cache_room() {}}+double FileExpertSource::cache_room_ratio() const {+ const uint64_t total = expert_bytes(), room = gate_cache_room();+ return total == 0 || room == 0 ? 0.0 : (double) room / (double) total;+}+void FileExpertSource::gate_capacity() {if (!gate_on_ || (gate_win_++ % 16) != 0) return;const uint64_t total = expert_bytes(), room = gate_cache_room();diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 01b99f06c..665ccc9c1 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -5515,18 +5515,26 @@ int main(int argc, char** argv) {// ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch). 0.2.0: on by// default on Linux, with the decode gate (STRATA_IO_PF_GATE) and zero-copy page-cached experts// (STRATA_COLD_OVERLAP=4); STRATA_IO_PREFETCH=0 / STRATA_IO_PF_GATE=0 / STRATA_COLD_OVERLAP=0 turn them off.- // the defaults are for the GGUF read in place (what was measured); an experts.bin pack keeps them off unless set+ // io prefetch + the gate: the GGUF read in place and (0.2.1) an experts.bin pack; zero-copy only for the GGUF in placeconst char* io_pf_env = std::getenv("STRATA_IO_PREFETCH");- const bool io_pf_default = (io_pf_env == nullptr || io_pf_env[0] == '\0') && src.linux_file_defaults();+ const bool io_pf_default = (io_pf_env == nullptr || io_pf_env[0] == '\0') && src.linux_io_defaults();+#if defined(__linux__)+ if ((io_pf_env == nullptr || io_pf_env[0] == '\0') && !io_pf_default && !src.linux_file_defaults() && !src.unbuffered())+ std::fprintf(stderr, "strata generate: file tier (experts.bin): io prefetch stays off by default here: the page cache has room "+ "for %.0f%% of the experts (it helps from 55%%: 32 GB-class RAM, decode +20..34%% measured); "+ "STRATA_IO_PREFETCH=1 turns it on anyway\n", src.cache_room_ratio() * 100.0);+#endifif (io_pf_default || (io_pf_env != nullptr && std::atoi(io_pf_env) != 0)) {const char* th = std::getenv("STRATA_IO_PF_THREADS");src.set_io_prefetch(true, th != nullptr ? std::atoi(th) : 0);if (src.io_prefetch() && io_pf_default) {const char* cv = std::getenv("STRATA_COLD_OVERLAP");- std::fprintf(stderr, "strata generate: file tier (Linux defaults, the GGUF read in place): page-cached experts "+ std::fprintf(stderr, "strata generate: file tier (Linux defaults, %s): page-cached experts ""computed from the mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, ""the decode's read-ahead %s; off: STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, "- "STRATA_IO_PF_GATE=0\n", cv != nullptr && cv[0] != '\0' ? cv : "4", th != nullptr ? th : "8",+ "STRATA_IO_PF_GATE=0\n",+ src.linux_file_defaults() ? "the GGUF read in place" : "experts.bin: its pages are read from the mapping as they are",+ cv != nullptr && cv[0] != '\0' ? cv : (src.linux_file_defaults() ? "4" : "0"), th != nullptr ? th : "8",src.gate_on() ? "only while the page cache has room for it and it pays (STRATA_IO_PF_GATE=1)": "always (STRATA_IO_PF_GATE=0)");} else if (src.io_prefetch())