Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Linux: io prefetch + decode gate by default for experts.bin packs when the page cache has room for 55% of the experts

Niko1221ecf2bdcb7f05
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 07c9b3e68..0a574939c 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -270,8 +270,9 @@ engine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) w
(Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet.
**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read
-from a GGUF read in place (the native packs, also with `--mmap-experts`), Linux now runs three things by default (an
-`experts.bin` pack, such as the NVFP4 ones, keeps them off unless you set them: not measured there yet).
+from a GGUF read in place (the native packs, also with `--mmap-experts`), Linux now runs three things by default
+(0.2.1: an `experts.bin` pack gets io prefetch and the decode gate when the page cache has room for at least 55% of the experts, see below; zero-copy
+does not apply to it, its blob is one range the kernels already read from the mapping).
(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the page
cache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer
(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates
@@ -293,6 +294,9 @@ host: before, the 18 GiB RAM tier in a 32 GiB cgroup re-checked "1.2 GiB availab
their cache in the group, and its file tier switched to unbuffered reads (which also left zero-copy off). Off switches: `STRATA_COLD_OVERLAP=0`, `STRATA_IO_PREFETCH=0`,
`STRATA_IO_PF_GATE=0`; the startup log names what is on. Windows and macOS are unchanged.
+**Linux, `experts.bin` packs (0.2.1):** io prefetch and the decode gate are on by default when the page cache has room for at least 55% of the experts
+(the tightest cgroup limit or MemAvailable over the pack's expert bytes, asked once at start; `STRATA_IO_PF_GATE_LO`). Below that the pack reads as before and the log says so; `STRATA_IO_PREFETCH=1` forces it, `0` turns it off. Measured on an RTX 3060 12 GB, IQ3_XXS pack (43 GB `experts.bin`), `--mmap-experts`, cgroup RAM limit, page cache dropped before each engine, 5 interleaved pairs: 32 GB decode 1.38x / 1.21x / 1.33x (story / code / doc8k, 5/5 pairs faster each), prompts 4K 1.04x, 20K 1.06x. At 16 GB io prefetch gave decode +3..8% but the cold 4K prompt 4-6% slower, so it stays off there (room 38%). Identity gates (fixed flags, no RAM limit and a 32 GB cgroup): 10/10 texts, exact_pp 4K / 20K residual dumps IDENTICAL.
+
**Linux file-tier I/O path (`STRATA_IO_PREFETCH=1`; 0.1.40.2, on by default on Linux since 0.2.0 - see above):** the mapped file tier reads experts through
page faults. With this on, the layer's uncached experts and the router-predicted ones of the next layer (`STRATA_LOOKAHEAD_K`,
`STRATA_IO_PREFETCH_DEPTH`, default 2 when on) are read by I/O threads (`STRATA_IO_PF_THREADS`, default 8) with whole-blob
diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp
index fb5e64281..fe8bf184c 100644
--- a/include/strata/core/expert_source.hpp
+++ b/include/strata/core/expert_source.hpp
@@ -997,6 +997,20 @@ public:
#if defined(__linux__)
bool linux_file_defaults() const override { return !role_ptr_.empty(); } // the GGUF in place only (measured)
#endif
+ /// 0.2.1: io prefetch + the decode gate by default also for an experts.bin pack (zero-copy is moot there: its blob is one
+ /// contiguous range the kernels already read from the mapping; the COLD_OVERLAP default stays with the GGUF in place).
+ /// Measured (RTX 3060, 43 GB pack): 32 GB of RAM decode +20..+34%, 16 GB decode +3..+8% but the cold 4K prompt -4..-6%, so
+ /// below the gate's room test the pack keeps reading as before
+ bool linux_io_defaults() const {
+#if defined(__linux__)
+ return role_ptr_.empty() ? cache_room_ratio() >= 0.55 : true; // experts.bin: only while the page cache has room for it
+#else
+ return false;
+#endif
+ }
+ /// the memory the page cache can use (the tightest cgroup limit less what is not page cache, else MemAvailable) over the
+ /// experts' bytes; 0 when unknown. The decode gate's room test (STRATA_IO_PF_GATE_LO), asked once at start.
+ double cache_room_ratio() const;
bool gate_on() const { return gate_on_; }
double gate_held() const { return gate_ewma_; }
int64_t gate_closed_windows() const { return gate_closed_w_; }
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
index 5cd037da2..7be407d29 100644
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -1596,6 +1596,11 @@ uint64_t gate_cache_room() {
}
}
+double FileExpertSource::cache_room_ratio() const {
+ const uint64_t total = expert_bytes(), room = gate_cache_room();
+ return total == 0 || room == 0 ? 0.0 : (double) room / (double) total;
+}
+
void FileExpertSource::gate_capacity() {
if (!gate_on_ || (gate_win_++ % 16) != 0) return;
const uint64_t total = expert_bytes(), room = gate_cache_room();
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 01b99f06c..665ccc9c1 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -5515,18 +5515,26 @@ int main(int argc, char** argv) {
// ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch). 0.2.0: on by
// default on Linux, with the decode gate (STRATA_IO_PF_GATE) and zero-copy page-cached experts
// (STRATA_COLD_OVERLAP=4); STRATA_IO_PREFETCH=0 / STRATA_IO_PF_GATE=0 / STRATA_COLD_OVERLAP=0 turn them off.
- // the defaults are for the GGUF read in place (what was measured); an experts.bin pack keeps them off unless set
+ // io prefetch + the gate: the GGUF read in place and (0.2.1) an experts.bin pack; zero-copy only for the GGUF in place
const char* io_pf_env = std::getenv("STRATA_IO_PREFETCH");
- const bool io_pf_default = (io_pf_env == nullptr || io_pf_env[0] == '\0') && src.linux_file_defaults();
+ const bool io_pf_default = (io_pf_env == nullptr || io_pf_env[0] == '\0') && src.linux_io_defaults();
+#if defined(__linux__)
+ if ((io_pf_env == nullptr || io_pf_env[0] == '\0') && !io_pf_default && !src.linux_file_defaults() && !src.unbuffered())
+ std::fprintf(stderr, "strata generate: file tier (experts.bin): io prefetch stays off by default here: the page cache has room "
+ "for %.0f%% of the experts (it helps from 55%%: 32 GB-class RAM, decode +20..34%% measured); "
+ "STRATA_IO_PREFETCH=1 turns it on anyway\n", src.cache_room_ratio() * 100.0);
+#endif
if (io_pf_default || (io_pf_env != nullptr && std::atoi(io_pf_env) != 0)) {
const char* th = std::getenv("STRATA_IO_PF_THREADS");
src.set_io_prefetch(true, th != nullptr ? std::atoi(th) : 0);
if (src.io_prefetch() && io_pf_default) {
const char* cv = std::getenv("STRATA_COLD_OVERLAP");
- std::fprintf(stderr, "strata generate: file tier (Linux defaults, the GGUF read in place): page-cached experts "
+ std::fprintf(stderr, "strata generate: file tier (Linux defaults, %s): page-cached experts "
"computed from the mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, "
"the decode's read-ahead %s; off: STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, "
- "STRATA_IO_PF_GATE=0\n", cv != nullptr && cv[0] != '\0' ? cv : "4", th != nullptr ? th : "8",
+ "STRATA_IO_PF_GATE=0\n",
+ src.linux_file_defaults() ? "the GGUF read in place" : "experts.bin: its pages are read from the mapping as they are",
+ cv != nullptr && cv[0] != '\0' ? cv : (src.linux_file_defaults() ? "4" : "0"), th != nullptr ? th : "8",
src.gate_on() ? "only while the page cache has room for it and it pays (STRATA_IO_PF_GATE=1)"
: "always (STRATA_IO_PF_GATE=0)");
} else if (src.io_prefetch())