Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

149: the Linux file-tier defaults only for the GGUF read in place

Niko1221c31367979d7c
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index e0861c7cf..2f90c88e8 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -262,7 +262,8 @@ engine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) w
(Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet.
**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read
-from the files (a GGUF read in place, `--mmap-experts`, or an `experts.bin` file tier), Linux now runs three things by default.
+from a GGUF read in place (the native packs, also with `--mmap-experts`), Linux now runs three things by default (an
+`experts.bin` pack, such as the NVFP4 ones, keeps them off unless you set them: not measured there yet).
(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the page
cache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer
(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates
diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp
index ab7ae4a4a..e14d3f4e2 100644
--- a/include/strata/core/expert_source.hpp
+++ b/include/strata/core/expert_source.hpp
@@ -259,6 +259,8 @@ public:
virtual void prefetch_end() {}
/// 149 STRATA_IO_PF_GATE: whether the decode's read-ahead runs now (the look-ahead then predicts one layer, not two)
virtual bool read_ahead_open() const { return true; }
+ /// 0.2.0: the source the Linux file-tier defaults apply to (the GGUF read in place); STRATA_COLD_OVERLAP's default
+ virtual bool linux_file_defaults() const { return false; }
/// 149-cold STRATA_COLD_OVERLAP=3: the last prefetch_begin left this expert unstaged because the page cache holds
/// all of it (the GGUF read in place): its gate, up and down rows in the mapping. False: ask `blob`.
virtual bool zc_roles(int64_t layer, int64_t expert, const uint8_t** p) { (void) layer; (void) expert; (void) p; return false; }
@@ -992,6 +994,9 @@ public:
static bool in_decode();
bool gate_open() const { return !gate_on_ || gate_open_.load(std::memory_order_relaxed); }
bool read_ahead_open() const override { return gate_open(); }
+#if defined(__linux__)
+ bool linux_file_defaults() const override { return !role_ptr_.empty(); } // the GGUF in place only (measured)
+#endif
bool gate_on() const { return gate_on_; }
double gate_held() const { return gate_ewma_; }
int64_t gate_closed_windows() const { return gate_closed_w_; }
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
index 38b8be0be..5cd037da2 100644
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -1822,14 +1822,11 @@ int64_t FileExpertSource::prefetch_begin(int64_t layer, const int64_t* experts,
prefetch_end();
async_todo_.clear();
async_fill_us_ = 0;
- static const int mode = [] { // as the dispatch reads it (0.2.0: 4 by default on Linux)
+ static const int mode_env = [] { // as the dispatch reads it (0.2.0: unset = 4 for the GGUF read in place on Linux)
const char* v = std::getenv("STRATA_COLD_OVERLAP");
-#if defined(__linux__)
- return v == nullptr || v[0] == '\0' ? 4 : std::atoi(v);
-#else
- return v == nullptr || v[0] == '\0' ? 0 : std::atoi(v);
-#endif
+ return v == nullptr || v[0] == '\0' ? -1 : std::atoi(v);
}();
+ const int mode = mode_env >= 0 ? mode_env : (linux_file_defaults() ? 4 : 0);
claim_todo(layer, experts, n, async_todo_, pending, mode >= 3);
if (mode == 4) { // zero-copy only: the blobs that must be read are read here, as `prefetch` reads them
const auto t0 = std::chrono::steady_clock::now();
@@ -4383,17 +4380,13 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
};
const auto c0 = std::chrono::steady_clock::now();
const FileExpertSource::DecodeScope decode_scope; // 149 STRATA_IO_PF_GATE: this thread reads for the decode
- // 149-cold STRATA_COLD_OVERLAP: 0 off (default), 1 overlap, 2 measure only, 3 overlap + zero-copy, 4 zero-copy only
- // 0.2.0: 4 by default on Linux (zero-copy needs the page cache's residency query); elsewhere 0
+ // 149-cold STRATA_COLD_OVERLAP: 0 off, 1 overlap, 2 measure only, 3 overlap + zero-copy, 4 zero-copy only.
+ // 0.2.0: unset, 4 for the GGUF read in place on Linux (zero-copy needs the page cache's residency query), else 0
static const int cold_env = [] {
const char* v = std::getenv("STRATA_COLD_OVERLAP");
-#if defined(__linux__)
- return v == nullptr || v[0] == '\0' ? 4 : std::clamp(std::atoi(v), 0, 4);
-#else
- return v == nullptr || v[0] == '\0' ? 0 : std::clamp(std::atoi(v), 0, 4);
-#endif
+ return v == nullptr || v[0] == '\0' ? -1 : std::clamp(std::atoi(v), 0, 4);
}();
- d.cold_mode = cold_env;
+ d.cold_mode = cold_env >= 0 ? cold_env : (d.src != nullptr && d.src->linux_file_defaults() ? 4 : 0);
pt("begin");
d.src->begin_layer(d.layers, ids, n_tok * k);
pt("begun");
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 95975f1ad..f5a5407c9 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -5449,19 +5449,16 @@ int main(int argc, char** argv) {
// ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch). 0.2.0: on by
// default on Linux, with the decode gate (STRATA_IO_PF_GATE) and zero-copy page-cached experts
// (STRATA_COLD_OVERLAP=4); STRATA_IO_PREFETCH=0 / STRATA_IO_PF_GATE=0 / STRATA_COLD_OVERLAP=0 turn them off.
+ // the defaults are for the GGUF read in place (what was measured); an experts.bin pack keeps them off unless set
const char* io_pf_env = std::getenv("STRATA_IO_PREFETCH");
-#if defined(__linux__)
- const bool io_pf_default = io_pf_env == nullptr || io_pf_env[0] == '\0';
-#else
- const bool io_pf_default = false;
-#endif
+ const bool io_pf_default = (io_pf_env == nullptr || io_pf_env[0] == '\0') && src.linux_file_defaults();
if (io_pf_default || (io_pf_env != nullptr && std::atoi(io_pf_env) != 0)) {
const char* th = std::getenv("STRATA_IO_PF_THREADS");
src.set_io_prefetch(true, th != nullptr ? std::atoi(th) : 0);
if (src.io_prefetch() && io_pf_default) {
const char* cv = std::getenv("STRATA_COLD_OVERLAP");
- std::fprintf(stderr, "strata generate: file tier (Linux defaults): page-cached experts computed from the "
- "mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, "
+ std::fprintf(stderr, "strata generate: file tier (Linux defaults, the GGUF read in place): page-cached experts "
+ "computed from the mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, "
"the decode's read-ahead %s; off: STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, "
"STRATA_IO_PF_GATE=0\n", cv != nullptr && cv[0] != '\0' ? cv : "4", th != nullptr ? th : "8",
src.gate_on() ? "only while the page cache has room for it and it pays (STRATA_IO_PF_GATE=1)"