Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
KVA ships OFF in 0.2.7: STRATA_KVA unset is the exact 0.2.6 engine, STRATA_KVA=1 turns it on with an 8192-token exact tail
Niko1221e3450e455218
diff --git a/docs/KVA.md b/docs/KVA.mdindex e94a4dfc6..15aacaae6 100644--- a/docs/KVA.md+++ b/docs/KVA.md@@ -1,4 +1,4 @@-# KVA: the faster prompt read for long cold prompts (on by default since 0.2.7)+# KVA: a faster prompt read for long cold prompts (opt-in, off by default)KVA ("approximate late-layer state") reads a long prompt exactly through layer 23, then for the late layers24–47 builds **only their recurrent state** — the GDN recurrence and convolution history, the QSA K/V pages and@@ -7,9 +7,31 @@ the indexer keys — instead of running the whole layer. The attention output, tresults. The last positions of the prompt are then read fully, exactly as before, and decode runs the ordinaryfull path against the state KVA built.-It is an approximation: the answers can differ from the exact read (see "Measured" for how much). **`STRATA_KVA=0`-turns it off and gives the exact 0.2.6 prompt path.** The Strata app shows this as the "Faster long prompts" switch-in Expert settings; it sends `STRATA_KVA=0` when the switch is off.+**It is off by default.** Unset, the engine is 0.2.6's, prompt path and draft layer included. Set `STRATA_KVA=1` to+turn it on (the Strata app shows it as the "Faster long prompts" switch in Expert settings, off until the user turns+it on; the Terminal passes `STRATA_KVA=1`). When it is on, the engine uses the safest configuration measured: an+**8192-token exact tail** (`STRATA_KVA_SUFFIX` changes it).++## Why it is off++KVA is an approximation, and the 0.2.7 quality runs found where it costs answers. The runs compare KVA off and on on the+same build, greedy, with a noise run (off against off):++- **Extraction of many scattered records** (a list of 8 register entries in a 14K document, 6 such tasks at 14K and+ 30K): with the 2048-token tail, 4 of 6 tasks passed against 6 of 6 exact, 249 of 270 facts found against 270.+ The model silently dropped 3 of 8 records at 14K and, at 30K, started its list with an invented row+ (the request's own reference id leaked into the answer). Neither exact run did either.+- **Agent loops with a ~10K-token system prompt** (handbook, tool docs, rules): the first 8192 tokens, about 84 % of+ the system prompt, were approximated. Two of three runs had defects the exact runs did not: a final answer that+ left out the totals, and a split-order quote with wrong arithmetic (the price of 80 units for 70, a shortfall of+ 40 instead of 50).+- Needle retrieval, unanswerable questions, short summaries, code and JSON were unchanged. The speed gain is real:+ about 1.3x prompt speed at 14K and 1.5x at 30K on a Tesla P100.++The 8192-token tail approximates nothing below roughly 16K tokens (the chunks are 8192 tokens, and a chunk is+approximated only when it ends before the tail) and about half of a 30K prompt. It was the only variant that passed the+same runs on the Radeon AI PRO R9700 and tied the exact arm on the RTX 5070. Its speed gain is correspondingly smaller+than the figures above, which were measured with the 2048-token tail.## When it runs@@ -21,14 +43,13 @@ KVA runs when all of these hold, and says so in the engine log:- the prompt (or the part of it read in one go, see below) has at least `STRATA_KVA_MIN_PROMPT` tokens (4096) andmore than the fully computed tail plus 64.-Otherwise the prompt is read exactly. An engine that was started with `STRATA_KVA=1` logs the reason for every-read it turned away (`strata prefill: KVA configured but NOT applied: <reason>`); with the default it logs only-a long prompt turned away for an unusual reason (a layer split, pictures), and once at start-up when the engine-configuration rules KVA out for the whole run.+Otherwise the prompt is read exactly. An engine started with `STRATA_KVA=1` logs the reason for every read it turned+away (`strata prefill: KVA configured but NOT applied: <reason>`), and once at start-up when the engine configuration+rules KVA out for the whole run. Without `STRATA_KVA=1` it logs nothing about KVA.## The fully computed tail and the draft layer-The last `suffix` positions (2048, or `STRATA_KVA_SUFFIX`) are always read exactly.+The last `suffix` positions (8192, or `STRATA_KVA_SUFFIX`) are always read exactly.The MTP draft layer builds its K/V from the final residuals of the prompt. An approximated chunk has no finalresidual of its own, so its draft cells are built from the rows that chunk does have. By default nothing about the@@ -37,8 +58,8 @@ attends to the same cells as with `STRATA_KVA=0`. The target model verifies everhow many tokens a verify window keeps (decode speed), never the answer.Measured on the P100 (IQ3_XXS, auto cache and prefill, 14K and 30K prompts, 256 tokens), the draft layer attends to-those cells with acceptance within noise of the exact read. A forced small window (the first default-on build set-`--mtp-window 2048` for every session) and the research option below both cost acceptance on faster cards.+those cells with acceptance within noise of the exact read at 14K and a few points lower at 30K (decode 6 % lower at+30K). A forced small window (`--mtp-window 2048`) and the research option below both cost acceptance on faster cards.`STRATA_KVA_DRAFT_FLOOR=1` (research): the draft layer gets a *cell floor* per session, the first cell built from a realrow. It builds no draft cell below it and attends to `[max(floor, pos - window), pos)`, the exact tail growing with@@ -62,7 +83,7 @@ greater than 0 and is read exactly.debug check) differ with KVA on, as they would between any two reads that take different paths.- **A cold read again** (the cache evicted the conversation, a new chat with the same system prompt) approximatesits own prefix again.-- **`STRATA_KVA=0`** restores 0.2.6: every prompt is read exactly, the draft window is 32768 cells, nothing else+- **Unset (or `STRATA_KVA=0`)** is 0.2.6: every prompt is read exactly, the draft window is 32768 cells, nothing elsechanges.## Layer splits, batch slots, SYCL, Metal@@ -76,9 +97,9 @@ there, with a line when it is set).| variable | default | what it does ||---|---:|---|-| `STRATA_KVA` | on | `0` turns KVA off (the exact 0.2.6 prompt read). `1` is the same as the default but logs every read it turned away |+| `STRATA_KVA` | off | `1` turns KVA on (with the 8192-token tail). Unset or `0`: the exact 0.2.6 prompt read and draft layer || `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |-| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; see above |+| `STRATA_KVA_SUFFIX` | 8192 | fully computed tail positions; see above. 2048 is the fastest setting and the one the extraction and agent defects were found with || `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path || `STRATA_KVA_DRAFT_FLOOR` | off | `1`: the draft layer skips the draft cells of approximated chunks (research) || `STRATA_KVA_SKIP` | state | `moe` (opt-in): the late layers of an approximated chunk run their whole attention half (GDN and QSA outputs into the residual) and skip only the MoE half and its expert transfers; the log line says `mode moe-skip`. Slower than the default state-only form, closer to the exact residual |@@ -92,10 +113,10 @@ there, with a line when it is set).Every read says what it did:-- applied: `strata prefill: KVA on: split 24, suffix 2048, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`+- applied (here with `STRATA_KVA_SUFFIX=2048`): `strata prefill: KVA on: split 24, suffix 2048, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`- turned away: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`-A quality comparison needs the control arm to say `STRATA_KVA=0` (an unset variable is now KVA).+A quality comparison needs the KVA arm to say `STRATA_KVA=1`; the control arm leaves it unset or says `STRATA_KVA=0`.## Measured (Tesla P100 16 GB, Qwen3.8-Flash-Next IQ3_XXS, chunk 1024, expert cache 1500, MTP window 2048)diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hppindex ea6727099..f765fb85d 100644--- a/include/strata/prefill/kva.hpp+++ b/include/strata/prefill/kva.hpp@@ -1,4 +1,4 @@-// include/strata/prefill/kva.hpp - STRATA_KVA: the approximate late-layer prompt read (on by default since 0.2.7).+// include/strata/prefill/kva.hpp - STRATA_KVA: the approximate late-layer prompt read (opt-in since 0.2.7).//// After layer `split` the prompt's late layers need only their recurrent state (GDN recurrence + conv history)// and their KV/indexer caches; the attention output, the output projection, the MoE half and its expert@@ -14,8 +14,10 @@// oracle a captured real input per late layer (STRATA_KVA_ORACLE=<dir>; the plumbing/parity mode),// linear a fitted low-rank projector (STRATA_KVA_PROJECTOR=<file>; not implemented in this step).//-// 0.2.7: the read is ON by default for a cold prompt of STRATA_KVA_MIN_PROMPT tokens or more on one CUDA/HIP-// device. STRATA_KVA=0 brings back the exact 0.2.6 prompt path and the 0.2.6 draft window. Everything is read from+// 0.2.7: the read is OFF unless STRATA_KVA=1 is set (the quality runs found lost records and invented rows in long+// extraction and agent tasks, docs/KVA.md). Unset, nothing here runs and the engine is 0.2.6's. Switched on, it applies+// to a cold prompt of STRATA_KVA_MIN_PROMPT tokens or more on one CUDA/HIP device, with an 8192-token exact tail+// (the safest configuration measured). Everything is read from// the environment once per process.//// What the engine tells this header at start-up (kva_engine()): whether its configuration rules the read out for the@@ -47,7 +49,7 @@ inline constexpr bool kKvaBackend = true;#endif/// The fully computed tail of every approximated read.-inline constexpr int64_t kKvaSuffixDefault = 2048;+inline constexpr int64_t kKvaSuffixDefault = 8192;/// What the running engine knows and the environment cannot say.struct KvaEngine {@@ -67,7 +69,7 @@ struct KvaConfig {/// the MoE half and its expert transfers.enum class Skip { StateOnly, Moe };- bool on = true; ///< on unless STRATA_KVA=0 (or this build has no path)+ bool on = false; ///< STRATA_KVA=1 (and this build has a path); unset or 0: offbool explicit_on = false; ///< STRATA_KVA was set to something other than 0: every declined read says whybool capture = false; ///< STRATA_KVA_CAPTURE=<dir>: dump every late layer's real input rowsbool capture_fp32 = false; ///< STRATA_KVA_CAPTURE_FP32=1: fp32 records (else bf16; fp32 is for parity runs)@@ -84,7 +86,7 @@ struct KvaConfig {std::string capture_dir; ///< STRATA_KVA_CAPTUREstd::string projector; ///< STRATA_KVA_PROJECTOR- /// The fully computed tail of a read: STRATA_KVA_SUFFIX when set, otherwise 2048.+ /// The fully computed tail of a read: STRATA_KVA_SUFFIX when set, otherwise 8192.int64_t effective_suffix() const { return suffix_env > 0 ? suffix_env : kKvaSuffixDefault; }/// Does the draft layer need the cell floor at all? Only when a read can approximate something: KVA on, not ruled@@ -96,17 +98,13 @@ struct KvaConfig {/// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a/// continuation, no layer split, no pictures, no DFlash2 taps), on an engine that allows it, long enough for a/// suffix to leave approximated positions behind it. `routine` (optional) is set when the reason is an- /// everyday one (a continuation, a short prompt) that an engine running KVA by default does not log.+ /// everyday one (a continuation, a short prompt) that is not worth a log line.const char* decline(const KvaEngine& e, int64_t pos0, int64_t n, int64_t lb, int64_t le, int64_t n_layers,- bool pictures = false, bool taps = false, bool* routine = nullptr,- bool unvalidated_model = false) const {+ bool pictures = false, bool taps = false, bool* routine = nullptr) const {if (routine) *routine = false;- if (!on) return "off (STRATA_KVA=0)";+ if (!on) return "off (STRATA_KVA is not 1)";if (capture) return "capture mode";if (!e.blocked.empty()) return e.blocked.c_str();- // The split layer, the state-only layers and the suffix rule were measured on one geometry (Qwen3.8-Flash-Next,- // 48 layers, 512 experts). Another model reads exactly unless STRATA_KVA=1 asks for it.- if (unvalidated_model && !explicit_on) return "KVA was not validated for this model (STRATA_KVA=1 forces it)";if (pos0 != 0) {if (routine) *routine = true;return "a continuation read (pos0 != 0)";@@ -143,8 +141,8 @@ inline KvaConfig kva_config_from_env() {KvaConfig k;const char* sw = std::getenv("STRATA_KVA");const bool set = sw != nullptr && *sw != '\0';- k.on = !set || sw[0] != '0';- k.explicit_on = set && sw[0] != '0';+ k.on = set && sw[0] != '0';+ k.explicit_on = k.on;if (!kKvaBackend) k.on = false;k.split = kva_env_i64("STRATA_KVA_SPLIT", 24);k.suffix_env = std::max<int64_t>(0, kva_env_i64("STRATA_KVA_SUFFIX", 0));diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cppindex 0789e3313..1715a90ab 100644--- a/src/prefill/kva_test.cpp+++ b/src/prefill/kva_test.cpp@@ -37,30 +37,34 @@ static void clear_all() {static bool is(const char* got, const char* want) { return got != nullptr && std::strcmp(got, want) == 0; }int main() {- // ---- the switch: on by default, STRATA_KVA=0 off, anything else on and chatty+ // ---- the switch: off unless STRATA_KVA=1 (0.2.7 ships KVA opt-in; unset is the exact 0.2.6 engine)clear_all();{const KvaConfig c = kva_config_from_env();- CHECK(c.on && !c.explicit_on);+ CHECK(!c.on && !c.explicit_on); // unset: offCHECK(c.split == 24 && c.min_prompt == 4096 && c.suffix_env == 0);CHECK(c.mode == KvaConfig::Mode::Identity && c.only == KvaConfig::Only::All && !c.capture);+ KvaEngine eng;+ CHECK(!c.wants_cell_floor(eng, true));+ CHECK(is(c.decline(eng, 0, 30000, 0, -1, 48), "off (STRATA_KVA is not 1)")); // a 30K cold prompt is read exactly}put("STRATA_KVA", "0");{ const KvaConfig c = kva_config_from_env(); CHECK(!c.on && !c.explicit_on); }put("STRATA_KVA", "1");{ const KvaConfig c = kva_config_from_env(); CHECK(c.on && c.explicit_on); }put("STRATA_KVA", ""); // an empty value is "not set"- { const KvaConfig c = kva_config_from_env(); CHECK(c.on && !c.explicit_on); }+ { const KvaConfig c = kva_config_from_env(); CHECK(!c.on && !c.explicit_on); }clear_all();+ put("STRATA_KVA", "1"); // everything below runs with KVA switched on- // ---- the tail: 2048, or STRATA_KVA_SUFFIX; the draft window plays no part (the cell floor protects the draft layer)+ // ---- the tail: 8192 when switched on (the safest configuration measured), or STRATA_KVA_SUFFIX{const KvaConfig c = kva_config_from_env();- CHECK(c.effective_suffix() == kKvaSuffixDefault && kKvaSuffixDefault == 2048);+ CHECK(c.on && c.effective_suffix() == kKvaSuffixDefault && kKvaSuffixDefault == 8192);put("STRATA_KVA_SUFFIX", "3000");const KvaConfig e = kva_config_from_env();CHECK(e.effective_suffix() == 3000);- clear_all();+ unset("STRATA_KVA_SUFFIX");}// ---- who is turned away, and why (the first reason that holds is the one named)@@ -68,42 +72,33 @@ int main() {const KvaConfig c = kva_config_from_env();KvaEngine eng;bool routine = false;- // a cold 14.6K prompt on one device: applies+ // a cold 14.6K prompt on one device: applies (14600 > 8192 + 64)CHECK(c.decline(eng, 0, 14600, 0, -1, 48) == nullptr);CHECK(c.decline(eng, 0, 14600, 0, 48, 48) == nullptr); // le == n_layers is the whole trunk too// a continuation (turn 2 of a chat, a resumed prompt): routineCHECK(is(c.decline(eng, 9000, 14600, 0, -1, 48, false, false, &routine), "a continuation read (pos0 != 0)") && routine);- // short prompts and a prompt the tail would swallow: routine+ // short prompts and a prompt the 8192-token tail would swallow: routineCHECK(is(c.decline(eng, 0, 3000, 0, -1, 48, false, false, &routine), "the prompt is shorter than STRATA_KVA_MIN_PROMPT") && routine);- CHECK(c.decline(eng, 0, 4096, 0, -1, 48) == nullptr); // 4096 > 2112 + 64: it applies- put("STRATA_KVA_SUFFIX", "5000");- { const KvaConfig s = kva_config_from_env();- CHECK(is(s.decline(eng, 0, 5000, 0, -1, 48, false, false, &routine), "the suffix leaves too few positions to approximate") && routine); }- clear_all();+ CHECK(is(c.decline(eng, 0, 8256, 0, -1, 48, false, false, &routine), "the suffix leaves too few positions to approximate") && routine);+ CHECK(c.decline(eng, 0, 8257, 0, -1, 48) == nullptr);+ put("STRATA_KVA_SUFFIX", "2048");+ { const KvaConfig s = kva_config_from_env(); CHECK(s.decline(eng, 0, 4096, 0, -1, 48) == nullptr); }+ unset("STRATA_KVA_SUFFIX");// not routine: the user may want to knowCHECK(is(c.decline(eng, 0, 14600, 24, -1, 48, false, false, &routine), "a layer split (not the whole trunk)") && !routine);CHECK(is(c.decline(eng, 0, 14600, 0, 24, 48), "a layer split (not the whole trunk)"));CHECK(is(c.decline(eng, 0, 14600, 0, -1, 48, true), "a prompt with pictures"));CHECK(is(c.decline(eng, 0, 14600, 0, -1, 48, false, true), "the DFlash2 drafter reads the late layers' hidden states"));- // a model the split and the tail were not measured on reads exactly unless STRATA_KVA=1 asks- CHECK(is(c.decline(eng, 0, 14600, 0, -1, 40, false, false, &routine, true),- "KVA was not validated for this model (STRATA_KVA=1 forces it)") && !routine);- put("STRATA_KVA", "1");- { const KvaConfig f = kva_config_from_env(); CHECK(f.decline(eng, 0, 14600, 0, -1, 40, false, false, nullptr, true) == nullptr); }- clear_all();- // no draft layer at all: nothing to cover// the engine rules the whole process out (batch slots, a layer split, a peer device)KvaEngine blocked = eng;blocked.blocked = "batch slots (--batch)";CHECK(is(c.decline(blocked, 0, 14600, 0, -1, 48, false, false, &routine), "batch slots (--batch)") && !routine);- // off wins over everything, capture over the rest- put("STRATA_KVA", "0");- { const KvaConfig o = kva_config_from_env(); CHECK(is(o.decline(eng, 0, 14600, 0, -1, 48), "off (STRATA_KVA=0)")); }- put("STRATA_KVA", "1");+ // capture wins over the restput("STRATA_KVA_CAPTURE", "/tmp/kva-cap");{ const KvaConfig o = kva_config_from_env(); CHECK(o.capture && is(o.decline(eng, 0, 14600, 0, -1, 48), "capture mode")); }- clear_all();+ unset("STRATA_KVA_CAPTURE");}+ clear_all();// ---- the research modes still parseput("STRATA_KVA_ORACLE", "/tmp/o");@@ -133,8 +128,9 @@ int main() {// (1) by default the draft layer is 0.2.6's own code; the cell floor is a research option, wired only when asked{KvaEngine eng;+ put("STRATA_KVA", "1");const KvaConfig on = kva_config_from_env();- CHECK(!on.draft_floor && !on.wants_cell_floor(eng, true));+ CHECK(on.on && !on.draft_floor && !on.wants_cell_floor(eng, true));put("STRATA_KVA_DRAFT_FLOOR", "1");const KvaConfig fl = kva_config_from_env();CHECK(fl.draft_floor && fl.wants_cell_floor(eng, true));diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cppindex 5ccfeb95b..7528c5d2b 100644--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@ -2720,8 +2720,15 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::striconst int64_t kva_suffix = kva.effective_suffix();bool kva_routine = false;const char* kva_why_not = kva.decline(kva_eng, pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers,- embd_rows != nullptr, !tap_layers_.empty(), &kva_routine,- g.n_layers != 48 || g.n_expert != 512);+ embd_rows != nullptr, !tap_layers_.empty(), &kva_routine);+ if (kva_why_not == nullptr && (g.n_layers != 48 || g.n_expert != 512)) {+ // the split layer and the tail were measured on one geometry; STRATA_KVA=1 is the user's call, but say so once+ static std::atomic<bool> warned{false};+ if (!warned.exchange(true))+ std::fprintf(stderr, "strata prefill: WARNING: KVA was only validated on Qwen3.8-Flash-Next (48 layers, 512 "+ "experts); this model has %lld layers and %lld experts\n", (long long) g.n_layers,+ (long long) g.n_expert);+ }const bool kva_run = kva_why_not == nullptr;const int64_t kva_split = std::min<int64_t>(std::max<int64_t>(kva.split, 1), g.n_layers);const int64_t kva_frontier = std::max<int64_t>(0, n - kva_suffix);@@ -5312,9 +5319,8 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri}} else if (kva.on && (kva.explicit_on || (n >= kva.min_prompt && !kva_routine && kva_eng.blocked.empty()))) {// KVA-8 (observability): a run that was configured for KVA but fell back completely must say so and why,- // or an operator cannot tell it apart from a run where the feature was never asked for. With KVA on by- // default only a long prompt that was turned away for an unusual reason is worth a line (STRATA_KVA=1 asks- // for every reason).+ // or an operator cannot tell it apart from a run where the feature was never asked for. (KVA only runs+ // when STRATA_KVA=1 asked for it, so every reason is worth a line.)std::fprintf(stderr, "strata prefill: KVA configured but NOT applied: %s (prompt %lld positions, ""min_prompt %lld, suffix %lld, split %lld)\n",kva_why_not != nullptr ? kva_why_not : "not applicable", (long long) n, (long long) kva.min_prompt,diff --git a/src/program/generate.cpp b/src/program/generate.cppindex dbc5b6b02..a718c9cfd 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -5509,7 +5509,7 @@ int main(int argc, char** argv) {// slots' draft-KV copies use this one toostatic const strata::core::ModelGeometry draft_geometry{};std::vector<std::unique_ptr<strata::core::MtpDrafter>> slot_mtp;- // KVA (0.2.7: on by default, STRATA_KVA=0 is the exact 0.2.6 path): what only the engine knows. A layer split and+ // KVA (0.2.7: opt-in with STRATA_KVA=1; unset is the exact 0.2.6 path): what only the engine knows. A layer split and// the batch slots read their prompts in pieces and in stages, which the one-device, one-cold-read rule of the// approximate prompt read cannot follow, so they stay on the exact path (and say so here, once). The draft layer// (MTP) keeps its window, whatever it is: a session KVA approximated tells it the first cell built from a real row