Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
KVA, when switched on, approximates layers 36-47 with a 2048-token exact tail (the measured configuration); STRATA_KVA_SPLIT / STRATA_KVA_SUFFIX override
Niko1221bda300508212
diff --git a/docs/KVA.md b/docs/KVA.mdindex 96a1ca3e1..457338553 100644--- a/docs/KVA.md+++ b/docs/KVA.md@@ -1,37 +1,36 @@# KVA: a faster prompt read for long cold prompts (opt-in, off by default)-KVA ("approximate late-layer state") reads a long prompt exactly through layer 23, then for the late layers-24–47 builds **only their recurrent state** — the GDN recurrence and convolution history, the QSA K/V pages and-the indexer keys — instead of running the whole layer. The attention output, the output projection, the MoE half+KVA ("approximate late-layer state") reads a long prompt exactly through the early layers, then for the late layers+builds **only their recurrent state** — the GDN recurrence and convolution history, the QSA K/V pages and the+indexer keys — instead of running the whole layer. The attention output, the output projection, the MoE half(router, experts, shared expert) and the late layers' expert transfers are skipped, because nothing reads theirresults. The last positions of the prompt are then read fully, exactly as before, and decode runs the ordinaryfull path against the state KVA built.**It is off by default.** Unset, the engine is 0.2.6's, prompt path and draft layer included. Set `STRATA_KVA=1` toturn it on (the Strata app shows it as the "Faster long prompts" switch in Expert settings, off until the user turns-it on; the Terminal passes `STRATA_KVA=1`). When it is on, the engine uses the safest configuration measured: an-**8192-token exact tail** (`STRATA_KVA_SUFFIX` changes it).--## Why it is off--KVA is an approximation, and the 0.2.7 quality runs found where it costs answers. The runs compare KVA off and on on the-same build, greedy, with a noise run (off against off):--- **Extraction of many scattered records** (a list of 8 register entries in a 14K document, 6 such tasks at 14K and- 30K): with the 2048-token tail, 4 of 6 tasks passed against 6 of 6 exact, 249 of 270 facts found against 270.- The model silently dropped 3 of 8 records at 14K and, at 30K, started its list with an invented row- (the request's own reference id leaked into the answer). Neither exact run did either.-- **Agent loops with a ~10K-token system prompt** (handbook, tool docs, rules): the first 8192 tokens, about 84 % of- the system prompt, were approximated. Two of three runs had defects the exact runs did not: a final answer that- left out the totals, and a split-order quote with wrong arithmetic (the price of 80 units for 70, a shortfall of- 40 instead of 50).-- Needle retrieval, unanswerable questions, short summaries, code and JSON were unchanged. The speed gain is real:- about 1.3x prompt speed at 14K and 1.5x at 30K on a Tesla P100.--The 8192-token tail approximates nothing below roughly 16K tokens (the chunks are 8192 tokens, and a chunk is-approximated only when it ends before the tail) and about half of a 30K prompt. It was the only variant that passed the-same runs on the Radeon AI PRO R9700 and tied the exact arm on the RTX 5070. Its speed gain is correspondingly smaller-than the figures above, which were measured with the 2048-token tail.+it on; the Terminal passes `STRATA_KVA=1`). Switched on, it uses the configuration the 0.2.7 quality runs found+cleanest: **split 36** (only layers 36–47 of 48 are approximated) and a **2048-token exact tail**. Both can be+changed with `STRATA_KVA_SPLIT` and `STRATA_KVA_SUFFIX`.++## Why it is off, and why split 36++KVA is an approximation, and the 0.2.7 quality runs (greedy, same build, KVA off against on, with a noise run of off+against off) show where it costs answers when more layers are approximated:++- **Split 24 (the first version, layers 24–47).** On long extraction tasks (a list of scattered register entries in a+ 14K–30K document) 4 of 6 tasks passed against 6 of 6 exact, 249 of 270 facts found against 270 on a Tesla P100:+ records were silently dropped (3 of 8 at 14K) and a 30K answer began with an invented row, the request's own reference+ id. Agent loops with a ~10K-token system prompt, of which the first 8192 tokens (about 84 %) were approximated, had+ a final answer without its totals and a quote with wrong arithmetic. On the RTX 5070 (`--agent-long`, 22 tasks) the+ split-24 default passed 0 of 10 agent runs and found 556 extraction facts, by applying the handbook's rules wrongly+ in the approximated region; exact passed 4 and 7 of 10 in two runs. A tail of 8192 tokens with split 24 found 574.+- **Split 36.** On the same 22 tasks on the RTX 5070: agents 6 of 10 (exact: 4 and 7), extraction 585 of 606 facts+ (exact: 585), with prompt speed +11 % at 14K and +18 % at 30K. This is the cleanest configuration measured and the+ one `STRATA_KVA=1` selects.+- Needle retrieval, unanswerable questions, short summaries, code and JSON were unchanged in every variant.+- Split 24 is faster (about 1.3x at 14K and 1.5x at 30K on the P100); the extra layers that stay exact are what buy+ the quality.## When it runs@@ -52,7 +51,7 @@ rules KVA out for the whole run. Without `STRATA_KVA=1` it logs nothing about KV## The fully computed tail and the draft layer-The last `suffix` positions (8192, or `STRATA_KVA_SUFFIX`) are always read exactly.+The last `suffix` positions (2048, or `STRATA_KVA_SUFFIX`) are always read exactly.The MTP draft layer builds its K/V from the final residuals of the prompt. An approximated chunk has no finalresidual of its own, so its draft cells are built from the rows that chunk does have. By default nothing about the@@ -100,10 +99,10 @@ there, with a line when it is set).| variable | default | what it does ||---|---:|---|-| `STRATA_KVA` | off | `1` turns KVA on (with the 8192-token tail). Unset or `0`: the exact 0.2.6 prompt read and draft layer |+| `STRATA_KVA` | off | `1` turns KVA on (split 36, 2048-token tail). Unset or `0`: the exact 0.2.6 prompt read and draft layer || `STRATA_KVA_ANY_MODEL` | off | `1` (research): also run KVA on a model whose shape (layers, experts) it was not measured on, with a warning |-| `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |-| `STRATA_KVA_SUFFIX` | 8192 | fully computed tail positions; see above. 2048 is the fastest setting and the one the extraction and agent defects were found with |+| `STRATA_KVA_SPLIT` | 36 | first approximated layer (zero-based); 24 approximates more, is faster and lost records and agent loops |+| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; see above || `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path || `STRATA_KVA_DRAFT_FLOOR` | off | `1`: the draft layer skips the draft cells of approximated chunks (research) || `STRATA_KVA_SKIP` | state | `moe` (opt-in): the late layers of an approximated chunk run their whole attention half (GDN and QSA outputs into the residual) and skip only the MoE half and its expert transfers; the log line says `mode moe-skip`. Slower than the default state-only form, closer to the exact residual |@@ -117,12 +116,12 @@ there, with a line when it is set).Every read says what it did:-- applied (here with `STRATA_KVA_SUFFIX=2048`): `strata prefill: KVA on: split 24, suffix 2048, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`+- applied: `strata prefill: KVA on: split 36, suffix 2048, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`- turned away: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`A quality comparison needs the KVA arm to say `STRATA_KVA=1`; the control arm leaves it unset or says `STRATA_KVA=0`.-## Measured (Tesla P100 16 GB, Qwen3.8-Flash-Next IQ3_XXS, chunk 1024, expert cache 1500, MTP window 2048)+## Measured with split 24 and a 2048-token tail (Tesla P100 16 GB, Qwen3.8-Flash-Next IQ3_XXS, chunk 1024, expert cache 1500, MTP window 2048)| prompt tokens | control prompt tok/s | KVA prompt tok/s | speedup | approximated | residual error on the suffix ||---:|---:|---:|---:|---:|---:|diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hppindex c98f43528..341cd72fe 100644--- a/include/strata/prefill/kva.hpp+++ b/include/strata/prefill/kva.hpp@@ -16,9 +16,9 @@//// 0.2.7: the read is OFF unless STRATA_KVA=1 is set (the quality runs found lost records and invented rows in long// extraction and agent tasks, docs/KVA.md). Unset, nothing here runs and the engine is 0.2.6's. Switched on, it applies-// to a cold prompt of STRATA_KVA_MIN_PROMPT tokens or more on one CUDA/HIP device, with an 8192-token exact tail-// (the safest configuration measured). Everything is read from-// the environment once per process.+// to a cold prompt of STRATA_KVA_MIN_PROMPT tokens or more on one CUDA/HIP device, approximating only the layers from+// 36 on, with a 2048-token exact tail (the configuration the 0.2.7 quality runs found cleanest; STRATA_KVA_SPLIT and+// STRATA_KVA_SUFFIX override). Everything is read from the environment once per process.//// What the engine tells this header at start-up (kva_engine()): whether its configuration rules the read out for the// whole process (batch slots, a layer split, a peer device).@@ -52,7 +52,9 @@ inline constexpr bool kKvaBackend = true;inline constexpr const char* kKvaShapeReason = "this model's shape is not one KVA was measured on";/// The fully computed tail of every approximated read.-inline constexpr int64_t kKvaSuffixDefault = 8192;+inline constexpr int64_t kKvaSuffixDefault = 2048;+/// The first approximated layer when switched on: 36 of 48 (the cleanest measured; 24 lost records and agent loops).+inline constexpr int64_t kKvaSplitDefault = 36;/// What the running engine knows and the environment cannot say.struct KvaEngine {@@ -80,7 +82,7 @@ struct KvaConfig {bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full pathbool any_model = false; ///< STRATA_KVA_ANY_MODEL=1 (research): also run on a shape KVA was not measured on, with a warningbool draft_floor = false; ///< STRATA_KVA_DRAFT_FLOOR=1: the draft layer skips the cells of approximated chunks (research; default: it reads them)- int64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)+ int64_t split = kKvaSplitDefault; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: kKvaSuffixDefault)int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full pathMode mode = Mode::Identity; ///< STRATA_KVA_MODE=identity|oracle|linear@@ -90,7 +92,7 @@ struct KvaConfig {std::string capture_dir; ///< STRATA_KVA_CAPTUREstd::string projector; ///< STRATA_KVA_PROJECTOR- /// The fully computed tail of a read: STRATA_KVA_SUFFIX when set, otherwise 8192.+ /// The fully computed tail of a read: STRATA_KVA_SUFFIX when set, otherwise 2048.int64_t effective_suffix() const { return suffix_env > 0 ? suffix_env : kKvaSuffixDefault; }/// Does the draft layer need the cell floor at all? Only when a read can approximate something: KVA on, not ruled@@ -152,7 +154,7 @@ inline KvaConfig kva_config_from_env() {k.on = set && sw[0] != '0';k.explicit_on = k.on;if (!kKvaBackend) k.on = false;- k.split = kva_env_i64("STRATA_KVA_SPLIT", 24);+ k.split = kva_env_i64("STRATA_KVA_SPLIT", kKvaSplitDefault);k.suffix_env = std::max<int64_t>(0, kva_env_i64("STRATA_KVA_SUFFIX", 0));k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);k.parity = kva_env_flag("STRATA_KVA_PARITY", false);diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cppindex 9c332185c..d87bece43 100644--- a/src/prefill/kva_test.cpp+++ b/src/prefill/kva_test.cpp@@ -42,7 +42,7 @@ int main() {{const KvaConfig c = kva_config_from_env();CHECK(!c.on && !c.explicit_on); // unset: off- CHECK(c.split == 24 && c.min_prompt == 4096 && c.suffix_env == 0);+ CHECK(c.split == 36 && c.min_prompt == 4096 && c.suffix_env == 0 && kKvaSplitDefault == 36);CHECK(c.mode == KvaConfig::Mode::Identity && c.only == KvaConfig::Only::All && !c.capture);KvaEngine eng;CHECK(!c.wants_cell_floor(eng, true));@@ -57,10 +57,13 @@ int main() {clear_all();put("STRATA_KVA", "1"); // everything below runs with KVA switched on- // ---- the tail: 8192 when switched on (the safest configuration measured), or STRATA_KVA_SUFFIX+ // ---- switched on: split 36 and a 2048-token tail (the configuration measured), or STRATA_KVA_SPLIT / STRATA_KVA_SUFFIX{const KvaConfig c = kva_config_from_env();- CHECK(c.on && c.effective_suffix() == kKvaSuffixDefault && kKvaSuffixDefault == 8192);+ CHECK(c.on && c.split == 36 && c.effective_suffix() == kKvaSuffixDefault && kKvaSuffixDefault == 2048);+ put("STRATA_KVA_SPLIT", "30");+ CHECK(kva_config_from_env().split == 30);+ unset("STRATA_KVA_SPLIT");put("STRATA_KVA_SUFFIX", "3000");const KvaConfig e = kva_config_from_env();CHECK(e.effective_suffix() == 3000);@@ -72,17 +75,18 @@ int main() {const KvaConfig c = kva_config_from_env();KvaEngine eng;bool routine = false;- // a cold 14.6K prompt on one device: applies (14600 > 8192 + 64)+ // a cold 14.6K prompt on one device: applies (14600 > 2048 + 64)CHECK(c.decline(eng, 0, 14600, 0, -1, 48) == nullptr);CHECK(c.decline(eng, 0, 14600, 0, 48, 48) == nullptr); // le == n_layers is the whole trunk too// a continuation (turn 2 of a chat, a resumed prompt): routineCHECK(is(c.decline(eng, 9000, 14600, 0, -1, 48, false, false, &routine), "a continuation read (pos0 != 0)") && routine);- // short prompts and a prompt the 8192-token tail would swallow: routine+ // short prompts and a prompt the tail would swallow: routineCHECK(is(c.decline(eng, 0, 3000, 0, -1, 48, false, false, &routine), "the prompt is shorter than STRATA_KVA_MIN_PROMPT") && routine);- CHECK(is(c.decline(eng, 0, 8256, 0, -1, 48, false, false, &routine), "the suffix leaves too few positions to approximate") && routine);- CHECK(c.decline(eng, 0, 8257, 0, -1, 48) == nullptr);- put("STRATA_KVA_SUFFIX", "2048");- { const KvaConfig s = kva_config_from_env(); CHECK(s.decline(eng, 0, 4096, 0, -1, 48) == nullptr); }+ CHECK(c.decline(eng, 0, 4096, 0, -1, 48) == nullptr); // 4096 is the first prompt that is long enough+ put("STRATA_KVA_SUFFIX", "5000"); // a longer tail swallows a prompt of 5000 and not one of 5065+ { const KvaConfig s = kva_config_from_env();+ CHECK(is(s.decline(eng, 0, 5000, 0, -1, 48, false, false, &routine), "the suffix leaves too few positions to approximate") && routine);+ CHECK(s.decline(eng, 0, 5065, 0, -1, 48) == nullptr); }unset("STRATA_KVA_SUFFIX");// not routine: the user may want to knowCHECK(is(c.decline(eng, 0, 14600, 24, -1, 48, false, false, &routine), "a layer split (not the whole trunk)") && !routine);