Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
KVA: the draft layer is untouched by default (window 32768, cells built for approximated chunks, no floor); the cell floor is STRATA_KVA_DRAFT_FLOOR=1
Niko12216c00851ac143
diff --git a/docs/KVA.md b/docs/KVA.mdindex c53c6abde..5274bd3e8 100644--- a/docs/KVA.md+++ b/docs/KVA.md@@ -30,26 +30,20 @@ configuration rules KVA out for the whole run.The last `suffix` positions (2048, or `STRATA_KVA_SUFFIX`) are always read exactly.-The MTP draft layer builds its K/V from the final residuals of the prompt, and an approximated chunk has none. The-draft layer's window is **not changed**: it stays what it was (`--mtp-window`, default 32768 cells). Instead the-engine keeps a *cell floor* per session, the first draft cell built from a real row:--- A read that KVA approximated raises the floor to the end of its approximated chunks. The draft layer builds no- cell below it and its attention starts no lower, so it reads the window `[max(floor, pos - window), pos)`: the- exactly computed tail, growing with the conversation, never an approximated cell. The target model verifies- every token either way.-- The floor follows the *tokens* of the state a request continues from (the live session, a conversation checkpoint,- a parked conversation): the engine remembers which token prefixes a KVA read approximated, and a state's floor- is how much of its tokens agree with one of them. A state no KVA read produced (a short chat, another document,- every prompt of a `STRATA_KVA=0` run) has floor 0.-- With floor 0 the draft layer attends to exactly the window it attended to with `STRATA_KVA=0`- (`window_start`, unit tested for every length). The only added work is one tiny kernel that picks the cells.- With `STRATA_KVA=0`, with the engine configurations that rule KVA out and with a build without a draft layer,- the draft layer's code is 0.2.6's own.--Measured on the P100 (14K prompt, 256 tokens, auto cache and prefill), the draft layer's window made no difference-to decode (38.7 tok/s with the 32768-cell window, 40.1 with 2048); on faster cards a smaller window costs-acceptance, which is why the window is left alone.+The MTP draft layer builds its K/V from the final residuals of the prompt. An approximated chunk has no final+residual of its own, so its draft cells are built from the rows that chunk does have. By default nothing about the+draft layer changes: its window is whatever it was (`--mtp-window`, default 32768 cells), it builds the same cells and+attends to the same cells as with `STRATA_KVA=0`. The target model verifies every token, so the draft only decides+how many tokens a verify window keeps (decode speed), never the answer.++Measured on the P100 (IQ3_XXS, auto cache and prefill, 14K and 30K prompts, 256 tokens), the draft layer attends to+those cells with acceptance within noise of the exact read. A forced small window (the first default-on build set+`--mtp-window 2048` for every session) and the research option below both cost acceptance on faster cards.++`STRATA_KVA_DRAFT_FLOOR=1` (research): the draft layer gets a *cell floor* per session, the first cell built from a real+row. It builds no draft cell below it and attends to `[max(floor, pos - window), pos)`, the exact tail growing with+the conversation. The floor follows the tokens of the state a request continues from (`kva_floor.hpp`); a state no KVA+read produced has floor 0. It is off by default because it costs draft acceptance after long prompts.## Chats, follow-up turns and checkpoints (`--serve`)@@ -86,6 +80,7 @@ there, with a line when it is set).| `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) || `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; see above || `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path |+| `STRATA_KVA_DRAFT_FLOOR` | off | `1`: the draft layer skips the draft cells of approximated chunks (research) || `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type || `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) || `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration (the read is exact while capturing) |diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hppindex 80e8bdb0e..874566d23 100644--- a/include/strata/prefill/kva.hpp+++ b/include/strata/prefill/kva.hpp@@ -22,11 +22,12 @@// whole process (batch slots, a layer split, a peer device).//// The draft layer (MTP) builds its K/V from the final rows of the prompt, which an approximated chunk does not have.-// Its window is NOT changed: the draft layer's attention starts at the "cell floor" (MtpDrafter::set_cell_floor), the-// first cell built from a real row, and the cells below it are neither built nor read. A session KVA did not-// approximate has a floor of 0 and is bit for bit what STRATA_KVA=0 runs (kva_floor.hpp keeps the floor with the-// tokens of the session, window_start.hpp is the one rule the kernel and the tests share). The target model verifies-// every token whatever the draft read.+// Its window is NOT changed, and by default neither is anything else: the draft cells of an approximated chunk are+// built from the rows that chunk has and the draft layer attends to them (measured: draft acceptance after a KVA+// read stays within noise on the P100). The research option STRATA_KVA_DRAFT_FLOOR=1 gives the draft layer a "cell+// floor" per session (MtpDrafter::set_cell_floor; kva_floor.hpp keeps it with the tokens of the session,+// window_start.hpp is the one rule the kernel and the tests share): the first cell built from a real row, below+// which no draft cell is built or read. The target model verifies every token whatever the draft read.#pragma once#include <algorithm>@@ -68,6 +69,7 @@ struct KvaConfig {bool capture_fp32 = false; ///< STRATA_KVA_CAPTURE_FP32=1: fp32 records (else bf16; fp32 is for parity runs)bool oracle_fp32 = false; ///< STRATA_KVA_ORACLE_FP32=1: the oracle files are fp32 (else bf16)bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full path+ bool draft_floor = false; ///< STRATA_KVA_DRAFT_FLOOR=1: the draft layer skips the cells of approximated chunks (research; default: it reads them)int64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: kKvaSuffixDefault)int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full path@@ -83,7 +85,7 @@ struct KvaConfig {/// Does the draft layer need the cell floor at all? Only when a read can approximate something: KVA on, not ruled/// out for the process, and a draft layer loaded. Otherwise the draft layer's attention is 0.2.6's own code.bool wants_cell_floor(const KvaEngine& e, bool has_drafter) const {- return kKvaBackend && on && e.blocked.empty() && has_drafter;+ return kKvaBackend && on && draft_floor && e.blocked.empty() && has_drafter;}/// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a@@ -143,6 +145,7 @@ inline KvaConfig kva_config_from_env() {k.suffix_env = std::max<int64_t>(0, kva_env_i64("STRATA_KVA_SUFFIX", 0));k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);k.parity = kva_env_flag("STRATA_KVA_PARITY", false);+ k.draft_floor = kva_env_flag("STRATA_KVA_DRAFT_FLOOR", false);if (const char* d = std::getenv("STRATA_KVA_CAPTURE"); d != nullptr && *d != '\0') {k.capture = true;k.capture_dir = d;diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cppindex 937d0ca14..be6f2ef85 100644--- a/src/prefill/kva_test.cpp+++ b/src/prefill/kva_test.cpp@@ -31,7 +31,7 @@ static void put(const char* k, const char* v) {static void clear_all() {for (const char* k : {"STRATA_KVA", "STRATA_KVA_SPLIT", "STRATA_KVA_SUFFIX", "STRATA_KVA_MIN_PROMPT", "STRATA_KVA_PARITY","STRATA_KVA_CAPTURE", "STRATA_KVA_CAPTURE_FP32", "STRATA_KVA_ORACLE", "STRATA_KVA_ORACLE_FP32",- "STRATA_KVA_PROJECTOR", "STRATA_KVA_MODE", "STRATA_KVA_ONLY"})+ "STRATA_KVA_PROJECTOR", "STRATA_KVA_MODE", "STRATA_KVA_ONLY", "STRATA_KVA_DRAFT_FLOOR"})unset(k);}static bool is(const char* got, const char* want) { return got != nullptr && std::strcmp(got, want) == 0; }@@ -118,6 +118,65 @@ int main() {}clear_all();+ // ---- "not applied => identical to STRATA_KVA=0", the parts that need no GPU+ // (1) by default the draft layer is 0.2.6's own code; the cell floor is a research option, wired only when asked+ {+ KvaEngine eng;+ const KvaConfig on = kva_config_from_env();+ CHECK(!on.draft_floor && !on.wants_cell_floor(eng, true));+ put("STRATA_KVA_DRAFT_FLOOR", "1");+ const KvaConfig fl = kva_config_from_env();+ CHECK(fl.draft_floor && fl.wants_cell_floor(eng, true));+ CHECK(!fl.wants_cell_floor(eng, false)); // no draft layer: nothing to wire+ KvaEngine blocked;+ blocked.blocked = "batch slots (--batch)";+ CHECK(!fl.wants_cell_floor(blocked, true)); // ruled out for the process: 0.2.6's own attention code+ put("STRATA_KVA", "0");+ const KvaConfig off = kva_config_from_env();+ CHECK(!off.wants_cell_floor(eng, true)); // STRATA_KVA=0: untouched+ clear_all();+ }+ // (2) with a floor of 0 the window is exactly the plain window, for every length and every window+ {+ using strata::kernels::window_start;+ bool same = true;+ for (int window : {1, 2, 64, 2048, 32768, 0x7fffffff})+ for (int n_kv : {0, 1, 2, 63, 64, 65, 2047, 2048, 2049, 4096, 14000, 32768, 32769, 100000}) {+ const int plain = n_kv > window ? n_kv - window : 0; // what window_ids always selected+ if (window_start(n_kv, window, 0) != plain) same = false;+ }+ CHECK(same);+ CHECK(window_start(14000, 32768, 11264) == 11264); // a floor inside the window raises the start+ CHECK(window_start(14000, 2048, 11264) == 11952); // a floor below the window changes nothing+ CHECK(window_start(14000, 32768, 20000) == 13999); // a floor past the end keeps one cell+ CHECK(window_start(14000, 32768, 14000) == 13999);+ CHECK(window_start(1, 32768, 5) == 0);+ CHECK(window_start(0, 32768, 5) == 0);+ }+ // (3) the floor follows the session's tokens: a state no KVA read produced has floor 0+ {+ KvaFloorRegistry reg;+ std::vector<int64_t> doc;+ for (int i = 0; i < 20000; ++i) doc.push_back(1000 + (i * 7919) % 5000);+ CHECK(reg.floor_for(doc.data(), (int64_t) doc.size()) == 0); // nothing remembered+ reg.note(doc.data(), 11264); // a KVA read approximated [0, 11264)+ CHECK(reg.floor_for(doc.data(), 20000) == 11264); // the live session / a later turn of it+ CHECK(reg.floor_for(doc.data(), 5000) == 5000); // a checkpoint inside the approximated part+ std::vector<int64_t> other(doc);+ for (int i = 0; i < 20000; ++i) other[(size_t) i] += 1; // a different prompt entirely+ CHECK(reg.floor_for(other.data(), 20000) == 0); // short chats, other documents: floor 0+ std::vector<int64_t> shared(doc.begin(), doc.begin() + 3000); // a new chat sharing the first 3000 tokens+ for (int i = 3000; i < 9000; ++i) shared.push_back(900000 + i);+ CHECK(reg.floor_for(shared.data(), 9000) == 3000);+ reg.note(doc.data(), 11264); // the same read again: no second entry+ CHECK(reg.size() == 1);+ KvaFloorRegistry small(2);+ for (int k = 0; k < 5; ++k) { std::vector<int64_t> d(100, k); small.note(d.data(), 100); }+ CHECK(small.size() == 2); // bounded+ reg.clear();+ CHECK(reg.floor_for(doc.data(), 20000) == 0);+ }+if (fails == 0) std::printf("kva_test: ok\n");return fails == 0 ? 0 : 1;}