Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

KVA: the draft layer is untouched by default (window 32768, cells built for approximated chunks, no floor); the cell floor is STRATA_KVA_DRAFT_FLOOR=1

Niko12216c00851ac143
diff --git a/docs/KVA.md b/docs/KVA.md
index c53c6abde..5274bd3e8 100644
--- a/docs/KVA.md
+++ b/docs/KVA.md
@@ -30,26 +30,20 @@ configuration rules KVA out for the whole run.
The last `suffix` positions (2048, or `STRATA_KVA_SUFFIX`) are always read exactly.
-The MTP draft layer builds its K/V from the final residuals of the prompt, and an approximated chunk has none. The
-draft layer's window is **not changed**: it stays what it was (`--mtp-window`, default 32768 cells). Instead the
-engine keeps a *cell floor* per session, the first draft cell built from a real row:
-
-- A read that KVA approximated raises the floor to the end of its approximated chunks. The draft layer builds no
- cell below it and its attention starts no lower, so it reads the window `[max(floor, pos - window), pos)`: the
- exactly computed tail, growing with the conversation, never an approximated cell. The target model verifies
- every token either way.
-- The floor follows the *tokens* of the state a request continues from (the live session, a conversation checkpoint,
- a parked conversation): the engine remembers which token prefixes a KVA read approximated, and a state's floor
- is how much of its tokens agree with one of them. A state no KVA read produced (a short chat, another document,
- every prompt of a `STRATA_KVA=0` run) has floor 0.
-- With floor 0 the draft layer attends to exactly the window it attended to with `STRATA_KVA=0`
- (`window_start`, unit tested for every length). The only added work is one tiny kernel that picks the cells.
- With `STRATA_KVA=0`, with the engine configurations that rule KVA out and with a build without a draft layer,
- the draft layer's code is 0.2.6's own.
-
-Measured on the P100 (14K prompt, 256 tokens, auto cache and prefill), the draft layer's window made no difference
-to decode (38.7 tok/s with the 32768-cell window, 40.1 with 2048); on faster cards a smaller window costs
-acceptance, which is why the window is left alone.
+The MTP draft layer builds its K/V from the final residuals of the prompt. An approximated chunk has no final
+residual of its own, so its draft cells are built from the rows that chunk does have. By default nothing about the
+draft layer changes: its window is whatever it was (`--mtp-window`, default 32768 cells), it builds the same cells and
+attends to the same cells as with `STRATA_KVA=0`. The target model verifies every token, so the draft only decides
+how many tokens a verify window keeps (decode speed), never the answer.
+
+Measured on the P100 (IQ3_XXS, auto cache and prefill, 14K and 30K prompts, 256 tokens), the draft layer attends to
+those cells with acceptance within noise of the exact read. A forced small window (the first default-on build set
+`--mtp-window 2048` for every session) and the research option below both cost acceptance on faster cards.
+
+`STRATA_KVA_DRAFT_FLOOR=1` (research): the draft layer gets a *cell floor* per session, the first cell built from a real
+row. It builds no draft cell below it and attends to `[max(floor, pos - window), pos)`, the exact tail growing with
+the conversation. The floor follows the tokens of the state a request continues from (`kva_floor.hpp`); a state no KVA
+read produced has floor 0. It is off by default because it costs draft acceptance after long prompts.
## Chats, follow-up turns and checkpoints (`--serve`)
@@ -86,6 +80,7 @@ there, with a line when it is set).
| `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |
| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; see above |
| `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path |
+| `STRATA_KVA_DRAFT_FLOOR` | off | `1`: the draft layer skips the draft cells of approximated chunks (research) |
| `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type |
| `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) |
| `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration (the read is exact while capturing) |
diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hpp
index 80e8bdb0e..874566d23 100644
--- a/include/strata/prefill/kva.hpp
+++ b/include/strata/prefill/kva.hpp
@@ -22,11 +22,12 @@
// whole process (batch slots, a layer split, a peer device).
//
// The draft layer (MTP) builds its K/V from the final rows of the prompt, which an approximated chunk does not have.
-// Its window is NOT changed: the draft layer's attention starts at the "cell floor" (MtpDrafter::set_cell_floor), the
-// first cell built from a real row, and the cells below it are neither built nor read. A session KVA did not
-// approximate has a floor of 0 and is bit for bit what STRATA_KVA=0 runs (kva_floor.hpp keeps the floor with the
-// tokens of the session, window_start.hpp is the one rule the kernel and the tests share). The target model verifies
-// every token whatever the draft read.
+// Its window is NOT changed, and by default neither is anything else: the draft cells of an approximated chunk are
+// built from the rows that chunk has and the draft layer attends to them (measured: draft acceptance after a KVA
+// read stays within noise on the P100). The research option STRATA_KVA_DRAFT_FLOOR=1 gives the draft layer a "cell
+// floor" per session (MtpDrafter::set_cell_floor; kva_floor.hpp keeps it with the tokens of the session,
+// window_start.hpp is the one rule the kernel and the tests share): the first cell built from a real row, below
+// which no draft cell is built or read. The target model verifies every token whatever the draft read.
#pragma once
#include <algorithm>
@@ -68,6 +69,7 @@ struct KvaConfig {
bool capture_fp32 = false; ///< STRATA_KVA_CAPTURE_FP32=1: fp32 records (else bf16; fp32 is for parity runs)
bool oracle_fp32 = false; ///< STRATA_KVA_ORACLE_FP32=1: the oracle files are fp32 (else bf16)
bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full path
+ bool draft_floor = false; ///< STRATA_KVA_DRAFT_FLOOR=1: the draft layer skips the cells of approximated chunks (research; default: it reads them)
int64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)
int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: kKvaSuffixDefault)
int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full path
@@ -83,7 +85,7 @@ struct KvaConfig {
/// Does the draft layer need the cell floor at all? Only when a read can approximate something: KVA on, not ruled
/// out for the process, and a draft layer loaded. Otherwise the draft layer's attention is 0.2.6's own code.
bool wants_cell_floor(const KvaEngine& e, bool has_drafter) const {
- return kKvaBackend && on && e.blocked.empty() && has_drafter;
+ return kKvaBackend && on && draft_floor && e.blocked.empty() && has_drafter;
}
/// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a
@@ -143,6 +145,7 @@ inline KvaConfig kva_config_from_env() {
k.suffix_env = std::max<int64_t>(0, kva_env_i64("STRATA_KVA_SUFFIX", 0));
k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);
k.parity = kva_env_flag("STRATA_KVA_PARITY", false);
+ k.draft_floor = kva_env_flag("STRATA_KVA_DRAFT_FLOOR", false);
if (const char* d = std::getenv("STRATA_KVA_CAPTURE"); d != nullptr && *d != '\0') {
k.capture = true;
k.capture_dir = d;
diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cpp
index 937d0ca14..be6f2ef85 100644
--- a/src/prefill/kva_test.cpp
+++ b/src/prefill/kva_test.cpp
@@ -31,7 +31,7 @@ static void put(const char* k, const char* v) {
static void clear_all() {
for (const char* k : {"STRATA_KVA", "STRATA_KVA_SPLIT", "STRATA_KVA_SUFFIX", "STRATA_KVA_MIN_PROMPT", "STRATA_KVA_PARITY",
"STRATA_KVA_CAPTURE", "STRATA_KVA_CAPTURE_FP32", "STRATA_KVA_ORACLE", "STRATA_KVA_ORACLE_FP32",
- "STRATA_KVA_PROJECTOR", "STRATA_KVA_MODE", "STRATA_KVA_ONLY"})
+ "STRATA_KVA_PROJECTOR", "STRATA_KVA_MODE", "STRATA_KVA_ONLY", "STRATA_KVA_DRAFT_FLOOR"})
unset(k);
}
static bool is(const char* got, const char* want) { return got != nullptr && std::strcmp(got, want) == 0; }
@@ -118,6 +118,65 @@ int main() {
}
clear_all();
+ // ---- "not applied => identical to STRATA_KVA=0", the parts that need no GPU
+ // (1) by default the draft layer is 0.2.6's own code; the cell floor is a research option, wired only when asked
+ {
+ KvaEngine eng;
+ const KvaConfig on = kva_config_from_env();
+ CHECK(!on.draft_floor && !on.wants_cell_floor(eng, true));
+ put("STRATA_KVA_DRAFT_FLOOR", "1");
+ const KvaConfig fl = kva_config_from_env();
+ CHECK(fl.draft_floor && fl.wants_cell_floor(eng, true));
+ CHECK(!fl.wants_cell_floor(eng, false)); // no draft layer: nothing to wire
+ KvaEngine blocked;
+ blocked.blocked = "batch slots (--batch)";
+ CHECK(!fl.wants_cell_floor(blocked, true)); // ruled out for the process: 0.2.6's own attention code
+ put("STRATA_KVA", "0");
+ const KvaConfig off = kva_config_from_env();
+ CHECK(!off.wants_cell_floor(eng, true)); // STRATA_KVA=0: untouched
+ clear_all();
+ }
+ // (2) with a floor of 0 the window is exactly the plain window, for every length and every window
+ {
+ using strata::kernels::window_start;
+ bool same = true;
+ for (int window : {1, 2, 64, 2048, 32768, 0x7fffffff})
+ for (int n_kv : {0, 1, 2, 63, 64, 65, 2047, 2048, 2049, 4096, 14000, 32768, 32769, 100000}) {
+ const int plain = n_kv > window ? n_kv - window : 0; // what window_ids always selected
+ if (window_start(n_kv, window, 0) != plain) same = false;
+ }
+ CHECK(same);
+ CHECK(window_start(14000, 32768, 11264) == 11264); // a floor inside the window raises the start
+ CHECK(window_start(14000, 2048, 11264) == 11952); // a floor below the window changes nothing
+ CHECK(window_start(14000, 32768, 20000) == 13999); // a floor past the end keeps one cell
+ CHECK(window_start(14000, 32768, 14000) == 13999);
+ CHECK(window_start(1, 32768, 5) == 0);
+ CHECK(window_start(0, 32768, 5) == 0);
+ }
+ // (3) the floor follows the session's tokens: a state no KVA read produced has floor 0
+ {
+ KvaFloorRegistry reg;
+ std::vector<int64_t> doc;
+ for (int i = 0; i < 20000; ++i) doc.push_back(1000 + (i * 7919) % 5000);
+ CHECK(reg.floor_for(doc.data(), (int64_t) doc.size()) == 0); // nothing remembered
+ reg.note(doc.data(), 11264); // a KVA read approximated [0, 11264)
+ CHECK(reg.floor_for(doc.data(), 20000) == 11264); // the live session / a later turn of it
+ CHECK(reg.floor_for(doc.data(), 5000) == 5000); // a checkpoint inside the approximated part
+ std::vector<int64_t> other(doc);
+ for (int i = 0; i < 20000; ++i) other[(size_t) i] += 1; // a different prompt entirely
+ CHECK(reg.floor_for(other.data(), 20000) == 0); // short chats, other documents: floor 0
+ std::vector<int64_t> shared(doc.begin(), doc.begin() + 3000); // a new chat sharing the first 3000 tokens
+ for (int i = 3000; i < 9000; ++i) shared.push_back(900000 + i);
+ CHECK(reg.floor_for(shared.data(), 9000) == 3000);
+ reg.note(doc.data(), 11264); // the same read again: no second entry
+ CHECK(reg.size() == 1);
+ KvaFloorRegistry small(2);
+ for (int k = 0; k < 5; ++k) { std::vector<int64_t> d(100, k); small.note(d.data(), 100); }
+ CHECK(small.size() == 2); // bounded
+ reg.clear();
+ CHECK(reg.floor_for(doc.data(), 20000) == 0);
+ }
+
if (fails == 0) std::printf("kva_test: ok\n");
return fails == 0 ? 0 : 1;
}