Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
KVA on by default for a cold prompt of 4096+ tokens (Qwen3.8 on one CUDA/HIP device); STRATA_KVA=0 gives the exact 0.2.6 prompt read
Niko1221750ca8c3b16f
diff --git a/CMakeLists.txt b/CMakeLists.txtindex c4e6e8a18..7180ed0fa 100644--- a/CMakeLists.txt+++ b/CMakeLists.txt@@ -1544,6 +1544,9 @@ if(STRATA_BUILD_TESTS)add_executable(arch_defaults_test src/core/arch_defaults_test.cpp src/core/arch_defaults.cpp)target_include_directories(arch_defaults_test PRIVATE include)add_test(NAME arch_defaults_test COMMAND arch_defaults_test)+ add_executable(kva_test src/prefill/kva_test.cpp) # KVA: the switch, the automatic tail, the reasons a read is turned away+ target_include_directories(kva_test PRIVATE include)+ add_test(NAME kva_test COMMAND kva_test)add_executable(adapt_async_default_test src/core/adapt_async_default_test.cpp)target_include_directories(adapt_async_default_test PRIVATE include)add_test(NAME adapt_async_default_test COMMAND adapt_async_default_test)diff --git a/docs/KVA.md b/docs/KVA.mdindex 556f71974..d07651f3c 100644--- a/docs/KVA.md+++ b/docs/KVA.md@@ -1,7 +1,4 @@-# KVA: approximate prompt processing for the late layers (opt-in, research)--This is an **experimental, opt-in** feature. Nothing in it runs unless `STRATA_KVA=1` is set, and a build without-that variable behaves exactly as before.+# KVA: the faster prompt read for long cold prompts (on by default since 0.2.7)KVA ("approximate late-layer state") reads a long prompt exactly through layer 23, then for the late layers24–47 builds **only their recurrent state** — the GDN recurrence and convolution history, the QSA K/V pages and@@ -10,41 +7,91 @@ the indexer keys — instead of running the whole layer. The attention output, tresults. The last positions of the prompt are then read fully, exactly as before, and decode runs the ordinaryfull path against the state KVA built.-## Turning it on+It is an approximation: the answers can differ from the exact read (see "Measured" for how much). **`STRATA_KVA=0`+turns it off and gives the exact 0.2.6 prompt path.** The Strata app shows this as the "Faster long prompts" switch+in Expert settings; it sends `STRATA_KVA=0` when the switch is off.-```sh-STRATA_KVA=1 \-STRATA_KVA_SUFFIX=2112 \-STRATA_KVA_MIN_PROMPT=4096 \- strata --serve ... --prefill 1024 --mtp-window 2048-```+## When it runs++KVA runs when all of these hold, and says so in the engine log:++- Qwen3.8-Flash-Next on CUDA or HIP (not SYCL, not Metal; Qwen3.6 and the other models read exactly);+- one GPU, no `--batch` slots, no `--peer-device`, no DFlash2 drafter;+- the read is cold (it starts at position 0 of the session) and the text has no pictures;+- the prompt (or the part of it read in one go, see below) has at least `STRATA_KVA_MIN_PROMPT` tokens (4096) and+ more than the fully computed tail plus 64.++Otherwise the prompt is read exactly. An engine that was started with `STRATA_KVA=1` logs the reason for every+read it turned away (`strata prefill: KVA configured but NOT applied: <reason>`); with the default it logs only+a long prompt turned away for an unusual reason (a layer split, pictures), and once at start-up when the engine+configuration rules KVA out for the whole run.++## The fully computed tail and the draft layer++The last `suffix` positions are always read exactly. The MTP draft layer builds its K/V from the final+residuals of the prompt, and those rows are only real where the layers ran in full, so the tail must cover the+draft layer's window plus the 64 cells its catch-up reaches back:++- no `--mtp-window` given (what the app and the Terminal write): with KVA on, the engine sets the draft window to+ 2048 cells and the tail to 2112 tokens. The 0.2.6 window was 32768, which is every cell of a chat up to that+ length and could never be covered. The engine prints one line at start-up when it does this.+- `--mtp-window N` given: the window stays N and the tail is N + 64 (a 32768-cell window never leaves a+ prompt shorter than 32 K approximated; the log says "the draft layer attends to every cell" when no tail can cover it).+- `STRATA_KVA_SUFFIX=N` fixes the tail by hand.+- no draft layer (`--mtp` not given): the tail is 2048.++A draft layer that reads an approximated cell costs only draft acceptance (speed); the target model verifies every+emitted token, so the answer does not depend on it.++## Chats, follow-up turns and checkpoints (`--serve`)++The server reads a prompt in parts: up to the end of a long system prompt (the shared checkpoint), up to the last+turn header, then the new turn's header and the last token. Each part that starts at position 0 and is long enough+can take the approximate path; its own last `suffix` positions are exact, and every later part starts at a position+greater than 0 and is read exactly.++- **Turn 2 and later.** The server continues from the live session or from a conversation checkpoint (the state+ after the earlier turns) and reads only the new tokens, at positions greater than 0, so they are read in full. They+ are read exactly *on top of* the cells the first read left: nothing is approximated twice. Questions about the+ old document in turn 5 are answered from the same cells turn 1 had, so their quality is that of the first read.+- **Checkpoints and parked conversations** hold whatever cells the session held, approximated or not, and give+ them back as they were. A checkpoint taken inside an approximated stretch is a consistent state: the read+ continues from it exactly. Restoring a checkpoint and re-reading the same prompt from 0 (the `STRATA_CKPT_REREAD`+ debug check) differ with KVA on, as they would between any two reads that take different paths.+- **A cold read again** (the cache evicted the conversation, a new chat with the same system prompt) approximates+ its own prefix again.+- **`STRATA_KVA=0`** restores 0.2.6: every prompt is read exactly, the draft window is 32768 cells, nothing else+ changes.++## Layer splits, batch slots, SYCL, Metal++These never take the approximate path and say why: a layer split reports "a layer split (not the whole trunk)", the+batch slots and a peer device switch KVA off for the process (one line at start-up with `STRATA_KVA=1`), the SYCL+engine has its own prompt path without KVA, and the Metal build has no state-only kernels (`STRATA_KVA` is ignored+there, with a line when it is set).++## Settings| variable | default | what it does ||---|---:|---|-| `STRATA_KVA` | off | `1` enables the feature for qualifying requests |+| `STRATA_KVA` | on | `0` turns KVA off (the exact 0.2.6 prompt read). `1` is the same as the default but logs every read it turned away || `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |-| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; rounded **up** to a whole chunk |+| `STRATA_KVA_SUFFIX` | 2112 with the default draft window | fully computed tail positions; see above || `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path || `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type || `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) |-| `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration |+| `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration (the read is exact while capturing) || `STRATA_KVA_CAPTURE_FP32` | off | write fp32 instead of bf16 records || `STRATA_KVA_ORACLE` | — | directory: replay a capture instead of a projector || `STRATA_KVA_GDN_NORM` | off | keep the full GDN recurrence (the A/B control for the state-only one) || `STRATA_LOGITS_DUMP` | — | debug (not KVA-specific): write every verify window's sampler logits to `path`, for KL / top-1 between two runs |-Requirements and limits for this first version:+Every read says what it did:++- applied: `strata prefill: KVA on: split 24, suffix 2112, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`+- turned away: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`-- One cold read of one prompt, one device (`pos0 == 0`, one stage). A follow-up read, a resumed prompt or a- layer split stays on the full path.-- The fully computed suffix must cover the MTP drafter's window: use `--mtp-window N` with- `STRATA_KVA_SUFFIX >= N + 64`, or the drafter reads state built from approximated positions.-- **The session state after a KVA read is approximate.** It must not be reused as an exact prefix: no- checkpoint of it, no follow-up prompt that reuses it. The engine prints a warning when this happens.-- The approximation is a quality tradeoff, not a lossless transformation. Measure before using it.-- Every run says what it did:- - applied: `strata prefill: KVA on: split 24, suffix 2112, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`- - configured but not applied: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`+A quality comparison needs the control arm to say `STRATA_KVA=0` (an unset variable is now KVA).## Measured (Tesla P100 16 GB, Qwen3.8-Flash-Next IQ3_XXS, chunk 1024, expert cache 1500, MTP window 2048)@@ -96,7 +143,7 @@ place of a projector; the replay must use the same prompt and the same chunking.## Implementation notes-- `src/prefill/kva.hpp` — the plan and the environment variables.+- `include/strata/prefill/kva.hpp` — the plan, the environment variables and the reasons a read is turned away (`src/prefill/kva_test.cpp` tests them).- `src/prefill/prefill.cpp` — the chunk plan (`kva_approx`), the capture/oracle taps, the state-only layerexits, the transfer-plan skip, and `give_back()` on a skipped MoE half: a layer that consumes no expert muststill release its stream entries, or the issuer thread fills the ring and the next fully computed layer waitsdiff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hppindex d97d26bdb..f63aea4de 100644--- a/include/strata/core/mtp.hpp+++ b/include/strata/core/mtp.hpp@@ -167,6 +167,8 @@ public:const void* tensor_q8(const char* name) const { return q8(name); }QsaState& kv_state_rw() { return st_; }int64_t first_needed() const { return (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0; }+ /// The cells the draft layer attends to (0 = every cell), as load() settled it.+ int64_t window() const { return window_; }int device() const { return device_; }bool idle(std::string& err) {if (cs_ && cudaStreamSynchronize(cs_) != cudaSuccess) { err = "mtp: its stream failed"; return false; }diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hppindex 212522a59..d9ce3dc07 100644--- a/include/strata/prefill/kva.hpp+++ b/include/strata/prefill/kva.hpp@@ -1,4 +1,4 @@-// include/strata/prefill/kva.hpp - STRATA_KVA: the optional approximate-prefill plan (research, opt-in).+// include/strata/prefill/kva.hpp - STRATA_KVA: the approximate late-layer prompt read (on by default since 0.2.7).//// After layer `split` the prompt's late layers need only their recurrent state (GDN recurrence + conv history)// and their KV/indexer caches; the attention output, the output projection, the MoE half and its expert@@ -14,27 +14,64 @@// oracle a captured real input per late layer (STRATA_KVA_ORACLE=<dir>; the plumbing/parity mode),// linear a fitted low-rank projector (STRATA_KVA_PROJECTOR=<file>; not implemented in this step).//-// Everything here is read from the environment once per process and is off unless STRATA_KVA is set, so a-// default build behaves exactly as before.+// 0.2.7: the read is ON by default for a cold prompt of STRATA_KVA_MIN_PROMPT tokens or more on one CUDA/HIP+// device. STRATA_KVA=0 brings back the exact 0.2.6 prompt path and the 0.2.6 draft window. Everything is read from+// the environment once per process.+//+// The two facts the engine tells this header at start-up (kva_engine()): whether a draft layer (MTP) is loaded and+// how many cells it attends to, and whether the engine configuration rules the read out for the whole process+// (batch slots, a layer split). The fully computed suffix must cover the drafter's window plus the 64 cells its+// catch-up reaches back (MtpDrafter::first_needed), or the drafter would build its K/V from approximated cells+// (that costs acceptance, never correctness: the target model verifies every token).#pragma once+#include <algorithm>#include <cstdint>#include <cstdlib>+#include <cstring>#include <string>namespace strata::prefill {+/// Does this build have the state-only late-layer path? CUDA and HIP: yes. The Metal engine compiles prefill.cpp+/// against its own kernel library, which has none.+#if defined(STRATA_GPU_METAL)+inline constexpr bool kKvaBackend = false;+#else+inline constexpr bool kKvaBackend = true;+#endif++/// The draft layer's window (--mtp-window) the engine picks when KVA is on, the draft layer is loaded and the user+/// gave no --mtp-window: 2048 cells (suffix 2112). The 0.2.6 default was 32768, i.e. every cell of a chat up to+/// that length, which no suffix could cover.+inline constexpr int64_t kKvaDrafterWindow = 2048;+/// The fully computed tail when no draft layer needs more.+inline constexpr int64_t kKvaSuffixDefault = 2048;++/// What the running engine knows and the environment cannot say.+struct KvaEngine {+ bool has_drafter = false; ///< an MTP draft layer is loaded (--mtp with --spec 2 or more)+ int64_t drafter_window = 0; ///< the cells it attends to; 0 = every cell+ std::string blocked; ///< non-empty: this process never reads approximately, and why+};++inline KvaEngine& kva_engine() {+ static KvaEngine e;+ return e;+}+struct KvaConfig {enum class Mode { Identity, Oracle, Linear };enum class Only { All, Gdn, Qsa };- bool on = false; ///< STRATA_KVA=1+ bool on = true; ///< on unless STRATA_KVA=0 (or this build has no path)+ bool explicit_on = false; ///< STRATA_KVA was set to something other than 0: every declined read says whybool capture = false; ///< STRATA_KVA_CAPTURE=<dir>: dump every late layer's real input rowsbool capture_fp32 = false; ///< STRATA_KVA_CAPTURE_FP32=1: fp32 records (else bf16; fp32 is for parity runs)bool oracle_fp32 = false; ///< STRATA_KVA_ORACLE_FP32=1: the oracle files are fp32 (else bf16)bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full pathint64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)- int64_t suffix = 2048; ///< STRATA_KVA_SUFFIX: fully computed tail positions (rounded up to a chunk)+ int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: effective_suffix())int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full pathMode mode = Mode::Identity; ///< STRATA_KVA_MODE=identity|oracle|linearOnly only = Only::All; ///< STRATA_KVA_ONLY=all|gdn|qsa: which late layers take the approximate path@@ -42,12 +79,48 @@ struct KvaConfig {std::string capture_dir; ///< STRATA_KVA_CAPTUREstd::string projector; ///< STRATA_KVA_PROJECTOR- /// Does this run qualify at all? A cold read of one prompt, on one stage, long enough for a suffix to- /// leave approximated positions behind it.- bool applies(int64_t pos0, int64_t n, int64_t lb, int64_t le, int64_t n_layers) const {- if (!on || capture) return false; // capture runs are ordinary full reads- if (pos0 != 0 || lb != 0 || (le >= 0 && le != n_layers)) return false;- return n >= min_prompt && n > suffix + 64;+ /// The fully computed tail of a read. STRATA_KVA_SUFFIX when set; otherwise 2048, or the draft layer's window ++ /// 64 when that is larger. (A draft layer that attends to every cell has no suffix that covers it:+ /// decline() says so.)+ int64_t effective_suffix(const KvaEngine& e) const {+ if (suffix_env > 0) return suffix_env;+ int64_t s = kKvaSuffixDefault;+ if (e.has_drafter && e.drafter_window > 0) s = std::max<int64_t>(s, e.drafter_window + 64);+ return s;+ }++ /// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a+ /// continuation, no layer split, no pictures, no DFlash2 taps), on an engine that allows it, long enough for a+ /// suffix to leave approximated positions behind it. `routine` (optional) is set when the reason is an+ /// everyday one (a continuation, a short prompt) that an engine running KVA by default does not log.+ const char* decline(const KvaEngine& e, int64_t pos0, int64_t n, int64_t lb, int64_t le, int64_t n_layers,+ bool pictures = false, bool taps = false, bool* routine = nullptr,+ bool unvalidated_model = false) const {+ if (routine) *routine = false;+ if (!on) return "off (STRATA_KVA=0)";+ if (capture) return "capture mode";+ if (!e.blocked.empty()) return e.blocked.c_str();+ // The split layer, the state-only layers and the suffix rule were measured on one geometry (Qwen3.8-Flash-Next,+ // 48 layers, 512 experts). Another model reads exactly unless STRATA_KVA=1 asks for it.+ if (unvalidated_model && !explicit_on) return "KVA was not validated for this model (STRATA_KVA=1 forces it)";+ if (pos0 != 0) {+ if (routine) *routine = true;+ return "a continuation read (pos0 != 0)";+ }+ if (lb != 0 || (le >= 0 && le != n_layers)) return "a layer split (not the whole trunk)";+ if (pictures) return "a prompt with pictures";+ if (taps) return "the DFlash2 drafter reads the late layers' hidden states";+ if (suffix_env <= 0 && e.has_drafter && e.drafter_window <= 0)+ return "the draft layer attends to every cell (set --mtp-window)";+ if (n < min_prompt) {+ if (routine) *routine = true;+ return "the prompt is shorter than STRATA_KVA_MIN_PROMPT";+ }+ if (n <= effective_suffix(e) + 64) {+ if (routine) *routine = true;+ return "the suffix leaves too few positions to approximate";+ }+ return nullptr;}};@@ -63,41 +136,48 @@ inline bool kva_env_flag(const char* name, bool dflt) {return v[0] != '0';}+/// The settings as the environment says now (uncached: the tests call it after changing the environment).+inline KvaConfig kva_config_from_env() {+ KvaConfig k;+ const char* sw = std::getenv("STRATA_KVA");+ const bool set = sw != nullptr && *sw != '\0';+ k.on = !set || sw[0] != '0';+ k.explicit_on = set && sw[0] != '0';+ if (!kKvaBackend) k.on = false;+ k.split = kva_env_i64("STRATA_KVA_SPLIT", 24);+ k.suffix_env = std::max<int64_t>(0, kva_env_i64("STRATA_KVA_SUFFIX", 0));+ k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);+ k.parity = kva_env_flag("STRATA_KVA_PARITY", false);+ if (const char* d = std::getenv("STRATA_KVA_CAPTURE"); d != nullptr && *d != '\0') {+ k.capture = true;+ k.capture_dir = d;+ k.capture_fp32 = kva_env_flag("STRATA_KVA_CAPTURE_FP32", false);+ }+ if (const char* d = std::getenv("STRATA_KVA_ORACLE"); d != nullptr && *d != '\0') {+ k.oracle_dir = d;+ k.mode = KvaConfig::Mode::Oracle;+ k.oracle_fp32 = kva_env_flag("STRATA_KVA_ORACLE_FP32", false);+ }+ if (const char* d = std::getenv("STRATA_KVA_PROJECTOR"); d != nullptr && *d != '\0') {+ k.projector = d;+ k.mode = KvaConfig::Mode::Linear;+ }+ if (const char* m = std::getenv("STRATA_KVA_MODE"); m != nullptr && *m != '\0') {+ if (std::string(m) == "oracle") k.mode = KvaConfig::Mode::Oracle;+ else if (std::string(m) == "linear") k.mode = KvaConfig::Mode::Linear;+ else k.mode = KvaConfig::Mode::Identity;+ }+ if (const char* o = std::getenv("STRATA_KVA_ONLY"); o != nullptr && *o != '\0') {+ if (std::string(o) == "gdn") k.only = KvaConfig::Only::Gdn;+ else if (std::string(o) == "qsa") k.only = KvaConfig::Only::Qsa;+ else k.only = KvaConfig::Only::All;+ }+ return k;+}+/// The process's KVA settings, parsed once (the engine's own options are parsed once too).inline const KvaConfig& kva_config() {- static const KvaConfig c = [] {- KvaConfig k;- k.on = kva_env_flag("STRATA_KVA", false);- k.split = kva_env_i64("STRATA_KVA_SPLIT", 24);- k.suffix = kva_env_i64("STRATA_KVA_SUFFIX", 2048);- k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);- k.parity = kva_env_flag("STRATA_KVA_PARITY", false);- if (const char* d = std::getenv("STRATA_KVA_CAPTURE"); d != nullptr && *d != '\0') {- k.capture = true;- k.capture_dir = d;- k.capture_fp32 = kva_env_flag("STRATA_KVA_CAPTURE_FP32", false);- }- if (const char* d = std::getenv("STRATA_KVA_ORACLE"); d != nullptr && *d != '\0') {- k.oracle_dir = d;- k.mode = KvaConfig::Mode::Oracle;- k.oracle_fp32 = kva_env_flag("STRATA_KVA_ORACLE_FP32", false);- }- if (const char* d = std::getenv("STRATA_KVA_PROJECTOR"); d != nullptr && *d != '\0') {- k.projector = d;- k.mode = KvaConfig::Mode::Linear;- }- if (const char* m = std::getenv("STRATA_KVA_MODE"); m != nullptr && *m != '\0') {- if (std::string(m) == "oracle") k.mode = KvaConfig::Mode::Oracle;- else if (std::string(m) == "linear") k.mode = KvaConfig::Mode::Linear;- else k.mode = KvaConfig::Mode::Identity;- }- if (const char* o = std::getenv("STRATA_KVA_ONLY"); o != nullptr && *o != '\0') {- if (std::string(o) == "gdn") k.only = KvaConfig::Only::Gdn;- else if (std::string(o) == "qsa") k.only = KvaConfig::Only::Qsa;- else k.only = KvaConfig::Only::All;- }- return k;- }();+ static const KvaConfig c = kva_config_from_env();return c;}diff --git a/metal/src/engine/stubs.cpp b/metal/src/engine/stubs.cppindex 78ff2891b..b6615813c 100644--- a/metal/src/engine/stubs.cpp+++ b/metal/src/engine/stubs.cpp@@ -340,6 +340,7 @@ namespace strata::prefill { void gather_rows16(const uint16_t * p0, const int32_namespace strata::prefill { void gdn_conv(float * p0, const float * p1, const float * p2, float * p3, int64_t p4, float p5, void * p6) { strata::metal_engine::missing("strata::prefill::gdn_conv"); } }namespace strata::prefill { void gdn_gates(const float * p0, const float * p1, const float * p2, float * p3, float * p4, int64_t p5, void * p6) { strata::metal_engine::missing("strata::prefill::gdn_gates"); } }namespace strata::prefill { void gdn_recurrence(float * p0, const float * p1, const float * p2, const float * p3, const float * p4, const float * p5, float p6, float * p7, uint16_t * p8, int64_t p9, void * p10, int64_t p11) { strata::metal_engine::missing("strata::prefill::gdn_recurrence"); } }+namespace strata::prefill { void gdn_recurrence_state_only(float * p0, const float * p1, const float * p2, const float * p3, float * p4, int64_t p5, void * p6) { strata::metal_engine::missing("strata::prefill::gdn_recurrence_state_only"); } }namespace strata::prefill { void gr_broadcast(const float * p0, float * p1, int64_t p2, void * p3) { strata::metal_engine::missing("strata::prefill::gr_broadcast"); } }namespace strata::prefill { void gr_mix(const float * p0, const float * p1, float * p2, uint16_t * p3, int64_t p4, void * p5, uint16_t * p6, uint16_t * p7) { strata::metal_engine::missing("strata::prefill::gr_mix"); } }namespace strata::prefill { void gr_mix_r(const float * p0, const float * p1, const float * p2, const float * p3, float * p4, uint16_t * p5, int64_t p6, void * p7, uint16_t * p8, uint16_t * p9) { strata::metal_engine::missing("strata::prefill::gr_mix_r"); } }diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cppnew file mode 100644index 000000000..bd468c34d--- /dev/null+++ b/src/prefill/kva_test.cpp@@ -0,0 +1,143 @@+// src/prefill/kva_test.cpp - the KVA switch, the automatic tail and the reasons a read is turned away (no GPU needed).+#include "strata/prefill/kva.hpp"++#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <string>++using namespace strata::prefill;++static int fails = 0;+#define CHECK(c) do { if (!(c)) { std::printf("FAIL line %d: %s\n", __LINE__, #c); ++fails; } } while (0)++static void unset(const char* k) {+#if defined(_WIN32)+ _putenv_s(k, "");+#else+ unsetenv(k);+#endif+}+static void put(const char* k, const char* v) {+#if defined(_WIN32)+ _putenv_s(k, v);+#else+ setenv(k, v, 1);+#endif+}+static void clear_all() {+ for (const char* k : {"STRATA_KVA", "STRATA_KVA_SPLIT", "STRATA_KVA_SUFFIX", "STRATA_KVA_MIN_PROMPT", "STRATA_KVA_PARITY",+ "STRATA_KVA_CAPTURE", "STRATA_KVA_CAPTURE_FP32", "STRATA_KVA_ORACLE", "STRATA_KVA_ORACLE_FP32",+ "STRATA_KVA_PROJECTOR", "STRATA_KVA_MODE", "STRATA_KVA_ONLY"})+ unset(k);+}+static bool is(const char* got, const char* want) { return got != nullptr && std::strcmp(got, want) == 0; }++int main() {+ // ---- the switch: on by default, STRATA_KVA=0 off, anything else on and chatty+ clear_all();+ {+ const KvaConfig c = kva_config_from_env();+ CHECK(c.on && !c.explicit_on);+ CHECK(c.split == 24 && c.min_prompt == 4096 && c.suffix_env == 0);+ CHECK(c.mode == KvaConfig::Mode::Identity && c.only == KvaConfig::Only::All && !c.capture);+ }+ put("STRATA_KVA", "0");+ { const KvaConfig c = kva_config_from_env(); CHECK(!c.on && !c.explicit_on); }+ put("STRATA_KVA", "1");+ { const KvaConfig c = kva_config_from_env(); CHECK(c.on && c.explicit_on); }+ put("STRATA_KVA", ""); // an empty value is "not set"+ { const KvaConfig c = kva_config_from_env(); CHECK(c.on && !c.explicit_on); }+ clear_all();++ // ---- the automatic tail: 2048, or the draft layer's window + 64 when that is larger; STRATA_KVA_SUFFIX wins+ {+ const KvaConfig c = kva_config_from_env();+ KvaEngine none; // no draft layer+ CHECK(c.effective_suffix(none) == kKvaSuffixDefault);+ KvaEngine d; // the window the engine picks (no --mtp-window given)+ d.has_drafter = true;+ d.drafter_window = kKvaDrafterWindow;+ CHECK(c.effective_suffix(d) == 2112);+ d.drafter_window = 8192; // --mtp-window 8192+ CHECK(c.effective_suffix(d) == 8256);+ d.drafter_window = 512; // a small window: the default tail still holds+ CHECK(c.effective_suffix(d) == 2048);+ d.drafter_window = 0; // every cell: no tail can cover it, the plain tail is returned+ CHECK(c.effective_suffix(d) == 2048);+ put("STRATA_KVA_SUFFIX", "3000");+ const KvaConfig e = kva_config_from_env();+ d.drafter_window = 8192;+ CHECK(e.effective_suffix(d) == 3000);+ clear_all();+ }++ // ---- who is turned away, and why (the first reason that holds is the one named)+ {+ const KvaConfig c = kva_config_from_env();+ KvaEngine eng;+ eng.has_drafter = true;+ eng.drafter_window = kKvaDrafterWindow;+ bool routine = false;+ // a cold 14.6K prompt on one device: applies+ CHECK(c.decline(eng, 0, 14600, 0, -1, 48) == nullptr);+ CHECK(c.decline(eng, 0, 14600, 0, 48, 48) == nullptr); // le == n_layers is the whole trunk too+ // a continuation (turn 2 of a chat, a resumed prompt): routine+ CHECK(is(c.decline(eng, 9000, 14600, 0, -1, 48, false, false, &routine), "a continuation read (pos0 != 0)") && routine);+ // short prompts and a prompt the tail would swallow: routine+ CHECK(is(c.decline(eng, 0, 3000, 0, -1, 48, false, false, &routine), "the prompt is shorter than STRATA_KVA_MIN_PROMPT") && routine);+ CHECK(c.decline(eng, 0, 4096, 0, -1, 48) == nullptr); // 4096 > 2112 + 64: it applies+ put("STRATA_KVA_SUFFIX", "5000");+ { const KvaConfig s = kva_config_from_env();+ CHECK(is(s.decline(eng, 0, 5000, 0, -1, 48, false, false, &routine), "the suffix leaves too few positions to approximate") && routine); }+ clear_all();+ // not routine: the user may want to know+ CHECK(is(c.decline(eng, 0, 14600, 24, -1, 48, false, false, &routine), "a layer split (not the whole trunk)") && !routine);+ CHECK(is(c.decline(eng, 0, 14600, 0, 24, 48), "a layer split (not the whole trunk)"));+ CHECK(is(c.decline(eng, 0, 14600, 0, -1, 48, true), "a prompt with pictures"));+ CHECK(is(c.decline(eng, 0, 14600, 0, -1, 48, false, true), "the DFlash2 drafter reads the late layers' hidden states"));+ // a model the split and the tail were not measured on reads exactly unless STRATA_KVA=1 asks+ CHECK(is(c.decline(eng, 0, 14600, 0, -1, 40, false, false, &routine, true),+ "KVA was not validated for this model (STRATA_KVA=1 forces it)") && !routine);+ put("STRATA_KVA", "1");+ { const KvaConfig f = kva_config_from_env(); CHECK(f.decline(eng, 0, 14600, 0, -1, 40, false, false, nullptr, true) == nullptr); }+ clear_all();+ // a draft layer that attends to every cell has no tail that covers it+ KvaEngine all = eng;+ all.drafter_window = 0;+ CHECK(is(c.decline(all, 0, 14600, 0, -1, 48), "the draft layer attends to every cell (set --mtp-window)"));+ put("STRATA_KVA_SUFFIX", "2112"); // ... unless the user fixes the tail by hand+ { const KvaConfig s = kva_config_from_env(); CHECK(s.decline(all, 0, 14600, 0, -1, 48) == nullptr); }+ clear_all();+ // no draft layer at all: nothing to cover+ KvaEngine nod;+ CHECK(c.decline(nod, 0, 14600, 0, -1, 48) == nullptr);+ // the engine rules the whole process out (batch slots, a layer split, a peer device)+ KvaEngine blocked = eng;+ blocked.blocked = "batch slots (--batch)";+ CHECK(is(c.decline(blocked, 0, 14600, 0, -1, 48, false, false, &routine), "batch slots (--batch)") && !routine);+ // off wins over everything, capture over the rest+ put("STRATA_KVA", "0");+ { const KvaConfig o = kva_config_from_env(); CHECK(is(o.decline(eng, 0, 14600, 0, -1, 48), "off (STRATA_KVA=0)")); }+ put("STRATA_KVA", "1");+ put("STRATA_KVA_CAPTURE", "/tmp/kva-cap");+ { const KvaConfig o = kva_config_from_env(); CHECK(o.capture && is(o.decline(eng, 0, 14600, 0, -1, 48), "capture mode")); }+ clear_all();+ }++ // ---- the research modes still parse+ put("STRATA_KVA_ORACLE", "/tmp/o");+ put("STRATA_KVA_ORACLE_FP32", "1");+ put("STRATA_KVA_ONLY", "qsa");+ put("STRATA_KVA_SPLIT", "30");+ put("STRATA_KVA_MIN_PROMPT", "1024");+ {+ const KvaConfig c = kva_config_from_env();+ CHECK(c.mode == KvaConfig::Mode::Oracle && c.oracle_fp32 && c.oracle_dir == "/tmp/o");+ CHECK(c.only == KvaConfig::Only::Qsa && c.split == 30 && c.min_prompt == 1024);+ }+ clear_all();++ if (fails == 0) std::printf("kva_test: ok\n");+ return fails == 0 ? 0 : 1;+}diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cppindex f0b09e558..741c3d8fb 100644--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@ -2716,9 +2716,15 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri// STRATA_KVA: the approximate-prefix plan of this run, if it qualifies (a cold read of one long prompt, one// stage, no capture). Positions [0, kva_frontier) are approximated in the layers at and after the split.const KvaConfig& kva = kva_config();- const bool kva_run = kva.applies(pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers);+ const KvaEngine& kva_eng = kva_engine();+ const int64_t kva_suffix = kva.effective_suffix(kva_eng);+ bool kva_routine = false;+ const char* kva_why_not = kva.decline(kva_eng, pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers,+ embd_rows != nullptr, !tap_layers_.empty(), &kva_routine,+ g.n_layers != 48 || g.n_expert != 512);+ const bool kva_run = kva_why_not == nullptr;const int64_t kva_split = std::min<int64_t>(std::max<int64_t>(kva.split, 1), g.n_layers);- const int64_t kva_frontier = std::max<int64_t>(0, n - kva.suffix);+ const int64_t kva_frontier = std::max<int64_t>(0, n - kva_suffix);static KvaTap kva_capture; // process-wide: one cold read at a time, so the per-layer position is enoughstatic KvaTap kva_oracle;int64_t kva_positions = 0, kva_chunks = 0; // what actually took the approximate path (diagnostics)@@ -5272,25 +5278,31 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri: kva.only == KvaConfig::Only::Qsa ? ", qsa layers only" : "";std::fprintf(stderr, "strata prefill: KVA %s: split %lld, suffix %lld, approximated %lld of %lld positions ""in %lld of %lld chunks, mode %s%s%s\n",- kva_run ? "on" : "off (capture)", (long long) kva_split, (long long) kva.suffix,+ kva_run ? "on" : "off (capture)", (long long) kva_split, (long long) kva_suffix,(long long) kva_positions, (long long) n, (long long) kva_chunks,(long long) (n > 0 && m.T > 0 ? (n + m.T - 1) / m.T : 0), mode, only,kva.capture ? ", capturing the late layers' inputs" : "");- if (kva_positions > 0)- std::fprintf(stderr, "strata prefill: KVA: the session state holds approximated cells; it is not an exact "- "prefix and must not be reused as one (no checkpoint, no follow-up reuse)\n");- } else if (kva.on) {+ if (kva_positions > 0) {+ // What a later turn does with this state (serve): it continues from the live session or a checkpoint of+ // it and reads only the new tokens, exactly, over the cells this read left; nothing is approximated twice.+ // The answer quality of those turns is that of this read. STRATA_KVA=0 reads exactly.+ static std::atomic<bool> said{false};+ if (!said.exchange(true))+ std::fprintf(stderr, "strata prefill: KVA: the late layers (%lld and up) hold approximated cells for "+ "the first %lld positions of this prompt; a follow-up turn that continues from "+ "this state reads its new tokens exactly on top of them (nothing is approximated "+ "twice); STRATA_KVA=0 reads every prompt exactly\n",+ (long long) kva_split, (long long) kva_frontier);+ }+ } else if (kva.on && (kva.explicit_on || (n >= kva.min_prompt && !kva_routine && kva_eng.blocked.empty()))) {// KVA-8 (observability): a run that was configured for KVA but fell back completely must say so and why,- // or an operator cannot tell it apart from a run where the feature was never asked for.- const char* why = pos0 != 0 ? "a continuation read (pos0 != 0)"- : (LB != 0 || (LE >= 0 && LE != g.n_layers)) ? "a layer split (not the whole trunk)"- : kva.capture ? "capture mode"- : n < kva.min_prompt ? "the prompt is shorter than STRATA_KVA_MIN_PROMPT"- : n <= kva.suffix + 64 ? "the suffix leaves too few positions to approximate"- : "not applicable";+ // or an operator cannot tell it apart from a run where the feature was never asked for. With KVA on by+ // default only a long prompt that was turned away for an unusual reason is worth a line (STRATA_KVA=1 asks+ // for every reason).std::fprintf(stderr, "strata prefill: KVA configured but NOT applied: %s (prompt %lld positions, ""min_prompt %lld, suffix %lld, split %lld)\n",- why, (long long) n, (long long) kva.min_prompt, (long long) kva.suffix, (long long) kva_split);+ kva_why_not != nullptr ? kva_why_not : "not applicable", (long long) n, (long long) kva.min_prompt,+ (long long) kva_suffix, (long long) kva_split);}if (std::getenv("STRATA_STATE_HASH_GDN") != nullptr) { // debug: the GDN states as the prompt path leaves themcudaStreamSynchronize(m.cs);diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 45aae75ac..213e3640a 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -71,6 +71,7 @@#include "strata/core/dflash2.hpp"#include "strata/core/draft_cap.hpp"#endif+#include "strata/prefill/kva.hpp"#include "strata/prefill/prefill.hpp"#if defined(STRATA_PREFILL_MMQ)#include "strata/prefill/moe_mmq.hpp"@@ -680,6 +681,7 @@ struct Options {/// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it.std::string mtp;int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell)+ bool mtp_window_given = false; ///< --mtp-window was given (KVA picks the window itself only when it was not)/// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the/// pinned arena while the CPU computes the rest (verify windows).double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs)@@ -2626,7 +2628,7 @@ int main(int argc, char** argv) {else if (a == "--spec-follow") o.spec_follow = next("--spec-follow");else if (a == "--window-hashes") o.window_hashes = next("--window-hashes");else if (a == "--mtp") o.mtp = next("--mtp");- else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window"));+ else if (a == "--mtp-window") { o.mtp_window = std::atoll(next("--mtp-window")); o.mtp_window_given = true; }else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac"));else if (a == "--adapt-every") { o.adapt_every = std::atoi(next("--adapt-every")); o.adapt_every_given = true; }else if (a == "--adapt-decay") o.adapt_decay = (float) std::atof(next("--adapt-decay"));@@ -5496,6 +5498,37 @@ int main(int argc, char** argv) {// slots' draft-KV copies use this one toostatic const strata::core::ModelGeometry draft_geometry{};std::vector<std::unique_ptr<strata::core::MtpDrafter>> slot_mtp;+ // KVA (0.2.7: on by default, STRATA_KVA=0 is the exact 0.2.6 path): what only the engine knows. A layer split and+ // the batch slots read their prompts in pieces and in stages, which the one-device, one-cold-read rule of the+ // approximate prompt read cannot follow, so they stay on the exact path (and say so here, once). The draft layer+ // (MTP) must find exact cells in its window: with no --mtp-window given the window becomes 2048 cells (the 0.2.6+ // default, 32768, is every cell of a chat up to that length, which no fully computed tail could cover) and the+ // tail 2112; a window the user gave keeps its value and sets the tail (window + 64).+ {+ const strata::prefill::KvaConfig& kc = strata::prefill::kva_config();+ strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();+ if (kc.explicit_on && !strata::prefill::kKvaBackend)+ std::fprintf(stderr, "strata generate: STRATA_KVA is set, but this build has no state-only late-layer path; "+ "every prompt is read exactly\n");+ if (kc.on) {+ if (multi_gpu) ke.blocked = "a layer split across GPUs";+ else if (o.batch > 0) ke.blocked = "batch slots (--batch)";+ else if (o.peer_device >= 0) ke.blocked = "a peer device (--peer-device)";+ if (!ke.blocked.empty()) {+ if (kc.explicit_on)+ std::fprintf(stderr, "strata generate: KVA (the approximate late-layer prompt read) is off here: %s\n",+ ke.blocked.c_str());+ } else if (!o.mtp.empty() && o.spec >= 2 && !o.mtp_window_given && kc.suffix_env <= 0) {+ if (o.mtp_window != strata::prefill::kKvaDrafterWindow)+ std::fprintf(stderr, "strata generate: KVA on (the faster prompt read for long cold prompts): the draft layer "+ "attends to the last %lld cells (--mtp-window %lld); STRATA_KVA=0 keeps the "+ "exact 0.2.6 prompt read and its %lld-cell window\n",+ (long long) strata::prefill::kKvaDrafterWindow, (long long) strata::prefill::kKvaDrafterWindow,+ (long long) o.mtp_window);+ o.mtp_window = strata::prefill::kKvaDrafterWindow;+ }+ }+ }if (!o.mtp.empty()) {if (o.spec < 2) {std::fprintf(stderr, "strata generate: --mtp is ignored without --spec T (T >= 2)\n");@@ -5515,6 +5548,11 @@ int main(int argc, char** argv) {if (o.pipeline_windows >= 2) mtp.set_force_capture(true);if (!o.mtp.empty() && !mtp.load(o.mtp, draft_geometry, last_st && !mtp_first ? last_st->ss : ss, mtp_t, err, o.mtp_window)) { std::fprintf(stderr, "strata generate: %s%s\n", err.c_str(), vram_free_note().c_str()); return 1; }mtp.set_ple_session(&ss);+ if (!o.mtp.empty()) { // KVA: the cells the draft layer attends to now (0 = every cell) set the tail that must be exact+ strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();+ ke.has_drafter = true;+ ke.drafter_window = mtp.window();+ }#if defined(STRATA_HAVE_DFLASH2)if (o.draft == "dflash2") {const char* why = o.mtp.empty() ? "it needs --mtp (the fallback) and --spec 2 or more"