Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

KVA on by default for a cold prompt of 4096+ tokens (Qwen3.8 on one CUDA/HIP device); STRATA_KVA=0 gives the exact 0.2.6 prompt read

Niko1221750ca8c3b16f
diff --git a/CMakeLists.txt b/CMakeLists.txt
index c4e6e8a18..7180ed0fa 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1544,6 +1544,9 @@ if(STRATA_BUILD_TESTS)
add_executable(arch_defaults_test src/core/arch_defaults_test.cpp src/core/arch_defaults.cpp)
target_include_directories(arch_defaults_test PRIVATE include)
add_test(NAME arch_defaults_test COMMAND arch_defaults_test)
+ add_executable(kva_test src/prefill/kva_test.cpp) # KVA: the switch, the automatic tail, the reasons a read is turned away
+ target_include_directories(kva_test PRIVATE include)
+ add_test(NAME kva_test COMMAND kva_test)
add_executable(adapt_async_default_test src/core/adapt_async_default_test.cpp)
target_include_directories(adapt_async_default_test PRIVATE include)
add_test(NAME adapt_async_default_test COMMAND adapt_async_default_test)
diff --git a/docs/KVA.md b/docs/KVA.md
index 556f71974..d07651f3c 100644
--- a/docs/KVA.md
+++ b/docs/KVA.md
@@ -1,7 +1,4 @@
-# KVA: approximate prompt processing for the late layers (opt-in, research)
-
-This is an **experimental, opt-in** feature. Nothing in it runs unless `STRATA_KVA=1` is set, and a build without
-that variable behaves exactly as before.
+# KVA: the faster prompt read for long cold prompts (on by default since 0.2.7)
KVA ("approximate late-layer state") reads a long prompt exactly through layer 23, then for the late layers
24–47 builds **only their recurrent state** — the GDN recurrence and convolution history, the QSA K/V pages and
@@ -10,41 +7,91 @@ the indexer keys — instead of running the whole layer. The attention output, t
results. The last positions of the prompt are then read fully, exactly as before, and decode runs the ordinary
full path against the state KVA built.
-## Turning it on
+It is an approximation: the answers can differ from the exact read (see "Measured" for how much). **`STRATA_KVA=0`
+turns it off and gives the exact 0.2.6 prompt path.** The Strata app shows this as the "Faster long prompts" switch
+in Expert settings; it sends `STRATA_KVA=0` when the switch is off.
-```sh
-STRATA_KVA=1 \
-STRATA_KVA_SUFFIX=2112 \
-STRATA_KVA_MIN_PROMPT=4096 \
- strata --serve ... --prefill 1024 --mtp-window 2048
-```
+## When it runs
+
+KVA runs when all of these hold, and says so in the engine log:
+
+- Qwen3.8-Flash-Next on CUDA or HIP (not SYCL, not Metal; Qwen3.6 and the other models read exactly);
+- one GPU, no `--batch` slots, no `--peer-device`, no DFlash2 drafter;
+- the read is cold (it starts at position 0 of the session) and the text has no pictures;
+- the prompt (or the part of it read in one go, see below) has at least `STRATA_KVA_MIN_PROMPT` tokens (4096) and
+ more than the fully computed tail plus 64.
+
+Otherwise the prompt is read exactly. An engine that was started with `STRATA_KVA=1` logs the reason for every
+read it turned away (`strata prefill: KVA configured but NOT applied: <reason>`); with the default it logs only
+a long prompt turned away for an unusual reason (a layer split, pictures), and once at start-up when the engine
+configuration rules KVA out for the whole run.
+
+## The fully computed tail and the draft layer
+
+The last `suffix` positions are always read exactly. The MTP draft layer builds its K/V from the final
+residuals of the prompt, and those rows are only real where the layers ran in full, so the tail must cover the
+draft layer's window plus the 64 cells its catch-up reaches back:
+
+- no `--mtp-window` given (what the app and the Terminal write): with KVA on, the engine sets the draft window to
+ 2048 cells and the tail to 2112 tokens. The 0.2.6 window was 32768, which is every cell of a chat up to that
+ length and could never be covered. The engine prints one line at start-up when it does this.
+- `--mtp-window N` given: the window stays N and the tail is N + 64 (a 32768-cell window never leaves a
+ prompt shorter than 32 K approximated; the log says "the draft layer attends to every cell" when no tail can cover it).
+- `STRATA_KVA_SUFFIX=N` fixes the tail by hand.
+- no draft layer (`--mtp` not given): the tail is 2048.
+
+A draft layer that reads an approximated cell costs only draft acceptance (speed); the target model verifies every
+emitted token, so the answer does not depend on it.
+
+## Chats, follow-up turns and checkpoints (`--serve`)
+
+The server reads a prompt in parts: up to the end of a long system prompt (the shared checkpoint), up to the last
+turn header, then the new turn's header and the last token. Each part that starts at position 0 and is long enough
+can take the approximate path; its own last `suffix` positions are exact, and every later part starts at a position
+greater than 0 and is read exactly.
+
+- **Turn 2 and later.** The server continues from the live session or from a conversation checkpoint (the state
+ after the earlier turns) and reads only the new tokens, at positions greater than 0, so they are read in full. They
+ are read exactly *on top of* the cells the first read left: nothing is approximated twice. Questions about the
+ old document in turn 5 are answered from the same cells turn 1 had, so their quality is that of the first read.
+- **Checkpoints and parked conversations** hold whatever cells the session held, approximated or not, and give
+ them back as they were. A checkpoint taken inside an approximated stretch is a consistent state: the read
+ continues from it exactly. Restoring a checkpoint and re-reading the same prompt from 0 (the `STRATA_CKPT_REREAD`
+ debug check) differ with KVA on, as they would between any two reads that take different paths.
+- **A cold read again** (the cache evicted the conversation, a new chat with the same system prompt) approximates
+ its own prefix again.
+- **`STRATA_KVA=0`** restores 0.2.6: every prompt is read exactly, the draft window is 32768 cells, nothing else
+ changes.
+
+## Layer splits, batch slots, SYCL, Metal
+
+These never take the approximate path and say why: a layer split reports "a layer split (not the whole trunk)", the
+batch slots and a peer device switch KVA off for the process (one line at start-up with `STRATA_KVA=1`), the SYCL
+engine has its own prompt path without KVA, and the Metal build has no state-only kernels (`STRATA_KVA` is ignored
+there, with a line when it is set).
+
+## Settings
| variable | default | what it does |
|---|---:|---|
-| `STRATA_KVA` | off | `1` enables the feature for qualifying requests |
+| `STRATA_KVA` | on | `0` turns KVA off (the exact 0.2.6 prompt read). `1` is the same as the default but logs every read it turned away |
| `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |
-| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; rounded **up** to a whole chunk |
+| `STRATA_KVA_SUFFIX` | 2112 with the default draft window | fully computed tail positions; see above |
| `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path |
| `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type |
| `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) |
-| `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration |
+| `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration (the read is exact while capturing) |
| `STRATA_KVA_CAPTURE_FP32` | off | write fp32 instead of bf16 records |
| `STRATA_KVA_ORACLE` | — | directory: replay a capture instead of a projector |
| `STRATA_KVA_GDN_NORM` | off | keep the full GDN recurrence (the A/B control for the state-only one) |
| `STRATA_LOGITS_DUMP` | — | debug (not KVA-specific): write every verify window's sampler logits to `path`, for KL / top-1 between two runs |
-Requirements and limits for this first version:
+Every read says what it did:
+
+- applied: `strata prefill: KVA on: split 24, suffix 2112, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`
+- turned away: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`
-- One cold read of one prompt, one device (`pos0 == 0`, one stage). A follow-up read, a resumed prompt or a
- layer split stays on the full path.
-- The fully computed suffix must cover the MTP drafter's window: use `--mtp-window N` with
- `STRATA_KVA_SUFFIX >= N + 64`, or the drafter reads state built from approximated positions.
-- **The session state after a KVA read is approximate.** It must not be reused as an exact prefix: no
- checkpoint of it, no follow-up prompt that reuses it. The engine prints a warning when this happens.
-- The approximation is a quality tradeoff, not a lossless transformation. Measure before using it.
-- Every run says what it did:
- - applied: `strata prefill: KVA on: split 24, suffix 2112, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`
- - configured but not applied: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`
+A quality comparison needs the control arm to say `STRATA_KVA=0` (an unset variable is now KVA).
## Measured (Tesla P100 16 GB, Qwen3.8-Flash-Next IQ3_XXS, chunk 1024, expert cache 1500, MTP window 2048)
@@ -96,7 +143,7 @@ place of a projector; the replay must use the same prompt and the same chunking.
## Implementation notes
-- `src/prefill/kva.hpp` — the plan and the environment variables.
+- `include/strata/prefill/kva.hpp` — the plan, the environment variables and the reasons a read is turned away (`src/prefill/kva_test.cpp` tests them).
- `src/prefill/prefill.cpp` — the chunk plan (`kva_approx`), the capture/oracle taps, the state-only layer
exits, the transfer-plan skip, and `give_back()` on a skipped MoE half: a layer that consumes no expert must
still release its stream entries, or the issuer thread fills the ring and the next fully computed layer waits
diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp
index d97d26bdb..f63aea4de 100644
--- a/include/strata/core/mtp.hpp
+++ b/include/strata/core/mtp.hpp
@@ -167,6 +167,8 @@ public:
const void* tensor_q8(const char* name) const { return q8(name); }
QsaState& kv_state_rw() { return st_; }
int64_t first_needed() const { return (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0; }
+ /// The cells the draft layer attends to (0 = every cell), as load() settled it.
+ int64_t window() const { return window_; }
int device() const { return device_; }
bool idle(std::string& err) {
if (cs_ && cudaStreamSynchronize(cs_) != cudaSuccess) { err = "mtp: its stream failed"; return false; }
diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hpp
index 212522a59..d9ce3dc07 100644
--- a/include/strata/prefill/kva.hpp
+++ b/include/strata/prefill/kva.hpp
@@ -1,4 +1,4 @@
-// include/strata/prefill/kva.hpp - STRATA_KVA: the optional approximate-prefill plan (research, opt-in).
+// include/strata/prefill/kva.hpp - STRATA_KVA: the approximate late-layer prompt read (on by default since 0.2.7).
//
// After layer `split` the prompt's late layers need only their recurrent state (GDN recurrence + conv history)
// and their KV/indexer caches; the attention output, the output projection, the MoE half and its expert
@@ -14,27 +14,64 @@
// oracle a captured real input per late layer (STRATA_KVA_ORACLE=<dir>; the plumbing/parity mode),
// linear a fitted low-rank projector (STRATA_KVA_PROJECTOR=<file>; not implemented in this step).
//
-// Everything here is read from the environment once per process and is off unless STRATA_KVA is set, so a
-// default build behaves exactly as before.
+// 0.2.7: the read is ON by default for a cold prompt of STRATA_KVA_MIN_PROMPT tokens or more on one CUDA/HIP
+// device. STRATA_KVA=0 brings back the exact 0.2.6 prompt path and the 0.2.6 draft window. Everything is read from
+// the environment once per process.
+//
+// The two facts the engine tells this header at start-up (kva_engine()): whether a draft layer (MTP) is loaded and
+// how many cells it attends to, and whether the engine configuration rules the read out for the whole process
+// (batch slots, a layer split). The fully computed suffix must cover the drafter's window plus the 64 cells its
+// catch-up reaches back (MtpDrafter::first_needed), or the drafter would build its K/V from approximated cells
+// (that costs acceptance, never correctness: the target model verifies every token).
#pragma once
+#include <algorithm>
#include <cstdint>
#include <cstdlib>
+#include <cstring>
#include <string>
namespace strata::prefill {
+/// Does this build have the state-only late-layer path? CUDA and HIP: yes. The Metal engine compiles prefill.cpp
+/// against its own kernel library, which has none.
+#if defined(STRATA_GPU_METAL)
+inline constexpr bool kKvaBackend = false;
+#else
+inline constexpr bool kKvaBackend = true;
+#endif
+
+/// The draft layer's window (--mtp-window) the engine picks when KVA is on, the draft layer is loaded and the user
+/// gave no --mtp-window: 2048 cells (suffix 2112). The 0.2.6 default was 32768, i.e. every cell of a chat up to
+/// that length, which no suffix could cover.
+inline constexpr int64_t kKvaDrafterWindow = 2048;
+/// The fully computed tail when no draft layer needs more.
+inline constexpr int64_t kKvaSuffixDefault = 2048;
+
+/// What the running engine knows and the environment cannot say.
+struct KvaEngine {
+ bool has_drafter = false; ///< an MTP draft layer is loaded (--mtp with --spec 2 or more)
+ int64_t drafter_window = 0; ///< the cells it attends to; 0 = every cell
+ std::string blocked; ///< non-empty: this process never reads approximately, and why
+};
+
+inline KvaEngine& kva_engine() {
+ static KvaEngine e;
+ return e;
+}
+
struct KvaConfig {
enum class Mode { Identity, Oracle, Linear };
enum class Only { All, Gdn, Qsa };
- bool on = false; ///< STRATA_KVA=1
+ bool on = true; ///< on unless STRATA_KVA=0 (or this build has no path)
+ bool explicit_on = false; ///< STRATA_KVA was set to something other than 0: every declined read says why
bool capture = false; ///< STRATA_KVA_CAPTURE=<dir>: dump every late layer's real input rows
bool capture_fp32 = false; ///< STRATA_KVA_CAPTURE_FP32=1: fp32 records (else bf16; fp32 is for parity runs)
bool oracle_fp32 = false; ///< STRATA_KVA_ORACLE_FP32=1: the oracle files are fp32 (else bf16)
bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full path
int64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)
- int64_t suffix = 2048; ///< STRATA_KVA_SUFFIX: fully computed tail positions (rounded up to a chunk)
+ int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: effective_suffix())
int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full path
Mode mode = Mode::Identity; ///< STRATA_KVA_MODE=identity|oracle|linear
Only only = Only::All; ///< STRATA_KVA_ONLY=all|gdn|qsa: which late layers take the approximate path
@@ -42,12 +79,48 @@ struct KvaConfig {
std::string capture_dir; ///< STRATA_KVA_CAPTURE
std::string projector; ///< STRATA_KVA_PROJECTOR
- /// Does this run qualify at all? A cold read of one prompt, on one stage, long enough for a suffix to
- /// leave approximated positions behind it.
- bool applies(int64_t pos0, int64_t n, int64_t lb, int64_t le, int64_t n_layers) const {
- if (!on || capture) return false; // capture runs are ordinary full reads
- if (pos0 != 0 || lb != 0 || (le >= 0 && le != n_layers)) return false;
- return n >= min_prompt && n > suffix + 64;
+ /// The fully computed tail of a read. STRATA_KVA_SUFFIX when set; otherwise 2048, or the draft layer's window +
+ /// 64 when that is larger. (A draft layer that attends to every cell has no suffix that covers it:
+ /// decline() says so.)
+ int64_t effective_suffix(const KvaEngine& e) const {
+ if (suffix_env > 0) return suffix_env;
+ int64_t s = kKvaSuffixDefault;
+ if (e.has_drafter && e.drafter_window > 0) s = std::max<int64_t>(s, e.drafter_window + 64);
+ return s;
+ }
+
+ /// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a
+ /// continuation, no layer split, no pictures, no DFlash2 taps), on an engine that allows it, long enough for a
+ /// suffix to leave approximated positions behind it. `routine` (optional) is set when the reason is an
+ /// everyday one (a continuation, a short prompt) that an engine running KVA by default does not log.
+ const char* decline(const KvaEngine& e, int64_t pos0, int64_t n, int64_t lb, int64_t le, int64_t n_layers,
+ bool pictures = false, bool taps = false, bool* routine = nullptr,
+ bool unvalidated_model = false) const {
+ if (routine) *routine = false;
+ if (!on) return "off (STRATA_KVA=0)";
+ if (capture) return "capture mode";
+ if (!e.blocked.empty()) return e.blocked.c_str();
+ // The split layer, the state-only layers and the suffix rule were measured on one geometry (Qwen3.8-Flash-Next,
+ // 48 layers, 512 experts). Another model reads exactly unless STRATA_KVA=1 asks for it.
+ if (unvalidated_model && !explicit_on) return "KVA was not validated for this model (STRATA_KVA=1 forces it)";
+ if (pos0 != 0) {
+ if (routine) *routine = true;
+ return "a continuation read (pos0 != 0)";
+ }
+ if (lb != 0 || (le >= 0 && le != n_layers)) return "a layer split (not the whole trunk)";
+ if (pictures) return "a prompt with pictures";
+ if (taps) return "the DFlash2 drafter reads the late layers' hidden states";
+ if (suffix_env <= 0 && e.has_drafter && e.drafter_window <= 0)
+ return "the draft layer attends to every cell (set --mtp-window)";
+ if (n < min_prompt) {
+ if (routine) *routine = true;
+ return "the prompt is shorter than STRATA_KVA_MIN_PROMPT";
+ }
+ if (n <= effective_suffix(e) + 64) {
+ if (routine) *routine = true;
+ return "the suffix leaves too few positions to approximate";
+ }
+ return nullptr;
}
};
@@ -63,41 +136,48 @@ inline bool kva_env_flag(const char* name, bool dflt) {
return v[0] != '0';
}
+/// The settings as the environment says now (uncached: the tests call it after changing the environment).
+inline KvaConfig kva_config_from_env() {
+ KvaConfig k;
+ const char* sw = std::getenv("STRATA_KVA");
+ const bool set = sw != nullptr && *sw != '\0';
+ k.on = !set || sw[0] != '0';
+ k.explicit_on = set && sw[0] != '0';
+ if (!kKvaBackend) k.on = false;
+ k.split = kva_env_i64("STRATA_KVA_SPLIT", 24);
+ k.suffix_env = std::max<int64_t>(0, kva_env_i64("STRATA_KVA_SUFFIX", 0));
+ k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);
+ k.parity = kva_env_flag("STRATA_KVA_PARITY", false);
+ if (const char* d = std::getenv("STRATA_KVA_CAPTURE"); d != nullptr && *d != '\0') {
+ k.capture = true;
+ k.capture_dir = d;
+ k.capture_fp32 = kva_env_flag("STRATA_KVA_CAPTURE_FP32", false);
+ }
+ if (const char* d = std::getenv("STRATA_KVA_ORACLE"); d != nullptr && *d != '\0') {
+ k.oracle_dir = d;
+ k.mode = KvaConfig::Mode::Oracle;
+ k.oracle_fp32 = kva_env_flag("STRATA_KVA_ORACLE_FP32", false);
+ }
+ if (const char* d = std::getenv("STRATA_KVA_PROJECTOR"); d != nullptr && *d != '\0') {
+ k.projector = d;
+ k.mode = KvaConfig::Mode::Linear;
+ }
+ if (const char* m = std::getenv("STRATA_KVA_MODE"); m != nullptr && *m != '\0') {
+ if (std::string(m) == "oracle") k.mode = KvaConfig::Mode::Oracle;
+ else if (std::string(m) == "linear") k.mode = KvaConfig::Mode::Linear;
+ else k.mode = KvaConfig::Mode::Identity;
+ }
+ if (const char* o = std::getenv("STRATA_KVA_ONLY"); o != nullptr && *o != '\0') {
+ if (std::string(o) == "gdn") k.only = KvaConfig::Only::Gdn;
+ else if (std::string(o) == "qsa") k.only = KvaConfig::Only::Qsa;
+ else k.only = KvaConfig::Only::All;
+ }
+ return k;
+}
+
/// The process's KVA settings, parsed once (the engine's own options are parsed once too).
inline const KvaConfig& kva_config() {
- static const KvaConfig c = [] {
- KvaConfig k;
- k.on = kva_env_flag("STRATA_KVA", false);
- k.split = kva_env_i64("STRATA_KVA_SPLIT", 24);
- k.suffix = kva_env_i64("STRATA_KVA_SUFFIX", 2048);
- k.min_prompt = kva_env_i64("STRATA_KVA_MIN_PROMPT", 4096);
- k.parity = kva_env_flag("STRATA_KVA_PARITY", false);
- if (const char* d = std::getenv("STRATA_KVA_CAPTURE"); d != nullptr && *d != '\0') {
- k.capture = true;
- k.capture_dir = d;
- k.capture_fp32 = kva_env_flag("STRATA_KVA_CAPTURE_FP32", false);
- }
- if (const char* d = std::getenv("STRATA_KVA_ORACLE"); d != nullptr && *d != '\0') {
- k.oracle_dir = d;
- k.mode = KvaConfig::Mode::Oracle;
- k.oracle_fp32 = kva_env_flag("STRATA_KVA_ORACLE_FP32", false);
- }
- if (const char* d = std::getenv("STRATA_KVA_PROJECTOR"); d != nullptr && *d != '\0') {
- k.projector = d;
- k.mode = KvaConfig::Mode::Linear;
- }
- if (const char* m = std::getenv("STRATA_KVA_MODE"); m != nullptr && *m != '\0') {
- if (std::string(m) == "oracle") k.mode = KvaConfig::Mode::Oracle;
- else if (std::string(m) == "linear") k.mode = KvaConfig::Mode::Linear;
- else k.mode = KvaConfig::Mode::Identity;
- }
- if (const char* o = std::getenv("STRATA_KVA_ONLY"); o != nullptr && *o != '\0') {
- if (std::string(o) == "gdn") k.only = KvaConfig::Only::Gdn;
- else if (std::string(o) == "qsa") k.only = KvaConfig::Only::Qsa;
- else k.only = KvaConfig::Only::All;
- }
- return k;
- }();
+ static const KvaConfig c = kva_config_from_env();
return c;
}
diff --git a/metal/src/engine/stubs.cpp b/metal/src/engine/stubs.cpp
index 78ff2891b..b6615813c 100644
--- a/metal/src/engine/stubs.cpp
+++ b/metal/src/engine/stubs.cpp
@@ -340,6 +340,7 @@ namespace strata::prefill { void gather_rows16(const uint16_t * p0, const int32_
namespace strata::prefill { void gdn_conv(float * p0, const float * p1, const float * p2, float * p3, int64_t p4, float p5, void * p6) { strata::metal_engine::missing("strata::prefill::gdn_conv"); } }
namespace strata::prefill { void gdn_gates(const float * p0, const float * p1, const float * p2, float * p3, float * p4, int64_t p5, void * p6) { strata::metal_engine::missing("strata::prefill::gdn_gates"); } }
namespace strata::prefill { void gdn_recurrence(float * p0, const float * p1, const float * p2, const float * p3, const float * p4, const float * p5, float p6, float * p7, uint16_t * p8, int64_t p9, void * p10, int64_t p11) { strata::metal_engine::missing("strata::prefill::gdn_recurrence"); } }
+namespace strata::prefill { void gdn_recurrence_state_only(float * p0, const float * p1, const float * p2, const float * p3, float * p4, int64_t p5, void * p6) { strata::metal_engine::missing("strata::prefill::gdn_recurrence_state_only"); } }
namespace strata::prefill { void gr_broadcast(const float * p0, float * p1, int64_t p2, void * p3) { strata::metal_engine::missing("strata::prefill::gr_broadcast"); } }
namespace strata::prefill { void gr_mix(const float * p0, const float * p1, float * p2, uint16_t * p3, int64_t p4, void * p5, uint16_t * p6, uint16_t * p7) { strata::metal_engine::missing("strata::prefill::gr_mix"); } }
namespace strata::prefill { void gr_mix_r(const float * p0, const float * p1, const float * p2, const float * p3, float * p4, uint16_t * p5, int64_t p6, void * p7, uint16_t * p8, uint16_t * p9) { strata::metal_engine::missing("strata::prefill::gr_mix_r"); } }
diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cpp
new file mode 100644
index 000000000..bd468c34d
--- /dev/null
+++ b/src/prefill/kva_test.cpp
@@ -0,0 +1,143 @@
+// src/prefill/kva_test.cpp - the KVA switch, the automatic tail and the reasons a read is turned away (no GPU needed).
+#include "strata/prefill/kva.hpp"
+
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <string>
+
+using namespace strata::prefill;
+
+static int fails = 0;
+#define CHECK(c) do { if (!(c)) { std::printf("FAIL line %d: %s\n", __LINE__, #c); ++fails; } } while (0)
+
+static void unset(const char* k) {
+#if defined(_WIN32)
+ _putenv_s(k, "");
+#else
+ unsetenv(k);
+#endif
+}
+static void put(const char* k, const char* v) {
+#if defined(_WIN32)
+ _putenv_s(k, v);
+#else
+ setenv(k, v, 1);
+#endif
+}
+static void clear_all() {
+ for (const char* k : {"STRATA_KVA", "STRATA_KVA_SPLIT", "STRATA_KVA_SUFFIX", "STRATA_KVA_MIN_PROMPT", "STRATA_KVA_PARITY",
+ "STRATA_KVA_CAPTURE", "STRATA_KVA_CAPTURE_FP32", "STRATA_KVA_ORACLE", "STRATA_KVA_ORACLE_FP32",
+ "STRATA_KVA_PROJECTOR", "STRATA_KVA_MODE", "STRATA_KVA_ONLY"})
+ unset(k);
+}
+static bool is(const char* got, const char* want) { return got != nullptr && std::strcmp(got, want) == 0; }
+
+int main() {
+ // ---- the switch: on by default, STRATA_KVA=0 off, anything else on and chatty
+ clear_all();
+ {
+ const KvaConfig c = kva_config_from_env();
+ CHECK(c.on && !c.explicit_on);
+ CHECK(c.split == 24 && c.min_prompt == 4096 && c.suffix_env == 0);
+ CHECK(c.mode == KvaConfig::Mode::Identity && c.only == KvaConfig::Only::All && !c.capture);
+ }
+ put("STRATA_KVA", "0");
+ { const KvaConfig c = kva_config_from_env(); CHECK(!c.on && !c.explicit_on); }
+ put("STRATA_KVA", "1");
+ { const KvaConfig c = kva_config_from_env(); CHECK(c.on && c.explicit_on); }
+ put("STRATA_KVA", ""); // an empty value is "not set"
+ { const KvaConfig c = kva_config_from_env(); CHECK(c.on && !c.explicit_on); }
+ clear_all();
+
+ // ---- the automatic tail: 2048, or the draft layer's window + 64 when that is larger; STRATA_KVA_SUFFIX wins
+ {
+ const KvaConfig c = kva_config_from_env();
+ KvaEngine none; // no draft layer
+ CHECK(c.effective_suffix(none) == kKvaSuffixDefault);
+ KvaEngine d; // the window the engine picks (no --mtp-window given)
+ d.has_drafter = true;
+ d.drafter_window = kKvaDrafterWindow;
+ CHECK(c.effective_suffix(d) == 2112);
+ d.drafter_window = 8192; // --mtp-window 8192
+ CHECK(c.effective_suffix(d) == 8256);
+ d.drafter_window = 512; // a small window: the default tail still holds
+ CHECK(c.effective_suffix(d) == 2048);
+ d.drafter_window = 0; // every cell: no tail can cover it, the plain tail is returned
+ CHECK(c.effective_suffix(d) == 2048);
+ put("STRATA_KVA_SUFFIX", "3000");
+ const KvaConfig e = kva_config_from_env();
+ d.drafter_window = 8192;
+ CHECK(e.effective_suffix(d) == 3000);
+ clear_all();
+ }
+
+ // ---- who is turned away, and why (the first reason that holds is the one named)
+ {
+ const KvaConfig c = kva_config_from_env();
+ KvaEngine eng;
+ eng.has_drafter = true;
+ eng.drafter_window = kKvaDrafterWindow;
+ bool routine = false;
+ // a cold 14.6K prompt on one device: applies
+ CHECK(c.decline(eng, 0, 14600, 0, -1, 48) == nullptr);
+ CHECK(c.decline(eng, 0, 14600, 0, 48, 48) == nullptr); // le == n_layers is the whole trunk too
+ // a continuation (turn 2 of a chat, a resumed prompt): routine
+ CHECK(is(c.decline(eng, 9000, 14600, 0, -1, 48, false, false, &routine), "a continuation read (pos0 != 0)") && routine);
+ // short prompts and a prompt the tail would swallow: routine
+ CHECK(is(c.decline(eng, 0, 3000, 0, -1, 48, false, false, &routine), "the prompt is shorter than STRATA_KVA_MIN_PROMPT") && routine);
+ CHECK(c.decline(eng, 0, 4096, 0, -1, 48) == nullptr); // 4096 > 2112 + 64: it applies
+ put("STRATA_KVA_SUFFIX", "5000");
+ { const KvaConfig s = kva_config_from_env();
+ CHECK(is(s.decline(eng, 0, 5000, 0, -1, 48, false, false, &routine), "the suffix leaves too few positions to approximate") && routine); }
+ clear_all();
+ // not routine: the user may want to know
+ CHECK(is(c.decline(eng, 0, 14600, 24, -1, 48, false, false, &routine), "a layer split (not the whole trunk)") && !routine);
+ CHECK(is(c.decline(eng, 0, 14600, 0, 24, 48), "a layer split (not the whole trunk)"));
+ CHECK(is(c.decline(eng, 0, 14600, 0, -1, 48, true), "a prompt with pictures"));
+ CHECK(is(c.decline(eng, 0, 14600, 0, -1, 48, false, true), "the DFlash2 drafter reads the late layers' hidden states"));
+ // a model the split and the tail were not measured on reads exactly unless STRATA_KVA=1 asks
+ CHECK(is(c.decline(eng, 0, 14600, 0, -1, 40, false, false, &routine, true),
+ "KVA was not validated for this model (STRATA_KVA=1 forces it)") && !routine);
+ put("STRATA_KVA", "1");
+ { const KvaConfig f = kva_config_from_env(); CHECK(f.decline(eng, 0, 14600, 0, -1, 40, false, false, nullptr, true) == nullptr); }
+ clear_all();
+ // a draft layer that attends to every cell has no tail that covers it
+ KvaEngine all = eng;
+ all.drafter_window = 0;
+ CHECK(is(c.decline(all, 0, 14600, 0, -1, 48), "the draft layer attends to every cell (set --mtp-window)"));
+ put("STRATA_KVA_SUFFIX", "2112"); // ... unless the user fixes the tail by hand
+ { const KvaConfig s = kva_config_from_env(); CHECK(s.decline(all, 0, 14600, 0, -1, 48) == nullptr); }
+ clear_all();
+ // no draft layer at all: nothing to cover
+ KvaEngine nod;
+ CHECK(c.decline(nod, 0, 14600, 0, -1, 48) == nullptr);
+ // the engine rules the whole process out (batch slots, a layer split, a peer device)
+ KvaEngine blocked = eng;
+ blocked.blocked = "batch slots (--batch)";
+ CHECK(is(c.decline(blocked, 0, 14600, 0, -1, 48, false, false, &routine), "batch slots (--batch)") && !routine);
+ // off wins over everything, capture over the rest
+ put("STRATA_KVA", "0");
+ { const KvaConfig o = kva_config_from_env(); CHECK(is(o.decline(eng, 0, 14600, 0, -1, 48), "off (STRATA_KVA=0)")); }
+ put("STRATA_KVA", "1");
+ put("STRATA_KVA_CAPTURE", "/tmp/kva-cap");
+ { const KvaConfig o = kva_config_from_env(); CHECK(o.capture && is(o.decline(eng, 0, 14600, 0, -1, 48), "capture mode")); }
+ clear_all();
+ }
+
+ // ---- the research modes still parse
+ put("STRATA_KVA_ORACLE", "/tmp/o");
+ put("STRATA_KVA_ORACLE_FP32", "1");
+ put("STRATA_KVA_ONLY", "qsa");
+ put("STRATA_KVA_SPLIT", "30");
+ put("STRATA_KVA_MIN_PROMPT", "1024");
+ {
+ const KvaConfig c = kva_config_from_env();
+ CHECK(c.mode == KvaConfig::Mode::Oracle && c.oracle_fp32 && c.oracle_dir == "/tmp/o");
+ CHECK(c.only == KvaConfig::Only::Qsa && c.split == 30 && c.min_prompt == 1024);
+ }
+ clear_all();
+
+ if (fails == 0) std::printf("kva_test: ok\n");
+ return fails == 0 ? 0 : 1;
+}
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index f0b09e558..741c3d8fb 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -2716,9 +2716,15 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
// STRATA_KVA: the approximate-prefix plan of this run, if it qualifies (a cold read of one long prompt, one
// stage, no capture). Positions [0, kva_frontier) are approximated in the layers at and after the split.
const KvaConfig& kva = kva_config();
- const bool kva_run = kva.applies(pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers);
+ const KvaEngine& kva_eng = kva_engine();
+ const int64_t kva_suffix = kva.effective_suffix(kva_eng);
+ bool kva_routine = false;
+ const char* kva_why_not = kva.decline(kva_eng, pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers,
+ embd_rows != nullptr, !tap_layers_.empty(), &kva_routine,
+ g.n_layers != 48 || g.n_expert != 512);
+ const bool kva_run = kva_why_not == nullptr;
const int64_t kva_split = std::min<int64_t>(std::max<int64_t>(kva.split, 1), g.n_layers);
- const int64_t kva_frontier = std::max<int64_t>(0, n - kva.suffix);
+ const int64_t kva_frontier = std::max<int64_t>(0, n - kva_suffix);
static KvaTap kva_capture; // process-wide: one cold read at a time, so the per-layer position is enough
static KvaTap kva_oracle;
int64_t kva_positions = 0, kva_chunks = 0; // what actually took the approximate path (diagnostics)
@@ -5272,25 +5278,31 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
: kva.only == KvaConfig::Only::Qsa ? ", qsa layers only" : "";
std::fprintf(stderr, "strata prefill: KVA %s: split %lld, suffix %lld, approximated %lld of %lld positions "
"in %lld of %lld chunks, mode %s%s%s\n",
- kva_run ? "on" : "off (capture)", (long long) kva_split, (long long) kva.suffix,
+ kva_run ? "on" : "off (capture)", (long long) kva_split, (long long) kva_suffix,
(long long) kva_positions, (long long) n, (long long) kva_chunks,
(long long) (n > 0 && m.T > 0 ? (n + m.T - 1) / m.T : 0), mode, only,
kva.capture ? ", capturing the late layers' inputs" : "");
- if (kva_positions > 0)
- std::fprintf(stderr, "strata prefill: KVA: the session state holds approximated cells; it is not an exact "
- "prefix and must not be reused as one (no checkpoint, no follow-up reuse)\n");
- } else if (kva.on) {
+ if (kva_positions > 0) {
+ // What a later turn does with this state (serve): it continues from the live session or a checkpoint of
+ // it and reads only the new tokens, exactly, over the cells this read left; nothing is approximated twice.
+ // The answer quality of those turns is that of this read. STRATA_KVA=0 reads exactly.
+ static std::atomic<bool> said{false};
+ if (!said.exchange(true))
+ std::fprintf(stderr, "strata prefill: KVA: the late layers (%lld and up) hold approximated cells for "
+ "the first %lld positions of this prompt; a follow-up turn that continues from "
+ "this state reads its new tokens exactly on top of them (nothing is approximated "
+ "twice); STRATA_KVA=0 reads every prompt exactly\n",
+ (long long) kva_split, (long long) kva_frontier);
+ }
+ } else if (kva.on && (kva.explicit_on || (n >= kva.min_prompt && !kva_routine && kva_eng.blocked.empty()))) {
// KVA-8 (observability): a run that was configured for KVA but fell back completely must say so and why,
- // or an operator cannot tell it apart from a run where the feature was never asked for.
- const char* why = pos0 != 0 ? "a continuation read (pos0 != 0)"
- : (LB != 0 || (LE >= 0 && LE != g.n_layers)) ? "a layer split (not the whole trunk)"
- : kva.capture ? "capture mode"
- : n < kva.min_prompt ? "the prompt is shorter than STRATA_KVA_MIN_PROMPT"
- : n <= kva.suffix + 64 ? "the suffix leaves too few positions to approximate"
- : "not applicable";
+ // or an operator cannot tell it apart from a run where the feature was never asked for. With KVA on by
+ // default only a long prompt that was turned away for an unusual reason is worth a line (STRATA_KVA=1 asks
+ // for every reason).
std::fprintf(stderr, "strata prefill: KVA configured but NOT applied: %s (prompt %lld positions, "
"min_prompt %lld, suffix %lld, split %lld)\n",
- why, (long long) n, (long long) kva.min_prompt, (long long) kva.suffix, (long long) kva_split);
+ kva_why_not != nullptr ? kva_why_not : "not applicable", (long long) n, (long long) kva.min_prompt,
+ (long long) kva_suffix, (long long) kva_split);
}
if (std::getenv("STRATA_STATE_HASH_GDN") != nullptr) { // debug: the GDN states as the prompt path leaves them
cudaStreamSynchronize(m.cs);
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 45aae75ac..213e3640a 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -71,6 +71,7 @@
#include "strata/core/dflash2.hpp"
#include "strata/core/draft_cap.hpp"
#endif
+#include "strata/prefill/kva.hpp"
#include "strata/prefill/prefill.hpp"
#if defined(STRATA_PREFILL_MMQ)
#include "strata/prefill/moe_mmq.hpp"
@@ -680,6 +681,7 @@ struct Options {
/// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it.
std::string mtp;
int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell)
+ bool mtp_window_given = false; ///< --mtp-window was given (KVA picks the window itself only when it was not)
/// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the
/// pinned arena while the CPU computes the rest (verify windows).
double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs)
@@ -2626,7 +2628,7 @@ int main(int argc, char** argv) {
else if (a == "--spec-follow") o.spec_follow = next("--spec-follow");
else if (a == "--window-hashes") o.window_hashes = next("--window-hashes");
else if (a == "--mtp") o.mtp = next("--mtp");
- else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window"));
+ else if (a == "--mtp-window") { o.mtp_window = std::atoll(next("--mtp-window")); o.mtp_window_given = true; }
else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac"));
else if (a == "--adapt-every") { o.adapt_every = std::atoi(next("--adapt-every")); o.adapt_every_given = true; }
else if (a == "--adapt-decay") o.adapt_decay = (float) std::atof(next("--adapt-decay"));
@@ -5496,6 +5498,37 @@ int main(int argc, char** argv) {
// slots' draft-KV copies use this one too
static const strata::core::ModelGeometry draft_geometry{};
std::vector<std::unique_ptr<strata::core::MtpDrafter>> slot_mtp;
+ // KVA (0.2.7: on by default, STRATA_KVA=0 is the exact 0.2.6 path): what only the engine knows. A layer split and
+ // the batch slots read their prompts in pieces and in stages, which the one-device, one-cold-read rule of the
+ // approximate prompt read cannot follow, so they stay on the exact path (and say so here, once). The draft layer
+ // (MTP) must find exact cells in its window: with no --mtp-window given the window becomes 2048 cells (the 0.2.6
+ // default, 32768, is every cell of a chat up to that length, which no fully computed tail could cover) and the
+ // tail 2112; a window the user gave keeps its value and sets the tail (window + 64).
+ {
+ const strata::prefill::KvaConfig& kc = strata::prefill::kva_config();
+ strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();
+ if (kc.explicit_on && !strata::prefill::kKvaBackend)
+ std::fprintf(stderr, "strata generate: STRATA_KVA is set, but this build has no state-only late-layer path; "
+ "every prompt is read exactly\n");
+ if (kc.on) {
+ if (multi_gpu) ke.blocked = "a layer split across GPUs";
+ else if (o.batch > 0) ke.blocked = "batch slots (--batch)";
+ else if (o.peer_device >= 0) ke.blocked = "a peer device (--peer-device)";
+ if (!ke.blocked.empty()) {
+ if (kc.explicit_on)
+ std::fprintf(stderr, "strata generate: KVA (the approximate late-layer prompt read) is off here: %s\n",
+ ke.blocked.c_str());
+ } else if (!o.mtp.empty() && o.spec >= 2 && !o.mtp_window_given && kc.suffix_env <= 0) {
+ if (o.mtp_window != strata::prefill::kKvaDrafterWindow)
+ std::fprintf(stderr, "strata generate: KVA on (the faster prompt read for long cold prompts): the draft layer "
+ "attends to the last %lld cells (--mtp-window %lld); STRATA_KVA=0 keeps the "
+ "exact 0.2.6 prompt read and its %lld-cell window\n",
+ (long long) strata::prefill::kKvaDrafterWindow, (long long) strata::prefill::kKvaDrafterWindow,
+ (long long) o.mtp_window);
+ o.mtp_window = strata::prefill::kKvaDrafterWindow;
+ }
+ }
+ }
if (!o.mtp.empty()) {
if (o.spec < 2) {
std::fprintf(stderr, "strata generate: --mtp is ignored without --spec T (T >= 2)\n");
@@ -5515,6 +5548,11 @@ int main(int argc, char** argv) {
if (o.pipeline_windows >= 2) mtp.set_force_capture(true);
if (!o.mtp.empty() && !mtp.load(o.mtp, draft_geometry, last_st && !mtp_first ? last_st->ss : ss, mtp_t, err, o.mtp_window)) { std::fprintf(stderr, "strata generate: %s%s\n", err.c_str(), vram_free_note().c_str()); return 1; }
mtp.set_ple_session(&ss);
+ if (!o.mtp.empty()) { // KVA: the cells the draft layer attends to now (0 = every cell) set the tail that must be exact
+ strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();
+ ke.has_drafter = true;
+ ke.drafter_window = mtp.window();
+ }
#if defined(STRATA_HAVE_DFLASH2)
if (o.draft == "dflash2") {
const char* why = o.mtp.empty() ? "it needs --mtp (the fallback) and --spec 2 or more"