Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
KVA: leave the draft window alone; a per-session cell floor keeps the draft layer off approximated cells
Niko12217a34c60d61c7
diff --git a/docs/KVA.md b/docs/KVA.mdindex d07651f3c..c53c6abde 100644--- a/docs/KVA.md+++ b/docs/KVA.md@@ -28,20 +28,28 @@ configuration rules KVA out for the whole run.## The fully computed tail and the draft layer-The last `suffix` positions are always read exactly. The MTP draft layer builds its K/V from the final-residuals of the prompt, and those rows are only real where the layers ran in full, so the tail must cover the-draft layer's window plus the 64 cells its catch-up reaches back:--- no `--mtp-window` given (what the app and the Terminal write): with KVA on, the engine sets the draft window to- 2048 cells and the tail to 2112 tokens. The 0.2.6 window was 32768, which is every cell of a chat up to that- length and could never be covered. The engine prints one line at start-up when it does this.-- `--mtp-window N` given: the window stays N and the tail is N + 64 (a 32768-cell window never leaves a- prompt shorter than 32 K approximated; the log says "the draft layer attends to every cell" when no tail can cover it).-- `STRATA_KVA_SUFFIX=N` fixes the tail by hand.-- no draft layer (`--mtp` not given): the tail is 2048.--A draft layer that reads an approximated cell costs only draft acceptance (speed); the target model verifies every-emitted token, so the answer does not depend on it.+The last `suffix` positions (2048, or `STRATA_KVA_SUFFIX`) are always read exactly.++The MTP draft layer builds its K/V from the final residuals of the prompt, and an approximated chunk has none. The+draft layer's window is **not changed**: it stays what it was (`--mtp-window`, default 32768 cells). Instead the+engine keeps a *cell floor* per session, the first draft cell built from a real row:++- A read that KVA approximated raises the floor to the end of its approximated chunks. The draft layer builds no+ cell below it and its attention starts no lower, so it reads the window `[max(floor, pos - window), pos)`: the+ exactly computed tail, growing with the conversation, never an approximated cell. The target model verifies+ every token either way.+- The floor follows the *tokens* of the state a request continues from (the live session, a conversation checkpoint,+ a parked conversation): the engine remembers which token prefixes a KVA read approximated, and a state's floor+ is how much of its tokens agree with one of them. A state no KVA read produced (a short chat, another document,+ every prompt of a `STRATA_KVA=0` run) has floor 0.+- With floor 0 the draft layer attends to exactly the window it attended to with `STRATA_KVA=0`+ (`window_start`, unit tested for every length). The only added work is one tiny kernel that picks the cells.+ With `STRATA_KVA=0`, with the engine configurations that rule KVA out and with a build without a draft layer,+ the draft layer's code is 0.2.6's own.++Measured on the P100 (14K prompt, 256 tokens, auto cache and prefill), the draft layer's window made no difference+to decode (38.7 tok/s with the 32768-cell window, 40.1 with 2048); on faster cards a smaller window costs+acceptance, which is why the window is left alone.## Chats, follow-up turns and checkpoints (`--serve`)@@ -76,7 +84,7 @@ there, with a line when it is set).|---|---:|---|| `STRATA_KVA` | on | `0` turns KVA off (the exact 0.2.6 prompt read). `1` is the same as the default but logs every read it turned away || `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |-| `STRATA_KVA_SUFFIX` | 2112 with the default draft window | fully computed tail positions; see above |+| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; see above || `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path || `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type || `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) |@@ -88,7 +96,7 @@ there, with a line when it is set).Every read says what it did:-- applied: `strata prefill: KVA on: split 24, suffix 2112, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`+- applied: `strata prefill: KVA on: split 24, suffix 2048, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`- turned away: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`A quality comparison needs the control arm to say `STRATA_KVA=0` (an unset variable is now KVA).diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hppindex f63aea4de..38fe8f011 100644--- a/include/strata/core/mtp.hpp+++ b/include/strata/core/mtp.hpp@@ -26,6 +26,7 @@#include <cuda_runtime.h>+#include <algorithm>#include <cstdint>#include <functional>#include <string>@@ -166,9 +167,25 @@ public:const uint16_t* tensor_bf16(const char* name) const { return bf16(name); }const void* tensor_q8(const char* name) const { return q8(name); }QsaState& kv_state_rw() { return st_; }- int64_t first_needed() const { return (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0; }+ /// The first cell whose K/V the draft layer can still read: the window's start for this prompt, and never below+ /// the KVA floor (cells built from approximated rows are not read, so they are not built either).+ int64_t first_needed() const {+ const int64_t w = (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0;+ return std::max<int64_t>(w, cell_floor_);+ }/// The cells the draft layer attends to (0 = every cell), as load() settled it.int64_t window() const { return window_; }+ /// KVA: call before load(). The draft layer's attention then reads its cell range through a floor, which stays 0+ /// (the plain window, cell for cell) until set_cell_floor() raises it. Off (the default): the code is 0.2.6's.+ void enable_cell_floor(bool on) { floor_on_ = on; }+ bool cell_floor_enabled() const { return floor_on_ && m_floor_ != nullptr; }+ /// KVA: the first draft cell that holds a K/V built from a real (not approximated) row. The attention starts no+ /// lower; draft_kv / prefill build no cell below it. Between requests (the drafter idle). 0 = every cell is real.+ void set_cell_floor(int64_t cell) {+ cell_floor_ = std::max<int64_t>(0, cell);+ if (m_floor_ != nullptr) *(volatile int32_t*) h_floor_ = (int32_t) std::min<int64_t>(cell_floor_, 0x7fffffff);+ }+ int64_t cell_floor() const { return cell_floor_; }int device() const { return device_; }bool idle(std::string& err) {if (cs_ && cudaStreamSynchronize(cs_) != cudaSuccess) { err = "mtp: its stream failed"; return false; }@@ -273,6 +290,9 @@ private:int64_t n_dvocab_ = 0;std::string rt_dir_;int64_t window_ = 0; // attention over the last window_ cells (0 = every cell)+ int64_t cell_floor_ = 0; // KVA: no draft cell below this one is read or built (set_cell_floor)+ bool floor_on_ = false; // enable_cell_floor: the attention goes through window_ids_floor+ int32_t *h_floor_ = nullptr, *m_floor_ = nullptr; // the floor, mapped: the kernel reads it when it runsint64_t prompt_len_ = 0;const int32_t* rr_res_ = nullptr;float rr_margin_ = 0.0f;diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hppindex a5e35181e..b2ff4a0d4 100644--- a/include/strata/kernels/verify_kernels.hpp+++ b/include/strata/kernels/verify_kernels.hpp@@ -192,5 +192,10 @@ void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream);/// A sliding attention window: for `n` step records (kStepCount ints each, n_kv at [1]) the selection becomes the/// last `window` cells: ids[q * ids_stride + j] = max(0, n_kv - window) + j and the record's width = the count.void window_ids(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_stride, void* stream);+/// window_ids whose window starts no lower than the cell `*floor_cell` (device-visible memory, read when the kernel+/// runs; KVA: the first draft cell built from a real row). `window` may be INT_MAX for "every cell". A floor of 0+/// selects exactly what window_ids selects. CUDA / HIP only.+void window_ids_floor(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_stride, const int32_t* floor_cell,+ void* stream);} // namespace strata::kernelsdiff --git a/include/strata/kernels/window_start.hpp b/include/strata/kernels/window_start.hppnew file mode 100644index 000000000..8b0944350--- /dev/null+++ b/include/strata/kernels/window_start.hpp@@ -0,0 +1,26 @@+// include/strata/kernels/window_start.hpp - the first cell the draft layer's attention reads.+//+// The draft layer attends to the last `window` cells. KVA (docs/KVA.md) can leave cells in the draft layer's K/V+// that were built from approximated rows; `floor` is the first cell that holds a real one, so the attention starts+// there when it is higher. floor = 0 (every session KVA did not approximate) gives exactly the plain window.+// One function for the device kernel and for the host test.+#pragma once++#if defined(__CUDACC__) || defined(__HIPCC__)+#define STRATA_WINDOW_START_HD __host__ __device__+#else+#define STRATA_WINDOW_START_HD+#endif++namespace strata::kernels {++/// The first cell of the attention for a sequence of `n_kv` cells. The width n_kv - start is at least 1 whenever+/// n_kv is (a floor at or past the end must not leave the attention with nothing to read).+STRATA_WINDOW_START_HD inline int window_start(int n_kv, int window, int floor) {+ int start = n_kv > window ? n_kv - window : 0;+ if (floor > start) start = floor;+ if (start > n_kv - 1) start = n_kv > 0 ? n_kv - 1 : 0;+ return start;+}++} // namespace strata::kernelsdiff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hppindex d9ce3dc07..80e8bdb0e 100644--- a/include/strata/prefill/kva.hpp+++ b/include/strata/prefill/kva.hpp@@ -18,11 +18,15 @@// device. STRATA_KVA=0 brings back the exact 0.2.6 prompt path and the 0.2.6 draft window. Everything is read from// the environment once per process.//-// The two facts the engine tells this header at start-up (kva_engine()): whether a draft layer (MTP) is loaded and-// how many cells it attends to, and whether the engine configuration rules the read out for the whole process-// (batch slots, a layer split). The fully computed suffix must cover the drafter's window plus the 64 cells its-// catch-up reaches back (MtpDrafter::first_needed), or the drafter would build its K/V from approximated cells-// (that costs acceptance, never correctness: the target model verifies every token).+// What the engine tells this header at start-up (kva_engine()): whether its configuration rules the read out for the+// whole process (batch slots, a layer split, a peer device).+//+// The draft layer (MTP) builds its K/V from the final rows of the prompt, which an approximated chunk does not have.+// Its window is NOT changed: the draft layer's attention starts at the "cell floor" (MtpDrafter::set_cell_floor), the+// first cell built from a real row, and the cells below it are neither built nor read. A session KVA did not+// approximate has a floor of 0 and is bit for bit what STRATA_KVA=0 runs (kva_floor.hpp keeps the floor with the+// tokens of the session, window_start.hpp is the one rule the kernel and the tests share). The target model verifies+// every token whatever the draft read.#pragma once#include <algorithm>@@ -41,17 +45,11 @@ inline constexpr bool kKvaBackend = false;inline constexpr bool kKvaBackend = true;#endif-/// The draft layer's window (--mtp-window) the engine picks when KVA is on, the draft layer is loaded and the user-/// gave no --mtp-window: 2048 cells (suffix 2112). The 0.2.6 default was 32768, i.e. every cell of a chat up to-/// that length, which no suffix could cover.-inline constexpr int64_t kKvaDrafterWindow = 2048;-/// The fully computed tail when no draft layer needs more.+/// The fully computed tail of every approximated read.inline constexpr int64_t kKvaSuffixDefault = 2048;/// What the running engine knows and the environment cannot say.struct KvaEngine {- bool has_drafter = false; ///< an MTP draft layer is loaded (--mtp with --spec 2 or more)- int64_t drafter_window = 0; ///< the cells it attends to; 0 = every cellstd::string blocked; ///< non-empty: this process never reads approximately, and why};@@ -71,7 +69,7 @@ struct KvaConfig {bool oracle_fp32 = false; ///< STRATA_KVA_ORACLE_FP32=1: the oracle files are fp32 (else bf16)bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full pathint64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)- int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: effective_suffix())+ int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: kKvaSuffixDefault)int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full pathMode mode = Mode::Identity; ///< STRATA_KVA_MODE=identity|oracle|linearOnly only = Only::All; ///< STRATA_KVA_ONLY=all|gdn|qsa: which late layers take the approximate path@@ -79,14 +77,13 @@ struct KvaConfig {std::string capture_dir; ///< STRATA_KVA_CAPTUREstd::string projector; ///< STRATA_KVA_PROJECTOR- /// The fully computed tail of a read. STRATA_KVA_SUFFIX when set; otherwise 2048, or the draft layer's window +- /// 64 when that is larger. (A draft layer that attends to every cell has no suffix that covers it:- /// decline() says so.)- int64_t effective_suffix(const KvaEngine& e) const {- if (suffix_env > 0) return suffix_env;- int64_t s = kKvaSuffixDefault;- if (e.has_drafter && e.drafter_window > 0) s = std::max<int64_t>(s, e.drafter_window + 64);- return s;+ /// The fully computed tail of a read: STRATA_KVA_SUFFIX when set, otherwise 2048.+ int64_t effective_suffix() const { return suffix_env > 0 ? suffix_env : kKvaSuffixDefault; }++ /// Does the draft layer need the cell floor at all? Only when a read can approximate something: KVA on, not ruled+ /// out for the process, and a draft layer loaded. Otherwise the draft layer's attention is 0.2.6's own code.+ bool wants_cell_floor(const KvaEngine& e, bool has_drafter) const {+ return kKvaBackend && on && e.blocked.empty() && has_drafter;}/// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a@@ -110,13 +107,11 @@ struct KvaConfig {if (lb != 0 || (le >= 0 && le != n_layers)) return "a layer split (not the whole trunk)";if (pictures) return "a prompt with pictures";if (taps) return "the DFlash2 drafter reads the late layers' hidden states";- if (suffix_env <= 0 && e.has_drafter && e.drafter_window <= 0)- return "the draft layer attends to every cell (set --mtp-window)";if (n < min_prompt) {if (routine) *routine = true;return "the prompt is shorter than STRATA_KVA_MIN_PROMPT";}- if (n <= effective_suffix(e) + 64) {+ if (n <= effective_suffix() + 64) {if (routine) *routine = true;return "the suffix leaves too few positions to approximate";}diff --git a/include/strata/prefill/kva_floor.hpp b/include/strata/prefill/kva_floor.hppnew file mode 100644index 000000000..50189c7dc--- /dev/null+++ b/include/strata/prefill/kva_floor.hpp@@ -0,0 +1,72 @@+// include/strata/prefill/kva_floor.hpp - which cells of a session were built from approximated rows (KVA).+//+// A KVA read leaves the draft layer's K/V with cells [0, floor) built from rows the late layers never computed. The+// draft layer must not attend to them (it would only cost draft acceptance, but that is the decode speed), and a+// session that KVA did not approximate must not lose any draft context at all. The floor therefore follows the+// SESSION'S TOKENS, not the process: after a KVA read the registry remembers the prefix (its token ids) that was+// approximated; whatever state a later request continues from (the live session, a conversation checkpoint, a parked+// conversation), the floor of that state is how much of its tokens agree with a remembered prefix, capped by the+// prefix's floor. A state no KVA read produced has no common prefix, so its floor is 0 and its draft layer attends+// to the whole window, as with STRATA_KVA=0.+//+// A mislabel can only cost draft cells (an exact state that shares a prefix with an approximated one reports the+// floor too high); it can never make the draft attend to approximated cells that were registered, and the target+// model verifies every token whatever the draft read.+#pragma once++#include <algorithm>+#include <cstdint>+#include <vector>++namespace strata::prefill {++class KvaFloorRegistry {+public:+ explicit KvaFloorRegistry(size_t cap = 8) : cap_(cap) {}++ /// A KVA read approximated the cells [0, floor) of the prompt `ids` (at least `floor` tokens).+ template <class Token>+ void note(const Token* ids, int64_t floor) {+ if (floor <= 0) return;+ ++clock_;+ for (Entry& e : v_)+ if ((int64_t) e.ids.size() == floor && std::equal(e.ids.begin(), e.ids.end(), ids)) { e.used = clock_; return; }+ Entry e;+ e.ids.assign(ids, ids + floor);+ e.used = clock_;+ if (v_.size() >= cap_) {+ size_t lru = 0;+ for (size_t i = 1; i < v_.size(); ++i) if (v_[i].used < v_[lru].used) lru = i;+ v_.erase(v_.begin() + (std::ptrdiff_t) lru);+ }+ v_.push_back(std::move(e));+ }++ /// The first cell of a state holding the tokens ids[0, n) that is NOT an approximated one: the longest common+ /// prefix with a remembered read, over the remembered reads. 0 when the tokens match none.+ template <class Token>+ int64_t floor_for(const Token* ids, int64_t n) const {+ int64_t best = 0;+ for (const Entry& e : v_) {+ const int64_t m = std::min<int64_t>(n, (int64_t) e.ids.size());+ int64_t i = 0;+ while (i < m && (int64_t) ids[i] == e.ids[(size_t) i]) ++i;+ best = std::max(best, i);+ }+ return best;+ }++ size_t size() const { return v_.size(); }+ void clear() { v_.clear(); }++private:+ struct Entry {+ std::vector<int64_t> ids;+ uint64_t used = 0;+ };+ std::vector<Entry> v_;+ size_t cap_;+ uint64_t clock_ = 0;+};++} // namespace strata::prefilldiff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hppindex f9bbca027..39ed85577 100644--- a/include/strata/prefill/prefill.hpp+++ b/include/strata/prefill/prefill.hpp@@ -209,6 +209,10 @@ public:/// for its grouping on the host; a policy may count them. Not called on the paths that keep the ids on the GPU.std::function<void(int64_t layer, const int32_t* ids, int64_t n)> on_routes;int64_t stopped_at() const { return stopped_at_; }+ /// KVA: the end of the positions the last run approximated (they are a prefix: chunks from the run's first position+ /// on), i.e. the first position whose final rows are real; 0 when that run approximated nothing. Raised chunk by+ /// chunk, before on_chunk reports the chunk, so the draft layer can skip the chunk's cells. Reset by every run.+ int64_t kva_floor() const { return kva_floor_; }/// Grows this stage's (and the later stages') pinned upload slots to what a text chunk of `T` tokens uploads, so/// a run of such chunks allocates no pinned memory (STRATA_PIPED_CONCURRENT: an allocation or free on the read's/// thread takes the driver's lock). Before the run, with nothing of this path in flight. False: a slot could@@ -270,6 +274,7 @@ private:Prefill* helper_ = nullptr; ///< set_stage_helperbool single_chunk_ = false; ///< a later stage: the prompt is one chunk (set by the stage before)int64_t stopped_at_ = -1; ///< stop_at_chunk ended the last run after this many tokens (-1: it read all)+ int64_t kva_floor_ = 0; ///< kva_floor()bool pipe_chunked_ = false; ///< a later stage: the chunks came from a --prefill-pipe plan (set likewise)int64_t force_chunk_ = 0; ///< force_chunk, for the next run (0: none)bool force_single_ = false, force_planned_ = false;diff --git a/metal/src/engine/stubs.cpp b/metal/src/engine/stubs.cppindex b6615813c..7facf9a29 100644--- a/metal/src/engine/stubs.cpp+++ b/metal/src/engine/stubs.cpp@@ -325,6 +325,7 @@ namespace strata::kernels { void topk_512_step(const float * p0, const QsaShapesnamespace strata::kernels { void wait_flag_ge(const uint32_t * p0, uint32_t p1, void * p2) { strata::metal_engine::missing("strata::kernels::wait_flag_ge"); } }namespace strata::kernels { void wait_flag_ge_or(const uint32_t * p0, uint32_t p1, const uint32_t * p2, void * p3) { strata::metal_engine::missing("strata::kernels::wait_flag_ge_or"); } }namespace strata::kernels { void window_ids(int32_t * p0, int p1, int p2, int32_t * p3, int64_t p4, void * p5) { strata::metal_engine::missing("strata::kernels::window_ids"); } }+namespace strata::kernels { void window_ids_floor(int32_t * p0, int p1, int p2, int32_t * p3, int64_t p4, const int32_t * p5, void * p6) { strata::metal_engine::missing("strata::kernels::window_ids_floor"); } }auto strata::prefill::Gemm::bf16(const uint16_t * p0, const uint16_t * p1, float * p2, int64_t p3, int64_t p4, int64_t p5, int64_t p6, float p7, int64_t p8, bool p9) -> void { strata::metal_engine::missing("strata::prefill::Gemm::bf16"); }auto strata::prefill::Gemm::bf16_converts_x() const -> bool { strata::metal_engine::missing("strata::prefill::Gemm::bf16_converts_x"); return {}; }auto strata::prefill::Gemm::bf16_hcd_exact(const uint16_t * p0, int64_t p1, const uint16_t * p2, float * p3, int64_t p4, int64_t p5, int64_t p6) -> bool { strata::metal_engine::missing("strata::prefill::Gemm::bf16_hcd_exact"); return {}; }diff --git a/src/core/mtp.cpp b/src/core/mtp.cppindex f4b82b334..292a5d2b4 100644--- a/src/core/mtp.cpp+++ b/src/core/mtp.cpp@@ -163,7 +163,7 @@ MtpDrafter::~MtpDrafter() {if (owns_draft_head_ && dvocab_) cudaFree(dvocab_);if (ev_chain_) cudaEventDestroy(ev_chain_);for (cudaEvent_t e : ev_step_) if (e) cudaEventDestroy(e);- void* hosts[] = {h_tok_, h_step_, h_pos_, h_row_, h_out_, h_prob_, h_force_};+ void* hosts[] = {h_tok_, h_step_, h_pos_, h_row_, h_out_, h_prob_, h_force_, h_floor_};for (void* h : hosts) if (h) cudaFreeHost(h);}@@ -370,6 +370,7 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session// ---- bufferswindow_ = (window > 0 && window < max_cells) ? window : 0;+ if (floor_on_ && !mapped(64, (void**) &h_floor_, (void**) &m_floor_)) { err = "mtp: the window floor does not fit"; return false; }cap_ = (((window_ > 0 ? window_ : max_cells) + 63) / 64) * 64;attn_scratch_floats_ = (int64_t) strata::kernels::qsa_decode_attn_scratch_floats(cap_, s);const uint64_t T = (uint64_t) max_t, N = (uint64_t) g.n_embd, HC = (uint64_t) g.hc, K = (uint64_t) ss.k;@@ -1027,7 +1028,9 @@ bool MtpDrafter::record_rest(int step_row, cudaStream_t cs, std::string& err) {if (st_.kv_rot) fwht256_inplace_cuda(qcur_, (int64_t) T * NH, cs);g_mp.stamp(5, cs);const QsaAttnPools pools = qsa_attn_pools(st_);- if (window_ > 0) window_ids(const_cast<int32_t*>(step), T, (int) window_, ident_, cap_, cs);+ if (floor_on_ && m_floor_ != nullptr) // KVA: the window (or every cell) from the first real draft cell on+ window_ids_floor(const_cast<int32_t*>(step), T, window_ > 0 ? (int) window_ : 0x7fffffff, ident_, cap_, m_floor_, cs);+ else if (window_ > 0) window_ids(const_cast<int32_t*>(step), T, (int) window_, ident_, cap_, cs);// STRATA_QSA_SPLIT_ATTN=1: the draft layer's attention over its window, split the same waystatic const bool split_attn = [] { const char* v = std::getenv("STRATA_QSA_SPLIT_ATTN"); return v != nullptr && std::atoi(v) != 0; }();if (!(split_attn && qsa_decode_attn_split(qcur_, pools, ident_, step, cap_, s, attn_scratch_, (uint64_t) attn_scratch_floats_, attn_, T, cs)))@@ -1423,7 +1426,7 @@ bool MtpDrafter::prefill(const float* R_rows, const int32_t* next_tokens, int64_const Clock::time_point t0 = Clock::now();const int64_t HCN = g_->hc * g_->n_embd;// cells the window can never reach again need no K/V- const int64_t first_needed = (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0;+ const int64_t first_needed = this->first_needed();// E-4: every group's token / step / position records uploaded at once; each group is then device copies and a// graph on the one stream, with a single sync at the end (a group of <= max_t rows used to be staged in mapped// memory and synced before the next: ~5,500 host round trips on a 32K prompt). The same work in the samediff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cuindex a9f570b9e..265664759 100644--- a/src/kernels/cuda/verify_kernels.cu+++ b/src/kernels/cuda/verify_kernels.cu@@ -6,6 +6,7 @@#include "strata/core/geometry.hpp"#include "strata/core/emulate.hpp"#include "strata/kernels/q8_1_finite.hpp" // #606: q8_1_ds+#include "strata/kernels/window_start.hpp" // KVA: the draft layer's window floor#include "strata/kernels/dp4a.hpp"#include "strata/kernels/pdl.hpp"@@ -1016,6 +1017,30 @@ void window_ids(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_strcheck("window_ids");}+namespace {+// window_ids with a floor read from (mapped) memory when the kernel RUNS: the captured graph keeps the pointer, the+// value changes between sessions. A floor of 0 gives window_ids' selection exactly (strata::kernels::window_start).+__global__ void window_ids_floor_kernel(int32_t* steps, int window, int32_t* ids, long long stride,+ const volatile int32_t* floor_cell) {+ const int q = blockIdx.y;+ int32_t* st = steps + q * 4;+ const int n_kv = st[1];+ const int start = strata::kernels::window_start(n_kv, window, *floor_cell);+ const int width = n_kv - start;+ for (int j = blockIdx.x * blockDim.x + threadIdx.x; j < width; j += gridDim.x * blockDim.x)+ ids[q * stride + j] = start + j;+ __syncthreads();+ if (blockIdx.x == 0 && threadIdx.x == 0) st[3] = width;+}+} // namespace++void window_ids_floor(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_stride, const int32_t* floor_cell,+ void* stream) {+ window_ids_floor_kernel<<<dim3(8, (unsigned) n), 256, 0, (cudaStream_t) stream>>>(steps, window, ids, (long long) ids_stride,+ floor_cell);+ check("window_ids_floor");+}+void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream) {dense_steps_kernel<<<1, 64, 0, (cudaStream_t) stream>>>(cells, n, steps);check("dense_steps");diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cppindex bd468c34d..937d0ca14 100644--- a/src/prefill/kva_test.cpp+++ b/src/prefill/kva_test.cpp@@ -1,10 +1,13 @@// src/prefill/kva_test.cpp - the KVA switch, the automatic tail and the reasons a read is turned away (no GPU needed).+#include "strata/kernels/window_start.hpp"#include "strata/prefill/kva.hpp"+#include "strata/prefill/kva_floor.hpp"#include <cstdio>#include <cstdlib>#include <cstring>#include <string>+#include <vector>using namespace strata::prefill;@@ -50,25 +53,13 @@ int main() {{ const KvaConfig c = kva_config_from_env(); CHECK(c.on && !c.explicit_on); }clear_all();- // ---- the automatic tail: 2048, or the draft layer's window + 64 when that is larger; STRATA_KVA_SUFFIX wins+ // ---- the tail: 2048, or STRATA_KVA_SUFFIX; the draft window plays no part (the cell floor protects the draft layer){const KvaConfig c = kva_config_from_env();- KvaEngine none; // no draft layer- CHECK(c.effective_suffix(none) == kKvaSuffixDefault);- KvaEngine d; // the window the engine picks (no --mtp-window given)- d.has_drafter = true;- d.drafter_window = kKvaDrafterWindow;- CHECK(c.effective_suffix(d) == 2112);- d.drafter_window = 8192; // --mtp-window 8192- CHECK(c.effective_suffix(d) == 8256);- d.drafter_window = 512; // a small window: the default tail still holds- CHECK(c.effective_suffix(d) == 2048);- d.drafter_window = 0; // every cell: no tail can cover it, the plain tail is returned- CHECK(c.effective_suffix(d) == 2048);+ CHECK(c.effective_suffix() == kKvaSuffixDefault && kKvaSuffixDefault == 2048);put("STRATA_KVA_SUFFIX", "3000");const KvaConfig e = kva_config_from_env();- d.drafter_window = 8192;- CHECK(e.effective_suffix(d) == 3000);+ CHECK(e.effective_suffix() == 3000);clear_all();}@@ -76,8 +67,6 @@ int main() {{const KvaConfig c = kva_config_from_env();KvaEngine eng;- eng.has_drafter = true;- eng.drafter_window = kKvaDrafterWindow;bool routine = false;// a cold 14.6K prompt on one device: appliesCHECK(c.decline(eng, 0, 14600, 0, -1, 48) == nullptr);@@ -102,16 +91,7 @@ int main() {put("STRATA_KVA", "1");{ const KvaConfig f = kva_config_from_env(); CHECK(f.decline(eng, 0, 14600, 0, -1, 40, false, false, nullptr, true) == nullptr); }clear_all();- // a draft layer that attends to every cell has no tail that covers it- KvaEngine all = eng;- all.drafter_window = 0;- CHECK(is(c.decline(all, 0, 14600, 0, -1, 48), "the draft layer attends to every cell (set --mtp-window)"));- put("STRATA_KVA_SUFFIX", "2112"); // ... unless the user fixes the tail by hand- { const KvaConfig s = kva_config_from_env(); CHECK(s.decline(all, 0, 14600, 0, -1, 48) == nullptr); }- clear_all();// no draft layer at all: nothing to cover- KvaEngine nod;- CHECK(c.decline(nod, 0, 14600, 0, -1, 48) == nullptr);// the engine rules the whole process out (batch slots, a layer split, a peer device)KvaEngine blocked = eng;blocked.blocked = "batch slots (--batch)";diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cppindex 741c3d8fb..8fd781a5a 100644--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@ -2717,7 +2717,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri// stage, no capture). Positions [0, kva_frontier) are approximated in the layers at and after the split.const KvaConfig& kva = kva_config();const KvaEngine& kva_eng = kva_engine();- const int64_t kva_suffix = kva.effective_suffix(kva_eng);+ const int64_t kva_suffix = kva.effective_suffix();bool kva_routine = false;const char* kva_why_not = kva.decline(kva_eng, pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers,embd_rows != nullptr, !tap_layers_.empty(), &kva_routine,@@ -2728,6 +2728,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stristatic KvaTap kva_capture; // process-wide: one cold read at a time, so the per-layer position is enoughstatic KvaTap kva_oracle;int64_t kva_positions = 0, kva_chunks = 0; // what actually took the approximate path (diagnostics)+ kva_floor_ = 0; // this run's: raised below as chunks are approximatedif (kva.capture && kva_capture.f.empty()) kva_capture.open(kva.capture_dir, kva.capture_fp32);if (kva.capture && pos0 == 0 && hand_in_ == nullptr) kva_capture.begin_prompt(); // one prompt per captureif (kva_run && kva.mode == KvaConfig::Mode::Oracle && kva_oracle.f.empty())@@ -2862,6 +2863,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::striif (kva_approx) {kva_positions += T;++kva_chunks;+ kva_floor_ = p0 + T; // the chunks are a prefix of the run: this is the first position with real rows}core::progress_at("reading the prompt (batched): preparing the chunk from token", p0); // #251#if defined(STRATA_HAVE_DFLASH2)diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 82a6c9457..dbc5b6b02 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -73,6 +73,7 @@#include "strata/core/draft_cap.hpp"#endif#include "strata/prefill/kva.hpp"+#include "strata/prefill/kva_floor.hpp"#include "strata/prefill/prefill.hpp"#if defined(STRATA_PREFILL_MMQ)#include "strata/prefill/moe_mmq.hpp"@@ -682,7 +683,6 @@ struct Options {/// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it.std::string mtp;int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell)- bool mtp_window_given = false; ///< --mtp-window was given (KVA picks the window itself only when it was not)/// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the/// pinned arena while the CPU computes the rest (verify windows).double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs)@@ -2629,7 +2629,7 @@ int main(int argc, char** argv) {else if (a == "--spec-follow") o.spec_follow = next("--spec-follow");else if (a == "--window-hashes") o.window_hashes = next("--window-hashes");else if (a == "--mtp") o.mtp = next("--mtp");- else if (a == "--mtp-window") { o.mtp_window = std::atoll(next("--mtp-window")); o.mtp_window_given = true; }+ else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window"));else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac"));else if (a == "--adapt-every") { o.adapt_every = std::atoi(next("--adapt-every")); o.adapt_every_given = true; }else if (a == "--adapt-decay") o.adapt_decay = (float) std::atof(next("--adapt-decay"));@@ -5512,9 +5512,9 @@ int main(int argc, char** argv) {// KVA (0.2.7: on by default, STRATA_KVA=0 is the exact 0.2.6 path): what only the engine knows. A layer split and// the batch slots read their prompts in pieces and in stages, which the one-device, one-cold-read rule of the// approximate prompt read cannot follow, so they stay on the exact path (and say so here, once). The draft layer- // (MTP) must find exact cells in its window: with no --mtp-window given the window becomes 2048 cells (the 0.2.6- // default, 32768, is every cell of a chat up to that length, which no fully computed tail could cover) and the- // tail 2112; a window the user gave keeps its value and sets the tail (window + 64).+ // (MTP) keeps its window, whatever it is: a session KVA approximated tells it the first cell built from a real row+ // (the cell floor, set per request below) and every session KVA did not approximate attends exactly as with+ // STRATA_KVA=0.{const strata::prefill::KvaConfig& kc = strata::prefill::kva_config();strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();@@ -5529,15 +5529,8 @@ int main(int argc, char** argv) {if (kc.explicit_on)std::fprintf(stderr, "strata generate: KVA (the approximate late-layer prompt read) is off here: %s\n",ke.blocked.c_str());- } else if (!o.mtp.empty() && o.spec >= 2 && !o.mtp_window_given && kc.suffix_env <= 0) {- if (o.mtp_window != strata::prefill::kKvaDrafterWindow)- std::fprintf(stderr, "strata generate: KVA on (the faster prompt read for long cold prompts): the draft layer "- "attends to the last %lld cells (--mtp-window %lld); STRATA_KVA=0 keeps the "- "exact 0.2.6 prompt read and its %lld-cell window\n",- (long long) strata::prefill::kKvaDrafterWindow, (long long) strata::prefill::kKvaDrafterWindow,- (long long) o.mtp_window);- o.mtp_window = strata::prefill::kKvaDrafterWindow;}+ mtp.enable_cell_floor(kc.wants_cell_floor(ke, !o.mtp.empty() && o.spec >= 2));}}if (!o.mtp.empty()) {@@ -5559,11 +5552,6 @@ int main(int argc, char** argv) {if (o.pipeline_windows >= 2) mtp.set_force_capture(true);if (!o.mtp.empty() && !mtp.load(o.mtp, draft_geometry, last_st && !mtp_first ? last_st->ss : ss, mtp_t, err, o.mtp_window)) { std::fprintf(stderr, "strata generate: %s%s\n", err.c_str(), vram_free_note().c_str()); return 1; }mtp.set_ple_session(&ss);- if (!o.mtp.empty()) { // KVA: the cells the draft layer attends to now (0 = every cell) set the tail that must be exact- strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();- ke.has_drafter = true;- ke.drafter_window = mtp.window();- }#if defined(STRATA_HAVE_DFLASH2)if (o.draft == "dflash2") {const char* why = o.mtp.empty() ? "it needs --mtp (the fallback) and --spec 2 or more"@@ -9473,6 +9461,9 @@ int main(int argc, char** argv) {// a reporter's prompts failed there every time and passed with --short-read 0.bool nf_prompt_windows = false, short_read_off_once = false;std::vector<ConvCheckpoint> checks;+ // KVA: which prefixes of token ids a KVA read approximated, so the draft layer knows, for whatever state a request+ // continues from, the first cell built from a real row (kva_floor.hpp). Empty without KVA.+ strata::prefill::KvaFloorRegistry kva_floors;// STRATA_CKPT_POOL: the dropped checkpoints' storage, reused. STRATA_SLOT_CKPT_MOVE turns it on too: a moved// checkpoint frees nothing at the next prune, so every save would fault in fresh pages (17 -> 40-60 ms)CheckpointPool ckpt_pool(ckpt_pool_env() || slot_ckpt_move_env());@@ -9982,6 +9973,9 @@ int main(int argc, char** argv) {// come from Q8_1 x Q8_0 MMQ rather than mmvq: drafts may differ, the target's tokens never.static const bool split_mtp_batch = strata::program::draft_kv::split_env_on(std::getenv("STRATA_SPLIT_MTP_BATCH"));sp.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool {+ // KVA: a chunk the late layers did not compute has no real final rows: its draft cells are neither built+ // nor read (the floor rises before the cells are built, so they are skipped)+ if (use_mtp && mtp.cell_floor_enabled() && sp.kva_floor() > mtp.cell_floor()) mtp.set_cell_floor(sp.kva_floor());std::vector<int32_t> nxt((size_t) T);for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) cur[(size_t) (p0 + t + 1)];// E-9: batched through the prompt path when it can (a layer split: only with STRATA_SPLIT_MTP_BATCH=1)@@ -13161,6 +13155,10 @@ int main(int argc, char** argv) {tr("request", n, geni ? 1 : 0);req_times = ReqTimes{};if (use_mtp) mtp.set_prompt_len(n);+ // KVA: the draft layer's first real cell for the state this request continues from (0 for a cold read - the+ // first segment raises it if KVA approximates it - and for any state no KVA read produced)+ if (use_mtp && mtp.cell_floor_enabled())+ mtp.set_cell_floor(resume > 0 && reread_to <= 0 ? kva_floors.floor_for(ids.data(), resume) : 0);const int64_t read_from = reread_to > 0 ? 0 : resume;conversations.limit_reuse(read_from);pp_total = n;@@ -13603,7 +13601,12 @@ int main(int argc, char** argv) {} seg_plan_scope{seg_plan};auto run = [&](int64_t q, int64_t r) -> bool {if (planned) sp.force_chunk(plan.chunk, plan_single, plan.chunked);- return sp.run(ids.data() + q, r - q, q, e);+ const bool run_ok = sp.run(ids.data() + q, r - q, q, e);+ if (run_ok && q == 0 && sp.kva_floor() > 0) { // KVA approximated the start of this prompt: remember which tokens+ kva_floors.note(ids.data(), sp.kva_floor());+ if (use_mtp && mtp.cell_floor_enabled()) mtp.set_cell_floor(sp.kva_floor());+ }+ return run_ok;};// (a layer split reads its stages as a pipeline over one run's chunks: beside slots a chunk at a time, with// none decoding in pieces of STRATA_BATCH_YIELD_CHUNKS chunks (default 8; 0 = one run) so a BYIELD is taken)@@ -15944,6 +15947,7 @@ int main(int argc, char** argv) {return 1;}prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool {+ if (mtp.cell_floor_enabled() && prefill.kva_floor() > mtp.cell_floor()) mtp.set_cell_floor(prefill.kva_floor()); // KVA// cell i pairs R_i with the token at i + 1 (every such token is in the prompt)std::vector<int32_t> nxt((size_t) T);for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) o.tokens[(size_t) (p0 + t + 1)];