Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

KVA: leave the draft window alone; a per-session cell floor keeps the draft layer off approximated cells

Niko12217a34c60d61c7
diff --git a/docs/KVA.md b/docs/KVA.md
index d07651f3c..c53c6abde 100644
--- a/docs/KVA.md
+++ b/docs/KVA.md
@@ -28,20 +28,28 @@ configuration rules KVA out for the whole run.
## The fully computed tail and the draft layer
-The last `suffix` positions are always read exactly. The MTP draft layer builds its K/V from the final
-residuals of the prompt, and those rows are only real where the layers ran in full, so the tail must cover the
-draft layer's window plus the 64 cells its catch-up reaches back:
-
-- no `--mtp-window` given (what the app and the Terminal write): with KVA on, the engine sets the draft window to
- 2048 cells and the tail to 2112 tokens. The 0.2.6 window was 32768, which is every cell of a chat up to that
- length and could never be covered. The engine prints one line at start-up when it does this.
-- `--mtp-window N` given: the window stays N and the tail is N + 64 (a 32768-cell window never leaves a
- prompt shorter than 32 K approximated; the log says "the draft layer attends to every cell" when no tail can cover it).
-- `STRATA_KVA_SUFFIX=N` fixes the tail by hand.
-- no draft layer (`--mtp` not given): the tail is 2048.
-
-A draft layer that reads an approximated cell costs only draft acceptance (speed); the target model verifies every
-emitted token, so the answer does not depend on it.
+The last `suffix` positions (2048, or `STRATA_KVA_SUFFIX`) are always read exactly.
+
+The MTP draft layer builds its K/V from the final residuals of the prompt, and an approximated chunk has none. The
+draft layer's window is **not changed**: it stays what it was (`--mtp-window`, default 32768 cells). Instead the
+engine keeps a *cell floor* per session, the first draft cell built from a real row:
+
+- A read that KVA approximated raises the floor to the end of its approximated chunks. The draft layer builds no
+ cell below it and its attention starts no lower, so it reads the window `[max(floor, pos - window), pos)`: the
+ exactly computed tail, growing with the conversation, never an approximated cell. The target model verifies
+ every token either way.
+- The floor follows the *tokens* of the state a request continues from (the live session, a conversation checkpoint,
+ a parked conversation): the engine remembers which token prefixes a KVA read approximated, and a state's floor
+ is how much of its tokens agree with one of them. A state no KVA read produced (a short chat, another document,
+ every prompt of a `STRATA_KVA=0` run) has floor 0.
+- With floor 0 the draft layer attends to exactly the window it attended to with `STRATA_KVA=0`
+ (`window_start`, unit tested for every length). The only added work is one tiny kernel that picks the cells.
+ With `STRATA_KVA=0`, with the engine configurations that rule KVA out and with a build without a draft layer,
+ the draft layer's code is 0.2.6's own.
+
+Measured on the P100 (14K prompt, 256 tokens, auto cache and prefill), the draft layer's window made no difference
+to decode (38.7 tok/s with the 32768-cell window, 40.1 with 2048); on faster cards a smaller window costs
+acceptance, which is why the window is left alone.
## Chats, follow-up turns and checkpoints (`--serve`)
@@ -76,7 +84,7 @@ there, with a line when it is set).
|---|---:|---|
| `STRATA_KVA` | on | `0` turns KVA off (the exact 0.2.6 prompt read). `1` is the same as the default but logs every read it turned away |
| `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |
-| `STRATA_KVA_SUFFIX` | 2112 with the default draft window | fully computed tail positions; see above |
+| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; see above |
| `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path |
| `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type |
| `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) |
@@ -88,7 +96,7 @@ there, with a line when it is set).
Every read says what it did:
-- applied: `strata prefill: KVA on: split 24, suffix 2112, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`
+- applied: `strata prefill: KVA on: split 24, suffix 2048, approximated 5120 of 7533 positions in 5 of 8 chunks, mode identity`
- turned away: `strata prefill: KVA configured but NOT applied: <reason> (prompt … positions, min_prompt …, suffix …, split …)`
A quality comparison needs the control arm to say `STRATA_KVA=0` (an unset variable is now KVA).
diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp
index f63aea4de..38fe8f011 100644
--- a/include/strata/core/mtp.hpp
+++ b/include/strata/core/mtp.hpp
@@ -26,6 +26,7 @@
#include <cuda_runtime.h>
+#include <algorithm>
#include <cstdint>
#include <functional>
#include <string>
@@ -166,9 +167,25 @@ public:
const uint16_t* tensor_bf16(const char* name) const { return bf16(name); }
const void* tensor_q8(const char* name) const { return q8(name); }
QsaState& kv_state_rw() { return st_; }
- int64_t first_needed() const { return (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0; }
+ /// The first cell whose K/V the draft layer can still read: the window's start for this prompt, and never below
+ /// the KVA floor (cells built from approximated rows are not read, so they are not built either).
+ int64_t first_needed() const {
+ const int64_t w = (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0;
+ return std::max<int64_t>(w, cell_floor_);
+ }
/// The cells the draft layer attends to (0 = every cell), as load() settled it.
int64_t window() const { return window_; }
+ /// KVA: call before load(). The draft layer's attention then reads its cell range through a floor, which stays 0
+ /// (the plain window, cell for cell) until set_cell_floor() raises it. Off (the default): the code is 0.2.6's.
+ void enable_cell_floor(bool on) { floor_on_ = on; }
+ bool cell_floor_enabled() const { return floor_on_ && m_floor_ != nullptr; }
+ /// KVA: the first draft cell that holds a K/V built from a real (not approximated) row. The attention starts no
+ /// lower; draft_kv / prefill build no cell below it. Between requests (the drafter idle). 0 = every cell is real.
+ void set_cell_floor(int64_t cell) {
+ cell_floor_ = std::max<int64_t>(0, cell);
+ if (m_floor_ != nullptr) *(volatile int32_t*) h_floor_ = (int32_t) std::min<int64_t>(cell_floor_, 0x7fffffff);
+ }
+ int64_t cell_floor() const { return cell_floor_; }
int device() const { return device_; }
bool idle(std::string& err) {
if (cs_ && cudaStreamSynchronize(cs_) != cudaSuccess) { err = "mtp: its stream failed"; return false; }
@@ -273,6 +290,9 @@ private:
int64_t n_dvocab_ = 0;
std::string rt_dir_;
int64_t window_ = 0; // attention over the last window_ cells (0 = every cell)
+ int64_t cell_floor_ = 0; // KVA: no draft cell below this one is read or built (set_cell_floor)
+ bool floor_on_ = false; // enable_cell_floor: the attention goes through window_ids_floor
+ int32_t *h_floor_ = nullptr, *m_floor_ = nullptr; // the floor, mapped: the kernel reads it when it runs
int64_t prompt_len_ = 0;
const int32_t* rr_res_ = nullptr;
float rr_margin_ = 0.0f;
diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp
index a5e35181e..b2ff4a0d4 100644
--- a/include/strata/kernels/verify_kernels.hpp
+++ b/include/strata/kernels/verify_kernels.hpp
@@ -192,5 +192,10 @@ void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream);
/// A sliding attention window: for `n` step records (kStepCount ints each, n_kv at [1]) the selection becomes the
/// last `window` cells: ids[q * ids_stride + j] = max(0, n_kv - window) + j and the record's width = the count.
void window_ids(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_stride, void* stream);
+/// window_ids whose window starts no lower than the cell `*floor_cell` (device-visible memory, read when the kernel
+/// runs; KVA: the first draft cell built from a real row). `window` may be INT_MAX for "every cell". A floor of 0
+/// selects exactly what window_ids selects. CUDA / HIP only.
+void window_ids_floor(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_stride, const int32_t* floor_cell,
+ void* stream);
} // namespace strata::kernels
diff --git a/include/strata/kernels/window_start.hpp b/include/strata/kernels/window_start.hpp
new file mode 100644
index 000000000..8b0944350
--- /dev/null
+++ b/include/strata/kernels/window_start.hpp
@@ -0,0 +1,26 @@
+// include/strata/kernels/window_start.hpp - the first cell the draft layer's attention reads.
+//
+// The draft layer attends to the last `window` cells. KVA (docs/KVA.md) can leave cells in the draft layer's K/V
+// that were built from approximated rows; `floor` is the first cell that holds a real one, so the attention starts
+// there when it is higher. floor = 0 (every session KVA did not approximate) gives exactly the plain window.
+// One function for the device kernel and for the host test.
+#pragma once
+
+#if defined(__CUDACC__) || defined(__HIPCC__)
+#define STRATA_WINDOW_START_HD __host__ __device__
+#else
+#define STRATA_WINDOW_START_HD
+#endif
+
+namespace strata::kernels {
+
+/// The first cell of the attention for a sequence of `n_kv` cells. The width n_kv - start is at least 1 whenever
+/// n_kv is (a floor at or past the end must not leave the attention with nothing to read).
+STRATA_WINDOW_START_HD inline int window_start(int n_kv, int window, int floor) {
+ int start = n_kv > window ? n_kv - window : 0;
+ if (floor > start) start = floor;
+ if (start > n_kv - 1) start = n_kv > 0 ? n_kv - 1 : 0;
+ return start;
+}
+
+} // namespace strata::kernels
diff --git a/include/strata/prefill/kva.hpp b/include/strata/prefill/kva.hpp
index d9ce3dc07..80e8bdb0e 100644
--- a/include/strata/prefill/kva.hpp
+++ b/include/strata/prefill/kva.hpp
@@ -18,11 +18,15 @@
// device. STRATA_KVA=0 brings back the exact 0.2.6 prompt path and the 0.2.6 draft window. Everything is read from
// the environment once per process.
//
-// The two facts the engine tells this header at start-up (kva_engine()): whether a draft layer (MTP) is loaded and
-// how many cells it attends to, and whether the engine configuration rules the read out for the whole process
-// (batch slots, a layer split). The fully computed suffix must cover the drafter's window plus the 64 cells its
-// catch-up reaches back (MtpDrafter::first_needed), or the drafter would build its K/V from approximated cells
-// (that costs acceptance, never correctness: the target model verifies every token).
+// What the engine tells this header at start-up (kva_engine()): whether its configuration rules the read out for the
+// whole process (batch slots, a layer split, a peer device).
+//
+// The draft layer (MTP) builds its K/V from the final rows of the prompt, which an approximated chunk does not have.
+// Its window is NOT changed: the draft layer's attention starts at the "cell floor" (MtpDrafter::set_cell_floor), the
+// first cell built from a real row, and the cells below it are neither built nor read. A session KVA did not
+// approximate has a floor of 0 and is bit for bit what STRATA_KVA=0 runs (kva_floor.hpp keeps the floor with the
+// tokens of the session, window_start.hpp is the one rule the kernel and the tests share). The target model verifies
+// every token whatever the draft read.
#pragma once
#include <algorithm>
@@ -41,17 +45,11 @@ inline constexpr bool kKvaBackend = false;
inline constexpr bool kKvaBackend = true;
#endif
-/// The draft layer's window (--mtp-window) the engine picks when KVA is on, the draft layer is loaded and the user
-/// gave no --mtp-window: 2048 cells (suffix 2112). The 0.2.6 default was 32768, i.e. every cell of a chat up to
-/// that length, which no suffix could cover.
-inline constexpr int64_t kKvaDrafterWindow = 2048;
-/// The fully computed tail when no draft layer needs more.
+/// The fully computed tail of every approximated read.
inline constexpr int64_t kKvaSuffixDefault = 2048;
/// What the running engine knows and the environment cannot say.
struct KvaEngine {
- bool has_drafter = false; ///< an MTP draft layer is loaded (--mtp with --spec 2 or more)
- int64_t drafter_window = 0; ///< the cells it attends to; 0 = every cell
std::string blocked; ///< non-empty: this process never reads approximately, and why
};
@@ -71,7 +69,7 @@ struct KvaConfig {
bool oracle_fp32 = false; ///< STRATA_KVA_ORACLE_FP32=1: the oracle files are fp32 (else bf16)
bool parity = false; ///< STRATA_KVA_PARITY=1: report the state-only path against the full path
int64_t split = 24; ///< STRATA_KVA_SPLIT: first approximated layer (zero-based)
- int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: effective_suffix())
+ int64_t suffix_env = 0; ///< STRATA_KVA_SUFFIX: fully computed tail positions (0 = not set: kKvaSuffixDefault)
int64_t min_prompt = 4096; ///< STRATA_KVA_MIN_PROMPT: shorter prompts stay on the full path
Mode mode = Mode::Identity; ///< STRATA_KVA_MODE=identity|oracle|linear
Only only = Only::All; ///< STRATA_KVA_ONLY=all|gdn|qsa: which late layers take the approximate path
@@ -79,14 +77,13 @@ struct KvaConfig {
std::string capture_dir; ///< STRATA_KVA_CAPTURE
std::string projector; ///< STRATA_KVA_PROJECTOR
- /// The fully computed tail of a read. STRATA_KVA_SUFFIX when set; otherwise 2048, or the draft layer's window +
- /// 64 when that is larger. (A draft layer that attends to every cell has no suffix that covers it:
- /// decline() says so.)
- int64_t effective_suffix(const KvaEngine& e) const {
- if (suffix_env > 0) return suffix_env;
- int64_t s = kKvaSuffixDefault;
- if (e.has_drafter && e.drafter_window > 0) s = std::max<int64_t>(s, e.drafter_window + 64);
- return s;
+ /// The fully computed tail of a read: STRATA_KVA_SUFFIX when set, otherwise 2048.
+ int64_t effective_suffix() const { return suffix_env > 0 ? suffix_env : kKvaSuffixDefault; }
+
+ /// Does the draft layer need the cell floor at all? Only when a read can approximate something: KVA on, not ruled
+ /// out for the process, and a draft layer loaded. Otherwise the draft layer's attention is 0.2.6's own code.
+ bool wants_cell_floor(const KvaEngine& e, bool has_drafter) const {
+ return kKvaBackend && on && e.blocked.empty() && has_drafter;
}
/// Why this read cannot take the approximate path, or nullptr when it can: a cold read of one prompt (not a
@@ -110,13 +107,11 @@ struct KvaConfig {
if (lb != 0 || (le >= 0 && le != n_layers)) return "a layer split (not the whole trunk)";
if (pictures) return "a prompt with pictures";
if (taps) return "the DFlash2 drafter reads the late layers' hidden states";
- if (suffix_env <= 0 && e.has_drafter && e.drafter_window <= 0)
- return "the draft layer attends to every cell (set --mtp-window)";
if (n < min_prompt) {
if (routine) *routine = true;
return "the prompt is shorter than STRATA_KVA_MIN_PROMPT";
}
- if (n <= effective_suffix(e) + 64) {
+ if (n <= effective_suffix() + 64) {
if (routine) *routine = true;
return "the suffix leaves too few positions to approximate";
}
diff --git a/include/strata/prefill/kva_floor.hpp b/include/strata/prefill/kva_floor.hpp
new file mode 100644
index 000000000..50189c7dc
--- /dev/null
+++ b/include/strata/prefill/kva_floor.hpp
@@ -0,0 +1,72 @@
+// include/strata/prefill/kva_floor.hpp - which cells of a session were built from approximated rows (KVA).
+//
+// A KVA read leaves the draft layer's K/V with cells [0, floor) built from rows the late layers never computed. The
+// draft layer must not attend to them (it would only cost draft acceptance, but that is the decode speed), and a
+// session that KVA did not approximate must not lose any draft context at all. The floor therefore follows the
+// SESSION'S TOKENS, not the process: after a KVA read the registry remembers the prefix (its token ids) that was
+// approximated; whatever state a later request continues from (the live session, a conversation checkpoint, a parked
+// conversation), the floor of that state is how much of its tokens agree with a remembered prefix, capped by the
+// prefix's floor. A state no KVA read produced has no common prefix, so its floor is 0 and its draft layer attends
+// to the whole window, as with STRATA_KVA=0.
+//
+// A mislabel can only cost draft cells (an exact state that shares a prefix with an approximated one reports the
+// floor too high); it can never make the draft attend to approximated cells that were registered, and the target
+// model verifies every token whatever the draft read.
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <vector>
+
+namespace strata::prefill {
+
+class KvaFloorRegistry {
+public:
+ explicit KvaFloorRegistry(size_t cap = 8) : cap_(cap) {}
+
+ /// A KVA read approximated the cells [0, floor) of the prompt `ids` (at least `floor` tokens).
+ template <class Token>
+ void note(const Token* ids, int64_t floor) {
+ if (floor <= 0) return;
+ ++clock_;
+ for (Entry& e : v_)
+ if ((int64_t) e.ids.size() == floor && std::equal(e.ids.begin(), e.ids.end(), ids)) { e.used = clock_; return; }
+ Entry e;
+ e.ids.assign(ids, ids + floor);
+ e.used = clock_;
+ if (v_.size() >= cap_) {
+ size_t lru = 0;
+ for (size_t i = 1; i < v_.size(); ++i) if (v_[i].used < v_[lru].used) lru = i;
+ v_.erase(v_.begin() + (std::ptrdiff_t) lru);
+ }
+ v_.push_back(std::move(e));
+ }
+
+ /// The first cell of a state holding the tokens ids[0, n) that is NOT an approximated one: the longest common
+ /// prefix with a remembered read, over the remembered reads. 0 when the tokens match none.
+ template <class Token>
+ int64_t floor_for(const Token* ids, int64_t n) const {
+ int64_t best = 0;
+ for (const Entry& e : v_) {
+ const int64_t m = std::min<int64_t>(n, (int64_t) e.ids.size());
+ int64_t i = 0;
+ while (i < m && (int64_t) ids[i] == e.ids[(size_t) i]) ++i;
+ best = std::max(best, i);
+ }
+ return best;
+ }
+
+ size_t size() const { return v_.size(); }
+ void clear() { v_.clear(); }
+
+private:
+ struct Entry {
+ std::vector<int64_t> ids;
+ uint64_t used = 0;
+ };
+ std::vector<Entry> v_;
+ size_t cap_;
+ uint64_t clock_ = 0;
+};
+
+} // namespace strata::prefill
diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp
index f9bbca027..39ed85577 100644
--- a/include/strata/prefill/prefill.hpp
+++ b/include/strata/prefill/prefill.hpp
@@ -209,6 +209,10 @@ public:
/// for its grouping on the host; a policy may count them. Not called on the paths that keep the ids on the GPU.
std::function<void(int64_t layer, const int32_t* ids, int64_t n)> on_routes;
int64_t stopped_at() const { return stopped_at_; }
+ /// KVA: the end of the positions the last run approximated (they are a prefix: chunks from the run's first position
+ /// on), i.e. the first position whose final rows are real; 0 when that run approximated nothing. Raised chunk by
+ /// chunk, before on_chunk reports the chunk, so the draft layer can skip the chunk's cells. Reset by every run.
+ int64_t kva_floor() const { return kva_floor_; }
/// Grows this stage's (and the later stages') pinned upload slots to what a text chunk of `T` tokens uploads, so
/// a run of such chunks allocates no pinned memory (STRATA_PIPED_CONCURRENT: an allocation or free on the read's
/// thread takes the driver's lock). Before the run, with nothing of this path in flight. False: a slot could
@@ -270,6 +274,7 @@ private:
Prefill* helper_ = nullptr; ///< set_stage_helper
bool single_chunk_ = false; ///< a later stage: the prompt is one chunk (set by the stage before)
int64_t stopped_at_ = -1; ///< stop_at_chunk ended the last run after this many tokens (-1: it read all)
+ int64_t kva_floor_ = 0; ///< kva_floor()
bool pipe_chunked_ = false; ///< a later stage: the chunks came from a --prefill-pipe plan (set likewise)
int64_t force_chunk_ = 0; ///< force_chunk, for the next run (0: none)
bool force_single_ = false, force_planned_ = false;
diff --git a/metal/src/engine/stubs.cpp b/metal/src/engine/stubs.cpp
index b6615813c..7facf9a29 100644
--- a/metal/src/engine/stubs.cpp
+++ b/metal/src/engine/stubs.cpp
@@ -325,6 +325,7 @@ namespace strata::kernels { void topk_512_step(const float * p0, const QsaShapes
namespace strata::kernels { void wait_flag_ge(const uint32_t * p0, uint32_t p1, void * p2) { strata::metal_engine::missing("strata::kernels::wait_flag_ge"); } }
namespace strata::kernels { void wait_flag_ge_or(const uint32_t * p0, uint32_t p1, const uint32_t * p2, void * p3) { strata::metal_engine::missing("strata::kernels::wait_flag_ge_or"); } }
namespace strata::kernels { void window_ids(int32_t * p0, int p1, int p2, int32_t * p3, int64_t p4, void * p5) { strata::metal_engine::missing("strata::kernels::window_ids"); } }
+namespace strata::kernels { void window_ids_floor(int32_t * p0, int p1, int p2, int32_t * p3, int64_t p4, const int32_t * p5, void * p6) { strata::metal_engine::missing("strata::kernels::window_ids_floor"); } }
auto strata::prefill::Gemm::bf16(const uint16_t * p0, const uint16_t * p1, float * p2, int64_t p3, int64_t p4, int64_t p5, int64_t p6, float p7, int64_t p8, bool p9) -> void { strata::metal_engine::missing("strata::prefill::Gemm::bf16"); }
auto strata::prefill::Gemm::bf16_converts_x() const -> bool { strata::metal_engine::missing("strata::prefill::Gemm::bf16_converts_x"); return {}; }
auto strata::prefill::Gemm::bf16_hcd_exact(const uint16_t * p0, int64_t p1, const uint16_t * p2, float * p3, int64_t p4, int64_t p5, int64_t p6) -> bool { strata::metal_engine::missing("strata::prefill::Gemm::bf16_hcd_exact"); return {}; }
diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp
index f4b82b334..292a5d2b4 100644
--- a/src/core/mtp.cpp
+++ b/src/core/mtp.cpp
@@ -163,7 +163,7 @@ MtpDrafter::~MtpDrafter() {
if (owns_draft_head_ && dvocab_) cudaFree(dvocab_);
if (ev_chain_) cudaEventDestroy(ev_chain_);
for (cudaEvent_t e : ev_step_) if (e) cudaEventDestroy(e);
- void* hosts[] = {h_tok_, h_step_, h_pos_, h_row_, h_out_, h_prob_, h_force_};
+ void* hosts[] = {h_tok_, h_step_, h_pos_, h_row_, h_out_, h_prob_, h_force_, h_floor_};
for (void* h : hosts) if (h) cudaFreeHost(h);
}
@@ -370,6 +370,7 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session
// ---- buffers
window_ = (window > 0 && window < max_cells) ? window : 0;
+ if (floor_on_ && !mapped(64, (void**) &h_floor_, (void**) &m_floor_)) { err = "mtp: the window floor does not fit"; return false; }
cap_ = (((window_ > 0 ? window_ : max_cells) + 63) / 64) * 64;
attn_scratch_floats_ = (int64_t) strata::kernels::qsa_decode_attn_scratch_floats(cap_, s);
const uint64_t T = (uint64_t) max_t, N = (uint64_t) g.n_embd, HC = (uint64_t) g.hc, K = (uint64_t) ss.k;
@@ -1027,7 +1028,9 @@ bool MtpDrafter::record_rest(int step_row, cudaStream_t cs, std::string& err) {
if (st_.kv_rot) fwht256_inplace_cuda(qcur_, (int64_t) T * NH, cs);
g_mp.stamp(5, cs);
const QsaAttnPools pools = qsa_attn_pools(st_);
- if (window_ > 0) window_ids(const_cast<int32_t*>(step), T, (int) window_, ident_, cap_, cs);
+ if (floor_on_ && m_floor_ != nullptr) // KVA: the window (or every cell) from the first real draft cell on
+ window_ids_floor(const_cast<int32_t*>(step), T, window_ > 0 ? (int) window_ : 0x7fffffff, ident_, cap_, m_floor_, cs);
+ else if (window_ > 0) window_ids(const_cast<int32_t*>(step), T, (int) window_, ident_, cap_, cs);
// STRATA_QSA_SPLIT_ATTN=1: the draft layer's attention over its window, split the same way
static const bool split_attn = [] { const char* v = std::getenv("STRATA_QSA_SPLIT_ATTN"); return v != nullptr && std::atoi(v) != 0; }();
if (!(split_attn && qsa_decode_attn_split(qcur_, pools, ident_, step, cap_, s, attn_scratch_, (uint64_t) attn_scratch_floats_, attn_, T, cs)))
@@ -1423,7 +1426,7 @@ bool MtpDrafter::prefill(const float* R_rows, const int32_t* next_tokens, int64_
const Clock::time_point t0 = Clock::now();
const int64_t HCN = g_->hc * g_->n_embd;
// cells the window can never reach again need no K/V
- const int64_t first_needed = (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0;
+ const int64_t first_needed = this->first_needed();
// E-4: every group's token / step / position records uploaded at once; each group is then device copies and a
// graph on the one stream, with a single sync at the end (a group of <= max_t rows used to be staged in mapped
// memory and synced before the next: ~5,500 host round trips on a 32K prompt). The same work in the same
diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu
index a9f570b9e..265664759 100644
--- a/src/kernels/cuda/verify_kernels.cu
+++ b/src/kernels/cuda/verify_kernels.cu
@@ -6,6 +6,7 @@
#include "strata/core/geometry.hpp"
#include "strata/core/emulate.hpp"
#include "strata/kernels/q8_1_finite.hpp" // #606: q8_1_ds
+#include "strata/kernels/window_start.hpp" // KVA: the draft layer's window floor
#include "strata/kernels/dp4a.hpp"
#include "strata/kernels/pdl.hpp"
@@ -1016,6 +1017,30 @@ void window_ids(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_str
check("window_ids");
}
+namespace {
+// window_ids with a floor read from (mapped) memory when the kernel RUNS: the captured graph keeps the pointer, the
+// value changes between sessions. A floor of 0 gives window_ids' selection exactly (strata::kernels::window_start).
+__global__ void window_ids_floor_kernel(int32_t* steps, int window, int32_t* ids, long long stride,
+ const volatile int32_t* floor_cell) {
+ const int q = blockIdx.y;
+ int32_t* st = steps + q * 4;
+ const int n_kv = st[1];
+ const int start = strata::kernels::window_start(n_kv, window, *floor_cell);
+ const int width = n_kv - start;
+ for (int j = blockIdx.x * blockDim.x + threadIdx.x; j < width; j += gridDim.x * blockDim.x)
+ ids[q * stride + j] = start + j;
+ __syncthreads();
+ if (blockIdx.x == 0 && threadIdx.x == 0) st[3] = width;
+}
+} // namespace
+
+void window_ids_floor(int32_t* steps, int n, int window, int32_t* ids, int64_t ids_stride, const int32_t* floor_cell,
+ void* stream) {
+ window_ids_floor_kernel<<<dim3(8, (unsigned) n), 256, 0, (cudaStream_t) stream>>>(steps, window, ids, (long long) ids_stride,
+ floor_cell);
+ check("window_ids_floor");
+}
+
void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream) {
dense_steps_kernel<<<1, 64, 0, (cudaStream_t) stream>>>(cells, n, steps);
check("dense_steps");
diff --git a/src/prefill/kva_test.cpp b/src/prefill/kva_test.cpp
index bd468c34d..937d0ca14 100644
--- a/src/prefill/kva_test.cpp
+++ b/src/prefill/kva_test.cpp
@@ -1,10 +1,13 @@
// src/prefill/kva_test.cpp - the KVA switch, the automatic tail and the reasons a read is turned away (no GPU needed).
+#include "strata/kernels/window_start.hpp"
#include "strata/prefill/kva.hpp"
+#include "strata/prefill/kva_floor.hpp"
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
+#include <vector>
using namespace strata::prefill;
@@ -50,25 +53,13 @@ int main() {
{ const KvaConfig c = kva_config_from_env(); CHECK(c.on && !c.explicit_on); }
clear_all();
- // ---- the automatic tail: 2048, or the draft layer's window + 64 when that is larger; STRATA_KVA_SUFFIX wins
+ // ---- the tail: 2048, or STRATA_KVA_SUFFIX; the draft window plays no part (the cell floor protects the draft layer)
{
const KvaConfig c = kva_config_from_env();
- KvaEngine none; // no draft layer
- CHECK(c.effective_suffix(none) == kKvaSuffixDefault);
- KvaEngine d; // the window the engine picks (no --mtp-window given)
- d.has_drafter = true;
- d.drafter_window = kKvaDrafterWindow;
- CHECK(c.effective_suffix(d) == 2112);
- d.drafter_window = 8192; // --mtp-window 8192
- CHECK(c.effective_suffix(d) == 8256);
- d.drafter_window = 512; // a small window: the default tail still holds
- CHECK(c.effective_suffix(d) == 2048);
- d.drafter_window = 0; // every cell: no tail can cover it, the plain tail is returned
- CHECK(c.effective_suffix(d) == 2048);
+ CHECK(c.effective_suffix() == kKvaSuffixDefault && kKvaSuffixDefault == 2048);
put("STRATA_KVA_SUFFIX", "3000");
const KvaConfig e = kva_config_from_env();
- d.drafter_window = 8192;
- CHECK(e.effective_suffix(d) == 3000);
+ CHECK(e.effective_suffix() == 3000);
clear_all();
}
@@ -76,8 +67,6 @@ int main() {
{
const KvaConfig c = kva_config_from_env();
KvaEngine eng;
- eng.has_drafter = true;
- eng.drafter_window = kKvaDrafterWindow;
bool routine = false;
// a cold 14.6K prompt on one device: applies
CHECK(c.decline(eng, 0, 14600, 0, -1, 48) == nullptr);
@@ -102,16 +91,7 @@ int main() {
put("STRATA_KVA", "1");
{ const KvaConfig f = kva_config_from_env(); CHECK(f.decline(eng, 0, 14600, 0, -1, 40, false, false, nullptr, true) == nullptr); }
clear_all();
- // a draft layer that attends to every cell has no tail that covers it
- KvaEngine all = eng;
- all.drafter_window = 0;
- CHECK(is(c.decline(all, 0, 14600, 0, -1, 48), "the draft layer attends to every cell (set --mtp-window)"));
- put("STRATA_KVA_SUFFIX", "2112"); // ... unless the user fixes the tail by hand
- { const KvaConfig s = kva_config_from_env(); CHECK(s.decline(all, 0, 14600, 0, -1, 48) == nullptr); }
- clear_all();
// no draft layer at all: nothing to cover
- KvaEngine nod;
- CHECK(c.decline(nod, 0, 14600, 0, -1, 48) == nullptr);
// the engine rules the whole process out (batch slots, a layer split, a peer device)
KvaEngine blocked = eng;
blocked.blocked = "batch slots (--batch)";
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index 741c3d8fb..8fd781a5a 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -2717,7 +2717,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
// stage, no capture). Positions [0, kva_frontier) are approximated in the layers at and after the split.
const KvaConfig& kva = kva_config();
const KvaEngine& kva_eng = kva_engine();
- const int64_t kva_suffix = kva.effective_suffix(kva_eng);
+ const int64_t kva_suffix = kva.effective_suffix();
bool kva_routine = false;
const char* kva_why_not = kva.decline(kva_eng, pos0, n, LB, LE < 0 ? -1 : LE, (int64_t) g.n_layers,
embd_rows != nullptr, !tap_layers_.empty(), &kva_routine,
@@ -2728,6 +2728,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
static KvaTap kva_capture; // process-wide: one cold read at a time, so the per-layer position is enough
static KvaTap kva_oracle;
int64_t kva_positions = 0, kva_chunks = 0; // what actually took the approximate path (diagnostics)
+ kva_floor_ = 0; // this run's: raised below as chunks are approximated
if (kva.capture && kva_capture.f.empty()) kva_capture.open(kva.capture_dir, kva.capture_fp32);
if (kva.capture && pos0 == 0 && hand_in_ == nullptr) kva_capture.begin_prompt(); // one prompt per capture
if (kva_run && kva.mode == KvaConfig::Mode::Oracle && kva_oracle.f.empty())
@@ -2862,6 +2863,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
if (kva_approx) {
kva_positions += T;
++kva_chunks;
+ kva_floor_ = p0 + T; // the chunks are a prefix of the run: this is the first position with real rows
}
core::progress_at("reading the prompt (batched): preparing the chunk from token", p0); // #251
#if defined(STRATA_HAVE_DFLASH2)
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 82a6c9457..dbc5b6b02 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -73,6 +73,7 @@
#include "strata/core/draft_cap.hpp"
#endif
#include "strata/prefill/kva.hpp"
+#include "strata/prefill/kva_floor.hpp"
#include "strata/prefill/prefill.hpp"
#if defined(STRATA_PREFILL_MMQ)
#include "strata/prefill/moe_mmq.hpp"
@@ -682,7 +683,6 @@ struct Options {
/// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it.
std::string mtp;
int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell)
- bool mtp_window_given = false; ///< --mtp-window was given (KVA picks the window itself only when it was not)
/// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the
/// pinned arena while the CPU computes the rest (verify windows).
double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs)
@@ -2629,7 +2629,7 @@ int main(int argc, char** argv) {
else if (a == "--spec-follow") o.spec_follow = next("--spec-follow");
else if (a == "--window-hashes") o.window_hashes = next("--window-hashes");
else if (a == "--mtp") o.mtp = next("--mtp");
- else if (a == "--mtp-window") { o.mtp_window = std::atoll(next("--mtp-window")); o.mtp_window_given = true; }
+ else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window"));
else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac"));
else if (a == "--adapt-every") { o.adapt_every = std::atoi(next("--adapt-every")); o.adapt_every_given = true; }
else if (a == "--adapt-decay") o.adapt_decay = (float) std::atof(next("--adapt-decay"));
@@ -5512,9 +5512,9 @@ int main(int argc, char** argv) {
// KVA (0.2.7: on by default, STRATA_KVA=0 is the exact 0.2.6 path): what only the engine knows. A layer split and
// the batch slots read their prompts in pieces and in stages, which the one-device, one-cold-read rule of the
// approximate prompt read cannot follow, so they stay on the exact path (and say so here, once). The draft layer
- // (MTP) must find exact cells in its window: with no --mtp-window given the window becomes 2048 cells (the 0.2.6
- // default, 32768, is every cell of a chat up to that length, which no fully computed tail could cover) and the
- // tail 2112; a window the user gave keeps its value and sets the tail (window + 64).
+ // (MTP) keeps its window, whatever it is: a session KVA approximated tells it the first cell built from a real row
+ // (the cell floor, set per request below) and every session KVA did not approximate attends exactly as with
+ // STRATA_KVA=0.
{
const strata::prefill::KvaConfig& kc = strata::prefill::kva_config();
strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();
@@ -5529,15 +5529,8 @@ int main(int argc, char** argv) {
if (kc.explicit_on)
std::fprintf(stderr, "strata generate: KVA (the approximate late-layer prompt read) is off here: %s\n",
ke.blocked.c_str());
- } else if (!o.mtp.empty() && o.spec >= 2 && !o.mtp_window_given && kc.suffix_env <= 0) {
- if (o.mtp_window != strata::prefill::kKvaDrafterWindow)
- std::fprintf(stderr, "strata generate: KVA on (the faster prompt read for long cold prompts): the draft layer "
- "attends to the last %lld cells (--mtp-window %lld); STRATA_KVA=0 keeps the "
- "exact 0.2.6 prompt read and its %lld-cell window\n",
- (long long) strata::prefill::kKvaDrafterWindow, (long long) strata::prefill::kKvaDrafterWindow,
- (long long) o.mtp_window);
- o.mtp_window = strata::prefill::kKvaDrafterWindow;
}
+ mtp.enable_cell_floor(kc.wants_cell_floor(ke, !o.mtp.empty() && o.spec >= 2));
}
}
if (!o.mtp.empty()) {
@@ -5559,11 +5552,6 @@ int main(int argc, char** argv) {
if (o.pipeline_windows >= 2) mtp.set_force_capture(true);
if (!o.mtp.empty() && !mtp.load(o.mtp, draft_geometry, last_st && !mtp_first ? last_st->ss : ss, mtp_t, err, o.mtp_window)) { std::fprintf(stderr, "strata generate: %s%s\n", err.c_str(), vram_free_note().c_str()); return 1; }
mtp.set_ple_session(&ss);
- if (!o.mtp.empty()) { // KVA: the cells the draft layer attends to now (0 = every cell) set the tail that must be exact
- strata::prefill::KvaEngine& ke = strata::prefill::kva_engine();
- ke.has_drafter = true;
- ke.drafter_window = mtp.window();
- }
#if defined(STRATA_HAVE_DFLASH2)
if (o.draft == "dflash2") {
const char* why = o.mtp.empty() ? "it needs --mtp (the fallback) and --spec 2 or more"
@@ -9473,6 +9461,9 @@ int main(int argc, char** argv) {
// a reporter's prompts failed there every time and passed with --short-read 0.
bool nf_prompt_windows = false, short_read_off_once = false;
std::vector<ConvCheckpoint> checks;
+ // KVA: which prefixes of token ids a KVA read approximated, so the draft layer knows, for whatever state a request
+ // continues from, the first cell built from a real row (kva_floor.hpp). Empty without KVA.
+ strata::prefill::KvaFloorRegistry kva_floors;
// STRATA_CKPT_POOL: the dropped checkpoints' storage, reused. STRATA_SLOT_CKPT_MOVE turns it on too: a moved
// checkpoint frees nothing at the next prune, so every save would fault in fresh pages (17 -> 40-60 ms)
CheckpointPool ckpt_pool(ckpt_pool_env() || slot_ckpt_move_env());
@@ -9982,6 +9973,9 @@ int main(int argc, char** argv) {
// come from Q8_1 x Q8_0 MMQ rather than mmvq: drafts may differ, the target's tokens never.
static const bool split_mtp_batch = strata::program::draft_kv::split_env_on(std::getenv("STRATA_SPLIT_MTP_BATCH"));
sp.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool {
+ // KVA: a chunk the late layers did not compute has no real final rows: its draft cells are neither built
+ // nor read (the floor rises before the cells are built, so they are skipped)
+ if (use_mtp && mtp.cell_floor_enabled() && sp.kva_floor() > mtp.cell_floor()) mtp.set_cell_floor(sp.kva_floor());
std::vector<int32_t> nxt((size_t) T);
for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) cur[(size_t) (p0 + t + 1)];
// E-9: batched through the prompt path when it can (a layer split: only with STRATA_SPLIT_MTP_BATCH=1)
@@ -13161,6 +13155,10 @@ int main(int argc, char** argv) {
tr("request", n, geni ? 1 : 0);
req_times = ReqTimes{};
if (use_mtp) mtp.set_prompt_len(n);
+ // KVA: the draft layer's first real cell for the state this request continues from (0 for a cold read - the
+ // first segment raises it if KVA approximates it - and for any state no KVA read produced)
+ if (use_mtp && mtp.cell_floor_enabled())
+ mtp.set_cell_floor(resume > 0 && reread_to <= 0 ? kva_floors.floor_for(ids.data(), resume) : 0);
const int64_t read_from = reread_to > 0 ? 0 : resume;
conversations.limit_reuse(read_from);
pp_total = n;
@@ -13603,7 +13601,12 @@ int main(int argc, char** argv) {
} seg_plan_scope{seg_plan};
auto run = [&](int64_t q, int64_t r) -> bool {
if (planned) sp.force_chunk(plan.chunk, plan_single, plan.chunked);
- return sp.run(ids.data() + q, r - q, q, e);
+ const bool run_ok = sp.run(ids.data() + q, r - q, q, e);
+ if (run_ok && q == 0 && sp.kva_floor() > 0) { // KVA approximated the start of this prompt: remember which tokens
+ kva_floors.note(ids.data(), sp.kva_floor());
+ if (use_mtp && mtp.cell_floor_enabled()) mtp.set_cell_floor(sp.kva_floor());
+ }
+ return run_ok;
};
// (a layer split reads its stages as a pipeline over one run's chunks: beside slots a chunk at a time, with
// none decoding in pieces of STRATA_BATCH_YIELD_CHUNKS chunks (default 8; 0 = one run) so a BYIELD is taken)
@@ -15944,6 +15947,7 @@ int main(int argc, char** argv) {
return 1;
}
prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool {
+ if (mtp.cell_floor_enabled() && prefill.kva_floor() > mtp.cell_floor()) mtp.set_cell_floor(prefill.kva_floor()); // KVA
// cell i pairs R_i with the token at i + 1 (every such token is in the prompt)
std::vector<int32_t> nxt((size_t) T);
for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) o.tokens[(size_t) (p0 + t + 1)];