Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

STRATA_LOGITS_DUMP: a post-window logits tap for KL quality measurement (off by default); the sampler itself runs inside a CUDA graph capture, so the copy lives in the verify window's host-side epilogue where STRATA_DBG_NAN already copies from

Niko122145c9e623e235
diff --git a/docs/KVA.md b/docs/KVA.md
index e6c6fbf57..556f71974 100644
--- a/docs/KVA.md
+++ b/docs/KVA.md
@@ -31,6 +31,7 @@ STRATA_KVA_MIN_PROMPT=4096 \
| `STRATA_KVA_CAPTURE_FP32` | off | write fp32 instead of bf16 records |
| `STRATA_KVA_ORACLE` | — | directory: replay a capture instead of a projector |
| `STRATA_KVA_GDN_NORM` | off | keep the full GDN recurrence (the A/B control for the state-only one) |
+| `STRATA_LOGITS_DUMP` | — | debug (not KVA-specific): write every verify window's sampler logits to `path`, for KL / top-1 between two runs |
Requirements and limits for this first version:
@@ -51,13 +52,17 @@ Requirements and limits for this first version:
|---:|---:|---:|---:|---:|---:|
| 5,642 | 203.3 | 265.7 | 1.31× | 54 % | 0.357 |
| 14,581 | 202.5 | 311.2 | 1.54× | 85 % | not measured |
-| 29,997 | 198.7 | 326.5 | 1.64× | 92 % | 0.451 |
+| 29,997 | 199.7 | 338.3 | 1.69× | 92 % | 0.451 |
- Decode throughput was unchanged within the run-to-run spread (24–28 tok/s) in every arm: KVA changes prompt
processing only.
- Quality: on a 14.6K-token prompt with a fact inserted at 10/50/90 %, the KVA arm retrieved the fact at all
three positions and produced the identical answer string. On a "summarise this documentation" prompt the
answers stayed coherent and semantically identical, with different wording after 4–12 tokens.
+- **Likelihood** (`STRATA_LOGITS_DUMP`): at the first generated token the approximation moves the next-token
+ distribution by KL 0.018 nats at a 5.6K prompt (two control runs differ by 0.0013) and 0.020 nats at 30K
+ (control floor 0.0032). The top-1 token is the same and 9 of the top 10 are shared in both cases. The effect
+ does not grow with context even though the residual error does.
- Layer type matters more than the count: approximating only the 6 late QSA layers cost a 0.153 residual error
at 1.08×; approximating 18 GDN layers cost 0.339 at 1.18×. A gated recurrence integrates its input over the
whole prefix, so a wrong input keeps biasing the state, while a wrong QSA K/V only misplaces attention keys.
diff --git a/src/core/verify.cpp b/src/core/verify.cpp
index 3d6f48b2e..b82181c25 100644
--- a/src/core/verify.cpp
+++ b/src/core/verify.cpp
@@ -3198,6 +3198,34 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
for (int t = 0; t < T; ++t) out[t] = ((volatile int32_t*) h_out_)[t];
if (tap_file_ != nullptr && !batch_rec_) tap_write(T, tokens, pos0);
note_nonfinite(T, nullptr);
+ // STRATA_LOGITS_DUMP=<path> (debug tap, off by default): the window's logits as the sampler saw them, written
+ // AFTER the window (the stream is synchronised above and the graph capture is over, so a plain copy is legal -
+ // the same place STRATA_DBG_NAN copies from). One record is
+ // {int32 magic, int32 T, int32 n_vocab, int32 record, int64 position}, then T * n_vocab FP32 rows; row 0 is
+ // the window's first position (the last prompt token of a prefill), which is what a KL comparison wants.
+ if (const char* logits_path = std::getenv("STRATA_LOGITS_DUMP")) {
+ static std::FILE* lf = std::fopen(logits_path, "wb");
+ static int32_t lrec = 0;
+ if (lf != nullptr) {
+ std::vector<float> h((size_t) T * (size_t) n_vocab_);
+ const cudaError_t le = cudaMemcpy(h.data(), head_logits_, h.size() * sizeof(float), cudaMemcpyDeviceToHost);
+ if (le == cudaSuccess) {
+ const int32_t hdr[4] = {0x53474F4C /*"LOGS"*/, T, (int32_t) n_vocab_, lrec++};
+ const int64_t hpos = pos0;
+ std::fwrite(hdr, sizeof hdr, 1, lf);
+ std::fwrite(&hpos, sizeof hpos, 1, lf);
+ std::fwrite(h.data(), sizeof(float), h.size(), lf);
+ std::fflush(lf);
+ } else {
+ static bool said = false;
+ if (!said) {
+ said = true;
+ std::fprintf(stderr, "strata verify: STRATA_LOGITS_DUMP copy failed: %s\n", cudaGetErrorString(le));
+ (void) cudaGetLastError();
+ }
+ }
+ }
+ }
if (static const bool dbg = std::getenv("STRATA_DBG_NAN") != nullptr; dbg) { // debug: the first non-finite head
static bool reported = false;
if (!reported) {