Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
KVA: what the opt-in approximate late-layer prefill does, how to enable and capture, and the measured frontier
Niko12214abef6a74ee6
diff --git a/docs/KVA.md b/docs/KVA.mdnew file mode 100644index 000000000..eeb3dee51--- /dev/null+++ b/docs/KVA.md@@ -0,0 +1,96 @@+# KVA: approximate prompt processing for the late layers (opt-in, research)++This is an **experimental, opt-in** feature. Nothing in it runs unless `STRATA_KVA=1` is set, and a build without+that variable behaves exactly as before.++KVA ("approximate late-layer state") reads a long prompt exactly through layer 23, then for the late layers+24–47 builds **only their recurrent state** — the GDN recurrence and convolution history, the QSA K/V pages and+the indexer keys — instead of running the whole layer. The attention output, the output projection, the MoE half+(router, experts, shared expert) and the late layers' expert transfers are skipped, because nothing reads their+results. The last positions of the prompt are then read fully, exactly as before, and decode runs the ordinary+full path against the state KVA built.++## Turning it on++```sh+STRATA_KVA=1 \+STRATA_KVA_SUFFIX=2112 \+STRATA_KVA_MIN_PROMPT=4096 \+ strata --serve ... --prefill 1024 --mtp-window 2048+```++| variable | default | what it does |+|---|---:|---|+| `STRATA_KVA` | off | `1` enables the feature for qualifying requests |+| `STRATA_KVA_SPLIT` | 24 | first approximated layer (zero-based) |+| `STRATA_KVA_SUFFIX` | 2048 | fully computed tail positions; rounded **up** to a whole chunk |+| `STRATA_KVA_MIN_PROMPT` | 4096 | shorter prompts stay on the full path |+| `STRATA_KVA_ONLY` | all | `gdn` or `qsa`: approximate only that layer type |+| `STRATA_KVA_MODE` | identity | `identity`, `oracle`, `linear` (linear is not implemented) |+| `STRATA_KVA_CAPTURE` | — | directory: write every late layer's real input for calibration |+| `STRATA_KVA_CAPTURE_FP32` | off | write fp32 instead of bf16 records |+| `STRATA_KVA_ORACLE` | — | directory: replay a capture instead of a projector |+| `STRATA_KVA_GDN_NORM` | off | keep the full GDN recurrence (the A/B control for the state-only one) |++Requirements and limits for this first version:++- One cold read of one prompt, one device (`pos0 == 0`, one stage). A follow-up read, a resumed prompt or a+ layer split stays on the full path.+- The fully computed suffix must cover the MTP drafter's window: use `--mtp-window N` with+ `STRATA_KVA_SUFFIX >= N + 64`, or the drafter reads state built from approximated positions.+- **The session state after a KVA read is approximate.** It must not be reused as an exact prefix: no+ checkpoint of it, no follow-up prompt that reuses it. The engine prints a warning when this happens.+- The approximation is a quality tradeoff, not a lossless transformation. Measure before using it.++## Measured (Tesla P100 16 GB, Qwen3.8-Flash-Next IQ3_XXS, chunk 1024, expert cache 1500, MTP window 2048)++| prompt tokens | control prompt tok/s | KVA prompt tok/s | speedup | approximated | residual error on the suffix |+|---:|---:|---:|---:|---:|---:|+| 5,642 | 203.3 | 265.7 | 1.31× | 54 % | 0.357 |+| 14,581 | 202.5 | 311.2 | 1.54× | 85 % | not measured |+| 29,997 | 198.7 | 326.5 | 1.64× | 92 % | 0.451 |++- Decode throughput was unchanged within the run-to-run spread (24–28 tok/s) in every arm: KVA changes prompt+ processing only.+- Quality: on a 14.6K-token prompt with a fact inserted at 10/50/90 %, the KVA arm retrieved the fact at all+ three positions and produced the identical answer string. On a "summarise this documentation" prompt the+ answers stayed coherent and semantically identical, with different wording after 4–12 tokens.+- Layer type matters more than the count: approximating only the 6 late QSA layers cost a 0.153 residual error+ at 1.08×; approximating 18 GDN layers cost 0.339 at 1.18×. A gated recurrence integrates its input over the+ whole prefix, so a wrong input keeps biasing the state, while a wrong QSA K/V only misplaces attention keys.+- The control is not bit-reproducible on this box (identical runs differ in the last chunks by ~0.85 %), so the+ residual figures are quoted against that measured floor, never as bit equality.++## What the block input is++The late layers' input is the 10,240-value hyper-connection residual (`hc * n_embd`) that the prompt path+carries. In `identity` mode every late layer reuses the residual entering layer 24 — no fitted weights, no extra+work. Capturing the *real* per-layer inputs (`STRATA_KVA_CAPTURE`) and replaying them (`STRATA_KVA_ORACLE`)+reproduces the full path's answer exactly; that is the mode a fitted projector would replace.++A per-position linear projector was tested and rejected: fitted on 5,636 positions, a rank-256 map from the+boundary to each late layer's input is *worse* than reusing the boundary for the first nine late layers and never+gets below 0.80 relative error (`bench/2026-10-25-predictability.md` in the research folder). The residual is+moved by routing and attention decisions a linear map cannot follow.++## Capturing inputs++```sh+STRATA_KVA_CAPTURE=/data/kva-cap STRATA_KVA_CAPTURE_FP32=1 \+ strata --serve ... --prefill 1024 --tokens-file ids.txt --max-new 1+```++writes `L24.bin … L47.bin` in that directory: one row of `hc * n_embd` floats per prompt position per late+layer, in position order. A second prompt overwrites them. `STRATA_KVA_ORACLE=<dir>` replays such a directory in+place of a projector; the replay must use the same prompt and the same chunking.++## Implementation notes++- `src/prefill/kva.hpp` — the plan and the environment variables.+- `src/prefill/prefill.cpp` — the chunk plan (`kva_approx`), the capture/oracle taps, the state-only layer+ exits, the transfer-plan skip, and `give_back()` on a skipped MoE half: a layer that consumes no expert must+ still release its stream entries, or the issuer thread fills the ring and the next fully computed layer waits+ forever.+- `src/prefill/kernels.cu` — `gdn_recurrence_state_only()`, the same recurrence dispatch without the output+ normalization.+- CUDA and HIP builds (`src/`); the SYCL engine (`sycl/`) does not have the feature.diff --git a/docs/README.md b/docs/README.mdindex e197f3f77..ed4ebe063 100644--- a/docs/README.md+++ b/docs/README.md@@ -35,6 +35,7 @@ Use the guide for your GPU and operating system. Model and feature support can d- [Contribution rules](../CONTRIBUTING.md)- [Community benchmarks](COMMUNITY_BENCHMARKS.md) and the [standard benchmark protocol](BENCHMARK_PROTOCOL.md)- [How the engine works](HOW_IT_WORKS.md), [technical reference](DETAILS.md) and [Terminal reference](TERMINAL.md)+- [KVA: approximate prompt processing for the late layers (experimental, opt-in)](KVA.md)- [App development](../apps/host/README.md) and [packaging](../packaging/README.md)## AI tools and media