Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: the KVA opt-in is split 36, final (the split 24 variant is a 0.2.8 study); release notes marker closed
Niko1221597df5e601bb
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex 2f553b5ce..0abc44b15 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -35,7 +35,7 @@ A variable or flag you set yourself still wins over the switch, and the older si## Off unless you turn it on-- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details. [pending: decision kva2 - split 24 plus MoE skip is confirmed on four boxes only if the owner takes `f-027-kva2`; the numbers are in `notes-027-kva.md`]+- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details.- **Prefill on one card, decode on another** (`--split-serve`, #4000). The idea and a manual setup came from @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers, with the state in a file. The words are the same as with one card (checked at 4,096 and 16,384 tokens: tokens and end state identical). [pending: f-027-split - TO FILL the speed table from `notes-027-split.md` before this bullet is kept; the first runs were slower than a layer split]- **Pipelined drafts** (`STRATA_PIPED_MTP`, #4121 by @wannagoxuf): the draft model also runs for batch slots. On two RTX A6000 cards with a batch of slots, two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%. [pending: f-027-gh-eng-pm2]- **Layer-major prefill on a CUDA layer split** (#3976, ports of two contributors). [pending: f-027-gh-eng-lm - state whether it is on or off and give the measured prompt gain, or delete the bullet]