Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

docs: release notes - split serving (#4000) filled from docs/SPLIT_SERVE.md: opt-in, exact, slower than the layer split on two A6000, credit to MarcoTool

Niko12219ac278e124a6
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.md
index 0abc44b15..0d09c9641 100644
--- a/docs/release-notes-0.2.7.md
+++ b/docs/release-notes-0.2.7.md
@@ -36,7 +36,7 @@ A variable or flag you set yourself still wins over the switch, and the older si
## Off unless you turn it on
- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details.
-- **Prefill on one card, decode on another** (`--split-serve`, #4000). The idea and a manual setup came from @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers, with the state in a file. The words are the same as with one card (checked at 4,096 and 16,384 tokens: tokens and end state identical). [pending: f-027-split - TO FILL the speed table from `notes-027-split.md` before this bullet is kept; the first runs were slower than a layer split]
+- **Prefill on one card, decode on another** (`--split-serve P,D`, or `split_serve` in the run config; #4000, first version). The idea, the hand-made setup (an RTX 3070 reading the prompt, two MI50 cards making the answer) and the first measurements are by @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers. The words are the same as with one engine, token for token (checked at 4,096 and 16,384 tokens, with equal state fingerprints). On two RTX A6000 (3 rounds, 6 values per cell, whole-request time) it was slower than the normal layer split on every workload: an 8K prompt took 16% longer, a 32K prompt 48% longer, a 3-turn chat 30% and 4% longer. So we do not recommend it on two alike cards. It can only pay with two different cards, one good at reading prompts and one good at making answers; we have not measured such a pair. `docs/SPLIT_SERVE.md` has the numbers and the limits.
- **Pipelined drafts** (`STRATA_PIPED_MTP`, #4121 by @wannagoxuf): the draft model also runs for batch slots. On two RTX A6000 cards with a batch of slots, two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%. [pending: f-027-gh-eng-pm2]
- **Layer-major prefill on a CUDA layer split** (#3976, ports of two contributors). [pending: f-027-gh-eng-lm - state whether it is on or off and give the measured prompt gain, or delete the bullet]
- **Huge pages for DeepSeek on Linux.** If you reserve 2 MB huge pages (`vm.nr_hugepages`) big enough for the expert copy, the engine uses them and copies prompt data in place: an 8K prompt on the 2.25 set read 651 tok/s without and 694 with (R9700 limited to 12 GB; that run also set two buffers by hand). Without a reserved pool nothing changes. Not run with a reserved pool on the release package.