Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: release notes - the two-card batch default marker uses the owner's labels (a) pipelined by default, (b) pipelined only when every expert fits, (c) leave serial; the serial against pipelined numbers are batch scheduling without drafting
Niko12218dce24a0a13a
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex 5c8f71380..5c16ca4e4 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -38,7 +38,7 @@ A variable or flag you set yourself still wins over the switch, and the older si- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details.- **Prefill on one card, decode on another** (`--split-serve P,D`, or `split_serve` in the run config; #4000, first version). The idea, the hand-made setup (an RTX 3070 reading the prompt, two MI50 cards making the answer) and the first measurements are by @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers. The words are the same as with one engine, token for token (checked at 4,096 and 16,384 tokens, with equal state fingerprints). On two RTX A6000 (3 rounds, 6 values per cell, whole-request time) it was slower than the normal layer split on every workload: an 8K prompt took 16% longer, a 32K prompt 48% longer, a 3-turn chat 30% and 4% longer. So we do not recommend it on two alike cards. It can only pay with two different cards, one good at reading prompts and one good at making answers; we have not measured such a pair. `docs/SPLIT_SERVE.md` has the numbers and the limits.-- **Pipelined drafts for batch slots** (`STRATA_PIPED_MTP=1` with `--batch-mtp`, #4121 by @wannagoxuf). The draft model also runs for the slots of a batch on a two-card layer split. On two RTX A6000 with a batch of slots, five paired runs before the merge: two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%; the merged build, three pairs: +12.2% and +0.2%. Every slot still decodes exactly the tokens it decodes alone (checked slot by slot against a solo run). Separately, on two NVIDIA cards the standard drafting for a batch is 15 to 18% slower than the pipelined way even without this switch (two streams 107.4 against 125.7 tok/s, four streams 141.0 against 171.4); it was already so in 0.2.6. [pending: decision on the two-card batch default - (a) keep it, (b) pipelined by default, (c) pipelined only when every expert is resident; branches f-027-gh-eng-opt-flip and f-027-gh-eng-opt-resid]+- **Pipelined drafts for batch slots** (`STRATA_PIPED_MTP=1` with `--batch-mtp`, #4121 by @wannagoxuf). The draft model also runs for the slots of a batch on a two-card layer split. On two RTX A6000 with a batch of slots, five paired runs before the merge: two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%; the merged build, three pairs: +12.2% and +0.2%. Every slot still decodes exactly the tokens it decodes alone (checked slot by slot against a solo run). Separately, how two NVIDIA cards share several chats at once: today's default is the serial way (since 0.2.1), and the pipelined way is faster at two and four chats. On two RTX A6000 (Qwen IQ3_S, no drafting) pipelined was 17% faster at two chats (125.7 against 107.4 tok/s) and 22% at four (171.4 against 141.0); on two RTX A4000 (Q2_0 and IQ3_XXS) two chats gain 7 to 12%, four chats 17 to 20%, and a single chat loses 4% (IQ3_XXS). The default for this is the owner's choice. [pending: decision on the two-card batch default - (a) pipelined by default on 2+ NVIDIA cards (recommended; branch f-027-gh-eng-opt-flip 7d44e972e), (b) pipelined only when every expert fits on the cards (f-027-gh-eng-opt-resid 7cfa9291d), (c) leave serial]- **Huge pages for DeepSeek on Linux.** If you reserve 2 MB huge pages (`vm.nr_hugepages`) big enough for the expert copy, the engine uses them and copies prompt data in place: an 8K prompt on the 2.25 set read 651 tok/s without and 694 with (R9700 limited to 12 GB; that run also set two buffers by hand). Without a reserved pool nothing changes. Not run with a reserved pool on the release package.- **Settings that help if your PC fits them, no engine change:** `--ple-io ram` (the 28.8 GB table in RAM; on the 72 GB card prompts +16.5% at 1K, +30.6% at 4K, +16.8% at 8K), `--mtp-draft-vocab draft_vocab_en.bin` (+2.7 to +3.1% decode for English text), `--slots 2` for two conversations at once (+10% to +12% aggregate throughput on RTX PRO 5000 and RTX 3060), `--prefill-pipe` on a two-card layer split (+15% at 4K, +23% at 8K prompts on two RTX A4000). `--kv q4_0` on a 12 GB card did not reproduce a gain, so it is a choice, not a default.