Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
release notes 0.2.7: layer-major prefill (on by default on a CUDA layer split) and pipelined drafts filled in
Niko12215ac66e5005e5
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex cdcfc12a1..b2196c2bd 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -20,8 +20,6 @@ DRAFT of 27 October. Text in square brackets that starts with "pending" or "TO C- **Four Radeon AI PRO R9700 cards run DeepSeek faster, with five experts per word by default.** With the launcher's own settings, DeepSeek 2.25: an 8K prompt 342 to 399 tok/s on the first read and 967 to 1,350 once warm (+17%, +40%), a 32K prompt 1,127 to 1,604 first and 1,183 to 1,697 warm (+42%, +43%), decode on code 88.9 to 95.0 tok/s (+7%) and on an agent task 73.5 to 79.7 (+8%). Loading takes 58 to 60 s instead of 51 s. The default is now "Faster" (5 of 6 experts per word); the quality run passed 19 of 19 tasks against 18 of 19 with six experts, and the distance to the reference model is the same (0.1102 against 0.1102). Choose Full quality in the app, or `--experts 6`, to use six.- **DeepSeek can use 512K and 1M contexts.** Setup, the Terminal and the app offer them up to the limit of the model files, with a plain "not tested yet" note past 256K. The default stays 8K, and the app now loads a DeepSeek model with the context you chose. On an RTX 5070 a 1M context loaded in 18 s and cost about 1 GB of graphics memory, which pushed 136 experts off the card. We did not test long prompts at these sizes. The idea comes from Project Maya v1.1.28 (#442, by @needmorevram).- **More cards get small gains from kernel changes that were measured.** On an RTX PRO 5000 (72 GB, Qwen IQ3_S, five paired runs): story answers 195.9 to 211.8 tok/s (+8.1%), code 240.6 to 244.7 (+1.7%), prompts +1.2% at 4K, +1.5% at 8K, +1.6% at 16K. On two RTX A4000: story +2.2%, code +0.5%. These switches sum numbers in another order, so a reply can differ from 0.2.6 at a near-tie. RTX 40 cards (compute capability 8.9) use measured speed tables for the quantised matrix products; they are bit for bit the same as before and we have no Ada card, so the reporter's measurements are the only ones. Off: `STRATA_MMVQ_IL_89=0`. A100-class cards (sm_80) get the bitwise-identical defaults of #3779 (wider dequantisation, tuned tables, measured only on a CMP 170HX by the contributor); its many opt-ins stay off, and the reporters are asked to test. Off: `STRATA_DEQUANT_WIDE=0`, `STRATA_MMVQ_IL_SM80=0`.-- **Long prompts on two NVIDIA cards read much faster.** On a layer split over two CUDA cards the prompt is now read layer by layer in large pieces, so each layer's experts are read once per piece instead of once per 8,192 tokens. On two RTX A4000 (Qwen IQ3_XXS, experts streamed, five paired runs): first token after a 32K prompt 141.8 s to 27.7 s (5.1x), after a 20K prompt 98.8 s to 17.5 s (5.7x), after an 8K prompt 41.8 s to 15.2 s (2.8x); the answer speed is the same (38.6 against 38.4 tok/s), the 12 needle questions at 16K and 32K were all found, and the 19-task quick suite passed 18 of 19 with and without it (13 answers byte-identical, the 6 others differ in wording only). The numbers differ slightly from the old path because the experts' results are added in another order: the first-token KL against the old path is at the level of the difference between two prompt chunk sizes, and the most likely token matched on every prompt. One card, Radeon and the approximate-read mode keep the old path. Off: `STRATA_PREFILL_LAYER_MAJOR=0`, or `STRATA_DEFAULTS=026`. From PR 3976 (@Norhiro, building on @vizdatom's #1670).-- **MTP drafts in the pipelined batch groups (opt-in).** `STRATA_PIPED_MTP=1` with `--batch-mtp` lets each slot of a pipelined layer split bring its own draft tokens: on two RTX A6000 with two concurrent streams +12 to +15% total tokens per second, four streams +0 to +5%, one stream unchanged, and every slot decodes exactly the tokens it decodes alone. From PR 4121 (@wannagoxuf).- **The served web chat page gets a large update.** Wide windows use the width and narrow screens fit (#3176). Saved chats with search, rename and delete, editing a question, rewinding to an earlier turn and branching a new chat from it, sending a message while the answer is still coming (wait, send after, or interrupt), math written in LaTeX rendered with KaTeX, a sandboxed preview of HTML and SVG the model writes, and Load and Unload buttons in the header. The chat view is translated into the same languages as the app, French included. The history features are off until you switch them on in Chat settings: with them off the page sends the same requests as before. Monitor, Images and About are still English.- **The desktop app.** French is the ninth language. Settings now has: a sound when a task finishes (on request), a desktop notification when Manual mode needs your approval, minimise to the system tray, a fixed port so API clients keep working across updates (with a warning when the port is busy), and an interface size up to 200%. The Monitor has a readings strip: power draw, an electricity price you set (in your local currency, found from the system region), disk temperature where Windows or Linux can report it. The DeepSeek card says how much free disk the set needs. Not tried inside the real packaged app on screen yet: tray icon, the native notification and the busy-port warning. [TO CONFIRM: run once in the release e2e]- **Speed switches.** In Models, Native engine settings, Speed (and in the Terminal's Advanced table): "Faster long prompts" (off by default), "Faster answers on 12 GB NVIDIA cards", "Faster answers on Radeon RX 9000 / AI PRO R9700", "DeepSeek speed-ups". The Terminal also has "Qwen: 0.2.6 behaviour" for troubleshooting.@@ -39,8 +37,8 @@ A variable or flag you set yourself still wins over the switch, and the older si- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details.- **Prefill on one card, decode on another** (`--split-serve P,D`, or `split_serve` in the run config; #4000, first version). The idea, the hand-made setup (an RTX 3070 reading the prompt, two MI50 cards making the answer) and the first measurements are by @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers. The words are the same as with one engine, token for token (checked at 4,096 and 16,384 tokens, with equal state fingerprints). On two RTX A6000 (3 rounds, 6 values per cell, whole-request time) it was slower than the normal layer split on every workload: an 8K prompt took 16% longer, a 32K prompt 48% longer, a 3-turn chat 30% and 4% longer. So we do not recommend it on two alike cards. It can only pay with two different cards, one good at reading prompts and one good at making answers; we have not measured such a pair. `docs/SPLIT_SERVE.md` has the numbers and the limits.-- **Pipelined drafts** (`STRATA_PIPED_MTP`, #4121 by @wannagoxuf): the draft model also runs for batch slots. On two RTX A6000 cards with a batch of slots, two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%. [pending: f-027-gh-eng-pm2]-- **Layer-major prefill on a CUDA layer split** (#3976, ports of two contributors). [pending: f-027-gh-eng-lm - state whether it is on or off and give the measured prompt gain, or delete the bullet]+- **Pipelined drafts** (`STRATA_PIPED_MTP`, #4121 by @wannagoxuf): the draft model also runs for batch slots. On two RTX A6000 cards with a batch of slots, two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%. Opt-in: `STRATA_PIPED_MTP=1` with `--batch-mtp`; every slot still decodes exactly the tokens it decodes alone.+- **Layer-major prefill on a CUDA layer split is ON by default (#3976, ports of two contributors).** Long prompts on two NVIDIA cards read much faster. On a layer split over two CUDA cards the prompt is now read layer by layer in large pieces, so each layer's experts are read once per piece instead of once per 8,192 tokens. On two RTX A4000 (Qwen IQ3_XXS, experts streamed, five paired runs): first token after a 32K prompt 141.8 s to 27.7 s (5.1x), after a 20K prompt 98.8 s to 17.5 s (5.7x), after an 8K prompt 41.8 s to 15.2 s (2.8x); the answer speed is the same (38.6 against 38.4 tok/s), the 12 needle questions at 16K and 32K were all found, and the 19-task quick suite passed 18 of 19 with and without it (13 answers byte-identical, the 6 others differ in wording only). The numbers differ slightly from the old path because the experts' results are added in another order: the first-token KL against the old path is at the level of the difference between two prompt chunk sizes, and the most likely token matched on every prompt. One card, Radeon and the approximate-read mode keep the old path. Off: `STRATA_PREFILL_LAYER_MAJOR=0`, or `STRATA_DEFAULTS=026`. From PR 3976 (@Norhiro, building on @vizdatom's #1670).- **Huge pages for DeepSeek on Linux.** If you reserve 2 MB huge pages (`vm.nr_hugepages`) big enough for the expert copy, the engine uses them and copies prompt data in place: an 8K prompt on the 2.25 set read 651 tok/s without and 694 with (R9700 limited to 12 GB; that run also set two buffers by hand). Without a reserved pool nothing changes. Not run with a reserved pool on the release package.- **Settings that help if your PC fits them, no engine change:** `--ple-io ram` (the 28.8 GB table in RAM; on the 72 GB card prompts +16.5% at 1K, +30.6% at 4K, +16.8% at 8K), `--mtp-draft-vocab draft_vocab_en.bin` (+2.7 to +3.1% decode for English text), `--slots 2` for two conversations at once (+10% to +12% aggregate throughput on RTX PRO 5000 and RTX 3060), `--prefill-pipe` on a two-card layer split (+15% at 4K, +23% at 8K prompts on two RTX A4000). `--kv q4_0` on a 12 GB card did not reproduce a gain, so it is a choice, not a default.