Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

docs: release notes 0.2.7 - the engine items from f-027-gh-eng filled in (layer-major prefill on by default for a two-card CUDA split, pipelined drafts opt-in, sm_80 and sm_89 defaults, audit fixes, opt-in switches), markers closed except the owner's two-card batch choice

Niko122112d0f0abe8bc
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.md
index 65b3850fe..5c8f71380 100644
--- a/docs/release-notes-0.2.7.md
+++ b/docs/release-notes-0.2.7.md
@@ -19,6 +19,7 @@ DRAFT of 27 October. Text in square brackets that starts with "pending" or "TO C
- **DeepSeek reads prompts faster on most PCs.** The engine no longer re-reads a batch of experts from the SSD after every prompt, picks the attention buffer size by itself and fills the RAM store more cleverly. On an RTX 5070 (12 GB, 64 GB RAM, 2.25 set, three paired runs against an earlier 0.2.7 candidate): 4K prompt 116 to 170 tok/s (+47%), 8K prompt 202 to 268 tok/s (+33%). On one R9700 with the 2.25 set: 32K prompt 382 to 444 tok/s (+16%), 8K prompt 200 to 286 tok/s (+43%, but that prompt is limited by SSD reads and varies a lot between runs). The cost: the first words after a long prompt can come slower, because the store is no longer pre-filled. On the RTX 5070 a long answer still finished 8.7% sooner overall (137.5 s against 150.6 s); on one R9700 a 300-token answer after an 8K prompt took 51.9 s without the pre-fill and 48.6 s with it. `DS4_SETTLE_STORE=1` brings the pre-fill back. The speed-up switch "DeepSeek speed-ups" turns the new behaviour off in one step.
- **Four Radeon AI PRO R9700 cards run DeepSeek faster, with five experts per word by default.** With the launcher's own settings, DeepSeek 2.25: an 8K prompt 342 to 399 tok/s on the first read and 967 to 1,350 once warm (+17%, +40%), a 32K prompt 1,127 to 1,604 first and 1,183 to 1,697 warm (+42%, +43%), decode on code 88.9 to 95.0 tok/s (+7%) and on an agent task 73.5 to 79.7 (+8%). Loading takes 58 to 60 s instead of 51 s. The default is now "Faster" (5 of 6 experts per word); the quality run passed 19 of 19 tasks against 18 of 19 with six experts, and the distance to the reference model is the same (0.1102 against 0.1102). Choose Full quality in the app, or `--experts 6`, to use six.
- **DeepSeek can use 512K and 1M contexts.** Setup, the Terminal and the app offer them up to the limit of the model files, with a plain "not tested yet" note past 256K. The default stays 8K, and the app now loads a DeepSeek model with the context you chose. On an RTX 5070 a 1M context loaded in 18 s and cost about 1 GB of graphics memory, which pushed 136 experts off the card. We did not test long prompts at these sizes. The idea comes from Project Maya v1.1.28 (#442, by @needmorevram).
+- **Two NVIDIA cards read long prompts much sooner on a layer split.** On a layer split over two CUDA cards the prompt is now read layer by layer in large pieces (layer-major prefill, on by default), so each layer's experts are read once per piece instead of once per 8,192 tokens. On two RTX A4000 (16 GB each, Qwen IQ3_XXS with most experts streamed from RAM, five paired runs): the first word after a 32K prompt came after 27.7 s instead of 141.8 s, after a 20K prompt after 17.5 s instead of 98.8 s, after an 8K prompt after 15.2 s instead of 41.8 s. Decode speed is the same (38.6 against 38.4 tok/s). Quality: 12 of 12 needles found with both, the quick suite passed 18 of 19 tasks in all three arms, and the first-token probabilities moved by 0.0007 to 0.009 (KL; the same top choice in every check), so wording can differ a little. If a card has no room for the bigger step, the engine reads the old way. From PR #3976 by @Norhiro, building on @vizdatom's #1670. One card, Radeon and the approximate-read mode keep the old path. Off: `STRATA_PREFILL_LAYER_MAJOR=0`; `STRATA_DEFAULTS=026` gives the old path.
- **More cards get small gains from kernel changes that were measured.** On an RTX PRO 5000 (72 GB, Qwen IQ3_S, five paired runs): story answers 195.9 to 211.8 tok/s (+8.1%), code 240.6 to 244.7 (+1.7%), prompts +1.2% at 4K, +1.5% at 8K, +1.6% at 16K. On two RTX A4000: story +2.2%, code +0.5%. These switches sum numbers in another order, so a reply can differ from 0.2.6 at a near-tie. RTX 40 cards (compute capability 8.9) use measured speed tables for the quantised matrix products; they are bit for bit the same as before and we have no Ada card, so the reporter's measurements are the only ones. Off: `STRATA_MMVQ_IL_89=0`. A100-class cards (sm_80) get the bitwise-identical defaults of #3779 (wider dequantisation, tuned tables, measured only on a CMP 170HX by the contributor); its many opt-ins stay off, and the reporters are asked to test. Off: `STRATA_DEQUANT_WIDE=0`, `STRATA_MMVQ_IL_SM80=0`.
- **The served web chat page gets a large update.** Wide windows use the width and narrow screens fit (#3176). Saved chats with search, rename and delete, editing a question, rewinding to an earlier turn and branching a new chat from it, sending a message while the answer is still coming (wait, send after, or interrupt), math written in LaTeX rendered with KaTeX, a sandboxed preview of HTML and SVG the model writes, and Load and Unload buttons in the header. The chat view is translated into the same languages as the app, French included. The history features are off until you switch them on in Chat settings: with them off the page sends the same requests as before. Monitor, Images and About are still English.
- **The desktop app.** French is the ninth language. Settings now has: a sound when a task finishes (on request), a desktop notification when Manual mode needs your approval, minimise to the system tray, a fixed port so API clients keep working across updates (with a warning when the port is busy), and an interface size up to 200%. The Monitor has a readings strip: power draw, an electricity price you set (in your local currency, found from the system region), disk temperature where Windows or Linux can report it. The DeepSeek card says how much free disk the set needs. Not tried inside the real packaged app on screen yet: tray icon, the native notification and the busy-port warning. [TO CONFIRM: run once in the release e2e]
@@ -37,8 +38,7 @@ A variable or flag you set yourself still wins over the switch, and the older si
- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details.
- **Prefill on one card, decode on another** (`--split-serve P,D`, or `split_serve` in the run config; #4000, first version). The idea, the hand-made setup (an RTX 3070 reading the prompt, two MI50 cards making the answer) and the first measurements are by @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers. The words are the same as with one engine, token for token (checked at 4,096 and 16,384 tokens, with equal state fingerprints). On two RTX A6000 (3 rounds, 6 values per cell, whole-request time) it was slower than the normal layer split on every workload: an 8K prompt took 16% longer, a 32K prompt 48% longer, a 3-turn chat 30% and 4% longer. So we do not recommend it on two alike cards. It can only pay with two different cards, one good at reading prompts and one good at making answers; we have not measured such a pair. `docs/SPLIT_SERVE.md` has the numbers and the limits.
-- **Pipelined drafts** (`STRATA_PIPED_MTP`, #4121 by @wannagoxuf): the draft model also runs for batch slots. On two RTX A6000 cards with a batch of slots, two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%. Opt-in: `STRATA_PIPED_MTP=1` with `--batch-mtp`; every slot still decodes exactly the tokens it decodes alone.
-- **Layer-major prefill on a CUDA layer split is ON by default (#3976, ports of two contributors).** Long prompts on two NVIDIA cards read much faster. On a layer split over two CUDA cards the prompt is now read layer by layer in large pieces, so each layer's experts are read once per piece instead of once per 8,192 tokens. On two RTX A4000 (Qwen IQ3_XXS, experts streamed, five paired runs): first token after a 32K prompt 141.8 s to 27.7 s (5.1x), after a 20K prompt 98.8 s to 17.5 s (5.7x), after an 8K prompt 41.8 s to 15.2 s (2.8x); the answer speed is the same (38.6 against 38.4 tok/s), the 12 needle questions at 16K and 32K were all found, and the 19-task quick suite passed 18 of 19 with and without it (13 answers byte-identical, the 6 others differ in wording only). The numbers differ slightly from the old path because the experts' results are added in another order: the first-token KL against the old path is at the level of the difference between two prompt chunk sizes, and the most likely token matched on every prompt. One card, Radeon and the approximate-read mode keep the old path. Off: `STRATA_PREFILL_LAYER_MAJOR=0`, or `STRATA_DEFAULTS=026`. From PR 3976 (@Norhiro, building on @vizdatom's #1670).
+- **Pipelined drafts for batch slots** (`STRATA_PIPED_MTP=1` with `--batch-mtp`, #4121 by @wannagoxuf). The draft model also runs for the slots of a batch on a two-card layer split. On two RTX A6000 with a batch of slots, five paired runs before the merge: two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%; the merged build, three pairs: +12.2% and +0.2%. Every slot still decodes exactly the tokens it decodes alone (checked slot by slot against a solo run). Separately, on two NVIDIA cards the standard drafting for a batch is 15 to 18% slower than the pipelined way even without this switch (two streams 107.4 against 125.7 tok/s, four streams 141.0 against 171.4); it was already so in 0.2.6. [pending: decision on the two-card batch default - (a) keep it, (b) pipelined by default, (c) pipelined only when every expert is resident; branches f-027-gh-eng-opt-flip and f-027-gh-eng-opt-resid]
- **Huge pages for DeepSeek on Linux.** If you reserve 2 MB huge pages (`vm.nr_hugepages`) big enough for the expert copy, the engine uses them and copies prompt data in place: an 8K prompt on the 2.25 set read 651 tok/s without and 694 with (R9700 limited to 12 GB; that run also set two buffers by hand). Without a reserved pool nothing changes. Not run with a reserved pool on the release package.
- **Settings that help if your PC fits them, no engine change:** `--ple-io ram` (the 28.8 GB table in RAM; on the 72 GB card prompts +16.5% at 1K, +30.6% at 4K, +16.8% at 8K), `--mtp-draft-vocab draft_vocab_en.bin` (+2.7 to +3.1% decode for English text), `--slots 2` for two conversations at once (+10% to +12% aggregate throughput on RTX PRO 5000 and RTX 3060), `--prefill-pipe` on a two-card layer split (+15% at 4K, +23% at 8K prompts on two RTX A4000). `--kv q4_0` on a 12 GB card did not reproduce a gain, so it is a choice, not a default.
@@ -48,21 +48,23 @@ Each issue number gets its own reply.
- Chat: scrolling up while an answer streams works (#4014); a request that timed out shows a badge (#3991).
- The Monitor asks the NVIDIA tool only while it is open or a model runs, which stops the Windows stutter (#4066). The app no longer insists on CUDA when you have a HIP card on Windows (#3892), and the source-checkout host starts an engine with an empty API key (#3851). Deleting a model in the app removes all files of a multi-file model and leaves your own files (#3716).
-- The Terminal: `strata launch --set dspark=on` works (#3959); `launch --show-key` knows the config setup wrote (#3570); a model on disk in a differently cased folder is found on Linux instead of being offered for download again (#3996); `--spec 5` can be saved with a warning (#3844) [pending: f-027-gh-eng-d].
+- The Terminal: `strata launch --set dspark=on` works (#3959); `launch --show-key` knows the config setup wrote (#3570); a model on disk in a differently cased folder is found on Linux instead of being offered for download again (#3996); `--spec 5` can be saved with a warning (#3844).
- Manual mode: read-only commands that have write or execute options now ask first (#3975).
-- Engine [pending: f-027-gh-eng]: a conversation cache no longer reuses another conversation's keys and values (#1503); a short draft-vocabulary row can't be written past (#3911); a checkpoint save that runs out of memory no longer exits the engine (#3350); a hand build for RTX 50 no longer ends up as sm_75 (#4029, #4023); the DeepSeek CUDA packages now include RTX 40 cards (sm_89, #3935); an undeclared tool call returned as text is recovered, declared-only kept (#3796).
-- Engine, audit fixes [pending: f-027-gh-eng-d]: a failed read in the per-layer table reader no longer leaves the next request waiting for ever (#3602); a conversation restore that would have to create a new copy stream on a card with almost no free memory goes back to the cache instead of failing (#2409); an engine that cannot allocate on a layer split now says how much memory it planned to keep free and how much is free (#3638, #3702); `--spec 5` and other values outside the tuner's range can be saved with a warning (#3844).
+- Engine: a conversation cache no longer reuses another conversation's keys and values (#1503); a short draft-vocabulary row can't be written past (#3911); a checkpoint save that runs out of memory no longer exits the engine (#3350); one cause of the free memory of the second card shrinking with every long cold prompt on a layer split is fixed, a leaked copy stream (#3470, #3350); a hand build for RTX 50 no longer ends up as sm_75 (#4029, #4023); the DeepSeek CUDA packages now include RTX 40 cards (sm_89, #3935); an undeclared tool call returned as text is recovered, declared-only kept (#3796); `--spec 5` and other values outside the tuner's range can be saved with a warning (#3844).
+- Engine, found by reading the code and the pull requests: a failed read in the per-layer table reader no longer leaves the next request waiting for ever (#3602; the reporter's NaN did not reproduce here, `STRATA_PLE_CHECK=1` re-reads a suspect row and prints it); a conversation restore that would have to create a new copy stream on a card with almost no free memory goes back to the cache instead of failing (#2409); an engine that cannot allocate on a layer split says how much memory it planned to keep free and how much is free (#3638, #3702); the fused int8 tails of `STRATA_QFUSE` used a division that differed from the matrix kernels in about one code per million (#3948, found by a parity test, that switch is off by default, the default path is unchanged); `/v1/responses` accepts -1 and 0 as output limits and reports timings (#4036, #3291); logprobs and allowed token ids on /v1/chat/completions (#3200, partly).
+- Opt-in switches from pull requests, each checked for exactness or by unit tests and builds, not for speed on your card: a bigger Pascal (GP100) set (`STRATA_DOWN_WS=1`, `STRATA_MTP_Q8_SM60=1`, `STRATA_MMQ_WS=1`, `STRATA_PATTN_SM60=1`, `STRATA_EXPERT_NIBBLE=1`), `STRATA_PF_FUSED_Q2N=1` for the Q2_0 pack on sm_80 and newer, `STRATA_SELECT_LONG=1` for decoding past 64K cells, `STRATA_VRAM_FREE=device` (Windows, NVML), `STRATA_RAM_ADAPT`, `STRATA_PIPE_ADAPT_EVERY`, `STRATA_SPEC_PROB=1` (a gated draft proposal that keeps the target probabilities, #3416), the sparse copy events on Windows HIP (#4005), and an ARM64 Linux port of the CPU engine whose x86 code compiles to the same objects (#3622, built and cross-compiled, never run on an ARM machine). The speed figures in those pull requests are their authors', not ours.
- Setup and Docker: setup accepts engines that come from the update feed (#4027); a silent re-download in Docker fails loudly (#3421); the Docker image stage carries the scripts the Terminal needs (#3974, #3644); images are tagged 0.2.7.
- Docs: TROUBLESHOOTING is back with every entry that the 0.2.5 rewrite dropped, updated for 0.2.7; the Strix Halo Windows guide (#3029, first written by @HaiYunChuang, measured again here); a Polish README (#3355, by @j-luwierski, the 8th README).
- A graphics card that went through GPU resets can hang engine starts until the PC is rebooted. We saw it on a 4 x R9700 box; 40 loads after a fresh boot had no stall.
-Names stay on the commits: 35 contributors' pull requests were ported with their authorship.
+Names stay on the commits: 55 contributors' pull requests were ported with their authorship.
## What we did not test
- Prompts longer than a few thousand tokens at 512K or 1M DeepSeek contexts.
- RTX 40 cards (no Ada card here), A100-class cards, RX 6000 on Windows (#3895) and a two-GPU PC with a slow CPU (#3930): shipped from the reporters' measurements; please tell us.
- Whether the new summation orders (RTX PRO and Ampere-class cards) change answers: a 61-task check on the RTX PRO 5000 passed 54 against 55 with the switches off, and its differences stayed inside the spread between two identical runs; it is a small sample.
+- Layer-major prefill on cards other than two RTX A4000 (speed) and two RTX A6000 (quality), on Radeon, and on three or more cards; the opt-in switches above on their own hardware.
- The huge-page path with a reserved pool on the release package.
- The DeepSeek Strix Halo package on a 64 GB or 32 GB machine, and the 2.25 set on it.
- The route prior on the owner's RTX 5070: it applies there, and the quality check ran on the RTX 3060.
@@ -77,7 +79,7 @@ Names stay on the commits: 35 contributors' pull requests were ported with their
## Downloads
-Three installers, the three Control packages (0.1.52), 13 Qwen engine packages and 4 DeepSeek engine packages (all 0.1.52: Windows CUDA 13, Windows AMD for Ryzen AI Max, Linux CUDA 13, Linux AMD rx9070), plus `strata-feed.json` with its signature, `latest.json`, `install-manifest.txt`, `install.sh`, `install.ps1` and `SHA256SUMS.txt`. The DeepSeek CUDA packages are built for sm_86, sm_89 and sm_120. [pending: f-027-gh-eng]
+Three installers, the three Control packages (0.1.52), 13 Qwen engine packages and 4 DeepSeek engine packages (all 0.1.52: Windows CUDA 13, Windows AMD for Ryzen AI Max, Linux CUDA 13, Linux AMD rx9070), plus `strata-feed.json` with its signature, `latest.json`, `install-manifest.txt`, `install.sh`, `install.ps1` and `SHA256SUMS.txt`. The DeepSeek CUDA packages are built for sm_86, sm_89 and sm_120.
---