Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: release notes 0.2.7 - two-card pipelined default (owner choice a), DeepSeek pictures by default, the shard 1 setup fix
Niko12215a843fd84117
diff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex 5c16ca4e4..41b2108db 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -20,6 +20,8 @@ DRAFT of 27 October. Text in square brackets that starts with "pending" or "TO C- **Four Radeon AI PRO R9700 cards run DeepSeek faster, with five experts per word by default.** With the launcher's own settings, DeepSeek 2.25: an 8K prompt 342 to 399 tok/s on the first read and 967 to 1,350 once warm (+17%, +40%), a 32K prompt 1,127 to 1,604 first and 1,183 to 1,697 warm (+42%, +43%), decode on code 88.9 to 95.0 tok/s (+7%) and on an agent task 73.5 to 79.7 (+8%). Loading takes 58 to 60 s instead of 51 s. The default is now "Faster" (5 of 6 experts per word); the quality run passed 19 of 19 tasks against 18 of 19 with six experts, and the distance to the reference model is the same (0.1102 against 0.1102). Choose Full quality in the app, or `--experts 6`, to use six.- **DeepSeek can use 512K and 1M contexts.** Setup, the Terminal and the app offer them up to the limit of the model files, with a plain "not tested yet" note past 256K. The default stays 8K, and the app now loads a DeepSeek model with the context you chose. On an RTX 5070 a 1M context loaded in 18 s and cost about 1 GB of graphics memory, which pushed 136 experts off the card. We did not test long prompts at these sizes. The idea comes from Project Maya v1.1.28 (#442, by @needmorevram).- **Two NVIDIA cards read long prompts much sooner on a layer split.** On a layer split over two CUDA cards the prompt is now read layer by layer in large pieces (layer-major prefill, on by default), so each layer's experts are read once per piece instead of once per 8,192 tokens. On two RTX A4000 (16 GB each, Qwen IQ3_XXS with most experts streamed from RAM, five paired runs): the first word after a 32K prompt came after 27.7 s instead of 141.8 s, after a 20K prompt after 17.5 s instead of 98.8 s, after an 8K prompt after 15.2 s instead of 41.8 s. Decode speed is the same (38.6 against 38.4 tok/s). Quality: 12 of 12 needles found with both, the quick suite passed 18 of 19 tasks in all three arms, and the first-token probabilities moved by 0.0007 to 0.009 (KL; the same top choice in every check), so wording can differ a little. If a card has no room for the bigger step, the engine reads the old way. From PR #3976 by @Norhiro, building on @vizdatom's #1670. One card, Radeon and the approximate-read mode keep the old path. Off: `STRATA_PREFILL_LAYER_MAJOR=0`; `STRATA_DEFAULTS=026` gives the old path.+- **Two NVIDIA cards share several chats faster.** On two or more NVIDIA cards with a batch of slots, the engine now runs the cards as a pipeline by default; the serial way of 0.2.1 to 0.2.6 was slower at two and four chats. Two RTX A6000 (Qwen IQ3_S, no drafting): two chats +17% (125.7 against 107.4 tok/s), four chats +22% (171.4 against 141.0). Two RTX A4000 (Q2_0 and IQ3_XXS): two chats +7 to 12%, four chats +17 to 20%, one chat 4% slower on IQ3_XXS and equal on Q2_0. Every chat still decodes exactly its own tokens. Off: `STRATA_BATCH_GROUPS_NV_SERIAL=1`, or `STRATA_DEFAULTS=026`.+- **DeepSeek reads pictures by default.** Setup and `strata launch` start DeepSeek-V4.1-Flash with the engine's own vision tower (319 MiB, no separate reader). It costs a one-time download of about 1 GB (the tower's weights, 972 MB, checked against a pinned hash) and about 1.3 GB more RAM on four R9700 (0.2 GB on one card); the graphics memory is taken from the expert cache, so the total does not grow. On four R9700 a picture question (left half red, right half blue) was answered correctly, 7.8 s for the first picture and 0.2 s once warm; on one R9700 12.7 s and 0.5 s; text requests after pictures work. The text-only answers on the agent prompt are identical with and without it; the distance to the reference model moved by +0.0014 on four R9700 and by -0.003 on one (gate 0.006). Not wanted: `--vision off` on launch, or `--vision no` in setup, leaves the tower out. Checked on Linux (R9700, boston); not on Windows or Strix Halo.- **More cards get small gains from kernel changes that were measured.** On an RTX PRO 5000 (72 GB, Qwen IQ3_S, five paired runs): story answers 195.9 to 211.8 tok/s (+8.1%), code 240.6 to 244.7 (+1.7%), prompts +1.2% at 4K, +1.5% at 8K, +1.6% at 16K. On two RTX A4000: story +2.2%, code +0.5%. These switches sum numbers in another order, so a reply can differ from 0.2.6 at a near-tie. RTX 40 cards (compute capability 8.9) use measured speed tables for the quantised matrix products; they are bit for bit the same as before and we have no Ada card, so the reporter's measurements are the only ones. Off: `STRATA_MMVQ_IL_89=0`. A100-class cards (sm_80) get the bitwise-identical defaults of #3779 (wider dequantisation, tuned tables, measured only on a CMP 170HX by the contributor); its many opt-ins stay off, and the reporters are asked to test. Off: `STRATA_DEQUANT_WIDE=0`, `STRATA_MMVQ_IL_SM80=0`.- **The served web chat page gets a large update.** Wide windows use the width and narrow screens fit (#3176). Saved chats with search, rename and delete, editing a question, rewinding to an earlier turn and branching a new chat from it, sending a message while the answer is still coming (wait, send after, or interrupt), math written in LaTeX rendered with KaTeX, a sandboxed preview of HTML and SVG the model writes, and Load and Unload buttons in the header. The chat view is translated into the same languages as the app, French included. The history features are off until you switch them on in Chat settings: with them off the page sends the same requests as before. Monitor, Images and About are still English.- **The desktop app.** French is the ninth language. Settings now has: a sound when a task finishes (on request), a desktop notification when Manual mode needs your approval, minimise to the system tray, a fixed port so API clients keep working across updates (with a warning when the port is busy), and an interface size up to 200%. The Monitor has a readings strip: power draw, an electricity price you set (in your local currency, found from the system region), disk temperature where Windows or Linux can report it. The DeepSeek card says how much free disk the set needs. Not tried inside the real packaged app on screen yet: tray icon, the native notification and the busy-port warning. [TO CONFIRM: run once in the release e2e]@@ -30,7 +32,7 @@ DRAFT of 27 October. Text in square brackets that starts with "pending" or "TO CTwo switches bring back every default that 0.2.7 changed, in one step, for a test or for a PC where something looks wrong.- **`DS4_DEFAULTS=026`** (DeepSeek): the staging ring, the prompt chunk rule, the attention buffer, the pinned copy and the huge-page arena, the matrix-core attention, the drafter's cut, the five-experts default, the RAM store's spare slots and its pre-fill after a prompt, the dense dequantiser, the head and gate/up operand formats and the RDNA3 dense GEMM go back to what 0.2.6 did.-- **`STRATA_DEFAULTS=026`** (Qwen): residency-biased routing, the wide and split hyper-connection reads, the int8 prompt attention, the four gfx12 decode switches, the PLE residency policy, the gfx12 decode split (back to 0.55), the RTX 40 (sm_89) tables, the sm_80 defaults and the layer-major prefill of a two-card CUDA split.+- **`STRATA_DEFAULTS=026`** (Qwen): residency-biased routing, the wide and split hyper-connection reads, the int8 prompt attention, the four gfx12 decode switches, the PLE residency policy, the gfx12 decode split (back to 0.55), the RTX 40 (sm_89) tables, the sm_80 defaults, the layer-major prefill of a two-card CUDA split and the pipelined two-card batch default.A variable or flag you set yourself still wins over the switch, and the older single switches work as before. In the app, "DeepSeek speed-ups" off sends `DS4_DEFAULTS=026`.@@ -38,7 +40,7 @@ A variable or flag you set yourself still wins over the switch, and the older si- **Faster long prompts** (`STRATA_KVA=1`, or the switch in Models, Native engine settings, Speed). Reads the last 12 layers of a very long prompt in an approximate way, for Qwen3.8-Flash-Next on one NVIDIA or AMD card, on the first read of a prompt of 4,096 tokens or more. On an RTX 5070 with IQ3_S it was 11% faster at 14K and 18% faster at 30K. It stays off because earlier, more aggressive settings dropped records from long lists and failed agent tasks. The setting that ships when you turn it on (layers 36 to 47 approximate, the last 2,048 tokens exact) found the same 585 of 606 facts as the exact read and finished 6 of 10 agent tasks where the exact read finished 4 and 7 in two runs. It can still lose details.- **Prefill on one card, decode on another** (`--split-serve P,D`, or `split_serve` in the run config; #4000, first version). The idea, the hand-made setup (an RTX 3070 reading the prompt, two MI50 cards making the answer) and the first measurements are by @MarcoTool. One engine reads the prompt and hands the finished state to a second engine that answers. The words are the same as with one engine, token for token (checked at 4,096 and 16,384 tokens, with equal state fingerprints). On two RTX A6000 (3 rounds, 6 values per cell, whole-request time) it was slower than the normal layer split on every workload: an 8K prompt took 16% longer, a 32K prompt 48% longer, a 3-turn chat 30% and 4% longer. So we do not recommend it on two alike cards. It can only pay with two different cards, one good at reading prompts and one good at making answers; we have not measured such a pair. `docs/SPLIT_SERVE.md` has the numbers and the limits.-- **Pipelined drafts for batch slots** (`STRATA_PIPED_MTP=1` with `--batch-mtp`, #4121 by @wannagoxuf). The draft model also runs for the slots of a batch on a two-card layer split. On two RTX A6000 with a batch of slots, five paired runs before the merge: two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%; the merged build, three pairs: +12.2% and +0.2%. Every slot still decodes exactly the tokens it decodes alone (checked slot by slot against a solo run). Separately, how two NVIDIA cards share several chats at once: today's default is the serial way (since 0.2.1), and the pipelined way is faster at two and four chats. On two RTX A6000 (Qwen IQ3_S, no drafting) pipelined was 17% faster at two chats (125.7 against 107.4 tok/s) and 22% at four (171.4 against 141.0); on two RTX A4000 (Q2_0 and IQ3_XXS) two chats gain 7 to 12%, four chats 17 to 20%, and a single chat loses 4% (IQ3_XXS). The default for this is the owner's choice. [pending: decision on the two-card batch default - (a) pipelined by default on 2+ NVIDIA cards (recommended; branch f-027-gh-eng-opt-flip 7d44e972e), (b) pipelined only when every expert fits on the cards (f-027-gh-eng-opt-resid 7cfa9291d), (c) leave serial]+- **Pipelined drafts for batch slots** (`STRATA_PIPED_MTP=1` with `--batch-mtp`, #4121 by @wannagoxuf). The draft model also runs for the slots of a batch on a two-card layer split. On two RTX A6000 with a batch of slots, five paired runs before the merge: two streams +15.4% (five of five pairs), four streams +4.7%, one stream +0.1%; the merged build, three pairs: +12.2% and +0.2%. Every slot still decodes exactly the tokens it decodes alone (checked slot by slot against a solo run).- **Huge pages for DeepSeek on Linux.** If you reserve 2 MB huge pages (`vm.nr_hugepages`) big enough for the expert copy, the engine uses them and copies prompt data in place: an 8K prompt on the 2.25 set read 651 tok/s without and 694 with (R9700 limited to 12 GB; that run also set two buffers by hand). Without a reserved pool nothing changes. Not run with a reserved pool on the release package.- **Settings that help if your PC fits them, no engine change:** `--ple-io ram` (the 28.8 GB table in RAM; on the 72 GB card prompts +16.5% at 1K, +30.6% at 4K, +16.8% at 8K), `--mtp-draft-vocab draft_vocab_en.bin` (+2.7 to +3.1% decode for English text), `--slots 2` for two conversations at once (+10% to +12% aggregate throughput on RTX PRO 5000 and RTX 3060), `--prefill-pipe` on a two-card layer split (+15% at 4K, +23% at 8K prompts on two RTX A4000). `--kv q4_0` on a 12 GB card did not reproduce a gain, so it is a choice, not a default.@@ -53,6 +55,7 @@ Each issue number gets its own reply.- Engine: a conversation cache no longer reuses another conversation's keys and values (#1503); a short draft-vocabulary row can't be written past (#3911); a checkpoint save that runs out of memory no longer exits the engine (#3350); one cause of the free memory of the second card shrinking with every long cold prompt on a layer split is fixed, a leaked copy stream (#3470, #3350); a hand build for RTX 50 no longer ends up as sm_75 (#4029, #4023); the DeepSeek CUDA packages now include RTX 40 cards (sm_89, #3935); an undeclared tool call returned as text is recovered, declared-only kept (#3796); `--spec 5` and other values outside the tuner's range can be saved with a warning (#3844).- Engine, found by reading the code and the pull requests: a failed read in the per-layer table reader no longer leaves the next request waiting for ever (#3602; the reporter's NaN did not reproduce here, `STRATA_PLE_CHECK=1` re-reads a suspect row and prints it); a conversation restore that would have to create a new copy stream on a card with almost no free memory goes back to the cache instead of failing (#2409); an engine that cannot allocate on a layer split says how much memory it planned to keep free and how much is free (#3638, #3702); the fused int8 tails of `STRATA_QFUSE` used a division that differed from the matrix kernels in about one code per million (#3948, found by a parity test, that switch is off by default, the default path is unchanged); `/v1/responses` accepts -1 and 0 as output limits and reports timings (#4036, #3291); logprobs and allowed token ids on /v1/chat/completions (#3200, partly).- Opt-in switches from pull requests, each checked for exactness or by unit tests and builds, not for speed on your card: a bigger Pascal (GP100) set (`STRATA_DOWN_WS=1`, `STRATA_MTP_Q8_SM60=1`, `STRATA_MMQ_WS=1`, `STRATA_PATTN_SM60=1`, `STRATA_EXPERT_NIBBLE=1`), `STRATA_PF_FUSED_Q2N=1` for the Q2_0 pack on sm_80 and newer, `STRATA_SELECT_LONG=1` for decoding past 64K cells, `STRATA_VRAM_FREE=device` (Windows, NVML), `STRATA_RAM_ADAPT`, `STRATA_PIPE_ADAPT_EVERY`, `STRATA_SPEC_PROB=1` (a gated draft proposal that keeps the target probabilities, #3416), the sparse copy events on Windows HIP (#4005), and an ARM64 Linux port of the CPU engine whose x86 code compiles to the same objects (#3622, built and cross-compiled, never run on an ARM machine). The speed figures in those pull requests are their authors', not ours.+- Setup: ISTA re-published the first shard of every DeepSeek set on 26 October with a larger chat template; setup refused a copy of it from `--model-path` with "cut short or another version". It accepts it now (by its own size and hash; downloads stay on the pinned revision). The same 0.2.6 setup bug is fixed for people who already have the files.- Setup and Docker: setup accepts engines that come from the update feed (#4027); a silent re-download in Docker fails loudly (#3421); the Docker image stage carries the scripts the Terminal needs (#3974, #3644); images are tagged 0.2.7.- Docs: TROUBLESHOOTING is back with every entry that the 0.2.5 rewrite dropped, updated for 0.2.7; the Strix Halo Windows guide (#3029, first written by @HaiYunChuang, measured again here); a Polish README (#3355, by @j-luwierski, the 8th README).- A graphics card that went through GPU resets can hang engine starts until the PC is rebooted. We saw it on a 4 x R9700 box; 40 loads after a fresh boot had no stall.@@ -65,6 +68,7 @@ Names stay on the commits: 55 contributors' pull requests were ported with their- RTX 40 cards (no Ada card here), A100-class cards, RX 6000 on Windows (#3895) and a two-GPU PC with a slow CPU (#3930): shipped from the reporters' measurements; please tell us.- Whether the new summation orders (RTX PRO and Ampere-class cards) change answers: a 61-task check on the RTX PRO 5000 passed 54 against 55 with the switches off, and its differences stayed inside the spread between two identical runs; it is a small sample.- Layer-major prefill on cards other than two RTX A4000 (speed) and two RTX A6000 (quality), on Radeon, and on three or more cards; the opt-in switches above on their own hardware.+- DeepSeek pictures on Windows, on Strix Halo and on NVIDIA cards other than the boston A6000 pair.- The huge-page path with a reserved pool on the release package.- The DeepSeek Strix Halo package on a 64 GB or 32 GB machine, and the 2.25 set on it.- The route prior on the owner's RTX 5070: it applies there, and the quality check ran on the RTX 3060.