Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: Windows Strix Halo leaves STRATA_SH_STREAM off, with the token-id check; DeepSeek numbers say which build they come from
Niko1221f932e410e652
diff --git a/docs/STRIX_HALO.md b/docs/STRIX_HALO.mdindex 72e55c066..7e0a23c39 100644--- a/docs/STRIX_HALO.md+++ b/docs/STRIX_HALO.md@@ -5,8 +5,9 @@ shared by the CPU and the GPU). Strata runs there through the same HIP backend a**Status: experimental.****DeepSeek V4.1-Flash runs here on Windows** (since 0.2.7): its own engine package is built for gfx1151 and the official IST-DAS-1.75 / 2.00 / 2.25 bpw sets run through it. Measured on the 128 GB box (64 GB BIOS carve-out plus 64 GB seen by Windows) with 2.00 bpw and the default Faster mode: answers 15-17-tokens/s on code, 10-11 on chat; a 4K prompt 87 tokens/s, an 8K prompt 125 and a 32K prompt 177. The graphics chip and the CPU share the RAM, so the+1.75 / 2.00 / 2.25 bpw sets run through it. Measured on the 128 GB box (64 GB BIOS carve-out plus 64 GB seen by Windows) with 2.00 bpw and the default Faster mode, on the+2026-10-25 build (before the DeepSeek defaults of 0.2.7): answers 15-17 tokens/s on code, 10-11 on chat; a 4K prompt 87 tokens/s, an 8K prompt 125 and a 32K prompt 177.+On the 0.2.7 build the same box reads a 4K prompt at 214-233 tokens/s; the table below says which number comes from which build. The graphics chip and the CPU share the RAM, so theengine counts it once and reads the rest of the experts from the SSD. [DeepSeek on Strix Halo](#deepseek-v41-flash) has the details,the driver it needs and the limits.@@ -47,7 +48,8 @@ package is the AMD one built for gfx1151 (`strata-dsv41-...-windows-x86_64-hip-r| Context | 8K default; 32K and 64K work (an 8-hour 32K soak passed) || Disk | the sets above; keep the files on the NVMe drive, they are read while it answers |-Measured on the maintainers' box (Ryzen AI Max+ 395, 128 GB unified, Windows 11, 2.00 bpw, Faster mode, DSpark auto, engine 0.1.51):+Measured on the maintainers' box (Ryzen AI Max+ 395, 128 GB unified, Windows 11, 2.00 bpw, Faster mode, DSpark auto, engine 0.1.51) on 2026-10-25, with the+build of that day and the DeepSeek stream's own test harness (before the 0.2.7 DeepSeek defaults; the 0.2.7 numbers are in the next table):| Workload | Result || --- | --- |@@ -59,6 +61,21 @@ Measured on the maintainers' box (Ryzen AI Max+ 395, 128 GB unified, Windows 11,| 32K prompt | about 177 tokens/s || Stability | 100-request, 1-hour (243 requests) and 8-hour 32K (646 requests) soaks passed on the engine line this package is built from |+Measured again on 2026-10-26 on the same box with the 0.2.7 release candidate (f-027-int2 `3e468f5de`, engine 0.1.51), installed through `strata launch` from a local+signed test feed, 2.00 bpw, Faster mode, DSpark auto, the default 8K context, `tools/strata_bench.py` through the server (three fresh prompts of 4,125+tokens, 24 new tokens; chat is the median of nine answers of up to 100 tokens). One change at a time, to see which one moved the 4K prompt:++| Setup on the 0.2.7 candidate | 4K prompt (tok/s) | Chat decode (tok/s) |+| --- | ---: | ---: |+| what `strata launch` writes (`--prefill-chunk 16384`) | 214-226 | 12.1-13.6 |+| `--prefill-chunk 8192` | 229 (214-234) | 14.9 (12.0-18.2) |+| `--prefill-chunk 8192` and `DS4_DEFAULTS=026` (the 0.2.6 DeepSeek defaults) | 139 (133-144) | 13.7 (11.9-22.0) |+| `--prefill-chunk 1024` and `DS4_DEFAULTS=026` | 133 (129-136) | 14.2 (12.3-22.1) |++The 0.2.7 DeepSeek defaults account for the gap between 139 and 229; the chunk size matters much less than that on this build. The 0.2.7+candidate with the 0.2.6 defaults still reads 133-139, above the 78.7-87.2 of 2026-10-25 below. That rest is not explained: the build of that+day, the harness and the prompt were different, and only this table changes one thing at a time. Compare numbers within one table, not across.+The verify window runs synchronously on this PC: setup writes `DS4_ASYNC_T=4` in the config's env (`STRATA_DSV41_ASYNC_T=8`restores the engine's default, any other value overrides). The engine's mailbox ("async") path is sized for cards that hold mostof the experts and is slower here: measured 2026-10-25 (2.00 bpw, greedy, 256 tokens): code 15.87-16.33 → 20.14-21.28 tok/s@@ -67,7 +84,7 @@ the scheduling changes, and the drafter's own single-token windows stay async. T(its experts leave no room for it) and does not change.Prompts are read in 8,192-token chunks: setup writes `--prefill-chunk 8192` on this PC (STRATA_DSV41_NO_BIG_CHUNK=1 leaves the-engine's 1,024-token default). Measured 2026-10-25: 4K 78.7 → 87.2, 8K 108.9 → 125.1, 32K 155.2 → 176.9 tok/s; decode unchanged+engine's 1,024-token default). Measured 2026-10-25 (the build of that day, before the 0.2.7 defaults): 4K 78.7 → 87.2, 8K 108.9 → 125.1, 32K 155.2 → 176.9 tok/s; decode unchanged(16.53 → 16.59), because the chunk buffers take ~5 GB from the GPU expert tier and the RAM store's hit rate rises to compensate. Thechunk's summation order changes, not the model (the repo's chunk-size KL check: 1024 0.1705, 2048 0.1692, 4096 0.1688, 8192 0.1695).@@ -169,7 +186,7 @@ kernel): they change speed, not answers. Nothing here applies to CUDA or to anot| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | Decode layouts and kernels: packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores. || `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | The grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append. || `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | The GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once. |-| `STRATA_SH_STREAM=1` | The shared expert on a second stream beside the routed experts. Off by default on other AMD cards, but it pays on gfx1151 (decode +1.8% UD-IQ4_XS, +6.7% UD-Q4_K_XL at 8K). |+| `STRATA_SH_STREAM=1` | The shared expert on a second stream beside the routed experts. Off by default on other AMD cards, but it pays on Linux gfx1151 (decode +1.8% UD-IQ4_XS, +6.7% UD-Q4_K_XL at 8K). **Off by default on Windows gfx1151 since 0.2.7:** it measured 6.9% slower there with the same token ids ([STRIX_HALO_WINDOWS.md](STRIX_HALO_WINDOWS.md)). `STRATA_SH_STREAM=1` turns it back on. |Two further defaults change bits. `STRATA_HC_Q8` reads the hyper-connection weights as 8-bit (`STRATA_HC_Q8=0` restores the BF16read), and on Windows the prompt is read in chunks of 8192 tokens (`STRATA_GFX1151_CHUNK=0` keeps 6144). About `STRATA_HC_Q8`: it@@ -274,7 +291,7 @@ more than the differences because other jobs shared the box, so read them as unc| IQ3_XXS | 8K | 1,127 -> 1,212 (+7.6%) | 51.64 -> 50.95 (-1.3%) || IQ3_XXS | 128K | 1,252 -> 1,250 (-0.1%) | 50.88 -> 50.22 (-1.3%) |-With the shared-expert stream (`STRATA_SH_STREAM=1`, now a default) at 8K, 4 pairs: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%),+With the shared-expert stream (`STRATA_SH_STREAM=1`, a default on Linux; Windows leaves it off since 0.2.7) at 8K, 4 pairs: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%),UD-Q4_K_XL 49.5 -> 52.8 (+6.7%), prompt unchanged.**Other switch measurements** (Measured on: the same Ryzen AI Max+ 395)diff --git a/docs/STRIX_HALO_WINDOWS.md b/docs/STRIX_HALO_WINDOWS.mdindex 63bdeb0d5..65681fa9b 100644--- a/docs/STRIX_HALO_WINDOWS.md+++ b/docs/STRIX_HALO_WINDOWS.md@@ -42,6 +42,10 @@ the default 8K context, the others 64K context with `--context 64k`):These are speed measurements. They say nothing about answer quality, and the short-chat range depends on the text. The DeepSeek 2.25 bpw setand the other Qwen sizes were not measured for this page.+The DeepSeek 4K prompt here (214 to 226 tok/s) is higher than the 87 tok/s on [STRIX_HALO.md](STRIX_HALO.md), which was measured on 2026-10-25 with an+earlier build. The 0.2.7 DeepSeek defaults explain most of the gap (the same build with `DS4_DEFAULTS=026` reads 133 to 139 tok/s); the table there+changes one thing at a time.+## 2. Memory and the BIOS- Allocate a **fixed carve-out** to the graphics chip in the BIOS (64 GB on the measured PC). The engine counts the carve-out and the@@ -76,7 +80,7 @@ environment and the package's `rocm/bin` folder on the library path.## 4. What is on by default-- On `gfx1151` the engine turns on 19 exact speed switches by itself (the start line lists them; `STRATA_GFX1151_DEFAULTS=0` turns them all+- On `gfx1151` the engine turns on exact speed switches by itself (19 on Linux, 18 on Windows, which leaves `STRATA_SH_STREAM` out, see below; the start line lists them; `STRATA_GFX1151_DEFAULTS=0` turns them alloff, and a switch you set is kept). They are the ones that give the same answers as without. `STRATA_HC_Q8=1` is also on here (decodeabout 9% faster, near-tie tokens can differ); `STRATA_HC_Q8=0` keeps the old read.- The hipBLASLt table for `gfx1151` ships in the package and is chosen without a setting (`prefill gemm: hipBLASLt tuning enabled (32 rows,@@ -85,13 +89,14 @@ environment and the package's `rocm/bin` folder on the library path.budget and decode fell to 28 tok/s on engine 0.1.41; the engine on this page sizes the cache from the shared pool and reaches every slot.- DeepSeek runs in "Faster" mode here: 5 of 6 experts per word. The app and `strata launch` say so; the quality change is small.-**`STRATA_SH_STREAM` (the shared expert on its own stream) is on by default on `gfx1151`, and on this Windows PC it is slower.** The first-Windows guide found the same on engine 0.1.41 (+13% decode with it off). Measured again on 0.2.7 with Qwen IQ3_S, three starts each, run-in turn (on, off, on, off, on, off), the same quick suite after every start:+**`STRATA_SH_STREAM` (the shared expert on its own stream) is off by default on Windows `gfx1151` in 0.2.7, because it is slower there.** It stays+on by default on Linux `gfx1151`, where it was measured to help ([STRIX_HALO.md](STRIX_HALO.md)). The first Windows guide found the Windows+result on engine 0.1.41 (+13% decode with it off). Measured again on the 0.2.7 build as it was before this default changed (the switch was+on), with Qwen IQ3_S, three starts each, run in turn (on, off, on, off, on, off), the same quick suite after every start:| Start | `STRATA_SH_STREAM` | Chat decode, median of 9 answers (tok/s) | 4K prompt (tok/s) || --- | --- | ---: | ---: |-| 1 | 1 (default) | 43.9 | 742.9 |+| 1 | 1 (the default then) | 43.9 | 742.9 || 2 | 0 | 48.3 | 742.9 || 3 | 1 | 43.6 | 746.8 || 4 | 0 | 46.0 | 740.8 |@@ -99,9 +104,16 @@ in turn (on, off, on, off, on, off), the same quick suite after every start:| 6 | 0 | 46.6 | 739.4 |With the switch off, decode was 6.9% faster (46.6 against 43.6, the median of the three starts) and the code-like question 14% faster-(67 against 59 tok/s); all three pairs went the same way. The prompt speed did not change. Setting it takes `strata launch ... --set-STRATA_SH_STREAM=0`; it changes only the order the work runs in, so the answers stay the same. Whether Windows `gfx1151` gets this by-default is a separate decision. This is one model, one PC and short answers; DeepSeek was not tested with it.+(67 against 59 tok/s); all three pairs went the same way. The prompt speed did not change.++The answers stay the same: on the same PC and build I ran five fixed greedy prompts (a short question, a code request, an explanation, a+4,091-token prompt and a 16,383-token prompt; 895 generated token ids) through the engine four times, with the switch on, off, on, off. The+token ids were identical in all four runs for every prompt.++What 0.2.7 does: on Windows `gfx1151` the engine no longer turns the switch on by itself. To turn it back on, use `strata launch ...+--set STRATA_SH_STREAM=1` (or put `STRATA_SH_STREAM=1` in the environment of an engine you start by hand); a value you set is always kept.+`STRATA_DEFAULTS=026` brings back everything 0.2.6 did, including the switch on. This is one model, one PC and short answers; DeepSeek was+not tested with it, and Linux `gfx1151` has not been measured against it.## 5. Trapsdiff --git a/docs/release-notes-0.2.7.md b/docs/release-notes-0.2.7.mdindex 6e806adef..a37622782 100644--- a/docs/release-notes-0.2.7.md+++ b/docs/release-notes-0.2.7.md@@ -41,6 +41,13 @@ A variable or flag you set yourself (`DS4_ATTN_SLAB=N`, `STRATA_ROUTE_PRIOR=2`,`--ple-io`) still wins over the switch. The older single switches (`DS4_FAST_ATTN=0`, `DS4_ATTN_SLAB=0`, `DS4_COPY_PIN=0`,`DS4_COPY_HUGETLB=0`, `DS4_STORE_SPARE=0`, `DS4_STORE_PUT=0`, `STRATA_ROUTE_PRIOR=0`, ...) work as before.+- **Strix Halo (`gfx1151`) on Windows: the shared expert no longer runs on its own stream.** `STRATA_SH_STREAM` was on by default on+ `gfx1151`; on a Windows PC it is slower. Qwen3.8-Flash-Next IQ3_S, three interleaved pairs: chat decode **43.6 -> 46.6 tokens/s (+6.9%)**+ with it off, prompts unchanged. The answers are the same: five fixed greedy prompts (up to 16,383 tokens) gave identical token ids with+ the switch on and off, in four runs. Linux `gfx1151` keeps it on, other cards are untouched. Turn it back on with+ `STRATA_SH_STREAM=1`; `STRATA_DEFAULTS=026` restores the 0.2.6 behaviour. Details and the other numbers of that PC:+ [docs/STRIX_HALO_WINDOWS.md](STRIX_HALO_WINDOWS.md).+## What is faster if your PC is set up for itThese need no engine change; they are settings. Each was measured on the same boxes.