Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs/STRIX_HALO_WINDOWS.md: a Strix Halo Windows guide, measured again on 0.2.7 (#3029)
Niko1221ee1a6efd5cea
diff --git a/docs/README.md b/docs/README.mdindex ed4ebe063..9baad4415 100644--- a/docs/README.md+++ b/docs/README.md@@ -22,7 +22,7 @@ Use the guide for your GPU and operating system. Model and feature support can d| --- | --- || NVIDIA RTX 20 / 30 / 40 / 50 | [Install](INSTALL.md#what-you-need) || AMD Radeon (RX and AI PRO R9700) | [AMD / HIP](AMD_HIP.md) |-| AMD Ryzen AI Max / Strix Halo | [Strix Halo](STRIX_HALO.md) |+| AMD Ryzen AI Max / Strix Halo | [Strix Halo](STRIX_HALO.md), [on Windows](STRIX_HALO_WINDOWS.md) || Intel Arc on Linux | [Intel Arc](INTEL_ARC.md) || Apple Silicon | [Mac / Metal](MAC.md) || More than one GPU | [Multiple GPUs](MULTI_GPU.md) |diff --git a/docs/STRIX_HALO.md b/docs/STRIX_HALO.mdindex e37db19fc..72e55c066 100644--- a/docs/STRIX_HALO.md+++ b/docs/STRIX_HALO.md@@ -32,6 +32,8 @@ change BIOS or kernel settings. The manual Linux toolchain below is for reproducDeepSeek V4.1-Flash comes with its own engine package on Windows and needs nothing extra: pick the model and the app or `strata launch`downloads it ([DeepSeek on Strix Halo](#deepseek-v41-flash)).+On Windows, [STRIX_HALO_WINDOWS.md](STRIX_HALO_WINDOWS.md) has the speeds measured on 0.2.7 with Qwen IQ3_S and DeepSeek, how memory is used and the traps.+## DeepSeek V4.1-FlashThe model runs through its own engine, `strata-dsv41`, as on the other platforms ([DEEPSEEK41.md](DEEPSEEK41.md)); on Windows thediff --git a/docs/STRIX_HALO_WINDOWS.md b/docs/STRIX_HALO_WINDOWS.mdnew file mode 100644index 000000000..3f4fca8b4--- /dev/null+++ b/docs/STRIX_HALO_WINDOWS.md@@ -0,0 +1,125 @@+# Strix Halo (Ryzen AI Max, gfx1151) on Windows: install, what you get, what to know++This page is for a Windows PC with a Ryzen AI Max "Strix Halo" chip (Radeon 8060S / 8050S, `gfx1151`). It says what Strata does there out+of the box, what speed to expect with Qwen3.8-Flash-Next IQ3_S and with DeepSeek V4.1-Flash, and the traps on this hardware. The Linux+build, the toolchain and the opt-in switches are on [STRIX_HALO.md](STRIX_HALO.md); the DeepSeek details are in+[DEEPSEEK41.md](DEEPSEEK41.md).++The structure of this page comes from the first Windows guide for this PC, written by HaiYunChuang (pull request #3029). It measured+engines 0.1.41 to 0.1.47 with a hand-made configuration (`--mmap-experts`, a list of switches, a batch file). Since then the app and+`strata launch` have taken over most of that, so this version describes the 0.2.7 install and every number in it was measured again.++## 1. What this PC reaches++Measured on 2026-10-26 on a GMKtec NucBox EVO-X2 (Ryzen AI Max+ 395, 128 GB shared memory: a 64 GB BIOS carve-out for the graphics chip+and 64 GB that Windows sees; Windows 11 Pro, AMD Adrenalin 26.10.1 / driver 32.0.32015.7029). Strata was the 0.2.7 release candidate+(Terminal and engine 0.1.51 built from commit `3e468f5de`, ROCm 10.2.0a20260930). The server was started with `strata launch` and+nothing else was set by hand; the speeds come from `tools/strata_bench.py` against the running server (three runs per test, medians,+the engine's own timings; the model is loaded before the clock starts). The numbers can still move before the release is cut.++**Qwen3.8-Flash-Next IQ3_S** (`strata launch --family qwen --model iq3_s`; 131,072 tokens of context, int8 KV, pictures on):++| Test | Result |+| --- | --- |+| Short chat, answer speed | **44.0 tok/s** (40.6 to 59.5: code-like answers are faster than prose) |+| Short chat, time to first token | 0.60 s |+| 4K prompt, read speed | **755 tok/s** (first token after 5.4 s) |+| 16K prompt | **821 tok/s** (19.6 s) |+| 32K prompt | **792 tok/s** (40.6 s) |+| Coding-agent session, 6 turns | 69% of the prompt came from the conversation cache; answers 47 tok/s; 4.5 s to the first token per turn |++**DeepSeek V4.1-Flash 2.00 bpw** (`strata launch --family deepseek41 --model ist-das-2.00bpw`, Faster mode, the drafter on; the 4K row is+the default 8K context, the others 64K context with `--context 64k`):++| Test | Result |+| --- | --- |+| Short chat, answer speed | **12.1 to 13.6 tok/s** |+| 4K prompt | **214 to 226 tok/s** (about 19 s to the first token) |+| 16K prompt | **326 tok/s** (51 s) |+| 32K prompt | **342 tok/s** (97 s) |+| Coding-agent session, 6 turns | 70% from the conversation cache; answers 14.2 tok/s; 17 to 31 s to the first token per turn |++These are speed measurements. They say nothing about answer quality, and the short-chat range depends on the text. The DeepSeek 2.25 bpw set+and the other Qwen sizes were not measured for this page.++## 2. Memory and the BIOS++- Allocate a **fixed carve-out** to the graphics chip in the BIOS (64 GB on the measured PC). The engine counts the carve-out and the+ RAM that Windows sees as one pool and does not change any BIOS or driver setting.+- The graphics chip and the CPU share that pool. For Qwen IQ3_S the engine reads the experts into RAM, keeps them there (it does not+ register the arena with the GPU driver on an integrated chip) and gives the expert cache every slot: the start log says+ `expert cache auto ... -> 24576 slots` and `expert arena: NOT registered with the GPU driver (integrated GPU, UMA mode)`. The process held+ about 49 GB and 10 GB of RAM stayed free. Close other large programs before you load a model this size.+- The default of four parallel slots takes 7.5 GiB of the shared pool for slot sessions (the log says `--batch: 4 slot sessions ... take+ 7.54 GiB`). With IQ3_S the cache still held every expert, so there was nothing to gain from fewer slots here; with a bigger model there may be.++## 3. Install++1. Use the Windows app (`Strata-<version>-windows-x64-setup.exe`) or the Terminal. The engine for this chip is the `windows-x86_64-hip-rdna3`+ package; the app or `strata launch` downloads it on the first start, and `strata update` keeps it current.+2. Choose the model in the app, or `strata launch --family qwen --model iq3_s`. With model files you already have, add+ `--model-path <folder>` (for IQ3_S the two shards are in one folder: shard 1 is the main model, shard 2 holds the per-layer embedding+ table and must stay a separate file).+3. The first start took about 20 minutes on the measured PC with the model files already on disk: the engine package (158 MB), the+ check of the 84 GB of model files, the picture reader (0.9 GB), the draft layer (about 5 GB) and the one-time pack. The next start+ took under two minutes.++What `strata launch` wrote for IQ3_S, with no switches of mine (these are the settings to copy if you drive the engine by hand):++```+--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <draft layer>+--max-context 131072 --kv int8 --vision --vram-reserve-mib 700 (backend hip, GPU picture reader)+```++and for DeepSeek 2.00 bpw: `--max-context 8192 --dspark auto --dspark-conf 0.65 --prefill-chunk 16384`, with `DS4_ASYNC_T=4` in the+environment and the package's `rocm/bin` folder on the library path.++## 4. What is on by default++- On `gfx1151` the engine turns on 19 exact speed switches by itself (the start line lists them; `STRATA_GFX1151_DEFAULTS=0` turns them all+ off, and a switch you set is kept). They are the ones that give the same answers as without. `STRATA_HC_Q8=1` is also on here (decode+ about 9% faster, near-tie tokens can differ); `STRATA_HC_Q8=0` keeps the old read.+- The hipBLASLt table for `gfx1151` ships in the package and is chosen without a setting (`prefill gemm: hipBLASLt tuning enabled (32 rows,+ gfx1151, version 100500)` in the log). A hand-built engine has to be pointed at it: [AMD_HIP.md](AMD_HIP.md).+- `--mmap-experts` is not needed. The first Windows guide found that without it the pinned expert arena used up the graphics memory+ budget and decode fell to 28 tok/s on engine 0.1.41; the engine on this page sizes the cache from the shared pool and reaches every slot.+- DeepSeek runs in "Faster" mode here: 5 of 6 experts per word. The app and `strata launch` say so; the quality change is small.++**`STRATA_SH_STREAM` (the shared expert on its own stream) is on by default on `gfx1151`, and on this Windows PC it is slower.** The first+Windows guide found the same on engine 0.1.41 (+13% decode with it off). Measured again on 0.2.7 with Qwen IQ3_S, three starts each, run+in turn (on, off, on, off, on, off), the same quick suite after every start:++| Start | `STRATA_SH_STREAM` | Chat decode, median of 9 answers (tok/s) | 4K prompt (tok/s) |+| --- | --- | ---: | ---: |+| 1 | 1 (default) | 43.9 | 742.9 |+| 2 | 0 | 48.3 | 742.9 |+| 3 | 1 | 43.6 | 746.8 |+| 4 | 0 | 46.0 | 740.8 |+| 5 | 1 | 43.2 | 745.5 |+| 6 | 0 | 46.6 | 739.4 |++With the switch off, decode was **6.9% faster** (46.6 against 43.6, the median of the three starts) and the code-like question 14% faster+(67 against 59 tok/s); all three pairs went the same way. The prompt speed did not change. Setting it takes `strata launch ... --set+STRATA_SH_STREAM=0`; it changes only the order the work runs in, so the answers stay the same. Whether Windows `gfx1151` gets this by+default is a separate decision. This is one model, one PC and short answers; DeepSeek was not tested with it.+++## 5. Traps++1. **Keep RAM free.** Qwen IQ3_S held 49 GB of 63.6 GB. A browser with many tabs, or a second model, is enough to push the PC into paging.+2. **Do not run two engines.** One model at a time on this PC; `strata launch --status` shows what is serving, `strata launch --stop` ends it.+3. **The graphics driver.** The 2025-03 driver failed at 32K context for DeepSeek. The measured driver is Adrenalin 26.10.1 (32.0.32015.7029).+4. **If Windows resets the driver** (screen flicker, the engine dies mid-answer), see the driver-reset entry in+ [TROUBLESHOOTING.md](TROUBLESHOOTING.md#amd-cards).+5. **Other tuning from the first guide** (a longer `--prefill`, `STRATA_PF_*` switches, `--lookup-chain`): those results are for engines+ 0.1.41 to 0.1.47 and a hand-made configuration. They were not repeated here, so they are not part of this page.++## 6. Reproduce it++```+strata launch --family qwen --model iq3_s # then, from a source checkout:+python tools/strata_bench.py --url http://127.0.0.1:8080 --runs 3 --long+```++`--long` adds the 32K prompt. For DeepSeek add `--context 64k` to `strata launch` first; the default 8K context skips the 16K prompt and+the agent session.