Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: --kv-grow on HIP with numbers, STRATA_PREFILL_HELP on two R9700
Niko1221d9070189c041
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex 822439631..f5df3cef1 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -434,7 +434,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 391,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.- - `STRATA_HIP_WMMA=auto` (f-151-amd): the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of+ - `STRATA_HIP_WMMA=auto`: the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of`STRATA_HIP_WMMA_MIN_Q` (64) or more queries it runs the matrix-core kernel, everything else the FP32 one (`=1` forces iton every call, `=0` or unset keeps the FP32 kernel; the default has not moved). A launch error switches the matrix-corekernel off for the rest of the process and the FP32 kernel runs (before, the engine exited). Kernel timing on an R9700diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 967fbf899..b035389d7 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -340,6 +340,14 @@ move on a Threadripper PRO 3975WX + R9700 (14.5 ms, +0.4%, 5 pairs); no case reareach, and the expert cache holds the rest, giving slots back as the context grows. It needs one GPU, an expertprofile, the whole K/V in VRAM (no KV streaming) and every expert in RAM (not the resident low-RAM mode); otherwise,and with `--batch`, `--vram-elastic` or `--peer-device`, the engine says so and stays off.+ On HIP (ROCm 6+) it runs through hipMemAddressReserve / hipMemCreate / hipMemMap (2 MiB chunks; the card must report+ virtual memory management, `STRATA_HIP_VMM=0` turns it off); until then every AMD card printed "--kv-grow is off".+ Radeon AI PRO R9700 with `--vram-reserve-mib 17084` (a 16 GB card), Q2_0 pack, `--max-context 196608`, 5 interleaved pairs:+ expert-cache slots 6,213 -> 7,968 and decode 96.3 -> 98.5 tok/s (x1.023, 5/5; the noise arm read x0.999); at+ `--max-context 262144` the default engine finds no VRAM left for the cache on that budget and does not start, `--kv-grow`+ starts with 7,877 slots (94.4 tok/s). Prompts of 20K and 60K tokens through the growing K/V read as fast as without it+ (x1.002, x1.001) and gave identical answers. At contexts up to 128K the K/V is small and nothing changes; it pays on cards of+ about 24 GB or less with contexts of 128K and more.- `--host-core last` (or `STRATA_HOST_CORE=last`, Windows): the host thread runs on the last physical core and theworkers take the first. Windows sends a GPU's interrupts to the first core, where a host spinning on the GPU's flagswaits for them (`--host-core first` is the default; the startup log names the cores). It leaves a hybrid CPU as itdiff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.mdindex e93bba2ed..3f7b09e4d 100644--- a/docs/MULTI_GPU.md+++ b/docs/MULTI_GPU.md@@ -117,6 +117,10 @@ helped (~360 MB at 3.3K tokens). The rows it computes round like a different MMQbit-identical to the default's (it is repeatable: same prompt, same output), which is why it is **opt-in**:`STRATA_PREFILL_HELP=1` turns it on. Native packs on the MMQ prompt path only (not with the fused prompt kernels,`STRATA_PF_FUSED=1`, nor with `--peer-device`). With it on, `STRATA_PREFILL_HELP_FRAC=f` fixes the share.+On two Radeon AI PRO R9700 (each limited to 16 GB with `--vram-reserve-mib 17084`, Q2_0 pack, layer split auto, 5 interleaved pairs,+noise arm within 0.2%), `STRATA_PREFILL_HELP=1` gave prompt tok/s without -> with: 700 tokens 638 -> 672 (x1.053, 5/5), 1.5K 839 -> 809+(x0.964), 3K 1,105 -> 929 (x0.840), 6K 1,284 -> 1,211 (x0.943): the PCIe 4.0 x16 links of this box (28.4 GB/s each) make the share lose+from 1.5K tokens up, so it stays off (`STRATA_PREFILL_HELP_FRAC` was not tuned for it).The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D1[,D2..]` (the later stages'devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing