Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
hip: 8-bit hyper-connection weights (STRATA_HC_Q8) on by default on gfx1151, STRATA_HC_Q8=0 keeps the bf16 read; one start-up line; docs and 0.2.2 release note with the KL table and the CPU-vs-GPU rounding baseline
Niko12216d4874a491cc
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex eab17bdf7..e2e759071 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -1097,3 +1097,27 @@ passes. Additional unpublished upstream fixture suites are not covered.The published performance table explicitly identifies the measured developmentsnapshot; it must not be read as a benchmark of every subsequent rebase. Vision, long-context stress, broad answer-qualityequivalence, other AMD cards, and mixed-vendor inference are not validated here.++## Strix Halo: 8-bit hyper-connection weights (default on gfx1151)++On a Strix Halo (gfx1151) the hyper-connection down / up projections are read as Q8_0 by default (quantized at load from the pack, or the GGUF's own Q8_0),+which halves the 13.3 MB each read moves. Decode is about 9% faster (5 interleaved pairs per prompt on IQ3_S, MTP 4 drafts / min-p 0.5; json / code / tool+call / chat / agent between +7.9% and +10.7%). `STRATA_HC_Q8=0` restores the bf16 read. Other cards are unchanged (gfx12 has had it on since 0.1.4x).+Not bit-identical to the bf16 read: a greedy answer can pick another token at a near tie and then go its own way.++How far it moves the model, measured with the quant-quality yardstick (teacher forced, 5 texts of 1,000 tokens, reference = bf16 read; RTX A4000, IQ3_XXS,+2026-10-22 — the Q8_0 weights are the same bytes on every card):++| text | top-1 same | KL |+| --- | ---: | ---: |+| code | 97.9% | 0.0077 |+| documents | 93.5% | 0.0229 |+| English | 94.7% | 0.0107 |+| other languages | 98.0% | 0.0060 |+| agent transcript | 93.0% | 0.0779 |+| all 4,995 positions | 95.4% | 0.0251 |++For scale, the same yardstick with the bf16 read on both sides and only the experts computed on the CPU instead of the GPU (a rounding-only difference the+engine takes whenever an expert is not resident) gives code 0.0062, documents 0.0195, English 0.0088, other languages 0.0048, agent 0.0745, all 0.0228+(top-1 95.4%). The 8-bit read is at that level. The agent text's number rests on a few near-tie positions (the top 20 of 999 carry half of it); finer blocks+(16 or 8 values) did not lower it, and 8-bit hyper-connection weights in the first six layers alone give the full figure, in the last 24 layers a third of it.diff --git a/docs/STRIX_HALO.md b/docs/STRIX_HALO.mdindex 5d8f69348..2b89cdf53 100644--- a/docs/STRIX_HALO.md+++ b/docs/STRIX_HALO.md@@ -100,7 +100,7 @@ they change speed, not answers. Nothing here applies to CUDA or to another AMD a## 5. What is not on by default (it changes bits)-**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_HC_Q8`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.+**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well. `STRATA_HC_Q8` is on by default on gfx1151 (below); `STRATA_HC_Q8=0` turns it off.These are faster on gfx1151 and **round differently** from the default path (a different summation order or a half-precisionintermediate), so they are opt-in and the maintainers gate them with KL against the default (`STRATA_PF_SWITCH_MIN_T=4096` keeps@@ -114,15 +114,14 @@ export STRATA_HC_UPMIX=1 # the hyper-connection up projection with its epexport STRATA_PA_FAST=1 # the prompt attention with single FP16 q and pexport STRATA_HIP_WMMA=1 # the prompt attention on matrix coresexport STRATA_SELECT_WMMA=1 # the block scorer on matrix cores-export STRATA_HC_Q8=1 # the hyper-connection read from the GGUF's own Q8_0 projectionsexport STRATA_PF_SWITCH_MIN_T=4096export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too (below)```-`STRATA_HC_Q8=1` measured on the Strix Halo (IQ3_S, gfx1151, 2026-10-21, MTP 4 drafts / min-p 0.5): **+8.4%**+`STRATA_HC_Q8` (on by default on gfx1151 since 0.2.2; `STRATA_HC_Q8=0` restores the bf16 read) measured on the Strix Halo (IQ3_S, gfx1151, 2026-10-21, MTP 4 drafts / min-p 0.5): **+8.4%**on `json_extract` at fixed windows (53.55 -> 58.03 tok/s), at the cost of **94 of 250 output tokens differing** fromthe BF16 read (first at 60). It halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read; cold read-93.8 -> 20.3 us, 142 -> 352 GB/s). Experimental, opt-in only, never a default.+93.8 -> 20.3 us, 142 -> 352 GB/s). The answers are not bit-identical to the bf16 read; how far they move is in docs/AMD_HIP.md ("Strix Halo: 8-bit hyper-connection weights").`STRATA_PF_FUSED=1` reaches only chunks of `STRATA_PREFILL_STREAM_MIN` tokens or more (1,024 by default, the floor measured ona discrete card, where a smaller chunk does not pay for streaming every expert); a smaller read runs its experts through MMQ.diff --git a/docs/release-notes-0.2.2.md b/docs/release-notes-0.2.2.mdindex 9486736b4..90ca53d21 100644--- a/docs/release-notes-0.2.2.md+++ b/docs/release-notes-0.2.2.md@@ -22,6 +22,7 @@- **The Linux CUDA 12 engine package has Pascal and Volta code again (#3503).** The 0.2.1 package was built for sm_75, 86 and 89 only because the repo held no recipe for it. `packaging/build_cuda_linux.sh` builds it for sm_60, 61, 70, 75, 80, 86 and 89, the packager refuses a CUDA 12 package without sm_60, 61 and 70, and the source Dockerfile takes `CUDA_IMAGE` and `CUDA_ARCHITECTURES`. A package built with it ran an IQ2_XS model on a Tesla P100. The 0.2.2 packages are built with it.- **HIP queue priming stands aside on a restricted queue count (#3480).** When `GPU_MAX_HW_QUEUES` is 1 or 2, the engine no longer opens its two priming queues and says so in one line (`STRATA_HIP_QUEUE_PRIME=nn` still forces them). It compiles for gfx1201; it was not measured on a gfx12 card.- **Docker.** An old (0.1.x) volume with `/data/models`, `/data/packs` and `/data/mtp` is moved into `/data/Strata-data` by rename instead of being downloaded and packed again (it used to start a 75 GB download). A card without `--gpus` stops at once with the `--gpus all` hint instead of after minutes of setup. A missing or empty `MODEL_PATH` gets a plain message. Settings that name another image's component folder (a CUDA 12 image on a CUDA 13 volume) are made again, and an engine folder renamed by an update no longer ends in `server_exited` when pictures are on. The terminal stage of the image build no longer needs git (it failed with `FileNotFoundError`); the commit comes in as a build argument, and all five image flavors built. Tested on one RTX 3060 PC with CUDA 12 and CUDA 13 images; no AMD or Intel image and no Docker Desktop.+- **Strix Halo decodes about 9% faster.** The hyper-connection weights are read as 8-bit (Q8_0) on gfx1151 by default: 5 interleaved pairs on IQ3_S gave +7.9% to +10.7% on JSON, code, tool calls, chat and an agent transcript. Answers are not bit-identical: they can differ in near-tie tokens, about as much as the engine's own CPU-versus-GPU expert rounding does (the numbers are in docs/AMD_HIP.md). `STRATA_HC_Q8=0` restores the old read.- **Re-runs do not tune again.** A tuning that found the standard settings already fastest counted as "not tuned", so `strata launch`, the TUI and the Docker entrypoint measured again at every start (20 minutes); it counts as tuned now. In the Terminal a first launch of an installed model took 521 s and the second 13 s on the test PC.- **Problems with the computer's own setup are repaired or explained, not refused.** If the Windows runtime folder lists another account, Strata locks it down itself instead of stopping. A Documents folder that cannot be used (Controlled folder access, an offline OneDrive) falls back to the Strata home folder and Settings > Storage says so. A port that is taken or reserved gets one sentence with the next step. Local requests ignore a system proxy. File-in-use errors and a full disk say what happened. `strata launch` asks plain questions in a terminal that reports a tty late. `strata update` copes with a slow `--version`, a busy graphics card is not held against an engine, pre-release Control builds hand over correctly, and a wrong PC clock is named as the reason a feed or certificate looks bad. install.sh and install.ps1 give clearer messages (ARM, TLS 1.2, proxy credentials, no `tar.exe`). The Terminal parts are already in 0.2.1.3; the Documents fallback and the port handling are app parts.- **A stale update notice goes away.** After a version change the app and the Terminal no longer keep showing an offer for a version you already have.diff --git a/src/core/native_dense.cpp b/src/core/native_dense.cppindex aba2882b8..25efb6f62 100644--- a/src/core/native_dense.cpp+++ b/src/core/native_dense.cpp@@ -46,12 +46,26 @@ bool in_range(const std::string& name) {bool hc_q8_env() {// STRATA_HC_Q8: 1 = on (the GGUF's own Q8_0 projections, or - a bf16 pack - a Q8_0 copy quantized at load); 0 = off;// unset / auto = on for gfx12 (RDNA4: the R9700 decode +2.5 to +4%, KL 0.0003-0.0010 over its noise floor) unless- // STRATA_GFX12_DEFAULTS=0, off elsewhere+ // STRATA_GFX12_DEFAULTS=0; also on for gfx1151 (Strix Halo, STRATA_HC_Q8=0 turns it off), off elsewherestatic const bool on = [] {const char* v = std::getenv("STRATA_HC_Q8");if (v != nullptr && v[0] == '1') return true;if (v != nullptr && v[0] != '\0' && std::strcmp(v, "auto") != 0) return false;#if defined(STRATA_USE_HIP)+ { // Strix Halo (gfx1151): on by default (decode +9.5%, json/code/tool/chat/agent +7.9..+10.7%); the answers differ in+ // near-tie tokens by about the size of the engine's own CPU-vs-GPU expert rounding (docs/AMD_HIP.md);+ // STRATA_HC_Q8=0 keeps the bf16 read+ int dev1151 = 0;+ cudaDeviceProp p1151;+ const bool g1151 = cudaGetDevice(&dev1151) == cudaSuccess && cudaGetDeviceProperties(&p1151, dev1151) == cudaSuccess &&+ strata::kernels::gfx_arch_is_gfx1151(p1151.gcnArchName);+ (void) cudaGetLastError();+ if (g1151) {+ std::fprintf(stderr, "strata: Strix Halo (gfx1151): 8-bit hyper-connection weights are on (decode about 9%% faster, "+ "answers can differ in near-tie tokens); STRATA_HC_Q8=0 keeps the bf16 read\n");+ return true;+ }+ }const char* d = std::getenv("STRATA_GFX12_DEFAULTS");if (d != nullptr && d[0] == '0') return false;int dev = 0;