Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge perf-PR group rdna2 (f-145-pp-rdna2)
Niko122121c579727c8c
diff --cc docs/AMD_HIP.mdindex ce702054d,29c77d04b..661152f6b--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@@ -528,17 -456,71 +536,82 @@@ run it; the report below is from a commmatrix cores; `hip_prefill_hipblaslt_gemm`, no hipBLASLt table) and 3 that fail for reasons outside the engine(`ple_parity` needs a Q2_0 PLE file that is not on that machine, `expert_multi_test` refuses the CPU without AVX-512,`platform_memory_test` cannot `mlock` at the shell's default `ulimit -l`).+- **The verifier's hyper-connection read as one persistent kernel:** `STRATA_HC_PERSIST=1` (opt-in, HIP, 32-lane+ waves) runs the read of 1-8 tokens as one kernel of 80 blocks with two grid barriers instead of three launches; a+ start-up run per token count with a barrier timeout decides where it is resident, and one `strata hc:` line per card+ says so. On 2x RX 6900 XT (ROCm 10.0, layer split): the read 1.35-2.10x, ms per verify window -4.2 % / -2.8 % at+ 4K / 32K (6 rounds), not bit-identical (as close to a double-accumulation reference as the default read):+ [bench/results/2026-10-12-hc-persist-gfx1030](../bench/results/2026-10-12-hc-persist-gfx1030/README.md). Other+ RDNA cards are untested; the `strata hc:` line and a `STRATA_DECODE_TIMING` run from one are welcome. Not offered on gfx11 and+ gfx12 (RDNA3, RDNA4): the barrier timeout needs `s_memrealtime`, which gfx1100 and gfx1200 / gfx1201 do not have, and an R9700 (#2226 thread) measured the+ read +0.7 % / +1.0 % per window slower than the default one; the start-up line says so. A barrier that times out at run time+ (never seen in a working run) leaves the partial outputs of that read in place and every later persistent read of the+ process aborts at once, without a log line; the opt-in is for measurement on gfx103x until that is closed.+ - **Prompt GEMMs as SGEMMs (engine 0.1.39 + `Gemm::rdna2_sgemm`):** the rocBLAS in setup's ROCm (10.2.0a20260930)+ has tuned gfx1030 kernels for FP16 -> FP16, int8 and FP32 -> FP32 only. The prompt path's dense GEMMs take FP16 or+ BF16 inputs and write FP32 (rocBLAS's HS / BS types), which run its fallback kernels at about 5 TFLOPS. With+ `STRATA_RDNA2_SGEMM=1` (opt-in; `=all` forces it on any AMD card) the engine widens both inputs to FP32 (exact) and runs SGEMM,+ which accumulates in FP32 like the native call (`src/prefill/gemm.cu`; N < 64 keeps the native call). The author ran it+ as the gfx103x default; here it is a switch because it is not bit-identical to the native call (below). Measured on an RX 6800 16 GB over OCuLink (PCIe 4.0 x4, 7.1 GB/s), Ryzen 7 8845HS, 28.8 GB RAM, Windows 11,+ Coder IQ1_M with setup's arguments (64K context, MTP), fresh 12K-token prompts, 4 per arm, A/B in one binary:++ | | native GEMMs | SGEMM route |+ |---|---|---|+ | prompt | 312 tok/s | 452 tok/s |+ | prompt GPU timeline (`STRATA_PREFILL_TIMING`) | 37.2 s | 25.5 s |+ | of which gdn / qsa proj / hc read | 9.4 / 3.6 / 3.85 s | 2.8 / 1.4 / 2.84 s |+ | decode | 29-34 tok/s | 29-35 tok/s |++ The exact calls on the card (opA = T, opB = N, FP32 out): N 10240 x T 8192 x K 2560 86.7 -> 27.7 ms including+ the widening; N 320 x T 8192 x K 10240 (BF16) 10.9 -> 7.5 ms. Against a float64 reference on random inputs in+ [-1, 1], the SGEMM route is as close as or closer than the native call on all 10 FP16 / BF16 shapes checked; the+ two differ by at most 4.3e-6 of the largest output. A 9-task code-repair benchmark (the model patches seeded bugs+ in a browser game until 38 browser checks pass) solved 9/9 on the first try with both, model time 458 -> 374 s.+ With the switch on and `STRATA_HIP_PROMPT_F16` unset the SGEMM route runs; with `STRATA_HIP_PROMPT_F16=1`+ the FP16 route above runs first and the SGEMM route takes the products it leaves (`beta != 0`). Same RX 6800, one+ session, fresh 12K-token prompts: native 310, SGEMM 439, FP16 (#835) 494 tok/s; `STRATA_F16_RANGE=1` on the Coder+ IQ1_M peaked at 105.3 (activations) and 424.8 (FP16 outputs) with nothing beyond 65504, and the repair benchmark+ solved 9/9 on the first try with both (model time 374 s SGEMM, 307 s FP16). The SGEMM route is the slower of the+ two; no value passes through FP16, so it has no 65504 limit to stay under. It is not bit-identical to the native+ call (at most 4.3e-6 of the largest output apart), which can flip a near-tied greedy token like any such change.+ - **rocBLAS solution table** (for the `STRATA_HIP_PROMPT_F16=1` route above; with it off, nothing runs through the+ table): rocBLAS picks the kernel for each of those FP16 GEMMs from its own table by shape, and+ on gfx1030 that pick is far from the best kernel the library holds for some shapes: the GDN input projection+ (N = 10240, K = 2560) below about 1,150 tokens runs at 5 TFLOPS where the best of rocBLAS's own 237 solutions for the+ same shape runs at 32-38 (7x; the kernel choice flips at T = 1152). `tools/hip/tune_rocblas` enumerates the library's+ solutions per shape and token bucket, checks each against the default kernel's output and writes the winners as+ `tools/hip/<arch>-rocblas-<version>.txt`; the engine reads the table through `STRATA_ROCBLAS_TUNING` (a file, or+ the directory, from which it takes the file of its own rocBLAS build) and runs those solutions in place of rocBLAS's+ choice, the default kernel for every shape without a row. Solution indices are valid for one architecture and one+ rocBLAS build (the full version string, `5.6.0.8d1ae90e` for ROCm 10.0.0's), and the engine refuses any other+ table. Shipped: `gfx1030-rocblas-5.6.0.8d1ae90e.txt` (ROCm 10.0.0, the `/opt/rocm` setup installs on Ubuntu), measured+ on an RX 6900 XT. Other gfx1030 cards (RX 6800 / 6800 XT / 6950 XT, Radeon PRO W6800 / V620) run it as well - the+ indices are valid on every gfx1030 - but have fewer CUs or other clocks, so their own table may pick differently;+ `tune_rocblas` makes one in about 5 minutes ([Tuning table](#tuning-table)), and a report of what it changed is welcome. Setup+ points the engine at the directory when a table for the card's architecture exists, and the engine's log says+ `rocBLAS tuning enabled (... rows ...)` or why not. Measured on the 2x RX 6900 XT machine above (one card, #835 ++ this, `STRATA_PREFILL_TIMING=1`, temperature 0, the same answers with and without): a 799-token prompt's GPU time+ 3,329 -> 2,985 ms (the GDN projections 443 -> 177 ms, `hc read` 270 -> 77), 495 tokens 2,477 -> 2,139 ms (365 ->+ 146), 8,874 tokens 11,978 -> 11,459 ms (the 682-token tail chunk's projections, `hc read` 1,200 -> 898); a+ 1,108-token prompt did not change (3,391 -> 3,395 ms): its GDN time fell 598 -> 201 ms and the whole saving+ reappeared as `wait copy`, the experts' PCIe transfer, which bounds that prompt. Per-run lines and the tuner's+ summary: [bench/results/2026-10-05-rdna2-rocblas-table](../bench/results/2026-10-05-rdna2-rocblas-table/README.md). Decode does not use these GEMMs and is unchanged. See [Tuning+ table](#tuning-table) for making a table for another rocBLAS build.+ - **QSA block scores on the FP32 tiled scorer (`STRATA_SELECT_SIMT=1`, opt-in, as on CUDA below sm_80):** without+ matrix cores the prompt path's QSA selection scores every (query, KV block) pair on the warp kernel, the one prompt+ term that grows with the context. The switch runs the FP32 tiled scorer instead (FP32 in another summation order:+ not bitwise the warp kernel, so off by default), tiled 32/128/16/8 on gfx103x. `qsa_select_bench`, 256 queries:+ 2.69 -> 0.587 ms at 128K (the original 32/64/32/4 tiling: 0.695), 0.67 -> 0.169 ms at 32K; FP64 gate passed,+ selections identical 256/256. Every tiling gives the same bytes (each score is one fmaf chain over k in order), and+ the fastest tiling was the same with 60 and 72 of the card's 80 CUs (`HSA_CU_MASK`). On the 2x RX 6900 XT split+ (0.1.40.3, IQ3_S, `--kv int8 --kv-resident 32768`, two cold starts per arm, ABBA) a 128K-token prompt reads 1,735 ->+ 1,861 and 1,727 -> 1,858 tok/s (+7.4%), 32K +0.8%; decode is not affected (the text changes with the scores' order,+ so its speed is not compared). Measurements:+ [bench/results/2026-10-07-rdna2-qsa-select-simt](../bench/results/2026-10-07-rdna2-qsa-select-simt/README.md).+ - **File-tier cache rule on gfx906 (`STRATA_FILE_CACHE_OLD_RULE`):** on an MI50 ROCm queues are suspended while the kernel reclaims around the process's HMM-tracked arena (#1752, KevinX8: stalls, prompt 46 -> 360 tok/s once the file tier stops churning the page cache). The gfx906 backend therefore keeps the old rule (the file cache is used only when the room left is at least the bytes read from the files); other HIP cards keep the current rule. `STRATA_FILE_CACHE_OLD_RULE=1` forces the old rule on any Linux build, `=0` the current one.+ - **Windows** (the report above): setup's ready-made Windows engine 0.1.39 runs the RX 6800 beside a 780M iGPU+ (HIP numbers the RX 6800 1; setup points the engine at it), and `tools/hip/build_windows.bat` with+ `STRATA_HIP_ARCHS=gfx1030` builds a working engine from the `gfx103X-all` 10.2.0a20260930 wheels.- **gfx1031** (RX 6700 XT, #524): setup knows it (the `gfx103X-all` wheels, unvalidated); its reporter runs it dailyon one card. More reports: an RX 6700 XT 12 GB run as gfx1030 on ROCm 7.2.4 (#1027: IQ2_XS, decode 30-32 tok/s, prompt about 300 tok/s, 6 of 6needles), and an RX 6800M 12 GB on Windows with a self-built engine (#915, #1078: Q2_0, decode 9-27 tok/s, prompt 50-114 tok/s; thediff --cc setup.pyindex e7434577f,6e28ec906..3dd2ca51a--- a/setup.py+++ b/setup.py@@@ -4522,8 -4414,8 +4522,8 @@@ def write_config(path: Path, cfg: dict)# "sampling" or "mcp_servers" block, "allowed_hosts", "cors_origins", "open_browser" - and is kept when setup runs againSETUP_KEYS = frozenset({"exe", "args", "cwd", "tokenizer", "model_name", "log", "lib_dirs", "port", "backend", "env","gpu", "gpus_asked", "layer_split", "host", "api_key", "draft_vocab", "vision"})- SETUP_ENV = frozenset({"STRATA_HIPBLASLT_TUNING", "STRATA_RESIDENT_PIN", "STRATA_NO_ARENA_THP", "STRATA_MMQ_KQ", "STRATA_HC_Q8",+ SETUP_ENV = frozenset({"STRATA_HIPBLASLT_TUNING", "STRATA_ROCBLAS_TUNING", "STRATA_RESIDENT_PIN", "STRATA_NO_ARENA_THP", "STRATA_MMQ_KQ", "STRATA_HC_Q8",- "STRATA_ROUTE_TAIL_SKIP"}) # the "env" entries setup writes+ "STRATA_ROUTE_TAIL_SKIP"}) | frozenset(GFX1103_SWITCHES) | frozenset(GFX1150_SWITCHES) # the "env" entries setup writes (the gfx1103 switches are setup's too: a re-run refreshes them, they do not linger)# 0.1.44 added these three, written for Unsloth's files only (unsloth_default_env). In any other model's config they are the# user's own (STRATA_ROUTE_TAIL_SKIP=0 to turn the tail skip off, STRATA_MMQ_KQ=1 ...): a setup run again keeps them there# (0.1.44 dropped them with the rest of SETUP_ENV).@@@ -7024,15 -6747,14 +7028,23 @@@ def main() -> intif GFX1103_OPT_IN and gfx_arch_is(gpu["arch"], "gfx1103") and not WIN:# the 780M's KFD queues are evicted (a GPU reset) while transparent huge pages move the pinned expert arenacfg.setdefault("env", {})["STRATA_NO_ARENA_THP"] = "1"+ # the exact gfx1151 speed switches measured on this chip (GFX1103_SWITCHES): the engine has no arch defaults+ # for gfx1103, so setup writes them into the config. They are setup's own (SETUP_ENV), so a re-run refreshes+ # them and turning the opt-in off takes them out again.+ cfg.setdefault("env", {}).update(GFX1103_SWITCHES)+ if GFX1150_OPT_IN and gfx_arch_is(gpu["arch"], "gfx1150"):+ # the exact gfx1151 speed switches measured on this chip (GFX1150_SWITCHES): the engine has no arch defaults+ # for gfx1150, so setup writes them into the config. They are setup's own (SETUP_ENV), so a re-run refreshes+ # them and turning the opt-in off takes them out again.+ cfg.setdefault("env", {}).update(GFX1150_SWITCHES)+ # RDNA2 (gfx103x): the prompt's FP16 GEMMs through rocBLAS's best kernel per shape (tools/hip/tune_rocblas;+ # the GDN input projection 6x faster below 1,152 tokens on a 6900 XT). The ids are valid for one rocBLAS+ # build, which only the engine can read, so setup points it at the directory and the engine picks the file+ # <arch>-rocblas-<version>.txt for its own library, or runs the default kernels and says so in its log.+ rocblas_tables = sorted(p.name for p in (ROOT / "tools" / "hip").glob(f"{gpu['arch']}-rocblas-*.txt"))+ if rocblas_tables:+ cfg.setdefault("env", {})["STRATA_ROCBLAS_TUNING"] = str(ROOT / "tools" / "hip")+ ok(f"rocBLAS tuning tables for {gpu['arch']}: {', '.join(rocblas_tables)} (the engine takes the one of its rocBLAS build)")if resident: # ROCm: large page-locked host allocations can fail or be slow for the CPU; keep the copy pageablecfg.setdefault("env", {})["STRATA_RESIDENT_PIN"] = "0"if gpu["count"] > 1 or a.gpu is not None:diff --cc src/core/peer_experts.cppindex 9ca2ee994,230f81699..ee0c2d0c3--- a/src/core/peer_experts.cpp+++ b/src/core/peer_experts.cpp@@@ -145,18 -106,9 +146,18 @@@ bool PeerExperts::open(int device, conscudaError_t e0 = cudaDeviceEnablePeerAccess(device, 0);if (e0 == cudaErrorPeerAccessAlreadyEnabled) { cudaGetLastError(); e0 = cudaSuccess; }On on1(device);- cudaError_t e1 = cudaDeviceEnablePeerAccess(0, 0);+ cudaError_t e1 = cudaDeviceEnablePeerAccess(primary_device(), 0);if (e1 == cudaErrorPeerAccessAlreadyEnabled) { cudaGetLastError(); e1 = cudaSuccess; }p2p_ = e0 == cudaSuccess && e1 == cudaSuccess;+ // recommend, never force: the direct path is used only if a test copy each way arrives exact (#2666)+ if (p2p_ && !p2p_copies_data(device)) {+ std::fprintf(stderr, "strata: peer GPU %d: peer access is enabled but a test copy between the two GPUs arrived wrong "+ "(a host bridge between them, no real P2P): the prompt path takes the host route instead "+ "(STRATA_PEER_NO_P2P=1 does that by hand)\n", device);+ { On q0(0); cudaDeviceDisablePeerAccess(device); cudaGetLastError(); }+ { On q1(device); cudaDeviceDisablePeerAccess(0); cudaGetLastError(); }+ p2p_ = false;+ }}}On on(device);diff --cc src/program/generate.cppindex 158f67854,a5fb822a3..aae516316--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@@ -3228,9 -2859,25 +3256,28 @@@ int main(int argc, char** argv)}}const bool native_pack = strata::kernels::cpu::expert_layout().native;+ // STRATA_MMQ_DIRECT: the room each sized expert slot keeps after its blob (0 unless it is on: the slot costs and+ // the arena are then today's, byte for byte)+ const int64_t slot_guard = native_pack ? (int64_t) strata::prefill::Prefill::slot_guard_bytes() : 0;+ // STRATA_PRIMARY_DEVICE: switch the engine's own thread to the chosen card BEFORE any GPU+ // context or allocation exists below (the helper preflights, the PCIe probe, the weights and the+ // arena all follow it). Every other device switch in the engine saves and restores the current+ // device, so this single call is where the primary is picked.+ if (primary_device != 0) {+ int n_vis = 0;+ if (cudaGetDeviceCount(&n_vis) != cudaSuccess || primary_device >= n_vis) {+ cudaGetLastError();+ std::fprintf(stderr, "strata generate: STRATA_PRIMARY_DEVICE=%d: no such visible device (%d visible)\n",+ primary_device, n_vis);+ return 2;+ }+ if (cudaSetDevice(primary_device) != cudaSuccess) {+ std::fprintf(stderr, "strata generate: STRATA_PRIMARY_DEVICE=%d: cudaSetDevice failed: %s\n",+ primary_device, cudaGetErrorString(cudaGetLastError()));+ return 1;+ }+ std::fprintf(stderr, "strata generate: primary device from STRATA_PRIMARY_DEVICE: CUDA%d\n", primary_device);+ }// STRATA_EARLY_REMOTE_CONTEXTS=1: create EVERY secondary context here, like CUDA1's. Under WSL2 the driver's// pinned/mapped host budget (dxg gpadl, ~1 GiB) is spent by CUDA0's weights and MTP before the later loop runs,// and a new context then fails with cudaErrorMemoryAllocation (CUDA2: "cudaSetDevice(2) failed: out of memory").diff --cc tools/calibrate.pyindex 7cd6d4eef,52dda9dbc..397cd8f1f--- a/tools/calibrate.py+++ b/tools/calibrate.py@@@ -13,14 -13,13 +13,20 @@@ measured on (a Ryzen 5 7600 + RTX 5070its use counts fade. A PC whose CPU reads the missed experts slowly (DDR3, ~24 GB/s) gains fromswapping more: 160 swaps every window measured +7.9% over the default tier on a Xeon E5-2673 v3 (DDR3) with anRTX 4060 Ti on PCIe 3.0 x8 (Q2_0; with the swaps taking effect a window later, #764).+ --prefill-pipe a layer split only: the prompt chunks that keep the stages reading at once. The best chunk count+ depends on b/a, the fixed cost of a chunk over that of a token, which differs by an order of+ magnitude between rigs (about 700 tokens on four GP100 dies, 5000 on two RTX 3090 Ti) and costs+ when it is wrong (a fixed 1024 read 22-36% slower on two RTX 4090) - so it is measured here, once,+ as the prompt read speed of fresh prompts, rather than guessed or learned while the PC serves.+ STRATA_SH_STREAM (AMD only)+ the shared expert on its own stream. Off by default on HIP (#816: an RX 6800 on Windows and 4x R9700+ decode 12-45% faster without it), but the same card family on another PC goes the other way (#1272:+ 2x RX 6900 XT on Linux, 6-11% faster with it), so it is measured here, per PC, and written into the+ config's "env" only when it wins. The output is identical either way (same token ids).-The first two are measured through one engine (per-request `strata_tune` keys); the worker count and the adaptive-tier need a restart per value. Decode speed only: the prompt path streams every expert whatever these settings say.+++The first two and the prompt chunks are measured through one engine (per-request `strata_tune` keys); the worker- count and the adaptive tier need a restart per value. Those are decode speed: the prompt path streams every expert- whatever they say.++count, the adaptive tier and the shared-expert stream need a restart per value. Those are decode speed: the prompt++path streams every expert whatever they say.A setting is kept only when it beats the default by more than MIN_GAIN in an interleaved re-measurement - theadaptive expert tier and the OS make single measurements noisy by a few percent.@@@ -212,7 -159,7 +222,8 @@@ def run(cfg: dict, say=print, start_engtok = ST.Tokenizer(toks, (tpath / "merges.txt").read_text(encoding="utf-8").split("\n"),json.loads((tpath / "token_type.json").read_text()))ids_list = [chat_ids(tok, p) for p in PROMPTS]- return measure(engine_args(cfg), ids_list, start_engine, say, host_worker_extras(), doc_prompt(tok))- return measure(engine_args(cfg), ids_list, start_engine, say, host_worker_extras(), hip=cfg.get("backend") == "hip")++ return measure(engine_args(cfg), ids_list, start_engine, say, host_worker_extras(), doc_prompt(tok),++ hip=cfg.get("backend") == "hip")def engine_args(cfg: dict) -> list[str]:@@@ -261,7 -208,7 +272,7 @@@ def sweep(s, values: list, base_tune: dreturn out- def measure(base_args: list[str], ids_list, start_engine, say=print, extra_workers=(), long_prompt=None) -> dict:-def measure(base_args: list[str], ids_list, start_engine, say=print, extra_workers=(), hip: bool = False) -> dict:++def measure(base_args: list[str], ids_list, start_engine, say=print, extra_workers=(), long_prompt=None, hip: bool = False) -> dict:t0 = time.time()report: dict = {}say(" Loading the model for the measurements ...")diff --cc tools/test_calibrate.pyindex 914634fbb,3fb4804a2..5ea6cabfe--- a/tools/test_calibrate.py+++ b/tools/test_calibrate.py@@@ -224,8 -199,7 +266,8 @@@ class Calibrate(unittest.TestCase)(tok / "token_type.json").write_text(json.dumps([1, 1]))seen = []saved = CAL.measure- CAL.measure = lambda args, ids_list, start_engine, say=print, extra=(), long_prompt=None: (- seen.append(args) or {})- CAL.measure = lambda args, ids_list, start_engine, say=print, extra=(), hip=False: seen.append((args, hip)) or {}++ CAL.measure = lambda args, ids_list, start_engine, say=print, extra=(), long_prompt=None, hip=False: (++ seen.append((args, hip)) or {})fake = type("ST", (), {"Tokenizer": lambda *a: type("T", (), {"encode": lambda s, t, **k: [0]})()})try:with mock.patch.dict(sys.modules, {"strata_tokenizer": fake}):