Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Merge f-022-numa: NUMA-split CPU pool on multi-node machines (#3201)

Niko12214bd591269537
diff --cc docs/release-notes-0.2.2.md
index 5d6794b29,9cad27fed..77499fea9
--- a/docs/release-notes-0.2.2.md
+++ b/docs/release-notes-0.2.2.md
@@@ -2,4 -2,4 +2,5 @@@
## What is new
+- **An engine package for the RX 6000 series (built, not tested on a real RX 6000 card).** Linux has `rx6000` and Windows has `rdna2`, each with GPU code for gfx1030 and gfx1031 (RX 6800, 6900 and 6950 series, RX 6700 XT). They are offered to a PC whose card reports one of those two and to no other: the RX 7000, RX 9000 and Strix Halo packages are unchanged, and a PC with an RX 7000 or 9000 card next to an RX 6000 card keeps the package it had. The engine runs plain kernels on these cards (RDNA2 has no matrix cores, so none of the RX 7000 / RX 9000 or Radeon 780M prompt paths apply) and plain hipBLAS (hipBLASLt ships no kernels for gfx1030, so there is no tuning table). Setup and the app say once that this package was built and not tested on a real RX 6000 card. What was checked: the Linux package builds, every program in it has code for exactly gfx1030 and gfx1031, the rocBLAS kernel files for both are in it, every library it imports is in the package or belongs to the system, and the whole engine (all targets, tests included) compiles for gfx1030 and gfx1031. What was not: a model on an RX 6000 card, on either system, and the Windows package itself (the Windows build scripts take the new architecture list, but no Windows `rdna2` package has been built). Earlier reports on these cards came from people who built the engine from source ([docs/AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030)). If something goes wrong, open an issue with the end of `strata-<model>.log`.
+ - **The CPU expert pool splits per NUMA node on a machine with two or more NUMA nodes (built from the report in #3201, not tested on a two-socket machine).** Until now the pool kept its workers on the host thread's socket, because workers on the other socket read the experts across the link and were slower. The split gives each socket its own slice of the pool with its own task counter, and places each socket's half of every CPU-side expert in that socket's memory, so a slice reads local memory only. The reporter measured +10 to +14 % decode on a dual Xeon Gold 6226 (59.3 against 54.05 tok/s with the IOMMU in passthrough); a second reporter's dual Xeon E5-2670 v2 preferred more workers across both sockets, which the split also uses. It starts by itself when the machine has two or more NUMA nodes with CPUs and runs only when it can (Linux, two nodes, pinned threads, enough free memory per node); otherwise the pool is exactly what it was and the log says why. **A machine with one NUMA node runs the same code path as 0.2.1:** the identity gate (10 prompts, exact_pp 4K and 20K) is byte-identical and the single-request speed A/B is not slower on the one-node test boxes (numbers in the notes of the branch). `STRATA_NUMA_POOL=0` turns the split off; `STRATA_NUMA_SPLIT=1` still asks for it on any machine. The start-up log says what it did: `NUMA pool on: 2 NUMA nodes found ...` or `NUMA pool off: <reason>`. On Windows the node count is read but the split itself is Linux-only for now, so a two-node Windows PC logs the reason and runs as before. The logic runs in unit tests on made-up two-, three- and four-node machines; we have no two-socket machine, so the placement and the speed on real ones are the reporters' numbers, not ours. Do not wrap the server in `numactl --cpunodebind` or `--membind`: it pulls both halves onto one node. Thanks to baptisterajaut for the branch this is built on, and to xrip and Tworks25 for their numbers. Please send the start-up lines and a decode run with and without `STRATA_NUMA_POOL=0` from a two-socket machine.