Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

docs: document the opt-in NUMA-split CPU expert pool

Baptiste Rajautafbd64bc4fa6
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 2d72e5922..6a85dfc42 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -2110,6 +2110,27 @@ never changes a result.
stalls above 100 us, in 7 and 8 of 12 requests off against 0 or 1 with `auto`. That preemption of the host causes the
loss is an inference; the counters do not prove it. The warm-up requests of each run were discarded.
+**NUMA-split CPU expert pool (opt-in, `STRATA_NUMA_SPLIT=1`):** on a 2-socket Linux box it splits the CPU expert pool into
+one slice per NUMA node, each with its own task counter and barrier, and at load it places each node's half of every
+CPU-side expert's rows in that node's memory (first touch by threads pinned to that node; the model file's page cache is
+released first), so a slice reads its own socket's memory. Under the split the default worker count is the physical cores
+of both sockets minus the host core, minus one more kept free from 16 workers on, capped at 32 (22 on 2x12 cores); an explicit `--pool-workers N` is spread alternately across the
+sockets. Measured on a dual Xeon Gold 6226 (2x12 cores, 64 GB, 4 of 6 memory channels per socket populated), RTX 3090,
+PCIe 3.0, `orcarouter-qwen3.8-flash-next-uncensored-iq4_xs` (the IQ4_XS pack) at 262K context, server decode median over 12
+samples: 60.0 tok/s with the split against 52.6 for the 0.2.0 default (11 workers, one socket), +14%. Without the split,
+22 workers gave 50.25 tok/s, and the split at 12 workers gave 51.5; prompt speed is unchanged. It needs Linux, exactly 2
+NUMA nodes and enough free memory on each node for its half: otherwise the log warns `NUMA placement at risk`, and after
+the load it judges the placement from move_pages samples and prints FAILED when a node holds under 95% of its slice. Only
+native expert layers are split (`ArenaExpertSource` and the `cudaHostAlloc` path are not, and the engine says so at start-up
+for the arena path). The pool refuses the split with `STRATA_EXCHANGE_ROTATE=1` and when the two sockets' worker counts are
+lopsided, logs the reason (`NUMA split off: ...`) and runs as usual. `STRATA_NUMA_SPLIT=test` builds the same slices by
+alternating the threads between two slices on any machine, with no node lookup, no pinning requirement and no memory
+placement; it is for testing the per-slice scheduling, not for speed. Off switch: leave `STRATA_NUMA_SPLIT` unset (the
+default; the default path is unchanged). Log: `strata cpu pool: NUMA split on: slice 0 = node 0 (N threads), slice 1 =
+node 1 (M threads)` at start, then `FileExpertSource: NUMA placement: node 0 <GiB> (xx.x % of its slices), node 1
+<GiB> (xx.x % of its slices); free before: ...; after the drop: ...; pages taken and page-locked in N s since the
+placement started`.
+
---
## Experimental speed projection (EXPERIMENTAL, off by default)