Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
NUMA pool: STRATA_TEST_SYSFS_ROOT lets a test give the pool a made-up two-node machine; docs
Niko12217274c10e9ac4
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 8360cb672..1bc4cd924 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -649,7 +649,7 @@ when the sockets wait for each other's counters. Prompt speed did not change inwith it translating every DMA (`cat /sys/kernel/iommu_groups/*/type | sort | uniq -c` shows `DMA-FQ`), pinned host-to-device copies ran at7.7 to 9.2 GB/s on PCIe 3.0 x16 and at 12.3 GB/s with `iommu=pt` on the kernel command line, which moved prompt speed by about 30% there.A NUMA-aware pool (one slice of workers per socket, each reading its own socket's copy of the CPU experts) gave +10 to +14% on-the first box in a contributor's branch; it is not in this build. Do not wrap the server in `numactl --cpunodebind` when you test+the first box in a contributor's branch; 0.2.2 has it (on by itself with 2 or more NUMA nodes, `STRATA_NUMA_POOL=0` turns it off: see "NUMA-split CPU expert pool" below). Do not wrap the server in `numactl --cpunodebind` when you testworkers across sockets: it pins them to one node.### Running it at startup (Task Scheduler)@@ -2110,7 +2110,7 @@ never changes a result.stalls above 100 us, in 7 and 8 of 12 requests off against 0 or 1 with `auto`. That preemption of the host causes theloss is an inference; the counters do not prove it. The warm-up requests of each run were discarded.-**NUMA-split CPU expert pool (opt-in, `STRATA_NUMA_SPLIT=1`):** on a 2-socket Linux box it splits the CPU expert pool into+**NUMA-split CPU expert pool (0.2.2: on by itself when the machine has 2 or more NUMA nodes; off with `STRATA_NUMA_POOL=0`):** on a 2-socket Linux box it splits the CPU expert pool intoone slice per NUMA node, each with its own task counter and barrier, and at load it places each node's half of everyCPU-side expert's rows in that node's memory (first touch by threads pinned to that node; the model file's page cache isreleased first), so a slice reads its own socket's memory. Under the split the default worker count is the physical cores@@ -2126,8 +2126,9 @@ native expert layers are split (`ArenaExpertSource` and the `cudaHostAlloc` pathfor the arena path). The pool refuses the split with `STRATA_EXCHANGE_ROTATE=1` and when the two sockets' worker counts arelopsided, logs the reason (`NUMA split off: ...`) and runs as usual. `STRATA_NUMA_SPLIT=test` builds the same slices byalternating the threads between two slices on any machine, with no node lookup, no pinning requirement and no memory-placement; it is for testing the per-slice scheduling, not for speed. Off switch: leave `STRATA_NUMA_SPLIT` unset (the-default; the default path is unchanged). Log: `strata cpu pool: NUMA split on: slice 0 = node 0 (N threads), slice 1 =+placement; it is for testing the per-slice scheduling, not for speed.++*0.2.2: when it runs.* The split was a contributor's opt-in (`STRATA_NUMA_SPLIT=1`, issue #3201, measured on one dual Xeon Gold 6226 box). From 0.2.2 the pool counts the NUMA nodes that have CPUs (Linux: `/sys/devices/system/node/node*/cpulist`; Windows: `GetNumaHighestNodeNumber`) and, with 2 or more and neither `STRATA_NUMA_POOL` nor `STRATA_NUMA_SPLIT` set, tries the split by itself. **One node is exactly the pool as before: nothing is read, nothing is logged, no worker count changes.** On 2 or more nodes the split runs only when it can (Linux, exactly 2 nodes holding the pool's threads, balanced slices, pinned threads, no `STRATA_EXCHANGE_ROTATE=1`); otherwise the pool keeps its old worker count and cores and logs why: `strata cpu pool: NUMA pool off: <reason> (N NUMA nodes found), the pool runs as before` (a 4-node machine, a container that sees one node's CPUs, Windows). When it runs the log says `strata cpu pool: NUMA pool on: N NUMA nodes found, one slice of the pool per node (STRATA_NUMA_POOL=0 turns it off)`, then the `NUMA split on: slice 0 = node ...` line. **Off switch: `STRATA_NUMA_POOL=0`** (also `off`, `false`, `no`); it wins over `STRATA_NUMA_SPLIT=1`, and on a 2-node machine logs `NUMA pool off (STRATA_NUMA_POOL=0)`. `STRATA_NUMA_SPLIT=1` still asks for the split on any machine (and keeps the contributor's worker counts when it cannot run), `STRATA_NUMA_SPLIT=0` is off. On Windows the node count is read but the split itself is Linux-only for now: a Windows machine with 2 nodes logs the reason and runs as before. The 0.2.2 default was built from the report and tested on one-node machines only (identity gate and speed A/B unchanged there; the node logic runs in `numa_pool_test` on made-up topologies); it has not been run on a two-socket machine by us. Log: `strata cpu pool: NUMA split on: slice 0 = node 0 (N threads), slice 1 =node 1 (M threads)` at start, then `FileExpertSource: NUMA placement: node 0 <GiB> (xx.x % of its slices), node 1<GiB> (xx.x % of its slices); free before: ...; after the drop: ...; pages taken and page-locked in N s since theplacement started`.diff --git a/src/kernels/cpu/pool.cpp b/src/kernels/cpu/pool.cppindex e3f113d33..cbbf8aad6 100644--- a/src/kernels/cpu/pool.cpp+++ b/src/kernels/cpu/pool.cpp@@ -533,12 +533,20 @@ bool pin_this_thread(int core, [[maybe_unused]] int worker = -1) {}#if defined(__linux__)+// Where the NUMA topology is read from: /sys. STRATA_TEST_SYSFS_ROOT=<dir> reads <dir>/devices/system/{node,cpu} instead,+// so a test can give the pool a made-up two-node machine (pool_tasks_test); nobody sets it for real.+std::string sysfs_root() {+ const char* e = std::getenv("STRATA_TEST_SYSFS_ROOT");+ return e != nullptr && *e != '\0' ? std::string(e) : std::string("/sys");+}+// The NUMA node of a logical CPU (sysfs), or -1.int cpu_numa_node(int cpu) {if (cpu < 0) return -1;- char path[96];+ const std::string root = sysfs_root();+ char path[512];for (int node = 0; node < 64; ++node) {- std::snprintf(path, sizeof path, "/sys/devices/system/cpu/cpu%d/node%d", cpu, node);+ std::snprintf(path, sizeof path, "%s/devices/system/cpu/cpu%d/node%d", root.c_str(), cpu, node);if (access(path, F_OK) == 0) return node;}return -1;@@ -549,7 +557,7 @@ int cpu_numa_node(int cpu) {// (numa_pool.hpp); Windows: GetNumaHighestNodeNumber; any other system, or a sandbox that hides the folder: 1.int numa_node_count() {#if defined(__linux__)- return (std::max)(1, numa_cpu_node_count_at("/sys/devices/system/node"));+ return (std::max)(1, numa_cpu_node_count_at(sysfs_root() + "/devices/system/node"));#elif defined(_WIN32)ULONG highest = 0;return GetNumaHighestNodeNumber(&highest) ? (int) highest + 1 : 1;diff --git a/src/kernels/cpu/pool_tasks_test.cpp b/src/kernels/cpu/pool_tasks_test.cppindex c3cce98df..aa272880d 100644--- a/src/kernels/cpu/pool_tasks_test.cpp+++ b/src/kernels/cpu/pool_tasks_test.cpp@@ -8,6 +8,9 @@#include <cstdio>#include <cstdlib>#include <cstring>+#include <ctime>+#include <filesystem>+#include <fstream>#include <limits>#include <random>#include <stdexcept>@@ -287,6 +290,82 @@ int main(int argc, char** argv) {std::printf("pool tasks: nothing set: split %s%s\n", plan.on ? "on" : "off", plan.why.empty() ? "" : (": " + plan.why).c_str());if (!plan.on && !plan.why.empty()) { std::fprintf(stderr, "an automatic split that did not run left a refusal reason\n"); return 1; }}+ { // 0.2.2: a made-up two-node machine (STRATA_TEST_SYSFS_ROOT): the pool's own node lookup, the automatic split, its log lines+ const std::vector<int> cores = cpu::detect_cpu_topology(true, cpu::PoolAffinity::All).worker_cores;+ const int host = cpu::detect_cpu_topology(true, cpu::PoolAffinity::All).host_core;+ if (cores.size() < 4 || host < 0) {+ std::printf("pool tasks: fake two-node machine SKIPPED (needs 4 worker cores and a host core)\n");+ } else {+ namespace fs = std::filesystem;+ const fs::path root = fs::temp_directory_path() / ("strata_pool_fake_sysfs_" + std::to_string((long long) std::time(nullptr)));+ // node k of worker core i: the workers alternate between the nodes, the host core is on node 0+ auto build = [&](int n_nodes) {+ fs::remove_all(root);+ fs::create_directories(root / "devices/system/node");+ std::vector<std::string> list((size_t) n_nodes);+ auto put = [&](int cpu, int node) {+ fs::create_directories(root / "devices/system/cpu" / ("cpu" + std::to_string(cpu)) / ("node" + std::to_string(node)));+ list[(size_t) node] += (list[(size_t) node].empty() ? "" : ",") + std::to_string(cpu);+ };+ put(host, 0);+ for (size_t i = 0; i < cores.size(); ++i) put(cores[i], (int) (i % (size_t) n_nodes));+ for (int k = 0; k < n_nodes; ++k) {+ fs::create_directories(root / "devices/system/node" / ("node" + std::to_string(k)));+ std::ofstream(root / "devices/system/node" / ("node" + std::to_string(k)) / "cpulist") << list[(size_t) k] << "\n";+ }+ };+ put_env("STRATA_TEST_SYSFS_ROOT", root.string().c_str());+ unset_env("STRATA_NUMA_SPLIT");+ unset_env("STRATA_NUMA_POOL");+ const int workers = (int) std::min<size_t>(cores.size(), 8) & ~1; // even: the nodes get the same count++ build(2); // two nodes, nothing set: the split runs by itself+ {+ cpu::ExpertPool pool(workers, true, true);+ const auto& plan = cpu::active_numa_split();+ const bool ok = plan.on && !plan.test && plan.nodes[0] == 0 && plan.nodes[1] == 1 && pool.numa_slices() == 2 &&+ !plan.cores[0].empty() && !plan.cores[1].empty();+ std::printf("pool tasks: fake two nodes, nothing set: split %s (%zu + %zu cores)\n", plan.on ? "on" : "off",+ plan.cores[0].size(), plan.cores[1].size());+ if (!ok) { std::fprintf(stderr, "the automatic split did not run on a two-node machine\n"); return 1; }+ }+ put_env("STRATA_NUMA_POOL", "0"); // the off switch+ {+ cpu::ExpertPool pool(workers, true, true);+ if (cpu::active_numa_split().on || pool.numa_slices() != 0) { std::fprintf(stderr, "STRATA_NUMA_POOL=0 did not switch the automatic split off\n"); return 1; }+ }+ unset_env("STRATA_NUMA_POOL");+ put_env("STRATA_EXCHANGE_ROTATE", "1"); // the split cannot run: the pool stays as it was, with no half-made plan+ {+ cpu::ExpertPool pool(workers, true, true);+ if (cpu::active_numa_split().on || pool.numa_slices() != 0 || !cpu::active_numa_split().why.empty()) {+ std::fprintf(stderr, "an automatic split that cannot run left a plan or a reason behind\n");+ return 1;+ }+ }+ unset_env("STRATA_EXCHANGE_ROTATE");+ for (int nodes : {1, 3}) { // one node: the pool before the split; three: not supported, the pool as it was+ build(nodes);+ cpu::ExpertPool pool(workers, true, true);+ if (cpu::active_numa_split().on || pool.numa_slices() != 0) { std::fprintf(stderr, "split on a %d-node machine\n", nodes); return 1; }+ }+ { // the default worker count and cores of a one-node machine do not depend on the split code at all+ build(1);+ cpu::ExpertPool a(0, true, true);+ const std::vector<int> a_cores = a.worker_cores();+ const int a_workers = a.workers();+ unset_env("STRATA_TEST_SYSFS_ROOT");+ cpu::ExpertPool b(0, true, true);+ if (a_workers != b.workers() || a_cores != b.worker_cores()) {+ std::fprintf(stderr, "a one-node fake machine changed the pool's workers or cores\n");+ return 1;+ }+ std::printf("pool tasks: one node: %d workers and the same cores as with no topology override\n", a_workers);+ }+ unset_env("STRATA_TEST_SYSFS_ROOT");+ fs::remove_all(root);+ }+ }#endifstd::printf("pool tasks: %d bitwise comparisons passed, %d of them in NUMA-split mode (%d with modes 0-4 ""interleaved)\n", checks + split_checks + mixed_checks, split_checks + mixed_checks, mixed_checks);