Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

CPU pool: say where the workers run, --pool-cores (experimental), names for the engine's threads

Niko12217a5599581881
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 96ebc4d66..632e32e93 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -2108,6 +2108,10 @@ if(STRATA_BUILD_TESTS)
add_executable(pool_test src/kernels/cpu/pool_test.cpp)
target_link_libraries(pool_test PRIVATE strata_kernels_cpu)
add_test(NAME pool_test COMMAND pool_test --selftest WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+ # where the workers are placed (one per physical core by default) and --pool-cores (real pools, no model)
+ add_executable(pool_cores_test src/kernels/cpu/pool_cores_test.cpp)
+ target_link_libraries(pool_cores_test PRIVATE strata_kernels_cpu)
+ add_test(NAME pool_cores_test COMMAND pool_cores_test)
# the default CPU expert worker count on mock CPUs (8c/16t, 16c/32t, 64c/128t, 2-socket 128c/256t, ...)
add_executable(pool_plan_test src/kernels/cpu/pool_plan_test.cpp)
target_include_directories(pool_plan_test PRIVATE include)
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 0a55c8eca..e5e94af57 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -2070,6 +2070,58 @@ the pairs in which the switched arm won). They are here so you can try them on y
option. The uniform cache (no profile) holds no resident expert when a native pack's prompt is read (serve needs the
profile, and a one-shot run fills the cache only while it decodes), so `expert_cache_slot_guard_test` covers it.
+## Which logical processors the CPU workers use, and `--pool-cores` (experimental, off by default)
+
+The CPU expert workers read the experts that are not in graphics memory, and their speed is set by how many physical cores
+read memory at the same time. So the engine starts one worker per physical core, never two workers on the two hardware
+threads (SMT siblings) of one core, and keeps the first core for the main thread (the one that waits for the graphics
+card). The start-up log says where everything sits:
+
+```
+strata generate: CPU workers on logical processors 2,4,...,30 (15 workers, one per physical core; host on 0)
+```
+
+The numbers depend on the system. Windows numbers the two threads of a core next to each other (core k is logical
+processors 2k and 2k+1), so the workers sit on the even numbers. Linux usually numbers the first thread of every core first
+and the siblings after them (a 6-core, 12-thread CPU: cores 0 to 5, their siblings 6 to 11), so the workers sit on the low
+numbers. The engine does not guess: it reads the pairing from the system (Windows `GetLogicalProcessorInformationEx`, Linux
+`/sys/devices/system/cpu/cpu*/topology`). Where the system does not say (some containers), the line says so
+("the OS did not say which logical processors share a core") and the default count is half of the logical processors. The
+line also says when two workers do share a core. `ps -L -o tid,psr,comm` on Linux, and Process Explorer or any tool that reads
+thread names on Windows, show the workers as `strata-pool-N` and the main thread as `strata-host`; the other helper threads
+carry names too (`strata-stage`, `strata-adapt`, `strata-io`, ...).
+
+Only the workers and the main thread are pinned. Every other thread (the prompt copy threads, the adaptive tier's job thread,
+the graphics driver's threads) is placed by the operating system, and the picture encoder is a separate process whose
+threads are not pinned either. A task manager that shows only the first few logical processors busy is therefore not
+necessarily the pool: while the experts are in graphics memory the workers have nothing to do. On a Ryzen AI MAX+ 395 (16
+cores, 32 threads, Windows 11) with every expert in graphics memory, five snapshots of a running engine showed the
+unpinned threads on logical processors 0 to 15 only (the first eight cores, both threads of each) and 16 to 31 idle, while
+the 15 workers sat parked on 2, 4, ..., 30. With the experts on the CPU the same machine's workers were each at 100% on
+exactly those even numbers (CPU pool alone on random expert data, so only the placement and the memory traffic are real).
+That is why the default is not "the first N logical processors": on that machine, 8 workers on logical processors 1 to 8
+(four cores, both threads) read 67 GB/s, 8 workers on real cores 2, 4, ..., 16 read 94 GB/s, and the default 15 workers on
+2, 4, ..., 30 read 146 GB/s (median of three interleaved 10-second runs; 31 workers on 1 to 31 read 123 GB/s).
+
+**`--pool-cores LIST`** (or `STRATA_POOL_CORES=LIST`, or `"pool_cores": "2,4,6,8"` in the run config, or
+`strata launch --pool-cores 2,4,6,8`) pins the workers to exactly these logical processors, in this order. It is for people who
+want to choose the cores themselves (leave some for other programs, use one chiplet, test a layout); the default is what was
+measured. The list is numbers, ranges and `first-last:step`: `0,2,4,6`, `2-14:2`. The numbers are the ones Task Manager and
+`lscpu` show. The worker count is the length of the list; `--pool-workers N` takes the first N entries (more workers than
+entries: the extra ones run unpinned). Nothing is refused; the engine warns:
+
+```
+strata cpu pool: --pool-cores: logical processors 0 and 1 are the two hardware threads of one core: two workers on it share the core's memory bandwidth and slow each other down (one per physical core is the default)
+strata cpu pool: --pool-cores: logical processor 0 is also the host thread's: a worker there competes with the thread that waits for the graphics card
+strata cpu pool: --pool-cores: left out: 99 is not a logical processor this process may run on
+```
+
+Processors the process may not use (not there, or outside a `taskset` / affinity limit) are left out; if none is left the
+pool uses its default placement. A list that cannot be read (`two,four`) is said and ignored. The main thread keeps the first
+core (`--host-core` still decides that), so leave that processor out of the list. The option moves threads only and never
+changes a result. It applies to the main engine; the Qwen3.6 engine (`strata-q36`) keeps its own `--pool-workers` and the
+default placement.
+
## Thread placement: `--aux-cpus` (Linux, off by default)
The CPU pool pins its workers (one logical CPU per physical core) and the host thread pins itself to the pool's reserved
diff --git a/include/strata/kernels/cpu/pool.hpp b/include/strata/kernels/cpu/pool.hpp
index 98320a925..4f49a3a40 100644
--- a/include/strata/kernels/cpu/pool.hpp
+++ b/include/strata/kernels/cpu/pool.hpp
@@ -113,6 +113,18 @@ enum class HostCore { First, Last, Sibling };
void set_host_core(HostCore where);
HostCore host_core_setting();
+/// `--pool-cores LIST` / STRATA_POOL_CORES / the config's "pool_cores" (experimental, off by default): pin the pool's
+/// workers to exactly these logical processors, in this order, instead of one per physical core. An empty list (the
+/// default) leaves the placement as it has always been. Set before the pool is built. Processors the process may not
+/// run on are left out with a warning; SMT siblings of one core and the host's processor are kept with a warning (a
+/// person who asks for them gets them). The worker count is the list's length unless --pool-workers says otherwise.
+void set_pool_cores(const std::vector<int>& cpus);
+const std::vector<int>& pool_cores_setting();
+
+/// Every logical processor this process may run on and the physical core it belongs to (see LogicalCpuInfo). Empty when
+/// the system does not say (macOS, a sandbox without the topology).
+std::vector<LogicalCpuInfo> logical_cpu_layout();
+
CpuTopology detect_cpu_topology(bool skip_first, PoolAffinity affinity = PoolAffinity::All);
/// The logical processor the host thread is pinned to under the current setting (-1: no topology).
@@ -198,6 +210,11 @@ public:
/// What bounded the default worker count (pool_plan.hpp; "" when a count was configured), the physical cores
/// and logical processors counted - for the start-up line.
const char* plan_limit() const { return plan_limit_; }
+ /// True when the workers sit where `--pool-cores` said (the list, minus what could not be used).
+ bool explicit_cores() const { return explicit_cores_; }
+ /// The start-up line about where the workers and the host run ("CPU workers on logical processors 2,4,...,30 (15
+ /// workers, one per physical core; host on 0)"); "" without workers.
+ std::string placement_line() const;
int plan_cores() const { return plan_cores_; }
int plan_logical() const { return plan_logical_; }
/// STRATA_NUMA_SPLIT: 2 when the native row phases are split per NUMA node, 0 when off.
@@ -283,6 +300,8 @@ private:
void publish();
int n_ = 0;
+ bool explicit_cores_ = false;
+ bool pinned_ = true;
const char* plan_limit_ = "";
int plan_cores_ = 0;
int plan_logical_ = 0;
diff --git a/include/strata/kernels/cpu/pool_plan.hpp b/include/strata/kernels/cpu/pool_plan.hpp
index 42e538bf1..1c791291f 100644
--- a/include/strata/kernels/cpu/pool_plan.hpp
+++ b/include/strata/kernels/cpu/pool_plan.hpp
@@ -145,6 +145,184 @@ inline std::string pool_workers_warning(int requested, const PoolPlanInput& in,
std::to_string(plan.workers) + " workers and the end of each layer waits for every one). " + rec;
}
+// ---------------------------------------------------------------------------------------------------------------------
+// Where the workers run: the start-up line, and `--pool-cores LIST` (experimental, off by default).
+// Pure functions again (no OS calls), unit tested in pool_plan_test.cpp.
+// ---------------------------------------------------------------------------------------------------------------------
+
+/// One logical processor the process may use and the physical core it belongs to (`core`: any number that is the
+/// same for the SMT siblings of one core and different between cores, across sockets too; -1: the OS did not say).
+/// `cpu` is the OS's number: Linux CPU id, on Windows group * 64 + the processor number inside the group (Task
+/// Manager's CPU number on a PC with one processor group).
+struct LogicalCpuInfo {
+ int cpu = -1;
+ long core = -1;
+};
+
+/// "2,4,6,8,10,12" -> "2,4,...,12" when the numbers step by a constant from the second on and there are at least four of
+/// them; "0,1,2,3" -> "0-3"; anything else is written out in full (at most 64 numbers, then "... and N more").
+inline std::string format_cpu_list(const std::vector<int>& v) {
+ if (v.empty()) return "none";
+ auto num = [](int x) { return std::to_string(x); };
+ if (v.size() >= 4) {
+ const int step = v[1] - v[0];
+ bool even = step != 0;
+ for (size_t i = 2; even && i < v.size(); ++i) even = v[i] - v[i - 1] == step;
+ if (even) return step == 1 ? num(v.front()) + "-" + num(v.back())
+ : num(v[0]) + "," + num(v[1]) + ",...," + num(v.back());
+ }
+ std::string s;
+ for (size_t i = 0; i < v.size() && i < 64; ++i) s += (i ? "," : "") + num(v[i]);
+ if (v.size() > 64) s += ",... and " + std::to_string(v.size() - 64) + " more";
+ return s;
+}
+
+/// `--pool-cores` / STRATA_POOL_CORES: "0,2,4,6" or "2-14:2" (first-last, optional :step) or a mix, in the order given.
+/// False with a plain `err` for anything else (empty items, letters, a reversed range, a step of 0); `out` is then empty.
+/// A number that is listed twice is kept once (the second is dropped).
+inline bool parse_pool_cores(const std::string& text, std::vector<int>& out, std::string& err) {
+ out.clear();
+ err.clear();
+ size_t i = 0;
+ auto skip = [&] { while (i < text.size() && (text[i] == ' ' || text[i] == '\t')) ++i; };
+ auto number = [&](long& v) {
+ skip();
+ if (i >= text.size() || text[i] < '0' || text[i] > '9') return false;
+ v = 0;
+ while (i < text.size() && text[i] >= '0' && text[i] <= '9') {
+ v = v * 10 + (text[i] - '0');
+ if (v > 100000) return false;
+ ++i;
+ }
+ return true;
+ };
+ skip();
+ if (i >= text.size()) { err = "the list is empty (write it like 0,2,4,6 or 2-14:2)"; return false; }
+ for (;;) {
+ long lo = 0, hi = 0, step = 1;
+ if (!number(lo)) { err = "'" + text + "' is not a list of logical processor numbers (write it like 0,2,4,6 or 2-14:2)"; out.clear(); return false; }
+ hi = lo;
+ skip();
+ if (i < text.size() && text[i] == '-') {
+ ++i;
+ if (!number(hi) || hi < lo) { err = "'" + text + "' has a range that does not go up (write it like 2-14:2)"; out.clear(); return false; }
+ skip();
+ if (i < text.size() && text[i] == ':') {
+ ++i;
+ if (!number(step) || step < 1) { err = "'" + text + "' has a step that is not a positive number"; out.clear(); return false; }
+ }
+ }
+ for (long c = lo; c <= hi; c += step)
+ if (std::find(out.begin(), out.end(), (int) c) == out.end()) out.push_back((int) c);
+ skip();
+ if (i == text.size()) break;
+ if (text[i] != ',') { err = "'" + text + "' is not a list of logical processor numbers (write it like 0,2,4,6 or 2-14:2)"; out.clear(); return false; }
+ ++i;
+ }
+ return true;
+}
+
+/// What `--pool-cores` made of the person's list: the processors the workers get, in the order given, and what to tell
+/// them. Recommend, never force: the only entries dropped are those that cannot be used at all (a processor the process
+/// may not run on, or that does not exist, cannot be pinned to). Two entries that are SMT siblings of one core, or the
+/// host thread's own processor, are kept with a warning: the workers then share a core, which is what the person asked for.
+struct PoolCoresCheck {
+ std::vector<int> cpus;
+ std::vector<std::string> warnings;
+};
+
+inline PoolCoresCheck check_pool_cores(const std::vector<int>& list, const std::vector<LogicalCpuInfo>& layout, int host_cpu) {
+ PoolCoresCheck r;
+ auto find = [&](int cpu) -> const LogicalCpuInfo* {
+ for (const auto& l : layout)
+ if (l.cpu == cpu) return &l;
+ return nullptr;
+ };
+ std::vector<std::pair<long, int>> seen; // (core, the first listed cpu of it)
+ std::vector<int> dropped;
+ for (int c : list) {
+ const LogicalCpuInfo* info = find(c);
+ if (info == nullptr) { dropped.push_back(c); continue; }
+ r.cpus.push_back(c);
+ if (info->core < 0) continue;
+ bool shared = false;
+ for (const auto& s : seen)
+ if (s.first == info->core) {
+ r.warnings.push_back("logical processors " + std::to_string(s.second) + " and " + std::to_string(c) +
+ " are the two hardware threads of one core: two workers on it share the core's memory "
+ "bandwidth and slow each other down (one per physical core is the default)");
+ shared = true;
+ break;
+ }
+ if (!shared) seen.emplace_back(info->core, c);
+ }
+ if (!dropped.empty())
+ r.warnings.push_back("left out: " + format_cpu_list(dropped) +
+ (dropped.size() == 1 ? " is not a logical processor this process may run on"
+ : " are not logical processors this process may run on"));
+ if (host_cpu >= 0) {
+ const LogicalCpuInfo* h = find(host_cpu);
+ for (int c : r.cpus) {
+ const LogicalCpuInfo* info = find(c);
+ if (c == host_cpu) {
+ r.warnings.push_back("logical processor " + std::to_string(c) + " is also the host thread's: a worker there "
+ "competes with the thread that waits for the graphics card");
+ break;
+ }
+ if (h != nullptr && info != nullptr && h->core >= 0 && info->core == h->core) {
+ r.warnings.push_back("logical processor " + std::to_string(c) + " shares a core with the host thread's " +
+ std::to_string(host_cpu));
+ break;
+ }
+ }
+ }
+ return r;
+}
+
+/// The start-up line about where the workers run, without the "strata generate: " prefix:
+/// "CPU workers on logical processors 2,4,...,30 (15 workers, one per physical core; host on 0)".
+/// `layout` empty or a worker/host processor not in it: the OS did not say which processors are SMT siblings, and the
+/// line says so instead of claiming "one per physical core". `list_given`: the workers are the person's --pool-cores.
+inline std::string describe_worker_placement(const std::vector<int>& workers, int host_cpu,
+ const std::vector<LogicalCpuInfo>& layout, bool list_given) {
+ if (workers.empty()) return "";
+ auto core_of = [&](int cpu) -> long {
+ for (const auto& l : layout)
+ if (l.cpu == cpu) return l.core;
+ return -1;
+ };
+ bool known = true;
+ std::vector<long> cores;
+ for (int w : workers) {
+ const long c = core_of(w);
+ if (c < 0) known = false;
+ cores.push_back(c);
+ }
+ const long host_core = host_cpu >= 0 ? core_of(host_cpu) : -1;
+ std::string s = "CPU workers on logical processors " + format_cpu_list(workers) + " (" + std::to_string(workers.size()) +
+ (workers.size() == 1 ? " worker" : " workers");
+ if (list_given) s += ", your --pool-cores list";
+ if (!known) {
+ s += ", the OS did not say which logical processors share a core";
+ } else {
+ std::vector<long> distinct = cores;
+ std::sort(distinct.begin(), distinct.end());
+ distinct.erase(std::unique(distinct.begin(), distinct.end()), distinct.end());
+ if (distinct.size() == workers.size()) {
+ s += ", one per physical core";
+ } else {
+ size_t doubled = 0; // cores that carry more than one worker
+ for (long d : distinct) doubled += std::count(cores.begin(), cores.end(), d) > 1;
+ s += " on " + std::to_string(distinct.size()) + " physical core" + (distinct.size() == 1 ? "" : "s") +
+ "; " + std::to_string(doubled) + (doubled == 1 ? " core carries" : " cores carry") + " more than one worker (they slow each other down)";
+ }
+ if (host_core >= 0 && std::find(cores.begin(), cores.end(), host_core) != cores.end())
+ s += "; a worker shares the host's core";
+ }
+ s += host_cpu >= 0 ? "; host on " + std::to_string(host_cpu) : "; host not pinned";
+ return s + ")";
+}
+
/// STRATA_NUMA_SPLIT: whether two slices of `a` and `b` threads may split the rows 50/50 - a slice with under half
/// the other's threads would be the straggler of every batch, so the pool refuses that split.
inline bool numa_slices_balanced(int a, int b) { return std::min(a, b) * 2 >= std::max(a, b); }
diff --git a/include/strata/platform/aux_cpus.hpp b/include/strata/platform/aux_cpus.hpp
index cd90e7009..f8641e735 100644
--- a/include/strata/platform/aux_cpus.hpp
+++ b/include/strata/platform/aux_cpus.hpp
@@ -19,6 +19,7 @@
// do not link each other. No output of the engine depends on where a thread runs.
#pragma once
+#include "strata/platform/thread_name.hpp"
#include <algorithm>
#include <atomic>
#include <cstdio>
diff --git a/include/strata/platform/thread_name.hpp b/include/strata/platform/thread_name.hpp
new file mode 100644
index 000000000..ec97533a4
--- /dev/null
+++ b/include/strata/platform/thread_name.hpp
@@ -0,0 +1,44 @@
+// strata/platform/thread_name.hpp - names the engine's long-lived threads, so that a person (or a script) can tell the
+// pool workers from the host thread and the helpers when looking at where they run: Linux `ps -L -o tid,psr,comm` and
+// /proc/<pid>/task/<tid>/comm, Windows GetThreadDescription (Process Explorer, WinDbg, the tools/ probes).
+// A name changes nothing the thread does. Header-only: the callers sit in several libraries that do not link each other.
+#pragma once
+
+#include <cstdio>
+
+#if defined(_WIN32)
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN
+#endif
+#include <windows.h>
+#elif defined(__linux__) || defined(__APPLE__)
+#include <pthread.h>
+#endif
+
+namespace strata::platform {
+
+/// Gives the calling thread a name: "strata-pool-3" for `name_this_thread("strata-pool", 3)`, "strata-host" for no index.
+/// Linux keeps 15 characters, the rest is cut. Never fails loudly: a system that cannot name a thread leaves it unnamed.
+inline void name_this_thread(const char* base, int index = -1) {
+ char name[64];
+ if (index >= 0) std::snprintf(name, sizeof name, "%s-%d", base, index);
+ else std::snprintf(name, sizeof name, "%s", base);
+#if defined(_WIN32)
+ // SetThreadDescription (Windows 10 1607 and later) is looked up, not linked: older SDKs and toolchains lack the import
+ using SetDesc = HRESULT(WINAPI*)(HANDLE, PCWSTR);
+ static const SetDesc fn = (SetDesc) GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "SetThreadDescription");
+ if (fn == nullptr) return;
+ wchar_t wide[64];
+ int i = 0;
+ for (; name[i] != '\0' && i < 63; ++i) wide[i] = (wchar_t) (unsigned char) name[i];
+ wide[i] = L'\0';
+ (void) fn(GetCurrentThread(), wide);
+#elif defined(__linux__)
+ name[15] = '\0';
+ (void) pthread_setname_np(pthread_self(), name);
+#elif defined(__APPLE__)
+ (void) pthread_setname_np(name);
+#endif
+}
+
+} // namespace strata::platform
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
index 5ac52d151..b8ad8fb42 100644
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -1199,6 +1199,7 @@ void FileExpertSource::io_enqueue(int64_t layer, int64_t expert, bool ahead) {
// A read-ahead worker: takes a predicted (layer, expert), skips it when its pages are cached or it is already
// staged, and otherwise reads the whole blob into a staging buffer (copy_from_files preads what is not cached).
void FileExpertSource::io_worker() {
+ strata::platform::name_this_thread("strata-io");
for (;;) {
int32_t layer, expert;
bool ahead = true; // a prediction (marked for the used / unused count), not a layer's own demand read
@@ -3032,6 +3033,7 @@ bool RouterLookahead::start(std::vector<std::vector<uint16_t>> routers, int64_t
src_ = src;
x_.assign((size_t) (8 * n_embd), 0.f);
thread_ = std::thread([this] {
+ strata::platform::name_this_thread("strata-lookahead");
strata::aux_cpus::pin_current_thread();
run();
});
@@ -4398,6 +4400,7 @@ bool FileExpertSource::ram_queued(int64_t layer, int64_t expert) const {
}
void FileExpertSource::ram_worker() {
+ strata::platform::name_this_thread("strata-ram");
for (;;) {
RamJob* job = nullptr;
{
@@ -6195,6 +6198,7 @@ ForesightSwap::~ForesightSwap() {
}
void ForesightSwap::fill_loop() {
+ strata::platform::name_this_thread("strata-fill");
int cur_dev = -1;
for (;;) {
int64_t idx;
diff --git a/src/core/session.cpp b/src/core/session.cpp
index b70f683c6..0a25df79d 100644
--- a/src/core/session.cpp
+++ b/src/core/session.cpp
@@ -571,6 +571,7 @@ bool SessionLoopScratch::init(size_t parts_bytes_in, std::string& err) {
pinned = pinned_core.valid;
}
}
+ strata::platform::name_this_thread("strata-host");
strata::aux_cpus::note_owned_thread(); // the host: --aux-cpus leaves it where it is
return true;
}
diff --git a/src/kernels/cpu/pool.cpp b/src/kernels/cpu/pool.cpp
index cbbf8aad6..6bc56e8e7 100644
--- a/src/kernels/cpu/pool.cpp
+++ b/src/kernels/cpu/pool.cpp
@@ -82,6 +82,60 @@ std::atomic<int> g_host_core{(int) HostCore::First};
void set_host_core(HostCore where) { g_host_core.store((int) where); }
HostCore host_core_setting() { return (HostCore) g_host_core.load(); }
+namespace {
+std::vector<int> g_pool_cores; // --pool-cores: set once at start-up, before the pool exists
+} // namespace
+void set_pool_cores(const std::vector<int>& cpus) { g_pool_cores = cpus; }
+const std::vector<int>& pool_cores_setting() { return g_pool_cores; }
+
+std::vector<LogicalCpuInfo> logical_cpu_layout() {
+ std::vector<LogicalCpuInfo> out;
+#if defined(_WIN32)
+ DWORD len = 0;
+ GetLogicalProcessorInformationEx(RelationProcessorCore, nullptr, &len);
+ if (len == 0) return out;
+ std::vector<char> buf(len);
+ if (!GetLogicalProcessorInformationEx(RelationProcessorCore, (PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX) buf.data(), &len))
+ return out;
+ // a `start /affinity` or a job limit on a PC with one processor group: only those processors are the process's
+ DWORD_PTR proc_mask = 0, sys_mask = 0;
+ const bool limited = GetActiveProcessorGroupCount() == 1 && GetProcessAffinityMask(GetCurrentProcess(), &proc_mask, &sys_mask);
+ long core = 0;
+ for (const char* p = buf.data(); p < buf.data() + len; p += ((const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) p)->Size) {
+ const auto* e = (const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) p;
+ if (e->Relationship != RelationProcessorCore) continue;
+ for (WORD group = 0; group < e->Processor.GroupCount; ++group) {
+ const GROUP_AFFINITY& g = e->Processor.GroupMask[group];
+ for (int bit = 0; bit < 64; ++bit)
+ if ((g.Mask & (KAFFINITY(1) << bit)) && (!limited || (proc_mask & (DWORD_PTR(1) << bit))))
+ out.push_back({(int) (g.Group * 64 + bit), core});
+ }
+ ++core;
+ }
+#elif defined(__linux__)
+ std::vector<unsigned long> mask;
+ std::vector<int> allowed;
+ if (detail::get_thread_affinity(mask, &allowed) != 0) return out;
+ auto read = [](int cpu, const char* what) -> long {
+ char path[96];
+ std::snprintf(path, sizeof path, "/sys/devices/system/cpu/cpu%d/topology/%s", cpu, what);
+ long v = -1;
+ if (std::FILE* f = std::fopen(path, "r")) {
+ if (std::fscanf(f, "%ld", &v) != 1) v = -1;
+ std::fclose(f);
+ }
+ return v;
+ };
+ for (int cpu : allowed) {
+ const long pkg = read(cpu, "physical_package_id"), core = read(cpu, "core_id");
+ out.push_back({cpu, pkg >= 0 && core >= 0 ? pkg * 1000000 + core : -1});
+ }
+#else
+ for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) out.push_back({(int) i, -1});
+#endif
+ return out;
+}
+
static CpuTopology detect_cpu_topology_impl(bool skip_first, PoolAffinity affinity);
// The fields the default worker count needs (pool_plan.hpp): the logical processors the process may use, the CPU
@@ -231,6 +285,10 @@ static CpuTopology detect_cpu_topology_impl(bool skip_first, PoolAffinity affini
topo.host_core = topo.worker_cores.front();
topo.worker_cores.erase(topo.worker_cores.begin());
}
+ // The OS did not say which logical processors share a core, and every one is listed as a core above: the default
+ // worker count must not count them as cores (30 workers on a 16c/32t PC would sit two to a core). With the layout
+ // unknown the plan takes half the logical processors (pool_plan.hpp), as on Linux without sysfs.
+ topo.layout_known = false;
return topo;
}
std::vector<char> buf(len);
@@ -685,6 +743,7 @@ void ExpertPool::diag(std::FILE* f) const {
ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, PoolAffinity affinity, int tasks)
: tasks_(tasks), host_works_(host_works), affinity_(affinity), topo_(detect_cpu_topology(true, affinity)) {
if (tasks < 0 || tasks > kMaxTasks) throw std::invalid_argument("expert pool tasks must be in 0..4096");
+ pinned_ = pin;
if (const char* e = std::getenv("STRATA_POOL_SPIN_US")) // a test knob; see kSpinBeforeSleep
spin_before_sleep_ = std::chrono::microseconds((std::max)(0, std::atoi(e)));
// STRATA_NUMA_SPLIT, read here for the worker count (the split itself is decided below, once the count is known).
@@ -700,10 +759,29 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, PoolAffinity af
bool env_known = auto_pool || (!env_off && (std::strcmp(env, "1") == 0 || std::strcmp(env, "test") == 0));
const PoolPlanInput plan_in = pool_plan_input(topo_);
const PoolPlan wplan = plan_pool_workers(plan_in);
+ // --pool-cores (experimental, off by default): the person's own list of logical processors for the workers. The plan above
+ // was made from the real topology; only the cores the workers are pinned to (and, by default, their count) come from the list.
+ if (!pool_cores_setting().empty()) {
+ const PoolCoresCheck chk = check_pool_cores(pool_cores_setting(), logical_cpu_layout(), topo_.host_core);
+ static std::atomic<bool> told{false}; // (a throw-away pool of --pool-affinity measure builds the same list again)
+ const bool say = !told.exchange(true);
+ if (say)
+ for (const std::string& w : chk.warnings) std::fprintf(stderr, "strata cpu pool: --pool-cores: %s\n", w.c_str());
+ if (chk.cpus.empty()) {
+ if (say)
+ std::fprintf(stderr, "strata cpu pool: --pool-cores: no usable logical processor in the list; the pool uses its "
+ "default placement\n");
+ } else {
+ topo_.worker_cores = chk.cpus;
+ topo_.worker_package.assign(chk.cpus.size(), -1);
+ explicit_cores_ = true;
+ }
+ }
// the split needs workers on both nodes: its cores are taken round-robin over the nodes (the packages when a core's node
// is unknown), so any count under the core count (the default or a configured one) spans them evenly (one node: the order
// is kept)
auto interleave_cores = [&] {
+ if (explicit_cores_) return; // --pool-cores: the person's order stays
if (topo_.worker_package.size() != topo_.worker_cores.size()) return;
std::vector<int> key = topo_.worker_package;
#if defined(__linux__)
@@ -722,6 +800,10 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, PoolAffinity af
// the worker count: configured, else the hybrid P cores, else the plan (`span`: every socket counts, the split's default)
auto pick_workers = [&](bool span, const char** limit) -> int {
if (n_workers > 0) return n_workers;
+ if (explicit_cores_) {
+ *limit = "your --pool-cores list";
+ return (int) topo_.worker_cores.size();
+ }
if (topo_.is_hybrid && affinity_ != PoolAffinity::All) return (std::max)(1, topo_.p_cores - 1);
if (span) {
PoolPlanInput span_in = plan_in;
@@ -841,6 +923,7 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, PoolAffinity af
for (int i = 0; i < n_; ++i) {
const int core = pin ? (i < (int) topo_.worker_cores.size() ? topo_.worker_cores[(size_t) i] : -1) : -1;
threads_.emplace_back([this, i, core] {
+ strata::platform::name_this_thread("strata-pool", i);
strata::aux_cpus::note_owned_thread(); // --aux-cpus never moves a worker
// a failed pin only costs locality under the split: the worker keeps the slice it was given
if (!pin_this_thread(core, i) && core >= 0 && numa_n_ > 0)
@@ -851,6 +934,16 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, PoolAffinity af
}
}
+std::string ExpertPool::placement_line() const {
+ if (n_ <= 0) return "";
+ if (!pinned_) return "CPU workers: " + std::to_string(n_) + " workers, not pinned to logical processors (the OS places them)";
+ // a worker beyond the cores listed runs unpinned (more --pool-workers than --pool-cores entries)
+ std::vector<int> on(topo_.worker_cores.begin(), topo_.worker_cores.begin() + (std::min)((size_t) n_, topo_.worker_cores.size()));
+ std::string s = describe_worker_placement(on, topo_.host_core, logical_cpu_layout(), explicit_cores_);
+ if ((int) on.size() < n_) s += " (" + std::to_string(n_ - (int) on.size()) + " more workers run unpinned)";
+ return s;
+}
+
ExpertPool::~ExpertPool() {
const ExpertPool* self = this;
g_diag_pool.compare_exchange_strong(self, nullptr);
diff --git a/src/kernels/cpu/pool_cores_test.cpp b/src/kernels/cpu/pool_cores_test.cpp
new file mode 100644
index 000000000..02b8d8f77
--- /dev/null
+++ b/src/kernels/cpu/pool_cores_test.cpp
@@ -0,0 +1,123 @@
+// pool_cores_test: where the CPU pool's workers are placed, and --pool-cores (experimental, off by default).
+// Builds real pools (no model, no GPU). The default placement must be what detect_cpu_topology() says (one logical
+// processor per physical core, the host's core left out); a list given to set_pool_cores() must be taken as it is.
+#include "strata/kernels/cpu/pool.hpp"
+
+#include <algorithm>
+#include <cstdio>
+#include <string>
+#include <vector>
+
+namespace cpu = strata::kernels::cpu;
+
+static int g_fail = 0;
+
+static void expect(bool ok, const char* what) {
+ if (!ok) {
+ ++g_fail;
+ std::fprintf(stderr, "FAIL: %s\n", what);
+ }
+}
+
+static std::string join(const std::vector<int>& v) {
+ std::string s;
+ for (size_t i = 0; i < v.size(); ++i) s += (i ? "," : "") + std::to_string(v[i]);
+ return s;
+}
+
+int main() {
+ const std::vector<cpu::LogicalCpuInfo> layout = cpu::logical_cpu_layout();
+ std::printf("pool_cores_test: %zu logical processors\n", layout.size());
+ expect(!layout.empty(), "the system names its logical processors");
+ const cpu::CpuTopology topo = cpu::detect_cpu_topology(true);
+ expect(!topo.worker_cores.empty() || topo.host_core >= 0, "the topology has cores");
+
+ // 1. the default: nothing set, the placement is the topology's, one worker per physical core
+ {
+ cpu::set_pool_cores({});
+ const int n = (int) std::min<size_t>(2, topo.worker_cores.size());
+ if (n >= 1) {
+ cpu::ExpertPool pool(n, true, true);
+ expect(!pool.explicit_cores(), "default: not an explicit list");
+ expect(pool.workers() == n, "default: the worker count asked for");
+ expect(std::equal(topo.worker_cores.begin(), topo.worker_cores.begin() + n, pool.worker_cores().begin()),
+ "default: the workers sit on the topology's cores, in its order");
+ const std::string line = pool.placement_line();
+ std::printf(" default: %s\n", line.c_str());
+ expect(line.rfind("CPU workers on logical processors ", 0) == 0, "the start-up line starts as documented");
+ std::vector<long> cores;
+ bool known = true;
+ for (int i = 0; i < n; ++i) {
+ long c = -1;
+ for (const auto& l : layout) if (l.cpu == pool.worker_cores()[(size_t) i]) c = l.core;
+ known = known && c >= 0;
+ cores.push_back(c);
+ }
+ if (known) {
+ std::sort(cores.begin(), cores.end());
+ expect(std::unique(cores.begin(), cores.end()) == cores.end(), "default: no two workers on one physical core");
+ expect(line.find("one per physical core") != std::string::npos, "the line says one per physical core");
+ } else {
+ expect(line.find("one per physical core") == std::string::npos, "unknown layout: the line does not claim it");
+ }
+ for (const auto& l : layout) if (l.cpu == topo.host_core)
+ for (int i = 0; i < n; ++i)
+ for (const auto& m : layout)
+ if (m.cpu == pool.worker_cores()[(size_t) i] && l.core >= 0)
+ expect(m.core != l.core, "default: no worker on the host's core");
+ }
+ }
+
+ // 2. --pool-cores: the list as given, in its order; the count is the list's length
+ if (topo.worker_cores.size() >= 3) {
+ const std::vector<int> want{topo.worker_cores[2], topo.worker_cores[0]};
+ cpu::set_pool_cores(want);
+ cpu::ExpertPool pool(0, true, true);
+ expect(pool.explicit_cores(), "list: explicit");
+ expect(pool.workers() == 2, "list: the count is the list's length");
+ expect(std::equal(want.begin(), want.end(), pool.worker_cores().begin()), "list: the workers sit on the listed processors, in order");
+ expect(std::string(pool.plan_limit()).find("--pool-cores") != std::string::npos, "list: the start-up line names the list");
+ const std::string line = pool.placement_line();
+ std::printf(" list %s: %s\n", join(want).c_str(), line.c_str());
+ expect(line.find("your --pool-cores list") != std::string::npos, "list: the line says it is the person's list");
+
+ // --pool-workers under the list: the first N entries
+ cpu::ExpertPool one(1, true, true);
+ expect(one.workers() == 1 && one.worker_cores()[0] == want[0], "list + --pool-workers 1: the first entry");
+ }
+
+ // 3. a processor that does not exist is left out (warning), the rest stays; nothing usable falls back to the default
+ if (topo.worker_cores.size() >= 2) {
+ cpu::set_pool_cores({99999, topo.worker_cores[1]});
+ cpu::ExpertPool pool(0, true, true);
+ expect(pool.explicit_cores() && pool.workers() == 1 && pool.worker_cores()[0] == topo.worker_cores[1],
+ "an unusable entry is left out, the usable one kept");
+ cpu::set_pool_cores({99999});
+ cpu::ExpertPool fallback(1, true, true);
+ expect(!fallback.explicit_cores() && fallback.worker_cores()[0] == topo.worker_cores[0],
+ "nothing usable: the default placement");
+ }
+
+ // 4. two SMT siblings of one core are kept (and warned about by check_pool_cores, tested in pool_plan_test)
+ for (const auto& a : layout) {
+ const cpu::LogicalCpuInfo* b = nullptr;
+ for (const auto& c : layout) if (c.cpu != a.cpu && a.core >= 0 && c.core == a.core) { b = &c; break; }
+ if (b == nullptr) continue;
+ cpu::set_pool_cores({a.cpu, b->cpu});
+ cpu::ExpertPool pool(0, true, true);
+ expect(pool.explicit_cores() && pool.workers() == 2 && pool.worker_cores()[0] == a.cpu && pool.worker_cores()[1] == b->cpu,
+ "SMT siblings in the list are kept as given");
+ const std::string line = pool.placement_line();
+ std::printf(" siblings %d,%d: %s\n", a.cpu, b->cpu, line.c_str());
+ expect(line.find("more than one worker") != std::string::npos, "the line says two workers share a core");
+ break;
+ }
+ cpu::set_pool_cores({});
+
+ if (g_fail) {
+ std::fprintf(stderr, "pool_cores_test: %d failure(s)\n", g_fail);
+ return 1;
+ }
+ std::printf("pool_cores_test: OK\n");
+ return 0;
+}
diff --git a/src/kernels/cpu/pool_plan_test.cpp b/src/kernels/cpu/pool_plan_test.cpp
index 3d7add149..55fcc860f 100644
--- a/src/kernels/cpu/pool_plan_test.cpp
+++ b/src/kernels/cpu/pool_plan_test.cpp
@@ -204,6 +204,70 @@ int main() {
expect(per(22, true, 0) == 11 && per(22, true, 1) == 11, "22 (the split default here): 11 + 11 workers");
}
+ // ---- --pool-cores and the start-up line (pure helpers) ----
+ {
+ std::vector<int> v;
+ std::string err;
+ expect(cpu::parse_pool_cores("0,2,4,6", v, err) && v == std::vector<int>({0, 2, 4, 6}), "list: 0,2,4,6");
+ expect(cpu::parse_pool_cores("2-14:2", v, err) && v == std::vector<int>({2, 4, 6, 8, 10, 12, 14}), "list: 2-14:2");
+ expect(cpu::parse_pool_cores("4-6", v, err) && v == std::vector<int>({4, 5, 6}), "list: 4-6");
+ expect(cpu::parse_pool_cores(" 6 , 2 ,2, 0-1 ", v, err) && v == std::vector<int>({6, 2, 0, 1}),
+ "list: spaces ok, the person's order kept, a repeated number once");
+ expect(!cpu::parse_pool_cores("", v, err) && v.empty() && !err.empty(), "list: empty is an error with a message");
+ expect(!cpu::parse_pool_cores("0,,2", v, err) && v.empty(), "list: an empty item");
+ expect(!cpu::parse_pool_cores("0,a", v, err), "list: letters");
+ expect(!cpu::parse_pool_cores("6-2", v, err), "list: a range that goes down");
+ expect(!cpu::parse_pool_cores("0-8:0", v, err), "list: a step of 0");
+ expect(!cpu::parse_pool_cores("-3", v, err), "list: a negative number");
+ expect(cpu::format_cpu_list({2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30}) == "2,4,...,30", "format: a stride");
+ expect(cpu::format_cpu_list({0, 1, 2, 3, 4}) == "0-4", "format: a run");
+ expect(cpu::format_cpu_list({2, 4, 7}) == "2,4,7", "format: short and irregular lists in full");
+ expect(cpu::format_cpu_list({1, 2, 4, 5, 9}) == "1,2,4,5,9", "format: irregular");
+ expect(cpu::format_cpu_list({}) == "none", "format: none");
+
+ // Windows-style numbering of 16 cores / 32 threads: core k = logical 2k and 2k+1
+ std::vector<cpu::LogicalCpuInfo> win;
+ for (int i = 0; i < 32; ++i) win.push_back({i, i / 2});
+ std::vector<int> w;
+ for (int c = 1; c < 16; ++c) w.push_back(2 * c);
+ const std::string line = cpu::describe_worker_placement(w, 0, win, false);
+ std::printf(" %s\n", line.c_str());
+ expect(line == "CPU workers on logical processors 2,4,...,30 (15 workers, one per physical core; host on 0)",
+ "the default line on 16c/32t");
+ // the mistake the person worried about: workers on logical 0..7 of an adjacent-numbering SMT PC
+ const std::string bad = cpu::describe_worker_placement({1, 2, 3, 4, 5, 6, 7}, 0, win, false);
+ std::printf(" %s\n", bad.c_str());
+ expect(bad.find("one per physical core") == std::string::npos && bad.find("carry more than one worker") != std::string::npos,
+ "workers on logical 1..7 of 16c/32t are reported as sharing cores");
+ expect(bad.find("shares the host's core") != std::string::npos, "... and the worker on the host's core is named");
+ const std::string unk = cpu::describe_worker_placement({1, 2, 3}, 0, {}, false);
+ expect(unk.find("did not say") != std::string::npos && unk.find("one per physical core") == std::string::npos,
+ "unknown layout: no claim");
+ expect(cpu::describe_worker_placement({2, 4}, -1, win, true).find("your --pool-cores list") != std::string::npos &&
+ cpu::describe_worker_placement({2, 4}, -1, win, true).find("host not pinned") != std::string::npos,
+ "a given list and an unpinned host are said");
+
+ // check_pool_cores: drop what cannot be used, warn about siblings and the host's processor, never refuse
+ cpu::PoolCoresCheck c = cpu::check_pool_cores({2, 4, 6, 8}, win, 0);
+ expect(c.cpus == std::vector<int>({2, 4, 6, 8}) && c.warnings.empty(), "check: one per core, no warning");
+ c = cpu::check_pool_cores({2, 3, 6}, win, 0);
+ expect(c.cpus == std::vector<int>({2, 3, 6}) && c.warnings.size() == 1 && c.warnings[0].find("two hardware threads") != std::string::npos,
+ "check: SMT siblings are kept with one warning");
+ c = cpu::check_pool_cores({0, 2}, win, 0);
+ expect(c.cpus == std::vector<int>({0, 2}) && c.warnings.size() == 1 && c.warnings[0].find("host thread's") != std::string::npos,
+ "check: the host's processor is kept with a warning");
+ c = cpu::check_pool_cores({1, 2}, win, 0);
+ expect(c.cpus == std::vector<int>({1, 2}) && c.warnings.size() == 1 && c.warnings[0].find("shares a core with the host") != std::string::npos,
+ "check: the host's SMT sibling is kept with a warning");
+ c = cpu::check_pool_cores({2, 40, 99}, win, 0);
+ expect(c.cpus == std::vector<int>({2}) && c.warnings.size() == 1 && c.warnings[0].find("40,99") != std::string::npos,
+ "check: processors this process may not use are left out and named");
+ c = cpu::check_pool_cores({40}, win, 0);
+ expect(c.cpus.empty() && c.warnings.size() == 1, "check: nothing usable");
+ c = cpu::check_pool_cores({2, 3}, {{2, -1}, {3, -1}}, 0);
+ expect(c.cpus.size() == 2 && c.warnings.empty(), "check: an unknown layout cannot tell siblings: no warning");
+ }
+
if (g_fail) {
std::fprintf(stderr, "pool_plan_test: %d failure(s)\n", g_fail);
return 1;
diff --git a/src/ngram/ple_reader.cpp b/src/ngram/ple_reader.cpp
index 9d2a05051..97b45ac45 100644
--- a/src/ngram/ple_reader.cpp
+++ b/src/ngram/ple_reader.cpp
@@ -364,7 +364,7 @@ bool PleReader::open(const std::string& path, uint64_t table_offset, uint64_t n_
impl_->threaded = io_thread;
if (io_thread) {
try {
- impl_->worker = std::thread([this] { strata::aux_cpus::pin_current_thread(); impl_->worker_loop(); });
+ impl_->worker = std::thread([this] { strata::platform::name_this_thread("strata-ple"); strata::aux_cpus::pin_current_thread(); impl_->worker_loop(); });
} catch (const std::exception& e) {
err = std::string("PleReader: cannot create I/O worker: ") + e.what();
close();
diff --git a/src/platform/direct_file.cpp b/src/platform/direct_file.cpp
index b09536c28..d3a0ede91 100644
--- a/src/platform/direct_file.cpp
+++ b/src/platform/direct_file.cpp
@@ -185,7 +185,7 @@ bool DirectFile::open(const std::string& path, std::string& err) {
// Completions of reads that finish synchronously are still queued to the port, so every issued read
// produces exactly one packet; `wait` is the only completion path.
const int n = io_threads(4);
- for (int i = 0; i < n; ++i) impl_->pool.emplace_back([this] { strata::aux_cpus::pin_current_thread(); impl_->worker(); });
+ for (int i = 0; i < n; ++i) impl_->pool.emplace_back([this] { strata::platform::name_this_thread("strata-dio"); strata::aux_cpus::pin_current_thread(); impl_->worker(); });
return true;
}
@@ -317,7 +317,7 @@ bool DirectFile::open(const std::string& path, std::string& err) {
if (fstat(impl_->fd, &st) != 0) { err = "DirectFile: cannot size " + path; close(); return false; }
impl_->size = (uint64_t) st.st_size;
const int n = io_threads(16);
- for (int i = 0; i < n; ++i) impl_->pool.emplace_back([this] { strata::aux_cpus::pin_current_thread(); impl_->worker(); });
+ for (int i = 0; i < n; ++i) impl_->pool.emplace_back([this] { strata::platform::name_this_thread("strata-dio"); strata::aux_cpus::pin_current_thread(); impl_->worker(); });
return true;
}
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index 14b23bccb..4505a9ab8 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -383,6 +383,7 @@ struct HandWorker {
std::thread th;
HandWorker() {
th = std::thread([this] {
+ strata::platform::name_this_thread("strata-hand");
strata::aux_cpus::pin_current_thread();
for (;;) {
std::packaged_task<bool()> job;
@@ -569,6 +570,7 @@ struct Stager {
threads.emplace_back([this] {
// --aux-cpus: the copy threads go to the spare CPUs with the other helpers; STRATA_AUX_STAGER=0 leaves them where
// they were created (a small spare set would then hold all of them: it limits the copy rate of a long read)
+ strata::platform::name_this_thread("strata-stage");
static const bool keep = [] { const char* v = std::getenv("STRATA_AUX_STAGER"); return v != nullptr && std::atoi(v) == 0; }();
if (keep) strata::aux_cpus::note_owned_thread();
else strata::aux_cpus::pin_current_thread();
@@ -2996,6 +2998,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
const bool threaded_issue = stream_all && issuer_on;
if (threaded_issue) {
issuer = std::thread([&] {
+ strata::platform::name_this_thread("strata-issue");
strata::aux_cpus::pin_current_thread();
const core::OnDevice od(m.device);
for (size_t idx = 0; idx < seq.size(); ++idx) {
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index b30d45c08..88bd21d75 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -518,6 +518,8 @@ struct Options {
bool pool_measure = false;
/// --host-core first|last|sibling (STRATA_HOST_CORE): the host thread's core (see HostCore in pool.hpp)
std::string host_core;
+ /// --pool-cores LIST (STRATA_POOL_CORES; experimental, off by default): the logical processors the pool's workers are pinned to
+ std::string pool_cores;
/// --aux-cpus off|auto|LIST (STRATA_AUX_CPUS): where the non-pool, non-host threads run (strata/platform/aux_cpus.hpp)
std::string aux_cpus;
/// R2.2's first half, as an A/B arm. **ON by default**, because the measurement that justifies it is the
@@ -1015,6 +1017,13 @@ void usage() {
" result. Not on hybrid CPUs. sibling: the first core's other hardware thread\n"
" (SMT), so the interrupts keep the first logical processor and the workers keep\n"
" every core; hybrid CPUs too, first without SMT (STRATA_HOST_CORE sets it too).\n"
+ " --pool-cores LIST Experimental, off by default: pin the CPU workers to exactly these logical\n"
+ " processors (0,2,4,6 or 2-14:2; the numbers Task Manager and lscpu show). The\n"
+ " default is one worker per physical core, which the start-up line names. Two\n"
+ " entries on one core, or the host's own, are kept with a warning; processors the\n"
+ " process may not use are left out. The worker count is the list's length unless\n"
+ " --pool-workers says otherwise. Moves threads only, never a result.\n"
+ " STRATA_POOL_CORES sets it too.\n"
" --aux-cpus SET Where the threads that are neither pool workers nor the host thread run (the\n"
" adaptive tier's job thread, the prefill helpers, the CUDA driver's threads, ...):\n"
" off (the default: wherever they were created), auto (the SMT siblings of the host's\n"
@@ -1641,6 +1650,7 @@ private:
// the job's memcpys stay off the host's core and the pool's: by default the spare SMT sibling of the last
// two-thread core (Windows); STRATA_ADAPT_JOB_CPU=<logical cpu>, or -1 for no pinning. Without either, --aux-cpus
// (off by default) puts the thread on the spare CPUs; an explicit STRATA_ADAPT_JOB_CPU is the user's word over it.
+ strata::platform::name_this_thread("strata-adapt-job");
const char* c = std::getenv("STRATA_ADAPT_JOB_CPU");
const int cpu = c != nullptr ? std::atoi(c) : smt_spare_cpu();
if (cpu >= 0) (void) strata::kernels::cpu::pin_current_thread(cpu);
@@ -1806,6 +1816,7 @@ private:
#endif
}
void loop(size_t k) {
+ strata::platform::name_this_thread("strata-ckpt", (int) k);
spread_current_thread();
uint64_t seen = 0;
std::unique_lock<std::mutex> lk(mu_);
@@ -2469,6 +2480,7 @@ int main(int argc, char** argv) {
return 2;
}
}
+ else if (a == "--pool-cores") o.pool_cores = next("--pool-cores");
else if (a == "--aux-cpus") {
o.aux_cpus = next("--aux-cpus");
std::string aerr;
@@ -2818,6 +2830,18 @@ int main(int argc, char** argv) {
if (hc == "last") strata::kernels::cpu::set_host_core(strata::kernels::cpu::HostCore::Last);
if (hc == "sibling") strata::kernels::cpu::set_host_core(strata::kernels::cpu::HostCore::Sibling);
}
+ { // --pool-cores / STRATA_POOL_CORES (experimental, off by default), before the pool is built. A list that cannot be read is
+ // said and left out: the pool keeps its default placement (recommend, never force).
+ std::string pc = o.pool_cores;
+ if (pc.empty())
+ if (const char* e = std::getenv("STRATA_POOL_CORES")) pc = e;
+ if (!pc.empty()) {
+ std::vector<int> cpus;
+ std::string perr;
+ if (strata::kernels::cpu::parse_pool_cores(pc, cpus, perr)) strata::kernels::cpu::set_pool_cores(cpus);
+ else std::fprintf(stderr, "strata generate: --pool-cores: %s; the pool uses its default placement\n", perr.c_str());
+ }
+ }
{ // --aux-cpus / STRATA_AUX_CPUS: read the CPUs this process may use now, before any thread is pinned; the plan
// follows once the pool exists (it needs the workers' CPUs)
std::string ac = o.aux_cpus;
@@ -5846,6 +5870,8 @@ int main(int argc, char** argv) {
strata::kernels::cpu::host_core_setting() == strata::kernels::cpu::HostCore::Last ? "last"
: strata::kernels::cpu::host_core_setting() == strata::kernels::cpu::HostCore::Sibling ? "sibling"
: "first");
+ // the same facts in plain words, and whether each worker has a physical core of its own
+ std::fprintf(stderr, "strata generate: %s\n", pool.placement_line().c_str());
// --aux-cpus: the plan, now the workers' CPUs are known; the threads that exist already (the CUDA driver's) move
// now when it is on, the ones created later pin themselves (strata/platform/aux_cpus.hpp)
const size_t nw = std::min<size_t>((size_t) pool.workers(), wc.size());
@@ -10692,6 +10718,7 @@ int main(int argc, char** argv) {
std::deque<std::string>& in_lines = in_shared->lines;
bool& in_eof = in_shared->eof;
std::thread([st = in_shared] {
+ strata::platform::name_this_thread("strata-stdin");
strata::aux_cpus::pin_current_thread();
// read(2) on the descriptor, not std::cin: glibc's exit() flushes every stdio stream and waits for
// stdin's lock, which getline holds while it waits for input - an engine ending on an error (every
@@ -10884,6 +10911,7 @@ int main(int argc, char** argv) {
const int io_limit = wio ? std::atoi(wio) : limit * 10;
if (limit > 0)
std::thread([limit, io_limit] {
+ strata::platform::name_this_thread("strata-watchdog");
strata::aux_cpus::pin_current_thread();
strata::core::Progress& p = strata::core::progress();
uint64_t last = p.beats.load(), ticks_at = p.ticks.load();
@@ -14195,7 +14223,7 @@ int main(int argc, char** argv) {
if (!drive.d.usage.empty() && o.adapt_every > 0 && (rounds % o.adapt_every) == 0 && pending.empty() &&
!exch_wait) {
pl_adapt_done = false;
- pl_adapt_thr = std::thread([&] { strata::aux_cpus::pin_current_thread(); pl_adapt_ok = adapt(); pl_adapt_done = true; });
+ pl_adapt_thr = std::thread([&] { strata::platform::name_this_thread("strata-adapt"); strata::aux_cpus::pin_current_thread(); pl_adapt_ok = adapt(); pl_adapt_done = true; });
}
return true;
};
@@ -14862,7 +14890,7 @@ int main(int argc, char** argv) {
bool adapt_ok = true;
// (--adapt-async 1: the asynchronous tier above instead, ticked before the window)
if (!drive.d.usage.empty() && !ajob && ((rounds + 1) % o.adapt_every) == 0)
- adapt_thr = std::thread([&] { strata::aux_cpus::pin_current_thread(); adapt_ok = adapt(); });
+ adapt_thr = std::thread([&] { strata::platform::name_this_thread("strata-adapt"); strata::aux_cpus::pin_current_thread(); adapt_ok = adapt(); });
if (!ver.commit(a + 1, err)) {
if (adapt_thr.joinable()) adapt_thr.join();
std::printf("ERR %s\n", err.c_str());
@@ -16240,7 +16268,7 @@ int main(int argc, char** argv) {
std::thread adapt_thr;
bool adapt_ok = true;
if (!drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0)
- adapt_thr = std::thread([&] { strata::aux_cpus::pin_current_thread(); adapt_ok = adapt(); });
+ adapt_thr = std::thread([&] { strata::platform::name_this_thread("strata-adapt"); strata::aux_cpus::pin_current_thread(); adapt_ok = adapt(); });
if (!ver.commit(a + 1, err)) {
if (adapt_thr.joinable()) adapt_thr.join();
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
diff --git a/sycl/src/program/generate.cpp b/sycl/src/program/generate.cpp
index a03c11f11..285066e8f 100644
--- a/sycl/src/program/generate.cpp
+++ b/sycl/src/program/generate.cpp
@@ -502,6 +502,8 @@ struct Options {
strata::kernels::cpu::PoolAffinity pool_affinity = strata::kernels::cpu::PoolAffinity::All;
/// --host-core first|last (STRATA_HOST_CORE): the host thread's core (see HostCore in pool.hpp)
std::string host_core;
+ /// --pool-cores LIST (STRATA_POOL_CORES; experimental, off by default): the logical processors the pool workers are pinned to
+ std::string pool_cores;
/// R2.2's first half, as an A/B arm. **ON by default**, because the measurement that justifies it is the
/// pool's own drain: 33.7 GB/s against 5/6 x 44.14 = 36.8 for five workers, on a machine whose sixth core
/// is reserved for a host thread that has nothing to do while the drain runs.
@@ -930,6 +932,9 @@ void usage() {
" interrupts to one logical processor, usually the first, and every copy that lands\n"
" raises one: a spinning host there waits for them. Moves threads only, never a\n"
" result. Not on hybrid CPUs (STRATA_HOST_CORE sets it too).\n"
+ " --pool-cores LIST Experimental, off by default: pin the CPU workers to exactly these logical\n"
+ " processors (0,2,4,6 or 2-14:2). Two entries on one core, or the host's own, are\n"
+ " kept with a warning. STRATA_POOL_CORES sets it too.\n"
" --pool-affinity MODE Worker CPU affinity: all (default: one worker per physical core, as\n"
" always), auto (hybrid CPUs: P-cores first, then their SMT siblings,\n"
" then E-cores) or p-cores (P-cores and their siblings only).\n"
@@ -1834,6 +1839,7 @@ int main(int argc, char **argv) try {
else if (a == "--graph-only") o.graph_only = true;
else if (a == "--gpu-only-full") o.gpu_only_full = true;
else if (a == "--pool-workers") o.pool_workers = std::atoi(next("--pool-workers"));
+ else if (a == "--pool-cores") o.pool_cores = next("--pool-cores");
else if (a == "--host-core") {
o.host_core = next("--host-core");
if (o.host_core != "first" && o.host_core != "last") {
@@ -2070,6 +2076,18 @@ int main(int argc, char **argv) try {
if (const char* e = std::getenv("STRATA_HOST_CORE")) hc = e;
if (hc == "last") strata::kernels::cpu::set_host_core(strata::kernels::cpu::HostCore::Last);
}
+ { // --pool-cores / STRATA_POOL_CORES (experimental, off by default), before the pool is built; a list that cannot be read is
+ // said and left out (recommend, never force)
+ std::string pc = o.pool_cores;
+ if (pc.empty())
+ if (const char* e = std::getenv("STRATA_POOL_CORES")) pc = e;
+ if (!pc.empty()) {
+ std::vector<int> cpus;
+ std::string perr;
+ if (strata::kernels::cpu::parse_pool_cores(pc, cpus, perr)) strata::kernels::cpu::set_pool_cores(cpus);
+ else std::fprintf(stderr, "strata generate: --pool-cores: %s; the pool uses its default placement\n", perr.c_str());
+ }
+ }
strata::core::set_peer_portable(o.peer_device >= 1); // multi-GPU: the Portable flag on mapped host buffers only with a peer device (before any allocation)
if (o.serve && o.conversation_cache_mib > 0 && (o.prompt_cache == 0 || o.conversation_cache_slots == 0))
std::fprintf(stderr, "strata serve: warning: conversation caching is disabled by %s\n",
@@ -4428,6 +4446,8 @@ int main(int argc, char **argv) try {
"(--host-core %s)\n", pool.workers(), on.c_str(), ht.host_core,
pool.host_works() ? " (draining too)" : "",
strata::kernels::cpu::host_core_setting() == strata::kernels::cpu::HostCore::Last ? "last" : "first");
+ // the same facts in plain words, and whether each worker has a physical core of its own
+ std::fprintf(stderr, "strata generate: %s\n", pool.placement_line().c_str());
}
if (o.no_ple_prefetch) strata::kernels::ple_prefetch_enable(false);
// ---- R4's slot storage. Allocated AFTER the weights and the session, so `cudaMemGetInfo` inside `open`