Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge perf-PR group batch-1441 (f-145-pp-batch-1441)
Niko1221da6d8d920577
diff --cc docs/MULTI_GPU.mdindex 6d2c8dc9b,40c7178cf..30e316d52--- a/docs/MULTI_GPU.md+++ b/docs/MULTI_GPU.md@@@ -215,33 -199,75 +215,102 @@@ that overlap instead of one, a 32K one32K +18% / +21% / +29%, 64K +4% / +12% / +17%, 8K unchanged. `STRATA_SPLIT_CHUNK16K=1` keeps 16384.) Conversation checkpoints save and restore every card's state; the adaptive expert swaps copyinto the card that owns the layer.+## Overlapped hand-off (`STRATA_SPLIT_OVERLAP=1`, opt-in)++By default each verify window runs the stages one after another: the host syncs stage 1, then stages and launches+stage 2's graph, and so on. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives the hand-off a ready+flag: the earlier stage's GPU writes the hand-off with volatile stores and raises the flag (`handoff_publish`), the+next stage's graph is staged and launched on a helper thread while the earlier stage is still running and waits for+the flag on its GPU (`wait_flag_ge`), and the host goes straight from serving one stage's layers to the next, syncing+the earlier stage only once the chain is done. It takes one stream sync and one graph launch per window off the+critical path. Batch windows (several conversations) keep the serial order. It is off beside `--pipeline-windows`, whose loop overlaps the stages its own way.++Measured on 2x RX 7900 XTX (gfx1100, ROCm, Linux), Ryzen 9 9900X, 52 GB RAM, Swift 1.5 IQ3_XXS, 128K context, 8-bit+KV, layers 0-25 / 26-47; 512-token greedy answers, median of 8 runs:++| | serial (default) | `STRATA_SPLIT_OVERLAP=1` | one card |+|---|---|---|---|+| decode, code prompt | 53 tok/s | 70.1 tok/s | ~70 tok/s |+| decode, prose prompt | 45 tok/s | 55.9 tok/s | ~68 tok/s |++Greedy output is byte-identical with the flag at 0 and 1; the quiz set scored the same (47/51). Not measured on+NVIDIA yet, where a sync and a graph launch cost less.++The reading stage's wait for the flag is bounded on the GPU: `STRATA_SPLIT_WAIT_MS` (default 30000, 0 = no bound).+If the flag does not arrive in time, the wait marks an error word and returns, the window's graph completes, and the+host fails that window with an error naming the setting (the server then restarts the engine) instead of using a+stale hand-off. Normally the host's own per-layer wait (20 s, #267) ends a stalled window first and releases every+GPU wait, the hand-off's included. Test hook: `STRATA_TEST_HANDOFF_DROP=N` withholds the flag in the first stage's+N-th window, as if the publish were lost.+ ## Prompt chunks sized for the pipeline (`--prefill-pipe`, opt-in; setup's `--calibrate` sets it)++ `--prefill auto` takes the largest chunk the buffers allow (up to 8192), which is what one card wants: a chunk costs a+ fixed pass over its layers' experts (b: dequantized, or streamed over PCIe where they are not in VRAM) plus a part per+ token (a), so few large chunks are cheapest. A layer split reads a prompt as a pipeline - stage s reads chunk c while+ stage s+1 reads chunk c-1 - and then takes about sum_i (b + a C_i) + (S - 1) max_i (b + a C_i) for chunks C_i over S+ stages: large or uneven chunks keep the later cards waiting (a 10.7K-token prompt in a 7680 and a 3011 chunk over+ four stages: the first stage carried 24 of the 36 s), small ones pay b again and again.++ `--prefill-pipe R` reads each prompt in the chunk count with the least pipeline time for b/a = R tokens, when that+ beats today's split by 3% or more: every count from the fewest the buffers allow is tried, each chunk evened out on the+ 256-token grid (at least 512, at most the buffers' chunk; keep `--prefill auto` so long prompts may use large chunks).+ b/a differs by an order of magnitude between rigs - about 700 tokens on four GP100 dies (every expert in VRAM, slow+ arithmetic), about 5000 on two RTX 3090 Ti (part of the experts streamed per chunk, fast arithmetic) - and a wrong one+ costs: a fixed 1024 read 19-24% slower on the two RTX 3090 Ti and 22-36% slower on two RTX 4090 (reported in the PR by+ MistyMoonR). So it is neither guessed nor learned from the prompts the engine serves (that made a prompt's chunks, and+ so its output bits, depend on how busy the PC happened to be): like the PCIe share and the draft floor, it is a+ setting of the PC that setup's `--calibrate` measures once (`tools/calibrate.py`: on a layer split, the prompt read+ speed of a 2K- and an 8K-token prompt - the same text every time, after a header of its own so nothing is reused -+ with the chunks off, evened out and b/a 768, 1536, 3072 and 6144; the best is kept only when it reads faster than off+ in each of three paired rounds and by more than 3% in the median) and writes into the config. Measured so: four GP100+ dies keep b/a 768 (355-356 against 219-220 tok/s in every round, +62%); two RTX 3090 Ti keep it off (nothing read+ faster than off, b/a 768 20% slower). A first version that read a different stretch of text per value and confirmed in+ two rounds picked 1536 on the RTX 3090 Ti pair from one fast read - where the experts are streamed, the read speed+ depends on the text. `--prefill-pipe 1` needs no+ b/a: it is `STRATA_PREFILL_EQUAL=1`'s rule (#693) - today's chunk count, evened out on the 256-token grid, and only+ when today's last chunk has 1024 tokens or more (3072 with `STRATA_PREFILL_CPU_SHARE` set). A chunk that size streams+ every expert its card does not hold, a shorter one only the experts it routes to, so a prompt just past the chunk size+ reads as one full chunk and a nearly free tail, and evening that out would add a second full pass over the experts:+ the first version evened every split out, and on two RTX 4090 MistyMoonR's calibration read 1,601 tok/s with it+ against 2,129 off. 0 (the default) is today's split;+ `STRATA_PREFILL_PIPE=<R>` is the same as the flag where the config cannot carry it. The chunk geometry changes the+ rounding, which is why it is opt-in.++ A prompt segment (the prompt between two of its cuts: the system-prompt root, the last turn, a `pin=`) is planned once,+ on its whole length and R, and every run of it reads in that chunk. So its chunks, and the loan its buffers are laid+ out for, are the same whether it is read in one run, in pieces between the windows of decoding `--batch` slots, or goes+ on after a `BYIELD` or from a periodic checkpoint taken inside it (the slot and the checkpoint keep the plan; the read+ goes on in it when it continues the same segment, else it plans again and says so in the log). The output bits then+ match as far as the experts the cards hold do (`--adapt-swaps 0` for byte-identical repeats, as without the pipe).+ Before this, a read beside decoding slots re-planned every piece, and one that went on after a `BYIELD` re-planned what+ was left. A chunk the pipe planned takes no `STRATA_PREFILL_CPU_SHARE`: the stages read+ such chunks at the same time, and which stage got the CPU pool first would decide which experts the CPU computed. The+ DONE line ends with `chunk=` (the first chunk of the request's longest batched segment) and `experts_streamed=` (the+ expert blobs its prompt copied to the cards), and the INFO line carries `prefill_cap`, `prefill_stages`,+ `stream_all_min` and `prefill_pipe`. One line in the log says what a plan chose:++ strata prefill: 10058 tokens in 1280-token chunks over 4 stages (--prefill-pipe 706: b/a)++ Four GP100 dies (PH402 SKU 200, `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context), prompt tok/s, fresh+ prompts, two interleaved rounds (each within 0.2%), with b/a 1024 (the same chunks as 700-890, measured on this rig):++ | Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill auto` + `--prefill-pipe 1024` (chunk) |+ | ---: | ---: | ---: | ---: |+ | 2,013 | 180.9 | 181.4 | 228.0 (1024) |+ | 3,865 | 207.6 | 276.5 | 312.4 (1024) |+ | 10,041 | 261.1 | 433.0 | 432.6 (2048) |+ | 28,998 | 475.7 | 554.1 | 559.7 (3328) |++ Two RTX 3090 Ti (CUDA 12.9 build for sm_86, two stages, Swift 1.5 IQ3_XXS, 1M context, 13,371 of 24,576 experts in+ VRAM), prompt tok/s, fresh prompts, two starts per arm, each reading the four prompts twice (the cold first pass after+ a card was freed is left out); b/a 5836 and 6432 (timed on the rig), against 1024:++ | Prompt tokens | `--prefill auto` | + b/a 5836 / 6432 (chunk) | for comparison: b/a 1024 |+ | ---: | ---: | ---: | ---: |+ | 2,035 | 694-815 | 698-816 (today's) | 560-562 |+ | 3,887 | 1,127-1,207 | 1,115-1,227 (today's) | 880-885 |+ | 10,063 | 1,437-1,553 | 1,654-1,681 (2 x 5120) | 1,109-1,113 |+ | 29,019 | 2,532-2,603 | 2,566-2,620 (today's) | 2,064-2,089 |## Limits (for now)diff --cc include/strata/prefill/prefill.hppindex 197dc25fe,79d23a4ac..6c4260824--- a/include/strata/prefill/prefill.hpp+++ b/include/strata/prefill/prefill.hpp@@@ -217,13 -233,11 +250,16 @@@ privatePrefill* next_ = nullptr;Prefill* helper_ = nullptr; ///< set_stage_helperbool single_chunk_ = false; ///< a later stage: the prompt is one chunk (set by the stage before)+ int64_t stopped_at_ = -1; ///< stop_at_chunk ended the last run after this many tokens (-1: it read all)+ bool pipe_chunked_ = false; ///< a later stage: the chunks came from a --prefill-pipe plan (set likewise)+ int64_t force_chunk_ = 0; ///< force_chunk, for the next run (0: none)+ bool force_single_ = false, force_planned_ = false;bool bind_stage_helper(int64_t T); // binds the helper's buffers for a one-chunk prompt of T tokensconst float* hand_in_ = nullptr; ///< the previous stage's rows of the chunk being read (host, pinned)+ /// STRATA_HANDOFF_PIPE: the previous stage's per-piece "landed in hand_in_" events (cudaEvent_t[hand_pieces_],+ /// on its device) when it copies the chunk in pieces without a host wait; null with the one-copy hand-off+ const void* hand_ev_ = nullptr;+ int hand_pieces_ = 0;std::string next_err_;std::future<bool> next_run_;diff --cc src/prefill/prefill.cppindex 3f8409212,a96753e2b..1bdee6772--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@@ -2352,17 -2295,10 +2472,17 @@@ bool Prefill::run_impl(const int64_t* tstd::future<bool> ple_next; // declared after everything it reads: an early return waits for itint ple_buf = 0;+ if (hand_in_ == nullptr) stopped_at_ = -1;- for (int64_t c0 = 0; c0 < n; c0 += m.T) {+ for (int64_t c0 = 0; c0 < n; c0 += C) {if (should_stop && should_stop()) { err = "cancelled"; return false; }+ // the first stage: end the run before this chunk (every chunk before it is handed on; the caller's drain+ // waits for the later stages to read them)+ if (c0 > 0 && hand_in_ == nullptr && stop_at_chunk && stop_at_chunk()) {+ stopped_at_ = c0;+ break;+ }if (std::getenv("STRATA_TRACE")) { std::fprintf(stderr, "strata trace: prompt chunk %lld of %lld\n", (long long) c0, (long long) n); std::fflush(stderr); }- const int64_t T = std::min(m.T, n - c0), p0 = pos0 + c0;+ const int64_t T = std::min(C, n - c0), p0 = pos0 + c0;core::progress_at("reading the prompt (batched): preparing the chunk from token", p0); // #251++stats_.chunks;pt.mark(kPfStart, cs);@@@ -4473,17 -4289,8 +4593,18 @@@if (next_run_.valid() && !next_run_.get()) { err = next_err_; return false; }next_err_.clear();next_->hand_in_ = h;+ next_->hand_ev_ = hp > 0 ? static_cast<const void*>(m.hand_ev[hand_buf_]) : nullptr;+ next_->hand_pieces_ = hp;next_->single_chunk_ = single_chunk;+ next_->pipe_chunked_ = pipe_chunked;+ if (handoff_pipe_worker()) {+ if (!m.hand_worker) m.hand_worker = std::make_unique<HandWorker>();+ next_run_ = m.hand_worker->submit(std::packaged_task<bool()>([this, tokens, c0, T, p0] {+ const bool r = next_->run_impl(tokens + c0, T, p0, next_err_);+ (void) cudaGetLastError(); // clear what a thread's end cleared (std::async)+ return r;+ }));+ } elsenext_run_ = std::async(std::launch::async, [this, tokens, c0, T, p0] {strata::aux_cpus::pin_current_thread();return next_->run_impl(tokens + c0, T, p0, next_err_);diff --cc src/program/generate.cppindex 09bdef8e7,ae76d8567..d4ea3891c--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@@ -8597,11 -8142,11 +8609,14 @@@ int main(int argc, char** argv)// #879: a window met non-finite logits: the next request drops every cached state first and reads from token 0bool nan_flush = false;std::vector<ConvCheckpoint> checks;+ // STRATA_CKPT_POOL: the dropped checkpoints' storage, reused. STRATA_SLOT_CKPT_MOVE turns it on too: a moved+ // checkpoint frees nothing at the next prune, so every save would fault in fresh pages (17 -> 40-60 ms)+ CheckpointPool ckpt_pool(ckpt_pool_env() || slot_ckpt_move_env());uint64_t check_clock = 0; // the checkpoints' LRU clock; creation and every use advance itint64_t tail_ckpt_len = -1; // --prompt-cache-tail: the length of the one tail checkpoint alive (-1 = none)+ // --prefill-pipe: the plan of the prompt segment being read (read_part sets it around its runs), for the+ // checkpoints taken inside it+ ConvCheckpoint::PromptPlan seg_plan;bool cvec_cached = true; // the control vector's state the live session and the checkpoints were read with// without --mtp the conversation cache stays on: a parked image carries the draft layer's K/V only when there is one// (the snapshots accept a null draft)@@@ -9027,13 -8481,12 +9042,14 @@@}}c.used = ++check_clock;+ if (seg_plan.chunk > 0 && L > seg_plan.from && L < seg_plan.to) c.plan = seg_plan; // inside a planned segmentif (as_tail) { // only one tail checkpoint stays alive: the previous request's goesconst int64_t old_tail = tail_ckpt_len;- checks.erase(std::remove_if(checks.begin(), checks.end(), [&](const ConvCheckpoint& k) {- return old_tail >= 0 && (int64_t) k.ids.size() == old_tail;- }), checks.end());+ auto old = [&](const ConvCheckpoint& k) { return old_tail >= 0 && (int64_t) k.ids.size() == old_tail; };+ if (ckpt_pool.on())+ ckpt_pool.erase_if(checks, old);+ else+ checks.erase(std::remove_if(checks.begin(), checks.end(), old), checks.end());tail_ckpt_len = L;} else if (L == tail_ckpt_len) {tail_ckpt_len = -1; // the flagged tail was dropped and a normal checkpoint takes its length@@@ -11927,21 -11226,29 +11962,43 @@@std::printf("ERR restoring a conversation checkpoint failed\n");return 1;}+ if (lent && lent_slot == admit_slot) {+ // the slot this request continues in drops its checkpoints at the admission: this one moves back+ // into the chain (and on into the slot then: moved_in)+ std::vector<ConvCheckpoint>& sc = bs[(size_t) lent_slot].checks;+ const auto it = sc.begin() + (c - sc.data());+ checks.push_back(std::move(*it));+ sc.erase(it);+ moved_in = resume;+ lent_slot = -1;+ }+ if (lent && trace)+ std::fprintf(stderr, "strata trace: the checkpoint at %lld restored from slot %d's%s\n",+ (long long) resume, lent_slot >= 0 ? lent_slot : admit_slot,+ lent_slot >= 0 ? "" : " (moved back into the chain)");}+ // --prefill-pipe: going on from a periodic checkpoint taken inside a planned segment (whatever restored it:+ // a checkpoint of this session, of a slot or of a parked conversation), the rest of that segment is read+ // in its chunks. STRATA_CKPT_REREAD reads [0, resume) again in the chunks the saving read had, which it+ // can only when that segment started at 0. The BYIELD park set resume_plan above.+ if (resume_plan.chunk <= 0 && resume > 0 && !stages.empty())+ for (const ConvCheckpoint& k : checks)+ if ((int64_t) k.ids.size() == resume && k.plan.chunk > 0) {+ if (reread_to <= 0) {+ resume_plan = k.plan;+ resume_plan_at = resume;+ resume_plan_end = k.plan.to;+ } else if (k.plan.from == 0) { // [0, resume) only: its end is the checkpoint+ resume_plan = k.plan;+ resume_plan_at = 0;+ resume_plan_end = resume;+ } else {+ std::fprintf(stderr, "strata serve: STRATA_CKPT_REREAD: the checkpoint is inside a "+ "segment that started at %lld, so a read from 0 has other chunks\n",+ (long long) k.plan.from);+ }+ break;+ }// KV streaming: the drafter's ring may hold cells past `resume` from a longer turn; the main layers'// host copies and slots are always current (every writer writes both), so they need nothingif (use_mtp && resume > 0 && reread_to <= 0) mtp.kv_restore(resume);@@@ -12327,377 -11688,17 +12439,383 @@@const char* v = std::getenv("STRATA_BATCH_YIELD_CHUNKS");return v != nullptr ? (int64_t) std::max(0, std::atoi(v)) : (int64_t) 8;}();- const int64_t C = std::max<int64_t>(sp.chunk(), 1);+ // a piece's chunk: the buffers' chunk, or with a plan as many whole planned chunks as that holds (the+ // stages still pipeline inside a piece, and every piece starts on the plan's grid)+ const int64_t C = planned ? std::max<int64_t>(1, sp.chunk() / plan.chunk) * plan.chunk+ : std::max<int64_t>(sp.chunk(), 1);+ const int64_t C_read = planned ? plan.chunk : C; // what the yield test calls a chunk- const int64_t piece = strata::program::batch_read_piece(o.batch > 0, piped, !stages.empty(), batch_on(), C, yield_chunks);+ // BYIELD <ys> with the read stopped at q (nothing in flight): the part read waits in slot ys and the+ // read ends with e = "yield" (true); false when it cannot give way there (the read goes on). Only+ // where the rest is read the same way after it (more than a chunk and more than a short read left),+ // text only, and into a slot that is not decoding.+ auto give_way = [&](int ys, int64_t q, std::string& err) -> bool {+ const bool can = ys < (int) bs.size() && !bs[(size_t) ys].active &&+ (admit_slot < 0 || ys == admit_slot) && req_imgs.empty() && o.prompt_cache > 0 &&- b0 - q > std::max<int64_t>(C, o.short_read);++ b0 - q > std::max<int64_t>(C_read, o.short_read);+ std::string ye;+ const auto ty = Clock::now();+ std::vector<int32_t> pre(ids.begin(), ids.begin() + q);+ if (can && copy_to_slot(ys, pre, ye)) {+ BSlot& sl = bs[(size_t) ys];+ sl = BSlot{};+ sl.ids = std::move(pre);+ sl.cached = true;+ sl.partial = true;+ sl.partial_from0 = read_from == 0 || resumed_from0;++ if (planned) sl.partial_plan = plan; // the resend reads the rest in these chunks+ sl.cvec = cvec_cached;+ for (const ConvCheckpoint& c : checks) // the root / periodic checkpoints of this read+ if ((int64_t) c.ids.size() <= q && std::equal(c.ids.begin(), c.ids.end(), ids.begin(),+ [](int32_t x, int64_t y) { return (int64_t) x == y; }))+ sl.checks.push_back(c);+ yielded_at = q;+ yielded_slot = ys;+ std::fprintf(stderr, "strata batch: the prompt read gives way at %lld of %lld tokens; slot %d "+ "holds it (copied in %.1f ms)\n", (long long) q, (long long) n, ys,+ std::chrono::duration<double, std::milli>(Clock::now() - ty).count());+ err = "yield";+ return true;+ }+ if (can) bs[(size_t) ys].cached = false; // half written+ std::fprintf(stderr, "strata batch: BYIELD %d not taken at %lld of %lld%s%s\n", ys, (long long) q,+ (long long) n, ye.empty() ? "" : ": ", ye.c_str());+ return false;+ };+ // Opt-in, experimental: STRATA_PIPED_CONCURRENT=1 reads the prompt on a thread of its own while+ // this thread keeps the slot groups decoding through the pipeline (pump), so they never stop for it+ // and the read has no drain inside it. The prompt and the windows use their own streams and memory+ // (the prompt's K/V goes to the main sessions, the slots' to theirs; the PLE table is locked), and+ // the adaptive tier stands still (its round landed before the read). The slots the prompt borrowed+ // were lent before this read with nothing in flight and are refilled after it, so the windows see+ // them missing and compute those experts on the CPU, as between pieces. A BYIELD ends the run at+ // its next chunk (Prefill::stop_at_chunk): the pipeline is drained there and the part read gives+ // way as below, or the read goes on from there.+ // Not where a read's thread allocates or frees (the idle stage's help, STRATA_PREFILL_HELP; the batched+ // draft K/V, STRATA_SPLIT_MTP_BATCH: a free waits for the device), where serving a doorbell makes a+ // CUDA call (--pcie-mode dma, the Foresight swap space STRATA_FS_SLOTS, the helper caches), and where+ // the expert source recycles the buffers it hands out (the GGUF in place, an unbuffered source, the+ // I/O read-ahead: one thread at a time). Pictures are read the usual way (their rows go up through+ // pinned slots of a size known only then).+ static const bool pr_conc = [&] {+ const char* v = std::getenv("STRATA_PIPED_CONCURRENT");+ if (v == nullptr || std::atoi(v) == 0) return false;+ auto env_on = [](const char* k) {+ const char* x = std::getenv(k);+ return x != nullptr && std::atoi(x) != 0;+ };+ strata::program::ConcurrentReadSetup s;+ s.prefill_help = env_on("STRATA_PREFILL_HELP");+ s.split_mtp_batch = env_on("STRATA_SPLIT_MTP_BATCH");+ s.fs_slots = env_on("STRATA_FS_SLOTS");+ s.pcie_dma = o.pcie_mode == "dma";+ // (--remote-expert-opt works only through a helper cache, so without one it does not count)+ s.helper_caches = prefill_src != srcp;+ for (int r = 0; r < 3; ++r) s.helper_caches |= o.expert_cache_remote[(size_t) r] > 0;+ s.staged_source = srcp == &src && src.stages_blobs();+ s.other_source = srcp != &src && srcp != &arena_src;+ if (const char* why = strata::program::concurrent_read_off(s)) {+ std::fprintf(stderr, "strata batch: STRATA_PIPED_CONCURRENT is off beside %s\n", why);+ return false;+ }+ return true;+ }();+ if (piped && pr_conc && req_imgs.empty()) { // (with no slot decoding it only watches for a BYIELD)+ // the drafter's prompt pass captures its graphs and grows its records at first use, and the+ // prompt path's pinned upload slots grow at their first larger upload: done here, with nothing in+ // flight (on the reader's thread a capture, an allocation or a free takes the driver's lock and+ // can wait for the device, #31)+ if (use_mtp && !mtp.reserve_prefill(C, e)) return false;+ if (!sp.reserve_uploads(C))+ std::fprintf(stderr, "strata batch: the prompt path's pinned upload slots could not be "+ "reserved (they grow at first use)\n");+ // where the read's thread and the doorbell thread run: on Linux a thread inherits its creator's+ // CPUs, and the host thread is pinned to the one core the pool leaves it, so both would share that+ // core with the host thread that pumps the windows - every window then waits its turn there for+ // the doorbell of each layer (4x RTX 5080, 4 stages: 6 rows/s during a read). They move off the+ // host's core and the pool workers' CPUs instead (--aux-cpus places them, when it is on).+ // STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B); STRATA_TEST_CONC_ON_HOST_CPU=1+ // (Windows, a test) puts them on the host thread's CPUs first, as Linux does.+ static const bool conc_unpin = [] {+ const char* v = std::getenv("STRATA_PIPED_CONCURRENT_UNPIN");+ return v == nullptr || std::atoi(v) != 0;+ }();+#if defined(_WIN32)+ static const bool conc_on_host = std::getenv("STRATA_TEST_CONC_ON_HOST_CPU") != nullptr;+ std::vector<ULONG> host_cpu_sets;+ if (conc_on_host) {+ const HANDLE self = GetCurrentThread();+ ULONG count = 0;+ if (GetThreadSelectedCpuSets(self, nullptr, 0, &count) ||+ GetLastError() == ERROR_INSUFFICIENT_BUFFER) {+ host_cpu_sets.resize(count);+ if (count && !GetThreadSelectedCpuSets(self, host_cpu_sets.data(), count, &count))+ host_cpu_sets.clear();+ }+ static bool said = false;+ if (!said) {+ said = true;+ std::fprintf(stderr, "strata batch: test: the read's and the doorbell threads start on the "+ "host thread's %zu CPU set(s)%s\n", host_cpu_sets.size(),+ conc_unpin ? ", then leave them (the fix)"+ : " and stay there (as on Linux)");+ }+ }+#endif+ // the threads that already inherited the host's core (the prompt path's copy threads, started by+ // the first prompt on the host thread) work during the read too: off that core as well (Linux)+ if (conc_unpin && !strata::aux_cpus::enabled())+ if (const int moved = strata::aux_cpus::release_host_inheritors(); moved > 0)+ std::fprintf(stderr, "strata batch: %d threads that ran on the host thread's core moved off "+ "it, for reads beside the slots\n", moved);+ auto conc_thread_cpus = [&] {+ strata::aux_cpus::pin_current_thread();+#if defined(_WIN32)+ const HANDLE self = GetCurrentThread();+ if (conc_on_host && !host_cpu_sets.empty())+ SetThreadSelectedCpuSets(self, host_cpu_sets.data(), (ULONG) host_cpu_sets.size());+ if (conc_unpin && conc_on_host) SetThreadSelectedCpuSets(self, nullptr, 0);+#else+ if (conc_unpin && !strata::aux_cpus::enabled()) (void) strata::aux_cpus::place_off_host();+#endif+ };+ // the slots' doorbells are served on a thread of their own for the read, which makes no CUDA+ // call: a call here or on the read's threads can wait for the driver's lock while its holder+ // waits for the GPU (a free; a module the first kernel of a shape loads), and a window on that+ // GPU waits for its doorbell (#31). Ends (and hands the service back to pump) on every exit.+ struct Doorbells {+ std::atomic<bool> on{true};+ std::thread th;+ std::function<void()> wake, off;+ ~Doorbells() {+ on.store(false, std::memory_order_release);+ if (wake) wake();+ if (th.joinable()) th.join();+ if (off) off();+ }+ } bells;+ for (int k = 0; k < n_pipe; ++k) stage_verifier(k).set_external_serve(true);+ conc_serving = true;+ bells.off = [&] {+ for (int k = 0; k < n_pipe; ++k) stage_verifier(k).set_external_serve(false);+ conc_serving = false;+ };+ bells.wake = [&] {+ std::lock_guard<std::mutex> lk(bell_mu);+ bell_cv.notify_one();+ };+ bells.th = std::thread([&] {+ conc_thread_cpus();+ auto any_busy = [&] {+ for (int k = 0; k < n_pipe; ++k)+ if (stage_verifier(k).batch_busy()) return true;+ return false;+ };+ while (bells.on.load(std::memory_order_acquire)) {+ bool busy = false;+ for (int k = 0; k < n_pipe; ++k) {+ auto& vk = stage_verifier(k);+ if (vk.batch_busy()) {+ busy = true;+ vk.batch_serve(win_pool_fn, win_pool_user);+ }+ }+ if (busy) continue;+ // STRATA_TEST_BELL_SLEEP_US=N (a test, an A/B of the wait below): sleep N us instead, as+ // the first form of this thread did (a whole timer tick on Windows)+ static const int test_sleep_us = [] {+ const char* t = std::getenv("STRATA_TEST_BELL_SLEEP_US");+ return t != nullptr ? std::max(0, std::atoi(t)) : 0;+ }();+ if (test_sleep_us > 0) {+ std::this_thread::sleep_for(std::chrono::microseconds(test_sleep_us));+ continue;+ }+ // no window in flight: wait for pump's next launch (it notifies when this is set; the+ // fences pair the flag with batch_launch's running flag, so one of the two sides sees the+ // other), or for the end of the read+ std::unique_lock<std::mutex> lk(bell_mu);+ bell_waiting.store(true, std::memory_order_relaxed);+ std::atomic_thread_fence(std::memory_order_seq_cst);+ bell_cv.wait_for(lk, std::chrono::milliseconds(2),+ [&] { return !bells.on.load(std::memory_order_acquire) || any_busy(); });+ bell_waiting.store(false, std::memory_order_relaxed);+ }+ });+ for (int64_t q0 = a0; q0 < b0;) {+ const auto tc = Clock::now();+ const int64_t rows0 = pipe_rows;+ // per stage, for the read's log line: its windows and the host time that set them up (the+ // PLE gather included, which waits while the prompt holds the table) - both this thread's+ std::vector<std::pair<int64_t, double>> st0((size_t) n_pipe);+ for (int k = 0; k < n_pipe; ++k)+ st0[(size_t) k] = {stage_verifier(k).windows, stage_verifier(k).ms_host};+ std::atomic<bool> done{false};+ std::atomic<int> want_yield{-1};+ bool read_ok = true;+ std::string read_err;+ in_prompt_read = true;+ sp.stop_at_chunk = [&] { return want_yield.load(std::memory_order_acquire) >= 0; };+ std::thread reader([&] {+ conc_thread_cpus();+ try { // (an exception leaving a std::thread ends the process: a failed read instead)++ if (planned) sp.force_chunk(plan.chunk, plan_single, plan.chunked);+ read_ok = sp.run(ids.data() + q0, b0 - q0, q0, read_err);+ } catch (const std::exception& x) {+ read_ok = false;+ read_err = std::string("the prompt read failed: ") + x.what();+ } catch (...) {+ read_ok = false;+ read_err = "the prompt read failed";+ }+ done.store(true, std::memory_order_release);+ { std::lock_guard<std::mutex> lk(in_mu); } // (the loop below may be waiting on in_cv)+ in_cv.notify_all();+ });+ struct ReadEnd { // on every way out of this round (an exception too): the reader joined first+ std::thread& t;+ std::function<bool()>& hook;+ bool& reading;+ ~ReadEnd() {+ if (t.joinable()) t.join();+ hook = nullptr;+ reading = false;+ }+ } read_end{reader, sp.stop_at_chunk, in_prompt_read};+ bool pump_ok = true;+ size_t seen = 0; // in_lines' length after the last look: a line that comes wakes the wait+ while (!done.load(std::memory_order_acquire)) {+ const bool decoding = pump_ok && (batch_on() || pipe_inflight());+ if (decoding) pump_ok = pump(true);+ std::unique_lock<std::mutex> lk(in_mu);+ if (!decoding) // nothing to pump: until the read ends or a line comes (both notify)+ in_cv.wait_for(lk, std::chrono::milliseconds(2), [&] {+ return done.load(std::memory_order_acquire) || in_lines.size() != seen;+ });+ // a BSTOP that came meanwhile ends its slot at its next window; a BYIELD ends the read at+ // its next chunk+ for (auto it = in_lines.begin(); it != in_lines.end();) {+ if (it->rfind("BSTOP ", 0) == 0) {+ const int b = std::atoi(it->c_str() + 6);+ if (b >= 0 && b < (int) bs.size()) bs[(size_t) b].stop = true;+ it = in_lines.erase(it);+ } else if (it->rfind("BYIELD ", 0) == 0 && want_yield.load() < 0) {+ want_yield.store(std::atoi(it->c_str() + 7), std::memory_order_release);+ it = in_lines.erase(it);+ } else {+ ++it;+ }+ }+ seen = in_lines.size();+ }+ reader.join(); // (a failed window leaves the read to finish: it is not stopped mid-run)+ sp.stop_at_chunk = nullptr;+ in_prompt_read = false;+ if (!pump_ok || !pipe_drain()) {+ batch_fatal = true;+ e = "a pipelined batch window failed";+ return false;+ }+ const int64_t q = read_ok && sp.stopped_at() >= 0 ? q0 + sp.stopped_at() : b0;+ const double read_ms = std::chrono::duration<double, std::milli>(Clock::now() - tc).count();+ std::string per;+ for (int k = 0; k < n_pipe; ++k) {+ const auto& v = stage_verifier(k);+ const int64_t w = v.windows - st0[(size_t) k].first;+ char b[96];+ const double host = w > 0 ? (v.ms_host - st0[(size_t) k].second) / (double) w : 0.0;+ std::snprintf(b, sizeof b, "%sstage %d: %lld windows, %.1f ms host each", k ? "; " : "",+ k + 1, (long long) w, host);+ per += b;+ }+ std::fprintf(stderr, "strata batch: read %lld tokens on a thread of its own in %.0f ms "+ "(concurrent, to %lld of %lld); the slots decoded %lld rows meanwhile "+ "(%.1f rows/s; %s)\n",+ (long long) (q - q0), read_ms, (long long) q, (long long) n,+ (long long) (pipe_rows - rows0),+ read_ms > 0.0 ? 1000.0 * (double) (pipe_rows - rows0) / read_ms : 0.0,+ per.c_str());+ if (!read_ok) {+ e = read_err;+ return false;+ }+ if (q < b0 && give_way(want_yield.load(), q, e)) return false;+ q0 = q; // done, or the give-way was not taken there: the rest goes on the same way+ }+ return true;+ }+ // Opt-in (#1906): on the pipelined slot groups a read goes in pieces of STRATA_PIPED_READ_PIECE tokens+ // (whole chunks). A piece's end has the pipeline drained, so a waiting shorter request can take+ // over there (BYIELD); with slots decoding, the groups then decode through the pipeline (pump) for+ // STRATA_BATCH_DECODE_SHARE x the piece's time, capped / floored by STRATA_PIPED_DECODE_MAX_MS /+ // _MIN_MS, and are drained (pipe_drain) before the next piece, as before any read. Unset: one run.+ static const int64_t pr_piece = [] {+ const char* v = std::getenv("STRATA_PIPED_READ_PIECE");+ return v != nullptr ? (int64_t) std::max(0, std::atoi(v)) : (int64_t) 0;+ }();+ static const double pr_cap = [] {+ const char* v = std::getenv("STRATA_PIPED_DECODE_MAX_MS");+ return v != nullptr ? std::max(0.0, std::atof(v)) : 0.0;+ }();+ static const double pr_floor = [] {+ const char* v = std::getenv("STRATA_PIPED_DECODE_MIN_MS");+ return v != nullptr ? std::max(0.0, std::atof(v)) : 0.0;+ }();+ const bool pr_on = piped && pr_piece > 0;+ auto piece_now = [&]() -> int64_t {+ return pr_on ? strata::program::piped_read_piece(pr_piece, C)+ : strata::program::batch_read_piece(o.batch > 0, piped, !stages.empty(), batch_on(), C,+ yield_chunks);+ };+ // the slots decode through the pipeline for budget_ms, drained before the next read+ auto piped_decode = [&](double budget_ms, std::string& err) -> bool {+ const auto td = Clock::now();+ in_prompt_read = true;+ do {+ if (!pump(true)) {+ in_prompt_read = false; batch_fatal = true; err = "a pipelined batch window failed";+ return false;+ }+ } while (batch_on() &&+ std::chrono::duration<double, std::milli>(Clock::now() - td).count() < budget_ms);+ if (!pipe_drain()) {+ in_prompt_read = false; batch_fatal = true; err = "a pipelined batch window failed";+ return false;+ }+ in_prompt_read = false;+ ++il_parts;+ il_ms += std::chrono::duration<double, std::milli>(Clock::now() - td).count();+ return true;+ };+ // prompts arriving back to back: one decode budget before a read that follows a piecewise read+ static Clock::time_point pr_prev_end{};+ static bool pr_have_prev = false;+ static double pr_prev_piece_ms = 0.0; // the previous read's last full piece+ if (pr_on) {+ const double lead =+ pr_prev_piece_ms > 0.0+ ? strata::program::piped_decode_budget_ms(pr_prev_piece_ms, decode_share, pr_cap, pr_floor)+ : (pr_cap > 0.0 ? pr_cap : std::max(pr_floor, 200.0));+ const double since =+ pr_have_prev ? std::chrono::duration<double, std::milli>(Clock::now() - pr_prev_end).count()+ : -1.0;+ if (strata::program::piped_read_lead_in(true, batch_on(), since, lead)) {+ const int64_t rows0 = pipe_rows;+ const auto tl = Clock::now();+ if (!piped_decode(lead, e)) return false;+ std::fprintf(stderr, "strata batch: a read right after a read: the slots decoded %lld rows in "+ "%.0f ms first\n", (long long) (pipe_rows - rows0),+ std::chrono::duration<double, std::milli>(Clock::now() - tl).count());+ }+ }+ int64_t piece = piece_now();- if (piece <= 0) return sp.run(ids.data() + a0, b0 - a0, a0, e);+ if (piece <= 0) return run(a0, b0);for (int64_t q = a0; q < b0;) {const int64_t r = std::min(b0, q + piece);const auto tq = Clock::now();- if (!sp.run(ids.data() + q, r - q, q, e)) return false;+ if (!run(q, r)) return false;+ const double piece_ms = std::chrono::duration<double, std::milli>(Clock::now() - tq).count();+ if (pr_on && r < b0) pr_prev_piece_ms = piece_ms;+ const int64_t piece_tokens = r - q;q = r;if (q >= b0) continue;int ys = -1;