Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Merge perf-PR group batch-1441 (f-145-pp-batch-1441)

Niko1221da6d8d920577
diff --cc docs/MULTI_GPU.md
index 6d2c8dc9b,40c7178cf..30e316d52
--- a/docs/MULTI_GPU.md
+++ b/docs/MULTI_GPU.md
@@@ -215,33 -199,75 +215,102 @@@ that overlap instead of one, a 32K one
32K +18% / +21% / +29%, 64K +4% / +12% / +17%, 8K unchanged. `STRATA_SPLIT_CHUNK16K=1` keeps 16384.) Conversation checkpoints save and restore every card's state; the adaptive expert swaps copy
into the card that owns the layer.
+## Overlapped hand-off (`STRATA_SPLIT_OVERLAP=1`, opt-in)
+
+By default each verify window runs the stages one after another: the host syncs stage 1, then stages and launches
+stage 2's graph, and so on. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives the hand-off a ready
+flag: the earlier stage's GPU writes the hand-off with volatile stores and raises the flag (`handoff_publish`), the
+next stage's graph is staged and launched on a helper thread while the earlier stage is still running and waits for
+the flag on its GPU (`wait_flag_ge`), and the host goes straight from serving one stage's layers to the next, syncing
+the earlier stage only once the chain is done. It takes one stream sync and one graph launch per window off the
+critical path. Batch windows (several conversations) keep the serial order. It is off beside `--pipeline-windows`, whose loop overlaps the stages its own way.
+
+Measured on 2x RX 7900 XTX (gfx1100, ROCm, Linux), Ryzen 9 9900X, 52 GB RAM, Swift 1.5 IQ3_XXS, 128K context, 8-bit
+KV, layers 0-25 / 26-47; 512-token greedy answers, median of 8 runs:
+
+| | serial (default) | `STRATA_SPLIT_OVERLAP=1` | one card |
+|---|---|---|---|
+| decode, code prompt | 53 tok/s | 70.1 tok/s | ~70 tok/s |
+| decode, prose prompt | 45 tok/s | 55.9 tok/s | ~68 tok/s |
+
+Greedy output is byte-identical with the flag at 0 and 1; the quiz set scored the same (47/51). Not measured on
+NVIDIA yet, where a sync and a graph launch cost less.
+
+The reading stage's wait for the flag is bounded on the GPU: `STRATA_SPLIT_WAIT_MS` (default 30000, 0 = no bound).
+If the flag does not arrive in time, the wait marks an error word and returns, the window's graph completes, and the
+host fails that window with an error naming the setting (the server then restarts the engine) instead of using a
+stale hand-off. Normally the host's own per-layer wait (20 s, #267) ends a stalled window first and releases every
+GPU wait, the hand-off's included. Test hook: `STRATA_TEST_HANDOFF_DROP=N` withholds the flag in the first stage's
+N-th window, as if the publish were lost.
+ ## Prompt chunks sized for the pipeline (`--prefill-pipe`, opt-in; setup's `--calibrate` sets it)
+
+ `--prefill auto` takes the largest chunk the buffers allow (up to 8192), which is what one card wants: a chunk costs a
+ fixed pass over its layers' experts (b: dequantized, or streamed over PCIe where they are not in VRAM) plus a part per
+ token (a), so few large chunks are cheapest. A layer split reads a prompt as a pipeline - stage s reads chunk c while
+ stage s+1 reads chunk c-1 - and then takes about sum_i (b + a C_i) + (S - 1) max_i (b + a C_i) for chunks C_i over S
+ stages: large or uneven chunks keep the later cards waiting (a 10.7K-token prompt in a 7680 and a 3011 chunk over
+ four stages: the first stage carried 24 of the 36 s), small ones pay b again and again.
+
+ `--prefill-pipe R` reads each prompt in the chunk count with the least pipeline time for b/a = R tokens, when that
+ beats today's split by 3% or more: every count from the fewest the buffers allow is tried, each chunk evened out on the
+ 256-token grid (at least 512, at most the buffers' chunk; keep `--prefill auto` so long prompts may use large chunks).
+ b/a differs by an order of magnitude between rigs - about 700 tokens on four GP100 dies (every expert in VRAM, slow
+ arithmetic), about 5000 on two RTX 3090 Ti (part of the experts streamed per chunk, fast arithmetic) - and a wrong one
+ costs: a fixed 1024 read 19-24% slower on the two RTX 3090 Ti and 22-36% slower on two RTX 4090 (reported in the PR by
+ MistyMoonR). So it is neither guessed nor learned from the prompts the engine serves (that made a prompt's chunks, and
+ so its output bits, depend on how busy the PC happened to be): like the PCIe share and the draft floor, it is a
+ setting of the PC that setup's `--calibrate` measures once (`tools/calibrate.py`: on a layer split, the prompt read
+ speed of a 2K- and an 8K-token prompt - the same text every time, after a header of its own so nothing is reused -
+ with the chunks off, evened out and b/a 768, 1536, 3072 and 6144; the best is kept only when it reads faster than off
+ in each of three paired rounds and by more than 3% in the median) and writes into the config. Measured so: four GP100
+ dies keep b/a 768 (355-356 against 219-220 tok/s in every round, +62%); two RTX 3090 Ti keep it off (nothing read
+ faster than off, b/a 768 20% slower). A first version that read a different stretch of text per value and confirmed in
+ two rounds picked 1536 on the RTX 3090 Ti pair from one fast read - where the experts are streamed, the read speed
+ depends on the text. `--prefill-pipe 1` needs no
+ b/a: it is `STRATA_PREFILL_EQUAL=1`'s rule (#693) - today's chunk count, evened out on the 256-token grid, and only
+ when today's last chunk has 1024 tokens or more (3072 with `STRATA_PREFILL_CPU_SHARE` set). A chunk that size streams
+ every expert its card does not hold, a shorter one only the experts it routes to, so a prompt just past the chunk size
+ reads as one full chunk and a nearly free tail, and evening that out would add a second full pass over the experts:
+ the first version evened every split out, and on two RTX 4090 MistyMoonR's calibration read 1,601 tok/s with it
+ against 2,129 off. 0 (the default) is today's split;
+ `STRATA_PREFILL_PIPE=<R>` is the same as the flag where the config cannot carry it. The chunk geometry changes the
+ rounding, which is why it is opt-in.
+
+ A prompt segment (the prompt between two of its cuts: the system-prompt root, the last turn, a `pin=`) is planned once,
+ on its whole length and R, and every run of it reads in that chunk. So its chunks, and the loan its buffers are laid
+ out for, are the same whether it is read in one run, in pieces between the windows of decoding `--batch` slots, or goes
+ on after a `BYIELD` or from a periodic checkpoint taken inside it (the slot and the checkpoint keep the plan; the read
+ goes on in it when it continues the same segment, else it plans again and says so in the log). The output bits then
+ match as far as the experts the cards hold do (`--adapt-swaps 0` for byte-identical repeats, as without the pipe).
+ Before this, a read beside decoding slots re-planned every piece, and one that went on after a `BYIELD` re-planned what
+ was left. A chunk the pipe planned takes no `STRATA_PREFILL_CPU_SHARE`: the stages read
+ such chunks at the same time, and which stage got the CPU pool first would decide which experts the CPU computed. The
+ DONE line ends with `chunk=` (the first chunk of the request's longest batched segment) and `experts_streamed=` (the
+ expert blobs its prompt copied to the cards), and the INFO line carries `prefill_cap`, `prefill_stages`,
+ `stream_all_min` and `prefill_pipe`. One line in the log says what a plan chose:
+
+ strata prefill: 10058 tokens in 1280-token chunks over 4 stages (--prefill-pipe 706: b/a)
+
+ Four GP100 dies (PH402 SKU 200, `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context), prompt tok/s, fresh
+ prompts, two interleaved rounds (each within 0.2%), with b/a 1024 (the same chunks as 700-890, measured on this rig):
+
+ | Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill auto` + `--prefill-pipe 1024` (chunk) |
+ | ---: | ---: | ---: | ---: |
+ | 2,013 | 180.9 | 181.4 | 228.0 (1024) |
+ | 3,865 | 207.6 | 276.5 | 312.4 (1024) |
+ | 10,041 | 261.1 | 433.0 | 432.6 (2048) |
+ | 28,998 | 475.7 | 554.1 | 559.7 (3328) |
+
+ Two RTX 3090 Ti (CUDA 12.9 build for sm_86, two stages, Swift 1.5 IQ3_XXS, 1M context, 13,371 of 24,576 experts in
+ VRAM), prompt tok/s, fresh prompts, two starts per arm, each reading the four prompts twice (the cold first pass after
+ a card was freed is left out); b/a 5836 and 6432 (timed on the rig), against 1024:
+
+ | Prompt tokens | `--prefill auto` | + b/a 5836 / 6432 (chunk) | for comparison: b/a 1024 |
+ | ---: | ---: | ---: | ---: |
+ | 2,035 | 694-815 | 698-816 (today's) | 560-562 |
+ | 3,887 | 1,127-1,207 | 1,115-1,227 (today's) | 880-885 |
+ | 10,063 | 1,437-1,553 | 1,654-1,681 (2 x 5120) | 1,109-1,113 |
+ | 29,019 | 2,532-2,603 | 2,566-2,620 (today's) | 2,064-2,089 |
## Limits (for now)
diff --cc include/strata/prefill/prefill.hpp
index 197dc25fe,79d23a4ac..6c4260824
--- a/include/strata/prefill/prefill.hpp
+++ b/include/strata/prefill/prefill.hpp
@@@ -217,13 -233,11 +250,16 @@@ private
Prefill* next_ = nullptr;
Prefill* helper_ = nullptr; ///< set_stage_helper
bool single_chunk_ = false; ///< a later stage: the prompt is one chunk (set by the stage before)
+ int64_t stopped_at_ = -1; ///< stop_at_chunk ended the last run after this many tokens (-1: it read all)
+ bool pipe_chunked_ = false; ///< a later stage: the chunks came from a --prefill-pipe plan (set likewise)
+ int64_t force_chunk_ = 0; ///< force_chunk, for the next run (0: none)
+ bool force_single_ = false, force_planned_ = false;
bool bind_stage_helper(int64_t T); // binds the helper's buffers for a one-chunk prompt of T tokens
const float* hand_in_ = nullptr; ///< the previous stage's rows of the chunk being read (host, pinned)
+ /// STRATA_HANDOFF_PIPE: the previous stage's per-piece "landed in hand_in_" events (cudaEvent_t[hand_pieces_],
+ /// on its device) when it copies the chunk in pieces without a host wait; null with the one-copy hand-off
+ const void* hand_ev_ = nullptr;
+ int hand_pieces_ = 0;
std::string next_err_;
std::future<bool> next_run_;
diff --cc src/prefill/prefill.cpp
index 3f8409212,a96753e2b..1bdee6772
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@@ -2352,17 -2295,10 +2472,17 @@@ bool Prefill::run_impl(const int64_t* t
std::future<bool> ple_next; // declared after everything it reads: an early return waits for it
int ple_buf = 0;
+ if (hand_in_ == nullptr) stopped_at_ = -1;
- for (int64_t c0 = 0; c0 < n; c0 += m.T) {
+ for (int64_t c0 = 0; c0 < n; c0 += C) {
if (should_stop && should_stop()) { err = "cancelled"; return false; }
+ // the first stage: end the run before this chunk (every chunk before it is handed on; the caller's drain
+ // waits for the later stages to read them)
+ if (c0 > 0 && hand_in_ == nullptr && stop_at_chunk && stop_at_chunk()) {
+ stopped_at_ = c0;
+ break;
+ }
if (std::getenv("STRATA_TRACE")) { std::fprintf(stderr, "strata trace: prompt chunk %lld of %lld\n", (long long) c0, (long long) n); std::fflush(stderr); }
- const int64_t T = std::min(m.T, n - c0), p0 = pos0 + c0;
+ const int64_t T = std::min(C, n - c0), p0 = pos0 + c0;
core::progress_at("reading the prompt (batched): preparing the chunk from token", p0); // #251
++stats_.chunks;
pt.mark(kPfStart, cs);
@@@ -4473,17 -4289,8 +4593,18 @@@
if (next_run_.valid() && !next_run_.get()) { err = next_err_; return false; }
next_err_.clear();
next_->hand_in_ = h;
+ next_->hand_ev_ = hp > 0 ? static_cast<const void*>(m.hand_ev[hand_buf_]) : nullptr;
+ next_->hand_pieces_ = hp;
next_->single_chunk_ = single_chunk;
+ next_->pipe_chunked_ = pipe_chunked;
+ if (handoff_pipe_worker()) {
+ if (!m.hand_worker) m.hand_worker = std::make_unique<HandWorker>();
+ next_run_ = m.hand_worker->submit(std::packaged_task<bool()>([this, tokens, c0, T, p0] {
+ const bool r = next_->run_impl(tokens + c0, T, p0, next_err_);
+ (void) cudaGetLastError(); // clear what a thread's end cleared (std::async)
+ return r;
+ }));
+ } else
next_run_ = std::async(std::launch::async, [this, tokens, c0, T, p0] {
strata::aux_cpus::pin_current_thread();
return next_->run_impl(tokens + c0, T, p0, next_err_);
diff --cc src/program/generate.cpp
index 09bdef8e7,ae76d8567..d4ea3891c
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@@ -8597,11 -8142,11 +8609,14 @@@ int main(int argc, char** argv)
// #879: a window met non-finite logits: the next request drops every cached state first and reads from token 0
bool nan_flush = false;
std::vector<ConvCheckpoint> checks;
+ // STRATA_CKPT_POOL: the dropped checkpoints' storage, reused. STRATA_SLOT_CKPT_MOVE turns it on too: a moved
+ // checkpoint frees nothing at the next prune, so every save would fault in fresh pages (17 -> 40-60 ms)
+ CheckpointPool ckpt_pool(ckpt_pool_env() || slot_ckpt_move_env());
uint64_t check_clock = 0; // the checkpoints' LRU clock; creation and every use advance it
int64_t tail_ckpt_len = -1; // --prompt-cache-tail: the length of the one tail checkpoint alive (-1 = none)
+ // --prefill-pipe: the plan of the prompt segment being read (read_part sets it around its runs), for the
+ // checkpoints taken inside it
+ ConvCheckpoint::PromptPlan seg_plan;
bool cvec_cached = true; // the control vector's state the live session and the checkpoints were read with
// without --mtp the conversation cache stays on: a parked image carries the draft layer's K/V only when there is one
// (the snapshots accept a null draft)
@@@ -9027,13 -8481,12 +9042,14 @@@
}
}
c.used = ++check_clock;
+ if (seg_plan.chunk > 0 && L > seg_plan.from && L < seg_plan.to) c.plan = seg_plan; // inside a planned segment
if (as_tail) { // only one tail checkpoint stays alive: the previous request's goes
const int64_t old_tail = tail_ckpt_len;
- checks.erase(std::remove_if(checks.begin(), checks.end(), [&](const ConvCheckpoint& k) {
- return old_tail >= 0 && (int64_t) k.ids.size() == old_tail;
- }), checks.end());
+ auto old = [&](const ConvCheckpoint& k) { return old_tail >= 0 && (int64_t) k.ids.size() == old_tail; };
+ if (ckpt_pool.on())
+ ckpt_pool.erase_if(checks, old);
+ else
+ checks.erase(std::remove_if(checks.begin(), checks.end(), old), checks.end());
tail_ckpt_len = L;
} else if (L == tail_ckpt_len) {
tail_ckpt_len = -1; // the flagged tail was dropped and a normal checkpoint takes its length
@@@ -11927,21 -11226,29 +11962,43 @@@
std::printf("ERR restoring a conversation checkpoint failed\n");
return 1;
}
+ if (lent && lent_slot == admit_slot) {
+ // the slot this request continues in drops its checkpoints at the admission: this one moves back
+ // into the chain (and on into the slot then: moved_in)
+ std::vector<ConvCheckpoint>& sc = bs[(size_t) lent_slot].checks;
+ const auto it = sc.begin() + (c - sc.data());
+ checks.push_back(std::move(*it));
+ sc.erase(it);
+ moved_in = resume;
+ lent_slot = -1;
+ }
+ if (lent && trace)
+ std::fprintf(stderr, "strata trace: the checkpoint at %lld restored from slot %d's%s\n",
+ (long long) resume, lent_slot >= 0 ? lent_slot : admit_slot,
+ lent_slot >= 0 ? "" : " (moved back into the chain)");
}
+ // --prefill-pipe: going on from a periodic checkpoint taken inside a planned segment (whatever restored it:
+ // a checkpoint of this session, of a slot or of a parked conversation), the rest of that segment is read
+ // in its chunks. STRATA_CKPT_REREAD reads [0, resume) again in the chunks the saving read had, which it
+ // can only when that segment started at 0. The BYIELD park set resume_plan above.
+ if (resume_plan.chunk <= 0 && resume > 0 && !stages.empty())
+ for (const ConvCheckpoint& k : checks)
+ if ((int64_t) k.ids.size() == resume && k.plan.chunk > 0) {
+ if (reread_to <= 0) {
+ resume_plan = k.plan;
+ resume_plan_at = resume;
+ resume_plan_end = k.plan.to;
+ } else if (k.plan.from == 0) { // [0, resume) only: its end is the checkpoint
+ resume_plan = k.plan;
+ resume_plan_at = 0;
+ resume_plan_end = resume;
+ } else {
+ std::fprintf(stderr, "strata serve: STRATA_CKPT_REREAD: the checkpoint is inside a "
+ "segment that started at %lld, so a read from 0 has other chunks\n",
+ (long long) k.plan.from);
+ }
+ break;
+ }
// KV streaming: the drafter's ring may hold cells past `resume` from a longer turn; the main layers'
// host copies and slots are always current (every writer writes both), so they need nothing
if (use_mtp && resume > 0 && reread_to <= 0) mtp.kv_restore(resume);
@@@ -12327,377 -11688,17 +12439,383 @@@
const char* v = std::getenv("STRATA_BATCH_YIELD_CHUNKS");
return v != nullptr ? (int64_t) std::max(0, std::atoi(v)) : (int64_t) 8;
}();
- const int64_t C = std::max<int64_t>(sp.chunk(), 1);
+ // a piece's chunk: the buffers' chunk, or with a plan as many whole planned chunks as that holds (the
+ // stages still pipeline inside a piece, and every piece starts on the plan's grid)
+ const int64_t C = planned ? std::max<int64_t>(1, sp.chunk() / plan.chunk) * plan.chunk
+ : std::max<int64_t>(sp.chunk(), 1);
+ const int64_t C_read = planned ? plan.chunk : C; // what the yield test calls a chunk
- const int64_t piece = strata::program::batch_read_piece(o.batch > 0, piped, !stages.empty(), batch_on(), C, yield_chunks);
+ // BYIELD <ys> with the read stopped at q (nothing in flight): the part read waits in slot ys and the
+ // read ends with e = "yield" (true); false when it cannot give way there (the read goes on). Only
+ // where the rest is read the same way after it (more than a chunk and more than a short read left),
+ // text only, and into a slot that is not decoding.
+ auto give_way = [&](int ys, int64_t q, std::string& err) -> bool {
+ const bool can = ys < (int) bs.size() && !bs[(size_t) ys].active &&
+ (admit_slot < 0 || ys == admit_slot) && req_imgs.empty() && o.prompt_cache > 0 &&
- b0 - q > std::max<int64_t>(C, o.short_read);
++ b0 - q > std::max<int64_t>(C_read, o.short_read);
+ std::string ye;
+ const auto ty = Clock::now();
+ std::vector<int32_t> pre(ids.begin(), ids.begin() + q);
+ if (can && copy_to_slot(ys, pre, ye)) {
+ BSlot& sl = bs[(size_t) ys];
+ sl = BSlot{};
+ sl.ids = std::move(pre);
+ sl.cached = true;
+ sl.partial = true;
+ sl.partial_from0 = read_from == 0 || resumed_from0;
++ if (planned) sl.partial_plan = plan; // the resend reads the rest in these chunks
+ sl.cvec = cvec_cached;
+ for (const ConvCheckpoint& c : checks) // the root / periodic checkpoints of this read
+ if ((int64_t) c.ids.size() <= q && std::equal(c.ids.begin(), c.ids.end(), ids.begin(),
+ [](int32_t x, int64_t y) { return (int64_t) x == y; }))
+ sl.checks.push_back(c);
+ yielded_at = q;
+ yielded_slot = ys;
+ std::fprintf(stderr, "strata batch: the prompt read gives way at %lld of %lld tokens; slot %d "
+ "holds it (copied in %.1f ms)\n", (long long) q, (long long) n, ys,
+ std::chrono::duration<double, std::milli>(Clock::now() - ty).count());
+ err = "yield";
+ return true;
+ }
+ if (can) bs[(size_t) ys].cached = false; // half written
+ std::fprintf(stderr, "strata batch: BYIELD %d not taken at %lld of %lld%s%s\n", ys, (long long) q,
+ (long long) n, ye.empty() ? "" : ": ", ye.c_str());
+ return false;
+ };
+ // Opt-in, experimental: STRATA_PIPED_CONCURRENT=1 reads the prompt on a thread of its own while
+ // this thread keeps the slot groups decoding through the pipeline (pump), so they never stop for it
+ // and the read has no drain inside it. The prompt and the windows use their own streams and memory
+ // (the prompt's K/V goes to the main sessions, the slots' to theirs; the PLE table is locked), and
+ // the adaptive tier stands still (its round landed before the read). The slots the prompt borrowed
+ // were lent before this read with nothing in flight and are refilled after it, so the windows see
+ // them missing and compute those experts on the CPU, as between pieces. A BYIELD ends the run at
+ // its next chunk (Prefill::stop_at_chunk): the pipeline is drained there and the part read gives
+ // way as below, or the read goes on from there.
+ // Not where a read's thread allocates or frees (the idle stage's help, STRATA_PREFILL_HELP; the batched
+ // draft K/V, STRATA_SPLIT_MTP_BATCH: a free waits for the device), where serving a doorbell makes a
+ // CUDA call (--pcie-mode dma, the Foresight swap space STRATA_FS_SLOTS, the helper caches), and where
+ // the expert source recycles the buffers it hands out (the GGUF in place, an unbuffered source, the
+ // I/O read-ahead: one thread at a time). Pictures are read the usual way (their rows go up through
+ // pinned slots of a size known only then).
+ static const bool pr_conc = [&] {
+ const char* v = std::getenv("STRATA_PIPED_CONCURRENT");
+ if (v == nullptr || std::atoi(v) == 0) return false;
+ auto env_on = [](const char* k) {
+ const char* x = std::getenv(k);
+ return x != nullptr && std::atoi(x) != 0;
+ };
+ strata::program::ConcurrentReadSetup s;
+ s.prefill_help = env_on("STRATA_PREFILL_HELP");
+ s.split_mtp_batch = env_on("STRATA_SPLIT_MTP_BATCH");
+ s.fs_slots = env_on("STRATA_FS_SLOTS");
+ s.pcie_dma = o.pcie_mode == "dma";
+ // (--remote-expert-opt works only through a helper cache, so without one it does not count)
+ s.helper_caches = prefill_src != srcp;
+ for (int r = 0; r < 3; ++r) s.helper_caches |= o.expert_cache_remote[(size_t) r] > 0;
+ s.staged_source = srcp == &src && src.stages_blobs();
+ s.other_source = srcp != &src && srcp != &arena_src;
+ if (const char* why = strata::program::concurrent_read_off(s)) {
+ std::fprintf(stderr, "strata batch: STRATA_PIPED_CONCURRENT is off beside %s\n", why);
+ return false;
+ }
+ return true;
+ }();
+ if (piped && pr_conc && req_imgs.empty()) { // (with no slot decoding it only watches for a BYIELD)
+ // the drafter's prompt pass captures its graphs and grows its records at first use, and the
+ // prompt path's pinned upload slots grow at their first larger upload: done here, with nothing in
+ // flight (on the reader's thread a capture, an allocation or a free takes the driver's lock and
+ // can wait for the device, #31)
+ if (use_mtp && !mtp.reserve_prefill(C, e)) return false;
+ if (!sp.reserve_uploads(C))
+ std::fprintf(stderr, "strata batch: the prompt path's pinned upload slots could not be "
+ "reserved (they grow at first use)\n");
+ // where the read's thread and the doorbell thread run: on Linux a thread inherits its creator's
+ // CPUs, and the host thread is pinned to the one core the pool leaves it, so both would share that
+ // core with the host thread that pumps the windows - every window then waits its turn there for
+ // the doorbell of each layer (4x RTX 5080, 4 stages: 6 rows/s during a read). They move off the
+ // host's core and the pool workers' CPUs instead (--aux-cpus places them, when it is on).
+ // STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B); STRATA_TEST_CONC_ON_HOST_CPU=1
+ // (Windows, a test) puts them on the host thread's CPUs first, as Linux does.
+ static const bool conc_unpin = [] {
+ const char* v = std::getenv("STRATA_PIPED_CONCURRENT_UNPIN");
+ return v == nullptr || std::atoi(v) != 0;
+ }();
+#if defined(_WIN32)
+ static const bool conc_on_host = std::getenv("STRATA_TEST_CONC_ON_HOST_CPU") != nullptr;
+ std::vector<ULONG> host_cpu_sets;
+ if (conc_on_host) {
+ const HANDLE self = GetCurrentThread();
+ ULONG count = 0;
+ if (GetThreadSelectedCpuSets(self, nullptr, 0, &count) ||
+ GetLastError() == ERROR_INSUFFICIENT_BUFFER) {
+ host_cpu_sets.resize(count);
+ if (count && !GetThreadSelectedCpuSets(self, host_cpu_sets.data(), count, &count))
+ host_cpu_sets.clear();
+ }
+ static bool said = false;
+ if (!said) {
+ said = true;
+ std::fprintf(stderr, "strata batch: test: the read's and the doorbell threads start on the "
+ "host thread's %zu CPU set(s)%s\n", host_cpu_sets.size(),
+ conc_unpin ? ", then leave them (the fix)"
+ : " and stay there (as on Linux)");
+ }
+ }
+#endif
+ // the threads that already inherited the host's core (the prompt path's copy threads, started by
+ // the first prompt on the host thread) work during the read too: off that core as well (Linux)
+ if (conc_unpin && !strata::aux_cpus::enabled())
+ if (const int moved = strata::aux_cpus::release_host_inheritors(); moved > 0)
+ std::fprintf(stderr, "strata batch: %d threads that ran on the host thread's core moved off "
+ "it, for reads beside the slots\n", moved);
+ auto conc_thread_cpus = [&] {
+ strata::aux_cpus::pin_current_thread();
+#if defined(_WIN32)
+ const HANDLE self = GetCurrentThread();
+ if (conc_on_host && !host_cpu_sets.empty())
+ SetThreadSelectedCpuSets(self, host_cpu_sets.data(), (ULONG) host_cpu_sets.size());
+ if (conc_unpin && conc_on_host) SetThreadSelectedCpuSets(self, nullptr, 0);
+#else
+ if (conc_unpin && !strata::aux_cpus::enabled()) (void) strata::aux_cpus::place_off_host();
+#endif
+ };
+ // the slots' doorbells are served on a thread of their own for the read, which makes no CUDA
+ // call: a call here or on the read's threads can wait for the driver's lock while its holder
+ // waits for the GPU (a free; a module the first kernel of a shape loads), and a window on that
+ // GPU waits for its doorbell (#31). Ends (and hands the service back to pump) on every exit.
+ struct Doorbells {
+ std::atomic<bool> on{true};
+ std::thread th;
+ std::function<void()> wake, off;
+ ~Doorbells() {
+ on.store(false, std::memory_order_release);
+ if (wake) wake();
+ if (th.joinable()) th.join();
+ if (off) off();
+ }
+ } bells;
+ for (int k = 0; k < n_pipe; ++k) stage_verifier(k).set_external_serve(true);
+ conc_serving = true;
+ bells.off = [&] {
+ for (int k = 0; k < n_pipe; ++k) stage_verifier(k).set_external_serve(false);
+ conc_serving = false;
+ };
+ bells.wake = [&] {
+ std::lock_guard<std::mutex> lk(bell_mu);
+ bell_cv.notify_one();
+ };
+ bells.th = std::thread([&] {
+ conc_thread_cpus();
+ auto any_busy = [&] {
+ for (int k = 0; k < n_pipe; ++k)
+ if (stage_verifier(k).batch_busy()) return true;
+ return false;
+ };
+ while (bells.on.load(std::memory_order_acquire)) {
+ bool busy = false;
+ for (int k = 0; k < n_pipe; ++k) {
+ auto& vk = stage_verifier(k);
+ if (vk.batch_busy()) {
+ busy = true;
+ vk.batch_serve(win_pool_fn, win_pool_user);
+ }
+ }
+ if (busy) continue;
+ // STRATA_TEST_BELL_SLEEP_US=N (a test, an A/B of the wait below): sleep N us instead, as
+ // the first form of this thread did (a whole timer tick on Windows)
+ static const int test_sleep_us = [] {
+ const char* t = std::getenv("STRATA_TEST_BELL_SLEEP_US");
+ return t != nullptr ? std::max(0, std::atoi(t)) : 0;
+ }();
+ if (test_sleep_us > 0) {
+ std::this_thread::sleep_for(std::chrono::microseconds(test_sleep_us));
+ continue;
+ }
+ // no window in flight: wait for pump's next launch (it notifies when this is set; the
+ // fences pair the flag with batch_launch's running flag, so one of the two sides sees the
+ // other), or for the end of the read
+ std::unique_lock<std::mutex> lk(bell_mu);
+ bell_waiting.store(true, std::memory_order_relaxed);
+ std::atomic_thread_fence(std::memory_order_seq_cst);
+ bell_cv.wait_for(lk, std::chrono::milliseconds(2),
+ [&] { return !bells.on.load(std::memory_order_acquire) || any_busy(); });
+ bell_waiting.store(false, std::memory_order_relaxed);
+ }
+ });
+ for (int64_t q0 = a0; q0 < b0;) {
+ const auto tc = Clock::now();
+ const int64_t rows0 = pipe_rows;
+ // per stage, for the read's log line: its windows and the host time that set them up (the
+ // PLE gather included, which waits while the prompt holds the table) - both this thread's
+ std::vector<std::pair<int64_t, double>> st0((size_t) n_pipe);
+ for (int k = 0; k < n_pipe; ++k)
+ st0[(size_t) k] = {stage_verifier(k).windows, stage_verifier(k).ms_host};
+ std::atomic<bool> done{false};
+ std::atomic<int> want_yield{-1};
+ bool read_ok = true;
+ std::string read_err;
+ in_prompt_read = true;
+ sp.stop_at_chunk = [&] { return want_yield.load(std::memory_order_acquire) >= 0; };
+ std::thread reader([&] {
+ conc_thread_cpus();
+ try { // (an exception leaving a std::thread ends the process: a failed read instead)
++ if (planned) sp.force_chunk(plan.chunk, plan_single, plan.chunked);
+ read_ok = sp.run(ids.data() + q0, b0 - q0, q0, read_err);
+ } catch (const std::exception& x) {
+ read_ok = false;
+ read_err = std::string("the prompt read failed: ") + x.what();
+ } catch (...) {
+ read_ok = false;
+ read_err = "the prompt read failed";
+ }
+ done.store(true, std::memory_order_release);
+ { std::lock_guard<std::mutex> lk(in_mu); } // (the loop below may be waiting on in_cv)
+ in_cv.notify_all();
+ });
+ struct ReadEnd { // on every way out of this round (an exception too): the reader joined first
+ std::thread& t;
+ std::function<bool()>& hook;
+ bool& reading;
+ ~ReadEnd() {
+ if (t.joinable()) t.join();
+ hook = nullptr;
+ reading = false;
+ }
+ } read_end{reader, sp.stop_at_chunk, in_prompt_read};
+ bool pump_ok = true;
+ size_t seen = 0; // in_lines' length after the last look: a line that comes wakes the wait
+ while (!done.load(std::memory_order_acquire)) {
+ const bool decoding = pump_ok && (batch_on() || pipe_inflight());
+ if (decoding) pump_ok = pump(true);
+ std::unique_lock<std::mutex> lk(in_mu);
+ if (!decoding) // nothing to pump: until the read ends or a line comes (both notify)
+ in_cv.wait_for(lk, std::chrono::milliseconds(2), [&] {
+ return done.load(std::memory_order_acquire) || in_lines.size() != seen;
+ });
+ // a BSTOP that came meanwhile ends its slot at its next window; a BYIELD ends the read at
+ // its next chunk
+ for (auto it = in_lines.begin(); it != in_lines.end();) {
+ if (it->rfind("BSTOP ", 0) == 0) {
+ const int b = std::atoi(it->c_str() + 6);
+ if (b >= 0 && b < (int) bs.size()) bs[(size_t) b].stop = true;
+ it = in_lines.erase(it);
+ } else if (it->rfind("BYIELD ", 0) == 0 && want_yield.load() < 0) {
+ want_yield.store(std::atoi(it->c_str() + 7), std::memory_order_release);
+ it = in_lines.erase(it);
+ } else {
+ ++it;
+ }
+ }
+ seen = in_lines.size();
+ }
+ reader.join(); // (a failed window leaves the read to finish: it is not stopped mid-run)
+ sp.stop_at_chunk = nullptr;
+ in_prompt_read = false;
+ if (!pump_ok || !pipe_drain()) {
+ batch_fatal = true;
+ e = "a pipelined batch window failed";
+ return false;
+ }
+ const int64_t q = read_ok && sp.stopped_at() >= 0 ? q0 + sp.stopped_at() : b0;
+ const double read_ms = std::chrono::duration<double, std::milli>(Clock::now() - tc).count();
+ std::string per;
+ for (int k = 0; k < n_pipe; ++k) {
+ const auto& v = stage_verifier(k);
+ const int64_t w = v.windows - st0[(size_t) k].first;
+ char b[96];
+ const double host = w > 0 ? (v.ms_host - st0[(size_t) k].second) / (double) w : 0.0;
+ std::snprintf(b, sizeof b, "%sstage %d: %lld windows, %.1f ms host each", k ? "; " : "",
+ k + 1, (long long) w, host);
+ per += b;
+ }
+ std::fprintf(stderr, "strata batch: read %lld tokens on a thread of its own in %.0f ms "
+ "(concurrent, to %lld of %lld); the slots decoded %lld rows meanwhile "
+ "(%.1f rows/s; %s)\n",
+ (long long) (q - q0), read_ms, (long long) q, (long long) n,
+ (long long) (pipe_rows - rows0),
+ read_ms > 0.0 ? 1000.0 * (double) (pipe_rows - rows0) / read_ms : 0.0,
+ per.c_str());
+ if (!read_ok) {
+ e = read_err;
+ return false;
+ }
+ if (q < b0 && give_way(want_yield.load(), q, e)) return false;
+ q0 = q; // done, or the give-way was not taken there: the rest goes on the same way
+ }
+ return true;
+ }
+ // Opt-in (#1906): on the pipelined slot groups a read goes in pieces of STRATA_PIPED_READ_PIECE tokens
+ // (whole chunks). A piece's end has the pipeline drained, so a waiting shorter request can take
+ // over there (BYIELD); with slots decoding, the groups then decode through the pipeline (pump) for
+ // STRATA_BATCH_DECODE_SHARE x the piece's time, capped / floored by STRATA_PIPED_DECODE_MAX_MS /
+ // _MIN_MS, and are drained (pipe_drain) before the next piece, as before any read. Unset: one run.
+ static const int64_t pr_piece = [] {
+ const char* v = std::getenv("STRATA_PIPED_READ_PIECE");
+ return v != nullptr ? (int64_t) std::max(0, std::atoi(v)) : (int64_t) 0;
+ }();
+ static const double pr_cap = [] {
+ const char* v = std::getenv("STRATA_PIPED_DECODE_MAX_MS");
+ return v != nullptr ? std::max(0.0, std::atof(v)) : 0.0;
+ }();
+ static const double pr_floor = [] {
+ const char* v = std::getenv("STRATA_PIPED_DECODE_MIN_MS");
+ return v != nullptr ? std::max(0.0, std::atof(v)) : 0.0;
+ }();
+ const bool pr_on = piped && pr_piece > 0;
+ auto piece_now = [&]() -> int64_t {
+ return pr_on ? strata::program::piped_read_piece(pr_piece, C)
+ : strata::program::batch_read_piece(o.batch > 0, piped, !stages.empty(), batch_on(), C,
+ yield_chunks);
+ };
+ // the slots decode through the pipeline for budget_ms, drained before the next read
+ auto piped_decode = [&](double budget_ms, std::string& err) -> bool {
+ const auto td = Clock::now();
+ in_prompt_read = true;
+ do {
+ if (!pump(true)) {
+ in_prompt_read = false; batch_fatal = true; err = "a pipelined batch window failed";
+ return false;
+ }
+ } while (batch_on() &&
+ std::chrono::duration<double, std::milli>(Clock::now() - td).count() < budget_ms);
+ if (!pipe_drain()) {
+ in_prompt_read = false; batch_fatal = true; err = "a pipelined batch window failed";
+ return false;
+ }
+ in_prompt_read = false;
+ ++il_parts;
+ il_ms += std::chrono::duration<double, std::milli>(Clock::now() - td).count();
+ return true;
+ };
+ // prompts arriving back to back: one decode budget before a read that follows a piecewise read
+ static Clock::time_point pr_prev_end{};
+ static bool pr_have_prev = false;
+ static double pr_prev_piece_ms = 0.0; // the previous read's last full piece
+ if (pr_on) {
+ const double lead =
+ pr_prev_piece_ms > 0.0
+ ? strata::program::piped_decode_budget_ms(pr_prev_piece_ms, decode_share, pr_cap, pr_floor)
+ : (pr_cap > 0.0 ? pr_cap : std::max(pr_floor, 200.0));
+ const double since =
+ pr_have_prev ? std::chrono::duration<double, std::milli>(Clock::now() - pr_prev_end).count()
+ : -1.0;
+ if (strata::program::piped_read_lead_in(true, batch_on(), since, lead)) {
+ const int64_t rows0 = pipe_rows;
+ const auto tl = Clock::now();
+ if (!piped_decode(lead, e)) return false;
+ std::fprintf(stderr, "strata batch: a read right after a read: the slots decoded %lld rows in "
+ "%.0f ms first\n", (long long) (pipe_rows - rows0),
+ std::chrono::duration<double, std::milli>(Clock::now() - tl).count());
+ }
+ }
+ int64_t piece = piece_now();
- if (piece <= 0) return sp.run(ids.data() + a0, b0 - a0, a0, e);
+ if (piece <= 0) return run(a0, b0);
for (int64_t q = a0; q < b0;) {
const int64_t r = std::min(b0, q + piece);
const auto tq = Clock::now();
- if (!sp.run(ids.data() + q, r - q, q, e)) return false;
+ if (!run(q, r)) return false;
+ const double piece_ms = std::chrono::duration<double, std::milli>(Clock::now() - tq).count();
+ if (pr_on && r < b0) pr_prev_piece_ms = piece_ms;
+ const int64_t piece_tokens = r - q;
q = r;
if (q >= b0) continue;
int ys = -1;