Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
batch slots: MTP drafts in the pipelined groups, STRATA_PIPED_MTP=N opt-in with --batch-mtp (#4121, #3794)
wannagoxuf17878ac57925
diff --git a/docs/BATCHING.md b/docs/BATCHING.mdindex 6e7b58d39..bc5e3a5ac 100644--- a/docs/BATCHING.md+++ b/docs/BATCHING.md@@ -390,6 +390,66 @@ compares the tier as a whole, not "batch windows only": the solo decode speed momore varied workload the gain may be smaller (not measured). Upstream main runs no tier in batch windows, and thiswas not measured against an upstream binary.+## MTP drafts in the pipelined groups (`STRATA_PIPED_MTP`, opt-in)++The pipelined groups run no drafts. `--batch-mtp` with serial windows (`--batch-groups 1`) keeps one card busy at a+time. `STRATA_PIPED_MTP=N` together with `--batch-mtp` gives both: the pipeline stays, and each slot drafts in it.+Off by default. Without the variable nothing changes.++How it works. A group's window carries, for each of its slots, the slot's token and up to N MTP proposals (at most+`--spec` - 1; the first always, each further one while the draft layer was at least `--spec-min-p` sure of the one+before). The last stage judges each slot's proposals on their own. How many rows to keep is known only then, and by+then the earlier stages already run other groups' windows, which reuse the scratch the commit replays from. So a+small save graph copies those rows right behind each window (`Verifier::batch_launch_rows`), and after the verdict every+stage queues the commit of the kept rows (`Verifier::batch_commit_rows`). The slot's drafter catches up on its own rows+and drafts the next proposals as an asynchronous chain (`MtpDrafter::chain_launch`). A group whose chain has not landed+lets another group start first. Anything that reads a slot's state (an admission, a parked conversation, the solo path+taking a slot back) first drains the pipeline, which also waits for the queued commits.++It needs a layer split with two or more groups, any number of slots per group (as long as a group's window,+slots x (1 + N), is at most 8 rows). When it cannot run, the engine says why and goes on without it: fewer than two+groups left, `STRATA_BATCH_SELF_COMMIT` on, or a window of more than 8 rows.++Measured on a box with 2x RTX A6000 48 GB (`--layer-split auto`), Qwen3.8-Flash-Next IQ3_S, every expert resident in+VRAM, `--batch 4 --batch-groups auto` (2 groups of 2 slots), `tools/serve_load.py`, 256 tokens per request, reasoning+off. Total output tokens per second: the median of the pairs, each pair the median of 3 bursts on a fresh engine start. The base arm+(pipelined groups, no drafts) and `STRATA_PIPED_MTP=2` were run in 5 interleaved pairs, the other arms in 3 pairs. "Default" is+`--batch 4` with no `--batch-groups`, which on exactly two NVIDIA cards means one serial group (`STRATA_BATCH_GROUPS_NV_SERIAL`):++| streams | pipelined, no drafts | `=1` | `=2` | `=3` | serial `--batch-mtp` (`--batch-groups 1`) | default (no flags) |+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |+| 1 (solo path) | 101.7 | 101.4 | 101.5 | 103.0 | 101.5 | 101.7 |+| 2 | 125.7 | 146.3 (+16.0%) | 145.1 (+15.4%) | 138.4 (+10.0%) | 124.0 (-1.3%) | 107.4 (-14.8%) |+| 4 | 171.4 | 188.8 (+10.2%) | 179.5 (+4.7%) | 165.6 (-3.4%) | 139.9 (-18.5%) | 141.0 (-17.8%) |++Gains are the median of the per-pair ratios. `=2` was faster than the base in 5 of 5 pairs at 2 and at 4 streams, `=1` in 3 of 3+pairs at both, `=3` in 3 of 3 at 2 streams and in none at 4. With `--batch 2` (2 groups of 1 slot) and 2 streams `=2` gave 146.8 against 125.7 tok/s (+16.4%, 5 of 5 pairs). One+stream runs on the solo path in every arm and does not change. About 55% of the checked proposals were kept at `=2`+and about 67% at `=1`. The acceptance falls as the window grows, which is why `=1` or `=2` is the choice and `=3` is not.++Best setting here: `=2` with two streams, `=1` with four. More than four streams was not measured, and the gain at+four streams is smaller than at two (not explained). Cost: the slot+drafters and a save buffer per stage (25.5 MiB at `=1` and 38.2 MiB at `=2` with 4 slots), taken from the expert cache.+On the A6000 box the cache stayed fully resident. On a 2x RTX A4000 box where the experts do not all fit, `--batch-mtp` took+165 of 6959 cache slots on the second card; that changes which experts are in VRAM and so, in the last digits, which tokens come out.++The first version of this was written by wannagoxuf (PR #4121), who measured +39% on 4x RTX 4090 with 4 streams+(366 to 510 tok/s at `=2`). That is a different rig and not repeated here.++Exact. Checked with the engine's batch protocol (the method of `tools/batch_test.py`: each prompt alone, then the+prompts together in the slots, token ids compared). On the A6000 box, 4 slots (2 groups of 2) and 2 slots (2 groups of 1)+with the feature unset and with `=1`, `=2` and `=3` (`=2` only for 2 slots): greedy streams, greedy and seeded sampled streams+together, slots that finish at different lengths, a slot cancelled in the middle of a draft, and a 3,000-token prompt read beside the+others. Every slot got the tokens it gets alone, and the same tokens as the build without the feature (a cancelled slot stops one+or two tokens apart, so its tokens are compared up to the shorter). Through the server the same cases (and two clients on four+slots) gave the same text as the build without the feature, apart from one or two streams of 19 per run; two runs of the+build without the feature differ from each other in the same way, so this is not the feature. On a box where the experts+do not all fit in VRAM the drafters' memory changes which experts are cached, and then the text of a feature run differs from a+run without it; inside one run every slot still equals its own solo decode.++Not run: AMD (HIP), the draft layer on the first stage, helper expert caches, more than two stages, SYCL (the SYCL+tree has its own copy of the batch code and does not have this).+## ExactnessA batch row's arithmetic is the single-token window's, so with greedy decoding **every conversation of a batch@@ -423,8 +483,9 @@ counter-based draw (Philox(seed, position)).- By default, batch windows carry no MTP drafts: a conversation in a slot decodes one token per window (the solopath keeps its drafts, which is why a request alone is not put in a slot, and goes back to it when left alone).- `--batch-mtp` uses one proposal per slot. With a layer split it needs each stage on its own GPU, does not combine- with `--batch-groups` above 1 (pipelined slot groups run no drafts, #1413) and has not been run with helper expert- caches.+ with `--batch-groups` above 1 (pipelined slot groups run no drafts, #1413) unless `STRATA_PIPED_MTP` is set with+ one slot per group ([above](#mtp-drafts-in-the-pipelined-groups-strata_piped_mtp-opt-in)), and has not been run+ with helper expert caches.- Repetition / frequency / presence penalties are not applied in batch windows.- A prompt shorter than one chunk is read in one piece (the slots wait for it); a read gives way only at a chunkboundary, and not for pictures.diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hppindex 2b2d9a0ec..e878b9ed9 100644--- a/include/strata/core/verify.hpp+++ b/include/strata/core/verify.hpp@@ -235,6 +235,23 @@ public:/// -1 = an error (err). Serves every layer that has rung so far.int batch_poll(PoolMultiFn pool, void* user, std::string& err);bool batch_busy() const { return b_running_.load(std::memory_order_acquire); }+ // ---- STRATA_PIPED_MTP: the pipelined groups with --batch-mtp's drafts. A group's window holds its slot's token+ // and its drafts, so how many rows to keep is known only once the LAST stage has picked - and by then this stage+ // runs another group's window, which reuses the scratch the commit replays from. So right behind the window a+ // save graph copies those rows (each GDN layer's conv and recurrence inputs, each QSA layer's raw indexer keys, the+ // PLE history snapshot) to the group's own rows [hbase, hbase + S) of the saved buffers, and the commit, queued+ // after the verdict, replays from there. Everything else the commit reads is per slot already.+ /// The saved buffers for `rows` hand-off rows (after init_slots).+ bool init_deferred_commit(int rows, std::string& err);+ bool deferred_commit_on() const { return dc_rows_ > 0; }+ /// The window over the slots `rows` (a slot's rows contiguous) at hand-off rows [hbase, hbase + S), its save right+ /// behind it, and no commit (batch_commit_rows, after the verdict). The PLE history advances at the commit.+ bool batch_launch_rows(const int* rows, int S, int hbase, const int32_t* tokens, const int64_t* pos,+ std::string& err);+ /// Queue the commit of that window's accepted prefixes on this stage's stream (no host wait): `keep` per slot ID,+ /// 1..the slot's rows. `rows`, `S`, `hbase`, `tokens` and `pos` are the window's, as batch_launch_rows got them.+ bool batch_commit_rows(const int* rows, int S, int hbase, const int* keep, const int32_t* tokens,+ const int64_t* pos, std::string& err);/// Serves the layers of this stage's window that have rung (the CPU pool and the flags), with no CUDA call: 1 once/// every layer is served (or the window raises no host doorbells), 0 while one is still to ring. batch_poll/// calls it, unless set_external_serve(true) moved it to a thread of its own (STRATA_PIPED_CONCURRENT: a CUDA@@ -419,6 +436,16 @@ private:void* arena_b_ = nullptr;int64_t last_pos_b_[8] = {};bool capture_batch(const int* rows, int S, int hbase, std::string& err);+ // STRATA_PIPED_MTP (init_deferred_commit): the saved rows, laid out as the window scratch (global layer index) with+ // dc_rows_ rows per layer instead of max_t_; the slots' accepted prefixes (mapped and device, 2 + max_t_ per slot)+ int dc_rows_ = 0;+ bool staging_deferred_ = false; ///< stage_batch for batch_launch_rows: no capture_commit_batch+ float *dc_qkv_ = nullptr, *dc_h_ = nullptr, *dc_gate_ = nullptr, *dc_beta_ = nullptr, *dc_idx_ = nullptr;+ float* dc_hist_ = nullptr;+ void* dc_arena_ = nullptr;+ int32_t *h_commitd_ = nullptr, *m_commitd_ = nullptr, *commitd_ = nullptr;+ std::map<std::vector<int>, cudaGraphExec_t> save_bm_, dcommit_bm_;+ bool capture_deferred(const int* rows, int S, int hbase, std::string& err);/// Free the graph pair of the least recently used batch layout other than `keep` (`evicted`: there was one).bool evict_batch_graph(const std::vector<int>& keep, bool& evicted, std::string& err);/// cudaGraphInstantiate; out of VRAM, batch layouts other than `key` are evicted (LRU) until it fits, and then (adiff --git a/include/strata/program/batch_groups.hpp b/include/strata/program/batch_groups.hppindex b1e3f32cc..9f614d454 100644--- a/include/strata/program/batch_groups.hpp+++ b/include/strata/program/batch_groups.hpp@@ -28,6 +28,7 @@ struct Input {bool adapt_async = false; ///< --adapt-async 1 was asked forbool mtp_serial = false; ///< STRATA_BATCH_MTP_SERIAL=1: --batch-mtp with no --batch-groups runs one serial group (0.1.43)bool nv_serial_faster = false; ///< PROPOSED (0.2.1, owner decides): NVIDIA build, STRATA_BATCH_GROUPS_NV_SERIAL not 0 - two cards run faster in serial windows (z440 2x A4000, aggregate tok/s: +16% at 2 slots, +12% at 4, +15% at 8)+ bool mtp_piped = false; ///< STRATA_PIPED_MTP=1: the pipelined groups draft too (each group carries its slots' rows)};struct Result {@@ -36,6 +37,7 @@ struct Result {Serial serial = Serial::None; ///< `auto` would have pipelined, but this feature needs the serial windowsint would_be = 1; ///< what `auto` would have picked without it (== groups unless `serial`)bool mtp_dropped = false; ///< --batch-mtp met the default pipeline: the pipeline wins, --batch-mtp is off+ bool mtp_piped = false; ///< STRATA_PIPED_MTP: --batch-mtp's drafts run in the pipelined groups};/// The most groups, at most one per stage of the layer split (`later_stages` + 1 GPUs), that divide `batch`.@@ -52,17 +54,24 @@ inline Result resolve(const Input& in) {r.groups = in.given;r.would_be = in.given;const bool defaulted = !in.set && in.later_stages > 0 && in.batch > 1; // the 0.1.41 default- if (!in.auto_given && !defaulted) return r;- r.from_auto = true;- r.would_be = auto_groups(in.later_stages, in.batch);- r.groups = r.would_be;- if (!in.set && r.would_be > 1) { // only the default gives way; `--batch-groups auto` asked for the pipeline- if (in.batch_mtp && in.mtp_serial) r.serial = Serial::BatchMtp;- else if (in.nv_serial_faster && in.later_stages == 1 && !in.batch_mtp && !in.adapt_async) r.serial = Serial::TwoCardsSerialFaster;- else if (in.adapt_async) r.serial = Serial::AdaptAsync;- r.mtp_dropped = in.batch_mtp && r.serial == Serial::None;- if (r.serial != Serial::None) r.groups = 1;+ if (in.auto_given || defaulted) {+ r.from_auto = true;+ r.would_be = auto_groups(in.later_stages, in.batch);+ r.groups = r.would_be;+ if (!in.set && r.would_be > 1) { // only the default gives way; `--batch-groups auto` asked for the pipeline+ if (in.batch_mtp && in.mtp_serial) r.serial = Serial::BatchMtp;+ else if (in.nv_serial_faster && in.later_stages == 1 && !in.batch_mtp && !in.adapt_async) r.serial = Serial::TwoCardsSerialFaster;+ else if (in.adapt_async) r.serial = Serial::AdaptAsync;+ if (r.serial != Serial::None) r.groups = 1;+ r.mtp_dropped = in.batch_mtp && r.serial == Serial::None;+ }}+ // STRATA_PIPED_MTP: the pipelined groups draft too. A group's window carries each of its slots' rows (the slot's+ // token and that slot's own proposals), so any number of slots per group drafts. This sits outside the block+ // above on purpose: with --batch-groups given (a number or `auto`) the operator asked for those groups, and the+ // drafts run in them too. Unset (mtp_piped false) nothing here changes a result.+ r.mtp_piped = in.batch_mtp && in.mtp_piped && r.serial == Serial::None && r.groups > 1;+ if (r.mtp_piped) r.mtp_dropped = false;return r;}diff --git a/src/core/verify.cpp b/src/core/verify.cppindex 5e97b8f52..531efacd1 100644--- a/src/core/verify.cpp+++ b/src/core/verify.cpp@@ -629,6 +629,11 @@ Verifier::~Verifier() {if (kv.second) cudaGraphExecDestroy(kv.second);if (arena_b_) cudaFree(arena_b_);if (h_commitb_) cudaFreeHost(h_commitb_);+ for (auto* m : {&save_bm_, &dcommit_bm_})+ for (auto& kv : *m)+ if (kv.second) cudaGraphExecDestroy(kv.second);+ if (dc_arena_) cudaFree(dc_arena_);+ if (h_commitd_) cudaFreeHost(h_commitd_);if (qcnt_) cudaFree(qcnt_);if (d_spec_) cudaFree(d_spec_);if (tap_file_) std::fclose(tap_file_);@@ -3668,6 +3673,234 @@ bool Verifier::capture_commit_batch(const int* rows, int S, int hbase, std::strireturn made;}+// ---- STRATA_PIPED_MTP: commits after the verdict (see the header)++bool Verifier::init_deferred_commit(int rows, std::string& err) {+ const OnDevice on_device(device_);+ if (slots_.empty()) { err = "verify: init_deferred_commit before init_slots"; return false; }+ if (rows < 1) { err = "verify: init_deferred_commit needs at least one row"; return false; }+ const ModelGeometry& g = *g_;+ const uint64_t R = (uint64_t) rows, C = (uint64_t) g.ssm_conv_channels, HV = (uint64_t) g.ssm_v_heads;+ const uint64_t ID = (uint64_t) g.idx_key_dim;+ const uint64_t nG = (uint64_t) std::max<int64_t>(g.n_gdn_layers(), 1), nQ = (uint64_t) std::max<int64_t>(g.n_qsa_layers(), 1);+ const uint64_t HS = (uint64_t) strata::kernels::NG_HIST * strata::kernels::NG_HC_DIM;+ const uint64_t CB = 2 + (uint64_t) max_t_, nS = (uint64_t) slots_.size();+ const uint64_t floats[6] = {nG * R * C, nG * R * C, nG * R * HV, nG * R * HV, nQ * R * ID, R * HS};+ uint64_t bytes = (nS * CB * 4 + 255) & ~255ull;+ for (uint64_t f : floats) bytes += (f * 4 + 255) & ~255ull;+ if (cudaMalloc(&dc_arena_, bytes) != cudaSuccess) {+ err = "verify: the saved commit rows (" + std::to_string(bytes >> 20) + " MiB) do not fit";+ return false;+ }+ uint8_t* p = (uint8_t*) dc_arena_;+ commitd_ = (int32_t*) p;+ p += (nS * CB * 4 + 255) & ~255ull;+ float** dst[6] = {&dc_qkv_, &dc_h_, &dc_gate_, &dc_beta_, &dc_idx_, &dc_hist_};+ for (int i = 0; i < 6; ++i) {+ *dst[i] = (float*) p;+ p += (floats[i] * 4 + 255) & ~255ull;+ }+ if (!mapped((size_t) (nS * CB * 4 + 16), (void**) &h_commitd_, (void**) &m_commitd_)) {+ err = "verify: the deferred commit staging failed";+ return false;+ }+ dc_rows_ = rows;+ std::fprintf(stderr, "strata verify: pipelined batch windows commit after the verdict (layers [%lld, %lld), %d rows, "+ "%.1f MiB saved rows)\n", (long long) lb_, (long long) le_, rows, (double) bytes / 1048576.0);+ return true;+}++bool Verifier::capture_deferred(const int* rows, int S, int hbase, std::string& err) {+ using namespace strata::kernels;+ const auto key = bkey(rows, S, hbase);+ cudaGraphExec_t& sex = save_bm_[key];+ cudaGraphExec_t& cex = dcommit_bm_[key];+ if (sex != nullptr && cex != nullptr) return true;+ const ModelGeometry& g = *g_;+ const QsaShapes s = shapes_of(g);+ const int64_t C = g.ssm_conv_channels, HV = g.ssm_v_heads, ZV = g.ssm_value_dim, ID = g.idx_key_dim, MT = max_t_;+ const int64_t R = dc_rows_, CB = 2 + MT, nQ = g.n_qsa_layers();+ const uint64_t gdn_floats = (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size ++ (uint64_t) g.ssm_conv_channels * (g.ssm_d_conv - 1);+ const int64_t TS = (s.idx_block - 1) * ID;+ const int64_t HS = (int64_t) NG_HIST * NG_HC_DIM;+ const bool ple = ss_->ple.ready() && ple_stage();+ auto layer_index = [&](int64_t& qsa_index, int64_t& gdn_index) {+ qsa_index = gdn_index = 0;+ for (int64_t l = 0; l < lb_; ++l) (is_qsa_layer(g, l) ? qsa_index : gdn_index) += 1;+ };+ auto finish = [&](cudaGraphExec_t& ex, bool ok, const char* what) {+ cudaGraph_t graph = nullptr;+ const cudaError_t ce = cudaStreamEndCapture(cs_, &graph);+ if (!ok || ce != cudaSuccess) {+ if (graph) cudaGraphDestroy(graph);+ if (ok) err = std::string(what) + cudaGetErrorString(ce);+ return false;+ }+ const bool made = instantiate_evicting(ex, graph, key, what, err);+ cudaGraphDestroy(graph);+ return made;+ };+ // the save: the window's rows [0, S) of the scratch the commit replays -> the group's saved rows [hbase, +S)+ if (sex == nullptr) {+ if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) {+ err = "verify: begin save capture failed";+ return false;+ }+ bool ok = true;+ int64_t qsa_index = 0, gdn_index = 0;+ layer_index(qsa_index, gdn_index);+ auto copy = [&](float* dst, const float* src, int64_t n) {+ ok = ok && cudaMemcpyAsync(dst, src, (size_t) n * sizeof(float), cudaMemcpyDeviceToDevice, cs_) == cudaSuccess;+ };+ for (int64_t l = lb_; l < le_; ++l) {+ if (!is_qsa_layer(g, l)) {+ copy(dc_qkv_ + (size_t) (gdn_index * R + hbase) * C, qkv_L_ + (size_t) gdn_index * MT * C, S * C);+ copy(dc_h_ + (size_t) (gdn_index * R + hbase) * C, h_L_ + (size_t) gdn_index * MT * C, S * C);+ copy(dc_gate_ + (size_t) (gdn_index * R + hbase) * HV, gate_L_ + (size_t) gdn_index * MT * HV, S * HV);+ copy(dc_beta_ + (size_t) (gdn_index * R + hbase) * HV, beta_L_ + (size_t) gdn_index * MT * HV, S * HV);+ ++gdn_index;+ } else {+ copy(dc_idx_ + (size_t) (qsa_index * R + hbase) * ID, idx_raw_L_ + (size_t) qsa_index * MT * ID, S * ID);+ ++qsa_index;+ }+ }+ if (ple) copy(dc_hist_ + (size_t) hbase * HS, hist_snap_, S * HS);+ if (!finish(sex, ok, "verify: save capture: ")) {+ if (ok && err.empty()) err = "verify: save capture failed";+ return false;+ }+ }+ // the commit: as capture_commit_batch, replaying the saved rows; the accepted prefixes come from commitd_+ if (cex == nullptr) {+ if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) {+ err = "verify: begin deferred commit capture failed";+ return false;+ }+ bool ok = true;+ try {+ for (int t = 0; t < S; ++t)+ if (t == 0 || rows[t] != rows[t - 1])+ copy_i32_from_mapped(commitd_ + (size_t) rows[t] * CB, m_commitd_ + (size_t) rows[t] * CB, CB, cs_);+ int64_t qsa_index = 0, gdn_index = 0;+ layer_index(qsa_index, gdn_index);+ for (int64_t l = lb_; l < le_ && ok; ++l) {+ const LayerView v(*wt_, l);+ if (!is_qsa_layer(g, l)) {+ const WeightRef* wnm = need(v, "ssm_norm.weight", err);+ if (!wnm) { ok = false; break; }+ for (int t = 0; t < S;) {+ const int first = t;+ while (t < S && rows[t] == rows[first]) ++t;+ SessionState& sx = *slots_[(size_t) rows[first]];+ const int32_t* keep = commitd_ + (size_t) rows[first] * CB;+ float* state = sx.gdn_state + (size_t) (gdn_index - sx.gdn_ord0) * gdn_floats;+ float* conv = state + (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size;+ const int64_t row = gdn_index * R + hbase + first;+ gdn_conv_commit(conv, dc_qkv_ + (size_t) row * C, (int) C, keep, cs_);+ // z only feeds the norm's output, which the commit discards (y_dummy_): the window scratch will do+ gdn_step_norm_multi(state, dc_h_ + (size_t) row * C, (int) C, dc_gate_ + (size_t) row * HV,+ dc_beta_ + (size_t) row * HV, z_ + (size_t) first * ZV,+ (const float*) wnm->data, EPS, y_dummy_ + (size_t) first * ZV,+ (int) g.ssm_k_heads, (int) HV, t - first, keep, cs_,+ t - first > 1 ? t - first : 0);+ }+ ++gdn_index;+ } else {+ const WeightRef* wikn = need(v, "indexer.k_norm.weight", err);+ if (!wikn) { ok = false; break; }+ for (int t = 0; t < S;) {+ const int first = t;+ while (t < S && rows[t] == rows[first]) ++t;+ const QsaState& st = slots_[(size_t) rows[first]]->qsa_states[qsa_index];+ copy_from_mapped(st.idx_tail, tail_snap_b_ + ((size_t) rows[first] * nQ + qsa_index) * TS, TS, cs_);+ const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos};+ for (int u = first; u < t; ++u)+ native_qsa_indexer_append(dc_idx_ + (size_t) (qsa_index * R + hbase + u) * ID,+ commitd_ + (size_t) rows[first] * CB + 2 + (u - first), 0,+ (const float*) wikn->data, EPS, ib, s, st.max_cells,+ rope_scaling(), cs_);+ }+ ++qsa_index;+ }+ }+ if (ok && ple)+ for (int t = 0; t < S;) {+ const int first = t;+ while (t < S && rows[t] == rows[first]) ++t;+ copy_indexed(slots_[(size_t) rows[first]]->ple_hist, dc_hist_ + (size_t) (hbase + first) * HS, HS,+ commitd_ + (size_t) rows[first] * CB + 1, HS, cs_);+ }+ } catch (const std::exception& e) {+ err = std::string("verify deferred commit: ") + e.what();+ ok = false;+ }+ if (!finish(cex, ok, "verify: deferred commit capture: ")) {+ if (ok && err.empty()) err = "verify: deferred commit capture failed";+ return false;+ }+ }+ return true;+}++bool Verifier::batch_launch_rows(const int* rows, int S, int hbase, const int32_t* tokens, const int64_t* pos,+ std::string& err) {+ const OnDevice on_device(device_);+ if (dc_rows_ < 1) { err = "verify: batch_launch_rows without init_deferred_commit"; return false; }+ if (b_running_) { err = "verify: batch_launch_rows while this stage is busy"; return false; }+ if (S < 1 || hbase < 0 || hbase + S > dc_rows_) { err = "verify: a pipelined window outside the saved rows"; return false; }+ if (pre_launch_ && !pre_launch_(cs_, err)) return false;+ staging_deferred_ = true;+ const bool staged = stage_batch(rows, S, hbase, tokens, pos, err);+ staging_deferred_ = false;+ if (!staged || !capture_deferred(rows, S, hbase, err)) return false;+ const auto key = bkey(rows, S, hbase);+ cudaError_t le = cudaGraphLaunch(exec_bm_[key], cs_);+ if (le == cudaSuccess) le = cudaGraphLaunch(save_bm_[key], cs_); // the rows the commit replays, kept for it+ if (le != cudaSuccess) { err = std::string("verify: batch launch: ") + cudaGetErrorString(le); return false; }+ (void) cudaStreamQuery(cs_);+ b_k_ = 0;+ b_steps_ = le_ - lb_;+ ring_now();+ b_running_.store(true, std::memory_order_release); // last: an external server reads the rest after it+ return true;+}++bool Verifier::batch_commit_rows(const int* rows, int S, int hbase, const int* keep, const int32_t* tokens,+ const int64_t* pos, std::string& err) {+ const OnDevice on_device(device_);+ const auto it = dcommit_bm_.find(bkey(rows, S, hbase));+ if (it == dcommit_bm_.end() || it->second == nullptr) { err = "verify: a deferred commit without its window"; return false; }+ const int64_t CB = 2 + max_t_;+ // Each slot's accepted prefix. The graph reads it from mapped memory when it runs; this slot's entry is written+ // again only at its next verdict, which comes after its next window has passed this stage - behind this commit.+ for (int t = 0; t < S;) {+ const int first = t;+ while (t < S && rows[t] == rows[first]) ++t;+ const int n = keep[rows[first]];+ if (n < 1 || n > t - first) { err = "verify: accepted prefix is outside its slot group"; return false; }+ int32_t* c = h_commitd_ + (size_t) rows[first] * CB;+ c[0] = n;+ c[1] = n - 1;+ for (int64_t j = 0; j < max_t_; ++j) c[2 + j] = j < n ? (int32_t) pos[first + j] : -1;+ }+ std::atomic_thread_fence(std::memory_order_seq_cst);+ const cudaError_t le = cudaGraphLaunch(it->second, cs_);+ if (le != cudaSuccess) { err = std::string("verify: deferred commit launch: ") + cudaGetErrorString(le); return false; }+ (void) cudaStreamQuery(cs_);+ if (ple_stage()) // the host's side of the commit (the hash's last two tokens), the accepted rows only+ for (int t = 0; t < S;) {+ const int first = t;+ while (t < S && rows[t] == rows[first]) ++t;+ SessionState& sx = *slots_[(size_t) rows[first]];+ for (int u = first; u < first + keep[rows[first]]; ++u) {+ sx.ple_prev[0] = sx.ple_prev[1];+ sx.ple_prev[1] = tokens[u];+ }+ }+ return true;+}+bool Verifier::evict_batch_graph(const std::vector<int>& keep, bool& evicted, std::string& err) {evicted = false;auto old = exec_bm_.end();@@ -3780,7 +4013,9 @@ bool Verifier::stage_batch(const int* rows, int S, int hbase, const int32_t* tok!evict_batch_graph(key, evicted, err))return false;bm_used_[key] = ++bm_tick_;- if (!capture_batch(rows, S, hbase, err) || !capture_commit_batch(rows, S, hbase, err)) return false;+ // (a pipelined window that commits after the verdict has its own commit graph: capture_deferred)+ if (!capture_batch(rows, S, hbase, err) || (!staging_deferred_ && !capture_commit_batch(rows, S, hbase, err)))+ return false;const Clock::time_point t0 = Clock::now();const QsaShapes s = shapes_of(g);for (int t = 0; t < S; ++t) {diff --git a/src/program/batch_groups_test.cpp b/src/program/batch_groups_test.cppindex 5debcd29d..564ad3dc9 100644--- a/src/program/batch_groups_test.cpp+++ b/src/program/batch_groups_test.cpp@@ -3,6 +3,7 @@// 1. `auto` picks the most groups, at most one per stage, that divide the slots;// 2. a layer split with --batch >= 2 and no --batch-groups means `auto` (0.1.41);// 3. --batch-mtp (or --adapt-async 1) turns that default into one group, and says which feature did it;+// with STRATA_PIPED_MTP and one slot per group, --batch-mtp keeps the pipeline and drafts in it;// 4. a number or `auto` given explicitly is honoured, with or without those features;// 5. no layer split: the number given (1) stays.#include "strata/program/batch_groups.hpp"@@ -58,6 +59,71 @@ int main() {"STRATA_BATCH_MTP_SERIAL: one group, because of --batch-mtp");check(bg::serial_what(r.serial)[0] == '-', "the reason names the feature");}+ { // STRATA_PIPED_MTP: the pipelined groups draft too, with any number of slots per group+ bg::Input i = in(4, 3);+ i.batch_mtp = true;+ i.mtp_piped = true;+ auto r = bg::resolve(i);+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,+ "STRATA_PIPED_MTP, 4 slots on 4 GPUs: 4 groups of 1 slot, and the drafts stay");+ i.batch = 8; // two slots per group: the group's window carries both slots' rows, each with its own drafts+ r = bg::resolve(i);+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,+ "STRATA_PIPED_MTP, 8 slots on 4 GPUs: 4 groups of 2 slots, and the drafts stay");+ i.batch = 4;+ i.mtp_serial = true; // asked for the serial windows: those win+ r = bg::resolve(i);+ check(r.groups == 1 && !r.mtp_piped && r.serial == bg::Serial::BatchMtp,+ "STRATA_PIPED_MTP with STRATA_BATCH_MTP_SERIAL: one serial group");+ i.mtp_serial = false;+ i.batch_mtp = false;+ r = bg::resolve(i);+ check(r.groups == 4 && !r.mtp_piped && !r.mtp_dropped, "STRATA_PIPED_MTP without --batch-mtp: nothing to draft");+ i.batch_mtp = true;+ i.set = true;+ i.given = 1; // one group: the serial windows, where --batch-mtp's own drafts run+ r = bg::resolve(i);+ check(r.groups == 1 && !r.mtp_piped && !r.mtp_dropped, "STRATA_PIPED_MTP, --batch-groups 1: the serial path, --batch-mtp runs there");+ }+ { // 0.1.49.2 port: --batch-groups given EXPLICITLY (a number or `auto`) must not silently+ // drop the drafts: the operator asked for those groups, and one slot per group still drafts.+ bg::Input i = in(4, 3);+ i.batch_mtp = true;+ i.mtp_piped = true;+ i.set = true;+ i.auto_given = true; // --batch-groups auto+ i.given = 4;+ auto r = bg::resolve(i);+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,+ "STRATA_PIPED_MTP, --batch-groups auto given: 4 groups of 1 slot, and the drafts stay");+ i.auto_given = false; // --batch-groups 4+ r = bg::resolve(i);+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,+ "STRATA_PIPED_MTP, --batch-groups 4 given: 4 groups of 1 slot, and the drafts stay");+ i.given = 2; // two slots per group: the group's window carries both slots' rows+ r = bg::resolve(i);+ check(r.groups == 2 && r.mtp_piped && !r.mtp_dropped,+ "STRATA_PIPED_MTP, --batch-groups 2 given: 2 groups of 2 slots, and the drafts stay");+ }+ { // STRATA_PIPED_MTP unset: nothing changes, whatever --batch-mtp and the groups say+ bg::Input i = in(4, 1);+ i.batch_mtp = true;+ auto r = bg::resolve(i);+ check(r.groups == 2 && !r.mtp_piped && r.mtp_dropped && r.serial == bg::Serial::None,+ "unset, --batch-mtp, nothing given: the pipeline wins, the drafts are dropped (as before)");+ i.set = true;+ i.given = 2;+ r = bg::resolve(i);+ check(r.groups == 2 && !r.mtp_piped && !r.mtp_dropped, "unset, --batch-mtp, --batch-groups 2: not 'dropped' here (as before)");+ i.given = 1;+ r = bg::resolve(i);+ check(r.groups == 1 && !r.mtp_piped && !r.mtp_dropped, "unset, --batch-mtp, --batch-groups 1: serial windows keep the drafts");+ i.set = false;+ i.mtp_serial = true;+ r = bg::resolve(i);+ check(r.groups == 1 && !r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::BatchMtp,+ "unset, STRATA_BATCH_MTP_SERIAL: one serial group (as before)");+ }{bg::Input i = in(2, 1);i.adapt_async = true;diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 75d404ac9..cb36cd56a 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -5381,6 +5381,7 @@ int main(int argc, char** argv) {#if !defined(STRATA_USE_HIP){ const char* nv = std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"); bin.nv_serial_faster = nv == nullptr || nv[0] != '0'; } // PROPOSED 0.2.1 default (NVIDIA, 2 cards)#endif+ { const char* mp = std::getenv("STRATA_PIPED_MTP"); bin.mtp_piped = mp != nullptr && mp[0] != '\0' && mp[0] != '0'; }return strata::program::batch_groups::resolve(bin);};const char* batch_mtp_env = std::getenv("STRATA_BATCH_MTP");@@ -5391,22 +5392,46 @@ int main(int argc, char** argv) {: o.spec < 2 ? "it needs --spec T (T >= 2)" : split_same? "it needs each stage of a layer split on its own GPU": !o.serve ? "it needs --serve" : nullptr;- // the groups it would run with (the default gives way to it: one group; a number or auto given pipelines)- if (why == nullptr && !stages.empty() && batch_groups_for(true).groups > 1)+ // the groups it would run with (the default gives way to it: one group; a number or auto given pipelines;+ // STRATA_PIPED_MTP with one slot per group: the pipelined groups draft)+ if (why == nullptr && !stages.empty() && batch_groups_for(true).groups > 1 && !batch_groups_for(true).mtp_piped)why = batch_groups_for(true).mtp_dropped? "a layer split with --batch pipelines the slots in groups by default and the pipelined groups run no slot ""drafts (4 streams on 2 GPUs: 25-30% faster than the serial MTP windows); --batch-groups 1 or ""STRATA_BATCH_MTP_SERIAL=1 keeps --batch-mtp with serial windows (fewer streams, or when prompts arrive ""rarely, it can be the faster choice)": "it does not combine with --batch-groups (pipelined slot groups) yet";- // (a layer split with helper-GPU expert caches or --remote-expert-opt has not been run with slot drafters)- if (why == nullptr && !stages.empty() && (remote_caches || o.remote_expert_opt))+ // (a layer split with helper-GPU expert caches has not been run with slot drafters. --remote-expert-opt alone does+ // not count: its object is only made with a helper cache on CUDA0 (`remote_opt` below), so without one it does nothing)+ if (why == nullptr && !stages.empty() && remote_caches)why = "it is not run with helper expert caches on a layer split";if (why != nullptr) {std::fprintf(stderr, "strata generate: WARNING: --batch-mtp is off: %s\n", why);batch_mtp = false;}}+ // STRATA_PIPED_MTP (opt-in): --batch-mtp's proposals in the pipelined groups of a layer split, one slot per group;+ // each stage commits a group's window once the last stage has judged its proposals+ // (Verifier::batch_launch_rows / batch_commit_rows)+ bool piped_mtp = batch_mtp && !stages.empty() && batch_groups_for(true).mtp_piped; // (final once the slots that fit are known, below)+ // STRATA_PIPED_MTP=N: up to N proposals per slot and window (at most --spec - 1); the first always, each further+ // one while the draft layer is at least --spec-min-p sure of it, as the solo path extends its windows+ int pm_drafts = 0;+ if (piped_mtp) {+ pm_drafts = std::clamp(std::atoi(std::getenv("STRATA_PIPED_MTP")), 1,+ std::max(1, std::min(o.spec - 1, (int) strata::kernels::kVerifyMaxT - 1)));+ }+ // The verifiers' window: the solo path's drafts (--spec rows), the plain batch windows (the slots) and, with+ // STRATA_PIPED_MTP, a group's rows: its slots, each its token and its proposals ((slots) * (1 + proposals), one slot+ // per group = at most --spec rows). A function, not a number: --batch (the slots that fit) and the groups are final+ // only after this point, at the verifiers' init, and the plain value must stay what it was there.+ auto batch_max_t = [&]() -> int {+ if (batch_mtp && !piped_mtp) return (int) strata::kernels::kVerifyMaxT;+ const int plain = std::max(o.spec, std::min(o.batch, (int) strata::kernels::kVerifyMaxT));+ const int group_rows = (piped_mtp && o.batch_groups > 1 && o.batch > 0)+ ? std::min<int>(strata::kernels::kVerifyMaxT, (o.batch / o.batch_groups) * (1 + pm_drafts)) : 0;+ return std::max(plain, group_rows);+ };// The most slots, up to n, whose windows hold at most kVerifyMaxT rows each: n itself up to that, more only as// pipelined groups on a layer split - the groups as resolved below (a number given, auto, or the default's one// group for --adapt-async 1), dividing the slots, each at most kVerifyMaxT, all at most one group's worth per@@ -5528,6 +5553,34 @@ int main(int argc, char** argv) {}}+ // STRATA_PIPED_MTP is decided again now that the slots that fit and the groups are final. Recommend, never force: when+ // it cannot run it is said and left off, and --batch-mtp goes on as it would without it.+ if (piped_mtp) {+ const char* why = nullptr;+ char buf[160];+ const char* sc = std::getenv("STRATA_BATCH_SELF_COMMIT");+ if (o.batch < 2 || o.batch_groups <= 1) {+ std::snprintf(buf, sizeof buf, "it needs 2 or more pipelined groups and %d slot%s in %d group%s are left", o.batch,+ o.batch == 1 ? "" : "s", o.batch_groups, o.batch_groups == 1 ? "" : "s");+ why = buf;+ } else if (sc != nullptr && sc[0] != '\0' && sc[0] != '0') {+ why = "it does not combine with STRATA_BATCH_SELF_COMMIT (the window would keep its rows itself)";+ } else if ((o.batch / o.batch_groups) * (1 + pm_drafts) > (int) strata::kernels::kVerifyMaxT) {+ std::snprintf(buf, sizeof buf, "a group's window would hold %d rows (%d slots x %d), more than %d",+ (o.batch / o.batch_groups) * (1 + pm_drafts), o.batch / o.batch_groups, 1 + pm_drafts,+ (int) strata::kernels::kVerifyMaxT);+ why = buf;+ }+ if (why != nullptr) {+ std::fprintf(stderr, "strata generate: WARNING: STRATA_PIPED_MTP is off: %s\n", why);+ piped_mtp = false;+ } else {+ std::fprintf(stderr, "strata generate: STRATA_PIPED_MTP: %d pipelined group%s of %d slot%s draft up to %d MTP "+ "proposal%s per slot each window\n", o.batch_groups, o.batch_groups == 1 ? "" : "s",+ o.batch / o.batch_groups, o.batch / o.batch_groups == 1 ? "" : "s", pm_drafts,+ pm_drafts > 1 ? "s" : "");+ }+ }// Secure MTP's CUDA0 allocations before the large host arena is registered with both CUDA contexts.// In particular WDDM can refuse the draft weights after mapping tens of GiB of host pages.strata::core::MtpDrafter mtp;@@ -5635,12 +5688,15 @@ int main(int argc, char** argv) {// runs on the first stage, as for the solo drafter (`mtp.set_ple_session(&ss)` above).for (int b = 0; b < o.batch; ++b) {auto d = std::make_unique<strata::core::MtpDrafter>();+ // STRATA_PIPED_MTP: the slot's proposal is drafted as an asynchronous chain (chain_launch), so the+ // pipeline goes on while it runs: the chain graphs carry the forcing kernel (a no-op here)+ if (piped_mtp) d->set_force_capture(true);if (!d->load(o.mtp, draft_geometry, *bslot_ss.back()[(size_t) b], o.spec, err, o.mtp_window, &mtp)) {std::fprintf(stderr, "strata generate: batch MTP slot %d: %s%s\n", b, err.c_str(),vram_free_note().c_str());return 1;}- d->set_max_drafts(1); // the first candidate verifies one proposal per slot+ d->set_max_drafts(piped_mtp ? pm_drafts : 1); // the serial windows verify one proposal per slotd->set_ple_session(bslot_ss[0][(size_t) b].get());slot_mtp.push_back(std::move(d));}@@ -9146,7 +9202,9 @@ int main(int argc, char** argv) {if (n_stages > 1) {// pipelined slot groups write their rows at hbase = group * rows-per-group, so with more than// kVerifyMaxT slots in groups the hand-off holds one row per slot (otherwise kVerifyMaxT, as before)- const int hrows = !batch_mtp && o.batch_groups > 1 ? std::max<int>(strata::kernels::kVerifyMaxT, o.batch)+ // (STRATA_PIPED_MTP: per slot its token and up to pm_drafts proposals)+ const int hrows = piped_mtp ? std::max<int>(strata::kernels::kVerifyMaxT, (1 + pm_drafts) * o.batch)+ : !batch_mtp && o.batch_groups > 1 ? std::max<int>(strata::kernels::kVerifyMaxT, o.batch): strata::kernels::kVerifyMaxT;const size_t hb = (size_t) hrows * (size_t) strata::core::Verifier::handoff_floats(g) * sizeof(float);std::vector<float*> hand((size_t) n_stages - 1, nullptr);@@ -9215,9 +9273,9 @@ int main(int argc, char** argv) {// --batch-mtp: every stage runs the same 2-rows-per-slot windows as the first (up to kVerifyMaxT// rows, the hand-off's size) and bounds its captured batch layouts like the first (below)if (batch_mtp) gs.ver.set_batch_graph_limit(64);- ok_s = gs.ver.init(gs.wt, g, gs.ss, vs, gs.head.loaded() ? &gs.head : nullptr,- batch_mtp ? strata::kernels::kVerifyMaxT- : std::max(o.spec, std::min(o.batch, (int) strata::kernels::kVerifyMaxT)), err);+ // (STRATA_PIPED_MTP: a group's window is its one slot's token and proposals, at most --spec rows,+ // so the stage keeps the windows it has without --batch-mtp - the solo path's too)+ ok_s = gs.ver.init(gs.wt, g, gs.ss, vs, gs.head.loaded() ? &gs.head : nullptr, batch_max_t(), err);split_drive.cache_base[st] = gs.cache.device_slot(0);split_drive.cache_slot_off[st] = gs.cache.slot_offsets();split_drive.pcie_num[st] = pcie_num_of(gs.pcie_frac);@@ -9297,7 +9355,7 @@ int main(int argc, char** argv) {if (pcie_balance() && n_stages == 1) ver.set_pcie_balance(&drive.d.pcie_model);if (batch_mtp) ver.set_batch_graph_limit(64); // --batch-mtp only: slot rotation makes many layouts (LRU); 0.1.39 keeps allif (!ver.init(wt, g, ss, vh, native_head.loaded() || prefill_head_alias_bound ? &native_head : nullptr,- batch_mtp ? strata::kernels::kVerifyMaxT : std::max(o.spec, std::min(o.batch, (int) strata::kernels::kVerifyMaxT)), err) ||+ batch_max_t(), err) ||(use_mtp && !prefill_head_alias_bound && !mtp.bind(last_st && !mtp_first ? last_st->wt : wt, last_st ? &last_st->head : &native_head,pipe ? pl_mtp_R : ver.final_R_all(), err))) {std::fprintf(stderr, "strata serve: %s\n", err.c_str());@@ -9321,7 +9379,8 @@ int main(int argc, char** argv) {}slot_mtp_rows.emplace_back(r, free_slot_mtp_rows);if (!slot_mtp[(size_t) b]->bind(last_st ? last_st->wt : wt, last_st ? &last_st->head : &native_head, r, err,- &mtp)) {+ &mtp) ||+ (piped_mtp && !slot_mtp[(size_t) b]->prepare_chain(err))) { // STRATA_PIPED_MTP: the chain graphsstd::fprintf(stderr, "strata serve: batch MTP slot %d: %s%s\n", b, err.c_str(),vram_free_note().c_str());return 1;@@ -9339,6 +9398,12 @@ int main(int argc, char** argv) {std::fprintf(stderr, "strata serve: --batch, stage %zu: %s\n", k + 1, err.c_str());return 1;}+ // STRATA_PIPED_MTP: the rows a group's commit replays after the verdict, one hand-off row each+ if (piped_mtp && bslot_ss.size() > 1 &&+ !vk.init_deferred_commit(std::max<int>(strata::kernels::kVerifyMaxT, (1 + pm_drafts) * o.batch), err)) {+ std::fprintf(stderr, "strata serve: STRATA_PIPED_MTP, stage %zu: %s\n", k + 1, err.c_str());+ return 1;+ }}}for (int st = 0; st < n_stages && n_stages > 1; ++st) {@@ -11490,6 +11555,7 @@ int main(int argc, char** argv) {int32_t x = 0; ///< the token the next window feeds (not yet in the slot's state)std::array<int32_t, strata::kernels::kVerifyMaxT> draft{}; ///< the slot's next MTP proposalbool draft_ready = false;+ int n_drafts = 1; ///< STRATA_PIPED_MTP: how many of `draft` its next window checksint64_t p = 0; ///< its positionint64_t produced = 0, max_new = 0;Clock::time_point t0;@@ -11912,12 +11978,19 @@ int main(int argc, char** argv) {const int n_pipe = (int) stages.size() + 1;const bool piped = o.batch > 0 && o.batch_groups > 1 && n_pipe > 1;strata::core::StageInFlightSampler stage_in_flight(n_pipe, piped && batch_telemetry_enabled);- if (piped && batch_mtp) // #1413: said, not silent+ const int GS = piped ? o.batch / o.batch_groups : o.batch;+ // STRATA_PIPED_MTP: the pipelined groups draft too - a group's window carries each of its slots' rows (the slot's+ // token and that slot's own proposals), so any number of slots per group drafts - and every stage saves the rows+ // their commit replays+ const int pmtp_rows_per_slot = 1 + pm_drafts; // the slot's token + its proposals+ const int pmtp_rows_per_group = GS * pmtp_rows_per_slot; // the group's slots+ const bool pmtp = piped && piped_mtp && ver.deferred_commit_on() &&+ pmtp_rows_per_group <= batch_max_t();+ if (piped && batch_mtp && !pmtp) // #1413: said, not silentstd::fprintf(stderr,"strata generate: WARNING: --batch-groups %d runs the pipelined path, which builds one row per slot: ""--batch-mtp's per-slot MTP drafts do not run there (their drafters still hold VRAM, about 0.9 GB per ""slot). Drop --batch-mtp, or use --batch-groups 1.\n", o.batch_groups);- const int GS = piped ? o.batch / o.batch_groups : o.batch;if (GS > strata::kernels::kVerifyMaxT && !batch_mtp) { // (slots_in_windows keeps it so: a group's window is// PGroup's rows, and the serial windows reach only the first kVerifyMaxT slots)std::fprintf(stderr, "strata generate: --batch %d in %d group%s: more than %d rows per window\n",@@ -11934,6 +12007,10 @@ int main(int argc, char** argv) {int S = 0; ///< slots in its window: up to its last active one (idle ones cost rows)int active = 0; ///< live slots at launch; S can also include idle padding rowsClock::time_point started; ///< the group window's first stage launch+ // STRATA_PIPED_MTP (pmtp): S counts ROWS - each of the group's slots' token, then that slot's proposals - at+ // hand-off rows [hbase, hbase + S)+ int rows[strata::kernels::kVerifyMaxT] = {};+ int hbase = 0;};std::vector<PGroup> pg((size_t) (piped ? o.batch_groups : 0));std::vector<int> stage_group((size_t) n_pipe, -1);@@ -11955,6 +12032,159 @@ int main(int argc, char** argv) {std::mutex bell_mu;std::condition_variable bell_cv;std::atomic<bool> bell_waiting{false};+ // STRATA_PIPED_MTP: a slot whose next proposals are drafting (its chain in flight on the drafter's stream), and+ // per slot the event after the copy of its rows on the last stage's stream, which the chain waits for+ std::vector<char> pm_drafting(bs.size(), 0);+ std::vector<cudaEvent_t> pm_copied(pmtp ? bs.size() : 0, nullptr);+ if (pmtp) {+ const strata::core::OnDevice on_last(stage_verifier(n_pipe - 1).device());+ for (cudaEvent_t& e : pm_copied)+ if (cudaEventCreateWithFlags(&e, cudaEventDisableTiming) != cudaSuccess) {+ std::fprintf(stderr, "strata serve: STRATA_PIPED_MTP: event creation failed\n");+ return 1;+ }+ }+ // the chain of slot b has landed: its proposals are the slot's next drafts (1), still drafting (0), or an error (-1)+ auto pmtp_poll_draft = [&](int b, bool wait) -> int {+ if (!pm_drafting[(size_t) b]) return 1;+ for (;;) {+ const int r = slot_mtp[(size_t) b]->chain_poll(err);+ if (r < 0) { std::printf("ERR batch MTP slot %d: %s\n", b, err.c_str()); return -1; }+ if (r == 1) break;+ if (!wait) return 0;+ }+ pm_drafting[(size_t) b] = 0;+ // the first proposal always; another while the draft layer was at least --spec-min-p sure of the one before+ // (the solo path's rule, MtpDrafter::draft)+ BSlot& sl = bs[(size_t) b];+ const int32_t* tok = slot_mtp[(size_t) b]->chain_tok();+ const float* prob = slot_mtp[(size_t) b]->chain_prob();+ sl.n_drafts = 1;+ sl.draft[0] = tok[0];+ while (sl.n_drafts < pm_drafts && prob[sl.n_drafts - 1] >= (float) o.spec_min_p) {+ sl.draft[(size_t) sl.n_drafts] = tok[sl.n_drafts];+ ++sl.n_drafts;+ }+ sl.draft_ready = true;+ return 1;+ };+ // STRATA_PIPED_MTP: the last stage has picked for group gi's window (its slot's token and G.S - 1 proposals).+ // Judge the proposals, queue every stage's commit of the kept rows (each stage runs it before that group's next+ // window there: the same stream), emit them, and start drafting the next proposals.+ auto pmtp_verdict = [&](PGroup& G, strata::core::Verifier& vk, const int32_t* outb) -> bool {+ auto is_eos = [&](int32_t y) {+ return std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) y) != o.eos_ids.end();+ };+ // each slot of the group owns a contiguous block of the window's rows: its token, then its own proposals+ struct Blk { int b = 0, f = 0, n = 0, keep = 1; };+ std::vector<Blk> blks;+ std::vector<int> keep(bs.size(), 1);+ for (int t = 0; t < G.S;) {+ const int b = G.rows[t];+ const int f = t;+ while (t < G.S && G.rows[t] == b) ++t;+ BSlot& sl = bs[(size_t) b];+ // proposal j stands when the target picked it after the row before it (and every earlier one stood); the+ // group was started only with proposals there is room to emit+ Blk blk;+ blk.b = b;+ blk.f = f;+ blk.n = t - f;+ while (blk.keep < blk.n && !sl.stop && outb[f + blk.keep - 1] == G.tok[f + blk.keep] &&+ !is_eos(outb[f + blk.keep - 1]))+ ++blk.keep;+ keep[(size_t) b] = blk.keep;+ bt_accepted += blk.keep - 1;+ bt_tokens += blk.n - 1; // (proposals checked)+ blks.push_back(blk);+ }+ for (int k = 0; k < n_pipe; ++k)+ if (!stage_verifier(k).batch_commit_rows(G.rows, G.S, G.hbase, keep.data(), G.tok, G.pos, err)) {+ std::printf("ERR %s\n", err.c_str());+ return false;+ }+ // #879: a row with non-finite logits: the slot that owns it ends, nothing of it is emitted+ for (int t = 0; t < G.S; ++t)+ if (vk.nonfinite_in(t)) {+ const int b = G.rows[t];+ BSlot& sl = bs[(size_t) b];+ std::fprintf(stderr, "strata batch: non-finite logits (#879) in slot %d at position %lld: it ends (BDONE "+ "nonfinite) and the next request reads its prompt from token 0\n", b, (long long) sl.p);+ const double ms = std::chrono::duration<double, std::milli>(Clock::now() - sl.t0).count();+ std::printf("BDONE %d %lld nonfinite %.1f\n", b, (long long) sl.produced, ms);+ sl.active = false;+ sl.cached = false;+ sl.draft_ready = false;+ sl.checks.clear();+ nan_flush = true;+ }+ for (const Blk& blk : blks) {+ BSlot& sl = bs[(size_t) blk.b];+ if (!sl.active) continue; // it ended above (non-finite logits) or before this window+ for (int j = 0; j < blk.keep; ++j) {+ const int32_t y = outb[blk.f + j];+ std::printf("BT %d %d\n", blk.b, (int) y);+ ++sl.produced;+ ++bt_rows;+ ++pipe_rows;+ const char* fin = is_eos(y) ? "stop" : sl.stop ? "cancel" : sl.produced >= sl.max_new ? "length"+ : sl.p + 2 > o.max_context ? "length" : nullptr;+ sl.ids.push_back(G.tok[blk.f + j]);+ if (fin != nullptr) {+ const double ms = std::chrono::duration<double, std::milli>(Clock::now() - sl.t0).count();+ std::printf("BDONE %d %lld %s %.1f\n", blk.b, (long long) sl.produced, fin, ms);+ sl.active = false;+ sl.draft_ready = false;+ // a cache again (#857), as on the pipelined path without drafts: final once its commits have run+ // (pmtp_settle, before anything reads the slot's sessions)+ sl.cached = o.prompt_cache > 0 && !sl.img && !sl.tainted;+ break;+ }+ sl.x = y;+ sl.p += 1;+ }+ }+ // the next proposal: each slot's drafter catches up on ITS OWN rows of this window and drafts from the last+ // kept one, as a chain on its own stream (pmtp_poll_draft collects it before the group's next window). The+ // rows - the last stage's final residuals - are copied on THAT stage's stream, so its next window, which+ // rewrites them, runs after the copy; the chain waits for the copy.+ const size_t stride = (size_t) g.hc * (size_t) g.n_embd;+ for (const Blk& blk : blks) {+ BSlot& sl = bs[(size_t) blk.b];+ if (!sl.active) continue; // it ended: no more drafts for it+ const strata::core::OnDevice on_slot_mtp(slot_mtp[(size_t) blk.b]->device());+ sl.draft_ready = false;+ const float* src = vk.final_R_all() + (size_t) blk.f * stride;+ if (cudaMemcpyAsync(slot_mtp_rows[(size_t) blk.b].get(), src, (size_t) blk.n * stride * sizeof(float),+ cudaMemcpyDeviceToDevice, vk.stream()) != cudaSuccess ||+ cudaEventRecord(pm_copied[(size_t) blk.b], vk.stream()) != cudaSuccess ||+ cudaStreamWaitEvent(slot_mtp[(size_t) blk.b]->stream(), pm_copied[(size_t) blk.b], 0) != cudaSuccess ||+ !slot_mtp[(size_t) blk.b]->chain_launch(blk.n, outb + blk.f, G.pos[blk.f], blk.keep - 1, nullptr, 0,+ pm_drafts, 0, err)) {+ std::printf("ERR batch MTP slot %d: %s\n", blk.b,+ err.empty() ? "the proposal's chain did not start" : err.c_str());+ return false;+ }+ pm_drafting[(size_t) blk.b] = 1;+ }+ return true;+ };+ // STRATA_PIPED_MTP: the commits queued after the verdicts have run on every stage, so whatever reads a slot's+ // sessions next (an admission, a parked conversation, the solo path taking a slot back) sees its final state+ auto pmtp_settle = [&]() -> bool {+ if (!pmtp) return true;+ for (int b = 0; b < (int) bs.size(); ++b) // (a chain in flight would refuse the drafter's next use)+ if (pmtp_poll_draft(b, true) < 0) return false;+ for (int k = 0; k < n_pipe; ++k) {+ strata::core::Verifier& vk = stage_verifier(k);+ const strata::core::OnDevice on_k(vk.device());+ if (cudaStreamSynchronize(vk.stream()) != cudaSuccess) {+ std::printf("ERR a pipelined commit failed on stage %d\n", k + 1);+ return false;+ }+ }+ return true;+ };// one turn: serve every running stage, hand finished groups on, start what can startauto pump = [&](bool may_start) -> bool {++pipe_tick;@@ -11976,7 +12206,8 @@ int main(int argc, char** argv) {if (k + 1 < n_pipe) { G.stage = k + 1; G.since = pipe_tick; continue; }// the last stage: the group's picksconst int32_t* outb = vk.batch_out();- for (int t = 0; t < G.S; ++t) {+ if (pmtp && !pmtp_verdict(G, vk, outb)) return false;+ for (int t = 0; t < G.S && !pmtp; ++t) {BSlot& sl = bs[(size_t) (gi * GS + t)];if (!sl.active) continue;if (vk.nonfinite_in(t)) { // #879: the slot ends, nothing emitted@@ -12037,22 +12268,58 @@ int main(int argc, char** argv) {if (!may_start) continue;for (int j = 0; j < (int) pg.size() && pick < 0; ++j) {const int gi = (int) ((rr + j) % (int64_t) pg.size());- if (!pg[(size_t) gi].inflight && group_active(gi)) pick = gi;+ if (pg[(size_t) gi].inflight || !group_active(gi)) continue;+ if (pmtp) { // its slots' proposal chains still running: another group goes first+ bool ready = true;+ for (int t = 0; t < GS; ++t) {+ const int dr = pmtp_poll_draft(gi * GS + t, false);+ if (dr < 0) return false;+ if (dr == 0) { ready = false; break; }+ }+ if (!ready) continue;+ }+ pick = gi;}if (pick < 0) continue;rr = pick + 1;PGroup& G = pg[(size_t) pick];G.S = 0;- G.active = 0;- for (int t = 0; t < GS; ++t) if (bs[(size_t) (pick * GS + t)].active) {- G.S = t + 1;- ++G.active;- }- for (int t = 0; t < G.S; ++t) {- BSlot& sl = bs[(size_t) (pick * GS + t)];- G.tok[t] = sl.active ? sl.x : 0;- G.pos[t] = sl.active ? sl.p : 0;- if (!sl.active) sl.cached = false; // its pad row writes its state+ if (pmtp) { // the group's slots: each its token, then its own proposals - as many as could be emitted+ G.hbase = pmtp_rows_per_group * pick;+ G.S = 0;+ G.active = 0;+ for (int t = 0; t < GS; ++t) {+ const int b = pick * GS + t;+ const BSlot& sl = bs[(size_t) b];+ if (!sl.active) continue; // an idle slot carries no row (its state is left untouched)+ ++G.active;+ G.rows[G.S] = b;+ G.tok[G.S] = sl.x;+ G.pos[G.S] = sl.p;+ ++G.S;+ int nd = sl.draft_ready && !sl.stop ? sl.n_drafts : 0;+ nd = (int) std::min<int64_t>(nd, std::min<int64_t>(sl.max_new - sl.produced - 1,+ o.max_context - sl.p - 2));+ for (int j = 0; j < nd; ++j) {+ G.rows[G.S] = b;+ G.tok[G.S] = sl.draft[(size_t) j];+ G.pos[G.S] = sl.p + 1 + j;+ ++G.S;+ }+ }+ if (G.S == 0) { G.inflight = false; continue; } // every slot of the group went idle+ } else {+ G.active = 0;+ for (int t = 0; t < GS; ++t) if (bs[(size_t) (pick * GS + t)].active) {+ G.S = t + 1;+ ++G.active;+ }+ for (int t = 0; t < G.S; ++t) {+ BSlot& sl = bs[(size_t) (pick * GS + t)];+ G.tok[t] = sl.active ? sl.x : 0;+ G.pos[t] = sl.active ? sl.p : 0;+ if (!sl.active) sl.cached = false; // its pad row writes its state+ }}G.inflight = true;G.stage = 0;@@ -12069,7 +12336,8 @@ int main(int argc, char** argv) {PGroup& G = pg[(size_t) pick];strata::core::progress().busy.store(true);if (batch_telemetry_enabled) stage_in_flight.launch(k, Clock::now());- if (!stage_verifier(k).batch_launch(pick * GS, G.S, G.tok, G.pos, err)) {+ if (!(pmtp ? stage_verifier(k).batch_launch_rows(G.rows, G.S, G.hbase, G.tok, G.pos, err)+ : stage_verifier(k).batch_launch(pick * GS, G.S, G.tok, G.pos, err))) {std::printf("ERR %s\n", err.c_str());return false;}@@ -12092,16 +12360,21 @@ int main(int argc, char** argv) {std::fflush(stdout);}}+ // every slot done: the last commits have run before anything reads a slot (between windows nothing does)+ if (pmtp && !pipe_inflight() && !batch_on() && !pmtp_settle()) return false;if (!pipe_inflight() && !batch_on() && bt_windows > 0) { // all idle: the timing line, as batch_stepconst double w = (double) bt_windows, wall = std::chrono::duration<double, std::milli>(Clock::now() - bt_start).count();std::fprintf(stderr, "strata batch (pipelined, %d groups of %d): %lld group-steps, %lld rows in %.0f ms = "- "%.1f rows/s (admissions included)\n", o.batch_groups, GS, (long long) bt_windows,+ "%.1f rows/s (admissions included)", o.batch_groups, GS, (long long) bt_windows,(long long) bt_rows, wall, 1000.0 * bt_rows / std::max(wall, 1e-9));+ if (pmtp)+ std::fprintf(stderr, ", %lld of %lld MTP proposals kept", (long long) bt_accepted, (long long) bt_tokens);+ std::fprintf(stderr, "\n");for (int k = 0; k < n_pipe; ++k) {const std::string pr = stage_verifier(k).profile_report();if (!pr.empty()) std::fprintf(stderr, "strata batch GPU stages, stage %d (ms/window):%s\n", k + 1, pr.c_str());}- bt_windows = bt_rows = 0;+ bt_windows = bt_rows = bt_tokens = bt_accepted = 0;// (not while a prompt is read in a request: its windows ran beside it, and the watchdog stays on for// the rest of the read, STRATA_PIPED_CONCURRENT; the request's end turns it off)if (!in_prompt_read) strata::core::progress().busy.store(false);@@ -12111,7 +12384,7 @@ int main(int argc, char** argv) {auto pipe_drain = [&]() -> bool {while (pipe_inflight())if (!pump(false)) return false;- return true;+ return pmtp_settle();};for (;;) {line_drain_ms = 0.0;@@ -15624,12 +15897,15 @@ int main(int argc, char** argv) {sl.t0 = Clock::now();sl.ids = live;if (batch_mtp) {+ int nd = 1; // (STRATA_PIPED_MTP: up to its proposals per window, as --spec-min-p allows)if (!slot_mtp[(size_t) admit_slot]->draft_first(1, ver.final_R_all(), sl.x,- sl.p - 1, sl.draft.data(), err)) {+ sl.p - 1, sl.draft.data(), err, nullptr,+ piped_mtp ? (float) o.spec_min_p : 0.0f, &nd)) {std::fprintf(stderr, "strata batch: MTP admission for slot %d failed: %s\n",admit_slot, err.c_str());return 1;}+ sl.n_drafts = std::max(1, nd);sl.draft_ready = true;}sl.cvec = cvec_cached;