Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

batch slots: MTP drafts in the pipelined groups, STRATA_PIPED_MTP=N opt-in with --batch-mtp (#4121, #3794)

wannagoxuf17878ac57925
diff --git a/docs/BATCHING.md b/docs/BATCHING.md
index 6e7b58d39..bc5e3a5ac 100644
--- a/docs/BATCHING.md
+++ b/docs/BATCHING.md
@@ -390,6 +390,66 @@ compares the tier as a whole, not "batch windows only": the solo decode speed mo
more varied workload the gain may be smaller (not measured). Upstream main runs no tier in batch windows, and this
was not measured against an upstream binary.
+## MTP drafts in the pipelined groups (`STRATA_PIPED_MTP`, opt-in)
+
+The pipelined groups run no drafts. `--batch-mtp` with serial windows (`--batch-groups 1`) keeps one card busy at a
+time. `STRATA_PIPED_MTP=N` together with `--batch-mtp` gives both: the pipeline stays, and each slot drafts in it.
+Off by default. Without the variable nothing changes.
+
+How it works. A group's window carries, for each of its slots, the slot's token and up to N MTP proposals (at most
+`--spec` - 1; the first always, each further one while the draft layer was at least `--spec-min-p` sure of the one
+before). The last stage judges each slot's proposals on their own. How many rows to keep is known only then, and by
+then the earlier stages already run other groups' windows, which reuse the scratch the commit replays from. So a
+small save graph copies those rows right behind each window (`Verifier::batch_launch_rows`), and after the verdict every
+stage queues the commit of the kept rows (`Verifier::batch_commit_rows`). The slot's drafter catches up on its own rows
+and drafts the next proposals as an asynchronous chain (`MtpDrafter::chain_launch`). A group whose chain has not landed
+lets another group start first. Anything that reads a slot's state (an admission, a parked conversation, the solo path
+taking a slot back) first drains the pipeline, which also waits for the queued commits.
+
+It needs a layer split with two or more groups, any number of slots per group (as long as a group's window,
+slots x (1 + N), is at most 8 rows). When it cannot run, the engine says why and goes on without it: fewer than two
+groups left, `STRATA_BATCH_SELF_COMMIT` on, or a window of more than 8 rows.
+
+Measured on a box with 2x RTX A6000 48 GB (`--layer-split auto`), Qwen3.8-Flash-Next IQ3_S, every expert resident in
+VRAM, `--batch 4 --batch-groups auto` (2 groups of 2 slots), `tools/serve_load.py`, 256 tokens per request, reasoning
+off. Total output tokens per second: the median of the pairs, each pair the median of 3 bursts on a fresh engine start. The base arm
+(pipelined groups, no drafts) and `STRATA_PIPED_MTP=2` were run in 5 interleaved pairs, the other arms in 3 pairs. "Default" is
+`--batch 4` with no `--batch-groups`, which on exactly two NVIDIA cards means one serial group (`STRATA_BATCH_GROUPS_NV_SERIAL`):
+
+| streams | pipelined, no drafts | `=1` | `=2` | `=3` | serial `--batch-mtp` (`--batch-groups 1`) | default (no flags) |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 1 (solo path) | 101.7 | 101.4 | 101.5 | 103.0 | 101.5 | 101.7 |
+| 2 | 125.7 | 146.3 (+16.0%) | 145.1 (+15.4%) | 138.4 (+10.0%) | 124.0 (-1.3%) | 107.4 (-14.8%) |
+| 4 | 171.4 | 188.8 (+10.2%) | 179.5 (+4.7%) | 165.6 (-3.4%) | 139.9 (-18.5%) | 141.0 (-17.8%) |
+
+Gains are the median of the per-pair ratios. `=2` was faster than the base in 5 of 5 pairs at 2 and at 4 streams, `=1` in 3 of 3
+pairs at both, `=3` in 3 of 3 at 2 streams and in none at 4. With `--batch 2` (2 groups of 1 slot) and 2 streams `=2` gave 146.8 against 125.7 tok/s (+16.4%, 5 of 5 pairs). One
+stream runs on the solo path in every arm and does not change. About 55% of the checked proposals were kept at `=2`
+and about 67% at `=1`. The acceptance falls as the window grows, which is why `=1` or `=2` is the choice and `=3` is not.
+
+Best setting here: `=2` with two streams, `=1` with four. More than four streams was not measured, and the gain at
+four streams is smaller than at two (not explained). Cost: the slot
+drafters and a save buffer per stage (25.5 MiB at `=1` and 38.2 MiB at `=2` with 4 slots), taken from the expert cache.
+On the A6000 box the cache stayed fully resident. On a 2x RTX A4000 box where the experts do not all fit, `--batch-mtp` took
+165 of 6959 cache slots on the second card; that changes which experts are in VRAM and so, in the last digits, which tokens come out.
+
+The first version of this was written by wannagoxuf (PR #4121), who measured +39% on 4x RTX 4090 with 4 streams
+(366 to 510 tok/s at `=2`). That is a different rig and not repeated here.
+
+Exact. Checked with the engine's batch protocol (the method of `tools/batch_test.py`: each prompt alone, then the
+prompts together in the slots, token ids compared). On the A6000 box, 4 slots (2 groups of 2) and 2 slots (2 groups of 1)
+with the feature unset and with `=1`, `=2` and `=3` (`=2` only for 2 slots): greedy streams, greedy and seeded sampled streams
+together, slots that finish at different lengths, a slot cancelled in the middle of a draft, and a 3,000-token prompt read beside the
+others. Every slot got the tokens it gets alone, and the same tokens as the build without the feature (a cancelled slot stops one
+or two tokens apart, so its tokens are compared up to the shorter). Through the server the same cases (and two clients on four
+slots) gave the same text as the build without the feature, apart from one or two streams of 19 per run; two runs of the
+build without the feature differ from each other in the same way, so this is not the feature. On a box where the experts
+do not all fit in VRAM the drafters' memory changes which experts are cached, and then the text of a feature run differs from a
+run without it; inside one run every slot still equals its own solo decode.
+
+Not run: AMD (HIP), the draft layer on the first stage, helper expert caches, more than two stages, SYCL (the SYCL
+tree has its own copy of the batch code and does not have this).
+
## Exactness
A batch row's arithmetic is the single-token window's, so with greedy decoding **every conversation of a batch
@@ -423,8 +483,9 @@ counter-based draw (Philox(seed, position)).
- By default, batch windows carry no MTP drafts: a conversation in a slot decodes one token per window (the solo
path keeps its drafts, which is why a request alone is not put in a slot, and goes back to it when left alone).
- `--batch-mtp` uses one proposal per slot. With a layer split it needs each stage on its own GPU, does not combine
- with `--batch-groups` above 1 (pipelined slot groups run no drafts, #1413) and has not been run with helper expert
- caches.
+ with `--batch-groups` above 1 (pipelined slot groups run no drafts, #1413) unless `STRATA_PIPED_MTP` is set with
+ one slot per group ([above](#mtp-drafts-in-the-pipelined-groups-strata_piped_mtp-opt-in)), and has not been run
+ with helper expert caches.
- Repetition / frequency / presence penalties are not applied in batch windows.
- A prompt shorter than one chunk is read in one piece (the slots wait for it); a read gives way only at a chunk
boundary, and not for pictures.
diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp
index 2b2d9a0ec..e878b9ed9 100644
--- a/include/strata/core/verify.hpp
+++ b/include/strata/core/verify.hpp
@@ -235,6 +235,23 @@ public:
/// -1 = an error (err). Serves every layer that has rung so far.
int batch_poll(PoolMultiFn pool, void* user, std::string& err);
bool batch_busy() const { return b_running_.load(std::memory_order_acquire); }
+ // ---- STRATA_PIPED_MTP: the pipelined groups with --batch-mtp's drafts. A group's window holds its slot's token
+ // and its drafts, so how many rows to keep is known only once the LAST stage has picked - and by then this stage
+ // runs another group's window, which reuses the scratch the commit replays from. So right behind the window a
+ // save graph copies those rows (each GDN layer's conv and recurrence inputs, each QSA layer's raw indexer keys, the
+ // PLE history snapshot) to the group's own rows [hbase, hbase + S) of the saved buffers, and the commit, queued
+ // after the verdict, replays from there. Everything else the commit reads is per slot already.
+ /// The saved buffers for `rows` hand-off rows (after init_slots).
+ bool init_deferred_commit(int rows, std::string& err);
+ bool deferred_commit_on() const { return dc_rows_ > 0; }
+ /// The window over the slots `rows` (a slot's rows contiguous) at hand-off rows [hbase, hbase + S), its save right
+ /// behind it, and no commit (batch_commit_rows, after the verdict). The PLE history advances at the commit.
+ bool batch_launch_rows(const int* rows, int S, int hbase, const int32_t* tokens, const int64_t* pos,
+ std::string& err);
+ /// Queue the commit of that window's accepted prefixes on this stage's stream (no host wait): `keep` per slot ID,
+ /// 1..the slot's rows. `rows`, `S`, `hbase`, `tokens` and `pos` are the window's, as batch_launch_rows got them.
+ bool batch_commit_rows(const int* rows, int S, int hbase, const int* keep, const int32_t* tokens,
+ const int64_t* pos, std::string& err);
/// Serves the layers of this stage's window that have rung (the CPU pool and the flags), with no CUDA call: 1 once
/// every layer is served (or the window raises no host doorbells), 0 while one is still to ring. batch_poll
/// calls it, unless set_external_serve(true) moved it to a thread of its own (STRATA_PIPED_CONCURRENT: a CUDA
@@ -419,6 +436,16 @@ private:
void* arena_b_ = nullptr;
int64_t last_pos_b_[8] = {};
bool capture_batch(const int* rows, int S, int hbase, std::string& err);
+ // STRATA_PIPED_MTP (init_deferred_commit): the saved rows, laid out as the window scratch (global layer index) with
+ // dc_rows_ rows per layer instead of max_t_; the slots' accepted prefixes (mapped and device, 2 + max_t_ per slot)
+ int dc_rows_ = 0;
+ bool staging_deferred_ = false; ///< stage_batch for batch_launch_rows: no capture_commit_batch
+ float *dc_qkv_ = nullptr, *dc_h_ = nullptr, *dc_gate_ = nullptr, *dc_beta_ = nullptr, *dc_idx_ = nullptr;
+ float* dc_hist_ = nullptr;
+ void* dc_arena_ = nullptr;
+ int32_t *h_commitd_ = nullptr, *m_commitd_ = nullptr, *commitd_ = nullptr;
+ std::map<std::vector<int>, cudaGraphExec_t> save_bm_, dcommit_bm_;
+ bool capture_deferred(const int* rows, int S, int hbase, std::string& err);
/// Free the graph pair of the least recently used batch layout other than `keep` (`evicted`: there was one).
bool evict_batch_graph(const std::vector<int>& keep, bool& evicted, std::string& err);
/// cudaGraphInstantiate; out of VRAM, batch layouts other than `key` are evicted (LRU) until it fits, and then (a
diff --git a/include/strata/program/batch_groups.hpp b/include/strata/program/batch_groups.hpp
index b1e3f32cc..9f614d454 100644
--- a/include/strata/program/batch_groups.hpp
+++ b/include/strata/program/batch_groups.hpp
@@ -28,6 +28,7 @@ struct Input {
bool adapt_async = false; ///< --adapt-async 1 was asked for
bool mtp_serial = false; ///< STRATA_BATCH_MTP_SERIAL=1: --batch-mtp with no --batch-groups runs one serial group (0.1.43)
bool nv_serial_faster = false; ///< PROPOSED (0.2.1, owner decides): NVIDIA build, STRATA_BATCH_GROUPS_NV_SERIAL not 0 - two cards run faster in serial windows (z440 2x A4000, aggregate tok/s: +16% at 2 slots, +12% at 4, +15% at 8)
+ bool mtp_piped = false; ///< STRATA_PIPED_MTP=1: the pipelined groups draft too (each group carries its slots' rows)
};
struct Result {
@@ -36,6 +37,7 @@ struct Result {
Serial serial = Serial::None; ///< `auto` would have pipelined, but this feature needs the serial windows
int would_be = 1; ///< what `auto` would have picked without it (== groups unless `serial`)
bool mtp_dropped = false; ///< --batch-mtp met the default pipeline: the pipeline wins, --batch-mtp is off
+ bool mtp_piped = false; ///< STRATA_PIPED_MTP: --batch-mtp's drafts run in the pipelined groups
};
/// The most groups, at most one per stage of the layer split (`later_stages` + 1 GPUs), that divide `batch`.
@@ -52,17 +54,24 @@ inline Result resolve(const Input& in) {
r.groups = in.given;
r.would_be = in.given;
const bool defaulted = !in.set && in.later_stages > 0 && in.batch > 1; // the 0.1.41 default
- if (!in.auto_given && !defaulted) return r;
- r.from_auto = true;
- r.would_be = auto_groups(in.later_stages, in.batch);
- r.groups = r.would_be;
- if (!in.set && r.would_be > 1) { // only the default gives way; `--batch-groups auto` asked for the pipeline
- if (in.batch_mtp && in.mtp_serial) r.serial = Serial::BatchMtp;
- else if (in.nv_serial_faster && in.later_stages == 1 && !in.batch_mtp && !in.adapt_async) r.serial = Serial::TwoCardsSerialFaster;
- else if (in.adapt_async) r.serial = Serial::AdaptAsync;
- r.mtp_dropped = in.batch_mtp && r.serial == Serial::None;
- if (r.serial != Serial::None) r.groups = 1;
+ if (in.auto_given || defaulted) {
+ r.from_auto = true;
+ r.would_be = auto_groups(in.later_stages, in.batch);
+ r.groups = r.would_be;
+ if (!in.set && r.would_be > 1) { // only the default gives way; `--batch-groups auto` asked for the pipeline
+ if (in.batch_mtp && in.mtp_serial) r.serial = Serial::BatchMtp;
+ else if (in.nv_serial_faster && in.later_stages == 1 && !in.batch_mtp && !in.adapt_async) r.serial = Serial::TwoCardsSerialFaster;
+ else if (in.adapt_async) r.serial = Serial::AdaptAsync;
+ if (r.serial != Serial::None) r.groups = 1;
+ r.mtp_dropped = in.batch_mtp && r.serial == Serial::None;
+ }
}
+ // STRATA_PIPED_MTP: the pipelined groups draft too. A group's window carries each of its slots' rows (the slot's
+ // token and that slot's own proposals), so any number of slots per group drafts. This sits outside the block
+ // above on purpose: with --batch-groups given (a number or `auto`) the operator asked for those groups, and the
+ // drafts run in them too. Unset (mtp_piped false) nothing here changes a result.
+ r.mtp_piped = in.batch_mtp && in.mtp_piped && r.serial == Serial::None && r.groups > 1;
+ if (r.mtp_piped) r.mtp_dropped = false;
return r;
}
diff --git a/src/core/verify.cpp b/src/core/verify.cpp
index 5e97b8f52..531efacd1 100644
--- a/src/core/verify.cpp
+++ b/src/core/verify.cpp
@@ -629,6 +629,11 @@ Verifier::~Verifier() {
if (kv.second) cudaGraphExecDestroy(kv.second);
if (arena_b_) cudaFree(arena_b_);
if (h_commitb_) cudaFreeHost(h_commitb_);
+ for (auto* m : {&save_bm_, &dcommit_bm_})
+ for (auto& kv : *m)
+ if (kv.second) cudaGraphExecDestroy(kv.second);
+ if (dc_arena_) cudaFree(dc_arena_);
+ if (h_commitd_) cudaFreeHost(h_commitd_);
if (qcnt_) cudaFree(qcnt_);
if (d_spec_) cudaFree(d_spec_);
if (tap_file_) std::fclose(tap_file_);
@@ -3668,6 +3673,234 @@ bool Verifier::capture_commit_batch(const int* rows, int S, int hbase, std::stri
return made;
}
+// ---- STRATA_PIPED_MTP: commits after the verdict (see the header)
+
+bool Verifier::init_deferred_commit(int rows, std::string& err) {
+ const OnDevice on_device(device_);
+ if (slots_.empty()) { err = "verify: init_deferred_commit before init_slots"; return false; }
+ if (rows < 1) { err = "verify: init_deferred_commit needs at least one row"; return false; }
+ const ModelGeometry& g = *g_;
+ const uint64_t R = (uint64_t) rows, C = (uint64_t) g.ssm_conv_channels, HV = (uint64_t) g.ssm_v_heads;
+ const uint64_t ID = (uint64_t) g.idx_key_dim;
+ const uint64_t nG = (uint64_t) std::max<int64_t>(g.n_gdn_layers(), 1), nQ = (uint64_t) std::max<int64_t>(g.n_qsa_layers(), 1);
+ const uint64_t HS = (uint64_t) strata::kernels::NG_HIST * strata::kernels::NG_HC_DIM;
+ const uint64_t CB = 2 + (uint64_t) max_t_, nS = (uint64_t) slots_.size();
+ const uint64_t floats[6] = {nG * R * C, nG * R * C, nG * R * HV, nG * R * HV, nQ * R * ID, R * HS};
+ uint64_t bytes = (nS * CB * 4 + 255) & ~255ull;
+ for (uint64_t f : floats) bytes += (f * 4 + 255) & ~255ull;
+ if (cudaMalloc(&dc_arena_, bytes) != cudaSuccess) {
+ err = "verify: the saved commit rows (" + std::to_string(bytes >> 20) + " MiB) do not fit";
+ return false;
+ }
+ uint8_t* p = (uint8_t*) dc_arena_;
+ commitd_ = (int32_t*) p;
+ p += (nS * CB * 4 + 255) & ~255ull;
+ float** dst[6] = {&dc_qkv_, &dc_h_, &dc_gate_, &dc_beta_, &dc_idx_, &dc_hist_};
+ for (int i = 0; i < 6; ++i) {
+ *dst[i] = (float*) p;
+ p += (floats[i] * 4 + 255) & ~255ull;
+ }
+ if (!mapped((size_t) (nS * CB * 4 + 16), (void**) &h_commitd_, (void**) &m_commitd_)) {
+ err = "verify: the deferred commit staging failed";
+ return false;
+ }
+ dc_rows_ = rows;
+ std::fprintf(stderr, "strata verify: pipelined batch windows commit after the verdict (layers [%lld, %lld), %d rows, "
+ "%.1f MiB saved rows)\n", (long long) lb_, (long long) le_, rows, (double) bytes / 1048576.0);
+ return true;
+}
+
+bool Verifier::capture_deferred(const int* rows, int S, int hbase, std::string& err) {
+ using namespace strata::kernels;
+ const auto key = bkey(rows, S, hbase);
+ cudaGraphExec_t& sex = save_bm_[key];
+ cudaGraphExec_t& cex = dcommit_bm_[key];
+ if (sex != nullptr && cex != nullptr) return true;
+ const ModelGeometry& g = *g_;
+ const QsaShapes s = shapes_of(g);
+ const int64_t C = g.ssm_conv_channels, HV = g.ssm_v_heads, ZV = g.ssm_value_dim, ID = g.idx_key_dim, MT = max_t_;
+ const int64_t R = dc_rows_, CB = 2 + MT, nQ = g.n_qsa_layers();
+ const uint64_t gdn_floats = (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size +
+ (uint64_t) g.ssm_conv_channels * (g.ssm_d_conv - 1);
+ const int64_t TS = (s.idx_block - 1) * ID;
+ const int64_t HS = (int64_t) NG_HIST * NG_HC_DIM;
+ const bool ple = ss_->ple.ready() && ple_stage();
+ auto layer_index = [&](int64_t& qsa_index, int64_t& gdn_index) {
+ qsa_index = gdn_index = 0;
+ for (int64_t l = 0; l < lb_; ++l) (is_qsa_layer(g, l) ? qsa_index : gdn_index) += 1;
+ };
+ auto finish = [&](cudaGraphExec_t& ex, bool ok, const char* what) {
+ cudaGraph_t graph = nullptr;
+ const cudaError_t ce = cudaStreamEndCapture(cs_, &graph);
+ if (!ok || ce != cudaSuccess) {
+ if (graph) cudaGraphDestroy(graph);
+ if (ok) err = std::string(what) + cudaGetErrorString(ce);
+ return false;
+ }
+ const bool made = instantiate_evicting(ex, graph, key, what, err);
+ cudaGraphDestroy(graph);
+ return made;
+ };
+ // the save: the window's rows [0, S) of the scratch the commit replays -> the group's saved rows [hbase, +S)
+ if (sex == nullptr) {
+ if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) {
+ err = "verify: begin save capture failed";
+ return false;
+ }
+ bool ok = true;
+ int64_t qsa_index = 0, gdn_index = 0;
+ layer_index(qsa_index, gdn_index);
+ auto copy = [&](float* dst, const float* src, int64_t n) {
+ ok = ok && cudaMemcpyAsync(dst, src, (size_t) n * sizeof(float), cudaMemcpyDeviceToDevice, cs_) == cudaSuccess;
+ };
+ for (int64_t l = lb_; l < le_; ++l) {
+ if (!is_qsa_layer(g, l)) {
+ copy(dc_qkv_ + (size_t) (gdn_index * R + hbase) * C, qkv_L_ + (size_t) gdn_index * MT * C, S * C);
+ copy(dc_h_ + (size_t) (gdn_index * R + hbase) * C, h_L_ + (size_t) gdn_index * MT * C, S * C);
+ copy(dc_gate_ + (size_t) (gdn_index * R + hbase) * HV, gate_L_ + (size_t) gdn_index * MT * HV, S * HV);
+ copy(dc_beta_ + (size_t) (gdn_index * R + hbase) * HV, beta_L_ + (size_t) gdn_index * MT * HV, S * HV);
+ ++gdn_index;
+ } else {
+ copy(dc_idx_ + (size_t) (qsa_index * R + hbase) * ID, idx_raw_L_ + (size_t) qsa_index * MT * ID, S * ID);
+ ++qsa_index;
+ }
+ }
+ if (ple) copy(dc_hist_ + (size_t) hbase * HS, hist_snap_, S * HS);
+ if (!finish(sex, ok, "verify: save capture: ")) {
+ if (ok && err.empty()) err = "verify: save capture failed";
+ return false;
+ }
+ }
+ // the commit: as capture_commit_batch, replaying the saved rows; the accepted prefixes come from commitd_
+ if (cex == nullptr) {
+ if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) {
+ err = "verify: begin deferred commit capture failed";
+ return false;
+ }
+ bool ok = true;
+ try {
+ for (int t = 0; t < S; ++t)
+ if (t == 0 || rows[t] != rows[t - 1])
+ copy_i32_from_mapped(commitd_ + (size_t) rows[t] * CB, m_commitd_ + (size_t) rows[t] * CB, CB, cs_);
+ int64_t qsa_index = 0, gdn_index = 0;
+ layer_index(qsa_index, gdn_index);
+ for (int64_t l = lb_; l < le_ && ok; ++l) {
+ const LayerView v(*wt_, l);
+ if (!is_qsa_layer(g, l)) {
+ const WeightRef* wnm = need(v, "ssm_norm.weight", err);
+ if (!wnm) { ok = false; break; }
+ for (int t = 0; t < S;) {
+ const int first = t;
+ while (t < S && rows[t] == rows[first]) ++t;
+ SessionState& sx = *slots_[(size_t) rows[first]];
+ const int32_t* keep = commitd_ + (size_t) rows[first] * CB;
+ float* state = sx.gdn_state + (size_t) (gdn_index - sx.gdn_ord0) * gdn_floats;
+ float* conv = state + (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size;
+ const int64_t row = gdn_index * R + hbase + first;
+ gdn_conv_commit(conv, dc_qkv_ + (size_t) row * C, (int) C, keep, cs_);
+ // z only feeds the norm's output, which the commit discards (y_dummy_): the window scratch will do
+ gdn_step_norm_multi(state, dc_h_ + (size_t) row * C, (int) C, dc_gate_ + (size_t) row * HV,
+ dc_beta_ + (size_t) row * HV, z_ + (size_t) first * ZV,
+ (const float*) wnm->data, EPS, y_dummy_ + (size_t) first * ZV,
+ (int) g.ssm_k_heads, (int) HV, t - first, keep, cs_,
+ t - first > 1 ? t - first : 0);
+ }
+ ++gdn_index;
+ } else {
+ const WeightRef* wikn = need(v, "indexer.k_norm.weight", err);
+ if (!wikn) { ok = false; break; }
+ for (int t = 0; t < S;) {
+ const int first = t;
+ while (t < S && rows[t] == rows[first]) ++t;
+ const QsaState& st = slots_[(size_t) rows[first]]->qsa_states[qsa_index];
+ copy_from_mapped(st.idx_tail, tail_snap_b_ + ((size_t) rows[first] * nQ + qsa_index) * TS, TS, cs_);
+ const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos};
+ for (int u = first; u < t; ++u)
+ native_qsa_indexer_append(dc_idx_ + (size_t) (qsa_index * R + hbase + u) * ID,
+ commitd_ + (size_t) rows[first] * CB + 2 + (u - first), 0,
+ (const float*) wikn->data, EPS, ib, s, st.max_cells,
+ rope_scaling(), cs_);
+ }
+ ++qsa_index;
+ }
+ }
+ if (ok && ple)
+ for (int t = 0; t < S;) {
+ const int first = t;
+ while (t < S && rows[t] == rows[first]) ++t;
+ copy_indexed(slots_[(size_t) rows[first]]->ple_hist, dc_hist_ + (size_t) (hbase + first) * HS, HS,
+ commitd_ + (size_t) rows[first] * CB + 1, HS, cs_);
+ }
+ } catch (const std::exception& e) {
+ err = std::string("verify deferred commit: ") + e.what();
+ ok = false;
+ }
+ if (!finish(cex, ok, "verify: deferred commit capture: ")) {
+ if (ok && err.empty()) err = "verify: deferred commit capture failed";
+ return false;
+ }
+ }
+ return true;
+}
+
+bool Verifier::batch_launch_rows(const int* rows, int S, int hbase, const int32_t* tokens, const int64_t* pos,
+ std::string& err) {
+ const OnDevice on_device(device_);
+ if (dc_rows_ < 1) { err = "verify: batch_launch_rows without init_deferred_commit"; return false; }
+ if (b_running_) { err = "verify: batch_launch_rows while this stage is busy"; return false; }
+ if (S < 1 || hbase < 0 || hbase + S > dc_rows_) { err = "verify: a pipelined window outside the saved rows"; return false; }
+ if (pre_launch_ && !pre_launch_(cs_, err)) return false;
+ staging_deferred_ = true;
+ const bool staged = stage_batch(rows, S, hbase, tokens, pos, err);
+ staging_deferred_ = false;
+ if (!staged || !capture_deferred(rows, S, hbase, err)) return false;
+ const auto key = bkey(rows, S, hbase);
+ cudaError_t le = cudaGraphLaunch(exec_bm_[key], cs_);
+ if (le == cudaSuccess) le = cudaGraphLaunch(save_bm_[key], cs_); // the rows the commit replays, kept for it
+ if (le != cudaSuccess) { err = std::string("verify: batch launch: ") + cudaGetErrorString(le); return false; }
+ (void) cudaStreamQuery(cs_);
+ b_k_ = 0;
+ b_steps_ = le_ - lb_;
+ ring_now();
+ b_running_.store(true, std::memory_order_release); // last: an external server reads the rest after it
+ return true;
+}
+
+bool Verifier::batch_commit_rows(const int* rows, int S, int hbase, const int* keep, const int32_t* tokens,
+ const int64_t* pos, std::string& err) {
+ const OnDevice on_device(device_);
+ const auto it = dcommit_bm_.find(bkey(rows, S, hbase));
+ if (it == dcommit_bm_.end() || it->second == nullptr) { err = "verify: a deferred commit without its window"; return false; }
+ const int64_t CB = 2 + max_t_;
+ // Each slot's accepted prefix. The graph reads it from mapped memory when it runs; this slot's entry is written
+ // again only at its next verdict, which comes after its next window has passed this stage - behind this commit.
+ for (int t = 0; t < S;) {
+ const int first = t;
+ while (t < S && rows[t] == rows[first]) ++t;
+ const int n = keep[rows[first]];
+ if (n < 1 || n > t - first) { err = "verify: accepted prefix is outside its slot group"; return false; }
+ int32_t* c = h_commitd_ + (size_t) rows[first] * CB;
+ c[0] = n;
+ c[1] = n - 1;
+ for (int64_t j = 0; j < max_t_; ++j) c[2 + j] = j < n ? (int32_t) pos[first + j] : -1;
+ }
+ std::atomic_thread_fence(std::memory_order_seq_cst);
+ const cudaError_t le = cudaGraphLaunch(it->second, cs_);
+ if (le != cudaSuccess) { err = std::string("verify: deferred commit launch: ") + cudaGetErrorString(le); return false; }
+ (void) cudaStreamQuery(cs_);
+ if (ple_stage()) // the host's side of the commit (the hash's last two tokens), the accepted rows only
+ for (int t = 0; t < S;) {
+ const int first = t;
+ while (t < S && rows[t] == rows[first]) ++t;
+ SessionState& sx = *slots_[(size_t) rows[first]];
+ for (int u = first; u < first + keep[rows[first]]; ++u) {
+ sx.ple_prev[0] = sx.ple_prev[1];
+ sx.ple_prev[1] = tokens[u];
+ }
+ }
+ return true;
+}
+
bool Verifier::evict_batch_graph(const std::vector<int>& keep, bool& evicted, std::string& err) {
evicted = false;
auto old = exec_bm_.end();
@@ -3780,7 +4013,9 @@ bool Verifier::stage_batch(const int* rows, int S, int hbase, const int32_t* tok
!evict_batch_graph(key, evicted, err))
return false;
bm_used_[key] = ++bm_tick_;
- if (!capture_batch(rows, S, hbase, err) || !capture_commit_batch(rows, S, hbase, err)) return false;
+ // (a pipelined window that commits after the verdict has its own commit graph: capture_deferred)
+ if (!capture_batch(rows, S, hbase, err) || (!staging_deferred_ && !capture_commit_batch(rows, S, hbase, err)))
+ return false;
const Clock::time_point t0 = Clock::now();
const QsaShapes s = shapes_of(g);
for (int t = 0; t < S; ++t) {
diff --git a/src/program/batch_groups_test.cpp b/src/program/batch_groups_test.cpp
index 5debcd29d..564ad3dc9 100644
--- a/src/program/batch_groups_test.cpp
+++ b/src/program/batch_groups_test.cpp
@@ -3,6 +3,7 @@
// 1. `auto` picks the most groups, at most one per stage, that divide the slots;
// 2. a layer split with --batch >= 2 and no --batch-groups means `auto` (0.1.41);
// 3. --batch-mtp (or --adapt-async 1) turns that default into one group, and says which feature did it;
+// with STRATA_PIPED_MTP and one slot per group, --batch-mtp keeps the pipeline and drafts in it;
// 4. a number or `auto` given explicitly is honoured, with or without those features;
// 5. no layer split: the number given (1) stays.
#include "strata/program/batch_groups.hpp"
@@ -58,6 +59,71 @@ int main() {
"STRATA_BATCH_MTP_SERIAL: one group, because of --batch-mtp");
check(bg::serial_what(r.serial)[0] == '-', "the reason names the feature");
}
+ { // STRATA_PIPED_MTP: the pipelined groups draft too, with any number of slots per group
+ bg::Input i = in(4, 3);
+ i.batch_mtp = true;
+ i.mtp_piped = true;
+ auto r = bg::resolve(i);
+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,
+ "STRATA_PIPED_MTP, 4 slots on 4 GPUs: 4 groups of 1 slot, and the drafts stay");
+ i.batch = 8; // two slots per group: the group's window carries both slots' rows, each with its own drafts
+ r = bg::resolve(i);
+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,
+ "STRATA_PIPED_MTP, 8 slots on 4 GPUs: 4 groups of 2 slots, and the drafts stay");
+ i.batch = 4;
+ i.mtp_serial = true; // asked for the serial windows: those win
+ r = bg::resolve(i);
+ check(r.groups == 1 && !r.mtp_piped && r.serial == bg::Serial::BatchMtp,
+ "STRATA_PIPED_MTP with STRATA_BATCH_MTP_SERIAL: one serial group");
+ i.mtp_serial = false;
+ i.batch_mtp = false;
+ r = bg::resolve(i);
+ check(r.groups == 4 && !r.mtp_piped && !r.mtp_dropped, "STRATA_PIPED_MTP without --batch-mtp: nothing to draft");
+ i.batch_mtp = true;
+ i.set = true;
+ i.given = 1; // one group: the serial windows, where --batch-mtp's own drafts run
+ r = bg::resolve(i);
+ check(r.groups == 1 && !r.mtp_piped && !r.mtp_dropped, "STRATA_PIPED_MTP, --batch-groups 1: the serial path, --batch-mtp runs there");
+ }
+ { // 0.1.49.2 port: --batch-groups given EXPLICITLY (a number or `auto`) must not silently
+ // drop the drafts: the operator asked for those groups, and one slot per group still drafts.
+ bg::Input i = in(4, 3);
+ i.batch_mtp = true;
+ i.mtp_piped = true;
+ i.set = true;
+ i.auto_given = true; // --batch-groups auto
+ i.given = 4;
+ auto r = bg::resolve(i);
+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,
+ "STRATA_PIPED_MTP, --batch-groups auto given: 4 groups of 1 slot, and the drafts stay");
+ i.auto_given = false; // --batch-groups 4
+ r = bg::resolve(i);
+ check(r.groups == 4 && r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::None,
+ "STRATA_PIPED_MTP, --batch-groups 4 given: 4 groups of 1 slot, and the drafts stay");
+ i.given = 2; // two slots per group: the group's window carries both slots' rows
+ r = bg::resolve(i);
+ check(r.groups == 2 && r.mtp_piped && !r.mtp_dropped,
+ "STRATA_PIPED_MTP, --batch-groups 2 given: 2 groups of 2 slots, and the drafts stay");
+ }
+ { // STRATA_PIPED_MTP unset: nothing changes, whatever --batch-mtp and the groups say
+ bg::Input i = in(4, 1);
+ i.batch_mtp = true;
+ auto r = bg::resolve(i);
+ check(r.groups == 2 && !r.mtp_piped && r.mtp_dropped && r.serial == bg::Serial::None,
+ "unset, --batch-mtp, nothing given: the pipeline wins, the drafts are dropped (as before)");
+ i.set = true;
+ i.given = 2;
+ r = bg::resolve(i);
+ check(r.groups == 2 && !r.mtp_piped && !r.mtp_dropped, "unset, --batch-mtp, --batch-groups 2: not 'dropped' here (as before)");
+ i.given = 1;
+ r = bg::resolve(i);
+ check(r.groups == 1 && !r.mtp_piped && !r.mtp_dropped, "unset, --batch-mtp, --batch-groups 1: serial windows keep the drafts");
+ i.set = false;
+ i.mtp_serial = true;
+ r = bg::resolve(i);
+ check(r.groups == 1 && !r.mtp_piped && !r.mtp_dropped && r.serial == bg::Serial::BatchMtp,
+ "unset, STRATA_BATCH_MTP_SERIAL: one serial group (as before)");
+ }
{
bg::Input i = in(2, 1);
i.adapt_async = true;
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 75d404ac9..cb36cd56a 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -5381,6 +5381,7 @@ int main(int argc, char** argv) {
#if !defined(STRATA_USE_HIP)
{ const char* nv = std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"); bin.nv_serial_faster = nv == nullptr || nv[0] != '0'; } // PROPOSED 0.2.1 default (NVIDIA, 2 cards)
#endif
+ { const char* mp = std::getenv("STRATA_PIPED_MTP"); bin.mtp_piped = mp != nullptr && mp[0] != '\0' && mp[0] != '0'; }
return strata::program::batch_groups::resolve(bin);
};
const char* batch_mtp_env = std::getenv("STRATA_BATCH_MTP");
@@ -5391,22 +5392,46 @@ int main(int argc, char** argv) {
: o.spec < 2 ? "it needs --spec T (T >= 2)" : split_same
? "it needs each stage of a layer split on its own GPU"
: !o.serve ? "it needs --serve" : nullptr;
- // the groups it would run with (the default gives way to it: one group; a number or auto given pipelines)
- if (why == nullptr && !stages.empty() && batch_groups_for(true).groups > 1)
+ // the groups it would run with (the default gives way to it: one group; a number or auto given pipelines;
+ // STRATA_PIPED_MTP with one slot per group: the pipelined groups draft)
+ if (why == nullptr && !stages.empty() && batch_groups_for(true).groups > 1 && !batch_groups_for(true).mtp_piped)
why = batch_groups_for(true).mtp_dropped
? "a layer split with --batch pipelines the slots in groups by default and the pipelined groups run no slot "
"drafts (4 streams on 2 GPUs: 25-30% faster than the serial MTP windows); --batch-groups 1 or "
"STRATA_BATCH_MTP_SERIAL=1 keeps --batch-mtp with serial windows (fewer streams, or when prompts arrive "
"rarely, it can be the faster choice)"
: "it does not combine with --batch-groups (pipelined slot groups) yet";
- // (a layer split with helper-GPU expert caches or --remote-expert-opt has not been run with slot drafters)
- if (why == nullptr && !stages.empty() && (remote_caches || o.remote_expert_opt))
+ // (a layer split with helper-GPU expert caches has not been run with slot drafters. --remote-expert-opt alone does
+ // not count: its object is only made with a helper cache on CUDA0 (`remote_opt` below), so without one it does nothing)
+ if (why == nullptr && !stages.empty() && remote_caches)
why = "it is not run with helper expert caches on a layer split";
if (why != nullptr) {
std::fprintf(stderr, "strata generate: WARNING: --batch-mtp is off: %s\n", why);
batch_mtp = false;
}
}
+ // STRATA_PIPED_MTP (opt-in): --batch-mtp's proposals in the pipelined groups of a layer split, one slot per group;
+ // each stage commits a group's window once the last stage has judged its proposals
+ // (Verifier::batch_launch_rows / batch_commit_rows)
+ bool piped_mtp = batch_mtp && !stages.empty() && batch_groups_for(true).mtp_piped; // (final once the slots that fit are known, below)
+ // STRATA_PIPED_MTP=N: up to N proposals per slot and window (at most --spec - 1); the first always, each further
+ // one while the draft layer is at least --spec-min-p sure of it, as the solo path extends its windows
+ int pm_drafts = 0;
+ if (piped_mtp) {
+ pm_drafts = std::clamp(std::atoi(std::getenv("STRATA_PIPED_MTP")), 1,
+ std::max(1, std::min(o.spec - 1, (int) strata::kernels::kVerifyMaxT - 1)));
+ }
+ // The verifiers' window: the solo path's drafts (--spec rows), the plain batch windows (the slots) and, with
+ // STRATA_PIPED_MTP, a group's rows: its slots, each its token and its proposals ((slots) * (1 + proposals), one slot
+ // per group = at most --spec rows). A function, not a number: --batch (the slots that fit) and the groups are final
+ // only after this point, at the verifiers' init, and the plain value must stay what it was there.
+ auto batch_max_t = [&]() -> int {
+ if (batch_mtp && !piped_mtp) return (int) strata::kernels::kVerifyMaxT;
+ const int plain = std::max(o.spec, std::min(o.batch, (int) strata::kernels::kVerifyMaxT));
+ const int group_rows = (piped_mtp && o.batch_groups > 1 && o.batch > 0)
+ ? std::min<int>(strata::kernels::kVerifyMaxT, (o.batch / o.batch_groups) * (1 + pm_drafts)) : 0;
+ return std::max(plain, group_rows);
+ };
// The most slots, up to n, whose windows hold at most kVerifyMaxT rows each: n itself up to that, more only as
// pipelined groups on a layer split - the groups as resolved below (a number given, auto, or the default's one
// group for --adapt-async 1), dividing the slots, each at most kVerifyMaxT, all at most one group's worth per
@@ -5528,6 +5553,34 @@ int main(int argc, char** argv) {
}
}
+ // STRATA_PIPED_MTP is decided again now that the slots that fit and the groups are final. Recommend, never force: when
+ // it cannot run it is said and left off, and --batch-mtp goes on as it would without it.
+ if (piped_mtp) {
+ const char* why = nullptr;
+ char buf[160];
+ const char* sc = std::getenv("STRATA_BATCH_SELF_COMMIT");
+ if (o.batch < 2 || o.batch_groups <= 1) {
+ std::snprintf(buf, sizeof buf, "it needs 2 or more pipelined groups and %d slot%s in %d group%s are left", o.batch,
+ o.batch == 1 ? "" : "s", o.batch_groups, o.batch_groups == 1 ? "" : "s");
+ why = buf;
+ } else if (sc != nullptr && sc[0] != '\0' && sc[0] != '0') {
+ why = "it does not combine with STRATA_BATCH_SELF_COMMIT (the window would keep its rows itself)";
+ } else if ((o.batch / o.batch_groups) * (1 + pm_drafts) > (int) strata::kernels::kVerifyMaxT) {
+ std::snprintf(buf, sizeof buf, "a group's window would hold %d rows (%d slots x %d), more than %d",
+ (o.batch / o.batch_groups) * (1 + pm_drafts), o.batch / o.batch_groups, 1 + pm_drafts,
+ (int) strata::kernels::kVerifyMaxT);
+ why = buf;
+ }
+ if (why != nullptr) {
+ std::fprintf(stderr, "strata generate: WARNING: STRATA_PIPED_MTP is off: %s\n", why);
+ piped_mtp = false;
+ } else {
+ std::fprintf(stderr, "strata generate: STRATA_PIPED_MTP: %d pipelined group%s of %d slot%s draft up to %d MTP "
+ "proposal%s per slot each window\n", o.batch_groups, o.batch_groups == 1 ? "" : "s",
+ o.batch / o.batch_groups, o.batch / o.batch_groups == 1 ? "" : "s", pm_drafts,
+ pm_drafts > 1 ? "s" : "");
+ }
+ }
// Secure MTP's CUDA0 allocations before the large host arena is registered with both CUDA contexts.
// In particular WDDM can refuse the draft weights after mapping tens of GiB of host pages.
strata::core::MtpDrafter mtp;
@@ -5635,12 +5688,15 @@ int main(int argc, char** argv) {
// runs on the first stage, as for the solo drafter (`mtp.set_ple_session(&ss)` above).
for (int b = 0; b < o.batch; ++b) {
auto d = std::make_unique<strata::core::MtpDrafter>();
+ // STRATA_PIPED_MTP: the slot's proposal is drafted as an asynchronous chain (chain_launch), so the
+ // pipeline goes on while it runs: the chain graphs carry the forcing kernel (a no-op here)
+ if (piped_mtp) d->set_force_capture(true);
if (!d->load(o.mtp, draft_geometry, *bslot_ss.back()[(size_t) b], o.spec, err, o.mtp_window, &mtp)) {
std::fprintf(stderr, "strata generate: batch MTP slot %d: %s%s\n", b, err.c_str(),
vram_free_note().c_str());
return 1;
}
- d->set_max_drafts(1); // the first candidate verifies one proposal per slot
+ d->set_max_drafts(piped_mtp ? pm_drafts : 1); // the serial windows verify one proposal per slot
d->set_ple_session(bslot_ss[0][(size_t) b].get());
slot_mtp.push_back(std::move(d));
}
@@ -9146,7 +9202,9 @@ int main(int argc, char** argv) {
if (n_stages > 1) {
// pipelined slot groups write their rows at hbase = group * rows-per-group, so with more than
// kVerifyMaxT slots in groups the hand-off holds one row per slot (otherwise kVerifyMaxT, as before)
- const int hrows = !batch_mtp && o.batch_groups > 1 ? std::max<int>(strata::kernels::kVerifyMaxT, o.batch)
+ // (STRATA_PIPED_MTP: per slot its token and up to pm_drafts proposals)
+ const int hrows = piped_mtp ? std::max<int>(strata::kernels::kVerifyMaxT, (1 + pm_drafts) * o.batch)
+ : !batch_mtp && o.batch_groups > 1 ? std::max<int>(strata::kernels::kVerifyMaxT, o.batch)
: strata::kernels::kVerifyMaxT;
const size_t hb = (size_t) hrows * (size_t) strata::core::Verifier::handoff_floats(g) * sizeof(float);
std::vector<float*> hand((size_t) n_stages - 1, nullptr);
@@ -9215,9 +9273,9 @@ int main(int argc, char** argv) {
// --batch-mtp: every stage runs the same 2-rows-per-slot windows as the first (up to kVerifyMaxT
// rows, the hand-off's size) and bounds its captured batch layouts like the first (below)
if (batch_mtp) gs.ver.set_batch_graph_limit(64);
- ok_s = gs.ver.init(gs.wt, g, gs.ss, vs, gs.head.loaded() ? &gs.head : nullptr,
- batch_mtp ? strata::kernels::kVerifyMaxT
- : std::max(o.spec, std::min(o.batch, (int) strata::kernels::kVerifyMaxT)), err);
+ // (STRATA_PIPED_MTP: a group's window is its one slot's token and proposals, at most --spec rows,
+ // so the stage keeps the windows it has without --batch-mtp - the solo path's too)
+ ok_s = gs.ver.init(gs.wt, g, gs.ss, vs, gs.head.loaded() ? &gs.head : nullptr, batch_max_t(), err);
split_drive.cache_base[st] = gs.cache.device_slot(0);
split_drive.cache_slot_off[st] = gs.cache.slot_offsets();
split_drive.pcie_num[st] = pcie_num_of(gs.pcie_frac);
@@ -9297,7 +9355,7 @@ int main(int argc, char** argv) {
if (pcie_balance() && n_stages == 1) ver.set_pcie_balance(&drive.d.pcie_model);
if (batch_mtp) ver.set_batch_graph_limit(64); // --batch-mtp only: slot rotation makes many layouts (LRU); 0.1.39 keeps all
if (!ver.init(wt, g, ss, vh, native_head.loaded() || prefill_head_alias_bound ? &native_head : nullptr,
- batch_mtp ? strata::kernels::kVerifyMaxT : std::max(o.spec, std::min(o.batch, (int) strata::kernels::kVerifyMaxT)), err) ||
+ batch_max_t(), err) ||
(use_mtp && !prefill_head_alias_bound && !mtp.bind(last_st && !mtp_first ? last_st->wt : wt, last_st ? &last_st->head : &native_head,
pipe ? pl_mtp_R : ver.final_R_all(), err))) {
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
@@ -9321,7 +9379,8 @@ int main(int argc, char** argv) {
}
slot_mtp_rows.emplace_back(r, free_slot_mtp_rows);
if (!slot_mtp[(size_t) b]->bind(last_st ? last_st->wt : wt, last_st ? &last_st->head : &native_head, r, err,
- &mtp)) {
+ &mtp) ||
+ (piped_mtp && !slot_mtp[(size_t) b]->prepare_chain(err))) { // STRATA_PIPED_MTP: the chain graphs
std::fprintf(stderr, "strata serve: batch MTP slot %d: %s%s\n", b, err.c_str(),
vram_free_note().c_str());
return 1;
@@ -9339,6 +9398,12 @@ int main(int argc, char** argv) {
std::fprintf(stderr, "strata serve: --batch, stage %zu: %s\n", k + 1, err.c_str());
return 1;
}
+ // STRATA_PIPED_MTP: the rows a group's commit replays after the verdict, one hand-off row each
+ if (piped_mtp && bslot_ss.size() > 1 &&
+ !vk.init_deferred_commit(std::max<int>(strata::kernels::kVerifyMaxT, (1 + pm_drafts) * o.batch), err)) {
+ std::fprintf(stderr, "strata serve: STRATA_PIPED_MTP, stage %zu: %s\n", k + 1, err.c_str());
+ return 1;
+ }
}
}
for (int st = 0; st < n_stages && n_stages > 1; ++st) {
@@ -11490,6 +11555,7 @@ int main(int argc, char** argv) {
int32_t x = 0; ///< the token the next window feeds (not yet in the slot's state)
std::array<int32_t, strata::kernels::kVerifyMaxT> draft{}; ///< the slot's next MTP proposal
bool draft_ready = false;
+ int n_drafts = 1; ///< STRATA_PIPED_MTP: how many of `draft` its next window checks
int64_t p = 0; ///< its position
int64_t produced = 0, max_new = 0;
Clock::time_point t0;
@@ -11912,12 +11978,19 @@ int main(int argc, char** argv) {
const int n_pipe = (int) stages.size() + 1;
const bool piped = o.batch > 0 && o.batch_groups > 1 && n_pipe > 1;
strata::core::StageInFlightSampler stage_in_flight(n_pipe, piped && batch_telemetry_enabled);
- if (piped && batch_mtp) // #1413: said, not silent
+ const int GS = piped ? o.batch / o.batch_groups : o.batch;
+ // STRATA_PIPED_MTP: the pipelined groups draft too - a group's window carries each of its slots' rows (the slot's
+ // token and that slot's own proposals), so any number of slots per group drafts - and every stage saves the rows
+ // their commit replays
+ const int pmtp_rows_per_slot = 1 + pm_drafts; // the slot's token + its proposals
+ const int pmtp_rows_per_group = GS * pmtp_rows_per_slot; // the group's slots
+ const bool pmtp = piped && piped_mtp && ver.deferred_commit_on() &&
+ pmtp_rows_per_group <= batch_max_t();
+ if (piped && batch_mtp && !pmtp) // #1413: said, not silent
std::fprintf(stderr,
"strata generate: WARNING: --batch-groups %d runs the pipelined path, which builds one row per slot: "
"--batch-mtp's per-slot MTP drafts do not run there (their drafters still hold VRAM, about 0.9 GB per "
"slot). Drop --batch-mtp, or use --batch-groups 1.\n", o.batch_groups);
- const int GS = piped ? o.batch / o.batch_groups : o.batch;
if (GS > strata::kernels::kVerifyMaxT && !batch_mtp) { // (slots_in_windows keeps it so: a group's window is
// PGroup's rows, and the serial windows reach only the first kVerifyMaxT slots)
std::fprintf(stderr, "strata generate: --batch %d in %d group%s: more than %d rows per window\n",
@@ -11934,6 +12007,10 @@ int main(int argc, char** argv) {
int S = 0; ///< slots in its window: up to its last active one (idle ones cost rows)
int active = 0; ///< live slots at launch; S can also include idle padding rows
Clock::time_point started; ///< the group window's first stage launch
+ // STRATA_PIPED_MTP (pmtp): S counts ROWS - each of the group's slots' token, then that slot's proposals - at
+ // hand-off rows [hbase, hbase + S)
+ int rows[strata::kernels::kVerifyMaxT] = {};
+ int hbase = 0;
};
std::vector<PGroup> pg((size_t) (piped ? o.batch_groups : 0));
std::vector<int> stage_group((size_t) n_pipe, -1);
@@ -11955,6 +12032,159 @@ int main(int argc, char** argv) {
std::mutex bell_mu;
std::condition_variable bell_cv;
std::atomic<bool> bell_waiting{false};
+ // STRATA_PIPED_MTP: a slot whose next proposals are drafting (its chain in flight on the drafter's stream), and
+ // per slot the event after the copy of its rows on the last stage's stream, which the chain waits for
+ std::vector<char> pm_drafting(bs.size(), 0);
+ std::vector<cudaEvent_t> pm_copied(pmtp ? bs.size() : 0, nullptr);
+ if (pmtp) {
+ const strata::core::OnDevice on_last(stage_verifier(n_pipe - 1).device());
+ for (cudaEvent_t& e : pm_copied)
+ if (cudaEventCreateWithFlags(&e, cudaEventDisableTiming) != cudaSuccess) {
+ std::fprintf(stderr, "strata serve: STRATA_PIPED_MTP: event creation failed\n");
+ return 1;
+ }
+ }
+ // the chain of slot b has landed: its proposals are the slot's next drafts (1), still drafting (0), or an error (-1)
+ auto pmtp_poll_draft = [&](int b, bool wait) -> int {
+ if (!pm_drafting[(size_t) b]) return 1;
+ for (;;) {
+ const int r = slot_mtp[(size_t) b]->chain_poll(err);
+ if (r < 0) { std::printf("ERR batch MTP slot %d: %s\n", b, err.c_str()); return -1; }
+ if (r == 1) break;
+ if (!wait) return 0;
+ }
+ pm_drafting[(size_t) b] = 0;
+ // the first proposal always; another while the draft layer was at least --spec-min-p sure of the one before
+ // (the solo path's rule, MtpDrafter::draft)
+ BSlot& sl = bs[(size_t) b];
+ const int32_t* tok = slot_mtp[(size_t) b]->chain_tok();
+ const float* prob = slot_mtp[(size_t) b]->chain_prob();
+ sl.n_drafts = 1;
+ sl.draft[0] = tok[0];
+ while (sl.n_drafts < pm_drafts && prob[sl.n_drafts - 1] >= (float) o.spec_min_p) {
+ sl.draft[(size_t) sl.n_drafts] = tok[sl.n_drafts];
+ ++sl.n_drafts;
+ }
+ sl.draft_ready = true;
+ return 1;
+ };
+ // STRATA_PIPED_MTP: the last stage has picked for group gi's window (its slot's token and G.S - 1 proposals).
+ // Judge the proposals, queue every stage's commit of the kept rows (each stage runs it before that group's next
+ // window there: the same stream), emit them, and start drafting the next proposals.
+ auto pmtp_verdict = [&](PGroup& G, strata::core::Verifier& vk, const int32_t* outb) -> bool {
+ auto is_eos = [&](int32_t y) {
+ return std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) y) != o.eos_ids.end();
+ };
+ // each slot of the group owns a contiguous block of the window's rows: its token, then its own proposals
+ struct Blk { int b = 0, f = 0, n = 0, keep = 1; };
+ std::vector<Blk> blks;
+ std::vector<int> keep(bs.size(), 1);
+ for (int t = 0; t < G.S;) {
+ const int b = G.rows[t];
+ const int f = t;
+ while (t < G.S && G.rows[t] == b) ++t;
+ BSlot& sl = bs[(size_t) b];
+ // proposal j stands when the target picked it after the row before it (and every earlier one stood); the
+ // group was started only with proposals there is room to emit
+ Blk blk;
+ blk.b = b;
+ blk.f = f;
+ blk.n = t - f;
+ while (blk.keep < blk.n && !sl.stop && outb[f + blk.keep - 1] == G.tok[f + blk.keep] &&
+ !is_eos(outb[f + blk.keep - 1]))
+ ++blk.keep;
+ keep[(size_t) b] = blk.keep;
+ bt_accepted += blk.keep - 1;
+ bt_tokens += blk.n - 1; // (proposals checked)
+ blks.push_back(blk);
+ }
+ for (int k = 0; k < n_pipe; ++k)
+ if (!stage_verifier(k).batch_commit_rows(G.rows, G.S, G.hbase, keep.data(), G.tok, G.pos, err)) {
+ std::printf("ERR %s\n", err.c_str());
+ return false;
+ }
+ // #879: a row with non-finite logits: the slot that owns it ends, nothing of it is emitted
+ for (int t = 0; t < G.S; ++t)
+ if (vk.nonfinite_in(t)) {
+ const int b = G.rows[t];
+ BSlot& sl = bs[(size_t) b];
+ std::fprintf(stderr, "strata batch: non-finite logits (#879) in slot %d at position %lld: it ends (BDONE "
+ "nonfinite) and the next request reads its prompt from token 0\n", b, (long long) sl.p);
+ const double ms = std::chrono::duration<double, std::milli>(Clock::now() - sl.t0).count();
+ std::printf("BDONE %d %lld nonfinite %.1f\n", b, (long long) sl.produced, ms);
+ sl.active = false;
+ sl.cached = false;
+ sl.draft_ready = false;
+ sl.checks.clear();
+ nan_flush = true;
+ }
+ for (const Blk& blk : blks) {
+ BSlot& sl = bs[(size_t) blk.b];
+ if (!sl.active) continue; // it ended above (non-finite logits) or before this window
+ for (int j = 0; j < blk.keep; ++j) {
+ const int32_t y = outb[blk.f + j];
+ std::printf("BT %d %d\n", blk.b, (int) y);
+ ++sl.produced;
+ ++bt_rows;
+ ++pipe_rows;
+ const char* fin = is_eos(y) ? "stop" : sl.stop ? "cancel" : sl.produced >= sl.max_new ? "length"
+ : sl.p + 2 > o.max_context ? "length" : nullptr;
+ sl.ids.push_back(G.tok[blk.f + j]);
+ if (fin != nullptr) {
+ const double ms = std::chrono::duration<double, std::milli>(Clock::now() - sl.t0).count();
+ std::printf("BDONE %d %lld %s %.1f\n", blk.b, (long long) sl.produced, fin, ms);
+ sl.active = false;
+ sl.draft_ready = false;
+ // a cache again (#857), as on the pipelined path without drafts: final once its commits have run
+ // (pmtp_settle, before anything reads the slot's sessions)
+ sl.cached = o.prompt_cache > 0 && !sl.img && !sl.tainted;
+ break;
+ }
+ sl.x = y;
+ sl.p += 1;
+ }
+ }
+ // the next proposal: each slot's drafter catches up on ITS OWN rows of this window and drafts from the last
+ // kept one, as a chain on its own stream (pmtp_poll_draft collects it before the group's next window). The
+ // rows - the last stage's final residuals - are copied on THAT stage's stream, so its next window, which
+ // rewrites them, runs after the copy; the chain waits for the copy.
+ const size_t stride = (size_t) g.hc * (size_t) g.n_embd;
+ for (const Blk& blk : blks) {
+ BSlot& sl = bs[(size_t) blk.b];
+ if (!sl.active) continue; // it ended: no more drafts for it
+ const strata::core::OnDevice on_slot_mtp(slot_mtp[(size_t) blk.b]->device());
+ sl.draft_ready = false;
+ const float* src = vk.final_R_all() + (size_t) blk.f * stride;
+ if (cudaMemcpyAsync(slot_mtp_rows[(size_t) blk.b].get(), src, (size_t) blk.n * stride * sizeof(float),
+ cudaMemcpyDeviceToDevice, vk.stream()) != cudaSuccess ||
+ cudaEventRecord(pm_copied[(size_t) blk.b], vk.stream()) != cudaSuccess ||
+ cudaStreamWaitEvent(slot_mtp[(size_t) blk.b]->stream(), pm_copied[(size_t) blk.b], 0) != cudaSuccess ||
+ !slot_mtp[(size_t) blk.b]->chain_launch(blk.n, outb + blk.f, G.pos[blk.f], blk.keep - 1, nullptr, 0,
+ pm_drafts, 0, err)) {
+ std::printf("ERR batch MTP slot %d: %s\n", blk.b,
+ err.empty() ? "the proposal's chain did not start" : err.c_str());
+ return false;
+ }
+ pm_drafting[(size_t) blk.b] = 1;
+ }
+ return true;
+ };
+ // STRATA_PIPED_MTP: the commits queued after the verdicts have run on every stage, so whatever reads a slot's
+ // sessions next (an admission, a parked conversation, the solo path taking a slot back) sees its final state
+ auto pmtp_settle = [&]() -> bool {
+ if (!pmtp) return true;
+ for (int b = 0; b < (int) bs.size(); ++b) // (a chain in flight would refuse the drafter's next use)
+ if (pmtp_poll_draft(b, true) < 0) return false;
+ for (int k = 0; k < n_pipe; ++k) {
+ strata::core::Verifier& vk = stage_verifier(k);
+ const strata::core::OnDevice on_k(vk.device());
+ if (cudaStreamSynchronize(vk.stream()) != cudaSuccess) {
+ std::printf("ERR a pipelined commit failed on stage %d\n", k + 1);
+ return false;
+ }
+ }
+ return true;
+ };
// one turn: serve every running stage, hand finished groups on, start what can start
auto pump = [&](bool may_start) -> bool {
++pipe_tick;
@@ -11976,7 +12206,8 @@ int main(int argc, char** argv) {
if (k + 1 < n_pipe) { G.stage = k + 1; G.since = pipe_tick; continue; }
// the last stage: the group's picks
const int32_t* outb = vk.batch_out();
- for (int t = 0; t < G.S; ++t) {
+ if (pmtp && !pmtp_verdict(G, vk, outb)) return false;
+ for (int t = 0; t < G.S && !pmtp; ++t) {
BSlot& sl = bs[(size_t) (gi * GS + t)];
if (!sl.active) continue;
if (vk.nonfinite_in(t)) { // #879: the slot ends, nothing emitted
@@ -12037,22 +12268,58 @@ int main(int argc, char** argv) {
if (!may_start) continue;
for (int j = 0; j < (int) pg.size() && pick < 0; ++j) {
const int gi = (int) ((rr + j) % (int64_t) pg.size());
- if (!pg[(size_t) gi].inflight && group_active(gi)) pick = gi;
+ if (pg[(size_t) gi].inflight || !group_active(gi)) continue;
+ if (pmtp) { // its slots' proposal chains still running: another group goes first
+ bool ready = true;
+ for (int t = 0; t < GS; ++t) {
+ const int dr = pmtp_poll_draft(gi * GS + t, false);
+ if (dr < 0) return false;
+ if (dr == 0) { ready = false; break; }
+ }
+ if (!ready) continue;
+ }
+ pick = gi;
}
if (pick < 0) continue;
rr = pick + 1;
PGroup& G = pg[(size_t) pick];
G.S = 0;
- G.active = 0;
- for (int t = 0; t < GS; ++t) if (bs[(size_t) (pick * GS + t)].active) {
- G.S = t + 1;
- ++G.active;
- }
- for (int t = 0; t < G.S; ++t) {
- BSlot& sl = bs[(size_t) (pick * GS + t)];
- G.tok[t] = sl.active ? sl.x : 0;
- G.pos[t] = sl.active ? sl.p : 0;
- if (!sl.active) sl.cached = false; // its pad row writes its state
+ if (pmtp) { // the group's slots: each its token, then its own proposals - as many as could be emitted
+ G.hbase = pmtp_rows_per_group * pick;
+ G.S = 0;
+ G.active = 0;
+ for (int t = 0; t < GS; ++t) {
+ const int b = pick * GS + t;
+ const BSlot& sl = bs[(size_t) b];
+ if (!sl.active) continue; // an idle slot carries no row (its state is left untouched)
+ ++G.active;
+ G.rows[G.S] = b;
+ G.tok[G.S] = sl.x;
+ G.pos[G.S] = sl.p;
+ ++G.S;
+ int nd = sl.draft_ready && !sl.stop ? sl.n_drafts : 0;
+ nd = (int) std::min<int64_t>(nd, std::min<int64_t>(sl.max_new - sl.produced - 1,
+ o.max_context - sl.p - 2));
+ for (int j = 0; j < nd; ++j) {
+ G.rows[G.S] = b;
+ G.tok[G.S] = sl.draft[(size_t) j];
+ G.pos[G.S] = sl.p + 1 + j;
+ ++G.S;
+ }
+ }
+ if (G.S == 0) { G.inflight = false; continue; } // every slot of the group went idle
+ } else {
+ G.active = 0;
+ for (int t = 0; t < GS; ++t) if (bs[(size_t) (pick * GS + t)].active) {
+ G.S = t + 1;
+ ++G.active;
+ }
+ for (int t = 0; t < G.S; ++t) {
+ BSlot& sl = bs[(size_t) (pick * GS + t)];
+ G.tok[t] = sl.active ? sl.x : 0;
+ G.pos[t] = sl.active ? sl.p : 0;
+ if (!sl.active) sl.cached = false; // its pad row writes its state
+ }
}
G.inflight = true;
G.stage = 0;
@@ -12069,7 +12336,8 @@ int main(int argc, char** argv) {
PGroup& G = pg[(size_t) pick];
strata::core::progress().busy.store(true);
if (batch_telemetry_enabled) stage_in_flight.launch(k, Clock::now());
- if (!stage_verifier(k).batch_launch(pick * GS, G.S, G.tok, G.pos, err)) {
+ if (!(pmtp ? stage_verifier(k).batch_launch_rows(G.rows, G.S, G.hbase, G.tok, G.pos, err)
+ : stage_verifier(k).batch_launch(pick * GS, G.S, G.tok, G.pos, err))) {
std::printf("ERR %s\n", err.c_str());
return false;
}
@@ -12092,16 +12360,21 @@ int main(int argc, char** argv) {
std::fflush(stdout);
}
}
+ // every slot done: the last commits have run before anything reads a slot (between windows nothing does)
+ if (pmtp && !pipe_inflight() && !batch_on() && !pmtp_settle()) return false;
if (!pipe_inflight() && !batch_on() && bt_windows > 0) { // all idle: the timing line, as batch_step
const double w = (double) bt_windows, wall = std::chrono::duration<double, std::milli>(Clock::now() - bt_start).count();
std::fprintf(stderr, "strata batch (pipelined, %d groups of %d): %lld group-steps, %lld rows in %.0f ms = "
- "%.1f rows/s (admissions included)\n", o.batch_groups, GS, (long long) bt_windows,
+ "%.1f rows/s (admissions included)", o.batch_groups, GS, (long long) bt_windows,
(long long) bt_rows, wall, 1000.0 * bt_rows / std::max(wall, 1e-9));
+ if (pmtp)
+ std::fprintf(stderr, ", %lld of %lld MTP proposals kept", (long long) bt_accepted, (long long) bt_tokens);
+ std::fprintf(stderr, "\n");
for (int k = 0; k < n_pipe; ++k) {
const std::string pr = stage_verifier(k).profile_report();
if (!pr.empty()) std::fprintf(stderr, "strata batch GPU stages, stage %d (ms/window):%s\n", k + 1, pr.c_str());
}
- bt_windows = bt_rows = 0;
+ bt_windows = bt_rows = bt_tokens = bt_accepted = 0;
// (not while a prompt is read in a request: its windows ran beside it, and the watchdog stays on for
// the rest of the read, STRATA_PIPED_CONCURRENT; the request's end turns it off)
if (!in_prompt_read) strata::core::progress().busy.store(false);
@@ -12111,7 +12384,7 @@ int main(int argc, char** argv) {
auto pipe_drain = [&]() -> bool {
while (pipe_inflight())
if (!pump(false)) return false;
- return true;
+ return pmtp_settle();
};
for (;;) {
line_drain_ms = 0.0;
@@ -15624,12 +15897,15 @@ int main(int argc, char** argv) {
sl.t0 = Clock::now();
sl.ids = live;
if (batch_mtp) {
+ int nd = 1; // (STRATA_PIPED_MTP: up to its proposals per window, as --spec-min-p allows)
if (!slot_mtp[(size_t) admit_slot]->draft_first(1, ver.final_R_all(), sl.x,
- sl.p - 1, sl.draft.data(), err)) {
+ sl.p - 1, sl.draft.data(), err, nullptr,
+ piped_mtp ? (float) o.spec_min_p : 0.0f, &nd)) {
std::fprintf(stderr, "strata batch: MTP admission for slot %d failed: %s\n",
admit_slot, err.c_str());
return 1;
}
+ sl.n_drafts = std::max(1, nd);
sl.draft_ready = true;
}
sl.cvec = cvec_cached;