Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Batch groups: PROPOSED NVIDIA two-card default is one serial group (measured +12..16% on 2x A4000); owner decides

Niko1221f680a80d885b
diff --git a/docs/BATCHING.md b/docs/BATCHING.md
index 55aac17e1..6e7b58d39 100644
--- a/docs/BATCHING.md
+++ b/docs/BATCHING.md
@@ -89,7 +89,7 @@ With a layer split, the engine options go into the config's `args`:
| --- | --- |
| `"parallel": N` / `--batch N` / `--slots N` (2..8 normally; with a layer split up to 8 per group, see below) | up to N conversations have batch slots; more requests wait for a free slot. Each slot gets its own state (a session carved like the stage's own: GDN recurrence, QSA K/V and indexer, PLE history) on every GPU of the split. With grouped MTP, more than 8 slots can rotate through eight-row windows if memory permits. |
| `--batch-groups G` | with a layer split (default: auto, below): the N slots in G groups that flow through the GPUs as a pipeline (GPU k runs one group while GPU k+1 runs another). G must divide N. 1 = all slots in one window, GPU after GPU. |
-| `--batch-groups auto` | the default on a layer split since 0.1.41 (give no `--batch-groups`): the engine pipelines one group per GPU stage (the most that divide the slots; 8 slots on 4 GPUs = 4 groups of 2) and says so (`INFO batch_groups=G`). `--batch-groups 1` turns it off (all slots in one window, GPU after GPU). Measured, 8 clients, total tok/s against one group: 4 x R9700 166 against 86, 2 GPUs 109 against 78, 3 GPUs 110 against 78. With `--adapt-async 1` and no `--batch-groups` the default is one group (the adaptive tier between batch windows does not run in pipelined groups) and the log says so; with `--batch-mtp` and no `--batch-groups` the pipeline wins and `--batch-mtp` is left off (`--batch-groups 1` or `STRATA_BATCH_MTP_SERIAL=1` keeps it); `--batch-groups G` or `--batch-groups auto` given with `--batch-mtp` pipelines the groups and turns `--batch-mtp` off, and given with `--adapt-async 1` pipelines the groups and turns `--adapt-async` off (the pipelined path does not adapt). |
+| `--batch-groups auto` | the default on a layer split since 0.1.41 (give no `--batch-groups`): the engine pipelines one group per GPU stage (the most that divide the slots; 8 slots on 4 GPUs = 4 groups of 2) and says so (`INFO batch_groups=G`). `--batch-groups 1` turns it off (all slots in one window, GPU after GPU). Measured, 8 clients, total tok/s against one group: 4 x R9700 166 against 86, 2 GPUs 109 against 78, 3 GPUs 110 against 78. With `--adapt-async 1` and no `--batch-groups` the default is one group (the adaptive tier between batch windows does not run in pipelined groups) and the log says so; with `--batch-mtp` and no `--batch-groups` the pipeline wins and `--batch-mtp` is left off (`--batch-groups 1` or `STRATA_BATCH_MTP_SERIAL=1` keeps it); `--batch-groups G` or `--batch-groups auto` given with `--batch-mtp` pipelines the groups and turns `--batch-mtp` off, and given with `--adapt-async 1` pipelines the groups and turns `--adapt-async` off (the pipelined path does not adapt). | PROPOSED for 0.2.1 (owner decides; branch f-021-split): on NVIDIA builds with exactly two stages the default is one group instead (2x RTX A4000, IQ3_XXS, aggregate tok/s against the pipelined default: +16% at 2 slots, +12% at 4, +15% at 8, 5 of 5 pairs each; `--batch-groups auto` or a number asks for the pipeline, `STRATA_BATCH_GROUPS_NV_SERIAL=0` restores the old default).
| `--trim-stage-weights` | with an **explicit** `--layer-split` (e.g. `12,24,36`, not `auto`): every GPU loads only the dense weights of its own layers instead of the whole model's (the same as `STRATA_STAGE_TRIM=1`, PR #639). The VRAM this frees goes to the expert cache. Useful without `--batch` too. |
The engine never refuses a count it cannot run: it says so in its log and runs what it can - at most 8 slots without a
diff --git a/include/strata/program/batch_groups.hpp b/include/strata/program/batch_groups.hpp
index 0abff1f88..b1e3f32cc 100644
--- a/include/strata/program/batch_groups.hpp
+++ b/include/strata/program/batch_groups.hpp
@@ -16,7 +16,7 @@
namespace strata::program::batch_groups {
/// What kept `auto` from pipelining.
-enum class Serial { None, BatchMtp, AdaptAsync };
+enum class Serial { None, BatchMtp, AdaptAsync, TwoCardsSerialFaster };
struct Input {
bool set = false; ///< --batch-groups was given (a number or `auto`)
@@ -27,6 +27,7 @@ struct Input {
bool batch_mtp = false; ///< --batch-mtp is on (it passed the gates that do not concern the groups)
bool adapt_async = false; ///< --adapt-async 1 was asked for
bool mtp_serial = false; ///< STRATA_BATCH_MTP_SERIAL=1: --batch-mtp with no --batch-groups runs one serial group (0.1.43)
+ bool nv_serial_faster = false; ///< PROPOSED (0.2.1, owner decides): NVIDIA build, STRATA_BATCH_GROUPS_NV_SERIAL not 0 - two cards run faster in serial windows (z440 2x A4000, aggregate tok/s: +16% at 2 slots, +12% at 4, +15% at 8)
};
struct Result {
@@ -57,6 +58,7 @@ inline Result resolve(const Input& in) {
r.groups = r.would_be;
if (!in.set && r.would_be > 1) { // only the default gives way; `--batch-groups auto` asked for the pipeline
if (in.batch_mtp && in.mtp_serial) r.serial = Serial::BatchMtp;
+ else if (in.nv_serial_faster && in.later_stages == 1 && !in.batch_mtp && !in.adapt_async) r.serial = Serial::TwoCardsSerialFaster;
else if (in.adapt_async) r.serial = Serial::AdaptAsync;
r.mtp_dropped = in.batch_mtp && r.serial == Serial::None;
if (r.serial != Serial::None) r.groups = 1;
@@ -65,7 +67,8 @@ inline Result resolve(const Input& in) {
}
inline const char* serial_what(Serial s) {
- return s == Serial::BatchMtp ? "--batch-mtp" : s == Serial::AdaptAsync ? "--adapt-async 1" : "";
+ return s == Serial::BatchMtp ? "--batch-mtp" : s == Serial::AdaptAsync ? "--adapt-async 1"
+ : s == Serial::TwoCardsSerialFaster ? "serial windows (measured faster than pipelined groups on two NVIDIA cards; STRATA_BATCH_GROUPS_NV_SERIAL=0 keeps the pipeline)" : "";
}
} // namespace strata::program::batch_groups
diff --git a/src/program/batch_groups_test.cpp b/src/program/batch_groups_test.cpp
index ecc519292..5debcd29d 100644
--- a/src/program/batch_groups_test.cpp
+++ b/src/program/batch_groups_test.cpp
@@ -78,6 +78,25 @@ int main() {
const auto r = bg::resolve(i);
check(r.groups == 1 && r.serial == bg::Serial::None && r.would_be == 1, "auto would be 1 group anyway: no reason");
}
+ { // PROPOSED NVIDIA default: two cards, nothing given: serial windows (measured +12..16% on 2x A4000)
+ bg::Input i = in(4, 1);
+ i.nv_serial_faster = true;
+ auto r = bg::resolve(i);
+ check(r.groups == 1 && r.from_auto && r.serial == bg::Serial::TwoCardsSerialFaster && r.would_be == 2,
+ "nv_serial_faster, 2 cards, 4 slots, nothing given: one group, named");
+ check(bg::serial_what(r.serial)[0] == 's', "the reason names the measurement");
+ check(bg::resolve(in(4, 1)).groups == 2, "flag off: the pipeline default stays");
+ i = in(8, 1); i.nv_serial_faster = true; // measured too: +15%
+ check(bg::resolve(i).groups == 1, "8 slots (measured +15%): serial too");
+ i = in(4, 3); i.nv_serial_faster = true;
+ check(bg::resolve(i).groups == 4, "4 cards (not measured): the pipeline stays");
+ i = in(4, 1); i.nv_serial_faster = true; i.batch_mtp = true;
+ check(bg::resolve(i).groups == 2 && bg::resolve(i).mtp_dropped, "with --batch-mtp the 0.1.43.1 rule is unchanged");
+ i = in(4, 1); i.nv_serial_faster = true; i.set = true; i.auto_given = true;
+ check(bg::resolve(i).groups == 2, "--batch-groups auto given: the pipeline");
+ i.auto_given = false; i.given = 2;
+ check(bg::resolve(i).groups == 2, "--batch-groups 2 given: 2");
+ }
{ // explicit values are honoured
bg::Input i = in(4, 1);
i.set = true;
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 01b99f06c..7ee272623 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -5108,6 +5108,9 @@ int main(int argc, char** argv) {
bin.batch_mtp = mtp_on;
bin.adapt_async = o.adapt_async != 0;
{ const char* ms = std::getenv("STRATA_BATCH_MTP_SERIAL"); bin.mtp_serial = ms != nullptr && ms[0] != '\0' && ms[0] != '0'; }
+#if !defined(STRATA_USE_HIP)
+ { const char* nv = std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"); bin.nv_serial_faster = nv == nullptr || nv[0] != '0'; } // PROPOSED 0.2.1 default (NVIDIA, 2 cards)
+#endif
return strata::program::batch_groups::resolve(bin);
};
const char* batch_mtp_env = std::getenv("STRATA_BATCH_MTP");
@@ -5179,7 +5182,11 @@ int main(int argc, char** argv) {
o.batch_groups = r.groups;
if (groups_said == r.groups) return; // (the second call, once the slots that fit are known, says nothing new)
groups_said = r.groups;
- if (r.serial != strata::program::batch_groups::Serial::None)
+ if (r.serial == strata::program::batch_groups::Serial::TwoCardsSerialFaster)
+ std::fprintf(stderr, "strata generate: --batch-groups auto: 1 group of %d slot%s - %s; --batch-groups %d asks for the "
+ "pipelined groups\n", o.batch, o.batch == 1 ? "" : "s",
+ strata::program::batch_groups::serial_what(r.serial), r.would_be);
+ else if (r.serial != strata::program::batch_groups::Serial::None)
std::fprintf(stderr, "strata generate: --batch-groups auto: 1 group of %d slot%s - %s does not run in pipelined "
"groups; --batch-groups %d would pipeline them and turn it off\n", o.batch,
o.batch == 1 ? "" : "s", strata::program::batch_groups::serial_what(r.serial), r.would_be);