Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

batch groups: two NVIDIA cards pipeline by default; the serial group is opt-in (STRATA_BATCH_GROUPS_NV_SERIAL=1, STRATA_DEFAULTS=026)

Niko1221ec8e405909f8
diff --git a/docs/BATCHING.md b/docs/BATCHING.md
index 20201490f..70c09fb51 100644
--- a/docs/BATCHING.md
+++ b/docs/BATCHING.md
@@ -449,6 +449,30 @@ compares the tier as a whole, not "batch windows only": the solo decode speed mo
more varied workload the gain may be smaller (not measured). Upstream main runs no tier in batch windows, and this
was not measured against an upstream binary.
+## Two NVIDIA cards: the pipelined groups are the default (0.2.7)
+
+With a layer split over exactly two NVIDIA cards, `--batch` and no `--batch-groups`, 0.2.1 to 0.2.6 ran one serial group
+(measured then on two RTX A4000, IQ3_XXS: +12 to 16% against the pipelined groups). From 0.2.7 the default is the pipelined
+groups, like on three or more cards. `STRATA_BATCH_GROUPS_NV_SERIAL=1` or `STRATA_DEFAULTS=026` gives the serial group back;
+`--batch-groups 1` asks for one group, `--batch-groups auto` or a number asks for the pipeline, and an explicit
+`STRATA_BATCH_GROUPS_NV_SERIAL=0` wins over `STRATA_DEFAULTS=026`.
+
+Why: with the 0.2.7 code the pipelined groups were faster at two and four streams on every box and quant measured
+(`tools/serve_load.py`, `--batch 4`, 256 tokens per request, interleaved pairs, total tokens per second, pipelined / serial):
+
+| box, model | experts in VRAM | 1 stream | 2 streams | 4 streams |
+| --- | --- | ---: | ---: | ---: |
+| 2x RTX A6000 48 GB, IQ3_S | all | 101.7 / 101.7 | 125.7 / 107.4 (+17%) | 171.4 / 141.0 (+22%) |
+| 2x RTX A4000 16 GB, Q2_0 | 17037 of 24576 pairs | 63.6-68.7 / 68.2-70.3 | 80.2-82.5 / 75.0-75.5 (+7 to 9%) | 110.8-111.5 / 94.8-95.3 (+17%) |
+| 2x RTX A4000 16 GB, IQ3_XXS | 13056 of 24576 pairs | 58.4-60.5 / 60.7-62.6 | 73.2-73.7 / 65.8-66.5 (+12%) | 100.5-100.7 / 82.9-83.6 (+20%) |
+
+The pipelined side was faster in every pair at two and four streams (3 pairs on the A6000, repeated in 3 more with the v0.2.6 binary; 4 clean pairs on the A4000
+with Q2_0, 3 on the A4000 with IQ3_XXS). The v0.2.6 binary measured the same on the A6000 (serial default 104.9 and 137.2
+tok/s at 2 and 4 streams, `--batch-groups auto` 124.6 and 169.6). A single stream runs on the solo path in both; on the A4000s
+the serial setting was about 4% faster for it (3 of 3 pairs with IQ3_XXS), on the A6000s it was equal. The 0.2.1 result
+(serial faster) did not repeat. Every slot's tokens equal its solo decode with the default (4 and 2 slots, greedy and seeded
+sampled streams, finishing slots, a cancelled slot).
+
## MTP drafts in the pipelined groups (`STRATA_PIPED_MTP`, opt-in)
The pipelined groups run no drafts. `--batch-mtp` with serial windows (`--batch-groups 1`) keeps one card busy at a
@@ -473,7 +497,7 @@ Measured on a box with 2x RTX A6000 48 GB (`--layer-split auto`), Qwen3.8-Flash-
VRAM, `--batch 4 --batch-groups auto` (2 groups of 2 slots), `tools/serve_load.py`, 256 tokens per request, reasoning
off. Total output tokens per second: the median of the pairs, each pair the median of 3 bursts on a fresh engine start. The base arm
(pipelined groups, no drafts) and `STRATA_PIPED_MTP=2` were run in 5 interleaved pairs, the other arms in 3 pairs. "Default" is
-`--batch 4` with no `--batch-groups`, which on exactly two NVIDIA cards means one serial group (`STRATA_BATCH_GROUPS_NV_SERIAL`):
+`--batch 4` with no `--batch-groups` as it ran in 0.2.1 to 0.2.6 on exactly two NVIDIA cards: one serial group (see the next section):
| streams | pipelined, no drafts | `=1` | `=2` | `=3` | serial `--batch-mtp` (`--batch-groups 1`) | default (no flags) |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
diff --git a/include/strata/program/batch_groups.hpp b/include/strata/program/batch_groups.hpp
index 168019b6b..345ace009 100644
--- a/include/strata/program/batch_groups.hpp
+++ b/include/strata/program/batch_groups.hpp
@@ -57,7 +57,7 @@ struct Input {
bool batch_mtp = false; ///< --batch-mtp is on (it passed the gates that do not concern the groups)
bool adapt_async = false; ///< --adapt-async 1 was asked for
bool mtp_serial = false; ///< STRATA_BATCH_MTP_SERIAL=1: --batch-mtp with no --batch-groups runs one serial group (0.1.43)
- bool nv_serial_faster = false; ///< PROPOSED (0.2.1, owner decides): NVIDIA build, STRATA_BATCH_GROUPS_NV_SERIAL not 0 - two cards run faster in serial windows (z440 2x A4000, aggregate tok/s: +16% at 2 slots, +12% at 4, +15% at 8)
+ bool nv_serial_faster = false; ///< opt-in (see nv_serial_wanted): NVIDIA build, two cards, STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 - one serial group, the 0.2.1-0.2.6 default
bool mtp_piped = false; ///< STRATA_PIPED_MTP=1: the pipelined groups draft too (each group carries its slots' rows)
};
@@ -70,6 +70,16 @@ struct Result {
bool mtp_piped = false; ///< STRATA_PIPED_MTP: --batch-mtp's drafts run in the pipelined groups
};
+/// Whether two NVIDIA cards run one serial group when --batch-groups is not given. Not by default since 0.2.7: the pipelined groups
+/// measured faster on 2x RTX A6000 (IQ3_S, every expert resident) and on 2x RTX A4000 (Q2_0 pack); the serial default of 0.2.1-0.2.6
+/// had measured faster on 2x A4000 with IQ3_XXS. STRATA_BATCH_GROUPS_NV_SERIAL asks for it with any value but 0 (an explicit 0 or
+/// non-zero value wins over the master switch); with the variable unset or empty, STRATA_DEFAULTS=026 (v0.2.6's behaviour) asks for
+/// it. `env_value` is the variable's value (null when unset).
+inline bool nv_serial_wanted(const char* env_value, bool defaults_026) {
+ if (env_value != nullptr && env_value[0] != '\0') return env_value[0] != '0';
+ return defaults_026;
+}
+
/// The most groups, at most one per stage of the layer split (`later_stages` + 1 GPUs), that divide `batch`.
inline int auto_groups(int later_stages, int batch) {
int best = 1;
@@ -107,7 +117,7 @@ inline Result resolve(const Input& in) {
inline const char* serial_what(Serial s) {
return s == Serial::BatchMtp ? "--batch-mtp" : s == Serial::AdaptAsync ? "--adapt-async 1"
- : s == Serial::TwoCardsSerialFaster ? "serial windows (measured faster than pipelined groups on two NVIDIA cards; STRATA_BATCH_GROUPS_NV_SERIAL=0 keeps the pipeline)" : "";
+ : s == Serial::TwoCardsSerialFaster ? "serial windows (STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 asked for them on two NVIDIA cards; --batch-groups auto pipelines)" : "";
}
} // namespace strata::program::batch_groups
diff --git a/src/program/batch_groups_test.cpp b/src/program/batch_groups_test.cpp
index 3c9788b4a..2d0657bf8 100644
--- a/src/program/batch_groups_test.cpp
+++ b/src/program/batch_groups_test.cpp
@@ -107,6 +107,20 @@ int main() {
check(r.groups == 2 && r.mtp_piped && !r.mtp_dropped,
"STRATA_PIPED_MTP, --batch-groups 2 given: 2 groups of 2 slots, and the drafts stay");
}
+ { // 0.2.7: two NVIDIA cards pipeline by default; the serial default is opt-in (STRATA_BATCH_GROUPS_NV_SERIAL=1, STRATA_DEFAULTS=026)
+ check(!bg::nv_serial_wanted(nullptr, false), "NV_SERIAL unset: the pipeline");
+ check(!bg::nv_serial_wanted("", false), "NV_SERIAL empty: the pipeline");
+ check(!bg::nv_serial_wanted("0", false), "NV_SERIAL=0: the pipeline");
+ check(bg::nv_serial_wanted("1", false), "NV_SERIAL=1: serial");
+ check(bg::nv_serial_wanted("yes", false), "NV_SERIAL=yes: serial");
+ check(bg::nv_serial_wanted(nullptr, true), "STRATA_DEFAULTS=026: serial (v0.2.6's behaviour)");
+ check(!bg::nv_serial_wanted("0", true), "an explicit NV_SERIAL=0 wins over STRATA_DEFAULTS=026");
+ check(bg::nv_serial_wanted("", true), "NV_SERIAL empty and STRATA_DEFAULTS=026: serial");
+ bg::Input i = in(4, 1);
+ i.nv_serial_faster = bg::nv_serial_wanted(nullptr, false);
+ auto r = bg::resolve(i);
+ check(r.groups == 2 && r.serial == bg::Serial::None, "2 cards, 4 slots, nothing given: 2 pipelined groups, whatever the residency");
+ }
{ // STRATA_PIPED_MTP unset: nothing changes, whatever --batch-mtp and the groups say
bg::Input i = in(4, 1);
i.batch_mtp = true;
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 65470b244..b69f71f89 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -5957,7 +5957,10 @@ int main(int argc, char** argv) {
bin.adapt_async = o.adapt_async != 0;
{ const char* ms = std::getenv("STRATA_BATCH_MTP_SERIAL"); bin.mtp_serial = ms != nullptr && ms[0] != '\0' && ms[0] != '0'; }
#if !defined(STRATA_USE_HIP)
- { const char* nv = std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"); bin.nv_serial_faster = nv == nullptr || nv[0] != '0'; } // PROPOSED 0.2.1 default (NVIDIA, 2 cards)
+ // NVIDIA, 2 cards: the pipelined groups by default (0.2.7); one serial group, the 0.2.1-0.2.6 default, with
+ // STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 (batch_groups.hpp nv_serial_wanted)
+ bin.nv_serial_faster = strata::program::batch_groups::nv_serial_wanted(std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"),
+ strata::qwen_defaults_026());
#endif
{ const char* mp = std::getenv("STRATA_PIPED_MTP"); bin.mtp_piped = mp != nullptr && mp[0] != '\0' && mp[0] != '0'; }
return strata::program::batch_groups::resolve(bin);