Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
batch groups: two NVIDIA cards pipeline by default; the serial group is opt-in (STRATA_BATCH_GROUPS_NV_SERIAL=1, STRATA_DEFAULTS=026)
Niko1221ec8e405909f8
diff --git a/docs/BATCHING.md b/docs/BATCHING.mdindex 20201490f..70c09fb51 100644--- a/docs/BATCHING.md+++ b/docs/BATCHING.md@@ -449,6 +449,30 @@ compares the tier as a whole, not "batch windows only": the solo decode speed momore varied workload the gain may be smaller (not measured). Upstream main runs no tier in batch windows, and thiswas not measured against an upstream binary.+## Two NVIDIA cards: the pipelined groups are the default (0.2.7)++With a layer split over exactly two NVIDIA cards, `--batch` and no `--batch-groups`, 0.2.1 to 0.2.6 ran one serial group+(measured then on two RTX A4000, IQ3_XXS: +12 to 16% against the pipelined groups). From 0.2.7 the default is the pipelined+groups, like on three or more cards. `STRATA_BATCH_GROUPS_NV_SERIAL=1` or `STRATA_DEFAULTS=026` gives the serial group back;+`--batch-groups 1` asks for one group, `--batch-groups auto` or a number asks for the pipeline, and an explicit+`STRATA_BATCH_GROUPS_NV_SERIAL=0` wins over `STRATA_DEFAULTS=026`.++Why: with the 0.2.7 code the pipelined groups were faster at two and four streams on every box and quant measured+(`tools/serve_load.py`, `--batch 4`, 256 tokens per request, interleaved pairs, total tokens per second, pipelined / serial):++| box, model | experts in VRAM | 1 stream | 2 streams | 4 streams |+| --- | --- | ---: | ---: | ---: |+| 2x RTX A6000 48 GB, IQ3_S | all | 101.7 / 101.7 | 125.7 / 107.4 (+17%) | 171.4 / 141.0 (+22%) |+| 2x RTX A4000 16 GB, Q2_0 | 17037 of 24576 pairs | 63.6-68.7 / 68.2-70.3 | 80.2-82.5 / 75.0-75.5 (+7 to 9%) | 110.8-111.5 / 94.8-95.3 (+17%) |+| 2x RTX A4000 16 GB, IQ3_XXS | 13056 of 24576 pairs | 58.4-60.5 / 60.7-62.6 | 73.2-73.7 / 65.8-66.5 (+12%) | 100.5-100.7 / 82.9-83.6 (+20%) |++The pipelined side was faster in every pair at two and four streams (3 pairs on the A6000, repeated in 3 more with the v0.2.6 binary; 4 clean pairs on the A4000+with Q2_0, 3 on the A4000 with IQ3_XXS). The v0.2.6 binary measured the same on the A6000 (serial default 104.9 and 137.2+tok/s at 2 and 4 streams, `--batch-groups auto` 124.6 and 169.6). A single stream runs on the solo path in both; on the A4000s+the serial setting was about 4% faster for it (3 of 3 pairs with IQ3_XXS), on the A6000s it was equal. The 0.2.1 result+(serial faster) did not repeat. Every slot's tokens equal its solo decode with the default (4 and 2 slots, greedy and seeded+sampled streams, finishing slots, a cancelled slot).+## MTP drafts in the pipelined groups (`STRATA_PIPED_MTP`, opt-in)The pipelined groups run no drafts. `--batch-mtp` with serial windows (`--batch-groups 1`) keeps one card busy at a@@ -473,7 +497,7 @@ Measured on a box with 2x RTX A6000 48 GB (`--layer-split auto`), Qwen3.8-Flash-VRAM, `--batch 4 --batch-groups auto` (2 groups of 2 slots), `tools/serve_load.py`, 256 tokens per request, reasoningoff. Total output tokens per second: the median of the pairs, each pair the median of 3 bursts on a fresh engine start. The base arm(pipelined groups, no drafts) and `STRATA_PIPED_MTP=2` were run in 5 interleaved pairs, the other arms in 3 pairs. "Default" is-`--batch 4` with no `--batch-groups`, which on exactly two NVIDIA cards means one serial group (`STRATA_BATCH_GROUPS_NV_SERIAL`):+`--batch 4` with no `--batch-groups` as it ran in 0.2.1 to 0.2.6 on exactly two NVIDIA cards: one serial group (see the next section):| streams | pipelined, no drafts | `=1` | `=2` | `=3` | serial `--batch-mtp` (`--batch-groups 1`) | default (no flags) || ---: | ---: | ---: | ---: | ---: | ---: | ---: |diff --git a/include/strata/program/batch_groups.hpp b/include/strata/program/batch_groups.hppindex 168019b6b..345ace009 100644--- a/include/strata/program/batch_groups.hpp+++ b/include/strata/program/batch_groups.hpp@@ -57,7 +57,7 @@ struct Input {bool batch_mtp = false; ///< --batch-mtp is on (it passed the gates that do not concern the groups)bool adapt_async = false; ///< --adapt-async 1 was asked forbool mtp_serial = false; ///< STRATA_BATCH_MTP_SERIAL=1: --batch-mtp with no --batch-groups runs one serial group (0.1.43)- bool nv_serial_faster = false; ///< PROPOSED (0.2.1, owner decides): NVIDIA build, STRATA_BATCH_GROUPS_NV_SERIAL not 0 - two cards run faster in serial windows (z440 2x A4000, aggregate tok/s: +16% at 2 slots, +12% at 4, +15% at 8)+ bool nv_serial_faster = false; ///< opt-in (see nv_serial_wanted): NVIDIA build, two cards, STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 - one serial group, the 0.2.1-0.2.6 defaultbool mtp_piped = false; ///< STRATA_PIPED_MTP=1: the pipelined groups draft too (each group carries its slots' rows)};@@ -70,6 +70,16 @@ struct Result {bool mtp_piped = false; ///< STRATA_PIPED_MTP: --batch-mtp's drafts run in the pipelined groups};+/// Whether two NVIDIA cards run one serial group when --batch-groups is not given. Not by default since 0.2.7: the pipelined groups+/// measured faster on 2x RTX A6000 (IQ3_S, every expert resident) and on 2x RTX A4000 (Q2_0 pack); the serial default of 0.2.1-0.2.6+/// had measured faster on 2x A4000 with IQ3_XXS. STRATA_BATCH_GROUPS_NV_SERIAL asks for it with any value but 0 (an explicit 0 or+/// non-zero value wins over the master switch); with the variable unset or empty, STRATA_DEFAULTS=026 (v0.2.6's behaviour) asks for+/// it. `env_value` is the variable's value (null when unset).+inline bool nv_serial_wanted(const char* env_value, bool defaults_026) {+ if (env_value != nullptr && env_value[0] != '\0') return env_value[0] != '0';+ return defaults_026;+}+/// The most groups, at most one per stage of the layer split (`later_stages` + 1 GPUs), that divide `batch`.inline int auto_groups(int later_stages, int batch) {int best = 1;@@ -107,7 +117,7 @@ inline Result resolve(const Input& in) {inline const char* serial_what(Serial s) {return s == Serial::BatchMtp ? "--batch-mtp" : s == Serial::AdaptAsync ? "--adapt-async 1"- : s == Serial::TwoCardsSerialFaster ? "serial windows (measured faster than pipelined groups on two NVIDIA cards; STRATA_BATCH_GROUPS_NV_SERIAL=0 keeps the pipeline)" : "";+ : s == Serial::TwoCardsSerialFaster ? "serial windows (STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 asked for them on two NVIDIA cards; --batch-groups auto pipelines)" : "";}} // namespace strata::program::batch_groupsdiff --git a/src/program/batch_groups_test.cpp b/src/program/batch_groups_test.cppindex 3c9788b4a..2d0657bf8 100644--- a/src/program/batch_groups_test.cpp+++ b/src/program/batch_groups_test.cpp@@ -107,6 +107,20 @@ int main() {check(r.groups == 2 && r.mtp_piped && !r.mtp_dropped,"STRATA_PIPED_MTP, --batch-groups 2 given: 2 groups of 2 slots, and the drafts stay");}+ { // 0.2.7: two NVIDIA cards pipeline by default; the serial default is opt-in (STRATA_BATCH_GROUPS_NV_SERIAL=1, STRATA_DEFAULTS=026)+ check(!bg::nv_serial_wanted(nullptr, false), "NV_SERIAL unset: the pipeline");+ check(!bg::nv_serial_wanted("", false), "NV_SERIAL empty: the pipeline");+ check(!bg::nv_serial_wanted("0", false), "NV_SERIAL=0: the pipeline");+ check(bg::nv_serial_wanted("1", false), "NV_SERIAL=1: serial");+ check(bg::nv_serial_wanted("yes", false), "NV_SERIAL=yes: serial");+ check(bg::nv_serial_wanted(nullptr, true), "STRATA_DEFAULTS=026: serial (v0.2.6's behaviour)");+ check(!bg::nv_serial_wanted("0", true), "an explicit NV_SERIAL=0 wins over STRATA_DEFAULTS=026");+ check(bg::nv_serial_wanted("", true), "NV_SERIAL empty and STRATA_DEFAULTS=026: serial");+ bg::Input i = in(4, 1);+ i.nv_serial_faster = bg::nv_serial_wanted(nullptr, false);+ auto r = bg::resolve(i);+ check(r.groups == 2 && r.serial == bg::Serial::None, "2 cards, 4 slots, nothing given: 2 pipelined groups, whatever the residency");+ }{ // STRATA_PIPED_MTP unset: nothing changes, whatever --batch-mtp and the groups saybg::Input i = in(4, 1);i.batch_mtp = true;diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 65470b244..b69f71f89 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -5957,7 +5957,10 @@ int main(int argc, char** argv) {bin.adapt_async = o.adapt_async != 0;{ const char* ms = std::getenv("STRATA_BATCH_MTP_SERIAL"); bin.mtp_serial = ms != nullptr && ms[0] != '\0' && ms[0] != '0'; }#if !defined(STRATA_USE_HIP)- { const char* nv = std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"); bin.nv_serial_faster = nv == nullptr || nv[0] != '0'; } // PROPOSED 0.2.1 default (NVIDIA, 2 cards)+ // NVIDIA, 2 cards: the pipelined groups by default (0.2.7); one serial group, the 0.2.1-0.2.6 default, with+ // STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 (batch_groups.hpp nv_serial_wanted)+ bin.nv_serial_faster = strata::program::batch_groups::nv_serial_wanted(std::getenv("STRATA_BATCH_GROUPS_NV_SERIAL"),+ strata::qwen_defaults_026());#endif{ const char* mp = std::getenv("STRATA_PIPED_MTP"); bin.mtp_piped = mp != nullptr && mp[0] != '\0' && mp[0] != '0'; }return strata::program::batch_groups::resolve(bin);