Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

docs/MULTI_GPU.md: say when a layer split pays on a slow CPU (#3930)

Norwinab48216a7b27
diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.md
index 36b35a81b..37bd82aa9 100644
--- a/docs/MULTI_GPU.md
+++ b/docs/MULTI_GPU.md
@@ -220,6 +220,14 @@ the Qwen layer split described above. On four Radeon AI PRO R9700 cards it reach
An explicit `--expert-cache` on the first card is capped to leave room for them.
- More cards pay off when the model's routed experts do not fit the faster ones. An extra slow card costs its own round per
window and can make a good pair slower.
+- **The split pays most when the CPU pool is the bottleneck.** On two identical RTX 2080 Ti 22 GB over a dual-socket Xeon E5-2696 v4 (AVX2 only,
+ 88 logical CPUs), a plain split moved decode from 35.2 to 51.0 tok/s (+45%) and an 11.6K-token prompt from 672 to 908 tok/s (+35%) at 180 W,
+ engine 0.1.49, six 300-token coding prompts. `STRATA_DECODE_TIMING=1` shows why: the window fell from 70.06 to 37.96 ms, almost entirely
+ because the CPU pool fell from 9.52 to 0.79 ms per window. The 5080 + 3090 measurement below, where decode with the second card is on par
+ with the 5080 alone, was taken over a fast CPU (Ryzen 9 9950X3D). When the CPU pool is what the window waits on, moving half the layers to a
+ second identical card removes that work. This is one pair of cards and one CPU, so there is no threshold, and the window breakdown from the
+ engine log is not the same statistic as the decode numbers beside it. Setup splits across two cards by itself; with two identical older cards
+ on a slow workstation CPU, compare one card against the split and tell us in [#3930](https://github.com/Niko1221/Strata/pull/3930).
- For several conversations at once, `--batch N --batch-groups G --trim-stage-weights` pipelines them through the cards
([BATCHING.md](BATCHING.md)).