Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs/MULTI_GPU.md: say when a layer split pays on a slow CPU (#3930)
Norwinab48216a7b27
diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.mdindex 36b35a81b..37bd82aa9 100644--- a/docs/MULTI_GPU.md+++ b/docs/MULTI_GPU.md@@ -220,6 +220,14 @@ the Qwen layer split described above. On four Radeon AI PRO R9700 cards it reachAn explicit `--expert-cache` on the first card is capped to leave room for them.- More cards pay off when the model's routed experts do not fit the faster ones. An extra slow card costs its own round perwindow and can make a good pair slower.+- **The split pays most when the CPU pool is the bottleneck.** On two identical RTX 2080 Ti 22 GB over a dual-socket Xeon E5-2696 v4 (AVX2 only,+ 88 logical CPUs), a plain split moved decode from 35.2 to 51.0 tok/s (+45%) and an 11.6K-token prompt from 672 to 908 tok/s (+35%) at 180 W,+ engine 0.1.49, six 300-token coding prompts. `STRATA_DECODE_TIMING=1` shows why: the window fell from 70.06 to 37.96 ms, almost entirely+ because the CPU pool fell from 9.52 to 0.79 ms per window. The 5080 + 3090 measurement below, where decode with the second card is on par+ with the 5080 alone, was taken over a fast CPU (Ryzen 9 9950X3D). When the CPU pool is what the window waits on, moving half the layers to a+ second identical card removes that work. This is one pair of cards and one CPU, so there is no threshold, and the window breakdown from the+ engine log is not the same statistic as the decode numbers beside it. Setup splits across two cards by itself; with two identical older cards+ on a slow workstation CPU, compare one card against the split and tell us in [#3930](https://github.com/Niko1221/Strata/pull/3930).- For several conversations at once, `--batch N --batch-groups G --trim-stage-weights` pipelines them through the cards([BATCHING.md](BATCHING.md)).