Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Docs: Main, Helper, Off in MULTI_GPU.md (the main card is the last stage), Docker GPU_ROLES, wording fixes

Niko12210fce78c1c377
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 7299034cf..6b6f87cfa 100755
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -15,6 +15,7 @@
# VISION off | on | gpu | cpu (pictures; the picture reader file is downloaded with the model)
# KV int8 | q4_0 | k8v4 | fp16 (conversation memory format)
# GPU / GPUS card numbers, e.g. 0 or 0,1 (empty or "all": the engine's choice)
+# GPU_ROLES a role for each card, e.g. 1=main,0=helper (main, helper or off; cards you do not name are off; not together with GPU / GPUS)
# TUNE quick | thorough | skip (done once; RETUNE=1 tunes again)
# PORT port inside the container (default 8080)
# API_KEY / API_KEY_FILE the key clients must send. The server listens on every address inside the container, and Strata does
@@ -201,6 +202,13 @@ case "$gpus" in ""|all|ALL) ;; *)
fi
done ;;
esac
+# GPU_ROLES=1=main,0=helper: a role for each card (main, helper or off; cards you do not name are off). `launch --gpu-roles` writes them into the
+# run settings at every start, so nothing saved has to be dropped.
+roles=$(printf '%s' "${GPU_ROLES:-}" | tr -d ' ')
+if [ -n "$roles" ]; then
+ case "$gpus" in ""|all|ALL) ;; *) die "Set GPU_ROLES or GPU / GPUS, not both." 2 ;; esac
+ set -- "$@" --gpu-roles "$roles"; have=1
+fi
[ -n "${STRATA_SET:-}" ] && { for kv in $STRATA_SET; do set -- "$@" --set "$kv"; done; have=1; }
[ -n "$key" ] && { set -- "$@" --api-key-env STRATA_LAUNCH_KEY; }
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.md
index 97313ab7c..9330f0cd7 100644
--- a/docs/AMD_HIP.md
+++ b/docs/AMD_HIP.md
@@ -46,7 +46,7 @@ the kernel's amdgpu driver (no ROCm install needed):
no hipBLASLt kernels) the same role falls to a rocBLAS table, `tools/hip/<arch>-rocblas-<version>.txt`, which the
engine picks for its own rocBLAS build when setup points it at the directory ([RDNA2](#rdna2-gfx1030)).
- **Several cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more:
- `./setup.sh --backend hip --gpus 1,0` splits the model's layers across them, the first one the main card (numbers
+ `./setup.sh --backend hip --gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md) (numbers
as setup lists them; `--gpus all` = every supported card, the most VRAM first). Every chosen card must be one of the
architectures above; the engine is compiled for each of them (cards of two families, e.g. gfx1100 + gfx1201, need
a system ROCm 7: AMD's wheels hold one family). A split pays only when no single card holds the model's experts
diff --git a/docs/DOCKER.md b/docs/DOCKER.md
index d6ac46ef0..f4b5ee62c 100644
--- a/docs/DOCKER.md
+++ b/docs/DOCKER.md
@@ -121,6 +121,7 @@ screens; leave out what you do not care about.
| `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` |
| `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` |
| `GPU` / `GPUS` | card numbers as the monitor lists them, `0` or `0,1`; empty or `all` = the engine's choice. Changing it later makes the model's run settings again (the files stay) | all that fit |
+| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) |
| `TUNE` | `quick` (up to about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |
| `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you |
| `PORT` | the port inside the container | `8080` |
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
index afb1e866e..100174f39 100644
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@@ -221,7 +221,7 @@ pipeline: on an RTX 5080 + RTX 3090 prompts were read 18-20% faster than on the
Every card must be an RTX 20 series or newer with 8 GB or more. See [MULTI_GPU.md](MULTI_GPU.md).
**Several AMD cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more:
-`--gpus 1,0` splits the model's layers across them, the first one the main card (numbers as setup lists them). A
+`--gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md; numbers as setup lists them). A
split pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201)).
## Docker (Linux)
diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.md
index 6d6659e03..b043bff11 100644
--- a/docs/MULTI_GPU.md
+++ b/docs/MULTI_GPU.md
@@ -13,7 +13,7 @@ is every card's share and skips those probes; there is no per-card setting yet.
**Two models on two cards (the app).** A layer split puts one model on several cards. The other way to use two cards is two
models, one per card, each answering on its own. In the app, turn on **Run multiple models** on the Models page: the card
-list then says which card a loaded model uses ("Already used by ..."), preselects a free card for the next model, and
+list then says which card a loaded model uses ("In use by ..."), starts that card as Off for the next model, and
explains a failed start on a card that is taken. Measured on a PC with Radeon AI PRO R9700 cards (two of four, Linux): Q2_0 on
one card and IQ3_XXS on another both loaded (33 s and 59 s with the files cached, 114 s the first time), both answered chats
(about 60 tokens/s each), the Monitor listed both, and unloading one left the other answering. This was run with the AMD build only;
@@ -41,19 +41,61 @@ Strata can use it:
Which GPUs? [1]:
```
-**Which cards a model uses (one rule, in setup, `strata launch` and the app).** Strata first tries the model on your
-**largest** card plus the computer's memory, the normal hybrid mode, and recommends that when it fits (with some RAM to
-spare). Only when it does not fit one card, or would be too tight, the layer split over the two largest cards is
-recommended instead. Every option is listed with plain numbers (what the model needs and what the cards and RAM give;
-how much would be read from the SSD when it does not fit) and the choice is yours. The cards that count: not an
-integrated chip (it shares the system memory and adds nothing beside a real card), at least 4 GB of graphics memory, and
-8 GB or more to share a model. The arithmetic is the app's `fitOn` (strata_control/gpu_fit.py keeps its constants equal);
+## Main, Helper, Off
+
+Every card has one of three roles:
+
+| Role | What it does |
+|---|---|
+| **Main** | Runs the **last** stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. |
+| **Helper** | Holds a share of the layers and of the expert cache, so more of the model fits on graphics cards instead of being read from the drive. Prompts are read about 20% faster. |
+| **Off** | Strata does not touch the card (a game, the screen, a card that is too small). |
+
+The Main card is the **last** card in the engine's order, not the first. The first card runs the first layers and, by default,
+the picture reader. (Older versions of this page said "the first is the main one"; that has not been true since 0.1.41.1.)
+
+**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`): the faster card
+is Main (speed is multiprocessors x max clock, the engine's own measure; cards within 5% are the same card, and the one with
+more memory, then the lower number, wins; a card with clearly more memory beats a faster one that is at least half as fast).
+Every other card starts as Helper, except a card with **8 GB or less** (Off), built-in graphics beside a card with memory of
+its own (Off) and a card the engine cannot run (Off). Every role is one click (the app), one key (`strata launch`: M, H, O;
+R goes back to the recommendation) or one line (below).
+
+Only a few things are warned about, in one short line each: built-in graphics as Main, a card the engine cannot run, too
+little memory for the chosen setup (part of the model would be read from the drive). **Every card Off** is allowed. The CUDA,
+HIP and SYCL engines cannot start without a card, so the server then says so in one line and stops; an engine that can run on
+the processor (the Mac engine) says "Performance is degraded on CPU only." and goes on.
+
+**Setting roles**
+
+```
+strata launch --gpu-roles 1=main,0=helper cards you do not name are Off
+./setup.sh --gpu-roles 1=main,0=helper (START-HERE.bat --gpu-roles ... on Windows); saved
+strata launch the Cards step: M Main, H Helper, O Off, R recommended, D details
+```
+
+or in the run settings (`strata-*.json`), which is what the app writes:
+
+```json
+"gpu_roles": {"1": "main", "0": "helper", "2": "off"},
+"gpu": [0, 1],
+"gpu_order": "as_given"
+```
+
+`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, with the Off
+cards left out. The server reads `gpu_roles`; when `gpu` was edited by hand and no longer matches, `gpu` wins and the log says
+so (a settings file that names its cards keeps them). Nothing is rewritten on upgrade: a file with only `"gpu": [1, 0]` shows
+as Main = the last card, Helper = the first, the rest Off. The recommendation itself is not saved, so a card you add later is
+recommended again; a card you add after choosing roles starts Off. Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles`.
+
+**One rule for which cards count** (`strata_control/gpu_fit.py` keeps the memory arithmetic): not an integrated chip beside
+a real card, at least 4 GB of graphics memory to be listed, 8 GB or less starts Off. The arithmetic is the app's `fitOn`;
each extra card of a split costs an estimated 2 GB of its own buffers.
-**Choosing yourself** (at setup or at any start):
+**Choosing the cards without roles** (at setup or at any start):
```
---gpus 0,2 these cards together, as nvidia-smi numbers them; the first is the main one. Remembered.
+--gpus 0,2 these cards together, as nvidia-smi numbers them; Strata puts the faster card last. Remembered.
--gpus all every card that can share the model
--gpu 0 one card (at a start: for that start only)
--layer-split auto (default) or the first layer of each later card, e.g. 18 or 16,32
diff --git a/setup.py b/setup.py
index c57e93f69..3d486fd32 100644
--- a/setup.py
+++ b/setup.py
@@ -1616,7 +1616,7 @@ def start_gpus(text):
def choose_gpus(a, found) -> list:
"""Which cards this install uses: --gpus / --gpu, or asked when two or more can share the model (the two best
- together recommended), else the supported card with the most VRAM. Returns their numbers, the main one first."""
+ together recommended), else the supported card with the most VRAM. Returns their numbers, in the engine order, the Main card last when roles are chosen."""
if a.gpus:
sel = parse_gpus(a.gpus, found)
check_gpus(sel, found, yes=a.yes, named=str(a.gpus).strip().lower() != "all")