Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docs: Main, Helper, Off in MULTI_GPU.md (the main card is the last stage), Docker GPU_ROLES, wording fixes
Niko12210fce78c1c377
diff --git a/docker/entrypoint.sh b/docker/entrypoint.shindex 7299034cf..6b6f87cfa 100755--- a/docker/entrypoint.sh+++ b/docker/entrypoint.sh@@ -15,6 +15,7 @@# VISION off | on | gpu | cpu (pictures; the picture reader file is downloaded with the model)# KV int8 | q4_0 | k8v4 | fp16 (conversation memory format)# GPU / GPUS card numbers, e.g. 0 or 0,1 (empty or "all": the engine's choice)+# GPU_ROLES a role for each card, e.g. 1=main,0=helper (main, helper or off; cards you do not name are off; not together with GPU / GPUS)# TUNE quick | thorough | skip (done once; RETUNE=1 tunes again)# PORT port inside the container (default 8080)# API_KEY / API_KEY_FILE the key clients must send. The server listens on every address inside the container, and Strata does@@ -201,6 +202,13 @@ case "$gpus" in ""|all|ALL) ;; *)fidone ;;esac+# GPU_ROLES=1=main,0=helper: a role for each card (main, helper or off; cards you do not name are off). `launch --gpu-roles` writes them into the+# run settings at every start, so nothing saved has to be dropped.+roles=$(printf '%s' "${GPU_ROLES:-}" | tr -d ' ')+if [ -n "$roles" ]; then+ case "$gpus" in ""|all|ALL) ;; *) die "Set GPU_ROLES or GPU / GPUS, not both." 2 ;; esac+ set -- "$@" --gpu-roles "$roles"; have=1+fi[ -n "${STRATA_SET:-}" ] && { for kv in $STRATA_SET; do set -- "$@" --set "$kv"; done; have=1; }[ -n "$key" ] && { set -- "$@" --api-key-env STRATA_LAUNCH_KEY; }diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex 97313ab7c..9330f0cd7 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -46,7 +46,7 @@ the kernel's amdgpu driver (no ROCm install needed):no hipBLASLt kernels) the same role falls to a rocBLAS table, `tools/hip/<arch>-rocblas-<version>.txt`, which theengine picks for its own rocBLAS build when setup points it at the directory ([RDNA2](#rdna2-gfx1030)).- **Several cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more:- `./setup.sh --backend hip --gpus 1,0` splits the model's layers across them, the first one the main card (numbers+ `./setup.sh --backend hip --gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md) (numbersas setup lists them; `--gpus all` = every supported card, the most VRAM first). Every chosen card must be one of thearchitectures above; the engine is compiled for each of them (cards of two families, e.g. gfx1100 + gfx1201, needa system ROCm 7: AMD's wheels hold one family). A split pays only when no single card holds the model's expertsdiff --git a/docs/DOCKER.md b/docs/DOCKER.mdindex d6ac46ef0..f4b5ee62c 100644--- a/docs/DOCKER.md+++ b/docs/DOCKER.md@@ -121,6 +121,7 @@ screens; leave out what you do not care about.| `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` || `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` || `GPU` / `GPUS` | card numbers as the monitor lists them, `0` or `0,1`; empty or `all` = the engine's choice. Changing it later makes the model's run settings again (the files stay) | all that fit |+| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) || `TUNE` | `quick` (up to about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` || `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you || `PORT` | the port inside the container | `8080` |diff --git a/docs/INSTALL.md b/docs/INSTALL.mdindex afb1e866e..100174f39 100644--- a/docs/INSTALL.md+++ b/docs/INSTALL.md@@ -221,7 +221,7 @@ pipeline: on an RTX 5080 + RTX 3090 prompts were read 18-20% faster than on theEvery card must be an RTX 20 series or newer with 8 GB or more. See [MULTI_GPU.md](MULTI_GPU.md).**Several AMD cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more:-`--gpus 1,0` splits the model's layers across them, the first one the main card (numbers as setup lists them). A+`--gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md; numbers as setup lists them). Asplit pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201)).## Docker (Linux)diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.mdindex 6d6659e03..b043bff11 100644--- a/docs/MULTI_GPU.md+++ b/docs/MULTI_GPU.md@@ -13,7 +13,7 @@ is every card's share and skips those probes; there is no per-card setting yet.**Two models on two cards (the app).** A layer split puts one model on several cards. The other way to use two cards is twomodels, one per card, each answering on its own. In the app, turn on **Run multiple models** on the Models page: the card-list then says which card a loaded model uses ("Already used by ..."), preselects a free card for the next model, and+list then says which card a loaded model uses ("In use by ..."), starts that card as Off for the next model, andexplains a failed start on a card that is taken. Measured on a PC with Radeon AI PRO R9700 cards (two of four, Linux): Q2_0 onone card and IQ3_XXS on another both loaded (33 s and 59 s with the files cached, 114 s the first time), both answered chats(about 60 tokens/s each), the Monitor listed both, and unloading one left the other answering. This was run with the AMD build only;@@ -41,19 +41,61 @@ Strata can use it:Which GPUs? [1]:```-**Which cards a model uses (one rule, in setup, `strata launch` and the app).** Strata first tries the model on your-**largest** card plus the computer's memory, the normal hybrid mode, and recommends that when it fits (with some RAM to-spare). Only when it does not fit one card, or would be too tight, the layer split over the two largest cards is-recommended instead. Every option is listed with plain numbers (what the model needs and what the cards and RAM give;-how much would be read from the SSD when it does not fit) and the choice is yours. The cards that count: not an-integrated chip (it shares the system memory and adds nothing beside a real card), at least 4 GB of graphics memory, and-8 GB or more to share a model. The arithmetic is the app's `fitOn` (strata_control/gpu_fit.py keeps its constants equal);+## Main, Helper, Off++Every card has one of three roles:++| Role | What it does |+|---|---|+| **Main** | Runs the **last** stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. |+| **Helper** | Holds a share of the layers and of the expert cache, so more of the model fits on graphics cards instead of being read from the drive. Prompts are read about 20% faster. |+| **Off** | Strata does not touch the card (a game, the screen, a card that is too small). |++The Main card is the **last** card in the engine's order, not the first. The first card runs the first layers and, by default,+the picture reader. (Older versions of this page said "the first is the main one"; that has not been true since 0.1.41.1.)++**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`): the faster card+is Main (speed is multiprocessors x max clock, the engine's own measure; cards within 5% are the same card, and the one with+more memory, then the lower number, wins; a card with clearly more memory beats a faster one that is at least half as fast).+Every other card starts as Helper, except a card with **8 GB or less** (Off), built-in graphics beside a card with memory of+its own (Off) and a card the engine cannot run (Off). Every role is one click (the app), one key (`strata launch`: M, H, O;+R goes back to the recommendation) or one line (below).++Only a few things are warned about, in one short line each: built-in graphics as Main, a card the engine cannot run, too+little memory for the chosen setup (part of the model would be read from the drive). **Every card Off** is allowed. The CUDA,+HIP and SYCL engines cannot start without a card, so the server then says so in one line and stops; an engine that can run on+the processor (the Mac engine) says "Performance is degraded on CPU only." and goes on.++**Setting roles**++```+strata launch --gpu-roles 1=main,0=helper cards you do not name are Off+./setup.sh --gpu-roles 1=main,0=helper (START-HERE.bat --gpu-roles ... on Windows); saved+strata launch the Cards step: M Main, H Helper, O Off, R recommended, D details+```++or in the run settings (`strata-*.json`), which is what the app writes:++```json+"gpu_roles": {"1": "main", "0": "helper", "2": "off"},+"gpu": [0, 1],+"gpu_order": "as_given"+```++`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, with the Off+cards left out. The server reads `gpu_roles`; when `gpu` was edited by hand and no longer matches, `gpu` wins and the log says+so (a settings file that names its cards keeps them). Nothing is rewritten on upgrade: a file with only `"gpu": [1, 0]` shows+as Main = the last card, Helper = the first, the rest Off. The recommendation itself is not saved, so a card you add later is+recommended again; a card you add after choosing roles starts Off. Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles`.++**One rule for which cards count** (`strata_control/gpu_fit.py` keeps the memory arithmetic): not an integrated chip beside+a real card, at least 4 GB of graphics memory to be listed, 8 GB or less starts Off. The arithmetic is the app's `fitOn`;each extra card of a split costs an estimated 2 GB of its own buffers.-**Choosing yourself** (at setup or at any start):+**Choosing the cards without roles** (at setup or at any start):```---gpus 0,2 these cards together, as nvidia-smi numbers them; the first is the main one. Remembered.+--gpus 0,2 these cards together, as nvidia-smi numbers them; Strata puts the faster card last. Remembered.--gpus all every card that can share the model--gpu 0 one card (at a start: for that start only)--layer-split auto (default) or the first layer of each later card, e.g. 18 or 16,32diff --git a/setup.py b/setup.pyindex c57e93f69..3d486fd32 100644--- a/setup.py+++ b/setup.py@@ -1616,7 +1616,7 @@ def start_gpus(text):def choose_gpus(a, found) -> list:"""Which cards this install uses: --gpus / --gpu, or asked when two or more can share the model (the two best- together recommended), else the supported card with the most VRAM. Returns their numbers, the main one first."""+ together recommended), else the supported card with the most VRAM. Returns their numbers, in the engine order, the Main card last when roles are chosen."""if a.gpus:sel = parse_gpus(a.gpus, found)check_gpus(sel, found, yes=a.yes, named=str(a.gpus).strip().lower() != "all")