Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Docker: launch gets SIGINT back so docker stop ends a download in 1 s; INSTALL.md points to DOCKER.md

Niko12217e490ef63cd0
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 3602abd51..de84c8bb1 100644
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -153,7 +153,7 @@ on_term() {
say "Stop requested: shutting down."
if [ -n "$launch_pid" ] && kill -0 "$launch_pid" 2>/dev/null; then
kill -INT "$launch_pid" 2>/dev/null # the launch is still downloading or loading: it stops what it started
- i=0; while kill -0 "$launch_pid" 2>/dev/null && [ "$i" -lt 25 ]; do sleep 1; i=$((i+1)); done
+ i=0; while kill -0 "$launch_pid" 2>/dev/null && [ "$i" -lt 30 ]; do sleep 1; i=$((i+1)); done
kill -TERM "$launch_pid" 2>/dev/null || true
fi
"$STRATA" launch --stop --preset "$PRESET" || true
@@ -161,12 +161,17 @@ on_term() {
}
trap on_term TERM INT
+# A command started with & by a non-interactive sh ignores SIGINT, so `kill -INT` would do nothing. The launch must see it (it is how it
+# stops a download or a load cleanly), so it is started through a one-line Python that puts SIGINT back before it runs the Terminal.
+PYBIN="$APP/terminal/python/bin/python3"; [ -x "$PYBIN" ] || PYBIN=$(command -v python3 || echo python3)
+RESET_INT='import os, signal, sys; signal.signal(signal.SIGINT, signal.SIG_DFL); os.execv(sys.argv[1], sys.argv[1:])'
+
phase preparing
say "Starting Strata: ${MODEL:-recommended model}${model_path:+ from $model_path} (preset '$PRESET')."
if [ -n "$key" ]; then
- STRATA_LAUNCH_KEY="$key" "$STRATA" "$@" &
+ STRATA_LAUNCH_KEY="$key" "$PYBIN" -c "$RESET_INT" "$STRATA" "$@" &
else
- "$STRATA" "$@" &
+ "$PYBIN" -c "$RESET_INT" "$STRATA" "$@" &
fi
launch_pid=$!
wait "$launch_pid"
diff --git a/docs/DOCKER.md b/docs/DOCKER.md
index dd9bcd8c8..6a519b93a 100644
--- a/docs/DOCKER.md
+++ b/docs/DOCKER.md
@@ -190,7 +190,7 @@ docker stop strata && docker rm strata
```
The models, the prepared pack, your preset and tuning stay in the volume `strata-data`; nothing is downloaded again. A new
-engine release can invalidate a tuning result: the first start says so and the preset is tuned again with `RETUNE=1`.
+engine can make an old tuning result less useful: start once with `-e TUNE=quick -e RETUNE=1` to tune again.
`strata update` inside the container is not the way to update: the engine is part of the image.
## Troubleshooting
@@ -217,7 +217,7 @@ use the `cuda12` image or update the driver.
container's user) must be allowed to use them. Add `--device /dev/kfd --device /dev/dri` and the groups
`--group-add video --group-add $(getent group render | cut -d: -f3)` (the render group's number differs from computer to
computer, so use the number, not the name). `ls -l /dev/kfd /dev/dri/renderD*` shows who owns them. On a computer with
-SELinux (Fedora, RHEL), add `--security-opt label=disable` for the device files to open.
+SELinux (Fedora, RHEL) that still refuses the device files, add `--security-opt label=disable` to the `docker run`.
**Intel: no device found.** Pass `--device /dev/dri` and `--group-add <render group number>` as above. An Arc card needs the
`i915` or `xe` driver on the host (`ls /dev/dri/renderD*` must show a device; `lspci -k | grep -A3 -i arc` shows the driver in use).
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
index 099db1179..eb8a1ee55 100644
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@@ -99,53 +99,10 @@ split pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_
## Docker (Linux)
-The same idea, in a container (NVIDIA cards).
-
-1. Host: Docker with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
- and a driver **580 or newer** (CUDA 13.0).
-2. Build (this compiles the engine into the image, so the container never compiles):
- `docker build -t strata .`
- `docker build -t strata --build-arg CUDA_ARCHITECTURES=89 .` builds for one card only (faster).
- The default covers RTX 30 (86), RTX 40 (89), RTX 50 (120) and A-series (80); a card outside that
- set needs a rebuild with its own arch. Add `--build-arg BUILD_VISION=0` to skip the image encoder.
-3. Run (the first start downloads the ~70 GB model, then starts; later starts go straight to serving):
- `docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 -v strata-data:/data strata`
-
- The setup choices are env vars: `-e MODEL=IQ2_XS -e FAMILY=qwen -e CONTEXT=32768 -e VISION=no`
- (or `MODEL=Q2_0|IQ3_XXS|IQ3_S`, `FAMILY=swift|coder`; the defaults above are the recommended ones).
- `-e VISION=cpu` keeps the image encoder on the CPU. `-e KV=int8|q4_0|k8v4` picks the KV cache
- precision; `k8v4` is INT8 K with 4-bit V and keeps its KV in VRAM from 64K up.
- Only the model files, the prepared pack, the MTP layer and the install config live in the
- `strata-data` volume; the engine is part of the image. Switching between models already on the
- volume needs no setup pass: `-e MODEL=Q2_0 -e FAMILY=coder` picks that model's config. Add
- `-e REINSTALL=1` only to change settings for a model already set up (context, vision, KV, host,
- api_key, LOW_RAM), since those are recorded in its config.
- `MODEL` (with `FAMILY`) selects `/data/config/strata-<model>.json` (the model name in lowercase), so
- `-e MODEL=IQ3_S.1X4-BATCH9` starts `/data/config/strata-iq3_s.1x4-batch9.json`: that is the way to keep
- several configs of one model on one volume. `-e CONFIG=/data/config/my.json` overrides it with any
- config file. A link `/opt/strata/strata-<model>.json` that already points into `/data/config/` (made by
- a pod command before the entrypoint runs) is kept. The entrypoint prints one `Config:` line naming the
- file the server starts with.
- Strata loads 32-62 GB into RAM. `--gpus all` on a host with two usable cards takes both: the
- layer split is setup's recommended default ([MULTI_GPU.md](MULTI_GPU.md)), and a volume
- set up for one card switches to the pair on its first start there. Pin one card with `-e GPU=0`,
- or name them with `-e GPUS=0,2` and where the later card's layers start with `-e LAYER_SPLIT=18`.
- A memory limit needs `-e LOW_RAM=on`, which maps the model's experts from the pack instead of
- keeping them in RAM: setup.py measures the host's RAM, not the container's limit, so it cannot
- see a cap. LOW_RAM runs on one card unless `GPUS` names several (then the experts the cards do not hold are
- read through the OS file cache, which can fill the RAM during long prompts).
- The server listens on `0.0.0.0:8080` by default; set `-e API_KEY=<secret>` before exposing the port
- to a network. The image has a `HEALTHCHECK` on `/health`, so `docker ps` shows the container
- healthy once the model is loaded, and `GET /v1/status` says what it is running.
- Model files you already have (setup's `--gguf-dir`): mount their folder and name it, e.g.
- `-v /models/UD-Q4_K_XL:/gguf -e GGUF_DIR=/gguf -e FAMILY=unsloth -e MODEL=UD-Q4_K_XL`.
- Mount it writable for the first start: setup checks the Unsloth file's SHA-256 once and keeps the
- result as `<shard>.done` next to each shard; later starts skip setup, so `:ro` is fine then.
- `-e RESIDENT_BUDGET_GIB=N` and `-e KV_STREAMING=on|off|auto` pass setup's `--resident-budget-gib`
- and `--kv-streaming` (with `REINSTALL=1` for a model already set up).
- Stopping: the engine releases its page-locked RAM (tens of GB) on SIGTERM, which takes longer than
- Docker's default 10 s; give it time, or `docker stop` kills it (exit 137):
- `docker run --stop-timeout 60 ...`, `docker stop -t 60 <name>`, or `stop_grace_period: 60s` in Compose.
+Ready-made images for NVIDIA (CUDA 13 and CUDA 12), AMD (RX 9000 / R9700 and RX 7000) and Intel Arc cards, with the Terminal
+inside: `docker run` one command, the model downloads on the first start into a volume, nothing is compiled on your computer.
+Which image for which card, the commands, Docker Compose, the API key, updating and troubleshooting: [DOCKER.md](DOCKER.md).
+Building from source: `docker build -t strata:src .` (CUDA 13; [DOCKER.md](DOCKER.md#build-it-yourself)).
## Older CPUs (experimental)