Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docker: launch gets SIGINT back so docker stop ends a download in 1 s; INSTALL.md points to DOCKER.md
Niko12217e490ef63cd0
diff --git a/docker/entrypoint.sh b/docker/entrypoint.shindex 3602abd51..de84c8bb1 100644--- a/docker/entrypoint.sh+++ b/docker/entrypoint.sh@@ -153,7 +153,7 @@ on_term() {say "Stop requested: shutting down."if [ -n "$launch_pid" ] && kill -0 "$launch_pid" 2>/dev/null; thenkill -INT "$launch_pid" 2>/dev/null # the launch is still downloading or loading: it stops what it started- i=0; while kill -0 "$launch_pid" 2>/dev/null && [ "$i" -lt 25 ]; do sleep 1; i=$((i+1)); done+ i=0; while kill -0 "$launch_pid" 2>/dev/null && [ "$i" -lt 30 ]; do sleep 1; i=$((i+1)); donekill -TERM "$launch_pid" 2>/dev/null || truefi"$STRATA" launch --stop --preset "$PRESET" || true@@ -161,12 +161,17 @@ on_term() {}trap on_term TERM INT+# A command started with & by a non-interactive sh ignores SIGINT, so `kill -INT` would do nothing. The launch must see it (it is how it+# stops a download or a load cleanly), so it is started through a one-line Python that puts SIGINT back before it runs the Terminal.+PYBIN="$APP/terminal/python/bin/python3"; [ -x "$PYBIN" ] || PYBIN=$(command -v python3 || echo python3)+RESET_INT='import os, signal, sys; signal.signal(signal.SIGINT, signal.SIG_DFL); os.execv(sys.argv[1], sys.argv[1:])'+phase preparingsay "Starting Strata: ${MODEL:-recommended model}${model_path:+ from $model_path} (preset '$PRESET')."if [ -n "$key" ]; then- STRATA_LAUNCH_KEY="$key" "$STRATA" "$@" &+ STRATA_LAUNCH_KEY="$key" "$PYBIN" -c "$RESET_INT" "$STRATA" "$@" &else- "$STRATA" "$@" &+ "$PYBIN" -c "$RESET_INT" "$STRATA" "$@" &filaunch_pid=$!wait "$launch_pid"diff --git a/docs/DOCKER.md b/docs/DOCKER.mdindex dd9bcd8c8..6a519b93a 100644--- a/docs/DOCKER.md+++ b/docs/DOCKER.md@@ -190,7 +190,7 @@ docker stop strata && docker rm strata```The models, the prepared pack, your preset and tuning stay in the volume `strata-data`; nothing is downloaded again. A new-engine release can invalidate a tuning result: the first start says so and the preset is tuned again with `RETUNE=1`.+engine can make an old tuning result less useful: start once with `-e TUNE=quick -e RETUNE=1` to tune again.`strata update` inside the container is not the way to update: the engine is part of the image.## Troubleshooting@@ -217,7 +217,7 @@ use the `cuda12` image or update the driver.container's user) must be allowed to use them. Add `--device /dev/kfd --device /dev/dri` and the groups`--group-add video --group-add $(getent group render | cut -d: -f3)` (the render group's number differs from computer tocomputer, so use the number, not the name). `ls -l /dev/kfd /dev/dri/renderD*` shows who owns them. On a computer with-SELinux (Fedora, RHEL), add `--security-opt label=disable` for the device files to open.+SELinux (Fedora, RHEL) that still refuses the device files, add `--security-opt label=disable` to the `docker run`.**Intel: no device found.** Pass `--device /dev/dri` and `--group-add <render group number>` as above. An Arc card needs the`i915` or `xe` driver on the host (`ls /dev/dri/renderD*` must show a device; `lspci -k | grep -A3 -i arc` shows the driver in use).diff --git a/docs/INSTALL.md b/docs/INSTALL.mdindex 099db1179..eb8a1ee55 100644--- a/docs/INSTALL.md+++ b/docs/INSTALL.md@@ -99,53 +99,10 @@ split pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_## Docker (Linux)-The same idea, in a container (NVIDIA cards).--1. Host: Docker with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)- and a driver **580 or newer** (CUDA 13.0).-2. Build (this compiles the engine into the image, so the container never compiles):- `docker build -t strata .`- `docker build -t strata --build-arg CUDA_ARCHITECTURES=89 .` builds for one card only (faster).- The default covers RTX 30 (86), RTX 40 (89), RTX 50 (120) and A-series (80); a card outside that- set needs a rebuild with its own arch. Add `--build-arg BUILD_VISION=0` to skip the image encoder.-3. Run (the first start downloads the ~70 GB model, then starts; later starts go straight to serving):- `docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 -v strata-data:/data strata`-- The setup choices are env vars: `-e MODEL=IQ2_XS -e FAMILY=qwen -e CONTEXT=32768 -e VISION=no`- (or `MODEL=Q2_0|IQ3_XXS|IQ3_S`, `FAMILY=swift|coder`; the defaults above are the recommended ones).- `-e VISION=cpu` keeps the image encoder on the CPU. `-e KV=int8|q4_0|k8v4` picks the KV cache- precision; `k8v4` is INT8 K with 4-bit V and keeps its KV in VRAM from 64K up.- Only the model files, the prepared pack, the MTP layer and the install config live in the- `strata-data` volume; the engine is part of the image. Switching between models already on the- volume needs no setup pass: `-e MODEL=Q2_0 -e FAMILY=coder` picks that model's config. Add- `-e REINSTALL=1` only to change settings for a model already set up (context, vision, KV, host,- api_key, LOW_RAM), since those are recorded in its config.- `MODEL` (with `FAMILY`) selects `/data/config/strata-<model>.json` (the model name in lowercase), so- `-e MODEL=IQ3_S.1X4-BATCH9` starts `/data/config/strata-iq3_s.1x4-batch9.json`: that is the way to keep- several configs of one model on one volume. `-e CONFIG=/data/config/my.json` overrides it with any- config file. A link `/opt/strata/strata-<model>.json` that already points into `/data/config/` (made by- a pod command before the entrypoint runs) is kept. The entrypoint prints one `Config:` line naming the- file the server starts with.- Strata loads 32-62 GB into RAM. `--gpus all` on a host with two usable cards takes both: the- layer split is setup's recommended default ([MULTI_GPU.md](MULTI_GPU.md)), and a volume- set up for one card switches to the pair on its first start there. Pin one card with `-e GPU=0`,- or name them with `-e GPUS=0,2` and where the later card's layers start with `-e LAYER_SPLIT=18`.- A memory limit needs `-e LOW_RAM=on`, which maps the model's experts from the pack instead of- keeping them in RAM: setup.py measures the host's RAM, not the container's limit, so it cannot- see a cap. LOW_RAM runs on one card unless `GPUS` names several (then the experts the cards do not hold are- read through the OS file cache, which can fill the RAM during long prompts).- The server listens on `0.0.0.0:8080` by default; set `-e API_KEY=<secret>` before exposing the port- to a network. The image has a `HEALTHCHECK` on `/health`, so `docker ps` shows the container- healthy once the model is loaded, and `GET /v1/status` says what it is running.- Model files you already have (setup's `--gguf-dir`): mount their folder and name it, e.g.- `-v /models/UD-Q4_K_XL:/gguf -e GGUF_DIR=/gguf -e FAMILY=unsloth -e MODEL=UD-Q4_K_XL`.- Mount it writable for the first start: setup checks the Unsloth file's SHA-256 once and keeps the- result as `<shard>.done` next to each shard; later starts skip setup, so `:ro` is fine then.- `-e RESIDENT_BUDGET_GIB=N` and `-e KV_STREAMING=on|off|auto` pass setup's `--resident-budget-gib`- and `--kv-streaming` (with `REINSTALL=1` for a model already set up).- Stopping: the engine releases its page-locked RAM (tens of GB) on SIGTERM, which takes longer than- Docker's default 10 s; give it time, or `docker stop` kills it (exit 137):- `docker run --stop-timeout 60 ...`, `docker stop -t 60 <name>`, or `stop_grace_period: 60s` in Compose.+Ready-made images for NVIDIA (CUDA 13 and CUDA 12), AMD (RX 9000 / R9700 and RX 7000) and Intel Arc cards, with the Terminal+inside: `docker run` one command, the model downloads on the first start into a volume, nothing is compiled on your computer.+Which image for which card, the commands, Docker Compose, the API key, updating and troubleshooting: [DOCKER.md](DOCKER.md).+Building from source: `docker build -t strata:src .` (CUDA 13; [DOCKER.md](DOCKER.md#build-it-yourself)).## Older CPUs (experimental)