Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Docker: DOCKER.md, MODEL_PATH in compose, device check for AMD and Intel

Niko12213641dafe98e1
diff --git a/docker/compose/.env.example b/docker/compose/.env.example
index eaf02c9e6..b8931d834 100644
--- a/docker/compose/.env.example
+++ b/docker/compose/.env.example
@@ -1,6 +1,7 @@
# Copy to .env next to the compose file you use. Every line is optional.
# STRATA_IMAGE=ghcr.io/niko1221/strata:0.2.0-cuda13 # TODO(owner): where the images will be published
# MODEL=IQ3_XXS
+# MODEL_PATH=/models # with a volume line - /srv/gguf/IQ3_XXS:/models:ro
# CONTEXT=32k
# PARALLEL=2
# VISION=off
diff --git a/docker/compose/docker-compose.amd.yml b/docker/compose/docker-compose.amd.yml
index ff20d5942..764244cca 100644
--- a/docker/compose/docker-compose.amd.yml
+++ b/docker/compose/docker-compose.amd.yml
@@ -14,6 +14,7 @@ services:
- "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080"
environment:
MODEL: ${MODEL:-}
+ MODEL_PATH: ${MODEL_PATH:-} # GGUF files you already have, as seen inside the container (mount them below)
CONTEXT: ${CONTEXT:-32k}
PARALLEL: ${PARALLEL:-}
VISION: ${VISION:-off}
diff --git a/docker/compose/docker-compose.intel.yml b/docker/compose/docker-compose.intel.yml
index 25f97f796..861314c6d 100644
--- a/docker/compose/docker-compose.intel.yml
+++ b/docker/compose/docker-compose.intel.yml
@@ -14,6 +14,7 @@ services:
- "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080"
environment:
MODEL: ${MODEL:-}
+ MODEL_PATH: ${MODEL_PATH:-} # GGUF files you already have, as seen inside the container (mount them below)
CONTEXT: ${CONTEXT:-32k}
PARALLEL: ${PARALLEL:-}
VISION: ${VISION:-off}
diff --git a/docker/compose/docker-compose.nvidia.yml b/docker/compose/docker-compose.nvidia.yml
index 198fc336b..f55fb2e0a 100644
--- a/docker/compose/docker-compose.nvidia.yml
+++ b/docker/compose/docker-compose.nvidia.yml
@@ -15,6 +15,7 @@ services:
- "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080" # 127.0.0.1 = this computer only. 0.0.0.0 = your network: keep the API key then.
environment:
MODEL: ${MODEL:-} # empty: Strata picks the one it recommends for your card, and says so in the log
+ MODEL_PATH: ${MODEL_PATH:-} # GGUF files you already have, as seen inside the container (mount them below)
CONTEXT: ${CONTEXT:-32k}
PARALLEL: ${PARALLEL:-} # empty: fit to the card
VISION: ${VISION:-off} # on: the model reads pictures (about 0.9 GB more to download)
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 2fb902cf7..3602abd51 100644
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -80,6 +80,13 @@ MSG
exit 3
fi
+# the card is found through sysfs even when no device file was passed in: say so now instead of failing when the engine starts
+backend=$(printf '%s' "$engine_json" | sed -n 's/.*"backend": *"\([a-z0-9]*\)".*/\1/p' | head -1)
+case "$backend" in
+ hip) [ -e /dev/kfd ] || { echo "[X] This is an AMD card, but /dev/kfd is not in the container. Add: --device /dev/kfd --device /dev/dri --group-add video --group-add <render group number> (docs/DOCKER.md)" >&2; exit 3; } ;;
+ sycl) ls /dev/dri/renderD* >/dev/null 2>&1 || { echo "[X] This is an Intel card, but /dev/dri is not in the container. Add: --device /dev/dri --group-add <render group number> (docs/DOCKER.md)" >&2; exit 3; } ;;
+esac
+
# ---- the API key
key="${API_KEY:-}"
if [ -z "$key" ] && [ -n "${API_KEY_FILE:-}" ]; then
diff --git a/docs/DOCKER.md b/docs/DOCKER.md
new file mode 100644
index 000000000..dd9bcd8c8
--- /dev/null
+++ b/docs/DOCKER.md
@@ -0,0 +1,288 @@
+# Strata in Docker
+
+Run Strata as a container: one command, nothing to install on the computer except Docker and your graphics card's driver.
+The images come with Strata's engine already built for your card, so nothing is compiled on your side. The model is
+downloaded on the first start into a volume (a download that stops continues where it stopped), and every later start goes
+straight to serving.
+
+> **On this page:** [Which image](#which-image-for-which-card) · [One command per card](#one-command-per-card) ·
+> [Docker Compose](#docker-compose) · [Settings](#settings) · [The API key](#the-api-key) · [Connect your apps](#connect-your-coding-apps) ·
+> [Watch and stop](#watch-it-and-stop-it) · [Update](#updating) · [Troubleshooting](#troubleshooting) ·
+> [From the older image](#from-the-older-image) · [Build it yourself](#build-it-yourself)
+
+Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the Mac app or the Terminal
+([MAC.md](MAC.md), [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)). Docker Desktop on **Windows** passes NVIDIA cards through
+(WSL 2) but not Radeon or Arc cards; on Windows the app is the easier way for every card.
+
+## Which image for which card
+
+| Your graphics card | Image tag | Needs on the computer |
+| --- | --- | --- |
+| NVIDIA **RTX 20, 30, 40, 50** series (and the matching RTX A / Quadro cards) | `0.2.0-cuda13` | NVIDIA driver **580 or newer**, [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) |
+| NVIDIA **GTX 10 (Pascal), Tesla P100, V100 (Volta)**, or any NVIDIA card whose driver is **575 to 579** | `0.2.0-cuda12` | NVIDIA driver **575 or newer**, the same toolkit |
+| AMD **RX 9000** series and Radeon AI PRO R9700 (RDNA 4, gfx1200 / gfx1201) | `0.2.0-rocm-gfx120x` | the `amdgpu` kernel driver (Linux 6.8+) |
+| AMD **RX 7000** series (RDNA 3, gfx1100 / gfx1101 / gfx1102) | `0.2.0-rocm-gfx110x` | the `amdgpu` kernel driver |
+| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.0-sycl` | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) |
+
+Not sure which NVIDIA image? Take `cuda13` when `nvidia-smi` shows driver 580 or newer and the card is an RTX; otherwise `cuda12`.
+A wrong pick does no harm: the container starts, says in plain words that the engine does not fit this card, and stops.
+
+TODO(owner): where the images are published (registry and names). The plan, to decide at the 0.2.0 release:
+`ghcr.io/niko1221/strata:0.2.0-cuda13`, `:0.2.0-cuda12`, `:0.2.0-rocm-gfx120x`, `:0.2.0-rocm-gfx110x`, `:0.2.0-sycl`, plus a
+moving `:cuda13` / `:cuda12` / `:rocm-gfx120x` / `:rocm-gfx110x` / `:sycl` per card for "the newest". Until then,
+[build them yourself](#build-it-yourself); the examples below use the plan's names.
+
+## One command per card
+
+Each command runs Strata in the background, keeps your files in the volume `strata-data`, and listens only on this computer
+(`127.0.0.1:8080`). Replace `MODEL` and the key. The first start downloads the model (tens of GB): watch it with
+`docker logs -f strata`.
+
+**NVIDIA**
+
+```sh
+docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \
+ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
+ -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
+ ghcr.io/niko1221/strata:0.2.0-cuda13
+```
+
+**AMD**
+
+```sh
+docker run -d --name strata --device /dev/kfd --device /dev/dri \
+ --group-add video --group-add $(getent group render | cut -d: -f3) -p 127.0.0.1:8080:8080 \
+ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
+ -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
+ ghcr.io/niko1221/strata:0.2.0-rocm-gfx120x
+```
+
+**Intel Arc**
+
+```sh
+docker run -d --name strata --device /dev/dri --group-add $(getent group render | cut -d: -f3) \
+ -p 127.0.0.1:8080:8080 --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
+ -v strata-data:/data -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \
+ -e API_KEY=$(openssl rand -hex 24) ghcr.io/niko1221/strata:0.2.0-sycl
+```
+
+What the options are for: `--ulimit memlock=-1:-1` lets the engine pin the RAM it uses for the model, `--shm-size 2g` gives
+it shared memory, and `--stop-timeout 120` gives the model two minutes to unload when you stop the container (Docker's
+default 10 seconds is too short for a big model). Do not put a memory limit on the container: Strata plans from the whole
+computer's RAM and loads tens of GB.
+
+To print the key you made: `echo` it before you start (it is in your shell history otherwise), or ask the running container:
+`docker exec strata strata launch --show-key --preset docker`.
+
+### Intel Arc
+
+Strata's downloader prepares NVIDIA and AMD cards. For an Arc card the container **uses prepared model files**: the model's
+GGUF files in a folder you mount (`-v /srv/gguf/IQ3_S:/models:ro` and `-e MODEL_PATH=/models`), and the prepared pack and MTP
+layer in the data volume (`/data/Strata-data/packs/<model>` and `/data/Strata-data/mtp/rt`, made by the Intel installer on
+this or another computer, see [INTEL.md](INTEL.md)). Without them the container tells you which file is missing and stops.
+Nothing is downloaded or prepared for Arc inside the container. Arc B-series cards (xe driver) stream the experts from the
+GGUF files, so the GGUF folder should be on a fast SSD.
+
+## Docker Compose
+
+Ready-made files are in [docker/compose/](../docker/compose): `docker-compose.nvidia.yml`, `docker-compose.amd.yml`,
+`docker-compose.intel.yml` and `.env.example`. Copy the one for your card and its `.env.example`, edit, start:
+
+```sh
+cd docker/compose
+cp .env.example .env # set MODEL=..., API_KEY=...
+docker compose -f docker-compose.nvidia.yml up -d
+docker compose -f docker-compose.nvidia.yml logs -f
+```
+
+They set what the commands above set: the GPU (`deploy.resources.reservations.devices` for NVIDIA, `/dev/kfd` and
+`/dev/dri` plus the `video` and `render` groups for AMD, `/dev/dri` for Intel), `ulimits.memlock`, `shm_size`, a two-minute
+`stop_grace_period`, the `strata-data` volume, and the port bound to `127.0.0.1`. To let other computers reach it, set
+`STRATA_BIND=0.0.0.0` in `.env` and keep a real `API_KEY`.
+
+## Settings
+
+Everything is an environment variable (`-e NAME=value`, or in compose). They are the same questions `strata launch` asks on its
+screens; leave out what you do not care about.
+
+| Variable | What it does | Default |
+| --- | --- | --- |
+| `MODEL` | model size: `IQ2_XS`, `IQ3_XXS`, `IQ3_S`, `Q2_0`, `UD-Q4_K_XL`, ... ([MODELS.md](MODELS.md)) | Strata picks what suits the card, and says so in the log |
+| `FAMILY` | `qwen`, `swift`, `coder`, ... when a size exists in several families | the usual one |
+| `MODEL_PATH` | GGUF files you already have: a folder or one `.gguf`, **inside the container** (mount it with `-v`, `:ro` is fine) | download |
+| `CONTEXT` | how much text it remembers at once: `32k`, `64k`, `128k`, `256k` or a number | the largest that fits |
+| `PARALLEL` | requests answered at the same time, 1 to 8 | fitted to the card |
+| `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` |
+| `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` |
+| `GPU` / `GPUS` | card numbers, `0` or `0,1`; empty or `all` = the engine's choice | all that fit |
+| `TUNE` | `quick` (about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |
+| `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you |
+| `PORT` | the port inside the container | `8080` |
+| `PRESET` | the preset name in `/data` | `docker` |
+| `STRATA_SET` | advanced engine settings, space separated, like `strata launch --set` (`"prefill=65536 STRATA_PIN_BY_NEED=1"`) | none |
+
+Your choices are saved as a preset in the volume. Start again with no settings and the saved preset is used; start with
+settings and they replace the saved ones (tuning results are kept). To change one thing, recreate the container with the new
+variable; the model and tuning stay in `/data`. Anything else: `docker exec strata strata launch --help`, or
+`docker run --rm IMAGE strata launch --plan --model q2_0` to see what a choice would do without starting anything.
+
+What is in the volume `/data`: `Strata-data/` (model files, the prepared pack), `state/` (presets, tuning, the key), `install/`
+(run settings and logs), `home/`. The engine and the Terminal are in the image, so updating the image never touches your files.
+
+### Files you already have
+
+Mount the folder with the model's GGUF files read-only and name it: `-v /srv/gguf/IQ3_XXS:/models:ro -e MODEL_PATH=/models
+-e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume.
+
+## The API key
+
+The server listens on every address **inside** the container (that is what lets the published port work), and Strata does not
+serve beyond this computer without a key. So a key is always on:
+
+- **You set it** with `API_KEY` (16 characters or more; `openssl rand -hex 24` makes one) or `API_KEY_FILE`.
+- **You do not**: the first start makes one, keeps it in the volume, and the log tells you how to show it:
+ `docker exec strata strata launch --show-key --preset docker`.
+
+Publish the port as `-p 127.0.0.1:8080:8080` to keep it on this computer. Publish it as `-p 8080:8080` (or bind `0.0.0.0`) only
+when other computers should reach it, and then use a key you chose. Strata warns in the log when the key is short. Every
+request must send `Authorization: Bearer <key>`; `GET /health` answers without one.
+
+## Connect your coding apps
+
+From outside the container, the address is the published port and the key is yours:
+
+- OpenAI-compatible apps: base URL `http://127.0.0.1:8080/v1`, API key = your key, model name from `GET /v1/models`.
+- Anthropic-compatible apps (Claude Code): base URL `http://127.0.0.1:8080`, same key.
+
+`strata connect` writes the settings file for OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent and OpenClaw, and
+checks that a prompt and a tool call work. **Inside** the container it fills in the key and the model for you:
+
+```sh
+docker exec strata strata connect opencode # prints the file and the environment
+docker exec strata strata connect opencode --check # one prompt and one tool call, the way that app sends them
+```
+
+It prints `http://127.0.0.1:8080`, which is the address *inside* the container; on your computer use the published address
+(`-p 127.0.0.1:9000:8080` makes it `http://127.0.0.1:9000`). If your coding app runs on your computer, install the `strata`
+Terminal there ([TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)) and run `strata connect opencode --url http://HOST:8080 --key <your key>`.
+
+## Watch it and stop it
+
+```sh
+docker logs -f strata # what it is doing (the first start: the download)
+docker exec -it strata strata monitor # the live monitor (q leaves it, the server keeps running)
+docker exec strata strata monitor --once # one snapshot, for scripts
+docker ps # "healthy" once the model is loaded; "starting" while it downloads or loads
+docker stop strata # drains, unloads the model, frees the graphics memory, exits 0
+docker start strata # serving again in about a minute: the model and your choices are in the volume
+```
+
+The health check asks `GET /v1/models` with the key. During the first download the container shows `starting`, not `unhealthy`.
+`docker stop` sends SIGTERM; the container runs `strata launch --stop`, so the engine releases its memory properly. Give it time
+(`--stop-timeout 120` or compose's `stop_grace_period: 2m`): with the default 10 seconds a big model can be killed (exit 137).
+
+## Updating
+
+```sh
+docker pull ghcr.io/niko1221/strata:0.2.0-cuda13 # the new tag (TODO(owner): registry)
+docker stop strata && docker rm strata
+# run the same docker run command again with the new tag (compose: docker compose pull && docker compose up -d)
+```
+
+The models, the prepared pack, your preset and tuning stay in the volume `strata-data`; nothing is downloaded again. A new
+engine release can invalidate a tuning result: the first start says so and the preset is tuned again with `RETUNE=1`.
+`strata update` inside the container is not the way to update: the engine is part of the image.
+
+## Troubleshooting
+
+**"The engine in this image does not match this computer's graphics card"** or the container stops right at the start.
+The card is not visible to the container, or the image is for another card. Check the options for your card in
+[the commands](#one-command-per-card) and the table in [Which image](#which-image-for-which-card).
+
+**NVIDIA: `could not select device driver "" with capabilities: [[gpu]]`, or `docker: Error response from daemon: unknown or
+invalid runtime name: nvidia`.** The NVIDIA Container Toolkit is missing or Docker was not told about it:
+
+```sh
+sudo apt install nvidia-container-toolkit # or your distribution's package; see NVIDIA's install guide
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+docker run --rm --gpus all ubuntu nvidia-smi # must list your card
+```
+
+On Fedora / RHEL with Podman, use CDI: `sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml`, then `--device nvidia.com/gpu=all`.
+If `nvidia-smi` inside the container works but Strata says the driver is too old, you picked `cuda13` with a driver below 580:
+use the `cuda12` image or update the driver.
+
+**AMD: `Permission denied` on `/dev/kfd`, or no card found.** The container needs the device files and your user (or the
+container's user) must be allowed to use them. Add `--device /dev/kfd --device /dev/dri` and the groups
+`--group-add video --group-add $(getent group render | cut -d: -f3)` (the render group's number differs from computer to
+computer, so use the number, not the name). `ls -l /dev/kfd /dev/dri/renderD*` shows who owns them. On a computer with
+SELinux (Fedora, RHEL), add `--security-opt label=disable` for the device files to open.
+
+**Intel: no device found.** Pass `--device /dev/dri` and `--group-add <render group number>` as above. An Arc card needs the
+`i915` or `xe` driver on the host (`ls /dev/dri/renderD*` must show a device; `lspci -k | grep -A3 -i arc` shows the driver in use).
+
+**SELinux (Fedora, RHEL, Rocky): `Permission denied` on a folder you mounted.** Add `:z` (shared) or `:Z` (private) to the
+mount: `-v /srv/gguf/IQ3_XXS:/models:ro,z`. A named volume like `strata-data` needs nothing.
+
+**The container is killed or the computer swaps.** Strata loads tens of GB into RAM: leave the container without a memory
+limit, close other big programs, and pick a smaller `MODEL` ([MODELS.md](MODELS.md)). The log prints a warning when it sees a
+memory limit.
+
+**`docker stop` took long or ended with exit code 137.** The model was still unloading when Docker's time ran out. Use
+`--stop-timeout 120` (compose: `stop_grace_period: 2m`).
+
+**Port already in use.** Publish another host port: `-p 127.0.0.1:9000:8080` (inside the container it stays 8080).
+
+**The first start says it is downloading but nothing moves.** `docker logs -f strata` shows the percentage every few seconds;
+an interrupted download continues where it stopped when you start the container again. The volume needs the model's size free
+(the log names it).
+
+**"No MODEL was set"** warning: fine, Strata chose a model for your card; set `MODEL` to choose yourself.
+
+## From the older image
+
+The 0.1.x image compiled the engine inside `docker build` and was set up with `setup.py`. The new ones are built from the engine
+package and start through `strata launch`. Mapping:
+
+| Older variable | Now |
+| --- | --- |
+| `MODEL`, `FAMILY`, `CONTEXT`, `API_KEY`, `PORT`, `KV`, `GPU`, `GPUS`, `VISION` | the same (`VISION` takes `off\|on\|gpu\|cpu`; `no` / `yes` still work) |
+| `GGUF_DIR` | `MODEL_PATH` (the old name still works) |
+| `HOST` | not needed: the server listens on every address inside the container; choose where it is reachable with `-p` |
+| `LOW_RAM`, `RESIDENT_BUDGET_GIB`, `KV_STREAMING`, `LAYER_SPLIT`, `CONFIG`, `REINSTALL` | ignored with a warning; use `STRATA_SET` for engine settings and `strata launch --help` for the rest |
+
+The old volume layout (`/data/config`, `/data/models`) is not read; the model files are found again if you point `MODEL_PATH` at
+the old model folder. The container name, the volume and the port can stay as they were.
+
+## Build it yourself
+
+You do not need to: the engine packages are published with the release. If you want to build the images (for a computer without
+registry access, or after changing Strata), you need an engine package folder (made by `packaging/package_engine.py`, or fetched
+by `strata update` into a components folder) and Docker on Linux:
+
+```sh
+docker/build.sh --flavor cuda --engine ~/components/engine/0.2.0-linux-x86_64-cuda-cuda13 --tag strata:0.2.0-cuda13
+docker/build.sh --flavor hip --engine ~/components/engine/0.2.0-linux-x86_64-hip-gfx120X --tag strata:0.2.0-rocm-gfx120x
+docker/build.sh --flavor sycl --engine ~/components/engine/0.2.0-linux-x86_64-sycl --tag strata:0.2.0-sycl
+```
+
+`build.sh` assembles the build context (the Terminal's files and the package) and runs `docker build`. The build makes the
+Terminal exactly as the release archive (Python 3.12 and the locked packages, fetched at build time only), then checks the
+engine package before it may enter the image: every file's SHA-256, and that every library it imports is bundled or belongs
+to the system or the graphics driver (`packaging/binary_deps.py`); finally it checks that the dynamic linker resolves every
+library inside the finished image. A package that fails any of these stops the build. Nothing is pushed anywhere.
+
+**From source.** `docker build -t strata:src .` in the repository compiles the CUDA 13 engine inside Docker, makes the package
+and puts it in the same kind of image (`--build-arg CUDA_ARCHITECTURES=89` for one card only, much faster). For AMD and Intel
+builds from source see [AMD_HIP.md](AMD_HIP.md) and [INTEL.md](INTEL.md); then package the result with
+`packaging/package_engine.py` and use `docker/build.sh`.
+
+## Known limits
+
+- Linux x86-64 hosts only. Docker on Windows reaches NVIDIA cards only; Docker on a Mac cannot use the GPU.
+- The images run as `root` inside the container (GPU device files, page-locked memory). Run the container on a computer you
+ trust the way you would run the app; do not mount folders you do not want it to read.
+- An Arc card needs prepared model files (see [Intel](#intel-arc)).
+- AMD **RX 7000** (gfx110x) and RDNA 3 iGPUs: the gfx110X image is built for gfx1100 / 1101 / 1102 (RX 7900, 7800, 7700, 7600). The
+ Radeon 780M (gfx1103) is experimental and has its own engine package, not in these images.