Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docker: DOCKER.md, MODEL_PATH in compose, device check for AMD and Intel
Niko12213641dafe98e1
diff --git a/docker/compose/.env.example b/docker/compose/.env.exampleindex eaf02c9e6..b8931d834 100644--- a/docker/compose/.env.example+++ b/docker/compose/.env.example@@ -1,6 +1,7 @@# Copy to .env next to the compose file you use. Every line is optional.# STRATA_IMAGE=ghcr.io/niko1221/strata:0.2.0-cuda13 # TODO(owner): where the images will be published# MODEL=IQ3_XXS+# MODEL_PATH=/models # with a volume line - /srv/gguf/IQ3_XXS:/models:ro# CONTEXT=32k# PARALLEL=2# VISION=offdiff --git a/docker/compose/docker-compose.amd.yml b/docker/compose/docker-compose.amd.ymlindex ff20d5942..764244cca 100644--- a/docker/compose/docker-compose.amd.yml+++ b/docker/compose/docker-compose.amd.yml@@ -14,6 +14,7 @@ services:- "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080"environment:MODEL: ${MODEL:-}+ MODEL_PATH: ${MODEL_PATH:-} # GGUF files you already have, as seen inside the container (mount them below)CONTEXT: ${CONTEXT:-32k}PARALLEL: ${PARALLEL:-}VISION: ${VISION:-off}diff --git a/docker/compose/docker-compose.intel.yml b/docker/compose/docker-compose.intel.ymlindex 25f97f796..861314c6d 100644--- a/docker/compose/docker-compose.intel.yml+++ b/docker/compose/docker-compose.intel.yml@@ -14,6 +14,7 @@ services:- "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080"environment:MODEL: ${MODEL:-}+ MODEL_PATH: ${MODEL_PATH:-} # GGUF files you already have, as seen inside the container (mount them below)CONTEXT: ${CONTEXT:-32k}PARALLEL: ${PARALLEL:-}VISION: ${VISION:-off}diff --git a/docker/compose/docker-compose.nvidia.yml b/docker/compose/docker-compose.nvidia.ymlindex 198fc336b..f55fb2e0a 100644--- a/docker/compose/docker-compose.nvidia.yml+++ b/docker/compose/docker-compose.nvidia.yml@@ -15,6 +15,7 @@ services:- "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080" # 127.0.0.1 = this computer only. 0.0.0.0 = your network: keep the API key then.environment:MODEL: ${MODEL:-} # empty: Strata picks the one it recommends for your card, and says so in the log+ MODEL_PATH: ${MODEL_PATH:-} # GGUF files you already have, as seen inside the container (mount them below)CONTEXT: ${CONTEXT:-32k}PARALLEL: ${PARALLEL:-} # empty: fit to the cardVISION: ${VISION:-off} # on: the model reads pictures (about 0.9 GB more to download)diff --git a/docker/entrypoint.sh b/docker/entrypoint.shindex 2fb902cf7..3602abd51 100644--- a/docker/entrypoint.sh+++ b/docker/entrypoint.sh@@ -80,6 +80,13 @@ MSGexit 3fi+# the card is found through sysfs even when no device file was passed in: say so now instead of failing when the engine starts+backend=$(printf '%s' "$engine_json" | sed -n 's/.*"backend": *"\([a-z0-9]*\)".*/\1/p' | head -1)+case "$backend" in+ hip) [ -e /dev/kfd ] || { echo "[X] This is an AMD card, but /dev/kfd is not in the container. Add: --device /dev/kfd --device /dev/dri --group-add video --group-add <render group number> (docs/DOCKER.md)" >&2; exit 3; } ;;+ sycl) ls /dev/dri/renderD* >/dev/null 2>&1 || { echo "[X] This is an Intel card, but /dev/dri is not in the container. Add: --device /dev/dri --group-add <render group number> (docs/DOCKER.md)" >&2; exit 3; } ;;+esac+# ---- the API keykey="${API_KEY:-}"if [ -z "$key" ] && [ -n "${API_KEY_FILE:-}" ]; thendiff --git a/docs/DOCKER.md b/docs/DOCKER.mdnew file mode 100644index 000000000..dd9bcd8c8--- /dev/null+++ b/docs/DOCKER.md@@ -0,0 +1,288 @@+# Strata in Docker++Run Strata as a container: one command, nothing to install on the computer except Docker and your graphics card's driver.+The images come with Strata's engine already built for your card, so nothing is compiled on your side. The model is+downloaded on the first start into a volume (a download that stops continues where it stopped), and every later start goes+straight to serving.++> **On this page:** [Which image](#which-image-for-which-card) · [One command per card](#one-command-per-card) ·+> [Docker Compose](#docker-compose) · [Settings](#settings) · [The API key](#the-api-key) · [Connect your apps](#connect-your-coding-apps) ·+> [Watch and stop](#watch-it-and-stop-it) · [Update](#updating) · [Troubleshooting](#troubleshooting) ·+> [From the older image](#from-the-older-image) · [Build it yourself](#build-it-yourself)++Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the Mac app or the Terminal+([MAC.md](MAC.md), [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)). Docker Desktop on **Windows** passes NVIDIA cards through+(WSL 2) but not Radeon or Arc cards; on Windows the app is the easier way for every card.++## Which image for which card++| Your graphics card | Image tag | Needs on the computer |+| --- | --- | --- |+| NVIDIA **RTX 20, 30, 40, 50** series (and the matching RTX A / Quadro cards) | `0.2.0-cuda13` | NVIDIA driver **580 or newer**, [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) |+| NVIDIA **GTX 10 (Pascal), Tesla P100, V100 (Volta)**, or any NVIDIA card whose driver is **575 to 579** | `0.2.0-cuda12` | NVIDIA driver **575 or newer**, the same toolkit |+| AMD **RX 9000** series and Radeon AI PRO R9700 (RDNA 4, gfx1200 / gfx1201) | `0.2.0-rocm-gfx120x` | the `amdgpu` kernel driver (Linux 6.8+) |+| AMD **RX 7000** series (RDNA 3, gfx1100 / gfx1101 / gfx1102) | `0.2.0-rocm-gfx110x` | the `amdgpu` kernel driver |+| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.0-sycl` | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) |++Not sure which NVIDIA image? Take `cuda13` when `nvidia-smi` shows driver 580 or newer and the card is an RTX; otherwise `cuda12`.+A wrong pick does no harm: the container starts, says in plain words that the engine does not fit this card, and stops.++TODO(owner): where the images are published (registry and names). The plan, to decide at the 0.2.0 release:+`ghcr.io/niko1221/strata:0.2.0-cuda13`, `:0.2.0-cuda12`, `:0.2.0-rocm-gfx120x`, `:0.2.0-rocm-gfx110x`, `:0.2.0-sycl`, plus a+moving `:cuda13` / `:cuda12` / `:rocm-gfx120x` / `:rocm-gfx110x` / `:sycl` per card for "the newest". Until then,+[build them yourself](#build-it-yourself); the examples below use the plan's names.++## One command per card++Each command runs Strata in the background, keeps your files in the volume `strata-data`, and listens only on this computer+(`127.0.0.1:8080`). Replace `MODEL` and the key. The first start downloads the model (tens of GB): watch it with+`docker logs -f strata`.++**NVIDIA**++```sh+docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \+ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \+ -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \+ ghcr.io/niko1221/strata:0.2.0-cuda13+```++**AMD**++```sh+docker run -d --name strata --device /dev/kfd --device /dev/dri \+ --group-add video --group-add $(getent group render | cut -d: -f3) -p 127.0.0.1:8080:8080 \+ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \+ -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \+ ghcr.io/niko1221/strata:0.2.0-rocm-gfx120x+```++**Intel Arc**++```sh+docker run -d --name strata --device /dev/dri --group-add $(getent group render | cut -d: -f3) \+ -p 127.0.0.1:8080:8080 --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \+ -v strata-data:/data -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \+ -e API_KEY=$(openssl rand -hex 24) ghcr.io/niko1221/strata:0.2.0-sycl+```++What the options are for: `--ulimit memlock=-1:-1` lets the engine pin the RAM it uses for the model, `--shm-size 2g` gives+it shared memory, and `--stop-timeout 120` gives the model two minutes to unload when you stop the container (Docker's+default 10 seconds is too short for a big model). Do not put a memory limit on the container: Strata plans from the whole+computer's RAM and loads tens of GB.++To print the key you made: `echo` it before you start (it is in your shell history otherwise), or ask the running container:+`docker exec strata strata launch --show-key --preset docker`.++### Intel Arc++Strata's downloader prepares NVIDIA and AMD cards. For an Arc card the container **uses prepared model files**: the model's+GGUF files in a folder you mount (`-v /srv/gguf/IQ3_S:/models:ro` and `-e MODEL_PATH=/models`), and the prepared pack and MTP+layer in the data volume (`/data/Strata-data/packs/<model>` and `/data/Strata-data/mtp/rt`, made by the Intel installer on+this or another computer, see [INTEL.md](INTEL.md)). Without them the container tells you which file is missing and stops.+Nothing is downloaded or prepared for Arc inside the container. Arc B-series cards (xe driver) stream the experts from the+GGUF files, so the GGUF folder should be on a fast SSD.++## Docker Compose++Ready-made files are in [docker/compose/](../docker/compose): `docker-compose.nvidia.yml`, `docker-compose.amd.yml`,+`docker-compose.intel.yml` and `.env.example`. Copy the one for your card and its `.env.example`, edit, start:++```sh+cd docker/compose+cp .env.example .env # set MODEL=..., API_KEY=...+docker compose -f docker-compose.nvidia.yml up -d+docker compose -f docker-compose.nvidia.yml logs -f+```++They set what the commands above set: the GPU (`deploy.resources.reservations.devices` for NVIDIA, `/dev/kfd` and+`/dev/dri` plus the `video` and `render` groups for AMD, `/dev/dri` for Intel), `ulimits.memlock`, `shm_size`, a two-minute+`stop_grace_period`, the `strata-data` volume, and the port bound to `127.0.0.1`. To let other computers reach it, set+`STRATA_BIND=0.0.0.0` in `.env` and keep a real `API_KEY`.++## Settings++Everything is an environment variable (`-e NAME=value`, or in compose). They are the same questions `strata launch` asks on its+screens; leave out what you do not care about.++| Variable | What it does | Default |+| --- | --- | --- |+| `MODEL` | model size: `IQ2_XS`, `IQ3_XXS`, `IQ3_S`, `Q2_0`, `UD-Q4_K_XL`, ... ([MODELS.md](MODELS.md)) | Strata picks what suits the card, and says so in the log |+| `FAMILY` | `qwen`, `swift`, `coder`, ... when a size exists in several families | the usual one |+| `MODEL_PATH` | GGUF files you already have: a folder or one `.gguf`, **inside the container** (mount it with `-v`, `:ro` is fine) | download |+| `CONTEXT` | how much text it remembers at once: `32k`, `64k`, `128k`, `256k` or a number | the largest that fits |+| `PARALLEL` | requests answered at the same time, 1 to 8 | fitted to the card |+| `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` |+| `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` |+| `GPU` / `GPUS` | card numbers, `0` or `0,1`; empty or `all` = the engine's choice | all that fit |+| `TUNE` | `quick` (about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |+| `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you |+| `PORT` | the port inside the container | `8080` |+| `PRESET` | the preset name in `/data` | `docker` |+| `STRATA_SET` | advanced engine settings, space separated, like `strata launch --set` (`"prefill=65536 STRATA_PIN_BY_NEED=1"`) | none |++Your choices are saved as a preset in the volume. Start again with no settings and the saved preset is used; start with+settings and they replace the saved ones (tuning results are kept). To change one thing, recreate the container with the new+variable; the model and tuning stay in `/data`. Anything else: `docker exec strata strata launch --help`, or+`docker run --rm IMAGE strata launch --plan --model q2_0` to see what a choice would do without starting anything.++What is in the volume `/data`: `Strata-data/` (model files, the prepared pack), `state/` (presets, tuning, the key), `install/`+(run settings and logs), `home/`. The engine and the Terminal are in the image, so updating the image never touches your files.++### Files you already have++Mount the folder with the model's GGUF files read-only and name it: `-v /srv/gguf/IQ3_XXS:/models:ro -e MODEL_PATH=/models+-e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume.++## The API key++The server listens on every address **inside** the container (that is what lets the published port work), and Strata does not+serve beyond this computer without a key. So a key is always on:++- **You set it** with `API_KEY` (16 characters or more; `openssl rand -hex 24` makes one) or `API_KEY_FILE`.+- **You do not**: the first start makes one, keeps it in the volume, and the log tells you how to show it:+ `docker exec strata strata launch --show-key --preset docker`.++Publish the port as `-p 127.0.0.1:8080:8080` to keep it on this computer. Publish it as `-p 8080:8080` (or bind `0.0.0.0`) only+when other computers should reach it, and then use a key you chose. Strata warns in the log when the key is short. Every+request must send `Authorization: Bearer <key>`; `GET /health` answers without one.++## Connect your coding apps++From outside the container, the address is the published port and the key is yours:++- OpenAI-compatible apps: base URL `http://127.0.0.1:8080/v1`, API key = your key, model name from `GET /v1/models`.+- Anthropic-compatible apps (Claude Code): base URL `http://127.0.0.1:8080`, same key.++`strata connect` writes the settings file for OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent and OpenClaw, and+checks that a prompt and a tool call work. **Inside** the container it fills in the key and the model for you:++```sh+docker exec strata strata connect opencode # prints the file and the environment+docker exec strata strata connect opencode --check # one prompt and one tool call, the way that app sends them+```++It prints `http://127.0.0.1:8080`, which is the address *inside* the container; on your computer use the published address+(`-p 127.0.0.1:9000:8080` makes it `http://127.0.0.1:9000`). If your coding app runs on your computer, install the `strata`+Terminal there ([TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)) and run `strata connect opencode --url http://HOST:8080 --key <your key>`.++## Watch it and stop it++```sh+docker logs -f strata # what it is doing (the first start: the download)+docker exec -it strata strata monitor # the live monitor (q leaves it, the server keeps running)+docker exec strata strata monitor --once # one snapshot, for scripts+docker ps # "healthy" once the model is loaded; "starting" while it downloads or loads+docker stop strata # drains, unloads the model, frees the graphics memory, exits 0+docker start strata # serving again in about a minute: the model and your choices are in the volume+```++The health check asks `GET /v1/models` with the key. During the first download the container shows `starting`, not `unhealthy`.+`docker stop` sends SIGTERM; the container runs `strata launch --stop`, so the engine releases its memory properly. Give it time+(`--stop-timeout 120` or compose's `stop_grace_period: 2m`): with the default 10 seconds a big model can be killed (exit 137).++## Updating++```sh+docker pull ghcr.io/niko1221/strata:0.2.0-cuda13 # the new tag (TODO(owner): registry)+docker stop strata && docker rm strata+# run the same docker run command again with the new tag (compose: docker compose pull && docker compose up -d)+```++The models, the prepared pack, your preset and tuning stay in the volume `strata-data`; nothing is downloaded again. A new+engine release can invalidate a tuning result: the first start says so and the preset is tuned again with `RETUNE=1`.+`strata update` inside the container is not the way to update: the engine is part of the image.++## Troubleshooting++**"The engine in this image does not match this computer's graphics card"** or the container stops right at the start.+The card is not visible to the container, or the image is for another card. Check the options for your card in+[the commands](#one-command-per-card) and the table in [Which image](#which-image-for-which-card).++**NVIDIA: `could not select device driver "" with capabilities: [[gpu]]`, or `docker: Error response from daemon: unknown or+invalid runtime name: nvidia`.** The NVIDIA Container Toolkit is missing or Docker was not told about it:++```sh+sudo apt install nvidia-container-toolkit # or your distribution's package; see NVIDIA's install guide+sudo nvidia-ctk runtime configure --runtime=docker+sudo systemctl restart docker+docker run --rm --gpus all ubuntu nvidia-smi # must list your card+```++On Fedora / RHEL with Podman, use CDI: `sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml`, then `--device nvidia.com/gpu=all`.+If `nvidia-smi` inside the container works but Strata says the driver is too old, you picked `cuda13` with a driver below 580:+use the `cuda12` image or update the driver.++**AMD: `Permission denied` on `/dev/kfd`, or no card found.** The container needs the device files and your user (or the+container's user) must be allowed to use them. Add `--device /dev/kfd --device /dev/dri` and the groups+`--group-add video --group-add $(getent group render | cut -d: -f3)` (the render group's number differs from computer to+computer, so use the number, not the name). `ls -l /dev/kfd /dev/dri/renderD*` shows who owns them. On a computer with+SELinux (Fedora, RHEL), add `--security-opt label=disable` for the device files to open.++**Intel: no device found.** Pass `--device /dev/dri` and `--group-add <render group number>` as above. An Arc card needs the+`i915` or `xe` driver on the host (`ls /dev/dri/renderD*` must show a device; `lspci -k | grep -A3 -i arc` shows the driver in use).++**SELinux (Fedora, RHEL, Rocky): `Permission denied` on a folder you mounted.** Add `:z` (shared) or `:Z` (private) to the+mount: `-v /srv/gguf/IQ3_XXS:/models:ro,z`. A named volume like `strata-data` needs nothing.++**The container is killed or the computer swaps.** Strata loads tens of GB into RAM: leave the container without a memory+limit, close other big programs, and pick a smaller `MODEL` ([MODELS.md](MODELS.md)). The log prints a warning when it sees a+memory limit.++**`docker stop` took long or ended with exit code 137.** The model was still unloading when Docker's time ran out. Use+`--stop-timeout 120` (compose: `stop_grace_period: 2m`).++**Port already in use.** Publish another host port: `-p 127.0.0.1:9000:8080` (inside the container it stays 8080).++**The first start says it is downloading but nothing moves.** `docker logs -f strata` shows the percentage every few seconds;+an interrupted download continues where it stopped when you start the container again. The volume needs the model's size free+(the log names it).++**"No MODEL was set"** warning: fine, Strata chose a model for your card; set `MODEL` to choose yourself.++## From the older image++The 0.1.x image compiled the engine inside `docker build` and was set up with `setup.py`. The new ones are built from the engine+package and start through `strata launch`. Mapping:++| Older variable | Now |+| --- | --- |+| `MODEL`, `FAMILY`, `CONTEXT`, `API_KEY`, `PORT`, `KV`, `GPU`, `GPUS`, `VISION` | the same (`VISION` takes `off\|on\|gpu\|cpu`; `no` / `yes` still work) |+| `GGUF_DIR` | `MODEL_PATH` (the old name still works) |+| `HOST` | not needed: the server listens on every address inside the container; choose where it is reachable with `-p` |+| `LOW_RAM`, `RESIDENT_BUDGET_GIB`, `KV_STREAMING`, `LAYER_SPLIT`, `CONFIG`, `REINSTALL` | ignored with a warning; use `STRATA_SET` for engine settings and `strata launch --help` for the rest |++The old volume layout (`/data/config`, `/data/models`) is not read; the model files are found again if you point `MODEL_PATH` at+the old model folder. The container name, the volume and the port can stay as they were.++## Build it yourself++You do not need to: the engine packages are published with the release. If you want to build the images (for a computer without+registry access, or after changing Strata), you need an engine package folder (made by `packaging/package_engine.py`, or fetched+by `strata update` into a components folder) and Docker on Linux:++```sh+docker/build.sh --flavor cuda --engine ~/components/engine/0.2.0-linux-x86_64-cuda-cuda13 --tag strata:0.2.0-cuda13+docker/build.sh --flavor hip --engine ~/components/engine/0.2.0-linux-x86_64-hip-gfx120X --tag strata:0.2.0-rocm-gfx120x+docker/build.sh --flavor sycl --engine ~/components/engine/0.2.0-linux-x86_64-sycl --tag strata:0.2.0-sycl+```++`build.sh` assembles the build context (the Terminal's files and the package) and runs `docker build`. The build makes the+Terminal exactly as the release archive (Python 3.12 and the locked packages, fetched at build time only), then checks the+engine package before it may enter the image: every file's SHA-256, and that every library it imports is bundled or belongs+to the system or the graphics driver (`packaging/binary_deps.py`); finally it checks that the dynamic linker resolves every+library inside the finished image. A package that fails any of these stops the build. Nothing is pushed anywhere.++**From source.** `docker build -t strata:src .` in the repository compiles the CUDA 13 engine inside Docker, makes the package+and puts it in the same kind of image (`--build-arg CUDA_ARCHITECTURES=89` for one card only, much faster). For AMD and Intel+builds from source see [AMD_HIP.md](AMD_HIP.md) and [INTEL.md](INTEL.md); then package the result with+`packaging/package_engine.py` and use `docker/build.sh`.++## Known limits++- Linux x86-64 hosts only. Docker on Windows reaches NVIDIA cards only; Docker on a Mac cannot use the GPU.+- The images run as `root` inside the container (GPU device files, page-locked memory). Run the container on a computer you+ trust the way you would run the app; do not mount folders you do not want it to read.+- An Arc card needs prepared model files (see [Intel](#intel-arc)).+- AMD **RX 7000** (gfx110x) and RDNA 3 iGPUs: the gfx110X image is built for gfx1100 / 1101 / 1102 (RX 7900, 7800, 7700, 7600). The+ Radeon 780M (gfx1103) is experimental and has its own engine package, not in these images.