Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docker: images built from the engine component packages (cuda13, cuda12, HIP gfx120X/gfx110X, SYCL), launch-based entrypoint, compose examples, from-source Dockerfile
Niko12215f1bd58e30b7
diff --git a/.gitattributes b/.gitattributesindex cfc1dc572..5248b4c8e 100644--- a/.gitattributes+++ b/.gitattributes@@ -16,3 +16,9 @@ data/quant_quality/** -text# the license text is shipped and checked as LF text on every checkoutLICENSE text eol=lf++# container scripts run in Linux: always LF+docker/*.sh text eol=lf+Dockerfile text eol=lf+docker/Dockerfile text eol=lf+docker/compose/* text eol=lfdiff --git a/Dockerfile b/Dockerfileindex 643060188..dc97a907f 100644--- a/Dockerfile+++ b/Dockerfile@@ -1,119 +1,93 @@-# syntax=docker/dockerfile:1+# Strata, built FROM SOURCE (NVIDIA CUDA 13). The engine is compiled here, packaged as an engine component package and then+# installed into the same runtime image as the ready-made ones (docker/Dockerfile): same Terminal, same entrypoint, same /data volume.#-# Strata: Qwen3.8-Flash-Next on NVIDIA GPUs (RTX 30/40/50, 12+ GB VRAM; two or-# three cards can share one model, 8 GB each - docs/MULTI_GPU.md).+# Most people want the ready-made images instead (nothing compiled): docs/DOCKER.md.#-# The engine is compiled during docker build, so the first container start only-# downloads the model (~70 GB) and starts the server. docker build has no GPU,-# so the CUDA architectures are fixed here instead of read from nvidia-smi: the-# engine is a fat binary with a cubin per listed arch, and the runtime picks the-# one matching your card. Narrow CUDA_ARCHITECTURES to your card for a faster-# build; a card outside the set needs a rebuild with its own arch.+# docker build -t strata:src . # RTX 20/30/40/50 (compiles for each, ~25 min)+# docker build -t strata:src --build-arg CUDA_ARCHITECTURES=86 . # one card only (faster); 86 = RTX 30, 89 = RTX 40, 120 = RTX 50+# docker run --rm --gpus all -p 127.0.0.1:8080:8080 --ulimit memlock=-1 --shm-size 2g --stop-timeout 120 \+# -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=change-me-please-123 strata:src#-# Build:-# docker build -t strata .-# docker build -t strata --build-arg CUDA_ARCHITECTURES=89 . # RTX 40 only+# docker build has no GPU, so the CUDA architectures are given here instead of read from nvidia-smi. The engine is a fat binary with+# a machine-code part per listed architecture (plus PTX for the newest, which later cards can JIT), and the runtime picks the one+# matching your card. Narrow CUDA_ARCHITECTURES to your card for a faster build. Native code for the CPU that builds the image is NOT+# used (STRATA_PORTABLE=ON), so the image runs on any x86-64 CPU with AVX2.#-# Run (host needs an NVIDIA driver >= 580 and nvidia-container-toolkit):-# docker run --rm --gpus all \-# -p 8080:8080 \-# --ulimit memlock=-1 \-# -v strata-data:/data \-# -e MODEL=IQ2_XS \-# strata-#-# Setup choices are env vars, read by docker-entrypoint.sh: FAMILY, MODEL, CONTEXT,-# VISION (no | yes | cpu), KV (int8 | q4_0 | k8v4), GPU (one card) or GPUS ("0,2"-# or "all", with LAYER_SPLIT), LOW_RAM (auto | on | off), HOST, PORT, API_KEY,-# GGUF_DIR (GGUF files you already have), RESIDENT_BUDGET_GIB, KV_STREAMING.-#-# Only the model files, the prepared pack, the MTP layer and the install config-# live in the /data volume; the engine is part of the image. Strata loads 32-62 GB-# into RAM, so a capped container needs -e LOW_RAM=on: setup.py reads the RAM from-# /proc/meminfo, which here is the host's total, not the container's limit. Add an-# API key before exposing the port to a network: -e API_KEY=<secret>. Pass-# -e REINSTALL=1 to change the model settings later.-#-# --gpus all on a host with two usable cards: setup takes both (the layer split is-# its recommended default). Pin one card with -e GPU=0, or name them with-# -e GPUS=0,2. A volume set up for one card switches to the pair on its first start-# on a two-card host unless GPU or GPUS pins it. LOW_RAM=on runs on one card.--FROM nvidia/cuda:13.0.0-devel-ubuntu24.04+# Plain legacy-builder features only (no heredocs, no --mount). AMD (HIP) and Intel (SYCL) source builds: docs/AMD_HIP.md, docs/INTEL.md.-# STRATA_EXECV=1: setup.py replaces itself with the server, so the server is PID 1-# and docker stop's SIGTERM reaches it (see setup.start). Normal Linux starts, which-# don't set it, keep spawning the server as a child.-ENV DEBIAN_FRONTEND=noninteractive PYTHONUNBUFFERED=1 LANG=C.UTF-8 STRATA_EXECV=1+ARG CUDA_IMAGE=nvidia/cuda:13.0.0-devel-ubuntu24.04+ARG RUNTIME_BASE=ubuntu:24.04+# ---- 1. compile the engine and make its component package+FROM ${CUDA_IMAGE} AS engine-build+ENV DEBIAN_FRONTEND=noninteractive PYTHONUNBUFFERED=1RUN apt-get update && apt-get install -y --no-install-recommends \- build-essential ca-certificates curl git libatomic1 libgomp1 \- python3 python3-pip python3-venv unzip \+ build-essential ca-certificates git libgomp1 libatomic1 python3 python3-pip python3-venv unzip \&& rm -rf /var/lib/apt/lists/*--WORKDIR /opt/strata+WORKDIR /srcCOPY . .--# RTX 20 (75), RTX 30 (86), RTX 40 (89), RTX 50 (120), plus 80 for A-series. CMakeLists-# refuses anything below 75. BUILD_VISION=0 skips the image encoder build.-ARG CUDA_ARCHITECTURES=75;80;86;89;120+ARG CUDA_ARCHITECTURES="75;80;86;89;120"ARG BUILD_VISION=1+ARG ENGINE_VERSION=+RUN python3 -m venv /venv && /venv/bin/pip install --no-cache-dir -r requirements.txt -r requirements-build.txt+# llama.cpp at the pinned commit (ggml for the engine, mtmd for pictures), exactly as setup.py fetches it+RUN /venv/bin/python -c "import setup; print(setup.get_llama_cpp())"+RUN set -e; \+ real=""; for a in $(echo "$CUDA_ARCHITECTURES" | tr ';,' ' '); do real="$real;${a}-real"; done; real=${real#;}; \+ last=$(echo "$CUDA_ARCHITECTURES" | tr ';,' ' ' | awk '{print $NF}'); \+ /venv/bin/cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \+ -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_PORTABLE=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON \+ "-DCMAKE_CUDA_ARCHITECTURES=${real};${last}-virtual" -DSTRATA_GGML_DIR=/src/third_party/llama.cpp; \+ /venv/bin/cmake --build build --target strata strata_kq -j "$(nproc)"; \+ if [ "$BUILD_VISION" = "1" ]; then \+ /venv/bin/cmake -G Ninja -S tools/vision -B build-vision -DCMAKE_BUILD_TYPE=Release -DLLAMA_DIR=/src/third_party/llama.cpp \+ -DSTRATA_VISION_CUDA=ON -DSTRATA_PORTABLE=ON "-DCMAKE_CUDA_ARCHITECTURES=${real};${last}-virtual"; \+ /venv/bin/cmake --build build-vision --target strata-vision -j "$(nproc)"; \+ fi+RUN set -e; \+ mkdir -p /stage /engine; \+ cp build/strata /stage/; \+ [ -f build/libstrata_kq.so ] && cp build/libstrata_kq.so /stage/ || true; \+ [ -f build-vision/bin/strata-vision ] && cp build-vision/bin/strata-vision /stage/ || true; \+ ver="${ENGINE_VERSION:-$(sed -n 's/^[[:space:]]*project([^)]*VERSION[[:space:]]*\([0-9][0-9.]*\).*/\1/p' CMakeLists.txt | head -1)}"; \+ echo "{\"version\": \"$ver\", \"source\": \"docker build\", \"archs\": \"$CUDA_ARCHITECTURES\", \"cuda\": \"13.0\", \"portable\": true}" > /stage/BUILD.json; \+ python3 packaging/collect_libs.py --binary build/strata --search /usr/local/cuda/lib64 --out /stage --also 'libcublasLt.so*'; \+ archs=""; for a in $(echo "$CUDA_ARCHITECTURES" | tr ';,' ' '); do archs="$archs --arch sm_$a"; done; \+ last=$(echo "$CUDA_ARCHITECTURES" | tr ';,' ' ' | awk '{print $NF}'); \+ python3 packaging/package_engine.py --src /stage --backend cuda --version "$ver" --out /engine $archs --arch "compute_$last" \+ --min-driver nvidia=580.65 --variant cuda13 --platform linux-x86_64 --include SYSTEM-LIBS.txt --include 'libstrata_kq*' \+ --include strata-vision; \+ for d in /engine/*/; do python3 packaging/package_engine.py --verify "$d"; python3 packaging/binary_deps.py "$d"; done-RUN python3 -m venv .venv \- && .venv/bin/pip install --no-cache-dir --upgrade pip \- && .venv/bin/pip install --no-cache-dir -r requirements.txt \- && chmod +x setup.sh docker-entrypoint.sh--# llama.cpp at the pinned commit, then the engine and the image encoder, built-# exactly the way setup.py builds them: native code for the CPU that builds the-# image, so build it on the PC it runs on. BUILD.json is what setup.py reads to-# decide whether an engine is current: source=local with a matching src hash-# means the first start reuses it instead of recompiling.-RUN .venv/bin/python - <<'PYEOF'-import json, os, pathlib, shutil-import setup--llama = setup.get_llama_cpp()-nvcc, _ = setup.find_nvcc()-arch = os.environ.get("CUDA_ARCHITECTURES", "75;80;86;89;120").strip().strip('"').replace(",", ";")-vision = "gpu" if os.environ.get("BUILD_VISION", "1") == "1" else "none"--setup.cmake_build(setup.ROOT, setup.ROOT / "build", "strata",- ["-DSTRATA_ENABLE_CUDA=ON", "-DSTRATA_Q6K_EXPERTS=ON", "-DSTRATA_MMQ_KQUANTS=ON", "-DSTRATA_BUILD_TESTS=OFF",- f"-DCMAKE_CUDA_ARCHITECTURES={arch}", f"-DCMAKE_CUDA_COMPILER={nvcc}",- f"-DSTRATA_GGML_DIR={llama}"], None, "build-strata.bat")-if vision != "none":- setup.cmake_build(setup.ROOT / "tools" / "vision", setup.ROOT / "build-vision", "strata-vision",- [f"-DLLAMA_DIR={llama}", "-DSTRATA_VISION_CUDA=ON", "-DSTRATA_PORTABLE=OFF",- f"-DCMAKE_CUDA_ARCHITECTURES={arch}", f"-DCMAKE_CUDA_COMPILER={nvcc}"], None, "build-vision.bat")--eng = setup.ROOT / "engine"-eng.mkdir(exist_ok=True)-shutil.copy2(setup.ROOT / "build" / setup.EXE, eng / setup.EXE)-if (setup.ROOT / "build" / setup.KQ_LIB).exists(): # the K-quant MMQ prompt library (UD-Q4/Q5/Q6_K_XL files)- shutil.copy2(setup.ROOT / "build" / setup.KQ_LIB, eng / setup.KQ_LIB)-if vision != "none":- shutil.copy2(setup.ROOT / "build-vision" / "bin" / setup.VEXE, eng / setup.VEXE)-bindir = pathlib.Path(nvcc).parent-meta = {"source": "local", "version": setup.source_version(),- "archs": [int(a.split("-")[0]) for a in arch.split(";") if a.split("-")[0].isdigit()], "vision": vision,- "cuda_dirs": [str(d) for d in (bindir, bindir / "x64", bindir.parent / "lib64") if d.is_dir()],- "src": setup.source_hash(setup.ENGINE_SOURCES),- "vision_src": setup.source_hash(setup.VISION_SOURCES) if vision != "none" else None}-(eng / "BUILD.json").write_text(json.dumps(meta, indent=1))-PYEOF--# the cmake trees are build-time only; the engine itself is what the container needs-RUN rm -rf build build-vision+# ---- 2. the Terminal (the release archive, built here)+FROM ${RUNTIME_BASE} AS terminal+ENV DEBIAN_FRONTEND=noninteractive+RUN apt-get update && apt-get install -y --no-install-recommends python3 ca-certificates && rm -rf /var/lib/apt/lists/*+COPY . /src/+RUN python3 /src/packaging/package_terminal.py --out /out --cache /cache \+ && mkdir -p /opt/strata \+ && tar -xzf /out/strata-terminal-*-linux-x86_64.tar.gz -C /opt/strata \+ && mv /opt/strata/strata-terminal-* /opt/strata/terminal \+ && /opt/strata/terminal/bin/strata --version+# ---- 3. the runtime image (the same as docker/Dockerfile's, FLAVOR=cuda)+FROM ${RUNTIME_BASE}+ARG STRATA_VERSION=dev+ENV DEBIAN_FRONTEND=noninteractive+RUN apt-get update && apt-get install -y --no-install-recommends tini ca-certificates libgomp1 libatomic1 libnuma1 zlib1g \+ && rm -rf /var/lib/apt/lists/* /var/cache/apt/*+COPY --from=terminal /opt/strata/terminal /opt/strata/terminal+COPY --from=engine-build /engine/ /opt/strata/components/engine/+COPY docker/entrypoint.sh docker/healthcheck.sh /opt/strata/+COPY docker/strata-wrapper.sh /usr/local/bin/strata+RUN chmod 0755 /opt/strata/entrypoint.sh /opt/strata/healthcheck.sh /usr/local/bin/strata+ENV STRATA_COMPONENTS_DIR=/opt/strata/components STRATA_INSTALL_DIR=/data/install XDG_STATE_HOME=/data/state \+ XDG_CONFIG_HOME=/data/config HOME=/data/home PORT=8080 LANG=C.UTF-8 PYTHONUNBUFFERED=1 PYTHONNOUSERSITE=1 STRATA_IN_DOCKER=1 \+ NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utilityVOLUME ["/data"]EXPOSE 8080--# /health is answered before the API key gate, so it works with or without one.-# The port only opens after the model loads (1-3 minutes, longer on a first run),-# so the start period is generous: a too short one marks a still-loading container-# unhealthy and a restart policy would kill it mid-download.-HEALTHCHECK --interval=30s --timeout=5s --start-period=600s --retries=3 \- CMD curl -fs "http://127.0.0.1:${PORT:-8080}/health" || exit 1--ENTRYPOINT ["./docker-entrypoint.sh"]+LABEL org.opencontainers.image.title="Strata" com.strata.flavor="cuda" com.strata.build="from-source" org.opencontainers.image.version="${STRATA_VERSION}"+HEALTHCHECK --interval=30s --timeout=8s --start-period=2h --retries=3 CMD ["/opt/strata/healthcheck.sh"]+STOPSIGNAL SIGTERM+ENTRYPOINT ["/usr/bin/tini", "--", "/opt/strata/entrypoint.sh"]+CMD ["run"]diff --git a/docker-entrypoint.sh b/docker-entrypoint.shdeleted file mode 100755index 9df74204d..000000000--- a/docker-entrypoint.sh+++ /dev/null@@ -1,106 +0,0 @@-#!/bin/sh-# Entry point for the Strata container. The engine is compiled during docker-# build and lives in the image, so the first start only downloads the model.-# The install config is kept on the /data volume so a recreated container skips-# the setup pass and goes straight to serving.-set -e-cd /opt/strata || exit 1--# #1584: the engine in the image is built with AVX2 CPU kernels. A host CPU without AVX2 (Sandy / Ivy Bridge Xeons,-# Bulldozer) runs it into an illegal instruction. A warning, not a stop: recommend, never force (the flags file-# can be wrong in a VM, and a build made on that host may be fine). STRATA_CPUINFO is for the test.-cpuinfo="${STRATA_CPUINFO:-/proc/cpuinfo}"-if [ -r "$cpuinfo" ] && grep -q '^flags' "$cpuinfo" && ! grep -m1 '^flags' "$cpuinfo" | grep -qw avx2; then- echo "WARNING: this host CPU has no AVX2. The engine in this image is built for CPUs with AVX2, so it may stop" >&2- echo " with 'illegal instruction' (exit 132). Build the image on this machine (docker build), which compiles the" >&2- echo " engine for the CPU it runs on, and see docs/INSTALL.md (Older CPUs, experimental) for the older-CPU build." >&2-fi--STRATA_DATA="${STRATA_DATA:-/data}"-FAMILY="${FAMILY:-qwen}"-MODEL="${MODEL:-IQ2_XS}"-CONTEXT="${CONTEXT:-32768}"-VISION="${VISION:-no}" # no | yes | cpu (the image encoder on the CPU)-HOST="${HOST:-0.0.0.0}"-PORT="${PORT:-8080}"-API_KEY="${API_KEY:-}"-KV="${KV:-}" # int8 | q4_0 | k8v4; empty: setup.py's own default (int8)-GPUS="${GPUS:-}" # "0,2" or "all": one model across several cards (docs/MULTI_GPU.md)-GPU="${GPU:-}" # one card, numbered as nvidia-smi numbers them-LAYER_SPLIT="${LAYER_SPLIT:-}" # with GPUS: where each later card's layers start (default: auto)-LOW_RAM="${LOW_RAM:-auto}" # on: the experts come from the pack's experts.bin, not from RAM-GGUF_DIR="${GGUF_DIR:-}" # a mounted folder with GGUF files you already have: no download-RESIDENT_BUDGET_GIB="${RESIDENT_BUDGET_GIB:-}" # UD-Q4_K_XL: GiB of experts kept in RAM (default: setup's pick)-KV_STREAMING="${KV_STREAMING:-}" # auto | on | off; empty: setup.py's own default (auto)-CONFIG="${CONFIG:-}" # a config file to start with (wins over MODEL's /data/config/strata-<model>.json)--# setup.py starts the newest strata-*.json it finds, so link in exactly the one-# this family and model were set up with. The config is the recorded output of-# that setup (the pack, the profile, the quant, the KV decision), not settings-# the entry point could rebuild from env vars. qwen has an empty family tag.-case "$FAMILY" in qwen) prefix="" ;; *) prefix="${FAMILY}-" ;; esac-tag="${prefix}$(printf '%s' "$MODEL" | tr 'A-Z' 'a-z')"-cfg="$STRATA_DATA/config/strata-$tag.json"-mkdir -p "$STRATA_DATA/config"--# REINSTALL is only needed to change settings for a model that is already set up-# (context, vision, KV, host, api_key). Switching between models already on the-# volume needs no setup pass: their config is already there.-#-# KV / GPUS / GPU / LAYER_SPLIT are passed only when set, so an unset one keeps-# setup.py's own default. LOW_RAM is always passed: setup.py measures the PC's RAM-# from /proc/meminfo, which in a container is the host's total, not the container's-# limit, so a memory-capped container has to ask for the low-RAM mode itself.-#-# Which config the server starts with (#1244): CONFIG when set; else a link in /opt/strata that already points-# into $STRATA_DATA/config/ (a pod command that picked one by linking); else MODEL's strata-<model>.json.-link="/opt/strata/strata-$tag.json"-keep=""-if [ -n "$CONFIG" ]; then- [ -f "$CONFIG" ] || { echo "CONFIG=$CONFIG does not exist." >&2; exit 1; }-elif [ "${REINSTALL:-0}" != "1" ] && [ -L "$link" ] && [ -f "$link" ]; then- case "$(readlink "$link")" in "$STRATA_DATA"/config/*) keep=1 ;; esac-fi--if [ -n "$CONFIG" ]; then- ln -sfn "$CONFIG" "$link"- echo "Config: $CONFIG (from CONFIG)"-elif [ -n "$keep" ]; then- echo "Config: $(readlink "$link") (existing link kept)"-elif [ "${REINSTALL:-0}" = "1" ] || [ ! -f "$cfg" ]; then- if [ -n "$GGUF_DIR" ]; then- echo "Setting up $tag from the GGUF files in $GGUF_DIR (the engine is already in the image)."- else- echo "Setting up $tag: downloading the model (~70 GB; the engine is already in the image)."- fi- set -- --family "$FAMILY" --model "$MODEL" --context "$CONTEXT" --vision "$VISION" \- --data-dir "$STRATA_DATA" --host "$HOST" --api-key "$API_KEY" \- --port "$PORT" --no-start --low-ram "$LOW_RAM"- if [ -n "$KV" ]; then set -- "$@" --kv "$KV"; fi- if [ -n "$GPUS" ]; then set -- "$@" --gpus "$GPUS"; fi- if [ -n "$GPU" ]; then set -- "$@" --gpu "$GPU"; fi- if [ -n "$LAYER_SPLIT" ]; then set -- "$@" --layer-split "$LAYER_SPLIT"; fi- if [ -n "$GGUF_DIR" ]; then set -- "$@" --gguf-dir "$GGUF_DIR"; fi- if [ -n "$RESIDENT_BUDGET_GIB" ]; then set -- "$@" --resident-budget-gib "$RESIDENT_BUDGET_GIB"; fi- if [ -n "$KV_STREAMING" ]; then set -- "$@" --kv-streaming "$KV_STREAMING"; fi- .venv/bin/python setup.py --setup --yes "$@"- [ -e "/opt/strata/strata-$tag.json" ] && { cmp -s "/opt/strata/strata-$tag.json" "$cfg" || cp -f "/opt/strata/strata-$tag.json" "$cfg"; }- echo "Config: $cfg (from MODEL $MODEL, just set up)"-else- # #1244: the copy on the volume is the one that counts, so a regular file left in /opt/strata by an earlier setup- # (or by an image built with one) must not stand in for it: edits to /data/config would be ignored- ln -sfn "$cfg" "/opt/strata/strata-$tag.json"- echo "Config: $cfg (from MODEL $MODEL)"-fi--# Later starts skip straight here: setup.py finds the installed config and-# launches serve/server.py (OpenAI- and Anthropic-compatible API on :8080).-# GPUS / GPU / LAYER_SPLIT are repeated on purpose. Given at the start they pin the-# cards for this model, and setup.py saves them in its config; without them a config-# that names one card is offered once to a pair, on its own, when the host has two-# cards that can share the model (setup.py's offer_together, docs/MULTI_GPU.md).-set -- --port "$PORT"-if [ -n "$GPUS" ]; then set -- "$@" --gpus "$GPUS"; fi-if [ -n "$GPU" ]; then set -- "$@" --gpu "$GPU"; fi-if [ -n "$LAYER_SPLIT" ]; then set -- "$@" --layer-split "$LAYER_SPLIT"; fi-exec .venv/bin/python setup.py "$@"diff --git a/docker/Dockerfile b/docker/Dockerfilenew file mode 100644index 000000000..2d6b5d8a7--- /dev/null+++ b/docker/Dockerfile@@ -0,0 +1,118 @@+# Strata container image, built from the engine COMPONENT PACKAGE: nothing is compiled on the user's side.+#+# one image = the Terminal (bundled Python 3.12 + strata_cli / strata_control / serve, built exactly as the release archive)+# + one engine package (CUDA 13, CUDA 12, HIP gfx120X, HIP gfx110X or SYCL) + the entrypoint+#+# Build it with docker/build.sh (it assembles the build context: the Terminal sources and the package folder):+#+# docker/build.sh --flavor cuda --engine /path/to/0.2.0-linux-x86_64-cuda-cuda13 --tag strata:0.2.0-cuda13+#+# Plain `docker build` also works when the context folder has this layout (build.sh makes it):+# terminal-src/ the files package_terminal.py needs (packaging/, strata_cli/, strata_control/, serve/, tools/, data/, ...)+# engine/ one folder per engine package, exactly as `package_engine.py` wrote it (engine-component.json inside)+# docker/ entrypoint.sh, strata-wrapper.sh, healthcheck.sh+# docker build --build-arg FLAVOR=cuda -t strata:0.2.0-cuda13 -f Dockerfile <context>+#+# FLAVOR picks the small set of system libraries the runtime image adds and its base:+# cuda NVIDIA: the engine package carries cudart/cublas; the driver's libcuda comes from nvidia-container-toolkit+# hip AMD: the engine package carries the ROCm runtime and kernels; the host's amdgpu driver and /dev/kfd are used+# sycl Intel: the engine package carries oneAPI's SYCL runtime; this image adds the Level Zero loader and Intel's compute runtime+# (Ubuntu 26.04 base: its compute runtime knows Arc A- and B-series; the 24.04 one does not know Battlemage)+#+# Only the legacy-builder-safe Dockerfile features are used (no heredocs, no --mount), so it builds with `docker build` as shipped+# in docker.io as well as with BuildKit.++ARG BASE_IMAGE=ubuntu:24.04++# ---- 1. the Terminal: the release archive, built here (pinned CPython 3.12 + hash-locked wheels, installed at build time only)+FROM ${BASE_IMAGE} AS terminal+ENV DEBIAN_FRONTEND=noninteractive+RUN apt-get update \+ && apt-get install -y --no-install-recommends python3 ca-certificates \+ && rm -rf /var/lib/apt/lists/*+COPY terminal-src/ /src/+RUN python3 /src/packaging/package_terminal.py --out /out --cache /cache \+ && mkdir -p /opt/strata \+ && tar -xzf /out/strata-terminal-*-linux-x86_64.tar.gz -C /opt/strata \+ && mv /opt/strata/strata-terminal-* /opt/strata/terminal \+ && /opt/strata/terminal/bin/strata --version++# ---- 2. the engine package, checked before it is allowed into the image: every file's hash, and every library it imports is+# bundled or belongs to the OS / GPU driver (packaging/binary_deps.py, the check the packager runs)+FROM ${BASE_IMAGE} AS verify+COPY --from=terminal /opt/strata/terminal /opt/strata/terminal+COPY terminal-src/packaging/package_engine.py terminal-src/packaging/binary_deps.py /opt/strata/terminal/packaging/+COPY engine/ /engine/+RUN set -e; n=0; for d in /engine/*/; do \+ echo "== $d"; \+ /opt/strata/terminal/python/bin/python3 /opt/strata/terminal/packaging/package_engine.py --verify "$d"; \+ /opt/strata/terminal/python/bin/python3 /opt/strata/terminal/packaging/binary_deps.py "$d"; \+ n=$((n+1)); \+ done; [ "$n" -ge 1 ] || { echo "no engine package in the build context (engine/<package>/engine-component.json)"; exit 1; }++# ---- 3. the runtime image+FROM ${BASE_IMAGE}+ARG FLAVOR=cuda+ARG STRATA_VERSION=dev+ARG ENGINE_LABEL=+ENV DEBIAN_FRONTEND=noninteractive++# system libraries the engine packages leave to the OS (SYSTEM-LIBS.txt), plus tini (PID 1: forwards docker stop, reaps children)+RUN set -e; \+ pkgs="tini ca-certificates libgomp1 libatomic1 libnuma1 zlib1g"; \+ case "$FLAVOR" in \+ cuda) ;; \+ hip) pkgs="$pkgs libdrm2 libdrm-amdgpu1 libelf1t64 libtinfo6 libpci3" ;; \+ sycl) pkgs="$pkgs libze1 libze-intel-gpu1 intel-opencl-icd" ;; \+ *) echo "FLAVOR must be cuda, hip or sycl"; exit 1 ;; \+ esac; \+ apt-get update; apt-get install -y --no-install-recommends $pkgs; \+ rm -rf /var/lib/apt/lists/* /var/cache/apt/*++COPY --from=terminal /opt/strata/terminal /opt/strata/terminal+COPY --from=verify /engine/ /opt/strata/components/engine/+COPY docker/entrypoint.sh docker/healthcheck.sh /opt/strata/+COPY docker/strata-wrapper.sh /usr/local/bin/strata+RUN chmod 0755 /opt/strata/entrypoint.sh /opt/strata/healthcheck.sh /usr/local/bin/strata++# Does the dynamic linker find every library of the package inside this image? (the GPU driver's libcuda / libnvidia-* come from the host)+RUN set -e; \+ for d in /opt/strata/components/engine/*/; do \+ miss=$(find "$d" -type f \( -name 'strata*' -o -name '*.so' -o -name '*.so.*' \) ! -name '*.json' ! -name '*.txt' | while read -r f; do \+ head -c4 "$f" | grep -q ELF || continue; \+ LD_LIBRARY_PATH="$d" ldd "$f" 2>/dev/null | grep 'not found' | grep -v -E 'libcuda\.so|libnvidia|libnvcuvid|libze_intel_gpu|libhsa-runtime64|libdrm_amdgpu' | sed "s|^|$(basename "$f"): |"; \+ done | sort -u); \+ if [ -n "$miss" ]; then echo "MISSING LIBRARIES in $d"; echo "$miss"; exit 1; fi; \+ echo "libraries of $d resolve inside this image"; \+ done++# Everything that is yours lives in /data: model files, the prepared pack, presets, tuning results, the API key, logs.+# The engine and the Terminal are in the image; updating the image never touches /data.+ENV STRATA_COMPONENTS_DIR=/opt/strata/components \+ STRATA_INSTALL_DIR=/data/install \+ XDG_STATE_HOME=/data/state \+ XDG_CONFIG_HOME=/data/config \+ HOME=/data/home \+ PORT=8080 \+ LANG=C.UTF-8 PYTHONUNBUFFERED=1 PYTHONNOUSERSITE=1 \+ STRATA_IN_DOCKER=1 \+ NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility+VOLUME ["/data"]+EXPOSE 8080++LABEL org.opencontainers.image.title="Strata" \+ org.opencontainers.image.description="Strata local LLM server (Terminal + engine package), flavor ${FLAVOR}" \+ org.opencontainers.image.version="${STRATA_VERSION}" \+ org.opencontainers.image.licenses="see LICENSE" \+ com.strata.flavor="${FLAVOR}" \+ com.strata.engine="${ENGINE_LABEL}"++# The port only opens after the model is downloaded, prepared and loaded (minutes; hours for a first 70 GB download), so the+# start period is long: "starting" is not "unhealthy".+HEALTHCHECK --interval=30s --timeout=8s --start-period=2h --retries=3 CMD ["/opt/strata/healthcheck.sh"]++# `docker stop` sends SIGTERM to tini, which passes it to the entrypoint; the entrypoint runs `strata launch --stop` (drain, unload,+# free the GPU memory). Give it time: docker run --stop-timeout 120 (compose: stop_grace_period: 2m).+STOPSIGNAL SIGTERM+ENTRYPOINT ["/usr/bin/tini", "--", "/opt/strata/entrypoint.sh"]+CMD ["run"]diff --git a/docker/build.sh b/docker/build.shnew file mode 100644index 000000000..bcab775b2--- /dev/null+++ b/docker/build.sh@@ -0,0 +1,104 @@+#!/bin/sh+# Build a Strata image from an engine component package. Run it on a Linux x86_64 machine with Docker (it needs the internet+# for the base image and for the Terminal's pinned Python and locked wheels; the engine package is local).+#+# docker/build.sh --flavor cuda --engine ~/pkgs/0.2.0-linux-x86_64-cuda-cuda13 --tag strata:0.2.0-cuda13+# docker/build.sh --flavor hip --engine ~/pkgs/0.2.0-linux-x86_64-hip-gfx120X --tag strata:0.2.0-rocm-gfx120x+# docker/build.sh --flavor sycl --engine ~/pkgs/0.2.0-linux-x86_64-sycl --tag strata:0.2.0-sycl+#+# --engine is the package FOLDER that packaging/package_engine.py wrote (engine-component.json inside). It can be given more than+# once to put several packages in one image (the Terminal picks the one that fits the card).+#+# --flavor cuda|hip|sycl which system libraries the runtime adds (see docker/Dockerfile)+# --tag NAME image name (default strata:<version>-<flavor>)+# --base IMAGE base image (default ubuntu:24.04; sycl: ubuntu:26.04)+# --repo DIR the Strata source tree (default: the one this script is in)+# --version V label (default: read from CMakeLists.txt)+# --ctx DIR keep the build context here (default: a temporary folder, removed after the build)+# --no-cache docker build --no-cache+# --ctx-only assemble the context and stop+# Environment: STRATA_DOCKER_BUILD_OPTS adds options to `docker build` (for example --network host).+# Nothing is pushed anywhere; the image stays in the local Docker.+set -eu++here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P)+repo=$(CDPATH= cd -- "$here/.." && pwd -P)+flavor=""; tag=""; base=""; version=""; ctx=""; nocache=""; ctxonly=""+engines=""+while [ $# -gt 0 ]; do+ case "$1" in+ --flavor) flavor=$2; shift 2 ;;+ --engine) engines="$engines+$2"; shift 2 ;;+ --tag) tag=$2; shift 2 ;;+ --base) base=$2; shift 2 ;;+ --repo) repo=$(CDPATH= cd -- "$2" && pwd -P); shift 2 ;;+ --version) version=$2; shift 2 ;;+ --ctx) ctx=$2; shift 2 ;;+ --no-cache) nocache=--no-cache; shift ;;+ --ctx-only) ctxonly=1; shift ;;+ -h|--help) sed -n '2,20p' "$0"; exit 0 ;;+ *) echo "build.sh: unknown option $1" >&2; exit 2 ;;+ esac+done+[ -n "$flavor" ] || { echo "build.sh: --flavor cuda|hip|sycl is required" >&2; exit 2; }+case "$flavor" in cuda|hip|sycl) ;; *) echo "build.sh: --flavor must be cuda, hip or sycl" >&2; exit 2 ;; esac+[ -n "$engines" ] || { echo "build.sh: --engine <package folder> is required" >&2; exit 2; }+[ -n "$base" ] || { [ "$flavor" = sycl ] && base=ubuntu:26.04 || base=ubuntu:24.04; }+if [ -z "$version" ]; then+ version=$(sed -n 's/^[[:space:]]*project([^)]*VERSION[[:space:]]*\([0-9][0-9.]*\).*/\1/p' "$repo/CMakeLists.txt" | head -1)+ version=${version:-dev}+fi++own_ctx=""+if [ -z "$ctx" ]; then ctx=$(mktemp -d "${TMPDIR:-/tmp}/strata-docker-ctx.XXXXXX"); own_ctx=1; fi+mkdir -p "$ctx/terminal-src" "$ctx/engine" "$ctx/docker"+cleanup() { [ -n "$own_ctx" ] && [ -z "$ctxonly" ] && rm -rf "$ctx"; }+trap cleanup EXIT++# 1. the Terminal's sources: exactly what packaging/package_terminal.py copies (the working tree, so local edits are in the image)+cd "$repo"+paths="packaging bin strata.py strata_control strata_cli serve requirements.txt requirements-monitor.txt docs/MANAGEMENT_PROTOCOL.md data setup.py ref/load.py CMakeLists.txt"+if git rev-parse --git-dir >/dev/null 2>&1; then+ # shellcheck disable=SC2086+ { git ls-files -z -- $paths ':(glob)tools/*.py'; } | tar --null -T - -cf - | tar -xf - -C "$ctx/terminal-src"+else+ # shellcheck disable=SC2086+ tar -cf - --exclude='__pycache__' $paths tools/*.py | tar -xf - -C "$ctx/terminal-src"+fi++# 2. the engine package(s): hard-linked when on the same file system (no second copy of GBs), copied otherwise+for e in $engines; do+ [ -f "$e/engine-component.json" ] || { echo "build.sh: $e has no engine-component.json (give the package folder)" >&2; exit 2; }+ dest="$ctx/engine/$(basename "$e")"+ rm -rf "$dest"+ cp -al "$e" "$dest" 2>/dev/null || { rm -rf "$dest"; cp -a "$e" "$dest"; } # (a failed hard-link attempt leaves a half-made folder)+ # a package that went through a zip or a Windows share lost its executable bit: the manifest hashes content only, so restore it on+ # OUR copy (break the hard link first, so the original is not changed)+ entry=$(sed -n 's/.*"entrypoint": *"\([^"]*\)".*/\1/p' "$dest/engine-component.json" | head -1)+ if [ -n "$entry" ] && [ -f "$dest/$entry" ] && [ ! -x "$dest/$entry" ]; then+ echo "note: $entry in $e is not executable; making it executable in the image copy"+ cp --remove-destination "$e/$entry" "$dest/$entry" && chmod 0755 "$dest/$entry"+ fi+done++# 3. the Dockerfile and the container scripts+cp "$here/Dockerfile" "$ctx/Dockerfile"+cp "$here/entrypoint.sh" "$here/strata-wrapper.sh" "$here/healthcheck.sh" "$ctx/docker/"+tr -d '\r' < "$here/entrypoint.sh" > "$ctx/docker/entrypoint.sh" # a checkout with CRLF line ends must not break the scripts+tr -d '\r' < "$here/strata-wrapper.sh" > "$ctx/docker/strata-wrapper.sh"+tr -d '\r' < "$here/healthcheck.sh" > "$ctx/docker/healthcheck.sh"++engine_label=$(for e in $engines; do basename "$e"; done | tr '\n' ' ' | sed 's/ $//')+[ -n "$tag" ] || tag="strata:$version-$flavor"+echo "context: $ctx"+echo "flavor $flavor, base $base, engine(s): $engine_label, tag $tag"+[ -n "$ctxonly" ] && { echo "context kept (--ctx-only)"; exit 0; }++# shellcheck disable=SC2086+docker build $nocache ${STRATA_DOCKER_BUILD_OPTS:-} \+ --build-arg "BASE_IMAGE=$base" --build-arg "FLAVOR=$flavor" \+ --build-arg "STRATA_VERSION=$version" --build-arg "ENGINE_LABEL=$engine_label" \+ -t "$tag" "$ctx"+docker image inspect "$tag" --format 'built {{.RepoTags}} {{.Size}} bytes'+echo "Not pushed anywhere. Size on disk: $(docker image inspect "$tag" --format '{{.Size}}') bytes."diff --git a/docker/compose/.env.example b/docker/compose/.env.examplenew file mode 100644index 000000000..eaf02c9e6--- /dev/null+++ b/docker/compose/.env.example@@ -0,0 +1,11 @@+# Copy to .env next to the compose file you use. Every line is optional.+# STRATA_IMAGE=ghcr.io/niko1221/strata:0.2.0-cuda13 # TODO(owner): where the images will be published+# MODEL=IQ3_XXS+# CONTEXT=32k+# PARALLEL=2+# VISION=off+# TUNE=skip+# API_KEY=put-a-long-random-key-here # openssl rand -hex 24+# STRATA_BIND=127.0.0.1 # 0.0.0.0 = reachable from your network (then use a real API_KEY)+# STRATA_PORT=8080+# RENDER_GID= # AMD / Intel only: getent group render | cut -d: -f3diff --git a/docker/compose/MAC.md b/docker/compose/MAC.mdnew file mode 100644index 000000000..a3017acdc--- /dev/null+++ b/docker/compose/MAC.md@@ -0,0 +1,9 @@+# Mac++Docker on a Mac runs Linux in a virtual machine that has no access to the Mac's GPU (no Metal), so a container could only+use the CPU, and Strata has no CPU-only mode worth running. Use the Mac app or the Terminal instead:++ curl -fsSL <release-folder>/install.sh | sh # then: strata launch++See docs/MAC.md and docs/TERMINAL_LAUNCH.md. The same goes for Windows with Docker Desktop and an AMD or Intel card:+the WSL2 virtual machine passes NVIDIA cards through, not Radeon or Arc cards. Use the Windows app or Terminal.diff --git a/docker/compose/docker-compose.amd.yml b/docker/compose/docker-compose.amd.ymlnew file mode 100644index 000000000..ff20d5942--- /dev/null+++ b/docker/compose/docker-compose.amd.yml@@ -0,0 +1,37 @@+# Strata on an AMD card. gfx120X image: RX 9000 series and Radeon AI PRO R9700 (RDNA4). gfx110X image: RX 7000 series (RDNA3).+# Needs the amdgpu kernel driver on the host (the ROCm user-space is in the image).+#+# cp .env.example .env+# docker compose -f docker-compose.amd.yml up -d+#+# TODO(owner): registry name. Until the images are published, build them (docker/build.sh) and use the local tag.+services:+ strata:+ image: ${STRATA_IMAGE:-strata:0.2.0-rocm-gfx120x}+ container_name: strata+ restart: unless-stopped+ ports:+ - "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080"+ environment:+ MODEL: ${MODEL:-}+ CONTEXT: ${CONTEXT:-32k}+ PARALLEL: ${PARALLEL:-}+ VISION: ${VISION:-off}+ TUNE: ${TUNE:-skip}+ API_KEY: ${API_KEY:-}+ volumes:+ - strata-data:/data+ # - /srv/gguf/IQ3_XXS:/models:ro+ devices:+ - /dev/kfd:/dev/kfd+ - /dev/dri:/dev/dri+ group_add:+ - video+ - "${RENDER_GID:-render}" # the host's render group id: getent group render | cut -d: -f3 (needed if you add `user:`; root does not need it)+ ulimits:+ memlock: { soft: -1, hard: -1 }+ shm_size: 2gb+ stop_grace_period: 2m+ # security_opt: [seccomp:unconfined] # only if the log says a ROCm call was blocked (rare)+volumes:+ strata-data:diff --git a/docker/compose/docker-compose.intel.yml b/docker/compose/docker-compose.intel.ymlnew file mode 100644index 000000000..25f97f796--- /dev/null+++ b/docker/compose/docker-compose.intel.yml@@ -0,0 +1,34 @@+# Strata on an Intel Arc card (A-series or B-series, including Arc Pro B70). Needs the xe or i915 kernel driver on the host.+# The Level Zero loader and Intel's compute runtime are in the image (Ubuntu 26.04 based).+#+# cp .env.example .env+# docker compose -f docker-compose.intel.yml up -d+#+# TODO(owner): registry name. Until the images are published, build them (docker/build.sh) and use the local tag.+services:+ strata:+ image: ${STRATA_IMAGE:-strata:0.2.0-sycl}+ container_name: strata+ restart: unless-stopped+ ports:+ - "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080"+ environment:+ MODEL: ${MODEL:-}+ CONTEXT: ${CONTEXT:-32k}+ PARALLEL: ${PARALLEL:-}+ VISION: ${VISION:-off}+ TUNE: ${TUNE:-skip}+ API_KEY: ${API_KEY:-}+ volumes:+ - strata-data:/data+ # - /srv/gguf/IQ3_XXS:/models:ro+ devices:+ - /dev/dri:/dev/dri+ group_add:+ - "${RENDER_GID:-render}" # the host's render group id: getent group render | cut -d: -f3 (needed if you add `user:`)+ ulimits:+ memlock: { soft: -1, hard: -1 }+ shm_size: 2gb+ stop_grace_period: 2m+volumes:+ strata-data:diff --git a/docker/compose/docker-compose.nvidia.yml b/docker/compose/docker-compose.nvidia.ymlnew file mode 100644index 000000000..198fc336b--- /dev/null+++ b/docker/compose/docker-compose.nvidia.yml@@ -0,0 +1,40 @@+# Strata on an NVIDIA card (RTX 20 to 50, and older cards with the cuda12 image). Needs the NVIDIA Container Toolkit on the host.+#+# cp .env.example .env # then edit: the model, the key+# docker compose -f docker-compose.nvidia.yml up -d+# docker compose -f docker-compose.nvidia.yml logs -f # the first start downloads the model (resumable)+#+# Image: cuda13 for RTX 20-50 with driver 580 or newer; cuda12 for Pascal / Volta (GTX 10, Tesla P100, V100) or drivers 575-579.+# TODO(owner): registry name. Until the images are published, build them (docker/build.sh) and use the local tag.+services:+ strata:+ image: ${STRATA_IMAGE:-strata:0.2.0-cuda13}+ container_name: strata+ restart: unless-stopped+ ports:+ - "${STRATA_BIND:-127.0.0.1}:${STRATA_PORT:-8080}:8080" # 127.0.0.1 = this computer only. 0.0.0.0 = your network: keep the API key then.+ environment:+ MODEL: ${MODEL:-} # empty: Strata picks the one it recommends for your card, and says so in the log+ CONTEXT: ${CONTEXT:-32k}+ PARALLEL: ${PARALLEL:-} # empty: fit to the card+ VISION: ${VISION:-off} # on: the model reads pictures (about 0.9 GB more to download)+ TUNE: ${TUNE:-skip} # quick: about 20 minutes once, then kept+ API_KEY: ${API_KEY:-} # empty: made on the first start; docker exec strata strata launch --show-key+ # GPUS: "0,1" # several cards: which ones+ volumes:+ - strata-data:/data # the model, presets, tuning, the key. Survives updates of the image.+ # - /srv/gguf/IQ3_XXS:/models:ro # GGUF files you already have (then also set MODEL_PATH: /models)+ deploy:+ resources:+ reservations:+ devices:+ - driver: nvidia+ count: all # or: device_ids: ["0"]+ capabilities: [gpu]+ # the short form of the same: gpus: all+ ulimits:+ memlock: { soft: -1, hard: -1 } # the engine pins tens of GB of RAM+ shm_size: 2gb+ stop_grace_period: 2m # the model drains and unloads on docker stop+volumes:+ strata-data:diff --git a/docker/entrypoint.sh b/docker/entrypoint.shnew file mode 100644index 000000000..2fb902cf7--- /dev/null+++ b/docker/entrypoint.sh@@ -0,0 +1,196 @@+#!/bin/sh+# Entry point of the Strata container: `strata launch --no-tui` with the settings the TUI asks for, read from environment+# variables, then stay up and stop cleanly on SIGTERM (`docker stop`).+#+# docker run ... strata start the server (default)+# docker run ... strata strata ... run any strata command instead (e.g. `strata launch --plan --model q2_0`)+# docker run ... strata sh a shell+#+# Settings (all optional; names match the screens of `strata launch`):+# MODEL model size, e.g. IQ3_XXS, Q2_0 (empty: Strata picks one for this card and says so)+# FAMILY model family when a size is offered by several (qwen, swift, coder, ...)+# MODEL_PATH GGUF files you already have, a folder or one .gguf, inside the container (mount it read-only)+# CONTEXT 32k, 64k, 128k, 256k or a number of tokens+# PARALLEL requests answered at the same time, 1 to 8 (empty: fit to the card)+# VISION off | on | gpu | cpu (pictures; the picture reader file is downloaded with the model)+# KV int8 | q4_0 | k8v4 | fp16 (conversation memory format)+# GPU / GPUS card numbers, e.g. 0 or 0,1 (empty or "all": the engine's choice)+# TUNE quick | thorough | skip (done once; RETUNE=1 tunes again)+# PORT port inside the container (default 8080)+# API_KEY / API_KEY_FILE the key clients must send. The server listens on every address inside the container, and Strata does+# not serve beyond loopback without a key, so one is always on: yours, or one made on the first start and kept in /data.+# PRESET preset name in /data (default "docker"). Settings above are saved into it; with no settings given a saved preset is reused.+# STRATA_SET advanced engine settings, space separated KEY=VALUE (like `strata launch --set`)+set -u++APP=/opt/strata+STRATA="$APP/terminal/bin/strata"+STATE_DIR="${XDG_STATE_HOME:-/data/state}/strata"+KEY_FILE="${STRATA_KEY_FILE:-/data/state/api-key}"+PHASE_FILE=/run/strata-phase+PRESET="${PRESET:-docker}"+PORT="${PORT:-8080}"++say() { printf '%s\n' "$*"; }+warn() { printf '[!] %s\n' "$*" >&2; }+die() { printf '[X] %s\n' "$*" >&2; exit "${2:-1}"; }+phase() { printf '%s\n' "$1" > "$PHASE_FILE" 2>/dev/null || true; }++# ---- other commands: `docker run image strata ...`, `docker run image sh`+if [ "$#" -gt 0 ] && [ "$1" != "run" ] && [ "$1" != "serve" ]; then+ case "$1" in strata) shift; exec "$STRATA" "$@" ;; *) exec "$@" ;; esac+fi++mkdir -p /data/state /data/home /data/install /data/Strata-data /run 2>/dev/null || die "/data is not writable. Mount a volume there: -v strata-data:/data"+if ! grep -q " /data " /proc/self/mountinfo 2>/dev/null; then+ warn "/data is not a volume, so the model and your settings are lost when this container is removed. Add: -v strata-data:/data"+fi++# ---- plain warnings (never refusals)+cpuinfo="${STRATA_CPUINFO:-/proc/cpuinfo}"+if [ -r "$cpuinfo" ] && grep -q '^flags' "$cpuinfo" && ! grep -m1 '^flags' "$cpuinfo" | grep -qw avx2; then+ warn "This computer's CPU has no AVX2. The engine is built for CPUs with AVX2 and may stop with 'illegal instruction'."+fi+limit=$(cat /sys/fs/cgroup/memory.max 2>/dev/null || echo max)+case "$limit" in max|"") ;; *[!0-9]*) ;; *)+ gb=$((limit / 1073741824))+ warn "This container has a memory limit of ${gb} GB. Strata plans from the host's RAM (/proc/meminfo) and loads tens of GB, so a low limit can end in the container being killed. Raise or remove the limit."+esac+if [ -n "${API_KEY:-}" ] && [ "${#API_KEY}" -lt 16 ]; then+ warn "API_KEY is short (${#API_KEY} characters). Use 16 or more; 'openssl rand -hex 24' makes a good one."+fi+for legacy in HOST LOW_RAM REINSTALL RESIDENT_BUDGET_GIB KV_STREAMING LAYER_SPLIT CONFIG; do+ eval "v=\${$legacy:-}"+ [ -n "$v" ] && warn "$legacy is from the older image and is ignored here. See docs/DOCKER.md (\"From the older image\")."+done++# ---- is there an engine for this card?+phase checking+engine_json=$("$STRATA" components engine --output json 2>/dev/null || true)+if ! printf '%s' "$engine_json" | grep -q '"usable": *true'; then+ echo "[X] The engine in this image does not match this computer's graphics card, or the card is not visible to the container." >&2+ "$STRATA" components engine 2>&1 | sed 's/^/ /' >&2 || true+ cat >&2 <<'MSG'+ Check, in this order:+ NVIDIA docker run --gpus all ... (needs nvidia-container-toolkit on the host; driver 580+ for the cuda13 image, 575+ for cuda12)+ AMD docker run --device /dev/kfd --device /dev/dri --group-add video --group-add render ...+ Intel docker run --device /dev/dri ... (add --group-add for the host's render group)+ Then pick the image for your card: docs/DOCKER.md.+MSG+ exit 3+fi++# ---- the API key+key="${API_KEY:-}"+if [ -z "$key" ] && [ -n "${API_KEY_FILE:-}" ]; then+ [ -r "$API_KEY_FILE" ] || die "API_KEY_FILE=$API_KEY_FILE cannot be read."+ key=$(tr -d '\r\n' < "$API_KEY_FILE")+fi++# ---- settings -> `strata launch` flags+set -- launch --no-tui --access lan --port "$PORT" --data-dir /data/Strata-data+have=""++preset_exists=""+if "$STRATA" launch --list-presets --output json 2>/dev/null | grep -q "\"name\": *\"$PRESET\""; then preset_exists=1; fi++model_path="${MODEL_PATH:-${GGUF_DIR:-}}"+[ -n "${MODEL:-}" ] && { set -- "$@" --model "$MODEL"; have=1; }+[ -n "${FAMILY:-}" ] && { set -- "$@" --family "$FAMILY"; have=1; }+[ -n "$model_path" ] && { set -- "$@" --model-path "$model_path"; have=1; }+[ -n "${CONTEXT:-}" ] && { set -- "$@" --context "$CONTEXT"; have=1; }+[ -n "${PARALLEL:-}" ] && [ "${PARALLEL}" != "auto" ] && { set -- "$@" --parallel "$PARALLEL"; have=1; }+[ -n "${KV:-}" ] && { set -- "$@" --kv "$KV"; have=1; }+if [ -n "${VISION:-}" ]; then+ case "$(printf '%s' "$VISION" | tr 'A-Z' 'a-z')" in+ off|no|false|0) v=off ;; on|yes|true|1) v=on ;; gpu) v=gpu ;; cpu) v=cpu ;;+ *) die "VISION=$VISION is not one of off, on, gpu, cpu." 2 ;;+ esac+ set -- "$@" --vision "$v"; have=1+fi+gpus="${GPUS:-${GPU:-}}"+case "$gpus" in ""|all|ALL) ;; *) set -- "$@" --gpu "$gpus"; have=1 ;; esac+[ -n "${STRATA_SET:-}" ] && { for kv in $STRATA_SET; do set -- "$@" --set "$kv"; done; have=1; }+[ -n "$key" ] && { set -- "$@" --api-key-env STRATA_LAUNCH_KEY; }++# tune once: ask only when this preset has not been tuned yet (a restart must not tune again for 20 minutes)+tune_mode=$(printf '%s' "${TUNE:-}" | tr 'A-Z' 'a-z')+if [ -n "$tune_mode" ] && [ "$tune_mode" != "skip" ]; then+ tuned=""+ if [ -n "$preset_exists" ] && "$STRATA" launch --list-presets 2>/dev/null | grep -E "^$PRESET[[:space:]]" | grep -qv "not tuned"; then tuned=1; fi+ if [ -n "$tuned" ] && [ "${RETUNE:-0}" != "1" ]; then+ say "This preset is already tuned; keeping the tuned settings (set RETUNE=1 to tune again)."+ else+ set -- "$@" --tune "$tune_mode"; have=1+ say "Tuning is on (TUNE=$tune_mode): the first start takes about $([ "$tune_mode" = quick ] && echo 20 || echo 120) minutes longer."+ fi+fi+[ "${TUNE_ALLOW_SMALL:-0}" = "1" ] && set -- "$@" --tune-allow-small++if [ -n "$preset_exists" ]; then+ set -- "$@" --preset "$PRESET"+ [ -n "$have" ] && set -- "$@" --save-preset "$PRESET"+elif [ -n "$have" ]; then+ set -- "$@" --save-preset "$PRESET"+else+ warn "No MODEL was set, so Strata picks the model it recommends for this card. Set MODEL (for example MODEL=IQ3_XXS) to choose."+ set -- "$@" --save-preset "$PRESET"+fi++# ---- start; `docker stop` during the download or the load ends it cleanly, after serving it drains and stops the model+stopping=""+launch_pid=""+on_term() {+ stopping=1+ phase stopping+ say "Stop requested: shutting down."+ if [ -n "$launch_pid" ] && kill -0 "$launch_pid" 2>/dev/null; then+ kill -INT "$launch_pid" 2>/dev/null # the launch is still downloading or loading: it stops what it started+ i=0; while kill -0 "$launch_pid" 2>/dev/null && [ "$i" -lt 25 ]; do sleep 1; i=$((i+1)); done+ kill -TERM "$launch_pid" 2>/dev/null || true+ fi+ "$STRATA" launch --stop --preset "$PRESET" || true+ exit 0+}+trap on_term TERM INT++phase preparing+say "Starting Strata: ${MODEL:-recommended model}${model_path:+ from $model_path} (preset '$PRESET')."+if [ -n "$key" ]; then+ STRATA_LAUNCH_KEY="$key" "$STRATA" "$@" &+else+ "$STRATA" "$@" &+fi+launch_pid=$!+wait "$launch_pid"+rc=$?+launch_pid=""+[ -n "$stopping" ] && exit 0+if [ "$rc" -ne 0 ]; then+ phase failed+ die "strata launch stopped with code $rc (the lines above say why). The container ends; a restart continues where it stopped (downloads resume)." "$rc"+fi++# keep the key where `docker exec ... strata monitor` and the health check can read it (private file on the volume)+( umask 077; "$STRATA" launch --show-key --preset "$PRESET" > "$KEY_FILE" 2>/dev/null ) || true+[ -s "$KEY_FILE" ] || rm -f "$KEY_FILE"+if [ -z "$key" ]; then+ warn "No API_KEY was set, so Strata made one (the server listens on all addresses in the container and will not run without a key)."+ warn "Show it with: docker exec <container> strata launch --show-key --preset $PRESET Set your own with -e API_KEY=..."+fi+phase serving+say "Serving on port $PORT. Watch it: docker exec -it <container> strata monitor Stop: docker stop <container>"++# ---- stay up; leave (so a restart policy can act) if the server is gone+bad=0+while [ -z "$stopping" ]; do+ sleep 10 &+ wait $!+ [ -n "$stopping" ] && break+ if "$STRATA" launch --status --output json 2>/dev/null | grep -q '"state": *"ready"'; then bad=0; else bad=$((bad+1)); fi+ if [ "$bad" -ge 3 ]; then+ phase failed+ die "The server is no longer running. Last lines of its log are in /data/install and 'docker logs'." 1+ fi+done+exit 0diff --git a/docker/healthcheck.sh b/docker/healthcheck.shnew file mode 100644index 000000000..e0bd0a49d--- /dev/null+++ b/docker/healthcheck.sh@@ -0,0 +1,18 @@+#!/bin/sh+# Container health: GET /v1/models with the API key. Healthy only when the server answers and has a model loaded.+# While the model downloads or loads (can take hours the first time) the container is "starting", not "unhealthy".+KEY_FILE="${STRATA_KEY_FILE:-/data/state/api-key}"+KEY="${API_KEY:-}"+[ -z "$KEY" ] && [ -r "$KEY_FILE" ] && KEY=$(cat "$KEY_FILE")+exec /opt/strata/terminal/python/bin/python3 - "http://127.0.0.1:${PORT:-8080}/v1/models" "$KEY" <<'PY'+import sys, urllib.request, json+url, key = sys.argv[1], sys.argv[2]+req = urllib.request.Request(url, headers={"Authorization": "Bearer " + key} if key else {})+try:+ with urllib.request.urlopen(req, timeout=4) as r:+ data = json.load(r)+ sys.exit(0 if r.status == 200 and data.get("data") else 1)+except Exception as e:+ print("not healthy: %s" % e, file=sys.stderr)+ sys.exit(1)+PYdiff --git a/docker/strata-wrapper.sh b/docker/strata-wrapper.shnew file mode 100644index 000000000..ce0ef91cb--- /dev/null+++ b/docker/strata-wrapper.sh@@ -0,0 +1,10 @@+#!/bin/sh+# `strata` inside the container: the Terminal, with the address and the API key of the server this container runs, so that+# `docker exec <name> strata monitor --once`, `strata status` and `strata connect opencode` work without typing either.+# A key or address you set yourself (STRATA_API_KEY, STRATA_URL, --url) always wins.+KEY_FILE="${STRATA_KEY_FILE:-/data/state/api-key}"+if [ -z "${STRATA_API_KEY:-}" ] && [ -r "$KEY_FILE" ]; then+ STRATA_API_KEY=$(cat "$KEY_FILE"); export STRATA_API_KEY+fi+STRATA_URL="${STRATA_URL:-http://127.0.0.1:${PORT:-8080}}"; export STRATA_URL+exec /opt/strata/terminal/bin/strata "$@"diff --git a/tools/test_docker_entrypoint.py b/tools/test_docker_entrypoint.pyindex 582415ffb..fe095b1c7 100644--- a/tools/test_docker_entrypoint.py+++ b/tools/test_docker_entrypoint.py@@ -1,7 +1,7 @@-"""docker-entrypoint.sh keeps the install config on the /data volume as the one that counts (#1244): with REINSTALL=0-and a config on the volume, /opt/strata/strata-<tag>.json is a link to it, even when a regular file is already there.-Runs the real script in a temporary tree with a stub for .venv/bin/python (POSIX sh and symlinks needed: skipped on-Windows).+"""docker/entrypoint.sh: environment variables become `strata launch` flags, the key never appears on a command line, an engine+that does not fit the card ends the start with a plain message, a started server is kept up, and SIGTERM runs `launch --stop`.++Runs the real script in a temporary tree with a stub for the `strata` command (POSIX sh needed: skipped on Windows).python -m unittest tools.test_docker_entrypoint"""@@ -9,108 +9,147 @@ from __future__ import annotationsimport osimport shutil+import signalimport subprocessimport tempfile+import timeimport unittestfrom pathlib import PathROOT = Path(__file__).resolve().parents[1]SH = shutil.which("sh")+STUB = r"""#!/bin/sh+# records every call; answers the few read-only queries the entrypoint makes+echo "$*" >> "$STUB_LOG"+case "$1 $2" in+ "components engine") if [ "${STUB_ENGINE:-ok}" = ok ]; then echo '{"usable": true}'; else echo '{"usable": false}'; echo "no engine for this card" >&2; fi; exit 0 ;;+esac+case "$*" in+ *"--list-presets --output json"*) if [ -n "${STUB_PRESET:-}" ]; then echo '{"presets": [{"name": "docker"}]}'; fi; exit 0 ;;+ *"--list-presets"*) if [ -n "${STUB_PRESET:-}" ]; then echo "docker iq3_xxs 32K fit to card ${STUB_TUNED:-not tuned}"; fi; exit 0 ;;+ *"launch --status"*) echo '{"running": [{"name": "docker", "state": "ready"}]}'; exit 0 ;;+ *"--show-key"*) echo "KEY-FROM-PRESET-0123456789"; exit 0 ;;+ *"launch --stop"*) echo "stopped" >> "$STUB_LOG"; exit 0 ;;+ *"launch --no-tui"*) echo "key-env=${STRATA_LAUNCH_KEY:-}" >> "$STUB_LOG"; [ -n "${STUB_FAIL:-}" ] && exit 6; exit 0 ;;+esac+exit 0+"""+[email protected](os.name == "nt" or SH is None, "needs POSIX sh and symlinks")[email protected](os.name == "nt" or SH is None, "needs POSIX sh")class Entrypoint(unittest.TestCase):def setUp(self):self.tmp = Path(tempfile.mkdtemp())self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True)- self.opt = self.tmp / "opt"- self.data = self.tmp / "data"- (self.opt / ".venv" / "bin").mkdir(parents=True)- (self.data / "config").mkdir(parents=True)- # the stub "python": setup.py --setup writes the config the way setup does; a plain start prints what it would load- stub = self.opt / ".venv" / "bin" / "python"- stub.write_text('#!/bin/sh\nif [ "$2" = "--setup" ]; then echo \'{"args": ["from-setup"]}\' > strata-iq3_s.json\n'- 'else echo "STARTED WITH: $(cat strata-iq3_s.json)"; fi\n', encoding="utf-8")+ (self.tmp / "opt" / "terminal" / "bin").mkdir(parents=True)+ stub = self.tmp / "opt" / "terminal" / "bin" / "strata"+ stub.write_text(STUB, encoding="utf-8")stub.chmod(0o755)- script = (ROOT / "docker-entrypoint.sh").read_text(encoding="utf-8").replace("/opt/strata", str(self.opt))+ self.log = self.tmp / "calls.log"+ text = (ROOT / "docker" / "entrypoint.sh").read_text(encoding="utf-8")+ text = (text.replace("/opt/strata", str(self.tmp / "opt")).replace("/data", str(self.tmp / "data"))+ .replace("/run/strata-phase", str(self.tmp / "phase")).replace("mkdir -p ", "mkdir -p " + str(self.tmp / "run") + " ", 1))self.script = self.tmp / "entry.sh"- self.script.write_text(script, encoding="utf-8")-- def run_entry(self, reinstall, **extra):- env = dict(os.environ, STRATA_DATA=str(self.data), MODEL="IQ3_S", REINSTALL=reinstall, **extra)- r = subprocess.run([SH, str(self.script)], env=env, capture_output=True, text=True, timeout=60)- self.assertEqual(r.returncode, 0, r.stderr)- return r.stdout-- def test_edited_volume_config_wins_over_a_regular_file_in_opt(self):- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["edited", "--kv-resident", "32768"]}\n')- (self.opt / "strata-iq3_s.json").write_text('{"args": ["stale"]}\n') # left by an earlier setup- out = self.run_entry("0")- self.assertIn("edited", out)- self.assertNotIn("stale", out)- self.assertTrue((self.opt / "strata-iq3_s.json").is_symlink())-- def test_link_is_made_when_there_is_no_file(self):- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["on-volume"]}\n')- self.assertIn("on-volume", self.run_entry("0"))-- def test_first_setup_copies_the_config_to_the_volume_and_the_next_start_links_it(self):- out = self.run_entry("0") # no config on the volume: setup runs- self.assertIn("from-setup", out)- cfg = self.data / "config" / "strata-iq3_s.json"- self.assertIn("from-setup", cfg.read_text())- cfg.write_text('{"args": ["edited-later"]}\n')- self.assertIn("edited-later", self.run_entry("0")) # the regular file setup left is replaced by the link-- def test_existing_link_into_the_data_config_dir_is_kept(self): # a pod command that picked a config by linking- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["plain"]}\n')- other = self.data / "config" / "strata-iq3_s.1x4.json"- other.write_text('{"args": ["batch8"]}\n')- os.symlink(other, self.opt / "strata-iq3_s.json")- out = self.run_entry("0")- self.assertIn("batch8", out)- self.assertNotIn("plain", out)- self.assertIn(f"Config: {other}", out)-- def test_link_to_somewhere_else_is_replaced(self):- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["on-volume"]}\n')- elsewhere = self.tmp / "elsewhere.json"- elsewhere.write_text('{"args": ["elsewhere"]}\n')- os.symlink(elsewhere, self.opt / "strata-iq3_s.json")- self.assertIn("on-volume", self.run_entry("0"))-- def test_config_env_wins(self):- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["plain"]}\n')- other = self.tmp / "mine.json"- other.write_text('{"args": ["mine"]}\n')- os.symlink(self.data / "config" / "strata-iq3_s.json", self.opt / "strata-iq3_s.json")- out = self.run_entry("0", CONFIG=str(other))- self.assertIn("mine", out)- self.assertNotIn("plain", out)- self.assertIn(f"Config: {other}", out)-- def test_model_names_the_config_and_it_is_printed(self):- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["by-model"]}\n')- out = self.run_entry("0")- self.assertIn("by-model", out)- self.assertIn(f"Config: {self.data}/config/strata-iq3_s.json", out)-- def warning_for(self, flags):- info = self.tmp / "cpuinfo"- info.write_text(f"processor\t: 0\nflags\t\t: {flags}\n")- (self.data / "config" / "strata-iq3_s.json").write_text('{"args": ["x"]}\n')- env = dict(os.environ, STRATA_DATA=str(self.data), MODEL="IQ3_S", REINSTALL="0", STRATA_CPUINFO=str(info))- r = subprocess.run([SH, str(self.script)], env=env, capture_output=True, text=True, timeout=60)- self.assertEqual(r.returncode, 0, r.stderr) # a warning, never a stop (#1584)- return r.stderr-- def test_a_cpu_without_avx2_is_warned_not_stopped(self):- err = self.warning_for("fpu sse4_2 avx aes")- self.assertIn("no AVX2", err)- self.assertIn("docker build", err)-- def test_a_cpu_with_avx2_gets_no_warning(self):- self.assertEqual(self.warning_for("fpu sse4_2 avx avx2 fma"), "")+ self.script.write_text(text, encoding="utf-8")++ def env(self, **extra):+ env = {k: v for k, v in os.environ.items() if k not in ("MODEL", "API_KEY", "CONTEXT", "VISION", "TUNE", "PORT", "GPU", "GPUS")}+ env.update(STUB_LOG=str(self.log), STRATA_CPUINFO=str(self.tmp / "cpuinfo"), **extra)+ (self.tmp / "cpuinfo").write_text("flags : fpu avx2 sse4_2\n")+ return env++ def start(self, **extra) -> subprocess.Popen:+ return subprocess.Popen([SH, str(self.script)], env=self.env(**extra), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)++ def wait_for(self, text: str, timeout: float = 20.0) -> str:+ end = time.time() + timeout+ while time.time() < end:+ if self.log.exists() and text in self.log.read_text():+ return self.log.read_text()+ time.sleep(0.1)+ self.fail(f"{text!r} not seen in {self.log.read_text() if self.log.exists() else '(no log)'}")++ def stop(self, p: subprocess.Popen) -> tuple[int, str, str]:+ p.send_signal(signal.SIGTERM)+ out, err = p.communicate(timeout=30)+ return p.returncode, out, err++ def launch_line(self, log: str) -> str:+ return next(line for line in log.splitlines() if line.startswith("launch --no-tui"))++ def test_settings_become_flags_and_sigterm_stops_cleanly(self):+ p = self.start(MODEL="IQ3_XXS", CONTEXT="64k", PARALLEL="2", VISION="yes", TUNE="quick", GPUS="0,1", PORT="8123", PRESET="office",+ API_KEY="a-long-enough-key-123456", MODEL_PATH="/models", STRATA_SET="prefill=65536 STRATA_PIN_BY_NEED=1")+ log = self.wait_for("--show-key")+ line = self.launch_line(log)+ for want in ("--access lan", "--port 8123", "--model IQ3_XXS", "--model-path /models", "--context 64k", "--parallel 2",+ "--vision on", "--gpu 0,1", "--tune quick", "--set prefill=65536", "--set STRATA_PIN_BY_NEED=1",+ "--save-preset office", "--api-key-env STRATA_LAUNCH_KEY"):+ self.assertIn(want, line)+ self.assertNotIn("a-long-enough-key", line) # the key is passed by environment, never on a command line+ self.assertIn("key-env=a-long-enough-key-123456", log)+ code, out, err = self.stop(p)+ self.assertEqual(code, 0, err)+ self.assertIn("launch --stop --preset office", self.log.read_text())+ self.assertIn("stopped", self.log.read_text())++ def test_saved_preset_is_reused_and_a_tuned_one_is_not_tuned_again(self):+ p = self.start(STUB_PRESET="1", STUB_TUNED="tuned 2026-10-17", TUNE="quick", MODEL="Q2_0")+ line = self.launch_line(self.wait_for("--show-key"))+ self.assertIn("--preset docker", line)+ self.assertIn("--model Q2_0", line)+ self.assertNotIn("--tune", line)+ self.stop(p)++ def test_untuned_preset_is_tuned_and_retune_forces_it(self):+ p = self.start(STUB_PRESET="1", TUNE="quick")+ self.assertIn("--tune quick", self.launch_line(self.wait_for("--show-key")))+ self.stop(p)+ self.log.unlink()+ p = self.start(STUB_PRESET="1", STUB_TUNED="tuned x", TUNE="quick", RETUNE="1")+ self.assertIn("--tune quick", self.launch_line(self.wait_for("--show-key")))+ self.stop(p)++ def test_no_settings_asks_for_the_recommended_model_and_warns(self):+ p = self.start()+ log = self.wait_for("--show-key")+ self.assertIn("--save-preset docker", self.launch_line(log))+ self.assertNotIn("--model ", self.launch_line(log))+ code, out, err = self.stop(p)+ self.assertIn("No MODEL was set", err)+ self.assertIn("made one", err) # no API_KEY given: told how to show the generated key++ def test_wrong_engine_for_this_card_ends_the_start_with_a_plain_message(self):+ r = subprocess.run([SH, str(self.script)], env=self.env(STUB_ENGINE="none"), capture_output=True, text=True, timeout=30)+ self.assertEqual(r.returncode, 3)+ self.assertIn("does not match this computer's graphics card", r.stderr)+ self.assertIn("docs/DOCKER.md", r.stderr)+ self.assertNotIn("launch --no-tui", self.log.read_text())++ def test_failed_launch_ends_the_container_with_its_code(self):+ r = subprocess.run([SH, str(self.script)], env=self.env(STUB_FAIL="1", MODEL="Q2_0"), capture_output=True, text=True, timeout=30)+ self.assertEqual(r.returncode, 6)+ self.assertIn("stopped with code 6", r.stderr)++ def test_bad_vision_value_is_a_plain_error(self):+ r = subprocess.run([SH, str(self.script)], env=self.env(VISION="maybe"), capture_output=True, text=True, timeout=30)+ self.assertEqual(r.returncode, 2)+ self.assertIn("VISION=maybe is not one of", r.stderr)++ def test_short_key_and_old_variables_warn(self):+ p = self.start(API_KEY="short", HOST="0.0.0.0", MODEL="Q2_0")+ self.wait_for("--show-key")+ code, out, err = self.stop(p)+ self.assertIn("API_KEY is short", err)+ self.assertIn("HOST is from the older image", err)++ def test_other_commands_run_instead_of_the_server(self):+ r = subprocess.run([SH, str(self.script), "strata", "launch", "--plan", "--model", "q2_0"], env=self.env(), capture_output=True,+ text=True, timeout=30)+ self.assertEqual(r.returncode, 0)+ self.assertIn("launch --plan --model q2_0", self.log.read_text())+ self.assertNotIn("--no-tui", self.log.read_text())if __name__ == "__main__":