Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat: publish signed backend and frontend images and run the server stack from them (#234)
Karlo Kajba Šimanićd5964054ebce
diff --git a/.github/workflows/images.yml b/.github/workflows/images.ymlindex 930f2ff..8d548c2 100644--- a/.github/workflows/images.yml+++ b/.github/workflows/images.yml@@ -1,9 +1,9 @@-# Builds the collector, Docker proxy and system helper images and publishes them to GHCR, signed-# with cosign (keyless, using this workflow's identity). Runs when a version tag is pushed; the tag must match+# Builds every Infradeck image (backend, frontend, collector, Docker proxy, system helper) and publishes them to+# GHCR, signed with cosign (keyless, using this workflow's identity). Runs when a version tag is pushed; the tag must match# the VERSION file:# git tag v0.1.0 && git push origin v0.1.0-# Remote hosts install from these images, pinned by digest, so no code is ever fetched from the-# dashboard. Third-party actions are pinned to commit hashes; update them deliberately.+# The server stack runs these images (docker-compose.server.yml names the version), and remote hosts install the+# collector ones pinned by digest, so no code is ever fetched from the dashboard. Third-party actions are pinned to commit hashes; update them deliberately.name: Publish imageson:@@ -22,6 +22,8 @@ jobs:fail-fast: falsematrix:include:+ - { name: backend, dockerfile: docker/backend.Dockerfile }+ - { name: frontend, dockerfile: docker/frontend.Dockerfile }- { name: collector, dockerfile: docker/collector.Dockerfile }- { name: docker-proxy, dockerfile: docker/docker-proxy.Dockerfile }- { name: system-helper, dockerfile: docker/system-helper.Dockerfile }@@ -66,8 +68,12 @@ jobs:- uses: sigstore/cosign-installer@398d4b0eeef1380460a10c8013a76f728fb906ac # v3- name: Sign the imagerun: cosign sign --yes "${IMAGE}@${{ steps.build.outputs.digest }}"- - name: Digest to put in .env+ # The collector and Docker proxy digests go into .env for Connect a server; the others are only listed.+ - name: Digestrun: |VAR=$(echo "${{ matrix.name }}" | tr 'a-z-' 'A-Z_')echo "### ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY"- echo "\`${VAR}_IMAGE=${IMAGE}@${{ steps.build.outputs.digest }}\`" >> "$GITHUB_STEP_SUMMARY"+ case "${{ matrix.name }}" in+ collector|docker-proxy) echo "\`${VAR}_IMAGE=${IMAGE}@${{ steps.build.outputs.digest }}\`" >> "$GITHUB_STEP_SUMMARY" ;;+ *) echo "\`${IMAGE}@${{ steps.build.outputs.digest }}\`" >> "$GITHUB_STEP_SUMMARY" ;;+ esacdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex c32dc96..ed8e196 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -34,6 +34,10 @@ The first public release. Earlier 0.x versions were private pre-releases.you can review and end, and read-only or write API tokens.- **Adding servers.** One command and one click: a join token, then approval on the dashboard. Collectorimages are signed with cosign and pinned by digest.+- **Published images.** Every Infradeck image (backend, frontend, collector, Docker proxy, system helper) is+ built by the release workflow for amd64 and arm64, signed with cosign and published on GHCR. A server install+ is a checkout of the release tag plus `docker compose pull && docker compose up -d`; nothing is built on the+ server unless you add `deploy/build.yml`.### Security@@ -69,5 +73,9 @@ that may need your attention:- **`GET /dashboard/summary`** was removed.- **Collectors.** Update them with a freshly generated command to get hard time limits on checks and the refusalto follow a redirect from the dashboard's address.+- **Images are pulled, not built.** Check out the tag (`git fetch --tags && git checkout v1.0.0`) and run+ `docker compose pull && docker compose up -d` instead of `git pull` and `--build`. Your volumes stay as they+ are. To keep building on the server, add `deploy/build.yml` to `COMPOSE_FILE`. Afterwards `docker image prune`+ removes the old local `infradeck-*` images.[1.0.0]: https://github.com/karloks2005/infradeck/releases/tag/v1.0.0diff --git a/README.md b/README.mdindex 9b50820..81130af 100644--- a/README.md+++ b/README.md@@ -81,14 +81,14 @@ To try **Connect a server** on the same machine, set `COLLECTOR_IMAGE` and `DOCKThe server stack (`docker-compose.server.yml`) publishes the dashboard through the reverse proxy you already run (a ready-made **Traefik** file is in [`deploy/`](deploy)). Collector endpoints are served only on a separate listener bound to your **VPN address**.-1. `cp example.env .env`, then set `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `DASHBOARD_DOMAIN`, `TRUSTED_PROXIES` and `COMPOSE_FILE=docker-compose.server.yml` (with Traefik: `docker-compose.server.yml:deploy/traefik.yml`, plus your Traefik names).+1. Clone the repository, check out a release (`git checkout v<version>`), `cp example.env .env`, then set `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `DASHBOARD_DOMAIN`, `TRUSTED_PROXIES` and `COMPOSE_FILE=docker-compose.server.yml` (with Traefik: `docker-compose.server.yml:deploy/traefik.yml`, plus your Traefik names).> `COMPOSE_FILE` matters: without it Compose runs the development stack.2. Generate `SECRETS_KEY` (needed for two-factor sign-in) and set `COLLECTOR_BIND` / `COLLECTOR_PUBLIC_URL` for your VPN.-3. `docker compose up -d --build`, then register the first account with the setup token from the backend log.+3. `docker compose pull && docker compose up -d` (the published, signed images of that release), then register the first account with the setup token from the backend log.4. Click **Connect a server** to add other machines.Every step, with the reasons and pitfalls: [Deploying on a server](docs/deployment.md), [Deploy behind Traefik](docs/deploy-traefik.md), [Adding servers](docs/adding-servers.md).-To update: `git pull && docker compose up -d --build`.+To update: `git fetch --tags && git checkout v<new-version> && docker compose pull && docker compose up -d`. To build from source instead, see [Building from source](docs/deployment.md#building-from-source).## Featuresdiff --git a/SECURITY.md b/SECURITY.mdindex 900d127..007a403 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -566,8 +566,11 @@ What runs on your machines is built from pinned inputs, and a release can't be sguard those tags: only a repository admin can create one, and nobody can move or delete one, so `v1.0.0`always names the same code. Changing a published tag means switching the ruleset off on purpose(Settings → Rules → Rulesets).-- **Images are signed.** The collector, Docker proxy and system helper images are signed with cosign (keyless,- tied to the workflow's identity) and published with provenance and an SBOM. Hosts install them by digest.+- **Images are signed.** Every Infradeck image (backend, frontend, collector, Docker proxy and system helper) is+ built by the release workflow, signed with cosign (keyless, tied to the workflow's identity) and published+ with provenance and an SBOM. The server stack pulls them by the release's version, and anyone can check the+ signature (`cosign verify`, see [Checking the signatures](docs/deployment.md#checking-the-signatures)); remote+ hosts install the collector images by digest. Nothing is built on the server unless you add `deploy/build.yml`.## Other exposure to be aware ofdiff --git a/TESTS.md b/TESTS.mdindex 0e86268..167c0c6 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,covers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **1632 test cases** from 882 test functions, plus 45 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).+- **1633 test cases** from 883 test functions, plus 45 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).- **About four minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collector@@ -189,7 +189,7 @@ The frontend's helper tests are separate, in `src/frontend/tests/` (see [Fronten| [`test_system_checks.py`](#test_system_checkspy) | 17 | Reboot, OOM, kernel-fault, clock and pending-reboot detection, and the System row's event list. || [`test_system_helper.py`](#test_system_helperpy) | 35 | The optional system helper: kernel-log parsing, `wtmp`, crash detection, hostile input. || [`test_trusted_proxies.py`](#test_trusted_proxiespy) | 17 | Whose `X-Forwarded-For` nginx believes: nobody by default, only addresses and CIDR ranges from `TRUSTED_PROXIES`. |-| [`test_version.py`](#test_versionpy) | 3 | The dashboard and the collector report the version from the one `VERSION` file. |+| [`test_version.py`](#test_versionpy) | 4 | The dashboard and the collector report the version from the `VERSION` file, and the server stack runs the images of that version. |## Every test, explained@@ -1480,14 +1480,16 @@ settings that decide whose `X-Forwarded-For` is believed. These tests run the re### test_version.py-The version is written once, in `VERSION` at the repository root. If these fail, the dashboard could call an-outdated collector "up to date", or the other way round.+The version is written in `VERSION` at the repository root, and the server compose file repeats it in its image+tags. If these fail, the dashboard could call an outdated collector "up to date", or a release could run another+release's images.| Test | What it checks ||---|---|| `test_the_dashboard_and_the_collector_report_the_version_in_the_version_file` | The backend and the collector both report the version in `VERSION`, and it looks like `1.2.3`. || `test_the_collector_image_reads_the_version_file_next_to_it` | In the collector image, `VERSION` sits next to `collector.py` and is read from there. || `test_a_collector_without_a_version_file_says_dev` | A collector copied somewhere without `VERSION` reports `dev` instead of failing. |+| `test_the_server_stack_runs_the_published_images_of_this_version` | `docker-compose.server.yml` runs every Infradeck service from its published GHCR image tagged with the version in `VERSION`, builds none of them, and `deploy/build.yml` builds each of them from source. |## Frontend helper testsdiff --git a/deploy/build.yml b/deploy/build.ymlnew file mode 100644index 0000000..984803c--- /dev/null+++ b/deploy/build.yml@@ -0,0 +1,41 @@+# Builds the server stack's images from this checkout instead of pulling the published, signed ones.+# For trying changes that aren't released yet, or a fork. Add it after the server file in .env (and after your+# proxy's file, if any):+# COMPOSE_FILE=docker-compose.server.yml:deploy/build.yml+# Then build and start with `docker compose up -d --build`. Without this file, `docker compose pull` and+# `docker compose up -d` run the published images of the version in docker-compose.server.yml.+services:+ backend:+ build: { context: ., dockerfile: docker/backend.Dockerfile }+ image: infradeck-backend:latest++ agent-worker:+ build: { context: ., dockerfile: docker/backend.Dockerfile }+ image: infradeck-backend:latest++ collector:+ build: { context: ., dockerfile: docker/collector.Dockerfile }+ image: infradeck-collector:latest++ docker-proxy:+ build: { context: ., dockerfile: docker/docker-proxy.Dockerfile }+ image: infradeck-docker-proxy:latest++ system-helper:+ build: { context: ., dockerfile: docker/system-helper.Dockerfile }+ image: infradeck-system-helper:latest++ port-owners:+ build: { context: ., dockerfile: docker/system-helper.Dockerfile }+ image: infradeck-system-helper:latest++ frontend:+ build:+ context: .+ dockerfile: docker/frontend.Dockerfile+ args:+ VITE_API_BASE_URL: /api+ # Only a build from source applies it; the published image refreshes every 3 s (each user can change+ # that in Account › Display).+ VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS: ${LIVE_GRAPH_REFRESH_INTERVAL_SECONDS:-3}+ image: infradeck-frontend:latestdiff --git a/deploy/traefik.yml b/deploy/traefik.ymlindex 8727860..4336ee0 100644--- a/deploy/traefik.yml+++ b/deploy/traefik.yml@@ -1,7 +1,7 @@# Publishes the server stack through an existing Traefik (step by step: docs/deploy-traefik.md).# Add it after the server file in .env:# COMPOSE_FILE=docker-compose.server.yml:deploy/traefik.yml-# Then a plain `docker compose up -d --build` uses both. All values come from .env: DASHBOARD_DOMAIN+# Then a plain `docker compose pull && docker compose up -d` uses both. All values come from .env: DASHBOARD_DOMAIN# (required), TRAEFIK_ROUTER, TRAEFIK_ENTRYPOINT, TRAEFIK_CERT_RESOLVER and TRAEFIK_NETWORK (defaults shown).# Also set TRUSTED_PROXIES to the subnet of Traefik's network, so each client keeps its own address.services:diff --git a/docker-compose.server.yml b/docker-compose.server.ymlindex 0ba5158..b21b3c4 100644--- a/docker-compose.server.yml+++ b/docker-compose.server.yml@@ -3,11 +3,14 @@# frontend attached to the proxy's network (as deploy/traefik.yml does); never attach the proxy to this stack's# `front` network, where the backend and the collector gate trust whatever reaches them.# Set COMPOSE_FILE in .env, adding the ready-made file for your proxy from deploy/ if there is one-# (see docs/deployment.md), then run a plain `docker compose up -d --build`:+# (see docs/deployment.md), then run a plain `docker compose pull && docker compose up -d`:# COMPOSE_FILE=docker-compose.server.yml:deploy/traefik.yml (Traefik)# COMPOSE_FILE=docker-compose.server.yml (any other reverse proxy)+# The Infradeck images are the published, signed ones of the version below, which is always the version of this+# checkout (check out a release tag, e.g. `git checkout v1.0.0`). To build them from source instead, add+# deploy/build.yml to COMPOSE_FILE and use `docker compose up -d --build`.# Nothing here needs editing: your settings come from .env (see example.env),-# so `git pull` can always update this file. Don't copy it under another name: Compose+# so checking out a newer tag can always update this file. Don't copy it under another name: Compose# prefers compose.yml (the development stack) when two default-named files exist.services:postgres:@@ -63,11 +66,8 @@ services:retries: 5backend:- build:- context: .- dockerfile: docker/backend.Dockerfile+ image: ghcr.io/karloks2005/infradeck/backend:0.10.0container_name: infradeck-backend- image: infradeck-backend:latestrestart: unless-stopped# Runs as an unprivileged user (see the Dockerfile) with no capabilities and a read-only filesystem.cap_drop: [ALL]@@ -134,10 +134,7 @@ services:# stuck model never holds up the API. Stopping it only delays diagnoses; everything else keeps working.# It needs the database, Redis and the internet (the model provider), and nothing else. See docs/containers.md.agent-worker:- build:- context: .- dockerfile: docker/backend.Dockerfile- image: infradeck-backend:latest+ image: ghcr.io/karloks2005/infradeck/backend:0.10.0container_name: infradeck-agent-workerrestart: unless-stoppedcommand: ["python", "-m", "src.agent.worker"]@@ -165,10 +162,7 @@ services:# Monitors the machine running this stack. Other servers use collector-compose.yml.collector:- build:- context: .- dockerfile: docker/collector.Dockerfile- image: infradeck-collector:latest+ image: ghcr.io/karloks2005/infradeck/collector:0.10.0container_name: infradeck-collectorrestart: unless-stoppednetworks: [collector]@@ -240,10 +234,7 @@ services:# It answers six kinds of read request and nothing else. It has no network at all: the collector# reaches it through a Unix socket in a volume only the two of them share. See SECURITY.md.docker-proxy:- build:- context: .- dockerfile: docker/docker-proxy.Dockerfile- image: infradeck-docker-proxy:latest+ image: ghcr.io/karloks2005/infradeck/docker-proxy:0.10.0container_name: infradeck-docker-proxyrestart: unless-stoppeduser: "0:0"@@ -262,10 +253,7 @@ services:# filesystem, and sees the host only through the read-only wtmp file. See SECURITY.md.system-helper:profiles: ["system-events"]- build:- context: .- dockerfile: docker/system-helper.Dockerfile- image: infradeck-system-helper:latest+ image: ghcr.io/karloks2005/infradeck/system-helper:0.10.0container_name: infradeck-system-helperrestart: unless-stoppeduser: "0:0"@@ -287,10 +275,7 @@ services:# read-only filesystem, and publishes only protocol, port and process name. Read SECURITY.md first.port-owners:profiles: ["port-owners"]- build:- context: .- dockerfile: docker/system-helper.Dockerfile- image: infradeck-system-helper:latest+ image: ghcr.io/karloks2005/infradeck/system-helper:0.10.0container_name: infradeck-port-ownersrestart: unless-stoppeduser: "0:0"@@ -308,12 +293,7 @@ services:- system_events:/outfrontend:- build:- context: .- dockerfile: docker/frontend.Dockerfile- args:- VITE_API_BASE_URL: /api- VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS: ${LIVE_GRAPH_REFRESH_INTERVAL_SECONDS:-3}+ image: ghcr.io/karloks2005/infradeck/frontend:0.10.0container_name: infradeck-frontendrestart: unless-stopped# Runs as the unprivileged nginx user (see the Dockerfile) with no capabilities and a read-only filesystem;diff --git a/docker/frontend.Dockerfile b/docker/frontend.Dockerfileindex 8efe03d..2b83c26 100644--- a/docker/frontend.Dockerfile+++ b/docker/frontend.Dockerfile@@ -1,4 +1,6 @@-FROM node:24.22.3-alpine@sha256:98e4e60b35fd50143e1838da2957488c25aa3a3efbd48bb679923f77d6d3d873 AS build+# The build stage runs on the builder's own platform: its output is plain JavaScript, the same for every+# platform, so an arm64 image doesn't have to run npm under emulation.+FROM --platform=$BUILDPLATFORM node:24.22.3-alpine@sha256:98e4e60b35fd50143e1838da2957488c25aa3a3efbd48bb679923f77d6d3d873 AS buildWORKDIR /appdiff --git a/docs/configuration.md b/docs/configuration.mdindex 85f11b2..6f329f0 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -40,7 +40,7 @@ Every setting below is listed, with no values, in [`example.env`](../example.env| `INFRADECK_SERVER_NAME` | host's hostname | Name shown for the stack's own machine. || `DOMAIN` | `localhost` | Domain for the optional Caddy profile. || `COMPOSE_FILE` | – | On a server: `docker-compose.server.yml`, with Traefik `docker-compose.server.yml:deploy/traefik.yml`, so plain `docker compose` uses them. |-| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes (server file; applied when the frontend image is built). |+| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes by default. Applies only when the frontend is built from source (`deploy/build.yml`); the published image uses 3 s. Each user can change it in **Account → Display**. || `INFRADECK_CHECK_INTERVAL_SECONDS` | `60` | Initial check interval of the stack's own collector (server file); the dashboard setting overrides it. |## Collector (environment of the collector container)diff --git a/docs/containers.md b/docs/containers.mdindex d0fde04..ee08e3e 100644--- a/docs/containers.md+++ b/docs/containers.md@@ -13,15 +13,15 @@ For the settings, see [Configuration](configuration.md).| Container | Image | Runs on | Required? | Job ||---|---|---|---|---|-| `infradeck-frontend` | built from `docker/frontend.Dockerfile` (nginx) | dashboard machine | yes | Serves the web app and passes `/api` to the backend. |-| `infradeck-backend` | built from `docker/backend.Dockerfile` (Python, FastAPI) | dashboard machine | yes | The API: sign-in, metrics intake, alerts, statistics, notifications, the assistant. |+| `infradeck-frontend` | published, built from `docker/frontend.Dockerfile` (nginx) | dashboard machine | yes | Serves the web app and passes `/api` to the backend. |+| `infradeck-backend` | published, built from `docker/backend.Dockerfile` (Python, FastAPI) | dashboard machine | yes | The API: sign-in, metrics intake, alerts, statistics, notifications, the assistant. || `infradeck-postgres` | `postgres:16-alpine` | dashboard machine | yes | Everything that must be kept: accounts, servers, metric history, incidents, settings. || `infradeck-redis` | `redis:7-alpine` | dashboard machine | yes | Short-lived state: sign-in sessions, rate-limit counters, pending two-factor logins, a small cache, the agent's job queue. || `infradeck-agent-worker` | the backend image | dashboard machine | for the incident agent | Runs the agent's jobs (diagnoses) from the queue in Redis, apart from the API. |-| `infradeck-collector` | built from `docker/collector.Dockerfile` | every monitored machine | yes, one per machine | Reads the machine's CPU, memory, disks, network and containers and sends them to the backend. |-| `infradeck-docker-proxy` | built from `docker/docker-proxy.Dockerfile` | every monitored machine | yes, next to each collector | The only container with the Docker socket; answers six read-only questions. |+| `infradeck-collector` | published, built from `docker/collector.Dockerfile` | every monitored machine | yes, one per machine | Reads the machine's CPU, memory, disks, network and containers and sends them to the backend. |+| `infradeck-docker-proxy` | published, built from `docker/docker-proxy.Dockerfile` | every monitored machine | yes, next to each collector | The only container with the Docker socket; answers six read-only questions. || `infradeck-collector-gate` | `nginx:alpine` | dashboard machine, server stack only | yes in `docker-compose.server.yml` | The only door to the collector endpoints, published on your VPN address. |-| `infradeck-system-helper` | built from `docker/system-helper.Dockerfile` | any monitored machine | optional (`--profile system-events`) | Reads the kernel log and wtmp: OOM kills, kernel errors, crash vs clean shutdown. |+| `infradeck-system-helper` | published, built from `docker/system-helper.Dockerfile` | any monitored machine | optional (`--profile system-events`) | Reads the kernel log and wtmp: OOM kills, kernel errors, crash vs clean shutdown. || `infradeck-port-owners` | same image as the system helper | any monitored machine | optional (`--profile port-owners`) | Names the process behind each listening port in the Network tab. || `infradeck-caddy` | `caddy:2-alpine` | dashboard machine, `compose.yml` only | optional (`--profile https`) | HTTPS with an automatic Let's Encrypt certificate when there is no Traefik. |diff --git a/docs/deploy-traefik.md b/docs/deploy-traefik.mdindex 987daf4..2eda717 100644--- a/docs/deploy-traefik.md+++ b/docs/deploy-traefik.md@@ -34,12 +34,18 @@ All commands below run on the server, in the `infradeck` folder from step 1.## 1. Get the code+Replace `<version>` with the release you want, for example `1.0.0` (see the repository's **Releases** page):+```bashgit clone https://github.com/karloks2005/Infradeck.git infradeckcd infradeck+git checkout v<version>cp example.env .env```+The stack runs that release's published, signed images; the checkout supplies the compose files and the nginx+settings of the same version.+`.env` holds your passwords and keys. Keep it on the server and never commit it.## 2. Fill in `.env`@@ -78,7 +84,8 @@ Run `docker compose` without `-f` in this step and every later one. `-f` replaceout `deploy/traefik.yml`, and Traefik stops serving the dashboard.```bash-docker compose up -d --build+docker compose pull+docker compose up -ddocker compose ps```@@ -199,26 +206,31 @@ TRUSTED_PROXIES=<traefik-subnet, see step 2>The dashboard's local port is `127.0.0.1:5173` again. Set `FRONTEND_PORT` if something on the server relies onanother one.-Update:+Update to the new release, with `<new-version>` its number:```bash-git pull-docker compose up -d --build+git fetch --tags+git checkout v<new-version>+docker compose pull+docker compose up -d```-Rebuild everything; a service you don't rebuild keeps running the old code. Check the result as in-[Verify](#verify).+Check the result as in [Verify](#verify). An install from before 1.0.0 built its images on the server; this+pulls the published ones instead and keeps your volumes.### Roll back-Go back to the commit you ran before, and rebuild:+Go back to the release you ran before:```bash-git log --oneline -5-git checkout <previous-commit>-docker compose up -d --build+git checkout v<previous-version>+docker compose pull+docker compose up -d```+A version from before 1.0.0 has no published dashboard images: check out the commit you ran (`git log --oneline`)+and run `docker compose up -d --build` instead.+If that version still had the Traefik labels in `docker-compose.server.yml`, set`COMPOSE_FILE=docker-compose.server.yml` in `.env` first.@@ -264,7 +276,8 @@ folder. The old volumes stay as they are, for rolling back.The labels mark the new volumes as the `infradeck` project's own, so Compose uses them without a warning.5. In the new folder, start the stack:```bash- docker compose up -d --build+ docker compose pull+ docker compose up -d```Check it as in [Verify](#verify). Accounts, history and the dashboard host's server are all still there.@@ -283,7 +296,7 @@ old volumes (`docker volume rm <old-project>_postgres_data ...`) only once the n| Everyone is locked out after a few failed sign-ins | `TRUSTED_PROXIES` is empty or wrong, so all visitors share Traefik's address. | Set it as in step 2, then `docker compose up -d`. || The frontend restarts over and over | `TRUSTED_PROXIES` contains something other than IP addresses and CIDR ranges, such as the network's name. | `docker compose logs frontend` names the bad value. Put the subnet from step 2 in `.env`, then `docker compose up -d`. || `service "postgres" is not running`, although the dashboard works | The stack runs from another folder, so this folder is a different Compose project. | Find the folder with `docker compose ls` and run the command there. |-| The frontend restarts over and over with `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)` | The stack was updated without `--build`, so an old frontend image runs under the new settings. | `docker compose up -d --build`. |+| The frontend restarts over and over with `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)` | An old frontend image runs under the new settings. | `docker compose pull && docker compose up -d`. |More symptoms: [Troubleshooting](troubleshooting.md).diff --git a/docs/deployment.md b/docs/deployment.mdindex cc1ec61..eb35034 100644--- a/docs/deployment.md+++ b/docs/deployment.md@@ -17,10 +17,27 @@ The server stack works behind any reverse proxy. For some proxies there is also| Traefik | `deploy/traefik.yml` | The Traefik labels (router, entrypoint, certificate resolver) and Traefik's Docker network. Step-by-step: [Deploy Infradeck behind Traefik](deploy-traefik.md). || Anything else | nothing | Point your proxy at the dashboard port, `127.0.0.1:5173` (see [Other reverse proxies](#other-reverse-proxies)). |+The server stack runs the **published, signed images** of one release from GHCR; nothing is built on the server.+To build them from source instead (unreleased changes, a fork), add `deploy/build.yml` as well: see+[Building from source](#building-from-source).++## 0. Get a release++Clone the repository and check out the release you want to run. The compose file names the images of that+version, and the nginx settings it mounts come from the same checkout, so the two always match:++```bash+git clone https://github.com/karloks2005/infradeck.git+cd infradeck+git checkout v<version>+```++The releases are listed on the repository's **Releases** page.+## 1. Create `.env` on the serverNothing in `docker-compose.server.yml` needs editing: everything specific to you comes from-`.env`, so `git pull` can always update the file. Start from the production template:+`.env`, so checking out a newer release can always update the file. Start from the production template:```bashcp example.env .env@@ -70,9 +87,13 @@ into `.env`, delete the copy, and continue with `docker-compose.server.yml`.## 3. Start it```bash-docker compose up -d --build+docker compose pull+docker compose up -d```+`pull` downloads the images of this release (amd64 and arm64); `up` starts them. To check that the images are the+ones the release workflow built and signed, see [Checking the signatures](#checking-the-signatures).+Open `https://<your-domain>` and register your account right away. The firstaccount needs the **setup token** (`docker compose logs backend | grep "setup token"`,or the value of `ADMIN_BOOTSTRAP_TOKEN` if you set one), so a stranger who finds a@@ -112,15 +133,18 @@ Things to get right:Remote machines install the collector and the Docker proxy from container images,pinned by digest, so no code is ever downloaded from the dashboard. The-[Publish images](../.github/workflows/images.yml) workflow builds them (amd64 and-arm64), signs them with cosign and prints their digests. The version lives in one place, the `VERSION` file-at the repository root; the dashboard and the collector both read it. Tag the version it holds:+[Publish images](../.github/workflows/images.yml) workflow builds every Infradeck image (backend, frontend,+collector, Docker proxy and system helper; amd64 and arm64), signs them with cosign and prints their digests.+The version lives in the `VERSION` file at the repository root; the dashboard and the collector both read it,+and `docker-compose.server.yml` repeats it in its image tags (a test fails if the two differ). Tag the version+it holds:```bashgit tag "v$(cat VERSION)" && git push origin "v$(cat VERSION)"```-For a new version, change `VERSION` in a pull request first and tag after it is merged. The workflow refuses a+For a new version, change `VERSION` and the image tags in `docker-compose.server.yml` in one pull request+first, and tag after it is merged. The workflow refuses atag that doesn't match `VERSION`, so the images never report a different version than their tag.Release tags are protected by two repository rulesets (Settings → Rules → Rulesets): only a repository admin can@@ -129,34 +153,49 @@ switch the "never moved or deleted" ruleset off first. Prefer a new patch versio1. Open the workflow run's summary and copy the `COLLECTOR_IMAGE=…` and`DOCKER_PROXY_IMAGE=…` lines into `.env`.-2. Make the three packages **public** (GitHub → your profile → Packages → each of `infradeck/collector`,- `infradeck/docker-proxy`, `infradeck/system-helper` → *Package settings* → *Change visibility*). New+2. Make the five packages **public** (GitHub → your profile → Packages → each of `infradeck/backend`,+ `infradeck/frontend`, `infradeck/collector`, `infradeck/docker-proxy`, `infradeck/system-helper` → *Package+ settings* → *Change visibility*). Newpackages are private even when the repository is public, and machines can't pull private ones withoutcredentials. Check from any machine that isn't signed in to GHCR:`docker manifest inspect ghcr.io/<owner>/infradeck/collector:<version>`.-3. Rebuild the stack so the backend knows the images:- `docker compose up -d --build`.+3. Restart the stack so the backend knows the images: `docker compose up -d`.**Connect a server** won't generate a command until both images are set.## Updating+Back up the database first (it stays in its Docker volume, but a backup costs nothing):+```bash-git pull-docker compose up -d --build+docker compose exec -T postgres pg_dump -U infradeck -d infradeck -Fc > infradeck-before-update.dump+```++Then check out the new release and pull its images:++```bash+git fetch --tags+git checkout v<new-version>+docker compose pull+docker compose up -d```-Always rebuild **all** services (don't name individual ones). The backend,-frontend, collector, Docker proxy and system helper each have their code built-into their image, so a service that isn't rebuilt keeps running the old version.-The database schema is updated automatically when the backend starts.+Your data lives in the stack's volumes, which an update never touches; don't run `docker compose down -v`, which+deletes them. The database schema is updated automatically when the backend starts. Read the+[changelog](../CHANGELOG.md) for anything a release asks you to do.++**Upgrading from a version that built its images on the server** (before 1.0.0): add nothing; the steps above pull+the published images instead. If you had added `--build` to your own scripts, drop it, or add+`deploy/build.yml` to `COMPOSE_FILE` to keep building from source. The old local images (`infradeck-backend`,+`infradeck-frontend` and the collector ones) are no longer used; remove them with `docker image prune` once+everything runs.**Upgrading from a version where the frontend listened on port 80 inside its container:** it now listens on8080 and runs without root. The compose files, `deploy/traefik.yml` and the Caddy profile already use the newport; only a reverse proxy you configured yourself to reach `frontend:80` over a Docker network must change to-`frontend:8080`. A proxy that uses the published port (`127.0.0.1:5173`) needs no change. Update with-`docker compose up -d --build`: without `--build` the old frontend image runs under the new settings and keeps-restarting with `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)`.+`frontend:8080`. A proxy that uses the published port (`127.0.0.1:5173`) needs no change. An old frontend image+under the new settings keeps restarting with `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission+denied)`; `docker compose pull` (or `--build` with `deploy/build.yml`) replaces it.**Upgrading from a version with the Traefik labels in `docker-compose.server.yml`:** they moved to`deploy/traefik.yml`. Before you update, change `.env` to@@ -177,7 +216,43 @@ freshly generated command on each machine (or pull the new digests into itscompose project). Collectors started from `collector-compose.yml` need`docker compose -f collector-compose.yml up -d --build` after a `git pull`.-<diagram - What each step updates: git pull → source on disk; docker compose up -d --build → backend, frontend, collector, docker-proxy and gate containers rebuilt and recreated; git tag → Publish images workflow → new digests → .env and a fresh Connect a server command for remote machines; show that skipping --build leaves the old image running>+<diagram - What each step updates: git tag → Publish images workflow → signed images on GHCR and new digests; git checkout of the tag → compose file and nginx settings on disk; docker compose pull → that version's images; docker compose up -d → containers recreated on the same volumes; new digests → .env and a fresh Connect a server command for remote machines>++### Checking the signatures++Every image is signed by the release workflow with cosign, without a key: the signature is tied to the+workflow's identity on GitHub. With [cosign](https://docs.sigstore.dev/cosign/system_config/installation/)+installed, check an image before you run it:++<!-- TODO: verify against the first published release -->+```bash+cosign verify ghcr.io/karloks2005/infradeck/backend:<version> \+ --certificate-identity "https://github.com/karloks2005/Infradeck/.github/workflows/images.yml@refs/tags/v<version>" \+ --certificate-oidc-issuer https://token.actions.githubusercontent.com+```++It prints the verified signature as JSON, and fails if the image wasn't signed by that workflow for that tag.+Repeat it for `frontend`, `collector`, `docker-proxy` and `system-helper`.++### Building from source++To run a commit that isn't released, or your own changes, build the images on the server. Add+`deploy/build.yml` to `COMPOSE_FILE`, after the server file and your proxy's file:++```bash+COMPOSE_FILE=docker-compose.server.yml:deploy/traefik.yml:deploy/build.yml # with Traefik+COMPOSE_FILE=docker-compose.server.yml:deploy/build.yml # any other proxy+```++Then build and start, and repeat after every `git pull`:++```bash+docker compose up -d --build+```++Always build **all** services (don't name individual ones): a service that isn't rebuilt keeps running the old+code. `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` applies only to a build like this; the published frontend refreshes+every 3 seconds, and each user can change that in **Account → Display**.## Other reverse proxiesdiff --git a/docs/troubleshooting.md b/docs/troubleshooting.mdindex c016c65..2824484 100644--- a/docs/troubleshooting.md+++ b/docs/troubleshooting.md@@ -7,9 +7,9 @@| A new machine never shows up under **New servers detected** | `docker logs infradeck-collector` on that machine. "Could not reach …" means the address in the command isn't reachable from it (firewall, `localhost`, wrong IP/domain). || Collector log says "Backend no longer accepts server …" | The server was deleted on the dashboard; it is asking to join again, so click **Add** or **Dismiss**. || `network traefik_default declared as external, but could not be found` | You're using `deploy/traefik.yml` on a machine without that Traefik network. Set `TRAEFIK_NETWORK` in `.env` to the network Traefik is really on, or, without Traefik, drop `deploy/traefik.yml` from `COMPOSE_FILE`. |-| Frontend keeps restarting; `docker compose logs frontend` shows `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)` | Its image is older than the compose file: the stack was started without `--build` after an update. Run `docker compose up -d --build`. |-| Dashboard shows old behaviour after a code change or `git pull` | Rebuild **all** services with `docker compose up -d --build` (on a server with `COMPOSE_FILE` set in `.env`) and hard-refresh the browser (Ctrl+Shift+R). |-| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec infradeck-collector grep -c dashboard_host /app/collector.py` prints `0`. Rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). |+| Frontend keeps restarting; `docker compose logs frontend` shows `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)` | Its image is older than the compose file. Run `docker compose pull && docker compose up -d` (when building from source with `deploy/build.yml`: `docker compose up -d --build`). |+| Dashboard shows old behaviour after an update or a code change | After checking out a new release: `docker compose pull && docker compose up -d`. After a code change (development stack, or `deploy/build.yml` on a server): rebuild **all** services with `docker compose up -d --build`. Then hard-refresh the browser (Ctrl+Shift+R). |+| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec infradeck-collector grep -c dashboard_host /app/collector.py` prints `0`. Pull the current one (`docker compose pull collector && docker compose up -d collector`), or with `deploy/build.yml` rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). || A server shows only `/` and no **Storage** bars for other disks | Its collector predates per-mount reporting. Install a newer collector image (run a freshly generated command, or `docker compose -f collector-compose.yml up -d --build`). A disk mounted after the collector started appears after a collector restart. || Logged out after opening the dashboard at a new address | Login cookies belong to the address they were set on; log in again. || Login fails behind HTTPS with `SESSION_COOKIE_SECURE=true` | The cookie is only sent over HTTPS; open the `https://` address, not `http://…:5173`. |diff --git a/tests/collectors/test_version.py b/tests/collectors/test_version.pyindex 435d47e..d6963ff 100644--- a/tests/collectors/test_version.py+++ b/tests/collectors/test_version.py@@ -1,8 +1,11 @@-"""VERSION at the repository root is the only place the version is written. The dashboard and the collector both-read it, so "Collector: up to date" compares two copies of the same value."""+"""VERSION at the repository root is where the version is written. The dashboard and the collector both read it, so+"Collector: up to date" compares two copies of the same value; the server compose file repeats it in its image tags,+and a test keeps the two equal."""import refrom pathlib import Path+import yaml+from src.backend import configfrom src.collector import collector@@ -29,3 +32,19 @@ def test_a_collector_without_a_version_file_says_dev(tmp_path, monkeypatch):monkeypatch.setattr(collector, "__file__", str(folder / "collector.py"))assert collector._read_version() == "dev"+++def test_the_server_stack_runs_the_published_images_of_this_version():+ """docker-compose.server.yml pulls Infradeck's images by the version in VERSION, so a checkout of a release tag+ runs that release; deploy/build.yml builds every one of them from source instead."""+ root = VERSION_FILE.parent+ version = VERSION_FILE.read_text().strip()+ services = yaml.safe_load((root / "docker-compose.server.yml").read_text())["services"]+ built = yaml.safe_load((root / "deploy/build.yml").read_text())["services"]++ ours = {name: service["image"] for name, service in services.items() if service["image"].startswith("ghcr.io/")}+ assert set(ours) == {"backend", "agent-worker", "collector", "docker-proxy", "system-helper", "port-owners", "frontend"}+ for name, image in ours.items():+ assert re.fullmatch(rf"ghcr\.io/karloks2005/infradeck/[a-z-]+:{re.escape(version)}", image), (name, image)+ assert "build" not in services[name], name+ assert set(built) == set(ours) and all("build" in service for service in built.values())