Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
fix(security): trust X-Forwarded-For only from the configured reverse proxy
Karlo Kajba Šimanićeb77fe48c68f
diff --git a/SECURITY.md b/SECURITY.mdindex ea8bfae..37964b2 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -413,9 +413,13 @@ URL, a keyword or a value in a JSON reply, a TCP port, a TLS certificate, a DNS## Client addresses and collector installs-- **Client IPs.** nginx accepts `X-Forwarded-For` only from private or Docker network addresses (for- example Traefik), so login and registration rate limits see the real client address and it can't be- spoofed from outside.+- **Client IPs.** nginx accepts `X-Forwarded-For` only from the addresses in `TRUSTED_PROXIES`: the reverse+ proxy in front of the dashboard, whichever one it is (Traefik, Caddy, nginx, HAProxy, ...). Anyone else's+ header is ignored and the address that connected counts, so another container, a LAN or a VPN client can't+ appear to come from a new address on every request and get around the sign-in, registration, two-factor+ and enrollment limits. With `TRUSTED_PROXIES` empty (the default) nobody is trusted; behind a proxy every+ client then shares the proxy's address and limits, which is safe but strict, so set it. The value is checked+ when the container starts (IP addresses and CIDR ranges only); anything else stops the container.- **No one-line installer.** The old `docker run` install is gone. Remote machines run the composecommand **Connect a server** generates (signed, digest-pinned images), and machines set up from`collector-compose.yml` build the images locally.@@ -442,8 +446,10 @@ URL, a keyword or a value in a JSON reply, a TCP port, a TLS certificate, a DNSnetwork separation are what stop that.- **Upgrade.** Sessions written in the old format no longer work (everyone signs in once) and are deleted fromRedis at start, so the old session IDs don't linger there.-- **Still open:** the backend trusts `X-Forwarded-For` from any address that can reach it (`--forwarded-allow-ips`).- With the separation, only the nginx containers can, but narrowing it further is tracked separately.+- **The backend** trusts `X-Forwarded-For` from any address that can reach it (`--forwarded-allow-ips *`). That+ is deliberate: every request from outside reaches it through nginx, which replaces the header with the address+ it settled on, and only the stack's own containers can connect to it directly. Narrowing it to the nginx+ containers would need fixed addresses on the Docker networks, which can clash with networks already on the host.## Statistics and historydiff --git a/TESTS.md b/TESTS.mdindex 0eac091..25061b0 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,covers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **1418 test cases** from 770 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).+- **1435 test cases** from 773 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collector@@ -177,6 +177,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_stats.py`](#test_statspy) | 46 | Fleet statistics (incidents over time, availability) and the busy-hours heatmap: grouping, ranges, caching and validation. || [`test_system_checks.py`](#test_system_checkspy) | 17 | Reboot, OOM, kernel-fault, clock and pending-reboot detection, and the System row's event list. || [`test_system_helper.py`](#test_system_helperpy) | 35 | The optional system helper: kernel-log parsing, `wtmp`, crash detection, hostile input. |+| [`test_trusted_proxies.py`](#test_trusted_proxiespy) | 17 | Whose `X-Forwarded-For` nginx believes: nobody by default, only addresses and CIDR ranges from `TRUSTED_PROXIES`. |## Every test, explained@@ -1113,6 +1114,17 @@ log lines, user names), so most of these tests feed it hostile input.| `test_helper_arriving_after_the_reboot_was_noticed_updates_the_event_and_alert` | If the helper finishes after the reboot was first noticed, the event and incident are updated to the right answer. || `test_invalid_last_shutdown_values_are_dropped` | An unexpected value for "last shutdown" is discarded. |+### test_trusted_proxies.py++`docker/trusted-proxies.sh` runs when the frontend container starts and turns `TRUSTED_PROXIES` into the nginx+settings that decide whose `X-Forwarded-For` is believed. These tests run the real script.++| Test | What it checks |+|---|---|+| `test_by_default_nobody_is_trusted` *(5 inputs)* | Unset, empty or only separators: no address is trusted, so the address that connected always counts. |+| `test_each_proxy_is_trusted_and_the_header_read_from_it` | IPv4 and IPv6 addresses and ranges, with or without spaces, each become a trusted proxy, and the header is read from them. |+| `test_anything_but_addresses_stops_the_container` *(11 inputs)* | Injected nginx directives, a line break, `*`, host names, shell commands and malformed ranges stop the container; nothing is written. |+## Frontend helper testsThe chart helpers (axis ticks, gaps, histograms, the hour-by-weekday grid, donut slices, thediff --git a/compose.yml b/compose.ymlindex b230e3c..57046fa 100644--- a/compose.yml+++ b/compose.yml@@ -253,6 +253,9 @@ services:VITE_API_BASE_URL: /apicontainer_name: infradeck-frontendrestart: unless-stopped+ environment:+ # Address or network of a reverse proxy in front of this one; empty trusts nobody's X-Forwarded-For.+ TRUSTED_PROXIES: ${TRUSTED_PROXIES:-}networks: [front]# Serves the dashboard and proxies /api to the backend. Reach it directly over# LAN/VPN, or put the `https` profile's Caddy in front for a public domain.diff --git a/docker-compose.server.yml b/docker-compose.server.ymlindex 082c90b..eb689ef 100644--- a/docker-compose.server.yml+++ b/docker-compose.server.yml@@ -289,6 +289,9 @@ services:VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS: ${LIVE_GRAPH_REFRESH_INTERVAL_SECONDS:-3}container_name: infradeck-frontendrestart: unless-stopped+ environment:+ # Address or network of your reverse proxy; empty trusts nobody's X-Forwarded-For.+ TRUSTED_PROXIES: ${TRUSTED_PROXIES:-}# Serves the dashboard and proxies /api to the backend; Traefik publishes it# over HTTPS on your domain. The plain-HTTP port is bound to localhost only# (set FRONTEND_BIND=0.0.0.0 to also allow direct LAN/VPN access).diff --git a/docker/frontend.Dockerfile b/docker/frontend.Dockerfileindex e60e6e1..08ea50e 100644--- a/docker/frontend.Dockerfile+++ b/docker/frontend.Dockerfile@@ -15,6 +15,8 @@ RUN npm run buildFROM nginx:alpineCOPY docker/nginx.conf /etc/nginx/conf.d/default.conf+# Writes conf.d/trusted-proxies.conf from TRUSTED_PROXIES before nginx starts.+COPY --chmod=755 docker/trusted-proxies.sh /docker-entrypoint.d/40-trusted-proxies.shCOPY --from=build /app/dist /usr/share/nginx/htmlEXPOSE 80\ No newline at end of filediff --git a/docker/nginx.conf b/docker/nginx.confindex b9b73be..7c30371 100644--- a/docker/nginx.conf+++ b/docker/nginx.conf@@ -15,13 +15,8 @@ server {add_header Cross-Origin-Opener-Policy "same-origin" always;add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self' data:; connect-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'" always;- # Trust X-Forwarded-For only from proxies on private/Docker networks (e.g. the- # optional Caddy container), so rate limits see the real client address.- set_real_ip_from 172.16.0.0/12;- set_real_ip_from 10.0.0.0/8;- set_real_ip_from 127.0.0.1;- real_ip_header X-Forwarded-For;- real_ip_recursive on;+ # Whose X-Forwarded-For is believed comes from TRUSTED_PROXIES (docker/trusted-proxies.sh writes+ # conf.d/trusted-proxies.conf at start). By default nobody's: $remote_addr is the address that connected.# Dashboard and API share one origin: no CORS, and collectors and browsers# use the same address (http(s)://<host>/api).diff --git a/docker/nginx.strict.conf b/docker/nginx.strict.confindex a4ad700..6bab987 100644--- a/docker/nginx.strict.conf+++ b/docker/nginx.strict.conf@@ -25,12 +25,9 @@ server {# since other apps on your domain may not be HTTPS-only.add_header Strict-Transport-Security "max-age=15552000" always;- # Trust X-Forwarded-For only from the reverse proxy in front (e.g. Traefik) on private/Docker- # networks, so rate limits see the real client address. Narrow these to your proxy's network.- set_real_ip_from 172.16.0.0/12;- set_real_ip_from 127.0.0.1;- real_ip_header X-Forwarded-For;- real_ip_recursive on;+ # Whose X-Forwarded-For is believed comes from TRUSTED_PROXIES, the address or network of your reverse+ # proxy (docker/trusted-proxies.sh writes conf.d/trusted-proxies.conf at start). By default nobody's:+ # $remote_addr is the address that connected.location ~ ^/api/(collector\.py$|monitoring/(collectors|metrics/)) {return 404;diff --git a/docker/trusted-proxies.sh b/docker/trusted-proxies.shnew file mode 100644index 0000000..8231d9e--- /dev/null+++ b/docker/trusted-proxies.sh@@ -0,0 +1,35 @@+#!/bin/sh+# Decides whose X-Forwarded-For nginx believes, from TRUSTED_PROXIES: the addresses or CIDR ranges of the reverse+# proxy in front of the dashboard (Traefik, Caddy, nginx, HAProxy, a tunnel connector...), separated by commas.+# Empty (the default): nobody is trusted and every request counts as coming from the address that connected.+#+# Runs from the nginx image's /docker-entrypoint.d before nginx starts. An entry that isn't an IP address or a+# CIDR range stops the container instead of being written into the configuration. The optional argument is the+# file to write (used by the tests).+set -euf++out="${1:-/etc/nginx/conf.d/trusted-proxies.conf}"+ipv4='^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?$'+ipv6='^[0-9A-Fa-f.]*:[0-9A-Fa-f:.]*(/[0-9]{1,3})?$'++entries=""+for entry in $(printf '%s' "${TRUSTED_PROXIES:-}" | tr ',' ' '); do+ if ! printf '%s' "$entry" | grep -Eq "$ipv4|$ipv6"; then+ echo "TRUSTED_PROXIES: '$entry' is not an IP address or CIDR range" >&2+ exit 1+ fi+ entries="$entries $entry"+done++{+ echo "# Generated from TRUSTED_PROXIES when the container starts; edit that setting instead."+ for entry in $entries; do+ echo "set_real_ip_from $entry;"+ done+ if [ -n "$entries" ]; then+ echo "real_ip_header X-Forwarded-For;"+ echo "real_ip_recursive on;"+ fi+} > "$out"++echo "Trusting X-Forwarded-For from:${entries:- nobody}"diff --git a/docs/configuration.md b/docs/configuration.mdindex 1f81fb5..472215e 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -23,6 +23,7 @@ Every setting below is listed, with no values, in [`example.env`](../example.env| `INGEST_MIN_SPACING_RATIO` | `0.25` | A check-in must arrive at least this share of the server's check interval (and never sooner than 5 seconds) after the previous one, or it is refused with 429. `0` switches the limit off. || `ENROLL_RATE_LIMIT` | `30` | Enrollments with the enrollment key allowed per address per hour (twice that per key). Raise it if many collectors reach the dashboard from one address (for example behind NAT). || `ENABLE_API_DOCS` | `false` | Serves the interactive API documentation at `/api/docs`. Leave off on a public install. |+| `TRUSTED_PROXIES` | empty | The address or network of the reverse proxy in front of the dashboard, comma-separated, so rate limits count each client by its real address. Only these may set `X-Forwarded-For`. Empty trusts nobody: behind a proxy all clients then share its address. See `example.env` for how to find the value. || `SECRETS_KEY` | empty | Key that encrypts two-factor secrets. Required for two-factor sign-in and for the "require two-factor for admins" policy to be enforced. Generate with `python -m src.backend.secrets_box`; comma-separated keys allow rotation. || `ADMIN_BOOTSTRAP_TOKEN` | generated at start | Setup token for the first account. Leave empty to have one generated and printed in the backend log until the first account exists. || `FRONTEND_BIND` | `0.0.0.0` (dev), `127.0.0.1` (server file) | Interface for the dashboard's plain-HTTP port. |diff --git a/docs/deployment.md b/docs/deployment.mdindex 985167f..cdf6894 100644--- a/docs/deployment.md+++ b/docs/deployment.md@@ -25,6 +25,7 @@ Then set at least these:| `POSTGRES_PASSWORD` | *(required)* | A long random password for the bundled database. || `REDIS_PASSWORD` | *(required)* | A long random password for Redis, which holds the login sessions. Letters and digits only (`openssl rand -hex 24`). || `DASHBOARD_DOMAIN` | *(required)* | Your domain, without `https://`. Used for the Traefik router. |+| `TRUSTED_PROXIES` | empty | The subnet of your reverse proxy's Docker network, for Traefik `docker network inspect traefik_default --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}'`. Without it every client shares Traefik's address, so one person's failed sign-ins can lock everyone out for a minute. See [Reverse proxies](#reverse-proxies-and-client-addresses). || `TRAEFIK_ENTRYPOINT` | `websecure` | Your Traefik HTTPS entrypoint name. || `TRAEFIK_CERT_RESOLVER` | `letsencrypt` | Your Traefik certificate resolver name. || `TRAEFIK_NETWORK` | `traefik_default` | The Docker network Traefik is attached to. |@@ -144,8 +145,8 @@ compose project). Collectors started from `collector-compose.yml` need- **Another reverse proxy (nginx, Caddy, HAProxy, ...):** use the **server stack** (`docker-compose.server.yml`),not the development stack, and point your proxy at its dashboard port (`127.0.0.1:5173` by default; theTraefik labels are simply unused). The server stack is the one that keeps the collector endpoints off the- public side, registration closed and the session cookie HTTPS-only. Terminate HTTPS in your proxy and have it- send `X-Forwarded-For`.+ public side, registration closed and the session cookie HTTPS-only. Terminate HTTPS in your proxy, have it+ send `X-Forwarded-For`, and set `TRUSTED_PROXIES` to its address (see below).- **Caddy profile (`--profile https`, development stack): private use only.** It publishes the developmentstack's single listener, which serves the collector endpoints on the same address as the dashboard, so itis not suitable for a dashboard that is open to the internet. Use it on a LAN or behind a VPN.@@ -154,3 +155,28 @@ compose project). Collectors started from `collector-compose.yml` need(collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly. This usesthe development stack's single listener; for a public dashboard with collectorson a VPN, use the server stack above.++## Reverse proxies and client addresses++Sign-in, registration, two-factor and enrollment are rate limited per client address. Behind a reverse proxy+every request arrives from the proxy, so the dashboard needs to know which addresses belong to your proxy:+only those may tell it the real client address in `X-Forwarded-For`. Put them in `TRUSTED_PROXIES` in `.env`,+comma-separated, and restart the frontend (`docker compose up -d frontend`). It works the same for any proxy.++| Your proxy runs | Set `TRUSTED_PROXIES` to | Find it with |+|---|---|---|+| In Docker (Traefik, Caddy, nginx, ...) | The subnet of the Docker network it shares with the dashboard | `docker network inspect <network> --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}'` |+| On the host itself | The gateway of the dashboard's `front` network, where traffic from the host arrives from | `docker network inspect <project>_front --format '{{range .IPAM.Config}}{{.Gateway}}{{end}}'` |+| On another machine | That machine's address as the dashboard sees it | Your network setup |++Not sure? Sign in through the proxy and open **Account › Sessions**: the address shown there is the proxy's.+After setting it, the same page shows your own address instead.++- **Empty (default): nobody is trusted.** Every client behind the proxy shares the proxy's address and limits,+ so one person's failed sign-ins can lock everyone out for a minute. Safe, but set it.+- **Trusting a whole network** also trusts every other container on it. Prefer a network only the proxy and+ the dashboard share.+- **Several proxies in a chain** (for example a CDN in front of Traefik): list each of them; the dashboard+ takes the last address in the chain that isn't one of yours.+- The value is checked when the frontend starts: only IP addresses and CIDR ranges are accepted, and anything+ else stops the container with a message in `docker compose logs frontend`.diff --git a/example.env b/example.envindex 791d57a..2c854df 100644--- a/example.env+++ b/example.env@@ -58,6 +58,22 @@ POSTGRES_USER=# To rotate, put the new key first and keep the old one after a comma: <new>,<old>SECRETS_KEY=+# =============================================================================+# Reverse proxy (both)+# =============================================================================++# (both) The address or network of the reverse proxy in front of the dashboard (Traefik, Caddy, nginx,+# HAProxy, a tunnel connector...), so sign-in and registration limits count each client by its real address.+# Only these addresses may say who the client is (X-Forwarded-For); anyone else could pretend to be a new+# client on every request. Several values are separated by commas: 172.18.0.0/16,10.8.0.1+# - Proxy in Docker (Traefik, Caddy, ...): the subnet of the network it shares with the dashboard:+# docker network inspect <network> --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}'+# - Proxy on the host itself: the gateway of the dashboard's network (traffic from the host arrives from it):+# docker network inspect <project>_front --format '{{range .IPAM.Config}}{{.Gateway}}{{end}}'+# Not sure? Sign in through the proxy and open Account > Sessions: the address shown is the proxy's.+# Empty (default): nobody is trusted, and every client behind a proxy shares the proxy's address and limits.+TRUSTED_PROXIES=+# =============================================================================# Traefik (server)# =============================================================================diff --git a/tests/security/test_trusted_proxies.py b/tests/security/test_trusted_proxies.pynew file mode 100644index 0000000..52a223e--- /dev/null+++ b/tests/security/test_trusted_proxies.py@@ -0,0 +1,46 @@+"""docker/trusted-proxies.sh turns TRUSTED_PROXIES into the nginx settings that decide whose X-Forwarded-For is+believed. Nobody's by default; anything that isn't an IP address or CIDR range stops the container instead of+reaching the nginx configuration."""+import os+import subprocess+from pathlib import Path++import pytest++SCRIPT = Path(__file__).resolve().parents[2] / "docker" / "trusted-proxies.sh"+++def _run(tmp_path, value):+ out = tmp_path / "trusted-proxies.conf"+ env = {"PATH": os.environ["PATH"], **({} if value is None else {"TRUSTED_PROXIES": value})}+ result = subprocess.run(["sh", str(SCRIPT), str(out)], env=env, capture_output=True, text=True, timeout=10)+ return result, out.read_text() if out.exists() else None+++def _directives(conf):+ return [line for line in conf.splitlines() if line and not line.startswith("#")]++[email protected]("value", [None, "", " ", ",", " , "])+def test_by_default_nobody_is_trusted(tmp_path, value):+ result, conf = _run(tmp_path, value)++ assert result.returncode == 0 and _directives(conf) == []+ assert "nobody" in result.stdout+++def test_each_proxy_is_trusted_and_the_header_read_from_it(tmp_path):+ result, conf = _run(tmp_path, "172.18.0.0/16, 10.8.0.1,fd00::/8,::1")++ assert result.returncode == 0+ assert _directives(conf) == ["set_real_ip_from 172.18.0.0/16;", "set_real_ip_from 10.8.0.1;", "set_real_ip_from fd00::/8;",+ "set_real_ip_from ::1;", "real_ip_header X-Forwarded-For;", "real_ip_recursive on;"]++[email protected]("value", ["1.2.3.4;include /etc/passwd", "1.2.3.4;", "0.0.0.0/0;real_ip_header X-Real-IP", "*", "proxy.example.com",+ "traefik", "$(id)", "`id`", "1.2.3.4/16/8", "1.2.3.4\nset_real_ip_from 0.0.0.0/0", "10.0.0.1,evil;"])+def test_anything_but_addresses_stops_the_container(tmp_path, value):+ result, conf = _run(tmp_path, value)++ assert result.returncode != 0 and "is not an IP address or CIDR range" in result.stderr+ assert conf is None