Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
fix(collector): never follow a redirect with the token and cap the dashboard's answers (#218)
Karlo Kajba Šimanić60b5d172f1ef
diff --git a/SECURITY.md b/SECURITY.mdindex 76d4deb..0d89e90 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -322,6 +322,9 @@ check results) don't exist on the public side at all.without them. The GHCR packages must be public for hosts to pull them without credentials.- **The VPN is yours to run.** Collector traffic over the VPN listener is plain HTTP inside thetunnel. Collectors authenticate with a per-server token that is stored hashed on the dashboard.+ A collector never follows a redirect from the dashboard's address, so a sign-in page or another proxy in+ front of the API can't make it send its token somewhere else; it logs the address it was sent to and+ waits for `INFRADECK_API_URL` to be fixed. It reads at most 1 MiB of any answer.What this still depends on:diff --git a/docs/troubleshooting.md b/docs/troubleshooting.mdindex 6f1ab8b..0efb268 100644--- a/docs/troubleshooting.md+++ b/docs/troubleshooting.md@@ -27,6 +27,7 @@| Two-factor setup says it is not available | `SECRETS_KEY` is missing or invalid in `.env`. Generate one (see [Two-factor sign-in](accounts.md#two-factor-sign-in)) and restart the backend. The backend log shows a SECURITY WARNING while the key is missing. || Lost the authenticator and the recovery codes | Run `docker compose exec backend python src/backend/reset_2fa.py <email>` on the host, then sign in with the password and set it up again. || Authenticator codes stopped working after a restore or `.env` change | `SECRETS_KEY` changed or was lost, so the stored secret can't be decrypted. Restore the old key (rotation: `new,old`), or use a recovery code and the reset command. |+| A collector log says "INFRADECK_API_URL redirects to …; set it to the final address" | Something in front of the dashboard's API answers with a redirect: an `http://` address that redirects to `https://`, or a sign-in page (Authelia, Authentik …) in front of `/api`. The collector never follows it, so its token can't go anywhere else. Set `INFRADECK_API_URL` to the address it names (with `/api`), or let collectors reach `/api` without the sign-in page. || Server marked offline | It missed 3 check-ins in a row. Check the machine, its collector log and its route to the dashboard. The incident resolves on the next report. || `Set REDIS_PASSWORD in .env` when starting the stack | Add a `REDIS_PASSWORD` to `.env` (`echo "REDIS_PASSWORD=$(openssl rand -hex 24)" >> .env`). Redis holds the login sessions, so it has a password; you are signed out once after adding it. || The backend log says `AuthenticationError` or `invalid password` for Redis, or `/health` shows Redis down | `REDIS_PASSWORD` in `.env` doesn't match the password Redis was started with, or contains characters that don't fit in a URL (`@`, `:`, `/`, `%`). Use letters and digits only, and recreate both containers: `docker compose up -d --force-recreate redis backend`. |diff --git a/src/collector/collector.py b/src/collector/collector.pyindex 3e3c7ce..abffd6e 100644--- a/src/collector/collector.py+++ b/src/collector/collector.py@@ -1756,6 +1756,32 @@ def run_check(check: dict[str, Any]) -> dict[str, Any]:return {"check_id": check["id"], "time": datetime.now(timezone.utc).isoformat(), **{key: value for key, value in outcome.items() if value is not None or key == "ok"}}+class _ApiRedirect(urllib.error.URLError):+ """The dashboard's address answered with a redirect, which calls to it never follow."""+++class _NoRedirects(urllib.request.HTTPRedirectHandler):+ def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: ANN001+ target = urllib.parse.urlsplit(urllib.parse.urljoin(req.full_url, newurl))+ raise _ApiRedirect(f"INFRADECK_API_URL redirects to {target.scheme}://{target.netloc}; set it to the final address")+++# Calls to the dashboard carry this collector's token, so they never follow a redirect: a proxy in front of the API+# (a sign-in page, an http-to-https redirect) would otherwise get the token sent to wherever it points (#205).+_API_OPENER = urllib.request.build_opener(_NoRedirects())+++def _api(request: urllib.request.Request) -> dict[str, Any]:+ """Send ``request`` to the dashboard and read its JSON answer, at most MAX_BODY_BYTES of it."""+ if urllib.parse.urlsplit(request.full_url).scheme not in ("http", "https"):+ raise ValueError("INFRADECK_API_URL must start with http:// or https://")+ with _API_OPENER.open(request, timeout=15) as response:+ body = response.read(MAX_BODY_BYTES + 1)+ if len(body) > MAX_BODY_BYTES:+ raise ValueError("The dashboard's answer is too large")+ return json.loads(body.decode())++def send_check_results(api_url: str, server_id: int, token: str, results: list[dict[str, Any]], tests: list[dict[str, Any]] | None = None) -> dict[str, Any]:request = urllib.request.Request(f"{api_url.rstrip('/')}/monitoring/metrics/checks",@@ -1763,8 +1789,7 @@ def send_check_results(api_url: str, server_id: int, token: str, results: list[dheaders={"Content-Type": "application/json", "Authorization": f"Bearer {token}"},method="POST",)- with urllib.request.urlopen(request, timeout=15) as response:- return json.loads(response.read(MAX_BODY_BYTES).decode())+ return _api(request)MAX_TESTS = 5@@ -1879,8 +1904,7 @@ def send_payload(api_url: str, server_id: int, token: str, payload: dict[str, Anheaders={"Content-Type": "application/json", "Authorization": f"Bearer {token}"},method="POST",)- with urllib.request.urlopen(request, timeout=15) as response:- return json.loads(response.read().decode())+ return _api(request)def _machine_id() -> str | None:@@ -1939,8 +1963,7 @@ def enroll(api_url: str, enroll_key: str) -> tuple[int, str]:headers={"Content-Type": "application/json"},method="POST",)- with urllib.request.urlopen(request, timeout=15) as response:- data = json.loads(response.read().decode())+ data = _api(request)return int(data["server_id"]), str(data["collector_token"])@@ -1949,8 +1972,7 @@ def _post_json(url: str, payload: dict[str, Any], bearer: str | None = None) ->if bearer:headers["Authorization"] = f"Bearer {bearer}"request = urllib.request.Request(url, data=json.dumps(payload).encode(), headers=headers, method="POST")- with urllib.request.urlopen(request, timeout=15) as response:- return json.loads(response.read().decode())+ return _api(request)def _error_detail(exc: urllib.error.HTTPError) -> str:diff --git a/tests/collectors/test_collector_checks.py b/tests/collectors/test_collector_checks.pyindex 60afe4d..29cb030 100644--- a/tests/collectors/test_collector_checks.py+++ b/tests/collectors/test_collector_checks.py@@ -321,3 +321,72 @@ def test_validate_check_rejects_credentials_in_a_url():base = {"id": 1, "interval": 60, "retry_interval": 20, "timeout": 10, "accepted_codes": ["200-299"]}assert collector.validate_check({**base, "type": "http", "target": "https://user:pass@host/"}) is Noneassert collector.validate_check({**base, "type": "http", "target": "https://host/?token=x"}) is not None+++# --- calls to the dashboard (#205) ------------------------------------------------------------------------------++def _serve(handler):+ server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)+ threading.Thread(target=server.serve_forever, daemon=True).start()+ return server, f"http://127.0.0.1:{server.server_address[1]}"+++def test_a_call_to_the_dashboard_never_follows_a_redirect_with_the_token():+ seen = []++ class Elsewhere(http.server.BaseHTTPRequestHandler):+ def do_GET(self): # noqa: N802+ seen.append(self.headers.get("Authorization"))+ self.send_response(200)+ self.end_headers()++ do_POST = do_GET++ def log_message(self, *args):+ pass++ elsewhere, elsewhere_url = _serve(Elsewhere)++ class SignInRedirect(http.server.BaseHTTPRequestHandler):+ def do_POST(self): # noqa: N802+ self.send_response(302)+ self.send_header("Location", f"{elsewhere_url}/login?rd=secret-return-path")+ self.end_headers()++ def log_message(self, *args):+ pass++ proxy, proxy_url = _serve(SignInRedirect)+ try:+ with pytest.raises(urllib.error.URLError) as raised:+ collector.send_payload(proxy_url, 1, "collector-token", {})+ assert seen == [] # the token never reached the other host+ assert str(raised.value.reason) == f"INFRADECK_API_URL redirects to {elsewhere_url}; set it to the final address" # no path or query+ finally:+ proxy.shutdown()+ elsewhere.shutdown()+++def test_a_huge_answer_from_the_dashboard_is_refused(monkeypatch):+ class Huge(http.server.BaseHTTPRequestHandler):+ def do_POST(self): # noqa: N802+ body = b"[" + b"0," * collector.MAX_BODY_BYTES + b"0]"+ self.send_response(200)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ try:+ self.wfile.write(body)+ except OSError: # the collector stops reading at its limit+ pass++ def log_message(self, *args):+ pass++ server, url = _serve(Huge)+ try:+ with pytest.raises(ValueError, match="too large"):+ collector.send_payload(url, 1, "collector-token", {})+ with pytest.raises(ValueError, match="http:// or https://"):+ collector.send_payload("file:///etc", 1, "collector-token", {})+ finally:+ server.shutdown()