Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat: system health checks, VPN collector gate and security hardening
karloks2005daffcbc6f0ce
diff --git a/.github/workflows/images.yml b/.github/workflows/images.ymlnew file mode 100644index 0000000..f28955d--- /dev/null+++ b/.github/workflows/images.yml@@ -0,0 +1,63 @@+# Builds the collector, Docker proxy and system helper images and publishes them to GHCR, signed+# with cosign (keyless, using this workflow's identity). Runs when a version tag is pushed:+# git tag v0.1.0 && git push origin v0.1.0+# Remote hosts install from these images, pinned by digest, so no code is ever fetched from the+# dashboard. Third-party actions are pinned to commit hashes; update them deliberately.+name: Publish images++on:+ push:+ tags: ["v*"]++permissions:+ contents: read+ packages: write+ id-token: write # keyless signing++jobs:+ publish:+ runs-on: ubuntu-latest+ strategy:+ fail-fast: false+ matrix:+ include:+ - { name: collector, dockerfile: docker/collector.Dockerfile }+ - { name: docker-proxy, dockerfile: docker/docker-proxy.Dockerfile }+ - { name: system-helper, dockerfile: docker/system-helper.Dockerfile }+ steps:+ - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4+ - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3+ - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3+ - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3+ with:+ registry: ghcr.io+ username: ${{ github.actor }}+ password: ${{ secrets.GITHUB_TOKEN }}+ - name: Image name+ run: echo "IMAGE=ghcr.io/${GITHUB_REPOSITORY,,}/${{ matrix.name }}" >> "$GITHUB_ENV"+ - id: meta+ uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5+ with:+ images: ${{ env.IMAGE }}+ tags: |+ type=semver,pattern={{version}}+ type=semver,pattern={{major}}.{{minor}}+ - id: build+ uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6+ with:+ context: .+ file: ${{ matrix.dockerfile }}+ platforms: linux/amd64,linux/arm64+ push: true+ tags: ${{ steps.meta.outputs.tags }}+ labels: ${{ steps.meta.outputs.labels }}+ provenance: true+ sbom: true+ - uses: sigstore/cosign-installer@398d4b0eeef1380460a10c8013a76f728fb906ac # v3+ - name: Sign the image+ run: cosign sign --yes "${IMAGE}@${{ steps.build.outputs.digest }}"+ - name: Digest to put in .env+ run: |+ VAR=$(echo "${{ matrix.name }}" | tr 'a-z-' 'A-Z_')+ echo "### ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY"+ echo "\`${VAR}_IMAGE=${IMAGE}@${{ steps.build.outputs.digest }}\`" >> "$GITHUB_STEP_SUMMARY"diff --git a/.gitignore b/.gitignoreindex d2547b7..ac38ee0 100644--- a/.gitignore+++ b/.gitignore@@ -246,3 +246,6 @@ dist-ssr/*.local.vscode/*!.vscode/extensions.json++# Local SQLite development database+dev.dbdiff --git a/README.md b/README.mdindex 494229f..fa35de2 100644--- a/README.md+++ b/README.md@@ -1,4 +1,4 @@-# Vexion - The Only Dashboard You Will Ever Need+# VexionA self-hosted dashboard for monitoring all your homelab machines in one place.One machine runs the dashboard; every other machine runs a small collector that@@ -9,55 +9,92 @@ It also includes **VertexAI**, an optional monitoring-only AI assistant (Gemini)that can run a fixed set of read-only health checks and never executes commands.It's main purpose is to report errors, diagnose errors and propose optimal fixes.-<img width="2853" height="1504" alt="image" src="https://github.com/user-attachments/assets/28c49288-bec8-425b-85ab-cdb3d75e099e" />+Security comes first: the collectors never get the Docker socket, remote machines+report over a VPN to a listener the public internet can't reach, and nothing+the dashboard serves is ever executed on your hosts. See [SECURITY.md](SECURITY.md)+for what every component can touch and the limits of each protection.++<ui - Full dashboard on desktop: the "This machine" CPU/memory/disk charts at the top, the three stat tiles, and the Fleet section with two or three servers (one expanded, the rest collapsed), plus an open incident in Active incidents>## Features-- **One dashboard for every machine**: CPU, RAM, disk, load, swap, inodes,- network totals and uptime per server, plus fleet-wide averages and trend charts.-- **Container details**: click any container to see CPU, memory used/limit,- processes, network in/out, disk I/O, uptime, ports, health status and- CPU/memory history.-- **Add servers with one command and one click**: run the command from the- dashboard on a new machine, then click **Add**. No keys, `.env` edits or restarts.-- **Incidents**: alerts for restarting/unhealthy containers and for servers that- stop reporting (resolved automatically when they come back).+- **One dashboard for every machine**: CPU, RAM, load, swap, network totals and+ uptime per server, plus disk and inode usage for **every mounted filesystem**+ (root, data disks, NFS/SMB shares, ZFS pools).+- **This machine at a glance**: live CPU, memory and disk charts for the machine+ running the dashboard, with the last 30 minutes of history.+- **Container details**: containers sit in a collapsible list per server, sorted by+ name, with problems (unhealthy, restarting, stopped) shown even when it's closed.+ Open a container to see CPU, memory used/limit, processes, network and disk I/O,+ uptime, ports, health status and CPU/memory history.+- **System health per server**: reboots, OOM-killer activity, kernel fault flags,+ clock synchronisation and pending reboots, plus (with the optional system helper)+ whether the last shutdown was clean and which process the OOM killer stopped.+- **Incidents with sensible alert rules**: disk (with per-mount overrides), inodes,+ memory, swap, load per CPU core, container restart loops, failing healthchecks,+ reboots and crashes, OOM kills, kernel faults, unsynchronised clocks and servers+ that stop reporting. One incident per problem, opened only after a sustained+ breach where that makes sense and resolved automatically when it clears.+ Thresholds are editable by admins.+- **Add servers with one command and one click**: click **Connect a server**, run+ the generated command on the new machine, then click **Add**. The command carries+ a single-use token, so only an admin can add machines.+- **Built to be exposed carefully**: a public dashboard behind Traefik, with+ collectors reporting over your VPN instead. Collectors run with no Docker+ socket and no capabilities; a separate read-only proxy answers Docker queries.- **Per-server check interval** (15 s to 15 min), set from the dashboard.-- **Accounts** with Redis-backed sessions and login/registration rate limiting.+- **Works on phones**: bottom navigation, touch-sized controls and layouts that+ fit from 320 px wide screens up.+- **Accounts with roles**: the first account (created with a setup token) is the+ admin; other accounts get a read-only view. Passwords must be at least 16+ characters. Redis-backed sessions and login/registration rate limiting.- **Homie chat assistant** (optional) restricted to predefined read-only checks.-<img width="2207" height="1389" alt="image" src="https://github.com/user-attachments/assets/612afa3a-52d8-4f7a-a9e6-f83fe1903f09" />-<img width="2207" height="1389" alt="image" src="https://github.com/user-attachments/assets/1a2c1574-afc4-4f78-ab08-649b91b11d1f" />--+<ui - Three phone screenshots side by side (about 390 px wide): the dashboard top with the bottom tab bar, a server card with the Containers list open, and Admin → Alerts thresholds>## Architecture```mermaidflowchart LR- subgraph main["Main machine (docker-compose)"]+ subgraph main["Dashboard machine (docker compose)"]direction TB- proxy["frontend: nginx<br/>dashboard + /api proxy"] --> backend["backend: FastAPI"]+ proxy["frontend: nginx<br/>dashboard + /api (public side)"] --> backend["backend: FastAPI"]+ gate["collector-gate: nginx<br/>collector endpoints only"] --> backendbackend --> pg[(PostgreSQL)]backend --> redis[(Redis)]- localc["collector"] --> backend+ localc["collector"] --> gate+ localc -->|"unix socket"| dproxy["docker-proxy<br/>3 read-only requests"]+ helper["system-helper (optional)<br/>CAP_SYSLOG, no network"] -.->|"summary file"| localcend- browser["Browser"] -->|"https://your-domain"| traefik["Traefik (optional)"]+ browser["Browser"] -->|"https://your-domain"| traefik["Traefik"]traefik --> proxy- c1["collector on server A"] -->|"push metrics to /api"| traefik- c2["collector on server B"] -->|"push metrics to /api"| traefik+ c1["collector on server A"] -->|"VPN"| gate+ c2["collector on server B"] -->|"VPN"| gate```- The **frontend** container (nginx) serves the dashboard and proxies `/api` to- the backend, so the browser and all collectors use a single address:- `http(s)://<host>/` for the UI and `http(s)://<host>/api` for the API.+ the backend. On a server install this is the only thing Traefik publishes, and+ it refuses every collector endpoint.+- The **collector gate** (nginx) is the only way to reach the collector endpoints+ (register, poll, enroll, ingest). It publishes one port on your **VPN address**+ and is not attached to Traefik's network.- The **backend** (FastAPI) stores metrics in PostgreSQL and sessions and rate- limits in Redis. Its own port is published on `127.0.0.1:8000` only, for debugging.+ limits in Redis. Its own port is published on `127.0.0.1:8000` only in the+ development stack.- **Collectors** only make outgoing requests. Monitored machines need no open- ports, only a route to the dashboard address.+ ports, only a route to the collector gate.+- The **Docker proxy** is the only container that mounts `/var/run/docker.sock`.+ It has no network; the collector reaches it through a Unix socket in a shared+ volume and it answers only container list, inspect and stats.+- The optional **system helper** reads the kernel log and `wtmp`, and writes a+ sanitised summary file the collector picks up (profile `system-events`).- PostgreSQL and Redis have no host ports; they are reachable only inside theCompose network.+The development stack (`compose.yml`) keeps a single listener: everything,+including the collector endpoints, is reachable through the frontend. The server+stack splits them as described above.+## Quick start (laptop / development)Requirements: Docker with Compose v2.@@ -67,25 +104,35 @@ cp .env.example .env # then set POSTGRES_PASSWORD to a long randomdocker compose up -d --build```-1. Open http://localhost:5173 and register an account.+1. Find the one-time **setup token** in the backend log:+ `docker compose logs backend | grep "setup token"`. Open http://localhost:5173+ and register an account with it. Use a password of at least 16 characters (a+ few random words work well). The first account is the admin.2. This machine appears under **New servers detected**. Click **Add**.-3. Metrics start arriving within a few seconds.+3. Metrics start arriving within a few seconds, and the charts at the top of the+ dashboard start following this machine.Health check: http://localhost:5173/api/health```bashdocker compose logs -f backend frontend collector # follow logs+docker compose --profile system-events up -d # also run the optional system helperdocker compose down # stop (data volumes are kept)```+To try **Connect a server** on the same machine, set `COLLECTOR_IMAGE` and+`DOCKER_PROXY_IMAGE` in `.env` to `homelab-collector:latest` and+`homelab-docker-proxy:latest` (the images Compose just built). Other machines+need published images; see [Publishing the collector images](#publishing-the-collector-images).+## Deploying on a server (behind Traefik)There are two Compose files:| File | Use | Differences ||---|---|---|-| `docker-compose.yml` | Laptop / development | Dashboard on port 5173 on all interfaces; plain HTTP; optional Caddy (`--profile https`). |-| `docker-compose.server.yml` | Server behind an existing Traefik | Traefik labels and network; HTTPS-only session cookie; only the first account can register; port 5173 bound to `127.0.0.1`; no Caddy. |+| `compose.yml` | Laptop / development (the default when you run `docker compose` without `-f`) | Dashboard on port 5173 on all interfaces; plain HTTP; optional Caddy (`--profile https`). |+| `docker-compose.server.yml` | Server behind an existing Traefik | Traefik labels and network; HTTPS-only session cookie; only the first account can register; port 5173 bound to `127.0.0.1`; no Caddy; **collector endpoints removed from the public side** and served by a separate collector gate on your VPN address. |### 1. Fill in the placeholders@@ -109,9 +156,19 @@ The entrypoint and resolver names are in Traefik's static configurationcp .env.example .env```-Set at least `POSTGRES_PASSWORD`. Optionally add-`COMPOSE_FILE=docker-compose.server.yml` so plain `docker compose ...`-commands use the server file without `-f`.+Set at least `POSTGRES_PASSWORD`, and add++```bash+COMPOSE_FILE=docker-compose.server.yml+```++so plain `docker compose ...` commands use the server file. Without it, Compose+picks `compose.yml` (the development stack), and an `up` would replace the+server containers with ones that have no Traefik labels and open registration.++Don't rename or delete the Compose files in the checkout. `git pull` then can't+update them (or refuses to run), and if both `compose.yml` and+`docker-compose.yml` exist in the folder, Compose silently uses `compose.yml`.### 3. Start it@@ -119,92 +176,332 @@ commands use the server file without `-f`.docker compose -f docker-compose.server.yml up -d --build```-Open `https://<your-domain>` and register your account right away: on the-server file only the first account can be created, then registration closes.-Traefik terminates HTTPS with its own-certificates; the stack needs no certificate configuration. Collectors on any-machine use `https://<your-domain>/api`.+Open `https://<your-domain>` and register your account right away. The first+account needs the **setup token** (`docker compose logs backend | grep "setup token"`,+or the value of `ADMIN_BOOTSTRAP_TOKEN` if you set one), so a stranger who finds a+fresh deployment can't claim it; then registration closes. That first account is+the admin. Traefik terminates HTTPS with its own certificates; the stack needs no+certificate configuration.++The public address serves only the dashboard. Collectors don't use it; set up+their route in the next step.++### 4. Let remote servers report over your VPN++Collectors reach the **collector gate**, not the public domain. In `.env`:++| Setting | Example | Meaning |+|---|---|---|+| `COLLECTOR_BIND` | `10.8.0.1` | Your VPN server's address on the dashboard machine (for OpenVPN, the `tun` address). The gate's port is published on this address only. |+| `COLLECTOR_PORT` | `8081` | Port of the gate on that address. |+| `COLLECTOR_PUBLIC_URL` | `http://10.8.0.1:8081/api` | What **Connect a server** tells new machines to use. |+| `COLLECTOR_IMAGE`, `DOCKER_PROXY_IMAGE` | `ghcr.io/…@sha256:…` | The images remote machines install; see below. |++Things to get right:++- **Bind to the VPN address, not `0.0.0.0`.** Docker's published ports bypass+ `ufw` and similar firewalls, so a "deny from outside" rule does not protect them.+- If Docker starts before the VPN is up, the bind fails until the address exists.+ `restart: unless-stopped` retries, or make Docker start after the VPN (for+ example a systemd drop-in with `After=openvpn-server@<name>.service`).+- Traffic inside the VPN is plain HTTP over the encrypted tunnel; collectors+ authenticate with a per-server token that the dashboard stores only as a hash.++#### Publishing the collector images++Remote machines install the collector and the Docker proxy from container images,+pinned by digest, so no code is ever downloaded from the dashboard. The+[Publish images](.github/workflows/images.yml) workflow builds them (amd64 and+arm64), signs them with cosign and prints their digests:++```bash+git tag v0.1.0 && git push origin v0.1.0+```++1. Open the workflow run's summary and copy the `COLLECTOR_IMAGE=…` and+ `DOCKER_PROXY_IMAGE=…` lines into `.env`.+2. If the repository is private, make the three packages **public** (GitHub →+ Packages → *Package settings* → *Change visibility*); otherwise machines can't+ pull them without credentials.+3. Rebuild the stack so the backend knows the images:+ `docker compose -f docker-compose.server.yml up -d --build`.++**Connect a server** won't generate a command until both images are set.++### Updating++```bash+git pull+docker compose -f docker-compose.server.yml up -d --build+```++Always rebuild **all** services (don't name individual ones). The backend,+frontend, collector, Docker proxy and system helper each have their code built+into their image, so a service that isn't rebuilt keeps running the old version.+The database schema is updated automatically when the backend starts.++Collectors on other machines run the image digest from the command you generated.+To update them, publish a new version tag, put the new digests in `.env`, and run a+freshly generated command on each machine (or pull the new digests into its+compose project). Collectors started from `collector-compose.yml` need+`docker compose -f collector-compose.yml up -d --build` after a `git pull`.++<diagram - What each step updates: git pull → source on disk; docker compose up -d --build → backend, frontend, collector, docker-proxy and gate containers rebuilt and recreated; git tag → Publish images workflow → new digests → .env and a fresh Connect a server command for remote machines; show that skipping --build leaves the old image running>### Without Traefik-- **Caddy (automatic Let's Encrypt):** with `docker-compose.yml`, set+- **Caddy (automatic Let's Encrypt):** with `compose.yml`, set`DOMAIN`, `SESSION_COOKIE_SECURE=true` and `FRONTEND_BIND=127.0.0.1` in`.env`, open ports 80/443, and run `docker compose --profile https up -d --build`.-- **Private, over a VPN:** install Tailscale or WireGuard on the server and your- devices, run `docker compose up -d --build`, and use `http://<vpn-ip>:5173`- (collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly.+- **Private, over a VPN:** install Tailscale, WireGuard or OpenVPN on the server+ and your devices, run `docker compose up -d --build`, and use `http://<vpn-ip>:5173`+ (collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly. This uses+ the development stack's single listener; for a public dashboard with collectors+ on a VPN, use the server stack above.## Adding servers-1. On the dashboard, click **Connect a server**. Check the address field: it is- prefilled with the address you opened the dashboard at and must be reachable- from the new machine (not `localhost`).-2. Copy the command and run it on the new machine. It needs only Docker.-3. Within a few seconds the machine appears under **New servers detected**- with a pairing code (also printed in `docker logs homelab-collector`).- Click **Add**. It starts reporting on its next check, without a restart.+Adding and removing servers, changing check intervals and managing the+enrollment key are admin-only.++1. On the dashboard, click **Connect a server**. Check the address field: with+ `COLLECTOR_PUBLIC_URL` set it shows your VPN address; otherwise it is prefilled+ with the address you opened the dashboard at and must be reachable from the+ new machine (not `localhost`).+2. Click **Generate command**. This creates a single-use token that expires+ (15 minutes by default) and puts it in the command.+3. Copy the command and run it on the new machine. It needs Docker with the+ compose plugin, and a route to the address above (the VPN).+4. Within a few seconds the machine appears under **New servers detected** with a+ **JOIN TOKEN** badge and a pairing code (also printed in+ `docker logs homelab-collector`). Click **Add**. It starts reporting on its+ next check, without a restart.++<ui - The Connect a server panel: the address field, the Generate command button with its single-use / expires note, and the copyable compose command>++<ui - The New servers detected list with one pending machine showing the JOIN TOKEN badge, its pairing code and the Add / Dismiss buttons>-The machine running the dashboard is detected automatically by the stack's own-collector.+The machine running the dashboard is added the same way (its collector starts+with the stack, over the gate's internal listener). That collector has+`HOMELAB_DASHBOARD_HOST=true` set in the Compose files, which makes it the machine+shown in the charts at the top of the dashboard and first in the server list.+Collectors installed on other machines don't set it.### What the command does+It starts a small compose project named `homelab-collector` from the YAML on its+standard input (shortened here):+```bash-docker run -d --name homelab-collector --restart unless-stopped \- -e HOMELAB_API_URL=https://dashboard.example.com/api \- -v /proc:/host/proc:ro -v /:/hostfs:ro \- -v /var/run/docker.sock:/var/run/docker.sock:ro \- -v homelab-collector:/var/lib/homelab-collector \- python:3.12-alpine sh -c 'wget -qO /tmp/collector.py "$HOMELAB_API_URL/collector.py" && exec python -u /tmp/collector.py'+docker compose -p homelab-collector -f - up -d <<'EOF'+services:+ collector:+ image: ghcr.io/…/collector@sha256:…+ environment:+ HOMELAB_API_URL: http://10.8.0.1:8081/api+ HOMELAB_JOIN_TOKEN: hlj_…+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sock+ volumes:+ - /proc:/host/proc:ro+ - /:/hostfs:ro+ - collector_state:/var/lib/homelab-collector+ - docker_proxy_socket:/run/docker-proxy:ro+ cap_drop: [ALL]+ read_only: true+ docker-proxy:+ image: ghcr.io/…/docker-proxy@sha256:…+ network_mode: none+ cap_drop: [ALL]+ read_only: true+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy+EOF```| Part | Purpose ||---|---|-| `--restart unless-stopped` | Survives crashes and reboots. |-| `HOMELAB_API_URL` | Where the dashboard is. The only setting. |-| `/proc:/host/proc:ro` | Host CPU, memory, load, network and uptime. |-| `/:/hostfs:ro` | Disk usage, hostname, OS name and `/etc/machine-id`. |-| `docker.sock` | Container list, status and resource usage. |-| `homelab-collector` volume | Keeps the token after approval, so restarts don't need approval again. |-| `python:3.12-alpine` + `wget` | Official Python image. The collector (one standard-library Python file) is downloaded from the dashboard at every start, so restarting a collector updates it. |+| `HOMELAB_API_URL` | Where the collector gate is. |+| `HOMELAB_JOIN_TOKEN` | Proves an admin asked this machine to join. Single use, expires. |+| `image: …@sha256:…` | Signed images pinned by digest. No code is downloaded from the dashboard. |+| `/proc:/host/proc:ro` | Host CPU, memory, load, network, uptime, boot ID, OOM counter and kernel flags. |+| `/:/hostfs:ro` | Disk usage of every mounted filesystem, hostname, OS name and `/etc/machine-id`. |+| `docker-proxy` | The only container with the Docker socket. It answers container list, inspect (without environment variables) and stats, and nothing else. |+| `cap_drop: [ALL]`, `read_only: true` | The collector can't read other users' private files through `/hostfs`, or write outside its state volume. |+| `collector_state` volume | Keeps the token after approval, so restarts don't need approval again. |To remove a collector completely:```bash-docker rm -f homelab-collector && docker volume rm homelab-collector+docker rm -f homelab-collector homelab-docker-proxy+docker volume rm homelab-collector_collector_state homelab-collector_docker_proxy_socket```### How registration works-- A collector without credentials asks to join and checks every 5 seconds- whether it has been approved. Clicking **Add** creates the server; the- collector then receives its token and saves it.-- Servers are matched on `/etc/machine-id`, so approving a reinstalled machine- reconnects it to its existing dashboard entry and history.-- If you delete a server, its collector asks to join again. It only comes back- if you click **Add**. **Dismiss** moves it to a collapsed "Dismissed" list.+- Over the VPN a collector can't even ask to join without a valid join token. It+ then asks, checking every 5 seconds whether it has been approved. Clicking+ **Add** creates the server and uses up the token; the collector receives its own+ token and saves it. If a token is wrong, expired or used, the collector prints+ why and you copy a fresh command.+- Servers are matched on `/etc/machine-id`. If a new request matches a server you+ already have, the dashboard warns that approving it **replaces that server's+ credentials and signs out its current collector**, and asks you to confirm.+ Only do that when you are reinstalling that machine.+- If you delete a server, its collector can no longer report. On a server install+ it can't ask to join again without a new command (its token was used up). In the+ development stack it asks again and only comes back if you click **Add**.+ **Dismiss** moves a request to a collapsed "Dismissed" list.++<diagram - Sequence diagram of joining: admin clicks Generate command → join token created → command run on the new machine → collector POST /collectors/register with the token over the VPN listener → pending entry with pairing code → collector polls every 5 s → admin clicks Add (token consumed) → poll returns the token → collector saves it and starts POST /metrics/ingest; include the delete → 404 → needs a new command loop>- Requests from collectors that stopped asking disappear after 2 minutes and aredeleted after a day.Alternatives:- **Enrollment key (no approval step):** for scripted installs, generate a key- under **Connect a server → Advanced** and add `-e HOMELAB_ENROLL_KEY=<key>`- to the command. Only a hash of the key is stored; rotating or revoking it does- not affect servers that are already connected.-- **Compose instead of `docker run`:** copy the repository to the machine, set- `HOMELAB_API_URL` in a `.env` next to `collector-compose.yml`, and run- `docker compose -f collector-compose.yml up -d --build`.+ under **Connect a server → Advanced** and add `HOMELAB_ENROLL_KEY: <key>` to the+ collector's environment. Only a hash of the key is stored; rotating or revoking+ it does not affect servers that are already connected.+- **Compose from the repository:** copy the repository to the machine, set+ `HOMELAB_API_URL` (and `HOMELAB_JOIN_TOKEN`) in a `.env` next to+ `collector-compose.yml`, and run+ `docker compose -f collector-compose.yml up -d --build`. This builds the images+ locally instead of pulling them.### Networking checklist-- Every monitored machine must reach the dashboard address; the dashboard never- connects to them.-- For machines outside your LAN, use a public HTTPS domain or a VPN (Tailscale,- WireGuard). Don't expose plain HTTP to the internet: collector tokens and the- collector code itself travel over this connection.+- Every monitored machine must reach the collector gate (or the dashboard address+ in the development stack); the dashboard never connects to them.+- For machines outside your LAN, put them on a VPN (OpenVPN, WireGuard, Tailscale)+ and publish the gate on the VPN address. Don't expose plain HTTP to the+ internet: collector tokens travel over this connection.- On a LAN, give the dashboard machine a fixed IP (e.g. a DHCP reservation).+## The dashboard++- **This machine** (top): live CPU, memory and root disk charts for the machine+ running the dashboard, loaded from the last 30 minutes of stored metrics.+- **Fleet**: one card per server, with the dashboard's own machine first. Click a+ server's name to collapse its card to a single line (status, CPU, RAM, fullest+ disk, incident count); **Collapse all** / **Expand all** appear with more than+ one server. Each card shows a **Storage** bar per filesystem, colored by the+ alert thresholds (including any per-mount overrides), a **System** row and a+ **Containers** list. Both lists start closed. A filter box appears for servers+ with more than 10 containers.+- **System** row: one line of status chips (when it booted, last shutdown, clock,+ OOM kills in the last 24 hours, kernel, pending updates) that opens into the+ last 7 days of reboots, OOM kills and kernel messages. A chip that says+ *unknown* means the collector couldn't read that value on this host, not that+ anything is wrong.+- **Active incidents**: open incidents, most severe first, with the server and+ when they started.++Which cards are collapsed is remembered per browser; the System and Containers+lists always start closed.++<ui - One expanded server card: CPU/RAM/Disk/Load tiles, Storage bars for several mounts (one amber or red), the System row open with a reboot and an OOM kill in its events, and the Containers list open with an unhealthy container and one container's details expanded (stat tiles and CPU/memory charts)>++<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>++## Alerts and incidents++Rules run on every collector check-in. Each problem opens at most one incident+(per server, per rule and, for disks, per mount); while it lasts the incident is+updated instead of duplicated, and it resolves automatically once the rule+stops firing.++| Rule | Default | Severity |+|---|---|---|+| Disk usage, per mount (overridable per mount) | 85% / 95% | warning / critical |+| Inode usage, per mount | 85% / 95% | warning / critical |+| Memory usage | 90% / 95% | warning / critical |+| Swap usage | 50% | warning |+| 5-minute load per CPU core | 1.5× / 2× | warning / critical |+| Container restart loop | 3 restarts within 30 minutes | critical |+| Container healthcheck `unhealthy` | – | critical |+| Server stopped reporting | 3 missed check-ins | critical |+| Host rebooted | stays open 60 minutes | info (critical after an unclean shutdown) |+| OOM killer fired | stays open 60 minutes | warning (critical at 3 or more) |+| Kernel fault flags (machine check, bad page, oops, warning, soft lockup) | – | warning, clears on reboot |+| Clock not synchronised | 3 check-ins in a row | warning |+| Reboot required | – | info |++Host rules (disk, inodes, memory, swap, load) open an incident only after+**3 check-ins in a row** breach the threshold, so a brief spike is ignored. An+open incident stays open while the latest value is still over the warning+threshold, and becomes critical only when the critical threshold is breached for+the same number of check-ins.++<diagram - Incident lifecycle for a host rule as a timeline of check-ins: value crosses the warning threshold → after 3 check-ins in a row a WARNING incident opens → stays over critical for 3 check-ins → escalates to CRITICAL → drops under warning → RESOLVED; show a single spike that opens nothing>++Admins can change every threshold, the number of check-ins in a row, how long+reboot and OOM incidents stay open, and **per-mount disk thresholds** under+**Admin → Alerts**. A per-mount override replaces the default disk thresholds for+that mount path on every server. Changes apply from each server's next check-in+and are recorded in the audit log.++<ui - Admin → Alerts: the Incident thresholds form with Warning / Critical fields per rule, Check-ins in a row, and the container restart loop row>++<ui - Active incidents section with a CRITICAL and a WARNING incident, each showing its server and start time>++Incidents are shown on the dashboard only; there are no push notifications yet.+(`ADMIN_ALERT_WEBHOOK_URL` is used by the assistant's `daily_checkup` only.)++## System health++Collectors report what the kernel exposes, without reading any logs:++| Check | Source | Notes |+|---|---|---|+| Reboots | `/proc/sys/kernel/random/boot_id` | A new boot ID between check-ins records a **Reboot** event. |+| OOM kills | `oom_kill` in `/proc/vmstat` | A rising counter records an **OOM kill** event. |+| Kernel faults | `/proc/sys/kernel/tainted` | Only fault bits count (machine check, bad page, oops, warning, soft lockup); proprietary or out-of-tree module flags are ignored. |+| Clock sync | `adjtimex` (read-only), else systemd-timesyncd's marker | Sync state only, not a measured offset. |+| Reboot required | `/run/reboot-required` | Debian and Ubuntu only; other distros show *unknown*. |++### Optional system helper++Process names, kernel error lines and crash-versus-clean shutdown detection need+the kernel log and the login records, which the collector deliberately can't+read. A separate container does:++```bash+docker compose --profile system-events up -d # add -f docker-compose.server.yml on a server+```++It runs with a single capability (`CAP_SYSLOG`), no network and a read-only+filesystem, sees only `/var/log/wtmp` (read-only), and writes a sanitised JSON+summary that the collector reads from a shared volume. With it you also get:++- the OOM-killed process name on each **OOM kill** event,+- recent kernel error lines in the **System** row (informational, they don't open incidents),+- **Last shutdown: clean / unclean**, and a **critical** incident after an unclean reboot.++Without it those show as *unknown* and everything else keeps working. Read+[SECURITY.md](SECURITY.md) before enabling it; the helper is about 200 lines of+standard-library Python.++<ui - The System row opened on a server with the helper running: status chips on top, then a list with an OOM kill (process name) and kernel error lines>++## Accounts and roles++- The first account created is the **admin**. It needs the install's **setup+ token** (printed in the backend log at start, or `ADMIN_BOOTSTRAP_TOKEN`). Later+ accounts are regular users (and on the server file, registration closes after+ the first account).+- **Passwords must be at least 16 characters** (and at most 72 bytes). A few random+ words make a good passphrase. The rule applies to new accounts; existing+ accounts keep working with their current password.+- Everyone signed in can see all servers, containers and incidents.+- Only admins can add, approve, dismiss or delete servers, change check+ intervals, manage the enrollment key and open the **Admin** page.+- **Admin** page settings that are connected today: alert thresholds (including+ per-mount disk thresholds), how long metrics are kept (default 30 days; older+ host, container, disk and system-event samples are deleted hourly), and turning+ chat with the assistant on or off. The other sections on the page show+ placeholder data and aren't saved yet.+## Configuration### Stack (`.env` next to the Compose file)@@ -218,53 +515,85 @@ Alternatives:| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. || `SESSION_TTL_SECONDS` | `86400` | Session lifetime. || `ALLOW_REGISTRATION` | `true` (dev), `false` (server file) | Let anyone create an account. When `false`, only the first account can be created. |+| `ADMIN_BOOTSTRAP_TOKEN` | generated at start | Setup token for the first account. Leave empty to have one generated and printed in the backend log until the first account exists. || `FRONTEND_BIND` | `0.0.0.0` (dev), `127.0.0.1` (server file) | Interface for the dashboard's plain-HTTP port. || `FRONTEND_PORT` | `5173` | Dashboard port on the host. |-| `CORS_ORIGINS` | `http://localhost:5173` | Comma-separated origins, only needed if the UI is served from another origin than `/api`. |+| `CORS_ORIGINS` | `http://localhost:5173` | Comma-separated origins. Also the allow-list for the cross-site request check: a state-changing request from any other origin is refused unless its `Origin` matches the `Host` the dashboard was reached on. Add your dashboard URL if something in front rewrites `Host`. || `LOGIN_RATE_LIMIT` / `LOGIN_RATE_WINDOW_SECONDS` | `5` / `60` | Login attempts per IP and per email. || `REGISTER_RATE_LIMIT` / `REGISTER_RATE_WINDOW_SECONDS` | `3` / `3600` | Registrations per IP. |+| `COLLECTOR_BIND` / `COLLECTOR_PORT` | `127.0.0.1` / `8081` | Server file: address and port where the collector gate is published. Set the address to your VPN's. |+| `COLLECTOR_PUBLIC_URL` | empty | What **Connect a server** tells new machines to use, e.g. `http://10.8.0.1:8081/api`. |+| `COLLECTOR_IMAGE` / `DOCKER_PROXY_IMAGE` | empty | Digest-pinned images remote machines install. **Connect a server** needs both. |+| `JOIN_TOKEN_TTL_SECONDS` | `900` | How long a generated join token stays valid. |+| `COLLECTOR_ACCESS` | `open` (dev), `strict` (server file) | `strict` makes the backend answer collector endpoints only when they came through the collector gate. || `HOMELAB_ENROLL_KEY` | empty | Lets the stack's own collector skip approval. || `HOMELAB_SERVER_NAME` | host's hostname | Name shown for the stack's own machine. || `DOMAIN` | `localhost` | Domain for the optional Caddy profile. |-| `COMPOSE_FILE` | – | e.g. `docker-compose.server.yml` to make it the default on a server. |+| `COMPOSE_FILE` | – | Set to `docker-compose.server.yml` on a server so plain `docker compose` uses it. |+| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes (server file; applied when the frontend image is built). |+| `HOMELAB_CHECK_INTERVAL_SECONDS` | `60` | Initial check interval of the stack's own collector (server file); the dashboard setting overrides it. |### Collector (environment of the collector container)| Variable | Default | Description ||---|---|---|| `HOMELAB_API_URL` | `http://localhost:8000` | Dashboard API address, e.g. `https://dashboard.example.com/api`. |+| `HOMELAB_JOIN_TOKEN` | empty | Single-use token from **Connect a server**; required over the VPN listener. || `HOMELAB_ENROLL_KEY` | empty | Join immediately instead of waiting for approval. || `HOMELAB_SERVER_NAME` | host's `/etc/hostname` | Name shown on the dashboard. || `HOMELAB_STATE_DIR` | `/var/lib/homelab-collector` | Where the token is saved (mount a volume here). || `HOMELAB_MACHINE_ID` | host's `/etc/machine-id` | Identity used to recognise reinstalled machines. |+| `HOMELAB_DASHBOARD_HOST` | empty | `true` marks this machine as the one running the dashboard (set in the Compose files for the stack's own collector; leave it unset elsewhere). || `HOMELAB_SERVER_ID` / `HOMELAB_COLLECTOR_TOKEN` | empty | Explicit credentials (legacy; overrides everything else). || `HOST_PROC` / `HOST_ROOT` | `/host/proc` and `/hostfs` if mounted, else `/proc` and `/` | Where the host's `/proc` and filesystem are. |-| `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Docker API socket; without it only host metrics are sent. |+| `HOMELAB_DOCKER_HOST` | empty | Socket of the read-only Docker proxy, e.g. `/run/docker-proxy/docker.sock`. Preferred over `DOCKER_SOCKET_PATH`. |+| `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Direct Docker socket, used only when `HOMELAB_DOCKER_HOST` is unset (not recommended). Without any Docker access only host metrics are sent. |+| `HOMELAB_HELPER_FILE` | `/helper/system.json` | Where the optional system helper's summary is read from. |++The system helper has `HOMELAB_HELPER_INTERVAL_SECONDS` (default `30`, minimum+`10`); the Docker proxy listens on `HOMELAB_DOCKER_PROXY_SOCKET` and forwards to+`DOCKER_SOCKET_PATH`. Both are set in their images and rarely need changing.The check interval is set per server on the dashboard; collectors pick it upfrom the backend's response to each report.## Security-- **Collector privileges.** The Docker socket mount is effectively root access- to the host, even with `:ro` (which only protects the socket file itself), and- `/` is mounted read-only in full. The collector only reads, but whoever- controls its code controls the machine. Serve the dashboard over HTTPS or a- VPN, and consider a restricted Docker API proxy such as- `tecnativa/docker-socket-proxy` that allows only container list/inspect/stats.-- **Pending servers.** Anyone who can reach the dashboard address can make a- machine appear under **New servers detected** (rate limited per IP), but it- only reports after someone clicks **Add**. Compare the pairing code with the- collector's log if unsure.-- **Accounts.** Every logged-in account can see and add pending servers and use- the assistant (your Gemini key). With `ALLOW_REGISTRATION=false` (the default- in `docker-compose.server.yml`) only the first account can be created; keep it- that way on a public domain.-- **Secrets at rest.** Collector tokens and enrollment keys are stored only as- SHA-256 hashes; passwords are hashed with bcrypt.+[SECURITY.md](SECURITY.md) has the full picture. In short:++- **Collector privileges.** In the Compose installs the collector has no Docker+ socket, drops all capabilities and has a read-only filesystem. Only the Docker+ proxy mounts the socket; it has no network, answers three read-only requests and+ strips environment variables from what it returns. The collector still mounts `/`+ read-only; without capabilities it can't read other users' private files through it.+- **Collector endpoints are not public.** On the server stack the public listener+ returns 404 for register, poll, enroll and ingest; they exist only on the+ collector gate, bound to your VPN address.+- **No code from the dashboard.** Remote machines install signed, digest-pinned+ images. The dashboard no longer serves collector code.+- **Join tokens.** Adding a machine needs a single-use, expiring token minted by an+ admin, then an admin's click. Re-pairing a machine that matches an existing+ server needs explicit confirmation.+- **Fresh installs can't be claimed.** The first account needs a setup token.+- **Accounts.** Passwords must be at least 16 characters. Every logged-in account+ can see all monitoring data and use the assistant (your Gemini key) unless an+ admin turns chat off. Only admins can add, approve or remove servers. With+ `ALLOW_REGISTRATION=false` (the default in `docker-compose.server.yml`) only the+ first account can be created; keep it that way on a public domain.+- **Browser protections.** State-changing requests from another site are refused+ (this covers other apps on sibling subdomains), and the frontend sends a strict+ Content-Security-Policy, clickjacking protection and HSTS on the public listener.+- **Secrets at rest.** Collector tokens, join tokens, the setup token and+ enrollment keys are stored only as SHA-256 hashes; passwords are hashed with bcrypt.- **Client IPs.** nginx accepts `X-Forwarded-For` only from private/Dockernetwork addresses (e.g. Traefik), so rate limits see the real client IP and itcan't be spoofed from outside.+- **Known limits.** The one-line `docker run` install is gone; machines you install+ with `collector-compose.yml` build locally. Anyone on your VPN can reach the+ collector gate (a token and an admin's click are still needed to join).+ `src/backend/seed_user.py` creates an admin straight in the database (from+ `SEED_ADMIN_EMAIL` and `SEED_ADMIN_PASSWORD`, or an interactive prompt), skipping the+ setup token; it is meant for people who already have database access.### Homie assistant (monitoring-only)@@ -325,33 +654,58 @@ After changing frontend or collector code, rebuild the containers:```bashsource venv/bin/activate-python -m pytest+pip install -r requirements-dev.txt # the app's requirements plus pytest+pytest```+No Docker, PostgreSQL or Redis is needed; the suite takes about 15 seconds. See+[TESTS.md](TESTS.md) for how the tests work, what they don't cover, and what every+test is for.+### Database schemaTables are created on backend startup (`Base.metadata.create_all`), and newcolumns are added with `ALTER TABLE ... IF NOT EXISTS` in the startup hook in-`src/backend/main.py`. `docker/postgres/init/01_create_users.sql` initialises+`src/backend/main.py`. Each check-in stores one row in `host_metrics` (which also+carries the boot ID, OOM counter, kernel flags, clock state and last shutdown), one+per filesystem in `mount_metrics` and one per container in `container_metrics`, all+with the same timestamp. `system_events` holds reboots, OOM kills and kernel+messages, `join_tokens` the hashed single-use tokens, and per-mount disk+thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialisesthe `users` table on a fresh volume.## Project layout```text-docker-compose.yml development stack+compose.yml development stack (default for `docker compose`)docker-compose.server.yml server stack behind Traefik+docker-compose-example.yml template for the server stackcollector-compose.yml collector-only stack for monitored machines+SECURITY.md what each component can touch, and its limits+TESTS.md how to run the tests and what each one checks+requirements-dev.txt app requirements plus pytest, for running the tests+pytest.ini test settings (lets plain `pytest` find the `src` package)+.github/workflows/+ images.yml builds, signs and publishes the collector images on a version tagdocker/backend.Dockerfile FastAPI backend (uvicorn)frontend.Dockerfile Vite build served by nginxcollector.Dockerfile standalone collector image- nginx.conf serves the UI, proxies /api to the backend+ docker-proxy.Dockerfile read-only Docker API proxy image+ system-helper.Dockerfile optional kernel-log / wtmp helper image+ nginx.conf development: serves the UI, proxies /api to the backend+ nginx.strict.conf server stack: public listener, collector endpoints refused+ nginx.collector.conf server stack: collector gate (VPN and internal listeners)Caddyfile optional HTTPS front door (dev compose, --profile https)postgres/init/ SQL run on a fresh PostgreSQL volumesrc/backend/ API: auth, monitoring, enrollment, rate limits, models+ alerts.py alert engine: one incident per rule, auto-resolve+ rules.py alert rules and default thresholdsagent/collector.py metrics collector (standard library only)+ docker_proxy.py read-only Docker API proxy (standard library only)+ system_helper.py optional kernel-log / wtmp helper (standard library only)health_checks.py read-only checks available to the assistantservice.py, prompts.py Gemini assistantfrontend/ React + Tailwind dashboard (Vite)@@ -366,18 +720,27 @@ All paths are relative to `/api` when going through the frontend/nginx.|---|---|---|| `GET /health` | – | Database and Redis status. || `POST /auth/register`, `POST /auth/login`, `POST /auth/logout`, `GET /auth/me` | session cookie | Accounts and sessions. |-| `GET /auth/config` | – | Whether registration is open. |-| `GET /monitoring/overview` | session | Servers, latest metrics, containers, pending servers, active alerts. |+| `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. |+| `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |+| `GET /monitoring/servers/{id}/history` | session | CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). || `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). |-| `PATCH /monitoring/servers/{id}/interval` | session | Set check interval (15–86400 s). |-| `DELETE /monitoring/servers/{id}` | session | Remove a server and its data. |-| `POST /monitoring/pending-collectors/{id}/approve` / `…/dismiss` | session | Add or dismiss a server asking to join. |-| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | session | Show status / create or rotate / revoke the enrollment key. |-| `GET /collector.py` | – | Collector source, used by the install command. |-| `POST /monitoring/collectors/register` | – (rate limited) | Collector asks to join; returns a poll secret and pairing code. |+| `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. |+| `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |+| `DELETE /monitoring/servers/{id}` | admin | Remove a server and its data. |+| `POST /monitoring/pending-collectors/{id}/approve` / `…/dismiss` | admin | Add or dismiss a server asking to join. Approving one that matches an existing server needs `{"replace": true}`. |+| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). |+| `GET /monitoring/install-info` | admin | The collector address and images the panel uses. |+| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |+| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, alert thresholds and per-mount disk thresholds. |+| `GET /agent/config` | session | Whether chat with the assistant is enabled. |+| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. || `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. || `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |-| `POST /monitoring/metrics/ingest` | collector token | Collector reports host and container metrics. |+| `POST /monitoring/metrics/ingest` | collector token | Collector reports host, container and system metrics. |++The four collector endpoints above exist only on the collector gate in the server+stack (the public listener answers 404); the development stack serves them on the+same address as everything else.| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Homie assistant chats. || `GET /logs`, `POST /logs`, `GET /dashboard/summary` | session | Audit log and summary counts. |@@ -387,10 +750,19 @@ All paths are relative to `/api` when going through the frontend/nginx.|---|---|| A new machine never shows up under **New servers detected** | `docker logs homelab-collector` on that machine. "Could not reach …" means the address in the command isn't reachable from it (firewall, `localhost`, wrong IP/domain). || Collector log says "Backend no longer accepts server …" | The server was deleted on the dashboard; it is asking to join again, so click **Add** or **Dismiss**. |-| `network traefik_default declared as external, but could not be found` | You're using `docker-compose.server.yml` on a machine without that Traefik network. Use `docker-compose.yml` for development, or fix the network name. |-| Dashboard shows old behaviour after a code change | Rebuild with `docker compose up -d --build` and hard-refresh the browser (Ctrl+Shift+R). |+| `network traefik_default declared as external, but could not be found` | You're using `docker-compose.server.yml` on a machine without that Traefik network. Use `compose.yml` for development, or fix the network name. |+| Dashboard shows old behaviour after a code change or `git pull` | Rebuild **all** services with `docker compose up -d --build` (with `-f docker-compose.server.yml` on a server unless `COMPOSE_FILE` is set) and hard-refresh the browser (Ctrl+Shift+R). |+| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec homelab-collector grep -c dashboard_host /app/collector.py` prints `0`. Rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). |+| A server shows only `/` and no **Storage** bars for other disks | Its collector predates per-mount reporting. Install a newer collector image (run a freshly generated command, or `docker compose -f collector-compose.yml up -d --build`). A disk mounted after the collector started appears after a collector restart. || Logged out after opening the dashboard at a new address | Login cookies belong to the address they were set on; log in again. || Login fails behind HTTPS with `SESSION_COOKIE_SECURE=true` | The cookie is only sent over HTTPS; open the `https://` address, not `http://…:5173`. |+| Collector log says "Registration refused (HTTP 403): The join token is invalid, expired or already used" | The token in the command has expired, was already used, or is wrong. Click **Generate command** again and run the new command. |+| **Connect a server** says no collector images are configured | Set `COLLECTOR_IMAGE` and `DOCKER_PROXY_IMAGE` in `.env` (see [Publishing the collector images](#publishing-the-collector-images)) and rebuild. |+| The register page asks for a setup token | The install has no accounts yet. Use `docker compose logs backend \| grep "setup token"`, or `ADMIN_BOOTSTRAP_TOKEN`. |+| "Password must be at least 16 characters long" | New passwords need 16 or more characters (at most 72 bytes). A few random words work well. |+| "Cross-site request refused" (403) on a button or form | The request came from another origin. Open the dashboard at the address you normally use. If a proxy in front rewrites the `Host` header, add your dashboard URL to `CORS_ORIGINS`. |+| The **System** row shows *unknown* for last shutdown or has no kernel messages | The optional system helper isn't running. Start it with `docker compose --profile system-events up -d`. *Unknown* for reboot-required on non-Debian hosts is expected. |+| `Error response … cannot assign requested address` for the collector gate | `COLLECTOR_BIND` is set to an address that doesn't exist yet (VPN not up). Start the VPN first, or fix the address. || Server marked offline | It missed 3 check-ins in a row. Check the machine, its collector log and its route to the dashboard. The incident resolves on the next report. |## Licensediff --git a/SECURITY.md b/SECURITY.mdnew file mode 100644index 0000000..8eff3af--- /dev/null+++ b/SECURITY.md@@ -0,0 +1,166 @@+# Security++Homelab Agent is meant to be self-hosted by people who care about the machines it watches.+This page says what each component can reach, so you can decide what to run.++## Components and what they can touch++| Component | Runs as | Reaches | Notes |+|---|---|---|---|+| Collector | container | host `/proc`, the host filesystem (read-only), the Docker proxy, the backend over HTTP | No Docker socket in the compose installs. |+| Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows three read-only requests. See below. |+| System helper (optional) | container, uid 0 with **one** capability | kernel log, `/var/log/wtmp` (read-only), one output volume | Off by default. Details below. |+| Collector gate (server stack) | nginx container, no capabilities | the backend; publishes one port on your VPN address | The only way to reach collector endpoints. See below. |+| Backend / frontend / database | containers | each other | Only the frontend port is published. |++## The optional system helper++Reading the kernel log (`dmesg`) needs a privilege, and crash-vs-clean-shutdown detection needs+the login records. The collector doesn't get either: a separate helper does the privileged reads+and the collector only receives its sanitized output.++Enable it with `docker compose --profile system-events up -d` (use the same `-f` file you use for+the collector). Without it, the dashboard still reports reboots, OOM kills (as a count), kernel+fault flags, clock sync and pending reboots from `/proc`; the rest shows as "unknown".++What it can and cannot do:++- **Capabilities:** all dropped except `CAP_SYSLOG`, the minimum for reading the kernel log when+ `kernel.dmesg_restrict=1` (Ubuntu's default). It runs as uid 0 inside the container because Docker+ does not grant capabilities to non-root users; with everything else dropped that is a small surface.+- **Network:** none (`network_mode: none`). It cannot send anything anywhere.+- **Filesystem:** read-only root. The only writable path is its output volume. The only host file it+ sees is `/var/log/wtmp`, read-only, and it reads only boot and shutdown records from it+ (not logins, users or addresses).+- **Reads, never writes, the kernel log:** it uses the non-destructive read-all call.+- **No code is fetched from the dashboard.** The helper is built from this repository. Review+ `src/agent/system_helper.py` (about 200 lines, standard library only) before enabling it.++How its output is treated as untrusted:++- Kernel log lines and `wtmp` content can be influenced by an attacker (a process can name itself+ anything). The helper matches fixed patterns, reduces text to printable characters, truncates it,+ and caps the number of events. Nothing is executed or put into a command.+- The collector opens the helper's file with `O_NOFOLLOW`, caps its size, requires it to be fresh and+ from the current boot, and forwards only the expected fields.+- The backend validates everything again (event kinds, key format, timestamps, lengths, counts).++## The Docker socket and the read-only proxy++A Docker socket mount is root on the host: the `:ro` flag only protects the socket file, not the+API behind it, and anything that can reach the API can start a privileged container. So in the+compose files the collector **does not mount the socket**. A separate `docker-proxy` container does:++- It answers exactly three requests: `GET /containers/json?all=1`, `GET /containers/<id>/json` and+ `GET /containers/<id>/stats?stream=false`. Everything else (other endpoints, other methods, extra+ query parameters, encoded or padded paths) gets a 403 and is logged. Matching is literal against the+ whole request target, with no decoding.+- It cuts the answers down to the fields the collector uses. Container inspect output includes each+ container's environment variables, which are often secrets; they never leave the proxy.+- It runs with all capabilities dropped, a read-only filesystem and `no-new-privileges`, with no+ network at all. It listens on a Unix socket (mode 0660) in a volume that only the collector shares.+- It is one standard-library file, `src/agent/docker_proxy.py`, built from this repository. Review it+ before trusting it: it is the one piece that holds the socket.++A compromised collector can therefore read what it already reports (container names, states,+stats) and nothing more through Docker.++## Protecting a fresh install++- **Setup token.** While an install has no accounts, creating the first one (which becomes the admin)+ needs a setup token. Without it, whoever finds a freshly deployed dashboard first (certificate+ transparency logs publish new HTTPS hostnames within minutes) could claim it. The backend generates+ a token at every start until the first account exists and prints it in its log+ (`docker compose logs backend | grep "setup token"`); only its hash is stored. To choose your own,+ set `ADMIN_BOOTSTRAP_TOKEN`. Once an account exists the token is deleted and nothing more is needed.+- **`src/backend/seed_user.py` creates an admin without the setup token**, so it is for people who+ already have access to the database (local development, recovering a lost login). It reads the+ email and password from `SEED_ADMIN_EMAIL` / `SEED_ADMIN_PASSWORD` or prompts for them, never prints+ the password, and applies the same 16-character rule. It has no built-in account or password.+- **Registration stays off.** `ALLOW_REGISTRATION` defaults to `false` in `docker-compose.server.yml`,+ so after the first account nobody else can sign up.++## Passwords and cross-site requests++- **Passwords must be at least 16 characters** (and at most 72 bytes, bcrypt's limit; longer ones are+ refused instead of being silently truncated). A few random words make a good passphrase. The rule+ applies when an account is created; existing accounts keep working with their current password, so+ change any short ones. Failed sign-ins are rate limited per address and per email.+- **Cross-site requests are refused.** Session cookies are `SameSite=Lax`, which still lets a page on a+ sibling subdomain (every other app behind your reverse proxy counts as the same site) send+ state-changing requests with your cookie. The backend therefore rejects any `POST`, `PUT`, `PATCH` or+ `DELETE` whose `Origin` isn't this dashboard (same host and port it was reached on, or an origin in+ `CORS_ORIGINS`), including `Origin: null`, and any request the browser marks cross-site+ (`Sec-Fetch-Site`) when there's no `Origin`. Collectors and `curl` send neither header and+ authenticate with tokens, so they're unaffected. Reads aren't covered; they don't change anything,+ and CORS stops other sites from reading the answers.+ If something between the browser and nginx rewrites the `Host` header, add your dashboard's URL to+ `CORS_ORIGINS` (for example `https://dashboard.example.com`).++## HTTP security headers++The frontend's nginx sends these on every response: a Content-Security-Policy that allows only the+app's own scripts and connections (`script-src 'self'`; styles also allow inline style attributes,+which React uses), `frame-ancestors 'none'` and `X-Frame-Options: DENY` against clickjacking,+`X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, a restrictive `Permissions-Policy`,+`Cross-Origin-Opener-Policy: same-origin`, and no server version. The public configuration+(`docker/nginx.strict.conf`) also sends HSTS (180 days, no `includeSubDomains`, no preload).++If you add a feature that loads anything from another origin, extend the policy in the nginx+configuration on purpose; the browser will otherwise refuse it.++## Connecting remote servers++Internet-facing dashboards shouldn't accept unauthenticated collector traffic, so with+`docker-compose.server.yml` the collector endpoints (register, poll, enroll, ingest) don't exist on+the public side at all.++- **Two doors.** The public listener (behind Traefik or another reverse proxy) serves only the+ dashboard and its API and answers 404 for every collector endpoint. Collectors use the+ `collector-gate` container instead, which publishes one port on your **VPN address** (set+ `COLLECTOR_BIND`, e.g. your OpenVPN server's tun address) and is not attached to your reverse+ proxy's network, so other apps can't reach it. The backend refuses collector endpoints unless they+ arrived through the gate (`COLLECTOR_ACCESS=strict`), as a second layer.+- **Bind to the VPN address, not `0.0.0.0` plus a firewall rule.** Docker's published ports bypass+ `ufw` and similar firewalls, so a "deny from outside" rule does not protect them.+- **Join tokens.** The Connect a server button creates a random single-use token that expires+ (15 minutes by default) and embeds it in the install command. Over the VPN listener a collector+ can't even ask to join without a valid token, and you still click Add. A token is consumed when+ its server is approved. Tokens are stored only as SHA-256 hashes, and at most 20 can be open.+- **Re-pairing an existing machine needs explicit confirmation.** If a new request matches the+ machine ID of a server you already have, approving it replaces that server's credentials and signs+ its collector out. The dashboard warns and asks, and the API refuses without `replace: true`.+- **No code comes from the dashboard.** The dashboard no longer serves `collector.py`. Remote+ hosts install the collector and the Docker proxy from container images you publish (the "Publish+ images" workflow builds them for a version tag, signs them with cosign and prints their digests).+ Put the digests in `COLLECTOR_IMAGE` and `DOCKER_PROXY_IMAGE`; the panel won't generate a command+ without them. The GHCR packages must be public for hosts to pull them without credentials.+- **The VPN is yours to run.** Collector traffic over the VPN listener is plain HTTP inside the+ tunnel. Collectors authenticate with a per-server token that is stored hashed on the dashboard.++What this still depends on:++- The backend trusts the `X-Collector-Via` header nginx sets. That is safe only while nothing+ untrusted can reach the backend directly (port 8000 is not published in the server stack; don't+ add other containers to its network).+- Anyone on your VPN can reach the gate. With a valid join token they can request to join, and an+ admin still has to approve them; ingest additionally needs a server's token.+- Rotating a collector token means re-adding that server.++## Other exposure to be aware of++- The collector mounts the host filesystem read-only at `/hostfs` to read disk usage and a few+ files such as `/etc/os-release`.+- The collector runs with all capabilities dropped, a read-only filesystem and `no-new-privileges`.+ Without `CAP_DAC_OVERRIDE`, even as root it can't read other users' private files (SSH keys,+ `.env`) through the `/hostfs` mount; only files readable by "other" or owned by root are visible.+- The Docker proxy is reached through a Unix socket in a volume shared only with the collector. It+ has no network at all, so there is no port to attack and nothing to authenticate. Don't mount that+ volume into other containers.+- The system helper and the Docker proxy are built from this repository or from signed images on+ purpose and are never fetched from the dashboard, because they hold privileges.++## Reporting a vulnerability++Please open a private security advisory on the repository, or contact the maintainer directly,+rather than filing a public issue.diff --git a/TESTS.md b/TESTS.mdnew file mode 100644index 0000000..02efaaa--- /dev/null+++ b/TESTS.md@@ -0,0 +1,439 @@+# Tests++Vexion ships with an automated test suite. This page explains how to run it, what it+covers (and what it doesn't), and what every single test is for, so you can decide+whether a failure matters before you upgrade or change your setup.++- **297 test cases** from 174 test functions (some functions run once per input).+- **About 15 seconds** to run. No Docker, PostgreSQL, Redis, network or root access needed.+- **Safe to run next to a live install.** The tests use a throwaway in-memory database+ and fakes; they never touch your running containers or your data. (Two collector+ tests read this machine's `/proc` and root filesystem, read-only.)++## Contents++- [Running the tests](#running-the-tests)+- [How the tests work](#how-the-tests-work)+- [What the tests do not cover](#what-the-tests-do-not-cover)+- [Words used below](#words-used-below)+- [What each test file protects](#what-each-test-file-protects)+- [Every test, explained](#every-test-explained)+- [Adding a test](#adding-a-test)+- [When something fails](#when-something-fails)++## Running the tests++You need Python 3.12 on the machine where you cloned the repository (not inside the+containers; the images don't include the test tools).++```bash+python -m venv venv && source venv/bin/activate+pip install -r requirements-dev.txt # the app's requirements plus pytest+pytest+```++A healthy run ends with a line like:++```text+297 passed, 3 warnings in 14.1s+```++The warnings are harmless notices from the libraries (FastAPI deprecating the+`on_event` startup hook, and a Starlette test-client notice). Anything marked+`FAILED` or `ERROR` is a real problem.++Useful variations:++| Command | What it does |+|---|---|+| `pytest -x` | Stop at the first failure. |+| `pytest -v` | List every test case with its result. |+| `pytest tests/test_rules.py` | Run one file. |+| `pytest -k "oom"` | Run only tests whose name contains `oom`. |+| `pytest tests/test_rules.py::test_single_spike_does_not_alert` | Run one test. |+| `pytest --lf` | Re-run only the tests that failed last time. |+| `pytest --collect-only -q` | List all tests without running them. |++`python -m pytest` works the same as `pytest`. Run it from anywhere inside the repository.++**When to run them:** after `git pull` and before `docker compose up -d --build`, and+after changing code or configuration you care about. A green run means the code's own+behaviour is intact; it does not prove your deployment is healthy (see below).++## How the tests work++Each test builds just what it needs and checks one behaviour:++- **A fresh in-memory SQLite database** per test (`DATABASE_URL=sqlite://`, set by the+ tests themselves). Your PostgreSQL is never used.+- **A fake Redis.** The code is told Redis is "there", but nothing connects to one.+- **Faked system access.** Tests that read `/proc`, mounted filesystems, the Docker+ API, the kernel log or `wtmp` point the collector at temporary files or stubs, so+ results don't depend on the machine running them.+- **Endpoint functions called directly**, with a pretend logged-in user, instead of+ starting a web server. A few tests start the real Docker proxy on a temporary Unix+ socket, and one drives the real FastAPI app through its test client.+- **Hostile input on purpose.** Security-sensitive code (the Docker proxy, the system+ helper, join tokens, the cross-site check) is tested with malformed, oversized and+ malicious values, not just the happy path.++Because of this, the same tests pass on a laptop, a VPS or a Raspberry Pi.++## What the tests do not cover++Being clear about the limits matters, especially for a security-focused project:++- **The web interface** (React) has no automated tests. Build errors show up when you+ run `docker compose up -d --build`; look at the dashboard after upgrading.+- **Your deployment**: Compose files, Traefik labels, nginx configuration, your VPN,+ firewall rules and the collector gate are not exercised by the suite. They were+ checked by hand against a running stack when written, not by an automated test.+- **Real Docker, kernel and disks.** Behaviour on your real hardware (an actual OOM+ kill, a real reboot, your filesystems) is covered by fakes, not by the real thing.+- **Image publishing** (the GitHub workflow) only runs when you push a version tag.++A quick check of a live install after an upgrade:++```bash+curl -s http://localhost:5173/api/health # database and Redis status+docker compose ps # everything Up / healthy+docker compose logs --tail 20 collector # "Sent host metrics and N container records"+```++## Words used below++| Term | Meaning |+|---|---|+| **Collector** | The small program on each monitored machine that reports to the dashboard. |+| **Check-in** | One report from a collector (host numbers, filesystems, containers, system state). The code calls it *ingest*. |+| **Incident / alert** | A problem shown on the dashboard. At most one is open per problem, and it closes by itself when the problem goes away. |+| **Sustained** | A value must break its limit for several check-ins in a row (3 by default) before an incident opens, so brief spikes are ignored. |+| **Join token** | The single-use code the **Connect a server** command carries so only an admin can add a machine. |+| **Gate / VPN listener** | The separate door collectors use in the server stack; the public side refuses collector traffic. |+| **System helper** | The optional container that reads the kernel log and `wtmp` for OOM process names and crash detection. |+| **Docker proxy** | The read-only middleman that is the only thing touching the Docker socket. |+| **Strict / open mode** | `COLLECTOR_ACCESS=strict` (server stack) or `open` (development stack). |++## What each test file protects++| File | Cases | Protects |+|---|---|---|+| [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |+| [`test_agent_errors.py`](#test_agent_errorspy) | 2 | Helpful, non-leaking error messages from the AI assistant. |+| [`test_agent_service.py`](#test_agent_servicepy) | 2 | The assistant can only run the fixed read-only checks. |+| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |+| [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. |+| [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |+| [`test_container_detail.py`](#test_container_detailpy) | 5 | Per-container details and parsing of Docker's stats. |+| [`test_conversations.py`](#test_conversationspy) | 1 | Deleting one assistant conversation leaves the others alone. |+| [`test_dashboard_host.py`](#test_dashboard_hostpy) | 8 | The "this machine" logic, history charts, container list and incident counts. |+| [`test_docker_proxy.py`](#test_docker_proxypy) | 91 | The read-only Docker proxy: what it allows, refuses and hides. |+| [`test_enrollment.py`](#test_enrollmentpy) | 10 | Enrollment keys, collector credentials and offline-server detection. |+| [`test_health_checks.py`](#test_health_checkspy) | 4 | The assistant's fixed health checks and the critical-only webhook. |+| [`test_pending_collectors.py`](#test_pending_collectorspy) | 28 | Adding servers: approval, join tokens, strict mode, re-pairing safety. |+| [`test_rules.py`](#test_rulespy) | 20 | Alert thresholds: sustained breaches, per-mount disks, settings validation. |+| [`test_system_checks.py`](#test_system_checkspy) | 12 | Reboot, OOM, kernel-fault, clock and pending-reboot detection. |+| [`test_system_helper.py`](#test_system_helperpy) | 35 | The optional system helper: kernel-log parsing, `wtmp`, crash detection, hostile input. |++## Every test, explained++Each table row is one test function. Where a function is marked *(N inputs)* it runs+once per input, which is why the totals above are larger than the number of rows.++### test_admin.py++Roles and permissions. If these fail, someone might be able to do something they+shouldn't, or an admin might be locked out.++| Test | What it checks |+|---|---|+| `test_first_registered_user_is_admin_and_later_ones_are_not` | The first account created becomes the admin; the second is a regular user. |+| `test_ensure_admin_exists_promotes_oldest_user_only_when_no_admin` | Upgrading an install from before roles existed promotes the oldest account to admin, only if there is no admin yet, and running it twice changes nothing more. |+| `test_require_admin_rejects_regular_and_disabled_users` | Admin-only actions check the role in the database: regular users and disabled admins are refused with 403. |+| `test_admin_can_change_retention_within_limits` | An admin can change how long metrics are kept; an invalid value (0 days) is rejected with 400. |+| `test_purge_old_metrics_deletes_only_samples_past_retention` | The hourly cleanup deletes only host and container samples older than the retention period and keeps recent ones. |+| `test_regular_user_sees_servers_but_not_pending_collectors` | Everyone signed in sees the servers, but only an admin sees the **New servers detected** list. |+| `test_regular_user_cannot_manage_servers_or_collectors` *(9 inputs)* | A regular user is refused (403) for every admin action: viewing, rotating or revoking the enrollment key, approving or dismissing a pending server, adding or deleting a server, changing a check interval, and reading admin settings. |+| `test_enrollment_key_stops_working_when_owner_is_no_longer_admin` | An enrollment key created by someone who was later demoted stops working. |+| `test_admin_can_turn_agent_chat_off_and_on` | The admin switch for the AI assistant works, regular users can see whether it is on, and the settings response has the expected shape. |+| `test_agent_chat_is_rejected_while_disabled` | With chat turned off, the chat endpoint refuses requests (403) and the AI is never called. |++### test_agent_errors.py++| Test | What it checks |+|---|---|+| `test_overloaded_provider_error_is_explained_to_the_user` | When the AI provider reports it is overloaded, the user gets a readable explanation. |+| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the chat or logs; it is replaced with `[redacted]`. |++### test_agent_service.py++| Test | What it checks |+|---|---|+| `test_predefined_health_check_runs_and_is_audited` | When the assistant asks for one of the fixed checks (such as `server_health`), the check runs, the result is fed back to it, and both the request and the result are written to the audit log. |+| `test_arbitrary_command_is_rejected_before_execution` | If the assistant ever asks for anything outside the allow-list (such as `rm -rf /`), it is refused before anything runs. |++### test_alerts.py++The incident engine and the container rules.++| Test | What it checks |+|---|---|+| `test_raise_alert_dedupes_and_refreshes` | Raising the same problem twice keeps one incident and updates its severity and text instead of duplicating it. |+| `test_raise_alert_rejects_unknown_severity` | Only the defined severities (info, warning, critical) are accepted. |+| `test_resolved_rule_can_fire_again_as_new_alert` | After an incident resolves, the same problem coming back opens a new incident and the old one stays in the history. |+| `test_unhealthy_container_alerts_once_and_resolves` | A container whose Docker healthcheck fails opens one critical incident, however many check-ins repeat it, and it closes when the container is healthy again. |+| `test_restart_loop_alerts_once_when_count_climbs` | A container whose restart count keeps rising within the window opens a restart-loop incident that reports how many restarts happened. |+| `test_restart_loop_ignores_old_restarts_and_status_alone` | A large restart count from long ago, or a single "restarting" sample, is not treated as a loop. |+| `test_removed_container_alert_resolves_when_others_report` | If a container disappears and other containers still report, its incident resolves. |+| `test_empty_container_list_keeps_container_alerts` | If a check-in has no containers at all (Docker unreachable), existing container incidents stay open instead of being wrongly closed. |+| `test_container_sync_leaves_other_rules_alone` | Updating container incidents never touches other incidents, such as "server offline". |+| `test_overview_lists_critical_first_with_hostname_and_true_count` | The dashboard's incident list shows the most severe first, includes the server name, and reports the true total even when the list is capped. |++### test_auth_backend.py++Accounts, passwords, first-run protection and the cross-site check.++| Test | What it checks |+|---|---|+| `test_register_user_creates_record_and_hashes_password` | Registering stores the account and a bcrypt hash, never the password itself. |+| `test_authenticate_user_returns_session_for_valid_credentials` | Signing in with the right email and password returns a session. |+| `test_record_audit_log_persists_database_entries` | Audit-log entries, including assistant events with steps and conversation IDs, are saved. |+| `test_registration_closed_still_allows_first_account` | With registration switched off, the first account can still be created on an empty install; once an account exists, registration closes and a stranger is refused (403). |+| `test_registration_open_by_default` | Registration is open when `ALLOW_REGISTRATION` is true. |+| `test_fresh_install_generates_a_token_and_stores_only_its_hash` | An install with no accounts generates a setup token and keeps only its hash in the database. |+| `test_the_token_is_regenerated_at_each_start_until_an_account_exists` | Each backend start issues a new token (the old one stops working) until the first account exists. |+| `test_configured_token_is_used_and_not_printed` | If you set `ADMIN_BOOTSTRAP_TOKEN`, that value is used and nothing is generated or printed. |+| `test_first_account_is_refused_without_the_right_token` *(5 inputs)* | Missing, empty, blank, wrong or huge setup tokens are all refused (403) and no account is created. |+| `test_first_account_with_the_token_becomes_admin_and_the_token_is_removed` | The right token creates the admin, the token is deleted, and it cannot create a second account. |+| `test_without_a_stored_token_the_first_account_is_refused` | If no token exists (the backend didn't start up properly), the first account is refused rather than allowed. |+| `test_existing_installs_need_no_token_and_leftovers_are_cleaned_up` | Installs that already have accounts need no token, and a leftover token is deleted. |+| `test_passwords_under_16_characters_are_refused` *(5 inputs)* | Passwords shorter than 16 characters are rejected. |+| `test_acceptable_passwords_pass` *(5 inputs)* | Passphrases of 16 characters or more are accepted, including spaces, up to 72 bytes, and non-English characters. |+| `test_passwords_over_bcrypts_72_byte_limit_are_refused_not_truncated` *(3 inputs)* | Passwords over 72 bytes are refused with a clear message instead of being silently cut short. |+| `test_registration_enforces_the_policy_and_creates_no_account` | A too-short password at registration is rejected and no account is created. |+| `test_existing_accounts_with_short_passwords_can_still_sign_in` | The 16-character rule applies to new passwords only; older accounts keep working. |+| `test_an_overlong_password_at_login_is_just_invalid_not_an_error_with_internals` | A very long password at login gets the normal "Invalid email or password", not an internal error message. |+| `test_a_request_from_another_site_is_refused` *(4 inputs)* | State-changing requests (POST, PUT, PATCH, DELETE) from a different website are refused. |+| `test_same_host_origin_is_allowed_whatever_the_scheme` | A request from the dashboard's own address is allowed, whether or not the proxy in front hides the `https`. |+| `test_the_port_must_match_too` | The same host on a different port counts as another site; the dashboard's own port is allowed. |+| `test_a_sibling_subdomain_is_a_different_site_for_this_purpose` | Another app on a sibling subdomain, or a look-alike such as `dashboard.example.com.evil.io`, is refused. |+| `test_origins_from_the_configured_list_are_allowed` | Origins you listed in `CORS_ORIGINS` are allowed. |+| `test_odd_origins_are_refused` *(7 inputs)* | Strange origins (`null`, empty, `file://`, `javascript:`, other schemes, missing host, bare hostnames) are refused. |+| `test_without_an_origin_the_fetch_metadata_header_decides` | When a browser sends no `Origin`, its cross-site hint is used: cross-site and same-site are refused, same-origin and direct navigation are allowed. |+| `test_non_browser_clients_and_safe_methods_are_not_affected` | Collectors and `curl` (which send neither header) and read-only requests are never blocked by this check. |+| `test_the_real_app_refuses_a_cross_site_post_before_any_handler_runs` | The real application rejects a forged cross-site request with 403 before any endpoint logic runs. |++### test_collector_mounts.py++| Test | What it checks |+|---|---|+| `test_collect_mounts_keeps_real_filesystems_once` | The collector reports real disks and network shares (such as NFS), decodes mount names containing spaces, and skips `/proc`, `tmpfs`, snaps, Docker's internal mounts and filesystems mounted after it started. Usage and inode percentages are computed correctly. |++### test_container_detail.py++| Test | What it checks |+|---|---|+| `test_container_detail_returns_latest_and_chronological_history` | A container's detail view returns its latest numbers and a history sorted oldest to newest. |+| `test_container_detail_is_visible_to_regular_users` | Regular users can open container details (they are read-only). |+| `test_container_detail_returns_404_for_unknown_server` | Asking for an unknown server gives a clean 404. |+| `test_container_usage_parses_docker_stats` | Docker's raw statistics are converted correctly into CPU %, memory used and percentage (ignoring reclaimable cache), network and disk totals, and process count. |+| `test_format_ports_lists_published_bindings` | Published ports are shown as `8080->80/tcp`, once even if both IPv4 and IPv6 are bound, and unpublished ports are skipped. |++### test_conversations.py++| Test | What it checks |+|---|---|+| `test_conversation_records_are_independent_for_safe_deletion` | Deleting one assistant conversation removes only its own messages and audit entries and leaves other conversations untouched. |++### test_dashboard_host.py++| Test | What it checks |+|---|---|+| `test_dashboard_host_is_listed_first_and_flagged` | The machine running the dashboard is listed first and flagged; other servers follow, newest first. |+| `test_only_latest_claimant_is_dashboard_host_and_flag_can_be_dropped` | Only one machine can be the dashboard host at a time, and a collector that stops claiming it loses the flag. |+| `test_history_returns_recent_samples_oldest_first` | The charts' history contains only the requested window, for the right server, in time order. |+| `test_history_of_unknown_server_is_404` | History for an unknown server is a clean 404. |+| `test_collector_sends_flag_only_when_configured` | The collector marks itself as the dashboard host only when `HOMELAB_DASHBOARD_HOST` is set. |+| `test_overview_lists_only_containers_from_latest_check_in` | The container list shows what the latest check-in reported, not every container ever seen. |+| `test_overview_lists_no_containers_when_latest_check_in_had_none` | If the latest check-in had no containers, none are shown even though older samples exist. |+| `test_overview_counts_active_alerts_per_server` | Each server's incident badge counts only its own open incidents, not resolved ones. |++### test_docker_proxy.py++The read-only Docker proxy is the one component that holds the Docker socket, so it is+tested hardest (91 cases). If any of these fail, do not run the proxy until it is fixed.++| Test | What it checks |+|---|---|+| `test_the_three_read_requests_are_allowed` *(5 inputs)* | The exact requests the collector needs (list containers, inspect one, one stats snapshot) are allowed. |+| `test_only_get_is_allowed` *(8 inputs)* | Every method except GET (POST, PUT, DELETE, PATCH, HEAD, OPTIONS, lowercase `get`, CONNECT) is refused. |+| `test_everything_else_is_refused` *(63 inputs)* | Everything outside the allow-list is refused: other endpoints (`/info`, images, exec, start, kill, export, archive, logs), extra or changed query strings, path traversal, encoded or padded paths, injected line breaks, odd container IDs. |+| `test_inspect_output_drops_environment_and_everything_unneeded` | The container description is cut down to the needed fields; environment variables (often secrets), mounts, labels and host configuration never leave the proxy. |+| `test_list_and_stats_output_are_cut_down` | The same filtering applies to the container list and stats. |+| `test_unexpected_upstream_shapes_do_not_crash_the_filters` *(7 inputs)* | Strange or malformed answers from Docker don't crash the proxy. |+| `test_collector_works_through_the_proxy_and_never_sees_secrets` | End to end: the real collector gets correct container data through the proxy and no secret reaches it. |+| `test_refused_requests_never_reach_the_docker_daemon` | Attack requests are rejected with 403 and Docker is never contacted. |+| `test_upstream_failures_become_a_generic_502` | If Docker errors or is down, the caller gets a plain "Upstream error" with no details. |+| `test_oversized_upstream_answers_are_rejected` | A huge answer from Docker is rejected instead of being passed on. |+| `test_proxy_socket_is_not_world_accessible_and_replaces_a_stale_one` | The proxy's socket file is created with owner and group access only, and a leftover socket from a crash is replaced. |+| `test_collector_still_supports_the_plain_socket_path` | The collector still works without Docker access (it just reports no containers). |++### test_enrollment.py++| Test | What it checks |+|---|---|+| `test_enrollment_key_is_shown_once_and_stored_hashed` | A new enrollment key is returned once, only its hash is stored, and the key's existence can still be shown. |+| `test_enroll_creates_server_owned_by_key_owner` | Using a valid key creates a server owned by the key's owner with a fresh token. |+| `test_reenroll_same_machine_reuses_server_and_replaces_token` | Enrolling the same machine again reconnects it to its existing server and history with a new token. |+| `test_different_machines_get_separate_servers` | Different machines never share a server entry. |+| `test_rotated_or_revoked_key_is_rejected` | Old keys stop working after rotation (401), and a revoked key is gone. |+| `test_offline_alert_raised_once_and_resolved_on_ingest` | A server that misses its check-ins gets one "offline" incident, not a new one each time, and it resolves when the server reports again. |+| `test_recently_seen_server_is_not_offline` | A server that checked in recently is not marked offline. |+| `test_collector_prefers_saved_credentials_over_enrolling` | A collector that already has saved credentials uses them and doesn't enroll again. |+| `test_collector_enrolls_and_saves_when_no_credentials` | Without credentials, the collector enrolls, saves its token, and the file is readable only by its owner (mode 600). |+| `test_machine_id_read_from_host_root` | The collector identifies the machine from the host's `/etc/machine-id`. |++### test_health_checks.py++| Test | What it checks |+|---|---|+| `test_server_health_uses_python_system_interfaces` | The assistant's server check reads memory and other facts through Python system interfaces, not shell commands. |+| `test_container_health_reports_unhealthy_container` | The container check marks the result critical and names an unhealthy container. |+| `test_unknown_check_is_rejected` | Asking for any check that isn't one of the fixed ones raises an error. |+| `test_daily_checkup_alerts_only_for_critical_findings` | The daily check-up notifies the webhook only when something is critical. |++### test_pending_collectors.py++Adding servers: approval, join tokens and strict mode. These protect your install+from machines or people you didn't invite.++| Test | What it checks |+|---|---|+| `test_registration_waits_until_approved_then_hands_out_token` | A joining collector waits as *pending*; after an admin approves it, it receives its token exactly once and the request is consumed. |+| `test_approved_server_appears_in_fleet_before_first_checkin` | An approved server shows up on the dashboard immediately, before its first report. |+| `test_pending_list_shows_live_requests_only` | Requests from collectors that stopped asking disappear from **New servers detected**. |+| `test_dismissed_request_stays_listed_and_can_still_be_added` | A dismissed request moves to the dismissed list and can still be approved later. |+| `test_restarted_collector_replaces_its_earlier_request` | A restarted collector replaces its own earlier request instead of piling up duplicates, while the same machine ID from another address stays separate. |+| `test_approving_known_machine_reuses_existing_server` | Approving a request that matches an existing server needs explicit confirmation (otherwise 409), and the warning names that server; once confirmed it reuses the existing server. |+| `test_unknown_poll_secret_is_rejected` | A wrong poll secret is refused (404). |+| `test_purge_removes_abandoned_requests` | Requests abandoned for a day or more are deleted. |+| `test_collector_waits_for_approval_without_restart` | The collector keeps polling until approved, then continues without being restarted. |+| `test_collector_without_key_or_credentials_requests_approval` | With no key or saved credentials, the collector asks to join and saves its credentials afterwards. |+| `test_collector_sends_its_join_token_when_registering` | The collector sends the join token from the install command (trimmed) when it registers. |+| `test_collector_without_a_join_token_sends_none` | Without a token configured, the collector sends none. |+| `test_collector_explains_a_refused_registration` | If registration is refused, the collector prints the backend's reason, such as an expired token, instead of failing silently. |+| `test_join_token_is_minted_hashed_and_expires` | Join tokens are random, stored only as hashes, and expire. |+| `test_join_token_minting_requires_admin_and_is_capped` | Only admins can create tokens, and no more than 20 unused ones can be open at once. |+| `test_registration_with_a_valid_token_is_marked_verified` | A request that presents a valid token is shown as verified. |+| `test_vpn_listener_refuses_registration_without_a_token` | Over the VPN listener, registration without a token is refused (403) and nothing is stored. |+| `test_wrong_tokens_are_refused` *(3 inputs)* | Blank, made-up or oversized tokens are refused. |+| `test_expired_token_is_refused` | An expired token cannot be used. |+| `test_token_is_consumed_when_the_server_is_approved` | Approving the server uses up its token; it cannot register another machine. |+| `test_same_host_can_retry_with_its_token_before_approval` | The same host can retry with its token until approval without creating duplicates. |+| `test_strict_mode_hides_collector_endpoints_from_the_public_listener` | In strict mode, requests that did not come through the VPN or internal listener get 404 on every collector endpoint, whatever they claim. |+| `test_internal_listener_needs_no_token_so_the_dashboards_own_collector_can_join` | The dashboard machine's own collector, using the internal listener, can join without a token (an admin still approves it). |+| `test_open_mode_keeps_working_without_tokens` | In the development stack (open mode) joining works without tokens, as before. |+| `test_a_bad_token_is_refused_even_in_open_mode` | Even in open mode, a token that is presented but wrong is refused. |+| `test_install_info_reports_the_configured_addresses` | **Connect a server** receives the configured VPN address and image names, and whether strict mode is on. |++### test_rules.py++The alert thresholds, and the settings that control them.++| Test | What it checks |+|---|---|+| `test_host_alert_needs_sustained_breach` | An incident opens only after the threshold is broken for 3 check-ins in a row, with the right wording and severity. |+| `test_single_spike_does_not_alert` | A brief spike that drops back down opens nothing. |+| `test_open_alert_stays_open_until_value_drops_and_escalates_only_when_sustained` | An open incident stays open while the value stays over the warning level, becomes critical only when the critical level is sustained, and resolves when the value drops. |+| `test_load_is_compared_per_core` | Load is judged per CPU core, so a busy 4-core machine and a busy 64-core machine are treated fairly. |+| `test_swap_only_warns` | Swap use produces a warning, never a critical incident. |+| `test_missing_metric_does_not_alert` | A metric the collector couldn't read never raises an incident. |+| `test_saved_thresholds_apply` | Thresholds saved by an admin take effect on the next check-in. |+| `test_threshold_update_is_validated_and_audited` | Invalid thresholds (warning above critical, zero, wrong types, unknown names) are rejected with 400, and valid changes are written to the audit log. |+| `test_corrupt_saved_thresholds_fall_back_to_defaults` | If the saved settings are damaged, the defaults are used instead of crashing. |+| `test_each_mount_gets_its_own_disk_alert` | Every filesystem gets its own disk and inode incidents with its own wording and sizes. |+| `test_unmounted_disk_alert_resolves` | When a disk disappears, its incident resolves. |+| `test_mount_needs_sustained_breach_of_its_own` | A newly seen disk must also stay over the limit for 3 check-ins before alerting. |+| `test_mount_rows_are_stored_and_bad_entries_ignored` | Malformed filesystem entries from a collector are ignored. |+| `test_mount_path_with_braces_does_not_break_titles` | An unusual mount name (containing `{braces}`) cannot break incident titles. |+| `test_mount_override_replaces_global_disk_thresholds_for_that_mount_only` | A per-mount override applies to that mount and no other. |+| `test_mount_override_can_be_looser_than_global` | A per-mount limit can be higher than the default, for disks you want to run nearly full. |+| `test_mount_override_applies_to_root_for_old_collectors` | Overrides for `/` also apply to older collectors that report only the root disk. |+| `test_mount_thresholds_are_validated_audited_and_replaced` | Per-mount settings are validated (absolute paths, 1 to 100, warning below critical, no extras), audited, and replace the old set; an empty set clears them. |+| `test_corrupt_mount_thresholds_are_ignored` | Damaged per-mount settings are ignored. |+| `test_overview_shows_latest_mounts_and_disk_thresholds` | The dashboard shows the latest filesystems and the thresholds used to colour them. |++### test_system_checks.py++Reboots, OOM kills, kernel faults, clock sync and pending reboots, using only what the+kernel exposes (no helper needed).++| Test | What it checks |+|---|---|+| `test_first_check_in_records_no_events` | The very first report cannot be compared with anything, so it creates no events or incidents. |+| `test_new_boot_id_records_a_reboot_and_raises_info_alert` | A new boot ID between check-ins records a reboot event and an informational incident. |+| `test_reboot_alert_clears_after_the_event_window` | The reboot incident closes by itself once the time window has passed. |+| `test_oom_counter_rise_raises_alert_but_a_reboot_resetting_it_does_not` | A rising kernel OOM-kill counter records an event and a warning; the counter dropping to zero after a reboot is not mistaken for an OOM event. |+| `test_repeated_oom_kills_are_critical` | Repeated OOM kills in the window escalate to critical. |+| `test_kernel_fault_taint_alerts_until_reboot_but_module_taint_is_ignored` | Kernel fault flags (machine check, oops and so on) raise a warning that clears on reboot; flags for proprietary or out-of-tree modules are ignored. |+| `test_ntp_alert_needs_sustained_unsynced_checks_and_resolves` | An unsynchronised clock must persist for several check-ins before alerting, and resolves when synced. |+| `test_unknown_values_never_alert` | If the collector couldn't read a value, nothing alerts. |+| `test_malformed_values_from_a_collector_are_dropped` | Wrong-typed or oversized system values are discarded instead of stored. |+| `test_reboot_required_raises_info_alert` | A pending-reboot flag raises an informational incident and clears when gone. |+| `test_collector_reads_kernel_state_from_proc` | The collector reads boot ID, OOM counter, kernel flags and the reboot-required file from the right places. |+| `test_collector_reports_unknown_when_files_are_missing` | Missing files give *unknown*, never a false "OK". |++### test_system_helper.py++The optional system helper reads text that an attacker can influence (process names,+log lines, user names), so most of these tests feed it hostile input.++| Test | What it checks |+|---|---|+| `test_oom_kill_is_parsed_with_process_name_and_wall_time` | An OOM line from the kernel log becomes an event with the killed process name and the correct wall-clock time. |+| `test_cgroup_oom_is_recognised` | OOM kills inside a container's memory limit are recognised too. |+| `test_kernel_errors_are_kept_but_info_lines_are_not` | Only error-level kernel messages are kept; ordinary informational lines are dropped. |+| `test_hostile_log_content_is_neutralised` | Escape codes, control characters and over-long names in log lines are made harmless and truncated. |+| `test_malformed_and_untimestamped_lines_are_ignored` | Garbage and unparseable lines are skipped. |+| `test_output_is_capped_to_the_newest_events` | Only the newest few events are published, so output size stays bounded. |+| `test_utmp_layout_is_384_bytes` | The `wtmp` record layout matches the real 384-byte format. |+| `test_clean_shutdown_is_detected` | A shutdown record before the latest boot means the previous shutdown was clean. |+| `test_missing_shutdown_record_is_unclean_when_shutdowns_were_recorded_before` | No shutdown record, on a system that normally records them, means a crash or power loss. |+| `test_never_recorded_shutdowns_are_unknown_not_unclean` | A system that never records shutdowns is reported as *unknown*, not as crashing on every reboot. |+| `test_unknown_when_wtmp_is_missing_short_or_out_of_date` | A missing, too-short or outdated `wtmp` gives *unknown*. |+| `test_truncated_or_symlinked_wtmp_is_safe` | A half-written `wtmp` is handled, and a symlinked one is refused. |+| `test_snapshot_is_written_atomically_with_a_fixed_schema` | The summary file is replaced atomically, with a fixed layout and fixed file permissions (readable by others, writable only by the helper), so a reader never sees a half-written file. |+| `test_collector_uses_a_fresh_file_from_the_current_boot` | The collector accepts the helper's file when it is recent and from the current boot. |+| `test_collector_ignores_stale_or_foreign_files` *(3 inputs)* | Old files, files from another boot, and files with a bad timestamp are ignored. |+| `test_collector_refuses_missing_symlinked_oversized_and_garbage_files` | Missing, symlinked, oversized, malformed or wrongly shaped files are refused. |+| `test_helper_events_are_stored_once_even_when_republished` | The helper republishes the same events every run, but each is stored once. |+| `test_with_the_helper_the_oom_counter_does_not_add_a_second_event` | With the helper running, one OOM kill produces one event, not two. |+| `test_malformed_helper_events_are_dropped` *(10 inputs)* | The backend rejects events with a disallowed kind, bad key, impossible time, wrong types or old age. |+| `test_helper_event_count_and_text_are_bounded` | At most 50 events per check-in are accepted, and long text is trimmed. |+| `test_unclean_shutdown_makes_the_reboot_critical` | A reboot after an unclean shutdown opens a critical incident whose text says so. |+| `test_clean_shutdown_keeps_the_reboot_informational` | A reboot after a clean shutdown stays informational. |+| `test_helper_arriving_after_the_reboot_was_noticed_updates_the_event_and_alert` | If the helper finishes after the reboot was first noticed, the event and incident are updated to the right answer. |+| `test_invalid_last_shutdown_values_are_dropped` | An unexpected value for "last shutdown" is discarded. |++## Adding a test++1. Put it in the file for the area it belongs to, or a new `tests/test_<area>.py`+ (files must start with `test_`, functions with `test_`).+2. Follow the existing style: build a fresh in-memory database, call the code+ directly, assert on the result. Look at a similar test and copy its setup.+3. For anything that handles outside input, add the awkward cases too (empty,+ oversized, wrongly typed, malicious). `pytest.mark.parametrize` runs one test+ over many inputs.+4. Run `pytest -x`, then the whole suite.+5. Add a row to this page for the new test.++## When something fails++| What you see | What to do |+|---|---|+| `ModuleNotFoundError: No module named 'pytest'` | Install the test tools: `pip install -r requirements-dev.txt`. |+| `ModuleNotFoundError: No module named 'src'` | Run `pytest` from inside the repository folder. |+| `ModuleNotFoundError` for another package | Install the app's requirements: `pip install -r requirements.txt` (or the dev file above). |+| One test `FAILED` | Read the assertion line (`assert ... == ...`) printed under it: the left side is what the code did, the right side is what was expected. Use `pytest -x -v` to focus on the first failure. |+| Failures after you edited code | The test name tells you what rule you broke. If the rule changed on purpose, update the test and its row on this page. |+| Failures after `git pull` with no changes of your own | Please open an issue with the output of `pytest -x`. |diff --git a/collector-compose.yml b/collector-compose.ymlindex ea911f8..ec2837e 100644--- a/collector-compose.yml+++ b/collector-compose.yml@@ -17,11 +17,66 @@ services:HOMELAB_API_URL: ${HOMELAB_API_URL:?Set HOMELAB_API_URL to your backend address}HOMELAB_ENROLL_KEY: ${HOMELAB_ENROLL_KEY:-}HOMELAB_SERVER_NAME: ${HOMELAB_SERVER_NAME:-}+ # Docker data comes through the read-only proxy; the collector has no Docker socket.+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sockvolumes:- /proc:/host/proc:ro- /:/hostfs:ro- - /var/run/docker.sock:/var/run/docker.sock:ro- collector_state:/var/lib/homelab-collector+ # Summary from the optional system helper (read-only; empty unless that profile runs).+ - system_events:/helper:ro+ - docker_proxy_socket:/run/docker-proxy:ro+ # Read-only host view, no capabilities: as root it then can't read other users' private+ # files (SSH keys, .env) even though / is mounted.+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:true++ # The only container that mounts the Docker socket (a mount flag can't make it read-only).+ # It answers three read requests and nothing else. It has no network at all: the collector+ # reaches it through a Unix socket in a volume only the two of them share. See SECURITY.md.+ docker-proxy:+ build:+ context: .+ dockerfile: docker/docker-proxy.Dockerfile+ image: homelab-docker-proxy:latest+ container_name: homelab-docker-proxy+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:true+ network_mode: none+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy++ # Optional: adds OOM-killer process names, kernel errors and crash-vs-clean-shutdown detection.+ # Enable with: docker compose --profile system-events up -d+ # Its only privilege is CAP_SYSLOG (to read the kernel log); it has no network, a read-only+ # filesystem, and sees the host only through the read-only wtmp file. See SECURITY.md.+ system-helper:+ profiles: ["system-events"]+ build:+ context: .+ dockerfile: docker/system-helper.Dockerfile+ image: homelab-system-helper:latest+ container_name: homelab-system-helper+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ cap_add: [SYSLOG]+ network_mode: none+ read_only: true+ security_opt:+ - no-new-privileges:true+ volumes:+ - /var/log/wtmp:/host/wtmp:ro+ - system_events:/outvolumes:collector_state:+ system_events:+ docker_proxy_socket:diff --git a/compose.yml b/compose.ymlindex a58017e..664c138 100644--- a/compose.yml+++ b/compose.yml@@ -46,6 +46,10 @@ services:SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-false}ALLOW_REGISTRATION: ${ALLOW_REGISTRATION:-true}CORS_ORIGINS: ${CORS_ORIGINS:-http://localhost:5173}+ # Connect a server panel: images remote hosts install, and the address they reach this dashboard at.+ COLLECTOR_IMAGE: ${COLLECTOR_IMAGE:-}+ DOCKER_PROXY_IMAGE: ${DOCKER_PROXY_IMAGE:-}+ COLLECTOR_PUBLIC_URL: ${COLLECTOR_PUBLIC_URL:-}# Browsers and remote collectors reach the API through the frontend at /api;# the direct port is only published on this machine for debugging.ports:@@ -77,15 +81,68 @@ services:# on the dashboard. An optional enrollment key skips that approval.HOMELAB_ENROLL_KEY: ${HOMELAB_ENROLL_KEY:-}HOMELAB_SERVER_NAME: ${HOMELAB_SERVER_NAME:-}+ # Docker data comes through the read-only proxy; the collector has no Docker socket.+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sockvolumes:- /proc:/host/proc:ro- /:/hostfs:ro- - /var/run/docker.sock:/var/run/docker.sock:ro- collector_state:/var/lib/homelab-collector+ # Summary from the optional system helper (read-only; empty unless that profile runs).+ - system_events:/helper:ro+ - docker_proxy_socket:/run/docker-proxy:ro+ # Read-only host view, no capabilities: as root it then can't read other users' private+ # files (SSH keys, .env) even though / is mounted.+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:truedepends_on:backend:condition: service_healthy+ # The only container that mounts the Docker socket (a mount flag can't make it read-only).+ # It answers three read requests and nothing else. It has no network at all: the collector+ # reaches it through a Unix socket in a volume only the two of them share. See SECURITY.md.+ docker-proxy:+ build:+ context: .+ dockerfile: docker/docker-proxy.Dockerfile+ image: homelab-docker-proxy:latest+ container_name: homelab-docker-proxy+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:true+ network_mode: none+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy++ # Optional: adds OOM-killer process names, kernel errors and crash-vs-clean-shutdown detection.+ # Enable with: docker compose --profile system-events up -d+ # Its only privilege is CAP_SYSLOG (to read the kernel log); it has no network, a read-only+ # filesystem, and sees the host only through the read-only wtmp file. See SECURITY.md.+ system-helper:+ profiles: ["system-events"]+ build:+ context: .+ dockerfile: docker/system-helper.Dockerfile+ image: homelab-system-helper:latest+ container_name: homelab-system-helper+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ cap_add: [SYSLOG]+ network_mode: none+ read_only: true+ security_opt:+ - no-new-privileges:true+ volumes:+ - /var/log/wtmp:/host/wtmp:ro+ - system_events:/out+frontend:build:context: .@@ -125,5 +182,7 @@ volumes:postgres_data:redis_data:collector_state:+ system_events:+ docker_proxy_socket:caddy_data:caddy_config:diff --git a/dev.db b/dev.dbdeleted file mode 100644index 8ea871b..0000000Binary files a/dev.db and /dev/null differdiff --git a/docker-compose-example.yml b/docker-compose-example.ymlindex 94a88f4..612b67d 100644--- a/docker-compose-example.yml+++ b/docker-compose-example.yml@@ -60,6 +60,12 @@ services:REGISTER_RATE_LIMIT: ${REGISTER_RATE_LIMIT:-3}REGISTER_RATE_WINDOW_SECONDS: ${REGISTER_RATE_WINDOW_SECONDS:-3600}ADMIN_ALERT_WEBHOOK_URL: ${ADMIN_ALERT_WEBHOOK_URL:-}+ # Collector endpoints answer only through the collector-gate container (VPN or internal).+ COLLECTOR_ACCESS: strict+ # Shown in the Connect a server panel: your VPN address for the gate, and pinned images.+ COLLECTOR_PUBLIC_URL: ${COLLECTOR_PUBLIC_URL:-}+ COLLECTOR_IMAGE: ${COLLECTOR_IMAGE:-}+ DOCKER_PROXY_IMAGE: ${DOCKER_PROXY_IMAGE:-}# No host port: nginx (frontend) reaches the backend over the Compose network,# and browsers/collectors use https://<domain>/api. Avoids clashing with other# services on the server. Debug with: docker compose exec backend ...@@ -83,23 +89,102 @@ services:container_name: homelab-collectorrestart: unless-stoppedenvironment:- HOMELAB_API_URL: http://backend:8000+ # Through the gate's internal listener (not published); the public listener refuses collectors.+ HOMELAB_API_URL: http://collector-gate:8080/api# Marks this collector as the machine running the dashboard (the charts at the top).HOMELAB_DASHBOARD_HOST: "true"# Nothing to configure: this machine shows up under "New servers detected"# on the dashboard. An optional enrollment key skips that approval.HOMELAB_ENROLL_KEY: ${HOMELAB_ENROLL_KEY:-}HOMELAB_SERVER_NAME: ${HOMELAB_SERVER_NAME:-}+ # Docker data comes through the read-only proxy; the collector has no Docker socket.+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sockHOMELAB_CHECK_INTERVAL_SECONDS: ${HOMELAB_CHECK_INTERVAL_SECONDS:-60}HOMELAB_MACHINE_ID: ${HOMELAB_MACHINE_ID:-}volumes:- /proc:/host/proc:ro- /:/hostfs:ro- - /var/run/docker.sock:/var/run/docker.sock:ro- collector_state:/var/lib/homelab-collector+ # Summary from the optional system helper (read-only; empty unless that profile runs).+ - system_events:/helper:ro+ - docker_proxy_socket:/run/docker-proxy:ro+ # Read-only host view, no capabilities: as root it then can't read other users' private+ # files (SSH keys, .env) even though / is mounted.+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:truedepends_on:backend:condition: service_healthy+ collector-gate:+ condition: service_started++ # The only way to reach the collector endpoints. Not on the reverse proxy's network, so other+ # apps can't register collectors. Publish its VPN listener on your VPN address only: Docker's+ # published ports bypass ufw and similar firewalls, so never bind it to 0.0.0.0 and rely on those.+ collector-gate:+ image: nginx:alpine+ container_name: homelab-collector-gate+ restart: unless-stopped+ ports:+ - "${COLLECTOR_BIND:-127.0.0.1}:${COLLECTOR_PORT:-8081}:8081"+ volumes:+ - ./docker/nginx.collector.conf:/etc/nginx/conf.d/default.conf:ro+ read_only: true+ tmpfs:+ - /var/cache/nginx+ - /var/run+ cap_drop: [ALL]+ cap_add: [CHOWN, SETGID, SETUID]+ security_opt:+ - no-new-privileges:true+ depends_on:+ backend:+ condition: service_healthy++ # The only container that mounts the Docker socket (a mount flag can't make it read-only).+ # It answers three read requests and nothing else. It has no network at all: the collector+ # reaches it through a Unix socket in a volume only the two of them share. See SECURITY.md.+ docker-proxy:+ build:+ context: .+ dockerfile: docker/docker-proxy.Dockerfile+ image: homelab-docker-proxy:latest+ container_name: homelab-docker-proxy+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:true+ network_mode: none+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy++ # Optional: adds OOM-killer process names, kernel errors and crash-vs-clean-shutdown detection.+ # Enable with: docker compose --profile system-events up -d+ # Its only privilege is CAP_SYSLOG (to read the kernel log); it has no network, a read-only+ # filesystem, and sees the host only through the read-only wtmp file. See SECURITY.md.+ system-helper:+ profiles: ["system-events"]+ build:+ context: .+ dockerfile: docker/system-helper.Dockerfile+ image: homelab-system-helper:latest+ container_name: homelab-system-helper+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ cap_add: [SYSLOG]+ network_mode: none+ read_only: true+ security_opt:+ - no-new-privileges:true+ volumes:+ - /var/log/wtmp:/host/wtmp:ro+ - system_events:/outfrontend:build:@@ -115,6 +200,9 @@ services:# (set FRONTEND_BIND=0.0.0.0 to also allow direct LAN/VPN access).ports:- "${FRONTEND_BIND:-127.0.0.1}:${FRONTEND_PORT:-5173}:80"+ volumes:+ # Public listener only: collector endpoints are refused here (see the collector-gate service).+ - ./docker/nginx.strict.conf:/etc/nginx/conf.d/default.conf:ronetworks:- default- traefik@@ -138,3 +226,5 @@ volumes:postgres_data:redis_data:collector_state:+ system_events:+ docker_proxy_socket:diff --git a/docker-compose.server.yml b/docker-compose.server.ymlindex 46606c7..927181c 100644--- a/docker-compose.server.yml+++ b/docker-compose.server.yml@@ -57,6 +57,12 @@ services:REGISTER_RATE_LIMIT: ${REGISTER_RATE_LIMIT:-3}REGISTER_RATE_WINDOW_SECONDS: ${REGISTER_RATE_WINDOW_SECONDS:-3600}ADMIN_ALERT_WEBHOOK_URL: ${ADMIN_ALERT_WEBHOOK_URL:-}+ # Collector endpoints answer only through the collector-gate container (VPN or internal).+ COLLECTOR_ACCESS: strict+ # Shown in the Connect a server panel: your VPN address for the gate, and pinned images.+ COLLECTOR_PUBLIC_URL: ${COLLECTOR_PUBLIC_URL:-}+ COLLECTOR_IMAGE: ${COLLECTOR_IMAGE:-}+ DOCKER_PROXY_IMAGE: ${DOCKER_PROXY_IMAGE:-}# No host port: nginx (frontend) reaches the backend over the Compose network,# and browsers/collectors use https://<domain>/api. Avoids clashing with other# services on the server. Debug with: docker compose exec backend ...@@ -80,23 +86,102 @@ services:container_name: homelab-collectorrestart: unless-stoppedenvironment:- HOMELAB_API_URL: http://backend:8000+ # Through the gate's internal listener (not published); the public listener refuses collectors.+ HOMELAB_API_URL: http://collector-gate:8080/api# Marks this collector as the machine running the dashboard (the charts at the top).HOMELAB_DASHBOARD_HOST: "true"# Nothing to configure: this machine shows up under "New servers detected"# on the dashboard. An optional enrollment key skips that approval.HOMELAB_ENROLL_KEY: ${HOMELAB_ENROLL_KEY:-}HOMELAB_SERVER_NAME: ${HOMELAB_SERVER_NAME:-}+ # Docker data comes through the read-only proxy; the collector has no Docker socket.+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sockHOMELAB_CHECK_INTERVAL_SECONDS: ${HOMELAB_CHECK_INTERVAL_SECONDS:-60}HOMELAB_MACHINE_ID: ${HOMELAB_MACHINE_ID:-}volumes:- /proc:/host/proc:ro- /:/hostfs:ro- - /var/run/docker.sock:/var/run/docker.sock:ro- collector_state:/var/lib/homelab-collector+ # Summary from the optional system helper (read-only; empty unless that profile runs).+ - system_events:/helper:ro+ - docker_proxy_socket:/run/docker-proxy:ro+ # Read-only host view, no capabilities: as root it then can't read other users' private+ # files (SSH keys, .env) even though / is mounted.+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:truedepends_on:backend:condition: service_healthy+ collector-gate:+ condition: service_started++ # The only way to reach the collector endpoints. Not on the reverse proxy's network, so other+ # apps can't register collectors. Publish its VPN listener on your VPN address only: Docker's+ # published ports bypass ufw and similar firewalls, so never bind it to 0.0.0.0 and rely on those.+ collector-gate:+ image: nginx:alpine+ container_name: homelab-collector-gate+ restart: unless-stopped+ ports:+ - "${COLLECTOR_BIND:-127.0.0.1}:${COLLECTOR_PORT:-8081}:8081"+ volumes:+ - ./docker/nginx.collector.conf:/etc/nginx/conf.d/default.conf:ro+ read_only: true+ tmpfs:+ - /var/cache/nginx+ - /var/run+ cap_drop: [ALL]+ cap_add: [CHOWN, SETGID, SETUID]+ security_opt:+ - no-new-privileges:true+ depends_on:+ backend:+ condition: service_healthy++ # The only container that mounts the Docker socket (a mount flag can't make it read-only).+ # It answers three read requests and nothing else. It has no network at all: the collector+ # reaches it through a Unix socket in a volume only the two of them share. See SECURITY.md.+ docker-proxy:+ build:+ context: .+ dockerfile: docker/docker-proxy.Dockerfile+ image: homelab-docker-proxy:latest+ container_name: homelab-docker-proxy+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ read_only: true+ security_opt:+ - no-new-privileges:true+ network_mode: none+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy++ # Optional: adds OOM-killer process names, kernel errors and crash-vs-clean-shutdown detection.+ # Enable with: docker compose --profile system-events up -d+ # Its only privilege is CAP_SYSLOG (to read the kernel log); it has no network, a read-only+ # filesystem, and sees the host only through the read-only wtmp file. See SECURITY.md.+ system-helper:+ profiles: ["system-events"]+ build:+ context: .+ dockerfile: docker/system-helper.Dockerfile+ image: homelab-system-helper:latest+ container_name: homelab-system-helper+ restart: unless-stopped+ user: "0:0"+ cap_drop: [ALL]+ cap_add: [SYSLOG]+ network_mode: none+ read_only: true+ security_opt:+ - no-new-privileges:true+ volumes:+ - /var/log/wtmp:/host/wtmp:ro+ - system_events:/outfrontend:build:@@ -112,6 +197,9 @@ services:# (set FRONTEND_BIND=0.0.0.0 to also allow direct LAN/VPN access).ports:- "${FRONTEND_BIND:-127.0.0.1}:${FRONTEND_PORT:-5173}:80"+ volumes:+ # Public listener only: collector endpoints are refused here (see the collector-gate service).+ - ./docker/nginx.strict.conf:/etc/nginx/conf.d/default.conf:ronetworks:- default- traefik@@ -135,3 +223,5 @@ volumes:postgres_data:redis_data:collector_state:+ system_events:+ docker_proxy_socket:diff --git a/docker/docker-proxy.Dockerfile b/docker/docker-proxy.Dockerfilenew file mode 100644index 0000000..8efc01f--- /dev/null+++ b/docker/docker-proxy.Dockerfile@@ -0,0 +1,13 @@+# Read-only Docker API proxy: the only container that mounts the Docker socket.+# Standard library only. See src/agent/docker_proxy.py for exactly what it allows.+FROM python:3.12-alpine++WORKDIR /app+COPY src/agent/docker_proxy.py ./docker_proxy.py++ENV DOCKER_SOCKET_PATH=/var/run/docker.sock \+ HOMELAB_DOCKER_PROXY_SOCKET=/run/docker-proxy/docker.sock \+ PYTHONUNBUFFERED=1++USER 0:0+CMD ["python", "/app/docker_proxy.py"]diff --git a/docker/nginx.collector.conf b/docker/nginx.collector.confnew file mode 100644index 0000000..863f8a9--- /dev/null+++ b/docker/nginx.collector.conf@@ -0,0 +1,51 @@+# Collector gateway: the only way to reach the collector endpoints when COLLECTOR_ACCESS=strict.+# It serves nothing else and is not attached to the reverse proxy's network.+#+# :8081 vpn publish this on your VPN address only (COLLECTOR_BIND), never on 0.0.0.0.+# Registration needs a join token from the dashboard's Connect a server button.+# :8080 internal for containers on this compose network (the dashboard machine's own+# collector). Never publish it: it accepts registration without a token.+#+# nginx sets X-Collector-Via itself; the backend trusts it only because nothing but this gateway+# and the frontend can reach the backend. No real_ip trust: a VPN client must not be able to spoof+# its address to dodge the per-IP rate limits.++server {+ listen 8081;+ server_name _;++ location ~ ^/api/monitoring/(collectors/register|collectors/register/poll|collectors/enroll|metrics/ingest)$ {+ rewrite ^/api/(.*)$ /$1 break;+ proxy_pass http://backend:8000;+ proxy_http_version 1.1;+ proxy_set_header Host $host;+ proxy_set_header X-Forwarded-For $remote_addr;+ proxy_set_header X-Collector-Via vpn;+ proxy_read_timeout 60s;+ client_max_body_size 2m;+ }++ location / {+ return 404;+ }+}++server {+ listen 8080;+ server_name _;++ location ~ ^/api/monitoring/(collectors/register|collectors/register/poll|collectors/enroll|metrics/ingest)$ {+ rewrite ^/api/(.*)$ /$1 break;+ proxy_pass http://backend:8000;+ proxy_http_version 1.1;+ proxy_set_header Host $host;+ proxy_set_header X-Forwarded-For $remote_addr;+ proxy_set_header X-Collector-Via internal;+ proxy_read_timeout 60s;+ client_max_body_size 2m;+ }++ location / {+ return 404;+ }+}diff --git a/docker/nginx.conf b/docker/nginx.confindex af0a037..04ba626 100644--- a/docker/nginx.conf+++ b/docker/nginx.conf@@ -5,6 +5,16 @@ server {root /usr/share/nginx/html;index index.html;+ # Security headers. The CSP matches what the built app loads: its own scripts and styles, no+ # external resources; style attributes (React inline styles) need 'unsafe-inline' for styles only.+ server_tokens off;+ add_header X-Content-Type-Options "nosniff" always;+ add_header X-Frame-Options "DENY" always;+ add_header Referrer-Policy "no-referrer" always;+ add_header Permissions-Policy "camera=(), microphone=(), geolocation=(), payment=(), usb=()" always;+ add_header Cross-Origin-Opener-Policy "same-origin" always;+ add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self' data:; connect-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'" always;+# Trust X-Forwarded-For only from proxies on private/Docker networks (e.g. the# optional Caddy container), so rate limits see the real client address.set_real_ip_from 172.16.0.0/12;@@ -18,7 +28,7 @@ server {location /api/ {proxy_pass http://backend:8000/;proxy_http_version 1.1;- proxy_set_header Host $host;+ proxy_set_header Host $http_host;proxy_set_header X-Forwarded-For $remote_addr;proxy_set_header X-Forwarded-Host $host;proxy_read_timeout 120s;diff --git a/docker/nginx.strict.conf b/docker/nginx.strict.confnew file mode 100644index 0000000..f5a1179--- /dev/null+++ b/docker/nginx.strict.conf@@ -0,0 +1,55 @@+# Public listener for installs that expose the dashboard through a reverse proxy (e.g. Traefik)+# while collectors report over a VPN. Mount over /etc/nginx/conf.d/default.conf in the frontend+# container (docker-compose.server.yml does this) and set COLLECTOR_ACCESS=strict on the backend.+#+# Every collector endpoint is refused here. Collectors use the separate collector-gate container+# (docker/nginx.collector.conf), which is not attached to the reverse proxy's network.++server {+ listen 80;+ server_name _;++ root /usr/share/nginx/html;+ index index.html;++ # Security headers. The CSP matches what the built app loads: its own scripts and styles, no+ # external resources; style attributes (React inline styles) need 'unsafe-inline' for styles only.+ server_tokens off;+ add_header X-Content-Type-Options "nosniff" always;+ add_header X-Frame-Options "DENY" always;+ add_header Referrer-Policy "no-referrer" always;+ add_header Permissions-Policy "camera=(), microphone=(), geolocation=(), payment=(), usb=()" always;+ add_header Cross-Origin-Opener-Policy "same-origin" always;+ add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self' data:; connect-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'" always;+ # Served over HTTPS by the reverse proxy; browsers ignore this on plain HTTP. No includeSubDomains,+ # since other apps on your domain may not be HTTPS-only.+ add_header Strict-Transport-Security "max-age=15552000" always;++ # Trust X-Forwarded-For only from the reverse proxy in front (e.g. Traefik) on private/Docker+ # networks, so rate limits see the real client address. Narrow these to your proxy's network.+ set_real_ip_from 172.16.0.0/12;+ set_real_ip_from 127.0.0.1;+ real_ip_header X-Forwarded-For;+ real_ip_recursive on;++ location ~ ^/api/(collector\.py$|monitoring/(collectors|metrics/ingest)) {+ return 404;+ }++ location /api/ {+ proxy_pass http://backend:8000/;+ proxy_http_version 1.1;+ proxy_set_header Host $http_host;+ proxy_set_header X-Forwarded-For $remote_addr;+ proxy_set_header X-Forwarded-Host $host;+ # The backend answers collector endpoints only when this says vpn or internal, so a client+ # must never be able to set it on the public listener.+ proxy_set_header X-Collector-Via "";+ proxy_read_timeout 120s;+ client_max_body_size 5m;+ }++ location / {+ try_files $uri $uri/ /index.html;+ }+}diff --git a/docker/system-helper.Dockerfile b/docker/system-helper.Dockerfilenew file mode 100644index 0000000..240454b--- /dev/null+++ b/docker/system-helper.Dockerfile@@ -0,0 +1,15 @@+# Optional system helper: reads the kernel log and wtmp, publishes a sanitized summary.+# Standard library only. Run it with only CAP_SYSLOG, no network and a read-only filesystem.+FROM python:3.12-alpine++WORKDIR /app+COPY src/agent/system_helper.py ./system_helper.py++ENV HOST_PROC=/proc \+ HOMELAB_WTMP_PATH=/host/wtmp \+ HOMELAB_HELPER_OUTPUT=/out/system.json \+ PYTHONUNBUFFERED=1++# Nothing to write to except the shared output volume.+USER 0:0+CMD ["python", "/app/system_helper.py"]diff --git a/example.env b/example.envindex 99d030b..10ed39f 100644--- a/example.env+++ b/example.env@@ -105,6 +105,21 @@ HOMELAB_MACHINE_ID=FRONTEND_BIND=127.0.0.1FRONTEND_PORT=5173+# --- Connecting remote servers over a VPN (docker-compose.server.yml) ---+# Collectors never use the public address. They report to the collector gate, which you+# publish on your VPN address only (for OpenVPN, the server's tun address, e.g. 10.8.0.1).+# Docker's published ports bypass ufw and similar firewalls, so bind to the VPN address+# instead of 0.0.0.0 and relying on a firewall rule.+COLLECTOR_BIND=127.0.0.1+COLLECTOR_PORT=8081+# What the "Connect a server" panel tells new machines to use, e.g. http://10.8.0.1:8081/api+COLLECTOR_PUBLIC_URL=+# Images remote hosts install, pinned by digest. Push a version tag (git tag v0.1.0 &&+# git push origin v0.1.0); the "Publish images" workflow prints these lines in its summary.+# The GHCR packages must be public for hosts to pull them without credentials.+COLLECTOR_IMAGE=+DOCKER_PROXY_IMAGE=+# How often live graphs refresh, in seconds (minimum 1). This is baked in when# the image is built, so rebuild after changing it:# docker compose up -d --build frontenddiff --git a/pytest.ini b/pytest.ininew file mode 100644index 0000000..1f83768--- /dev/null+++ b/pytest.ini@@ -0,0 +1,4 @@+[pytest]+# Lets the bare `pytest` command import the `src` package; `python -m pytest` works without this.+pythonpath = .+testpaths = testsdiff --git a/requirements-dev.txt b/requirements-dev.txtnew file mode 100644index 0000000..d719d18--- /dev/null+++ b/requirements-dev.txt@@ -0,0 +1,3 @@+# Everything needed to run the test suite: pip install -r requirements-dev.txt+-r requirements.txt+pytestdiff --git a/src/agent/collector.py b/src/agent/collector.pyindex 240bfd0..33190bc 100644--- a/src/agent/collector.py+++ b/src/agent/collector.py@@ -3,14 +3,17 @@Only HOMELAB_API_URL is required. On first start the collector asks to join andappears on the dashboard under "New servers detected"; once someone clicks Addit receives its token (no restart) and saves it under HOMELAB_STATE_DIR.+HOMELAB_JOIN_TOKEN (from the dashboard's Connect a server command) proves who asked it to join.HOMELAB_ENROLL_KEY skips the approval step for automated installs, andHOMELAB_SERVER_ID / HOMELAB_COLLECTOR_TOKEN still work as explicit credentials.Run with ``python -m src.agent.collector`` or as a standalone script (standard-library only; the backend serves it at /collector.py). Docker access is optional; when+library only; it ships as a container image). Docker access is optional; whenthe socket is unavailable, host metrics continue to be sent."""from __future__ import annotations+import ctypes+import ctypes.utilimport http.clientimport jsonimport os@@ -19,6 +22,7 @@ import socketimport timeimport urllib.errorimport urllib.request+from datetime import datetimefrom pathlib import Pathfrom typing import Any@@ -132,6 +136,108 @@ def collect_mounts() -> list[dict[str, Any]]:return sorted(by_device.values(), key=lambda entry: entry["mount"])+def _read_text(path: Path) -> str | None:+ try:+ return path.read_text().strip()+ except OSError:+ return None+++def _oom_kill_total() -> int | None:+ """Processes the kernel OOM killer has stopped since boot (needs Linux 4.13+)."""+ for line in (_read_text(HOST_PROC / "vmstat") or "").splitlines():+ if line.startswith("oom_kill "):+ return int(line.split()[1])+ return None+++class _Timex(ctypes.Structure):+ _fields_ = [+ ("modes", ctypes.c_uint), ("offset", ctypes.c_long), ("freq", ctypes.c_long), ("maxerror", ctypes.c_long),+ ("esterror", ctypes.c_long), ("status", ctypes.c_int), ("constant", ctypes.c_long), ("precision", ctypes.c_long),+ ("tolerance", ctypes.c_long), ("time_sec", ctypes.c_long), ("time_usec", ctypes.c_long), ("tick", ctypes.c_long),+ ("ppsfreq", ctypes.c_long), ("jitter", ctypes.c_long), ("shift", ctypes.c_int), ("stabil", ctypes.c_long),+ ("jitcnt", ctypes.c_long), ("calcnt", ctypes.c_long), ("errcnt", ctypes.c_long), ("stbcnt", ctypes.c_long),+ ("tai", ctypes.c_int), ("padding", ctypes.c_int * 11),+ ]+++_TIME_ERROR = 5+++def _ntp_synced() -> bool | None:+ """Whether the kernel clock is disciplined by a time service; None when it can't be read."""+ # A read-only adjtimex (modes=0) covers chrony, ntpd and timesyncd alike. Docker's default+ # seccomp profile may refuse it, in which case timesyncd's marker file is the fallback.+ try:+ libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True)+ state = libc.adjtimex(ctypes.byref(_Timex()))+ if state >= 0:+ return state != _TIME_ERROR+ except (OSError, AttributeError):+ pass+ if (HOST_ROOT / "run/systemd/timesyncd/synchronized").exists():+ return True+ return None+++def _reboot_required() -> bool | None:+ """Debian and Ubuntu flag pending reboots with a file; other distros aren't checked, so None."""+ if (HOST_ROOT / "run/reboot-required").exists() or (HOST_ROOT / "var/run/reboot-required").exists():+ return True+ return False if (HOST_ROOT / "etc/debian_version").exists() else None+++HELPER_FILE_MAX_BYTES = 256 * 1024+HELPER_MAX_AGE_SECONDS = 300+++def _helper_path() -> Path:+ return Path(os.getenv("HOMELAB_HELPER_FILE", "/helper/system.json"))+++def collect_helper(boot_id: str | None, path: Path | None = None, now: float | None = None) -> dict[str, Any]:+ """Summary from the optional system helper; {} when it isn't installed, stale or from another boot.++ The file is treated as untrusted input: symlinks are refused, size is capped, and only+ expected fields are passed on (the backend validates them again).+ """+ path = path or _helper_path()+ try:+ fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)+ except OSError:+ return {}+ try:+ raw = os.read(fd, HELPER_FILE_MAX_BYTES + 1)+ except OSError:+ return {}+ finally:+ os.close(fd)+ try:+ data = json.loads(raw) if len(raw) <= HELPER_FILE_MAX_BYTES else None+ generated = datetime.fromisoformat(data["generated_at"]).timestamp()+ except (ValueError, KeyError, TypeError):+ return {}+ if not isinstance(data, dict) or data.get("boot_id") != boot_id or abs((now if now is not None else time.time()) - generated) > HELPER_MAX_AGE_SECONDS:+ return {}+ events = data.get("events")+ return {"last_shutdown": data.get("last_shutdown"), "helper_events": events if isinstance(events, list) else []}+++def collect_system() -> dict[str, Any]:+ """Kernel-reported system state; the optional helper adds what needs log access."""+ tainted = _read_text(HOST_PROC / "sys/kernel/tainted")+ boot_id = _read_text(HOST_PROC / "sys/kernel/random/boot_id")+ return {+ **collect_helper(boot_id),+ "boot_id": boot_id,+ "oom_kill_total": _oom_kill_total(),+ "kernel_tainted": int(tainted) if tainted and tainted.isdigit() else None,+ "ntp_synced": _ntp_synced(),+ "reboot_required": _reboot_required(),+ }++def collect_host() -> tuple[dict[str, Any], dict[str, Any]]:before_total, before_idle = _cpu_times()time.sleep(0.15)@@ -162,6 +268,7 @@ def collect_host() -> tuple[dict[str, Any], dict[str, Any]]:"net_bytes_recv_mb": round(received / MiB, 3),"system_uptime_seconds": round(uptime, 1),"mounts": collect_mounts(),+ **collect_system(),}server = {"hostname": _host_hostname(),@@ -236,7 +343,8 @@ def _docker_request(socket_path: str, endpoint: str) -> Any:def collect_containers(socket_path: str | None = None) -> list[dict[str, Any]]:- socket_path = socket_path or os.getenv("DOCKER_SOCKET_PATH", "/var/run/docker.sock")+ # The read-only proxy is preferred: it keeps the Docker socket, which is root on the host, out of this container.+ socket_path = socket_path or os.getenv("HOMELAB_DOCKER_HOST") or os.getenv("DOCKER_SOCKET_PATH", "/var/run/docker.sock")if not Path(socket_path).exists():return []containers = _docker_request(socket_path, "/containers/json?all=1")@@ -379,14 +487,23 @@ def _post_json(url: str, payload: dict[str, Any], bearer: str | None = None) ->return json.loads(response.read().decode())+def _error_detail(exc: urllib.error.HTTPError) -> str:+ """The backend's explanation, e.g. that a join token has expired, without trusting its size or shape."""+ try:+ detail = json.loads(exc.read(2048)).get("detail")+ return " ".join(str(detail).split())[:200] if detail else "no details"+ except (ValueError, AttributeError, OSError):+ return "no details"++def request_approval(api_url: str) -> tuple[int, str]:"""Ask to join, then wait until someone clicks Add on the dashboard. No restart needed."""base = api_url.rstrip("/")while True:try:- registration = _post_json(f"{base}/monitoring/collectors/register", {"hostname": _host_hostname(), "machine_id": _machine_id(), "os_info": _host_os_info()})+ registration = _post_json(f"{base}/monitoring/collectors/register", {"hostname": _host_hostname(), "machine_id": _machine_id(), "os_info": _host_os_info(), "join_token": os.getenv("HOMELAB_JOIN_TOKEN", "").strip() or None})except urllib.error.HTTPError as exc:- print(f"Registration refused (HTTP {exc.code}); retrying in 5 minutes", flush=True)+ print(f"Registration refused (HTTP {exc.code}): {_error_detail(exc)}; retrying in 5 minutes", flush=True)time.sleep(300)continueexcept (urllib.error.URLError, TimeoutError, OSError, ValueError) as exc:diff --git a/src/agent/docker_proxy.py b/src/agent/docker_proxy.pynew file mode 100644index 0000000..c79b812--- /dev/null+++ b/src/agent/docker_proxy.py@@ -0,0 +1,155 @@+"""Read-only Docker API proxy for the collector.++The Docker socket is all-or-nothing: whatever can talk to it can start a privileged container,+which is root on the host. Only this proxy mounts the socket. It listens on a Unix socket in a+volume that only the collector shares (no network port at all), and answers exactly three kinds of request:++ GET /containers/json?all=1+ GET /containers/<id>/json+ GET /containers/<id>/stats?stream=false++Everything else is refused, and the answers are cut down to the fields the collector uses: the+inspect output is where a container's environment variables (often secrets) live, and they never+leave this process. Paths are matched literally, with no decoding or normalisation, so there is+nothing for traversal or encoding tricks to work on.++Standard library only. Run with ``python -m src.agent.docker_proxy`` or as a standalone script.+"""+from __future__ import annotations++import http.client+import json+import os+import re+import socket+import socketserver+from http.server import BaseHTTPRequestHandler+from pathlib import Path+from typing import Any, Callable++SOCKET_PATH = os.getenv("DOCKER_SOCKET_PATH", "/var/run/docker.sock")+LISTEN_PATH = os.getenv("HOMELAB_DOCKER_PROXY_SOCKET", "/run/docker-proxy/docker.sock")+MAX_RESPONSE_BYTES = 8 * 1024 * 1024+UPSTREAM_TIMEOUT_SECONDS = 10++_ID = r"[0-9a-f]{12,64}"+# Matched against the whole request target, query string included.+_LIST = re.compile(r"/containers/json(\?all=1)?")+_INSPECT = re.compile(rf"/containers/{_ID}/json")+_STATS = re.compile(rf"/containers/{_ID}/stats\?stream=false")+++def _pick(source: Any, keys: tuple[str, ...]) -> dict[str, Any]:+ return {key: source[key] for key in keys if isinstance(source, dict) and key in source}+++def project_list(data: Any) -> list[dict[str, Any]]:+ return [_pick(item, ("Id", "Names", "Image", "State")) for item in data if isinstance(item, dict)] if isinstance(data, list) else []+++def project_inspect(data: Any) -> dict[str, Any]:+ """The inspect fields the collector reads. Config.Env, mounts, labels and the rest are dropped."""+ if not isinstance(data, dict):+ return {}+ state = data.get("State") if isinstance(data.get("State"), dict) else {}+ health = state.get("Health") if isinstance(state.get("Health"), dict) else {}+ network = data.get("NetworkSettings") if isinstance(data.get("NetworkSettings"), dict) else {}+ config = data.get("Config") if isinstance(data.get("Config"), dict) else {}+ return {+ **_pick(data, ("Id", "Name", "RestartCount")),+ "Config": _pick(config, ("Image",)),+ "State": {**_pick(state, ("Status", "StartedAt")), "Health": _pick(health, ("Status",))},+ "NetworkSettings": _pick(network, ("Ports",)),+ }+++def project_stats(data: Any) -> dict[str, Any]:+ if not isinstance(data, dict):+ return {}+ return _pick(data, ("cpu_stats", "precpu_stats", "memory_stats", "networks", "blkio_stats", "pids_stats"))+++def route(method: str, target: str) -> tuple[str, Callable[[Any], Any]] | None:+ """The upstream request and response filter for an allowed request, or None to refuse it."""+ if method != "GET" or not target.isascii() or not target.isprintable():+ return None+ for pattern, project in ((_LIST, project_list), (_INSPECT, project_inspect), (_STATS, project_stats)):+ if pattern.fullmatch(target):+ return target, project+ return None+++class _UnixHTTPConnection(http.client.HTTPConnection):+ def __init__(self, socket_path: str):+ super().__init__("localhost", timeout=UPSTREAM_TIMEOUT_SECONDS)+ self.socket_path = socket_path++ def connect(self) -> None:+ self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)+ self.sock.settimeout(self.timeout)+ self.sock.connect(self.socket_path)+++class ProxyHandler(BaseHTTPRequestHandler):+ server_version = "homelab-docker-proxy"+ socket_path = SOCKET_PATH++ def _send(self, status: int, body: bytes = b"", content_type: str = "text/plain") -> None:+ self.send_response(status)+ self.send_header("Content-Type", content_type)+ self.send_header("Content-Length", str(len(body)))+ self.send_header("Connection", "close")+ self.end_headers()+ self.wfile.write(body)++ def _handle(self) -> None:+ allowed = route(self.command, self.path)+ if allowed is None:+ print(f"Refused {self.command} {self.path[:80]!r}", flush=True)+ self._send(403, b"Forbidden")+ return+ target, project = allowed+ try:+ connection = _UnixHTTPConnection(self.socket_path)+ try:+ connection.request("GET", target, headers={"Host": "localhost", "Connection": "close"})+ response = connection.getresponse()+ body = response.read(MAX_RESPONSE_BYTES + 1)+ status = response.status+ finally:+ connection.close()+ if status != 200 or len(body) > MAX_RESPONSE_BYTES:+ self._send(502, b"Upstream error")+ return+ self._send(200, json.dumps(project(json.loads(body))).encode(), "application/json")+ except (OSError, ValueError, http.client.HTTPException):+ self._send(502, b"Upstream error")++ # Every method goes through the same check, so anything but an allowed GET is refused.+ do_GET = do_POST = do_PUT = do_DELETE = do_PATCH = do_HEAD = do_OPTIONS = _handle++ def log_message(self, format: str, *args: Any) -> None: # noqa: A002 - quiet by default; refusals are logged above+ pass+++class _Server(socketserver.ThreadingUnixStreamServer):+ daemon_threads = True+++def make_server(listen_path: str = LISTEN_PATH, socket_path: str = SOCKET_PATH) -> socketserver.ThreadingUnixStreamServer:+ """Listen on a Unix socket only readable and writable by its owner (and group)."""+ listen = Path(listen_path)+ listen.parent.mkdir(parents=True, exist_ok=True)+ listen.unlink(missing_ok=True)+ handler = type("BoundProxyHandler", (ProxyHandler,), {"socket_path": socket_path})+ previous = os.umask(0o117) # created as 0660, never briefly world-accessible+ try:+ server = _Server(str(listen), handler)+ finally:+ os.umask(previous)+ return server+++if __name__ == "__main__":+ print(f"Docker read-only proxy listening on {LISTEN_PATH}, upstream {SOCKET_PATH}", flush=True)+ make_server().serve_forever()diff --git a/src/agent/system_helper.py b/src/agent/system_helper.pynew file mode 100644index 0000000..b9f53fd--- /dev/null+++ b/src/agent/system_helper.py@@ -0,0 +1,179 @@+"""Optional privileged helper: reads the kernel log and wtmp, writes a small sanitized JSON summary.++Runs in its own container (the ``system-events`` compose profile) with a single capability,+CAP_SYSLOG, no network, a read-only filesystem and wtmp mounted read-only. The collector+reads the summary from a shared volume; it never touches the kernel log itself.++Everything parsed here is attacker-influenced (a process can name itself anything, usernames+and hostnames land in wtmp), so nothing is ever executed or interpolated: lines are matched+with fixed patterns, strings are reduced to printable text and truncated, and the output has a+fixed schema. The helper is stateless; each run re-reads the kernel ring buffer and publishes+a snapshot, and the backend de-duplicates events by their key.++Standard library only. Run with ``python -m src.agent.system_helper`` or as a standalone script.+"""+from __future__ import annotations++import ctypes+import ctypes.util+import json+import os+import re+import struct+import time+from datetime import datetime, timezone+from pathlib import Path+from typing import Any++HOST_PROC = Path(os.getenv("HOST_PROC") or "/proc")+WTMP_PATH = Path(os.getenv("HOMELAB_WTMP_PATH", "/host/wtmp"))+OUTPUT_PATH = Path(os.getenv("HOMELAB_HELPER_OUTPUT", "/out/system.json"))++SYSLOG_ACTION_READ_ALL = 3 # reads the ring buffer without clearing it+SYSLOG_ACTION_SIZE_BUFFER = 10+MAX_KLOG_BYTES = 8 * 1024 * 1024+MAX_OOM_EVENTS = 20+MAX_ERROR_EVENTS = 10+MAX_TEXT = 200+MAX_PROCESS_NAME = 64++# "<6>[ 3800.756804] message": priority, seconds since boot, text. Lines without a timestamp are ignored.+_KLOG_LINE = re.compile(r"^<(\d{1,3})>\[\s*(\d{1,10}\.\d{6})\] (.*)$")+_OOM_KILL = re.compile(r"[Kk]illed process (\d{1,10}) \(([^)]{0,200})\)")+ERROR_PRIORITY = 3 # kern.err and worse++# struct utmp on 64-bit Linux (glibc): 384 bytes, fixed layout.+_UTMP = struct.Struct("<h2xi32s4s32s256shhiii4i20x")+_RUN_LVL, _BOOT_TIME = 1, 2+MAX_WTMP_RECORDS = 5000+BOOT_TIME_TOLERANCE_SECONDS = 300+++def sanitize(text: str, limit: int = MAX_TEXT) -> str:+ """Printable characters only, so log content can't carry control sequences or hide in the output."""+ return "".join(char if char.isprintable() else "?" for char in text).strip()[:limit]+++def _decode(raw: bytes) -> str:+ return raw.split(b"\0", 1)[0].decode("utf-8", errors="replace")+++def read_kernel_log() -> str:+ """The kernel ring buffer; needs CAP_SYSLOG when kernel.dmesg_restrict is set."""+ libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True)+ size = libc.klogctl(SYSLOG_ACTION_SIZE_BUFFER, None, 0)+ if size <= 0:+ raise OSError(ctypes.get_errno(), "cannot size the kernel log")+ size = min(size, MAX_KLOG_BYTES)+ buffer = ctypes.create_string_buffer(size)+ read = libc.klogctl(SYSLOG_ACTION_READ_ALL, buffer, size)+ if read < 0:+ raise OSError(ctypes.get_errno(), "cannot read the kernel log")+ return buffer.raw[:read].decode("utf-8", errors="replace")+++def parse_kernel_log(text: str, boot_id: str, monotonic_now: float, wall_now: float) -> list[dict[str, str]]:+ """OOM kills and kernel errors as events with wall-clock times, oldest first."""+ oom: list[dict[str, str]] = []+ errors: list[dict[str, str]] = []+ for line in text.splitlines():+ match = _KLOG_LINE.match(line)+ if not match:+ continue+ priority, stamp, message = int(match.group(1)) & 7, match.group(2), match.group(3)+ # printk timestamps are CLOCK_MONOTONIC, the same clock as time.monotonic().+ when = datetime.fromtimestamp(wall_now - (monotonic_now - float(stamp)), timezone.utc).isoformat(timespec="seconds")+ killed = _OOM_KILL.search(message)+ if killed and "ut of memory" in message:+ oom.append({"kind": "oom_kill", "time": when, "key": f"{boot_id}:oom:{stamp}",+ "detail": f"The kernel OOM killer stopped {sanitize(killed.group(2), MAX_PROCESS_NAME) or 'a process'} (pid {killed.group(1)})"})+ elif priority <= ERROR_PRIORITY:+ errors.append({"kind": "kernel_error", "time": when, "key": f"{boot_id}:err:{stamp}", "detail": sanitize(message)})+ return oom[-MAX_OOM_EVENTS:] + errors[-MAX_ERROR_EVENTS:]+++def read_wtmp_records(path: Path = WTMP_PATH) -> list[tuple[int, str, int]]:+ """(type, user, unix time) for the most recent records; [] when unreadable."""+ try:+ fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)+ except OSError:+ return []+ try:+ size = os.fstat(fd).st_size+ records = min(size // _UTMP.size, MAX_WTMP_RECORDS)+ os.lseek(fd, (size // _UTMP.size - records) * _UTMP.size, os.SEEK_SET)+ data = b""+ while len(data) < records * _UTMP.size:+ chunk = os.read(fd, records * _UTMP.size - len(data))+ if not chunk:+ break+ data += chunk+ except OSError:+ return []+ finally:+ os.close(fd)+ result = []+ for offset in range(0, len(data) - _UTMP.size + 1, _UTMP.size):+ fields = _UTMP.unpack_from(data, offset)+ result.append((fields[0], _decode(fields[4]), fields[9]))+ return result+++def classify_last_shutdown(records: list[tuple[int, str, int]], boot_time: float) -> str:+ """How the boot before this one ended: "clean", "unclean" or "unknown".++ A clean shutdown writes a "shutdown" record before the next boot record; a crash or power+ loss leaves none. It only counts as unclean when wtmp has recorded shutdowns before, so a+ system that never writes them isn't reported as crashing on every reboot.+ """+ boots = [index for index, (kind, _user, _time) in enumerate(records) if kind == _BOOT_TIME]+ if len(boots) < 2 or abs(records[boots[-1]][2] - boot_time) > BOOT_TIME_TOLERANCE_SECONDS:+ return "unknown"+ previous, latest = boots[-2], boots[-1]+ is_shutdown = lambda record: record[0] == _RUN_LVL and record[1] == "shutdown" # noqa: E731+ if any(is_shutdown(record) for record in records[previous + 1:latest]):+ return "clean"+ return "unclean" if any(is_shutdown(record) for record in records[:latest]) else "unknown"+++def build_snapshot() -> dict[str, Any]:+ boot_id = (HOST_PROC / "sys/kernel/random/boot_id").read_text().strip()[:64]+ wall_now, monotonic_now = time.time(), time.monotonic()+ snapshot: dict[str, Any] = {+ "version": 1,+ "generated_at": datetime.fromtimestamp(wall_now, timezone.utc).isoformat(timespec="seconds"),+ "boot_id": boot_id,+ "last_shutdown": "unknown",+ "events": [],+ }+ try:+ snapshot["events"] = parse_kernel_log(read_kernel_log(), boot_id, monotonic_now, wall_now)+ except OSError as error:+ print(f"Kernel log unavailable ({error}); is CAP_SYSLOG granted?", flush=True)+ uptime = float((HOST_PROC / "uptime").read_text().split()[0])+ snapshot["last_shutdown"] = classify_last_shutdown(read_wtmp_records(), wall_now - uptime)+ return snapshot+++def write_snapshot(snapshot: dict[str, Any], path: Path = OUTPUT_PATH) -> None:+ """Atomic replace, so the collector never reads a half-written file."""+ temporary = path.with_name(path.name + ".tmp")+ temporary.write_text(json.dumps(snapshot, separators=(",", ":")))+ temporary.chmod(0o644)+ temporary.replace(path)+++def run() -> None:+ interval = max(10, int(os.getenv("HOMELAB_HELPER_INTERVAL_SECONDS", "30")))+ print(f"System helper started; writing {OUTPUT_PATH} every {interval}s", flush=True)+ while True:+ started = time.monotonic()+ try:+ write_snapshot(build_snapshot())+ except (OSError, ValueError) as error:+ print(f"Snapshot failed ({type(error).__name__}): {error}", flush=True)+ time.sleep(max(1, interval - (time.monotonic() - started)))+++if __name__ == "__main__":+ run()diff --git a/src/backend/auth.py b/src/backend/auth.pyindex 49a5e6f..edb0174 100644--- a/src/backend/auth.py+++ b/src/backend/auth.py@@ -116,6 +116,20 @@ def get_recent_logs(return result+PASSWORD_MIN_LENGTH = 16+# bcrypt only uses the first 72 bytes (and refuses longer input), so longer passwords are rejected+# up front instead of being silently truncated or crashing.+PASSWORD_MAX_BYTES = 72+++def validate_password(password: str) -> None:+ """New passwords must be long enough to resist guessing; existing accounts keep working as they are."""+ if len(password) < PASSWORD_MIN_LENGTH:+ raise ValueError(f"Password must be at least {PASSWORD_MIN_LENGTH} characters long. A few random words make a good passphrase.")+ if len(password.encode("utf-8")) > PASSWORD_MAX_BYTES:+ raise ValueError(f"Password must be at most {PASSWORD_MAX_BYTES} bytes long.")++def register_user(db, redis_client, *, name: str, email: str, password: str):normalized_name = name.strip()normalized_email = email.strip().lower()@@ -124,8 +138,7 @@ def register_user(db, redis_client, *, name: str, email: str, password: str):raise ValueError("Name is required.")if not normalized_email:raise ValueError("Email is required.")- if len(password) < 6:- raise ValueError("Password must be at least 6 characters long.")+ validate_password(password)existing = db.query(User).filter(User.email == normalized_email).first()if existing:@@ -164,7 +177,8 @@ def authenticate_user(db, redis_client, *, email: str, password: str):if not user or not user.is_active:raise ValueError("Invalid email or password.")- if not bcrypt.checkpw(password.encode("utf-8"), user.password_hash.encode("utf-8")):+ # Anything over bcrypt's limit can't be a valid password here; bcrypt would raise with a different message.+ if len(password.encode("utf-8")) > PASSWORD_MAX_BYTES or not bcrypt.checkpw(password.encode("utf-8"), user.password_hash.encode("utf-8")):raise ValueError("Invalid email or password.")record_audit_log(db, user_id=user.id, level="INFO", message="User signed in", source="auth")diff --git a/src/backend/config.py b/src/backend/config.pyindex ef6b659..e314a7f 100644--- a/src/backend/config.py+++ b/src/backend/config.py@@ -26,3 +26,18 @@ CORS_ORIGINS = [for origin in os.getenv("CORS_ORIGINS", "http://localhost:5173,http://localhost:5174,http://localhost:5175,http://localhost:5176").split(",")if origin.strip()]++# "strict" makes the collector endpoints answer only requests that arrived through the VPN or+# internal listener (see docker/nginx.strict.conf); "open" keeps the single-listener behaviour.+COLLECTOR_ACCESS = os.getenv("COLLECTOR_ACCESS", "open").lower()+# What the Connect a server panel shows: the collector listener's address on your VPN, and the+# pinned images remote hosts install. All optional.+COLLECTOR_PUBLIC_URL = os.getenv("COLLECTOR_PUBLIC_URL", "").strip().rstrip("/")+COLLECTOR_IMAGE = os.getenv("COLLECTOR_IMAGE", "").strip()+DOCKER_PROXY_IMAGE = os.getenv("DOCKER_PROXY_IMAGE", "").strip()+JOIN_TOKEN_TTL_SECONDS = int(os.getenv("JOIN_TOKEN_TTL_SECONDS", "900"))++# Creating the first account (which becomes the admin) needs a setup token, so nobody else can claim a+# fresh install. Leave unset to have one generated and printed in the backend log at every start until+# the first account exists; set it to choose your own.+ADMIN_BOOTSTRAP_TOKEN = os.getenv("ADMIN_BOOTSTRAP_TOKEN", "").strip()diff --git a/src/backend/main.py b/src/backend/main.pyindex 61bf292..d78656e 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -10,18 +10,19 @@ import timefrom pathlib import Pathfrom datetime import datetime, timedelta, timezonefrom typing import Annotated+from urllib.parse import urlsplitimport redisfrom fastapi import Cookie, Depends, FastAPI, Header, HTTPException, Query, Request, Response, statusfrom fastapi.middleware.cors import CORSMiddleware-from fastapi.responses import PlainTextResponse+from fastapi.responses import JSONResponse, PlainTextResponsefrom pydantic import BaseModel, EmailStrfrom sqlalchemy import case, func, inspect, textfrom .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alertsfrom .auth import authenticate_user, get_recent_logs, get_user_from_session, logout_user, record_audit_log, register_userfrom .db import SessionLocal, check_database, get_db-from .models import AppSetting, AuditLog, Base, ChatMessage, ContainerMetric, Conversation, EnrollmentKey, HostMetric, IncidentAlert, MountMetric, PendingCollector, Server, User+from .models import AppSetting, AuditLog, Base, ChatMessage, ContainerMetric, Conversation, EnrollmentKey, HostMetric, IncidentAlert, JoinToken, MountMetric, PendingCollector, Server, SystemEvent, Userfrom .db import enginefrom .config import (@@ -30,13 +31,19 @@ from .config import (REDIS_URL,REGISTER_RATE_LIMIT,REGISTER_RATE_WINDOW_SECONDS,+ ADMIN_BOOTSTRAP_TOKEN,ALLOW_REGISTRATION,+ COLLECTOR_ACCESS,+ COLLECTOR_IMAGE,+ COLLECTOR_PUBLIC_URL,+ DOCKER_PROXY_IMAGE,+ JOIN_TOKEN_TTL_SECONDS,CORS_ORIGINS,SESSION_COOKIE_NAME,SESSION_COOKIE_SECURE,SESSION_TTL_SECONDS,)-from .rules import DEFAULT_THRESHOLDS, HOST_RULE_PREFIX, container_rules, host_rules, validate_thresholds+from .rules import DEFAULT_THRESHOLDS, HOST_RULE_PREFIX, container_rules, host_rules, validate_thresholds, detect_system_events, system_rules, SYSTEM_RULE_PREFIX, validate_mount_thresholds, validate_helper_events, REBOOT_DETAILS, REBOOT_KEY_PREFIXfrom .rate_limit import enforce_ip_limit, enforce_login_limit, enforce_register_limit, reset_limitsfrom ..agent.service import delete_agent_conversation, send_agent_message@@ -45,6 +52,8 @@ class RegisterRequest(BaseModel):name: stremail: EmailStrpassword: str+ # Only needed for the very first account; see ensure_bootstrap_token.+ bootstrap_token: str | None = Noneclass LoginRequest(BaseModel):@@ -78,6 +87,12 @@ class CollectorRegisterRequest(BaseModel):hostname: strmachine_id: str | None = Noneos_info: str | None = None+ join_token: str | None = None+++class ApproveRequest(BaseModel):+ # Required to approve a request whose machine matches a server that already exists.+ replace: bool = Falseclass AdminSettingsRequest(BaseModel):@@ -85,6 +100,8 @@ class AdminSettingsRequest(BaseModel):metrics_retention_days: int | None = Noneagent_chat_enabled: bool | None = Nonealert_thresholds: dict | None = None+ # Replaces all per-mount disk overrides: {mount: {disk_warning_pct, disk_critical_pct}}.+ mount_thresholds: dict | None = Noneclass MetricsIngestRequest(BaseModel):@@ -119,6 +136,36 @@ def agent_error_message(exc: Exception) -> str:redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)app = FastAPI(title="Homelab Agent API")+UNSAFE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}+++def cross_site_request(method: str, headers, allowed_origins: list[str]) -> bool:+ """True for a browser request that came from another site and must be refused.++ Cookies are SameSite=Lax, which still lets a page on a sibling subdomain (other apps behind the same+ reverse proxy count as same-site) send state-changing requests with the session cookie. Browsers+ always say where such a request came from, so refuse it unless that is this dashboard. Tools that+ aren't browsers (collectors, curl) send neither header and are unaffected; they authenticate with+ tokens, not cookies.+ """+ if method not in UNSAFE_METHODS:+ return False+ origin = headers.get("origin")+ if origin is not None:+ if origin in allowed_origins:+ return False+ parsed = urlsplit(origin)+ # Same host and port as the dashboard was reached on; the scheme can't be compared behind a TLS-terminating proxy.+ return not (parsed.scheme in ("http", "https") and parsed.netloc and parsed.netloc == headers.get("host"))+ return headers.get("sec-fetch-site") not in (None, "same-origin", "none")++[email protected]("http")+async def refuse_cross_site_requests(request: Request, call_next):+ if cross_site_request(request.method, request.headers, CORS_ORIGINS):+ return JSONResponse({"detail": "Cross-site request refused."}, status_code=403)+ return await call_next(request)+app.add_middleware(CORSMiddleware,# Only needed when the dashboard is served from a different origin than /api.@@ -144,6 +191,14 @@ def startup_event():("started_at", "VARCHAR(64)"), ("ports", "VARCHAR(512)"),):connection.execute(text(f"ALTER TABLE container_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))+ for column, column_type in (+ ("boot_id", "VARCHAR(64)"), ("oom_kill_total", "INTEGER"), ("kernel_tainted", "INTEGER"),+ ("ntp_synced", "BOOLEAN"), ("reboot_required", "BOOLEAN"), ("last_shutdown", "VARCHAR(16)"),+ ):+ connection.execute(text(f"ALTER TABLE host_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))+ connection.execute(text("ALTER TABLE system_events ADD COLUMN IF NOT EXISTS event_key VARCHAR(128)"))+ connection.execute(text("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS join_token_id INTEGER REFERENCES join_tokens(id) ON DELETE SET NULL"))+ connection.execute(text("CREATE INDEX IF NOT EXISTS ix_system_events_event_key ON system_events (event_key)"))connection.execute(text("ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS rule_key VARCHAR(255)"))connection.execute(text("ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS resolved_at TIMESTAMP WITH TIME ZONE"))connection.execute(text("CREATE INDEX IF NOT EXISTS ix_incident_alerts_rule_key ON incident_alerts (rule_key)"))@@ -167,6 +222,9 @@ def startup_event():db = SessionLocal()try:ensure_admin_exists(db)+ setup_token = ensure_bootstrap_token(db)+ if setup_token:+ print(f"First-run setup token: {setup_token} (needed to create the first account; shown again at every start until it exists)", flush=True)finally:db.close()threading.Thread(target=_offline_watch_loop, name="offline-watch", daemon=True).start()@@ -191,13 +249,41 @@ def health():}+BOOTSTRAP_TOKEN_KEY = "bootstrap_token_hash"++def registration_open(db) -> bool:return ALLOW_REGISTRATION or db.query(User).count() == 0+def ensure_bootstrap_token(db) -> str | None:+ """Make sure a fresh install can only be claimed by someone who can read its configuration or logs.++ While there are no accounts, the first one needs a setup token: ADMIN_BOOTSTRAP_TOKEN if set,+ otherwise a random one generated here and returned so the caller can print it. Only its hash is+ stored. Once an account exists the token is deleted and nothing is required any more.+ """+ setting = db.get(AppSetting, BOOTSTRAP_TOKEN_KEY)+ if db.query(User).count() > 0:+ if setting:+ db.delete(setting)+ db.commit()+ return None+ token = ADMIN_BOOTSTRAP_TOKEN or f"hlb_{secrets.token_urlsafe(24)}"+ _set_setting(db, BOOTSTRAP_TOKEN_KEY, _hash_secret(token))+ db.commit()+ return None if ADMIN_BOOTSTRAP_TOKEN else token+++def _bootstrap_token_valid(db, token: str | None) -> bool:+ setting = db.get(AppSetting, BOOTSTRAP_TOKEN_KEY)+ # No stored token (the backend hasn't started up properly) means refuse, never allow.+ return bool(setting and token and hmac.compare_digest(setting.value, _hash_secret(token.strip())))++@app.get("/auth/config")def auth_config(db=Depends(get_db)):- return {"registration_enabled": registration_open(db)}+ return {"registration_enabled": registration_open(db), "bootstrap_required": db.query(User).count() == 0}@app.post("/auth/register")@@ -210,6 +296,9 @@ def auth_register(payload: RegisterRequest, request: Request, response: Responselimit=REGISTER_RATE_LIMIT,window_seconds=REGISTER_RATE_WINDOW_SECONDS,)+ first_account = db.query(User).count() == 0+ if first_account and not _bootstrap_token_valid(db, payload.bootstrap_token):+ raise HTTPException(status_code=403, detail="This install needs its setup token to create the first account. Find it in the backend log: docker compose logs backend | grep \"setup token\"")try:result = register_user(db,@@ -221,6 +310,8 @@ def auth_register(payload: RegisterRequest, request: Request, response: Responseexcept ValueError as exc:raise HTTPException(status_code=400, detail=str(exc)) from exc+ if first_account:+ ensure_bootstrap_token(db) # an account exists now: this removes the tokenresponse.set_cookie(key=SESSION_COOKIE_NAME,value=result["session_id"],@@ -333,6 +424,7 @@ MAX_METRICS_RETENTION_DAYS = 3650AGENT_CHAT_ENABLED_KEY = "agent_chat_enabled"ALERT_THRESHOLDS_KEY = "alert_thresholds"+MOUNT_THRESHOLDS_KEY = "mount_thresholds"def _set_setting(db, key: str, value: str) -> None:@@ -366,8 +458,17 @@ def get_alert_thresholds(db) -> dict:return dict(DEFAULT_THRESHOLDS)+def get_mount_thresholds(db) -> dict:+ """Per-mount disk threshold overrides; anything unreadable is ignored in favor of the global thresholds."""+ setting = db.get(AppSetting, MOUNT_THRESHOLDS_KEY)+ try:+ return validate_mount_thresholds(json.loads(setting.value)) if setting else {}+ except ValueError:+ return {}++def _admin_settings(db) -> dict:- return {"metrics_retention_days": get_metrics_retention_days(db), "agent_chat_enabled": is_agent_chat_enabled(db), "alert_thresholds": get_alert_thresholds(db)}+ return {"metrics_retention_days": get_metrics_retention_days(db), "agent_chat_enabled": is_agent_chat_enabled(db), "alert_thresholds": get_alert_thresholds(db), "mount_thresholds": get_mount_thresholds(db)}def purge_old_metrics(db, now: datetime | None = None) -> int:@@ -376,6 +477,7 @@ def purge_old_metrics(db, now: datetime | None = None) -> int:deleted = db.query(HostMetric).filter(HostMetric.time < cutoff).delete(synchronize_session=False)deleted += db.query(ContainerMetric).filter(ContainerMetric.time < cutoff).delete(synchronize_session=False)deleted += db.query(MountMetric).filter(MountMetric.time < cutoff).delete(synchronize_session=False)+ deleted += db.query(SystemEvent).filter(SystemEvent.time < cutoff).delete(synchronize_session=False)db.commit()return deleted@@ -409,6 +511,14 @@ def update_admin_settings(payload: AdminSettingsRequest, session_id: Annotated[s_set_setting(db, ALERT_THRESHOLDS_KEY, json.dumps(thresholds))if changed:changes.append("Changed alert thresholds: " + ", ".join(f"{key}={value}" for key, value in changed.items()))+ if payload.mount_thresholds is not None:+ try:+ mount_thresholds = validate_mount_thresholds(payload.mount_thresholds)+ except ValueError as exc:+ raise HTTPException(status_code=400, detail=str(exc))+ if mount_thresholds != get_mount_thresholds(db):+ _set_setting(db, MOUNT_THRESHOLDS_KEY, json.dumps(mount_thresholds))+ changes.append("Changed per-mount disk thresholds: " + (", ".join(f"{mount}={limits['disk_warning_pct']:g}/{limits['disk_critical_pct']:g}" for mount, limits in mount_thresholds.items()) or "none"))db.commit()for change in changes:record_audit_log(db, user_id=user["id"], level="INFO", message=change, source="admin")@@ -425,6 +535,9 @@ def agent_config(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NOVERVIEW_ALERT_LIMIT = 50+SYSTEM_EVENTS_SHOWN = 10++@app.get("/monitoring/overview")def monitoring_overview(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):# Servers belong to the install: everyone signed in can view them, only admins manage them.@@ -432,6 +545,11 @@ def monitoring_overview(session_id: Annotated[str | None, Cookie(alias=SESSION_C# The dashboard's own machine first, then the newest servers.servers = db.query(Server).order_by(Server.is_dashboard_host.desc(), Server.created_at.desc()).all()alert_counts = dict(db.query(IncidentAlert.server_id, func.count()).filter(IncidentAlert.status == "active").group_by(IncidentAlert.server_id).all())+ since = datetime.now(timezone.utc) - timedelta(days=7)+ events: dict[int, list[SystemEvent]] = {}+ for event in db.query(SystemEvent).filter(SystemEvent.time >= since).order_by(SystemEvent.time.desc()):+ if len(events.setdefault(event.server_id, [])) < SYSTEM_EVENTS_SHOWN:+ events[event.server_id].append(event)result = []for server in servers:metric = db.query(HostMetric).filter(HostMetric.server_id == server.id).order_by(HostMetric.time.desc()).first()@@ -443,7 +561,8 @@ def monitoring_overview(session_id: Annotated[str | None, Cookie(alias=SESSION_C"check_interval_seconds": server.check_interval_seconds, "is_dashboard_host": server.is_dashboard_host,"active_alert_count": alert_counts.get(server.id, 0),"last_seen_at": server.last_seen_at.isoformat() if server.last_seen_at else None,- "host": ({key: getattr(metric, key) for key in ("time", "cpu_utilization_pct", "load_1m", "load_5m", "load_15m", "ram_used_gb", "ram_available_gb", "ram_pct", "swap_pct", "disk_usage_pct", "disk_inodes_pct", "net_bytes_sent_mb", "net_bytes_recv_mb", "system_uptime_seconds")} if metric else None),+ "host": ({key: getattr(metric, key) for key in ("time", "cpu_utilization_pct", "load_1m", "load_5m", "load_15m", "ram_used_gb", "ram_available_gb", "ram_pct", "swap_pct", "disk_usage_pct", "disk_inodes_pct", "net_bytes_sent_mb", "net_bytes_recv_mb", "system_uptime_seconds", "oom_kill_total", "kernel_tainted", "ntp_synced", "reboot_required", "last_shutdown")} if metric else None),+ "system_events": [{"time": e.time.isoformat(), "kind": e.kind, "detail": e.detail} for e in events.get(server.id, [])],"mounts": [{key: getattr(m, key) for key in ("mount", "fs_type", "total_gb", "used_gb", "used_pct", "inodes_pct")} for m in mounts],"containers": [{"container_id": c.container_id, "name": c.name, "image": c.image, "status": c.status, "restart_count": c.restart_count, "cpu_pct": c.cpu_pct, "memory_used_mb": c.memory_used_mb, "memory_pct": c.memory_pct, "healthcheck_status": c.healthcheck_status, "time": c.time.isoformat()} for c in latest.values()],})@@ -457,6 +576,7 @@ def monitoring_overview(session_id: Annotated[str | None, Cookie(alias=SESSION_Creturn {"servers": result, "pending_collectors": pending, "alert_count": active_alerts.count(),# Lets the dashboard color disk bars the same way the alert rules judge them.+ "mount_thresholds": get_mount_thresholds(db),"disk_thresholds": {key: thresholds[key] for key in ("disk_warning_pct", "disk_critical_pct", "inodes_warning_pct", "inodes_critical_pct")},"alerts": [{"id": a.id, "server_id": a.server_id, "hostname": hostnames.get(a.server_id), "severity": a.severity, "title": a.title, "description": a.description, "created_at": a.created_at.isoformat()} for a in alerts],}@@ -525,6 +645,22 @@ def _hash_secret(value: str) -> str:return hashlib.sha256(value.encode()).hexdigest()+def _collector_via(request) -> str:+ """Which listener the request came in on: "vpn", "internal", or "" (public or direct).++ nginx sets the header on its collector listeners and clears it on the public one, so a client+ can't claim a listener it didn't use. In strict mode only collector listeners are answered.+ """+ via = getattr(request, "headers", {}).get("x-collector-via", "")+ if COLLECTOR_ACCESS == "strict" and via not in ("vpn", "internal"):+ raise HTTPException(status_code=404, detail="Not Found")+ return via+++def require_collector_listener(request: Request) -> None:+ _collector_via(request)++@app.get("/monitoring/enrollment-key")def get_enrollment_key(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):user = _require_admin(session_id, db)@@ -556,7 +692,7 @@ def revoke_enrollment_key(session_id: Annotated[str | None, Cookie(alias=SESSIONreturn {"status": "revoked"}[email protected]("/monitoring/collectors/enroll")[email protected]("/monitoring/collectors/enroll", dependencies=[Depends(require_collector_listener)])def enroll_collector(payload: EnrollRequest, db=Depends(get_db)):"""Register (or re-register) a collector using the owner's enrollment key.@@ -590,28 +726,57 @@ PAIRING_ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"PENDING_VISIBLE_SECONDS = 120PENDING_RETENTION = timedelta(days=1)MAX_PENDING_COLLECTORS = 100-COLLECTOR_SOURCE = Path(__file__).resolve().parents[1] / "agent" / "collector.py"+MAX_OPEN_JOIN_TOKENS = 20+JOIN_TOKEN_PREFIX = "hlj_"def _client_ip(request: Request) -> str:return request.client.host if request.client else "unknown"[email protected]("/collector.py", response_class=PlainTextResponse)-def collector_source():- """Serve the collector so new machines can run it from a stock Python image."""- return PlainTextResponse(COLLECTOR_SOURCE.read_text(), media_type="text/x-python")[email protected]("/monitoring/join-tokens")+def create_join_token(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+ """Mint the single-use token the Connect a server command carries. The plaintext is returned only once."""+ user = _require_admin(session_id, db)+ now = datetime.now(timezone.utc)+ if db.query(JoinToken).filter(JoinToken.used_at.is_(None), JoinToken.expires_at > now).count() >= MAX_OPEN_JOIN_TOKENS:+ raise HTTPException(status_code=429, detail="Too many unused join tokens. Wait for them to expire or add the servers already waiting.")+ plaintext = f"{JOIN_TOKEN_PREFIX}{secrets.token_urlsafe(32)}"+ db.add(JoinToken(token_hash=_hash_secret(plaintext), created_by=user["id"], expires_at=now + timedelta(seconds=JOIN_TOKEN_TTL_SECONDS)))+ db.commit()+ return {"join_token": plaintext, "expires_in_seconds": JOIN_TOKEN_TTL_SECONDS}++[email protected]("/monitoring/install-info")+def install_info(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+ """What the Connect a server panel needs to build its command."""+ _require_admin(session_id, db)+ return {"strict": COLLECTOR_ACCESS == "strict", "collector_url": COLLECTOR_PUBLIC_URL or None, "collector_image": COLLECTOR_IMAGE or None, "proxy_image": DOCKER_PROXY_IMAGE or None}[email protected]("/monitoring/collectors/register")[email protected]("/monitoring/collectors/register", dependencies=[Depends(require_collector_listener)])def register_collector(payload: CollectorRegisterRequest, request: Request, db=Depends(get_db)):- """Unauthenticated: a new collector asks to join and waits until a user clicks Add."""+ """A new collector asks to join and waits until a user clicks Add.++ Over the VPN listener a join token from the dashboard's Connect a server command is required.+ """+ via = _collector_via(request)ip_address = _client_ip(request)enforce_ip_limit(redis_client, scope="collector-register", ip_address=ip_address, limit=30, window_seconds=3600)hostname = payload.hostname.strip()[:255]if not hostname:raise HTTPException(status_code=400, detail="Hostname is required.")machine_id = (payload.machine_id or "").strip()[:64] or None+ join_token = None+ if payload.join_token:+ join_token = db.query(JoinToken).filter(JoinToken.token_hash == _hash_secret(payload.join_token.strip()), JoinToken.used_at.is_(None), JoinToken.expires_at > datetime.now(timezone.utc)).first()+ if not join_token:+ raise HTTPException(status_code=403, detail="The join token is invalid, expired or already used. Copy a fresh command from the dashboard.")+ elif COLLECTOR_ACCESS == "strict" and via == "vpn":+ raise HTTPException(status_code=403, detail="A join token is required. Copy the command from the dashboard's Connect a server button.")+ if join_token:+ # The same host retrying with its token replaces its earlier request until someone approves it.+ db.query(PendingCollector).filter(PendingCollector.join_token_id == join_token.id, PendingCollector.status != "approved").delete(synchronize_session=False)if machine_id:# A restarted collector replaces its own earlier request instead of piling up duplicates.db.query(PendingCollector).filter(PendingCollector.machine_id == machine_id, PendingCollector.ip_address == ip_address, PendingCollector.status != "approved").delete(synchronize_session=False)@@ -622,13 +787,14 @@ def register_collector(payload: CollectorRegisterRequest, request: Request, db=Dpending = PendingCollector(poll_secret_hash=_hash_secret(secret), pairing_code=f"{code[:3]}-{code[3:]}", hostname=hostname,machine_id=machine_id, ip_address=ip_address[:64], os_info=(payload.os_info or "")[:255] or None,+ join_token_id=join_token.id if join_token else None,)db.add(pending)db.commit()return {"request_secret": secret, "pairing_code": pending.pairing_code, "poll_interval_seconds": 5}[email protected]("/monitoring/collectors/register/poll")[email protected]("/monitoring/collectors/register/poll", dependencies=[Depends(require_collector_listener)])def poll_collector_registration(authorization: Annotated[str | None, Header()] = None, db=Depends(get_db)):secret = authorization.removeprefix("Bearer ").strip() if authorization and authorization.startswith("Bearer ") else ""pending = db.query(PendingCollector).filter(PendingCollector.poll_secret_hash == _hash_secret(secret)).first() if secret else None@@ -650,6 +816,10 @@ def poll_collector_registration(authorization: Annotated[str | None, Header()] =return {"status": "approved", "server_id": server.id, "collector_token": token, "check_interval_seconds": server.check_interval_seconds}+def _server_for_machine(db, machine_id: str | None) -> Server | None:+ return db.query(Server).filter(Server.machine_id == machine_id).first() if machine_id else None++def _pending_for_user(pending_id: int, db) -> PendingCollector:pending = db.query(PendingCollector).filter(PendingCollector.id == pending_id, PendingCollector.status != "approved").first()if not pending:@@ -658,18 +828,21 @@ def _pending_for_user(pending_id: int, db) -> PendingCollector:@app.post("/monitoring/pending-collectors/{pending_id}/approve")-def approve_pending_collector(pending_id: int, session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+def approve_pending_collector(pending_id: int, payload: ApproveRequest | None = None, session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):user = _require_admin(session_id, db)pending = _pending_for_user(pending_id, db)- server = None- if pending.machine_id:- server = db.query(Server).filter(Server.machine_id == pending.machine_id).first()+ server = _server_for_machine(db, pending.machine_id)+ if server and not (payload and payload.replace):+ # Approving mints new credentials for that server and locks out its current collector, so never do it silently.+ raise HTTPException(status_code=409, detail=f"This machine matches the existing server {server.hostname}. Approving replaces that server's credentials and signs its current collector out. Confirm to continue.")if not server:server = Server(owner_id=user["id"], hostname=pending.hostname, machine_id=pending.machine_id, ip_address=pending.ip_address, os_info=pending.os_info)db.add(server)db.flush()pending.status = "approved"pending.approved_server_id = server.id+ if pending.join_token_id:+ db.query(JoinToken).filter(JoinToken.id == pending.join_token_id).update({"used_at": datetime.now(timezone.utc)}, synchronize_session=False)db.commit()return {"status": "approved", "server_id": server.id}@@ -691,13 +864,19 @@ def visible_pending_collectors(db, now: datetime | None = None) -> list[dict]:for row in rows:last_poll = row.last_poll_at if row.last_poll_at.tzinfo else row.last_poll_at.replace(tzinfo=timezone.utc)if last_poll >= cutoff:- visible.append({"id": row.id, "hostname": row.hostname, "pairing_code": row.pairing_code, "ip_address": row.ip_address, "os_info": row.os_info, "status": row.status, "created_at": row.created_at.isoformat()})+ existing = _server_for_machine(db, row.machine_id)+ visible.append({+ "id": row.id, "hostname": row.hostname, "pairing_code": row.pairing_code, "ip_address": row.ip_address, "os_info": row.os_info, "status": row.status, "created_at": row.created_at.isoformat(),+ "verified": row.join_token_id is not None,+ "replaces": {"id": existing.id, "hostname": existing.hostname} if existing else None,+ })return visibledef purge_stale_pending(db, now: datetime | None = None) -> None:cutoff = (now or datetime.now(timezone.utc)) - PENDING_RETENTIONdb.query(PendingCollector).filter(PendingCollector.last_poll_at < cutoff).delete(synchronize_session=False)+ db.query(JoinToken).filter(JoinToken.expires_at < (now or datetime.now(timezone.utc)) - timedelta(days=1)).delete(synchronize_session=False)db.commit()@@ -765,6 +944,7 @@ def delete_monitored_server(server_id: int, session_id: Annotated[str | None, Codb.query(HostMetric).filter(HostMetric.server_id == server.id).delete(synchronize_session=False)db.query(ContainerMetric).filter(ContainerMetric.server_id == server.id).delete(synchronize_session=False)db.query(MountMetric).filter(MountMetric.server_id == server.id).delete(synchronize_session=False)+ db.query(SystemEvent).filter(SystemEvent.server_id == server.id).delete(synchronize_session=False)db.query(IncidentAlert).filter(IncidentAlert.server_id == server.id).delete(synchronize_session=False)db.query(PendingCollector).filter(PendingCollector.approved_server_id == server.id).delete(synchronize_session=False)db.delete(server)@@ -785,7 +965,30 @@ def update_check_interval(server_id: int, payload: dict, session_id: Annotated[sreturn {"id": server.id, "check_interval_seconds": server.check_interval_seconds}[email protected]("/monitoring/metrics/ingest")+def record_helper_events(db, server_id: int, raw, now: datetime) -> None:+ """Store the helper's kernel-log events once each; the helper republishes the whole ring buffer."""+ events = validate_helper_events(raw, now)+ if not events:+ return+ known = {key for (key,) in db.query(SystemEvent.event_key).filter(SystemEvent.server_id == server_id, SystemEvent.event_key.in_([event[3] for event in events]))}+ for kind, time, detail, key in events:+ if key not in known:+ known.add(key)+ db.add(SystemEvent(server_id=server_id, time=time, kind=kind, detail=detail, event_key=key))+++def _clean_system_fields(fields: dict) -> None:+ """Drop system-state values of the wrong type; a collector reports None when it can't tell."""+ for key, kind in (("oom_kill_total", int), ("kernel_tainted", int), ("ntp_synced", bool), ("reboot_required", bool)):+ if key in fields and not (isinstance(fields[key], kind) and (kind is bool or not isinstance(fields[key], bool))):+ fields[key] = None+ if "last_shutdown" in fields and fields["last_shutdown"] not in {"clean", "unclean", "unknown"}:+ fields["last_shutdown"] = None+ if "boot_id" in fields:+ fields["boot_id"] = fields["boot_id"][:64] if isinstance(fields["boot_id"], str) and fields["boot_id"] else None++[email protected]("/monitoring/metrics/ingest", dependencies=[Depends(require_collector_listener)])def ingest_metrics(payload: MetricsIngestRequest, authorization: Annotated[str | None, Header()] = None, session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):server = db.query(Server).filter(Server.id == payload.server_id).first()if not server:@@ -804,7 +1007,18 @@ def ingest_metrics(payload: MetricsIngestRequest, authorization: Annotated[str |if server.is_dashboard_host:db.query(Server).filter(Server.id != server.id, Server.is_dashboard_host.is_(True)).update({"is_dashboard_host": False}, synchronize_session=False)now = datetime.now(timezone.utc)- db.add(HostMetric(server_id=server.id, time=now, **{k: v for k, v in payload.host.items() if k in HostMetric.__table__.columns.keys() and k not in {"id", "server_id", "time"}}))+ host_fields = {k: v for k, v in payload.host.items() if k in HostMetric.__table__.columns.keys() and k not in {"id", "server_id", "time"}}+ _clean_system_fields(host_fields)+ previous = db.query(HostMetric).filter(HostMetric.server_id == server.id).order_by(HostMetric.time.desc(), HostMetric.id.desc()).first()+ # Only a collector that found the system helper sends its events; then they replace the OOM counter.+ helper_active = isinstance(payload.host.get("helper_events"), list)+ for kind, detail, key in detect_system_events(previous, host_fields, helper_active):+ db.add(SystemEvent(server_id=server.id, time=now, kind=kind, detail=detail, event_key=key))+ record_helper_events(db, server.id, payload.host.get("helper_events"), now)+ # The helper can finish after a reboot was first noticed; fill in how the last shutdown went.+ if host_fields.get("boot_id") and host_fields.get("last_shutdown") in REBOOT_DETAILS:+ db.query(SystemEvent).filter(SystemEvent.server_id == server.id, SystemEvent.event_key == f"{REBOOT_KEY_PREFIX}{host_fields['boot_id']}").update({"detail": REBOOT_DETAILS[host_fields["last_shutdown"]]}, synchronize_session=False)+ db.add(HostMetric(server_id=server.id, time=now, **host_fields))mounts = payload.host.get("mounts")for mount in mounts if isinstance(mounts, list) else []:if isinstance(mount, dict) and isinstance(mount.get("mount"), str) and mount["mount"]:@@ -816,7 +1030,8 @@ def ingest_metrics(payload: MetricsIngestRequest, authorization: Annotated[str |db.add(ContainerMetric(server_id=server.id, time=now, **allowed))db.flush() # the rules read this check-in back from the history tablesthresholds = get_alert_thresholds(db)- sync_alerts(db, server.id, HOST_RULE_PREFIX, host_rules(db, server, thresholds, active_rule_keys(db, server.id, HOST_RULE_PREFIX)))+ sync_alerts(db, server.id, HOST_RULE_PREFIX, host_rules(db, server, thresholds, active_rule_keys(db, server.id, HOST_RULE_PREFIX), get_mount_thresholds(db)))+ sync_alerts(db, server.id, SYSTEM_RULE_PREFIX, system_rules(db, server, thresholds, now))# The collector sends no containers when Docker is unreachable; keep container alerts as they are rather than resolving them all.if containers:sync_alerts(db, server.id, CONTAINER_RULE_PREFIX, container_rules(db, server.id, containers, thresholds, now))diff --git a/src/backend/models.py b/src/backend/models.pyindex 61e8a73..ca5a06c 100644--- a/src/backend/models.py+++ b/src/backend/models.py@@ -122,10 +122,26 @@ class PendingCollector(Base):os_info: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)status: Mapped[str] = mapped_column(String(16), default="pending", nullable=False) # pending | dismissed | approvedapproved_server_id: Mapped[Optional[int]] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=True)+ # The join token this request presented, if any (shown as "verified" on the dashboard).+ join_token_id: Mapped[Optional[int]] = mapped_column(ForeignKey("join_tokens.id", ondelete="SET NULL"), nullable=True)created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)last_poll_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)+class JoinToken(Base):+ """Minted by the Connect a server button and embedded in its command; proves who asked a collector to join."""++ __tablename__ = "join_tokens"++ id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)+ token_hash: Mapped[str] = mapped_column(String(64), unique=True, nullable=False)+ created_by: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), nullable=False)+ created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)+ expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)+ # Set when the server it created is approved; a used token can't register anything else.+ used_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)++class EnrollmentKey(Base):__tablename__ = "enrollment_keys"@@ -154,6 +170,28 @@ class HostMetric(Base):net_bytes_sent_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)net_bytes_recv_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)system_uptime_seconds: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ # System state read from the kernel; None means the collector couldn't tell.+ boot_id: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)+ oom_kill_total: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)+ kernel_tainted: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)+ ntp_synced: Mapped[Optional[bool]] = mapped_column(Boolean, nullable=True)+ reboot_required: Mapped[Optional[bool]] = mapped_column(Boolean, nullable=True)+ # How the previous boot ended (clean | unclean | unknown), from the optional system helper.+ last_shutdown: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)+++class SystemEvent(Base):+ """Something that happened on a host (reboot, OOM kill, kernel fault), found by comparing check-ins."""++ __tablename__ = "system_events"++ id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)+ time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)+ kind: Mapped[str] = mapped_column(String(32), nullable=False) # reboot | oom_kill | kernel_fault | kernel_error+ # Identifies the occurrence so a snapshot that repeats it doesn't record it twice.+ event_key: Mapped[Optional[str]] = mapped_column(String(128), index=True, nullable=True)+ detail: Mapped[str] = mapped_column(String(512), nullable=False)class MountMetric(Base):diff --git a/src/backend/rules.py b/src/backend/rules.pyindex 62df58e..617b19f 100644--- a/src/backend/rules.py+++ b/src/backend/rules.py@@ -5,14 +5,16 @@ opened, refreshed and resolved alerts."""from __future__ import annotations-from datetime import datetime, timedelta+import re+from datetime import datetime, timedelta, timezonefrom sqlalchemy import func-from .alerts import CONTAINER_RULE_PREFIX, CRITICAL, WARNING-from .models import ContainerMetric, HostMetric, MountMetric+from .alerts import CONTAINER_RULE_PREFIX, CRITICAL, INFO, WARNING+from .models import ContainerMetric, HostMetric, MountMetric, SystemEventHOST_RULE_PREFIX = "host:"+SYSTEM_RULE_PREFIX = "system:"DEFAULT_THRESHOLDS: dict[str, float | int] = {"disk_warning_pct": 85, "disk_critical_pct": 95,@@ -23,6 +25,8 @@ DEFAULT_THRESHOLDS: dict[str, float | int] = {# How many check-ins in a row a host metric must breach before an alert opens."sustained_checks": 3,"restart_loop_count": 3, "restart_loop_window_minutes": 30,+ # How long a reboot or OOM kill keeps its alert open after it happened.+ "event_window_minutes": 60,}_LIMITS: dict[str, tuple[float, float]] = {@@ -30,8 +34,9 @@ _LIMITS: dict[str, tuple[float, float]] = {"load_warning_per_core": (0.1, 50), "load_critical_per_core": (0.1, 50),"sustained_checks": (1, 20),"restart_loop_count": (1, 100), "restart_loop_window_minutes": (5, 1440),+ "event_window_minutes": (5, 1440),}-_INTEGER_KEYS = {"sustained_checks", "restart_loop_count", "restart_loop_window_minutes"}+_INTEGER_KEYS = {"sustained_checks", "restart_loop_count", "restart_loop_window_minutes", "event_window_minutes"}def validate_thresholds(values: dict, current: dict) -> dict:@@ -57,6 +62,25 @@ def validate_thresholds(values: dict, current: dict) -> dict:return merged+def validate_mount_thresholds(value) -> dict[str, dict[str, float]]:+ """Per-mount disk thresholds as {mount: {"disk_warning_pct", "disk_critical_pct"}}. Raises ValueError."""+ if not isinstance(value, dict):+ raise ValueError("Mount thresholds must be an object keyed by mount path.")+ cleaned: dict[str, dict[str, float]] = {}+ for mount, pair in value.items():+ if not isinstance(mount, str) or not mount.startswith("/") or len(mount) > 255:+ raise ValueError("Each mount must be an absolute path such as /media/data.")+ if not isinstance(pair, dict) or set(pair) != {"disk_warning_pct", "disk_critical_pct"}:+ raise ValueError(f"{mount} needs a warning and a critical threshold.")+ for key, number in pair.items():+ if isinstance(number, bool) or not isinstance(number, (int, float)) or not 1 <= number <= 100:+ raise ValueError(f"{mount}: {key} must be a number between 1 and 100.")+ if pair["disk_warning_pct"] >= pair["disk_critical_pct"]:+ raise ValueError(f"{mount}: the warning threshold must be below its critical threshold.")+ cleaned[mount] = {key: pair[key] for key in ("disk_warning_pct", "disk_critical_pct")}+ return cleaned++def _level(value: float | None, warning: float, critical: float | None) -> int:"""0 = fine, 1 = warning, 2 = critical."""if value is None:@@ -85,7 +109,7 @@ def _gb(value: float | None) -> str:return f"{value / 1024:.1f} TB" if value >= 1024 else f"{value:.0f} GB"-def host_rules(db, server, thresholds: dict, active_keys: set[str]) -> dict[str, tuple[str, str, str]]:+def host_rules(db, server, thresholds: dict, active_keys: set[str], mount_overrides: dict | None = None) -> dict[str, tuple[str, str, str]]:"""Evaluate the host rules against the server's most recent check-ins."""sustained = thresholds["sustained_checks"]samples = db.query(HostMetric).filter(HostMetric.server_id == server.id).order_by(HostMetric.time.desc(), HostMetric.id.desc()).limit(sustained).all()@@ -118,13 +142,15 @@ def host_rules(db, server, thresholds: dict, active_keys: set[str]) -> dict[str,mount = _escape(raw_mount)history = [by_mount[raw_mount].get(time) for time in times]size = f"{_gb(row.used_gb)} of {_gb(row.total_gb)} used"- checks.append((f"{HOST_RULE_PREFIX}disk:{raw_mount}", [r.used_pct if r else None for r in history], thresholds["disk_warning_pct"], thresholds["disk_critical_pct"],+ disk = {"disk_warning_pct": thresholds["disk_warning_pct"], "disk_critical_pct": thresholds["disk_critical_pct"], **(mount_overrides or {}).get(raw_mount, {})}+ checks.append((f"{HOST_RULE_PREFIX}disk:{raw_mount}", [r.used_pct if r else None for r in history], disk["disk_warning_pct"], disk["disk_critical_pct"],f"Disk usage on {host}:{mount} is {{value:.0f}}%", f"{mount} is {{value:.1f}}% full ({size}). Free up space before it fills completely."))checks.append((f"{HOST_RULE_PREFIX}inodes:{raw_mount}", [r.inodes_pct if r else None for r in history], thresholds["inodes_warning_pct"], thresholds["inodes_critical_pct"],f"Inode usage on {host}:{mount} is {{value:.0f}}%", f"{{value:.1f}}% of inodes on {mount} are used. Many small files (caches, logs, mail queues) are the usual cause."))else:# Collectors from before per-mount reporting only send the root filesystem.- checks.append((f"{HOST_RULE_PREFIX}disk:/", [s.disk_usage_pct for s in samples], thresholds["disk_warning_pct"], thresholds["disk_critical_pct"],+ root = {"disk_warning_pct": thresholds["disk_warning_pct"], "disk_critical_pct": thresholds["disk_critical_pct"], **(mount_overrides or {}).get("/", {})}+ checks.append((f"{HOST_RULE_PREFIX}disk:/", [s.disk_usage_pct for s in samples], root["disk_warning_pct"], root["disk_critical_pct"],f"Disk usage on {host}:/ is {{value:.0f}}%", "/ is {value:.1f}% full. Free up space before it fills completely."))checks.append((f"{HOST_RULE_PREFIX}inodes:/", [s.disk_inodes_pct for s in samples], thresholds["inodes_warning_pct"], thresholds["inodes_critical_pct"],f"Inode usage on {host}:/ is {{value:.0f}}%", "{value:.1f}% of inodes on / are used. Many small files (caches, logs, mail queues) are the usual cause."))@@ -137,6 +163,112 @@ def host_rules(db, server, thresholds: dict, active_keys: set[str]) -> dict[str,return firing+# Bits of /proc/sys/kernel/tainted that mean the kernel hit a fault; the rest (proprietary or+# out-of-tree modules, livepatch) are normal on many healthy machines.+TAINT_FAULTS = {4: "machine check", 5: "bad page", 7: "kernel oops", 9: "kernel warning", 14: "soft lockup"}+++def taint_faults(value: int | None) -> list[str]:+ return [name for bit, name in TAINT_FAULTS.items() if value and value >> bit & 1]+++REBOOT_DETAIL = "The host rebooted"+CLEAN_REBOOT_DETAIL = "The host rebooted after a clean shutdown"+UNCLEAN_REBOOT_DETAIL = "The host restarted after an unclean shutdown (crash, power loss or forced reset)"+REBOOT_DETAILS = {"clean": CLEAN_REBOOT_DETAIL, "unclean": UNCLEAN_REBOOT_DETAIL}+REBOOT_KEY_PREFIX = "reboot:"+++def detect_system_events(previous: HostMetric | None, host: dict, helper_active: bool = False) -> list[tuple[str, str, str | None]]:+ """(kind, detail, event key) for what changed between the previous check-in and this one's host fields.++ With the system helper running, OOM kills come from its kernel log events instead of the counter.+ """+ if previous is None:+ return []+ events: list[tuple[str, str, str | None]] = []+ boot_id = host.get("boot_id")+ rebooted = bool(boot_id and previous.boot_id and boot_id != previous.boot_id)+ if rebooted:+ events.append(("reboot", REBOOT_DETAILS.get(host.get("last_shutdown"), REBOOT_DETAIL), f"{REBOOT_KEY_PREFIX}{boot_id}"))+ # Kernel counters restart with each boot, so compare them only within one boot.+ oom_total = host.get("oom_kill_total")+ if not helper_active and not rebooted and oom_total is not None and previous.oom_kill_total is not None and oom_total > previous.oom_kill_total:+ killed = oom_total - previous.oom_kill_total+ events.append(("oom_kill", f"The kernel OOM killer stopped {killed} {'process' if killed == 1 else 'processes'}", None))+ faults = taint_faults(host.get("kernel_tainted"))+ new_faults = [fault for fault in faults if rebooted or fault not in taint_faults(previous.kernel_tainted)]+ if new_faults:+ events.append(("kernel_fault", f"The kernel reported: {', '.join(new_faults)}", None))+ return events+++HELPER_EVENT_KINDS = {"oom_kill", "kernel_error"}+HELPER_EVENTS_MAX = 50+HELPER_EVENT_MAX_AGE = timedelta(days=7)+_EVENT_KEY = re.compile(r"[A-Za-z0-9:._-]{1,128}")+++def validate_helper_events(raw, now: datetime) -> list[tuple[str, datetime, str, str]]:+ """(kind, time, detail, key) for the well-formed, recent events a collector forwarded from the helper.++ The collector reads them from a file, so each field is checked again here and anything odd is dropped.+ """+ if not isinstance(raw, list):+ return []+ events = []+ for item in raw[:HELPER_EVENTS_MAX]:+ if not isinstance(item, dict) or item.get("kind") not in HELPER_EVENT_KINDS:+ continue+ detail, key, when = item.get("detail"), item.get("key"), item.get("time")+ if not all(isinstance(value, str) for value in (detail, key, when)) or not _EVENT_KEY.fullmatch(key):+ continue+ try:+ time = datetime.fromisoformat(when)+ except ValueError:+ continue+ if time.tzinfo is None:+ time = time.replace(tzinfo=timezone.utc)+ if not now - HELPER_EVENT_MAX_AGE <= time <= now + timedelta(minutes=5):+ continue+ events.append((item["kind"], time, "".join(char if char.isprintable() else "?" for char in detail).strip()[:300], key))+ return events+++def system_rules(db, server, thresholds: dict, now: datetime) -> dict[str, tuple[str, str, str]]:+ """Evaluate the system-state rules (reboots, OOM kills, kernel faults, clock sync, pending reboot)."""+ sustained = thresholds["sustained_checks"]+ samples = db.query(HostMetric).filter(HostMetric.server_id == server.id).order_by(HostMetric.time.desc(), HostMetric.id.desc()).limit(sustained).all()+ if not samples:+ return {}+ latest, host = samples[0], server.hostname+ since = now - timedelta(minutes=thresholds["event_window_minutes"])+ events = db.query(SystemEvent).filter(SystemEvent.server_id == server.id, SystemEvent.time >= since).order_by(SystemEvent.time.desc()).all()+ firing: dict[str, tuple[str, str, str]] = {}++ if reboots := [event for event in events if event.kind == "reboot"]:+ unclean = reboots[0].detail == UNCLEAN_REBOOT_DETAIL+ firing[f"{SYSTEM_RULE_PREFIX}reboot"] = (+ CRITICAL if unclean else INFO, f"{host} restarted after an unclean shutdown" if unclean else f"{host} rebooted",+ f"The host restarted at {reboots[0].time:%Y-%m-%d %H:%M} UTC. " + (+ "The previous shutdown was not recorded as clean, which points to a crash, power loss or forced reset." if unclean+ else "If you didn't expect this, check for a crash or power loss."),+ )+ if oom := [event for event in events if event.kind == "oom_kill"]:+ firing[f"{SYSTEM_RULE_PREFIX}oom"] = (+ CRITICAL if len(oom) >= 3 else WARNING, f"{host} is killing processes to free memory",+ f"The kernel OOM killer fired {len(oom)} time(s) in the last {thresholds['event_window_minutes']} minutes ({'; '.join(dict.fromkeys(event.detail for event in oom[:3]))}). Check which containers or processes are using the most memory.",+ )+ if faults := taint_faults(latest.kernel_tainted):+ # The taint flags stay set until the next boot, so this alert clears on reboot.+ firing[f"{SYSTEM_RULE_PREFIX}kernel"] = (WARNING, f"Kernel on {host} reported faults", f"The kernel has flagged: {', '.join(faults)}. Check dmesg on the host for details; the flag clears after a reboot.")+ if len(samples) >= sustained and all(sample.ntp_synced is False for sample in samples):+ firing[f"{SYSTEM_RULE_PREFIX}ntp"] = (WARNING, f"Clock on {host} is not synchronized", "The system clock is not synced to a time source. Drifting clocks break TLS, logs and scheduled jobs. Check that chrony, systemd-timesyncd or ntpd is running.")+ if latest.reboot_required:+ firing[f"{SYSTEM_RULE_PREFIX}reboot_required"] = (INFO, f"{host} needs a reboot", "Installed updates (often a new kernel) only take effect after a restart.")+ return firing++def container_rules(db, server_id: int, containers: list[dict], thresholds: dict, now: datetime) -> dict[str, tuple[str, str, str]]:"""Evaluate the container rules for one check-in's container records."""window = thresholds["restart_loop_window_minutes"]diff --git a/src/backend/seed_user.py b/src/backend/seed_user.pyindex f14d387..a186ae4 100644--- a/src/backend/seed_user.py+++ b/src/backend/seed_user.py@@ -1,58 +1,75 @@+"""Kreira administratorski račun iz naredbenog retka, za razvoj ili oporavak pristupa.++Podaci se čitaju iz okoline ili se traže interaktivno (lozinka se ne prikazuje i nikad se ne ispisuje):++ [email protected] SEED_ADMIN_PASSWORD='...' python src/backend/seed_user.py+ docker compose exec -it backend python src/backend/seed_user.py++Vrijedi ista pravila kao pri registraciji (lozinka od najmanje 16 znakova). Ova skripta zaobilazi+setup token, pa je koristi samo onaj tko ionako ima pristup bazi.+"""+import getpassimport osimport sys# Dodaj roditeljski direktorij (src/) u Python path kako bi 'backend' postao dostupan paketsys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))-from passlib.context import CryptContext+import bcrypt# Sada uvozi preko backend paketa (što rješava .config u db.py)+from backend.auth import validate_passwordfrom backend.db import SessionLocalfrom backend.models import User-pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")++def _ask(name, prompt, secret=False):+ value = os.getenv(name, "").strip()+ if value or not sys.stdin.isatty():+ return value+ return (getpass.getpass if secret else input)(prompt).strip()def create_initial_user():- email = "[email protected]"- raw_password = "adminspass123"- name = "Admin User"- role = "admin"+ email = _ask("SEED_ADMIN_EMAIL", "Email: ").lower()+ password = _ask("SEED_ADMIN_PASSWORD", "Lozinka (najmanje 16 znakova): ", secret=True)+ name = os.getenv("SEED_ADMIN_NAME", "").strip() or "Admin User"++ if not email or not password:+ print("❌ Postavi SEED_ADMIN_EMAIL i SEED_ADMIN_PASSWORD ili pokreni skriptu interaktivno (docker compose exec -it).")+ return 1+ try:+ validate_password(password)+ except ValueError as e:+ print(f"❌ {e}")+ return 1db = SessionLocal()try:- existing_user = db.query(User).filter(User.email == email).first()- if existing_user:+ if db.query(User).filter(User.email == email).first():print(f"⚠️ Korisnik s emailom '{email}' već postoji u bazi!")- return-- hashed_password = pwd_context.hash(raw_password)+ return 1new_user = User(email=email,name=name,- password_hash=hashed_password,- role=role,+ password_hash=bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt()).decode("utf-8"),+ role="admin",is_active=True,)-db.add(new_user)db.commit()db.refresh(new_user)- print("--------------------------------------------------")- print("✅ Korisnik je uspješno dodan u bazu!")- print(f" ID: {new_user.id}")- print(f" Email: {new_user.email}")- print(f" Lozinka: {raw_password}")- print("--------------------------------------------------")-+ print(f"✅ Administrator je dodan u bazu (ID {new_user.id}, {new_user.email}).")+ return 0except Exception as e:db.rollback()print(f"❌ Greška: {e}")+ return 1finally:db.close()if __name__ == "__main__":- create_initial_user()\ No newline at end of file+ sys.exit(create_initial_user())diff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsxindex c1a8da2..f7c1bd1 100644--- a/src/frontend/src/App.jsx+++ b/src/frontend/src/App.jsx@@ -299,21 +299,23 @@ function AuthCard({ title, subtitle, children, footerText, footerLink, footerLin}// null while loading; the backend decides (ALLOW_REGISTRATION, or no accounts exist yet).+const MIN_PASSWORD_LENGTH = 16+function useRegistrationEnabled() {- const [enabled, setEnabled] = useState(null)+ const [config, setConfig] = useState({ enabled: null, bootstrapRequired: false })useEffect(() => {let active = trueapiRequest('/auth/config', { method: 'GET' })- .then((data) => { if (active) setEnabled(Boolean(data.registration_enabled)) })- .catch(() => { if (active) setEnabled(true) })+ .then((data) => { if (active) setConfig({ enabled: Boolean(data.registration_enabled), bootstrapRequired: Boolean(data.bootstrap_required) }) })+ .catch(() => { if (active) setConfig({ enabled: true, bootstrapRequired: false }) })return () => { active = false }}, [])- return enabled+ return config}function LoginPage({ onLogin }) {const navigate = useNavigate()- const registrationEnabled = useRegistrationEnabled()+ const { enabled: registrationEnabled } = useRegistrationEnabled()const [email, setEmail] = useState('')const [password, setPassword] = useState('')const [error, setError] = useState('')@@ -388,8 +390,8 @@ function LoginPage({ onLogin }) {function RegisterPage({ onRegister }) {const navigate = useNavigate()- const registrationEnabled = useRegistrationEnabled()- const [form, setForm] = useState({ name: '', email: '', password: '' })+ const { enabled: registrationEnabled, bootstrapRequired } = useRegistrationEnabled()+ const [form, setForm] = useState({ name: '', email: '', password: '', bootstrapToken: '' })const [error, setError] = useState('')const handleChange = (event) => {@@ -404,8 +406,8 @@ function RegisterPage({ onRegister }) {const trimmedName = form.name.trim()const trimmedEmail = form.email.trim()- if (!trimmedName || !trimmedEmail || form.password.length < 6) {- setError('Please provide a name, email, and a password with at least 6 characters.')+ if (!trimmedName || !trimmedEmail || form.password.length < MIN_PASSWORD_LENGTH) {+ setError(`Please provide a name, an email, and a password of at least ${MIN_PASSWORD_LENGTH} characters.`)return}@@ -416,6 +418,7 @@ function RegisterPage({ onRegister }) {name: trimmedName,email: trimmedEmail,password: form.password,+ bootstrap_token: bootstrapRequired ? form.bootstrapToken.trim() : undefined,}),})@@ -471,11 +474,31 @@ function RegisterPage({ onRegister }) {value={form.password}onChange={handleChange}className="w-full rounded-2xl border border-neutral-700 bg-neutral-900/70 px-4 py-3 text-white outline-none transition focus:border-emerald-400"- placeholder="At least 6 characters"+ placeholder={`At least ${MIN_PASSWORD_LENGTH} characters`}+ minLength={MIN_PASSWORD_LENGTH}+ autoComplete="new-password"required/>+ <span className="mt-2 block text-xs text-[#7f8589]">A few random words make a good passphrase, for example “correct horse battery staple”. Up to 72 bytes.</span></label>+ {bootstrapRequired && (+ <label className="block">+ <span className="mb-2 block text-sm text-[#bcbcbc]">Setup token</span>+ <input+ type="password"+ name="bootstrapToken"+ value={form.bootstrapToken}+ onChange={handleChange}+ autoComplete="off"+ className="w-full rounded-2xl border border-neutral-700 bg-neutral-900/70 px-4 py-3 font-mono text-sm text-white outline-none transition focus:border-emerald-400"+ placeholder="hlb_…"+ required+ />+ <span className="mt-2 block text-xs text-[#7f8589]">This install has no accounts yet, so the first one needs its setup token. Find it in the backend log: <code>docker compose logs backend | grep "setup token"</code>, or use the value of <code>ADMIN_BOOTSTRAP_TOKEN</code> if you set one. The first account becomes the admin.</span>+ </label>+ )}+{error && <div className="rounded-xl border border-red-500/30 bg-red-500/10 px-3 py-2 text-sm text-red-300">{error}</div>}<button@@ -530,7 +553,7 @@ function thresholdLevel(value, warning, critical) {return value >= critical ? 2 : value >= warning ? 1 : 0}-function StorageBars({ mounts, thresholds }) {+function StorageBars({ mounts, thresholds, mountThresholds }) {if (!mounts.length) return nullconst t = { disk_warning_pct: 85, disk_critical_pct: 95, inodes_warning_pct: 85, inodes_critical_pct: 95, ...thresholds }return (@@ -538,7 +561,8 @@ function StorageBars({ mounts, thresholds }) {<div className="mb-2 text-xs uppercase tracking-[0.2em] text-[#7f8589]">Storage</div><div className="space-y-2">{mounts.map((mount) => {- const style = DISK_LEVEL_STYLES[thresholdLevel(mount.used_pct, t.disk_warning_pct, t.disk_critical_pct)]+ const own = mountThresholds?.[mount.mount]+ const style = DISK_LEVEL_STYLES[thresholdLevel(mount.used_pct, own?.disk_warning_pct ?? t.disk_warning_pct, own?.disk_critical_pct ?? t.disk_critical_pct)]const inodeLevel = thresholdLevel(mount.inodes_pct, t.inodes_warning_pct, t.inodes_critical_pct)const used = Math.min(100, Math.max(0, mount.used_pct ?? 0))return (@@ -588,6 +612,18 @@ function useStoredIdSet(storageKey) {return [ids, toggle, (list) => setIds(new Set(list))]}+// Like useStoredIdSet but forgotten on reload, so these sections always start collapsed.+function useIdSet() {+ const [ids, setIds] = useState(() => new Set())+ const toggle = (id) => setIds((current) => {+ const next = new Set(current)+ if (next.has(id)) next.delete(id)+ else next.add(id)+ return next+ })+ return [ids, toggle]+}+function Chevron({ open, className = '' }) {return <span aria-hidden="true" className={`inline-block text-[10px] text-[#7f8589] transition-transform ${open ? 'rotate-90' : ''} ${className}`}>▶</span>}@@ -626,6 +662,60 @@ function ContainerSection({ server, open, onToggle }) {)}+const TAINT_FAULT_BITS = [4, 5, 7, 9, 14]+const SYSTEM_EVENT_LABELS = { reboot: 'Reboot', oom_kill: 'OOM kill', kernel_fault: 'Kernel fault', kernel_error: 'Kernel error' }+const SYSTEM_EVENT_CLASSES = { reboot: 'text-sky-300', oom_kill: 'text-amber-300', kernel_fault: 'text-red-300', kernel_error: 'text-amber-300' }+const CHIP_CLASSES = { ok: 'text-emerald-300', warn: 'text-amber-300', info: 'text-sky-300', unknown: 'text-[#7f8589]' }++function formatAgo(value) {+ const seconds = Math.max(0, Math.floor((Date.now() - new Date(value).getTime()) / 1000))+ if (seconds < 90) return 'just now'+ if (seconds < 5400) return `${Math.round(seconds / 60)}m ago`+ if (seconds < 172800) return `${Math.round(seconds / 3600)}h ago`+ return `${Math.round(seconds / 86400)}d ago`+}++function systemChips(host, events) {+ const oomRecently = events.some((event) => event.kind === 'oom_kill' && Date.now() - new Date(event.time).getTime() < 24 * 3600 * 1000)+ const faulty = host.kernel_tainted != null && TAINT_FAULT_BITS.some((bit) => (host.kernel_tainted >> bit) & 1)+ return [+ ['Booted', host.system_uptime_seconds == null ? 'unknown' : formatAgo(Date.now() - host.system_uptime_seconds * 1000), host.system_uptime_seconds == null ? 'unknown' : events.some((event) => event.kind === 'reboot' && Date.now() - new Date(event.time).getTime() < 3600 * 1000) ? 'info' : 'ok'],+ ['Last shutdown', host.last_shutdown == null || host.last_shutdown === 'unknown' ? 'unknown' : host.last_shutdown, host.last_shutdown === 'clean' ? 'ok' : host.last_shutdown === 'unclean' ? 'warn' : 'unknown'],+ ['Clock', host.ntp_synced == null ? 'unknown' : host.ntp_synced ? 'synced' : 'not synced', host.ntp_synced == null ? 'unknown' : host.ntp_synced ? 'ok' : 'warn'],+ ['OOM kills (24h)', host.oom_kill_total == null ? 'unknown' : oomRecently ? 'yes' : 'none', host.oom_kill_total == null ? 'unknown' : oomRecently ? 'warn' : 'ok'],+ ['Kernel', host.kernel_tainted == null ? 'unknown' : faulty ? 'faults reported' : 'clean', host.kernel_tainted == null ? 'unknown' : faulty ? 'warn' : 'ok'],+ ['Updates', host.reboot_required == null ? 'unknown' : host.reboot_required ? 'reboot needed' : 'applied', host.reboot_required == null ? 'unknown' : host.reboot_required ? 'info' : 'ok'],+ ]+}++function SystemSection({ server, open, onToggle }) {+ const host = server.host+ const events = server.system_events || []+ const chips = systemChips(host, events)+ return (+ <div className="mt-4 rounded-xl border border-neutral-800">+ <button type="button" onClick={onToggle} aria-expanded={open} className="flex w-full flex-wrap items-center gap-x-4 gap-y-1 rounded-xl px-3 py-2.5 text-left text-sm transition hover:bg-white/3">+ <Chevron open={open} />+ <span className="text-white">System</span>+ {chips.map(([label, value, tone]) => <span key={label} className="whitespace-nowrap text-xs text-[#7f8589]">{label} <span className={`font-medium ${CHIP_CLASSES[tone]}`}>{value}</span></span>)}+ </button>+ {open && (+ <div className="border-t border-neutral-800 px-3 py-3">+ <div className="mb-2 text-xs uppercase tracking-[0.2em] text-[#7f8589]">Recent events · 7 days</div>+ {events.length ? <ul className="space-y-1.5 text-sm">{events.map((event, index) => (+ <li key={`${event.time}-${event.kind}-${index}`} className="flex flex-wrap items-baseline gap-x-3 gap-y-0.5">+ <span className="w-24 shrink-0 text-xs text-[#7f8589]" title={new Date(event.time).toLocaleString()}>{formatAgo(event.time)}</span>+ <span className={`w-24 shrink-0 text-xs font-medium ${SYSTEM_EVENT_CLASSES[event.kind] || 'text-[#cfcfcf]'}`}>{SYSTEM_EVENT_LABELS[event.kind] || event.kind}</span>+ <span className="min-w-0 flex-1 basis-60 break-words text-[#bcbcbc]">{event.detail}</span>+ </li>+ ))}</ul> : <p className="text-sm text-[#9aa0a6]">No reboots, OOM kills or kernel faults in the last 7 days.</p>}+ {host.last_shutdown == null && <p className="mt-3 text-xs text-[#7f8589]">“Unknown” means the collector couldn't read that value on this host, not that anything is wrong. Last shutdown, OOM process names and kernel errors need the optional system helper (see SECURITY.md).</p>}+ </div>+ )}+ </div>+ )+}+const HEALTH_CLASSES = { healthy: 'text-[#7f8589]', unhealthy: 'text-red-300', starting: 'text-amber-300' }const ALERT_STYLES = {@@ -688,7 +778,8 @@ const refresh = async () => {const servers = overview.servers || []// Remembered per browser: which server cards are collapsed and which container lists are open.const [collapsedServers, toggleServer, setCollapsedServers] = useStoredIdSet('homelab.collapsedServers')- const [openContainerLists, toggleContainerList] = useStoredIdSet('homelab.openContainerLists')+ const [openContainerLists, toggleContainerList] = useIdSet()+ const [openSystemLists, toggleSystemList] = useIdSet()const allCollapsed = servers.length > 0 && servers.every((server) => collapsedServers.has(server.id))const dashboardHost = servers.find((server) => server.is_dashboard_host)const dashboardHostId = dashboardHost?.id@@ -787,7 +878,8 @@ const refresh = async () => {</div>{!collapsed && <>{metric ? <div className="mt-5 grid grid-cols-2 gap-3 lg:grid-cols-4">{[['CPU', metric.cpu_utilization_pct, '%'], ['RAM', metric.ram_pct, '%'], [fullest ? `Disk · ${fullest.mount}` : 'Disk', fullest?.used_pct, '%'], ['Load (1m)', metric.load_1m, '']].map(([label, value, unit]) => <div key={label} className="rounded-xl bg-[#171b1f] p-3"><div className="truncate text-xs text-[#9aa0a6]" title={label}>{label}</div><div className="mt-1 text-xl font-semibold text-white">{value == null ? '—' : `${Number(value).toFixed(1)}${unit}`}</div></div>)}</div> : <div className="mt-4 rounded-xl bg-[#171b1f] p-4 text-sm text-[#9aa0a6]">Collector has not sent a measurement yet.</div>}- {metric && <StorageBars mounts={mounts} thresholds={overview.disk_thresholds} />}+ {metric && <StorageBars mounts={mounts} thresholds={overview.disk_thresholds} mountThresholds={overview.mount_thresholds} />}+ {metric && <SystemSection server={server} open={openSystemLists.has(server.id)} onToggle={() => toggleSystemList(server.id)} />}{server.containers.length > 0 && <ContainerSection server={server} open={openContainerLists.has(server.id)} onToggle={() => toggleContainerList(server.id)} />}<div className="mt-3 text-xs text-[#7f8589]">Last check: {server.last_seen_at ? new Date(server.last_seen_at).toLocaleString() : 'Never'}</div></>}@@ -879,32 +971,105 @@ function EnrollmentKeySettings() {)}+function buildJoinCommand({ url, token, collectorImage, proxyImage }) {+ return `docker compose -p homelab-collector -f - up -d <<'EOF'+services:+ collector:+ image: ${collectorImage}+ container_name: homelab-collector+ restart: unless-stopped+ environment:+ HOMELAB_API_URL: ${url}+ HOMELAB_JOIN_TOKEN: ${token}+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sock+ volumes:+ - /proc:/host/proc:ro+ - /:/hostfs:ro+ - collector_state:/var/lib/homelab-collector+ - docker_proxy_socket:/run/docker-proxy:ro+ cap_drop: [ALL]+ read_only: true+ security_opt: [no-new-privileges:true]+ docker-proxy:+ image: ${proxyImage}+ container_name: homelab-docker-proxy+ restart: unless-stopped+ user: "0:0"+ network_mode: none+ cap_drop: [ALL]+ read_only: true+ security_opt: [no-new-privileges:true]+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy+volumes:+ collector_state:+ docker_proxy_socket:+EOF`+}+function ConnectServerPanel() {- // API_BASE is usually relative ('/api'); collectors need the absolute address this dashboard is reached at.- const [apiUrl, setApiUrl] = useState(() => new URL(API_BASE, window.location.origin).href.replace(/\/$/, ''))- const url = apiUrl.trim().replace(/\/$/, '') || API_BASE+ const [info, setInfo] = useState(null)+ const [apiUrl, setApiUrl] = useState('')+ const [token, setToken] = useState(null)+ const [expiresAt, setExpiresAt] = useState(null)+ const [busy, setBusy] = useState(false)+ const [error, setError] = useState('')++ useEffect(() => {+ apiRequest('/monitoring/install-info', { method: 'GET' })+ .then((data) => {+ setInfo(data)+ // API_BASE is usually relative ('/api'); a collector needs an absolute address. The VPN address wins when one is configured.+ setApiUrl(data.collector_url || new URL(API_BASE, window.location.origin).href.replace(/\/$/, ''))+ })+ .catch((requestError) => setError(requestError.message || 'Could not load the install settings.'))+ }, [])++ const generate = async () => {+ setBusy(true); setError('')+ try {+ const data = await apiRequest('/monitoring/join-tokens', { method: 'POST' })+ setToken(data.join_token)+ setExpiresAt(Date.now() + data.expires_in_seconds * 1000)+ } catch (requestError) { setError(requestError.message || 'Could not create a join token.') }+ finally { setBusy(false) }+ }++ const url = apiUrl.trim().replace(/\/$/, '')const isLocalhost = /\/\/(localhost|127\.0\.0\.1)(:|\/|$)/.test(url)- const dockerRun = `docker run -d --name homelab-collector --restart unless-stopped \\- -e HOMELAB_API_URL=${url} \\- -v /proc:/host/proc:ro -v /:/hostfs:ro \\- -v /var/run/docker.sock:/var/run/docker.sock:ro \\- -v homelab-collector:/var/lib/homelab-collector \\- python:3.12-alpine sh -c 'wget -qO /tmp/collector.py "$HOMELAB_API_URL/collector.py" && exec python -u /tmp/collector.py'`+ const ready = info?.collector_image && info?.proxy_image+ const minutes = expiresAt ? Math.max(1, Math.round((expiresAt - Date.now()) / 60000)) : null+ const command = token && ready ? buildJoinCommand({ url, token, collectorImage: info.collector_image, proxyImage: info.proxy_image }) : nullreturn (<div className="mt-4 rounded-2xl border border-cyan-300/20 bg-cyan-300/5 p-4"><div className="font-medium text-cyan-100">Connect a server</div>- <p className="mt-1 text-sm text-[#bcbcbc]">Run this once on the machine you want to monitor. Within a few seconds it shows up below under <span className="text-white">New servers detected</span>. Click <span className="text-white">Add</span> and it starts reporting. No keys, config files or restarts.</p>+ <p className="mt-1 text-sm text-[#bcbcbc]">Generate a command, run it once on the machine you want to monitor, then click <span className="text-white">Add</span> when it appears below under <span className="text-white">New servers detected</span>. The command carries a single-use token that expires, so only you can add servers.</p>- <label className="mt-4 block text-xs text-[#9aa0a6]">Dashboard address the new machine will use- <input value={apiUrl} onChange={(e) => setApiUrl(e.target.value)} className="mt-1 w-full rounded-xl border border-neutral-700 bg-[#0d1013] px-3 py-2 text-sm text-white" aria-describedby="api-url-hint" />+ <label className="mt-4 block text-xs text-[#9aa0a6]">Address the new machine will use to reach the dashboard+ <input value={apiUrl} onChange={(e) => { setApiUrl(e.target.value); setToken(null) }} className="mt-1 w-full rounded-xl border border-neutral-700 bg-[#0d1013] px-3 py-2 text-sm text-white" aria-describedby="api-url-hint" /></label>- {isLocalhost- ? <p id="api-url-hint" className="mt-1 text-[11px] text-amber-200">Other machines can't reach <code>localhost</code>. Replace it with this machine's LAN IP, Tailscale address or domain (e.g. <code>http://192.168.1.10:5173/api</code>).</p>- : <p id="api-url-hint" className="mt-1 text-[11px] text-[#7f8589]">Prefilled with the address you opened this dashboard at.</p>}+ {info?.collector_url+ ? <p id="api-url-hint" className="mt-1 text-[11px] text-[#7f8589]">Your configured collector (VPN) address.</p>+ : info?.strict+ ? <p id="api-url-hint" className="mt-1 text-[11px] text-amber-200">Collectors can't use the public address. Set <code>COLLECTOR_PUBLIC_URL</code> to the collector listener on your VPN (for example <code>http://10.8.0.1:8081/api</code>).</p>+ : isLocalhost+ ? <p id="api-url-hint" className="mt-1 text-[11px] text-amber-200">Other machines can't reach <code>localhost</code>. Use this machine's LAN IP, VPN address or domain (e.g. <code>http://192.168.1.10:5173/api</code>).</p>+ : <p id="api-url-hint" className="mt-1 text-[11px] text-[#7f8589]">Prefilled with the address you opened this dashboard at. Over plain HTTP, use it only on a trusted LAN or VPN.</p>}++ {info && !ready && <div className="mt-4 rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-sm text-amber-200">No collector images are configured, so there is nothing safe to install from. Set <code>COLLECTOR_IMAGE</code> and <code>DOCKER_PROXY_IMAGE</code> in <code>.env</code> to your published, digest-pinned images (see SECURITY.md). The dashboard never serves collector code itself.</div>}++ <div className="mt-4 flex flex-wrap items-center gap-3">+ <button type="button" onClick={generate} disabled={busy || !ready || !url} className="rounded-xl bg-white px-4 py-2 text-sm font-medium text-slate-950 disabled:cursor-not-allowed disabled:opacity-40">{busy ? 'Generating…' : token ? 'Generate a new command' : 'Generate command'}</button>+ {token && <span className="text-xs text-[#9aa0a6]">Single use · expires in about {minutes} min. A new command makes this one's token stop working once it is used.</span>}+ </div>+ {error && <div className="mt-3 rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-sm text-amber-200">{error}</div>}- <CopyableCommand label="Run on the new machine" text={dockerRun} />- <p className="mt-2 text-[11px] text-[#7f8589]">The machine running this dashboard is detected automatically. The collector downloads itself from this dashboard on every start, so it stays up to date. Over plain HTTP, use it only on a trusted LAN or VPN.</p>+ {command && <>+ <CopyableCommand label="Run on the new machine (needs Docker with the compose plugin)" text={command} />+ <p className="mt-2 text-[11px] text-[#7f8589]">Installs the collector with no Docker socket and no capabilities; a separate read-only proxy is the only container that touches the socket. The machine running this dashboard is detected automatically.</p>+ </>}<details className="mt-4 rounded-xl border border-neutral-800 p-3"><summary className="cursor-pointer text-xs text-[#9aa0a6]">Advanced: skip approval with an enrollment key</summary>@@ -922,9 +1087,11 @@ function PendingServers({ pending, onChanged }) {if (!pending.length) return nullconst act = async (item, action) => {+ // Approving a machine that matches an existing server replaces that server's credentials and signs out its collector.+ if (action === 'approve' && item.replaces && !window.confirm(`This machine matches the existing server "${item.replaces.hostname}". Adding it replaces that server's credentials and signs its current collector out. Only continue if you are re-installing that machine.`)) returnsetBusyId(item.id); setError('')try {- await apiRequest(`/monitoring/pending-collectors/${item.id}/${action}`, { method: 'POST' })+ await apiRequest(`/monitoring/pending-collectors/${item.id}/${action}`, { method: 'POST', body: JSON.stringify({ replace: Boolean(item.replaces) }) })await onChanged()} catch (requestError) { setError(requestError.message || 'Could not update the server.') }finally { setBusyId(null) }@@ -933,7 +1100,8 @@ function PendingServers({ pending, onChanged }) {const row = (item) => (<div key={item.id} className="flex flex-wrap items-center justify-between gap-3 rounded-xl border border-neutral-800 bg-[#0d1013]/70 p-3"><div>- <div className="font-medium text-white">{item.hostname}</div>+ <div className="flex flex-wrap items-center gap-2"><span className="font-medium text-white">{item.hostname}</span>{item.verified ? <span className="rounded-full bg-emerald-400/15 px-2 py-0.5 text-[10px] font-semibold tracking-wider text-emerald-300">JOIN TOKEN</span> : <span className="rounded-full bg-amber-400/15 px-2 py-0.5 text-[10px] font-semibold tracking-wider text-amber-300">NO TOKEN</span>}</div>+ {item.replaces && <div className="mt-1 text-xs text-amber-200">Replaces the existing server “{item.replaces.hostname}”: its current collector will be signed out.</div>}<div className="mt-0.5 text-xs text-[#9aa0a6]">{[item.ip_address, item.os_info].filter(Boolean).join(' · ')} · code <span className="font-mono text-[#cfcfcf]">{item.pairing_code}</span></div></div><div className="flex w-full gap-2 sm:w-auto">@@ -1620,7 +1788,7 @@ function AdminAgentTab() {// [label, hint, warning key, critical key (null when the rule only warns), unit, step]const thresholdRows = [- ['Disk usage', 'Root filesystem', 'disk_warning_pct', 'disk_critical_pct', '%', 1],+ ['Disk usage', 'Default for every mount', 'disk_warning_pct', 'disk_critical_pct', '%', 1],['Inode usage', 'Root filesystem', 'inodes_warning_pct', 'inodes_critical_pct', '%', 1],['Memory usage', null, 'memory_warning_pct', 'memory_critical_pct', '%', 1],['Swap usage', 'Warning only', 'swap_warning_pct', null, '%', 1],@@ -1639,27 +1807,38 @@ function ThresholdInput({ label, value, step, onChange, disabled, caption }) {function IncidentThresholdsSection() {const [saved, setSaved] = useState(null)const [values, setValues] = useState(null)+ const [savedMounts, setSavedMounts] = useState([])+ const [mountRows, setMountRows] = useState([])const [saving, setSaving] = useState(false)const [message, setMessage] = useState(null)+ // The API keys overrides by mount; the form edits them as rows so a path can be typed freely.+ const toRows = (mountThresholds) => Object.entries(mountThresholds || {}).map(([mount, limits]) => ({ mount, warning: limits.disk_warning_pct, critical: limits.disk_critical_pct }))+ const toMountThresholds = (rows) => Object.fromEntries(rows.map((row) => [row.mount.trim(), { disk_warning_pct: row.warning, disk_critical_pct: row.critical }]))+useEffect(() => {apiRequest('/admin/settings', { method: 'GET' }).then((data) => {setSaved(data.alert_thresholds)setValues(data.alert_thresholds)+ setSavedMounts(toRows(data.mount_thresholds))+ setMountRows(toRows(data.mount_thresholds))}).catch((error) => setMessage({ ok: false, text: error.message || 'Could not load the thresholds.' }))}, [])const set = (key) => (value) => setValues((current) => ({ ...current, [key]: value }))+ const setMountRow = (index, field) => (value) => setMountRows((rows) => rows.map((row, i) => (i === index ? { ...row, [field]: value } : row)))const save = async () => {setSaving(true)setMessage(null)try {- const data = await apiRequest('/admin/settings', { method: 'PUT', body: JSON.stringify({ alert_thresholds: values }) })+ const data = await apiRequest('/admin/settings', { method: 'PUT', body: JSON.stringify({ alert_thresholds: values, mount_thresholds: toMountThresholds(mountRows) }) })setSaved(data.alert_thresholds)setValues(data.alert_thresholds)+ setSavedMounts(toRows(data.mount_thresholds))+ setMountRows(toRows(data.mount_thresholds))setMessage({ ok: true, text: 'Saved. New thresholds apply from each server\'s next check-in.' })} catch (error) {setMessage({ ok: false, text: error.message || 'Could not save the thresholds.' })@@ -1669,8 +1848,9 @@ function IncidentThresholdsSection() {}const loading = values === null- const unchanged = loading || JSON.stringify(values) === JSON.stringify(saved)- const incomplete = !loading && Object.values(values).some((value) => value === '')+ const unchanged = loading || (JSON.stringify(values) === JSON.stringify(saved) && JSON.stringify(mountRows) === JSON.stringify(savedMounts))+ const mountPaths = mountRows.map((row) => row.mount.trim())+ const incomplete = !loading && (Object.values(values).some((value) => value === '') || mountRows.some((row) => !row.mount.trim() || row.warning === '' || row.critical === '') || new Set(mountPaths).size !== mountPaths.length)return (<AdminSection@@ -1705,6 +1885,25 @@ function IncidentThresholdsSection() {<ThresholdInput label="Restart window in minutes" value={values?.restart_loop_window_minutes} step={1} onChange={set('restart_loop_window_minutes')} disabled={loading} />min</span></div>+ <label className="flex flex-col gap-3 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3 text-sm sm:flex-row sm:items-center sm:justify-between sm:gap-4 sm:py-2.5">+ <span><span className="text-[#bcbcbc]">Reboot and OOM-kill incidents</span><span className="block text-xs text-[#7f8589]">Minutes an incident stays open after the event</span></span>+ <ThresholdInput label="Event incident window in minutes" value={values?.event_window_minutes} step={1} onChange={set('event_window_minutes')} disabled={loading} />+ </label>+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3 text-sm">+ <div className="flex flex-wrap items-center justify-between gap-2">+ <span><span className="text-[#bcbcbc]">Disk usage per mount</span><span className="block text-xs text-[#7f8589]">Replaces the default disk thresholds for that mount, on every server</span></span>+ <button type="button" disabled={loading} onClick={() => setMountRows((rows) => [...rows, { mount: '', warning: values?.disk_warning_pct ?? 85, critical: values?.disk_critical_pct ?? 95 }])} className={secondaryButtonClass}>Add override</button>+ </div>+ {mountRows.length > 0 && <div className="mt-3 space-y-2">{mountRows.map((row, index) => (+ <div key={index} className="flex flex-wrap items-end gap-2 sm:items-center">+ <input type="text" aria-label="Mount path" placeholder="/mnt/data" value={row.mount} onChange={(event) => setMountRow(index, 'mount')(event.target.value)} className="h-10 min-w-0 flex-1 basis-40 rounded-xl border border-[#1f2326] bg-[#0d1013] px-3 font-mono text-xs text-white outline-none focus:border-white/30 sm:h-8" />+ <ThresholdInput label={`${row.mount || 'Mount'} warning`} caption="Warning" value={row.warning} step={1} onChange={setMountRow(index, 'warning')} />+ <ThresholdInput label={`${row.mount || 'Mount'} critical`} caption="Critical" value={row.critical} step={1} onChange={setMountRow(index, 'critical')} />+ <span className="pb-2 text-xs text-[#9aa0a6] sm:pb-0">%</span>+ <button type="button" aria-label={`Remove override for ${row.mount || 'mount'}`} onClick={() => setMountRows((rows) => rows.filter((_, i) => i !== index))} className="pb-2 text-lg leading-none text-[#9aa0a6] hover:text-red-300 sm:pb-0">×</button>+ </div>+ ))}</div>}+ </div></div>{message && <p className={`mt-3 text-xs ${message.ok ? 'text-emerald-300' : 'text-amber-200'}`}>{message.text}</p>}</AdminSection>diff --git a/tests/test_admin.py b/tests/test_admin.pyindex a25f3af..d70c21b 100644--- a/tests/test_admin.py+++ b/tests/test_admin.py@@ -37,8 +37,8 @@ def _login_as(monkeypatch, user_id):def test_first_registered_user_is_admin_and_later_ones_are_not(db):- first = register_user(db, FakeRedis(), name="First", email="[email protected]", password="secret123")- second = register_user(db, FakeRedis(), name="Second", email="[email protected]", password="secret123")+ first = register_user(db, FakeRedis(), name="First", email="[email protected]", password="correct-horse-battery-staple")+ second = register_user(db, FakeRedis(), name="Second", email="[email protected]", password="correct-horse-battery-staple")assert first["user"]["role"] == "admin"assert second["user"]["role"] == "user"@@ -164,7 +164,7 @@ def test_admin_can_turn_agent_chat_off_and_on(fleet, monkeypatch):result = main.update_admin_settings(main.AdminSettingsRequest(agent_chat_enabled=False), session_id="s", db=fleet)- assert result == {"metrics_retention_days": main.DEFAULT_METRICS_RETENTION_DAYS, "agent_chat_enabled": False, "alert_thresholds": rules.DEFAULT_THRESHOLDS}+ assert result == {"metrics_retention_days": main.DEFAULT_METRICS_RETENTION_DAYS, "agent_chat_enabled": False, "alert_thresholds": rules.DEFAULT_THRESHOLDS, "mount_thresholds": {}}_login_as(monkeypatch, 2)assert main.agent_config(session_id="s", db=fleet) == {"chat_enabled": False}diff --git a/tests/test_auth_backend.py b/tests/test_auth_backend.pyindex f1842cd..2c4b13d 100644--- a/tests/test_auth_backend.py+++ b/tests/test_auth_backend.py@@ -39,13 +39,13 @@ def test_register_user_creates_record_and_hashes_password():redis_client=redis,name="Ada Lovelace",email="[email protected]",- password="secret123",+ password="correct-horse-battery-staple",)assert result["user"]["email"] == "[email protected]"assert result["session_id"]assert session.query(User).count() == 1- assert session.query(User).first().password_hash != "secret123"+ assert session.query(User).first().password_hash != "correct-horse-battery-staple"def test_authenticate_user_returns_session_for_valid_credentials():@@ -60,14 +60,14 @@ def test_authenticate_user_returns_session_for_valid_credentials():redis_client=redis,name="Grace Hopper",email="[email protected]",- password="hunter2",+ password="correct-horse-battery-staple",)result = authenticate_user(db=session,redis_client=redis,email="[email protected]",- password="hunter2",+ password="correct-horse-battery-staple",)assert result["user"]["email"] == "[email protected]"@@ -86,7 +86,7 @@ def test_record_audit_log_persists_database_entries():redis_client=redis,name="Linus Torvalds",email="[email protected]",- password="linux123",+ password="correct-horse-battery-staple",)record_audit_log(@@ -126,13 +126,13 @@ def test_registration_closed_still_allows_first_account(monkeypatch):session = sessionmaker(bind=engine)()monkeypatch.setattr(main, "ALLOW_REGISTRATION", False)- assert main.auth_config(db=session) == {"registration_enabled": True}+ assert main.auth_config(db=session) == {"registration_enabled": True, "bootstrap_required": True}session.add(User(name="Owner", email="[email protected]", password_hash="x"))session.commit()- assert main.auth_config(db=session) == {"registration_enabled": False}+ assert main.auth_config(db=session) == {"registration_enabled": False, "bootstrap_required": False}with pytest.raises(HTTPException) as exc:- main.auth_register(main.RegisterRequest(name="Stranger", email="[email protected]", password="secret123"), request=None, response=None, db=session)+ main.auth_register(main.RegisterRequest(name="Stranger", email="[email protected]", password="correct-horse-battery-staple"), request=None, response=None, db=session)assert exc.value.status_code == 403@@ -147,3 +147,229 @@ def test_registration_open_by_default(monkeypatch):monkeypatch.setattr(main, "ALLOW_REGISTRATION", True)assert main.registration_open(session) is True+++import pytest+from fastapi import HTTPException+++# --- first-account setup token --------------------------------------------------------------++def _fresh(monkeypatch, token=""):+ import os+ from src.backend import main+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ monkeypatch.setattr(main, "ALLOW_REGISTRATION", False)+ monkeypatch.setattr(main, "ADMIN_BOOTSTRAP_TOKEN", token)+ monkeypatch.setattr(main, "enforce_register_limit", lambda *a, **k: None)+ monkeypatch.setattr(main, "redis_client", FakeRedisForRegister())+ return main, session+++class FakeRedisForRegister:+ def __init__(self):+ self.store = {}++ def setex(self, key, ttl, value):+ self.store[key] = value++ def set(self, key, value, *a, **k):+ self.store[key] = value++ def get(self, key):+ return self.store.get(key)++ def delete(self, *keys):+ for key in keys:+ self.store.pop(key, None)+++def _register(main, session, token, email="[email protected]"):+ from types import SimpleNamespace+ from fastapi import Response+ request = SimpleNamespace(client=SimpleNamespace(host="1.2.3.4"))+ return main.auth_register(main.RegisterRequest(name="Owner", email=email, password="correct-horse-battery-staple", bootstrap_token=token), request=request, response=Response(), db=session)+++def test_fresh_install_generates_a_token_and_stores_only_its_hash(monkeypatch):+ main, session = _fresh(monkeypatch)+ token = main.ensure_bootstrap_token(session)++ assert token.startswith("hlb_") and len(token) > 20+ assert session.get(main.AppSetting, main.BOOTSTRAP_TOKEN_KEY).value == main._hash_secret(token) != token+++def test_the_token_is_regenerated_at_each_start_until_an_account_exists(monkeypatch):+ main, session = _fresh(monkeypatch)+ first, second = main.ensure_bootstrap_token(session), main.ensure_bootstrap_token(session)+ assert first != second+ assert main._bootstrap_token_valid(session, second) and not main._bootstrap_token_valid(session, first)+++def test_configured_token_is_used_and_not_printed(monkeypatch):+ main, session = _fresh(monkeypatch, token="my-own-setup-token")+ assert main.ensure_bootstrap_token(session) is None+ assert main._bootstrap_token_valid(session, "my-own-setup-token")++[email protected]("bad", [None, "", " ", "hlb_wrong", "x" * 1000])+def test_first_account_is_refused_without_the_right_token(monkeypatch, bad):+ main, session = _fresh(monkeypatch)+ main.ensure_bootstrap_token(session)+ with pytest.raises(HTTPException) as exc:+ _register(main, session, bad)+ assert exc.value.status_code == 403 and "setup token" in exc.value.detail+ assert session.query(User).count() == 0+++def test_first_account_with_the_token_becomes_admin_and_the_token_is_removed(monkeypatch):+ main, session = _fresh(monkeypatch)+ token = main.ensure_bootstrap_token(session)++ result = _register(main, session, token)++ assert result["user"]["role"] == "admin"+ assert session.get(main.AppSetting, main.BOOTSTRAP_TOKEN_KEY) is None+ # The token can't create a second account, and registration is closed anyway.+ with pytest.raises(HTTPException) as exc:+ _register(main, session, token, email="[email protected]")+ assert exc.value.status_code == 403+++def test_without_a_stored_token_the_first_account_is_refused(monkeypatch):+ main, session = _fresh(monkeypatch) # startup never ran+ with pytest.raises(HTTPException) as exc:+ _register(main, session, "anything")+ assert exc.value.status_code == 403+++def test_existing_installs_need_no_token_and_leftovers_are_cleaned_up(monkeypatch):+ main, session = _fresh(monkeypatch)+ main.ensure_bootstrap_token(session)+ session.add(User(name="Owner", email="[email protected]", password_hash="x"))+ session.commit()++ assert main.ensure_bootstrap_token(session) is None+ assert session.get(main.AppSetting, main.BOOTSTRAP_TOKEN_KEY) is None+++# --- password policy ------------------------------------------------------------------------++from src.backend import auth as auth_module++[email protected]("password", ["", "short", "a" * 15, "fifteen-chars-ok"[:15], "12345678901234 "[:15]])+def test_passwords_under_16_characters_are_refused(password):+ with pytest.raises(ValueError, match="at least 16"):+ auth_module.validate_password(password)++[email protected]("password", ["a" * 16, "correct horse battery staple", "p" * 72, "ü" * 36, "пароль-пароль-пароль"])+def test_acceptable_passwords_pass(password):+ auth_module.validate_password(password)++[email protected]("password", ["p" * 73, "ü" * 37, "x" * 5000])+def test_passwords_over_bcrypts_72_byte_limit_are_refused_not_truncated(password):+ with pytest.raises(ValueError, match="at most 72 bytes"):+ auth_module.validate_password(password)+++def test_registration_enforces_the_policy_and_creates_no_account():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ with pytest.raises(ValueError, match="at least 16"):+ register_user(session, FakeRedis(), name="A", email="[email protected]", password="hunter2hunter2") # 14 characters+ assert session.query(User).count() == 0+++def test_existing_accounts_with_short_passwords_can_still_sign_in():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(name="Old", email="[email protected]", password_hash=bcrypt.hashpw(b"old6pw", bcrypt.gensalt()).decode(), role="admin"))+ session.commit()++ assert authenticate_user(session, FakeRedis(), email="[email protected]", password="old6pw")["user"]["email"] == "[email protected]"+++def test_an_overlong_password_at_login_is_just_invalid_not_an_error_with_internals():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(name="U", email="[email protected]", password_hash=bcrypt.hashpw(b"x" * 20, bcrypt.gensalt()).decode()))+ session.commit()++ with pytest.raises(ValueError) as exc:+ authenticate_user(session, FakeRedis(), email="[email protected]", password="x" * 200)+ assert str(exc.value) == "Invalid email or password."+++# --- cross-site request check ---------------------------------------------------------------++from src.backend.main import cross_site_request++ALLOWED = ["http://localhost:5173"]+++def _h(**headers):+ return {key.replace("_", "-"): value for key, value in headers.items()}++[email protected]("method", ["POST", "PUT", "PATCH", "DELETE"])+def test_a_request_from_another_site_is_refused(method):+ assert cross_site_request(method, _h(origin="https://evil.example", host="dashboard.example.com"), ALLOWED) is True+++def test_same_host_origin_is_allowed_whatever_the_scheme():+ for origin in ("https://dashboard.example.com", "http://dashboard.example.com"):+ assert cross_site_request("POST", _h(origin=origin, host="dashboard.example.com"), ALLOWED) is False+++def test_the_port_must_match_too():+ assert cross_site_request("POST", _h(origin="http://dashboard.example.com:8080", host="dashboard.example.com"), ALLOWED) is True+ assert cross_site_request("POST", _h(origin="http://localhost:5173", host="localhost:5173"), ALLOWED) is False+++def test_a_sibling_subdomain_is_a_different_site_for_this_purpose():+ assert cross_site_request("POST", _h(origin="https://other-app.example.com", host="dashboard.example.com"), ALLOWED) is True+ assert cross_site_request("POST", _h(origin="https://dashboard.example.com.evil.io", host="dashboard.example.com"), ALLOWED) is True+++def test_origins_from_the_configured_list_are_allowed():+ assert cross_site_request("POST", _h(origin="http://localhost:5173", host="localhost:8000"), ALLOWED) is False++[email protected]("origin", ["null", "", "file://", "javascript:alert(1)", "ftp://dashboard.example.com", "https://", "dashboard.example.com"])+def test_odd_origins_are_refused(origin):+ assert cross_site_request("POST", _h(origin=origin, host="dashboard.example.com"), ALLOWED) is True+++def test_without_an_origin_the_fetch_metadata_header_decides():+ assert cross_site_request("POST", _h(sec_fetch_site="cross-site", host="d"), ALLOWED) is True+ assert cross_site_request("POST", _h(sec_fetch_site="same-site", host="d"), ALLOWED) is True+ assert cross_site_request("POST", _h(sec_fetch_site="same-origin", host="d"), ALLOWED) is False+ assert cross_site_request("POST", _h(sec_fetch_site="none", host="d"), ALLOWED) is False+++def test_non_browser_clients_and_safe_methods_are_not_affected():+ # Collectors and curl send neither header and authenticate with tokens.+ assert cross_site_request("POST", _h(host="d"), ALLOWED) is False+ # Reads are never state-changing here, and CORS covers cross-origin reads.+ for method in ("GET", "HEAD", "OPTIONS"):+ assert cross_site_request(method, _h(origin="https://evil.example", host="d"), ALLOWED) is False+++def test_the_real_app_refuses_a_cross_site_post_before_any_handler_runs():+ from starlette.testclient import TestClient+ from src.backend import main as app_main+ client = TestClient(app_main.app)++ response = client.post("/monitoring/pending-collectors/1/approve", headers={"Origin": "https://evil.example"})+ assert response.status_code == 403 and response.json() == {"detail": "Cross-site request refused."}+ # Not a CSRF block: it fails later for lack of a session or a database, but never with the cross-site message.+ allowed = client.post("/auth/logout", headers={"Origin": "http://testserver"})+ assert allowed.json().get("detail") != "Cross-site request refused."diff --git a/tests/test_docker_proxy.py b/tests/test_docker_proxy.pynew file mode 100644index 0000000..3bb11c4--- /dev/null+++ b/tests/test_docker_proxy.py@@ -0,0 +1,177 @@+import http.client+import json+import socketserver+import threading+from http.server import BaseHTTPRequestHandler++import pytest++from src.agent import collector, docker_proxy as proxy++CID = "a" * 64+SECRET = "SUPER_SECRET_TOKEN_123"++INSPECT = {+ "Id": CID, "Name": "/web", "RestartCount": 2, "Image": "sha256:abc",+ "Config": {"Image": "nginx:1", "Env": [f"DB_PASSWORD={SECRET}"], "Labels": {"k": "v"}, "Cmd": ["nginx"]},+ "State": {"Status": "running", "StartedAt": "2026-01-01T00:00:00Z", "Pid": 99, "Health": {"Status": "healthy", "Log": [{"Output": SECRET}]}},+ "NetworkSettings": {"Ports": {"80/tcp": [{"HostIp": "0.0.0.0", "HostPort": "8080"}]}, "SecondaryIPAddresses": None, "Networks": {"x": {"Gateway": "1"}}},+ "Mounts": [{"Source": "/etc/shadow"}], "HostConfig": {"Privileged": True}, "GraphDriver": {"Data": {}},+}+LISTING = [{"Id": CID, "Names": ["/web"], "Image": "nginx:1", "State": "running", "Labels": {"secret": SECRET}, "Mounts": []}]+STATS = {"cpu_stats": {"cpu_usage": {"total_usage": 10}, "system_cpu_usage": 100, "online_cpus": 2}, "precpu_stats": {"cpu_usage": {"total_usage": 5}, "system_cpu_usage": 50},+ "memory_stats": {"usage": 1048576, "limit": 2097152, "stats": {"inactive_file": 0}}, "networks": {"eth0": {"rx_bytes": 1, "tx_bytes": 2}},+ "blkio_stats": {"io_service_bytes_recursive": []}, "pids_stats": {"current": 3}, "name": "/web", "id": CID}+++# --- routing: the allow-list ---------------------------------------------------------------+[email protected]("target", [+ "/containers/json", "/containers/json?all=1", f"/containers/{CID}/json", f"/containers/{CID}/stats?stream=false", f"/containers/{CID[:12]}/json",+])+def test_the_three_read_requests_are_allowed(target):+ assert proxy.route("GET", target) is not None++[email protected]("method", ["POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS", "get", "CONNECT"])+def test_only_get_is_allowed(method):+ assert proxy.route(method, "/containers/json") is None++[email protected]("target", [+ # other endpoints, including the ones that read container contents or start things+ "/", "/version", "/info", "/images/json", "/volumes", "/networks", "/system/df", "/events", "/_ping", "/containers", "/containers/",+ f"/containers/{CID}/export", f"/containers/{CID}/archive?path=/etc/passwd", f"/containers/{CID}/logs", f"/containers/{CID}/top",+ f"/containers/{CID}/attach", f"/containers/{CID}/exec", f"/containers/{CID}/start", f"/containers/{CID}/kill", f"/containers/{CID}/changes",+ "/containers/create", "/containers/prune", "/exec/abc/start", "/build", "/commit", "/swarm", "/secrets", "/configs",+ # query abuse+ "/containers/json?all=1&filters=x", "/containers/json?all=0", "/containers/json?all=1&all=1", "/containers/json?size=1", "/containers/json?",+ f"/containers/{CID}/stats", f"/containers/{CID}/stats?stream=true", f"/containers/{CID}/stats?stream=false&one-shot=1", f"/containers/{CID}/json?size=1",+ # traversal, encoding, padding+ f"/containers/{CID}/../../images/json", "/containers/../images/json", f"/containers/{CID}/json/..", "//containers/json", "/containers//json",+ "/containers/json/", f"/containers/{CID}/json/", "/containers/%6Ason", "/containers/json%00", f"/containers/{CID}%2fjson", "/%2e%2e/containers/json",+ "/containers/json#frag", "/containers/json ", " /containers/json", "/containers/json\r\nX: y", "/containers/json\n", "/containers/jsón",+ "/v1.43/containers/json", "http://localhost/containers/json", "*",+ # ids that are not plain hex+ "/containers/web/json", "/containers/ABC123ABC123/json", "/containers/abc/json", f"/containers/{'a' * 65}/json", "/containers/zzzzzzzzzzzz/json", "/containers/;rm/json",+])+def test_everything_else_is_refused(target):+ assert proxy.route("GET", target) is None+++# --- response filtering --------------------------------------------------------------------++def test_inspect_output_drops_environment_and_everything_unneeded():+ result = proxy.project_inspect(INSPECT)++ assert SECRET not in json.dumps(result)+ assert result == {+ "Id": CID, "Name": "/web", "RestartCount": 2, "Config": {"Image": "nginx:1"},+ "State": {"Status": "running", "StartedAt": "2026-01-01T00:00:00Z", "Health": {"Status": "healthy"}},+ "NetworkSettings": {"Ports": {"80/tcp": [{"HostIp": "0.0.0.0", "HostPort": "8080"}]}},+ }+++def test_list_and_stats_output_are_cut_down():+ assert SECRET not in json.dumps(proxy.project_list(LISTING))+ assert proxy.project_list(LISTING) == [{"Id": CID, "Names": ["/web"], "Image": "nginx:1", "State": "running"}]+ assert set(proxy.project_stats(STATS)) == {"cpu_stats", "precpu_stats", "memory_stats", "networks", "blkio_stats", "pids_stats"}++[email protected]("junk", [None, 5, "x", [], {}, [1, "a", None], {"State": "bad", "Config": 3, "NetworkSettings": []}])+def test_unexpected_upstream_shapes_do_not_crash_the_filters(junk):+ proxy.project_list(junk), proxy.project_inspect(junk), proxy.project_stats(junk)+++# --- end to end: collector -> proxy -> fake Docker daemon ----------------------------------++class _FakeDocker(BaseHTTPRequestHandler):+ seen: list[str] = []++ def do_GET(self):+ _FakeDocker.seen.append(self.path)+ routes = {"/containers/json?all=1": LISTING, f"/containers/{CID}/json": INSPECT, f"/containers/{CID}/stats?stream=false": STATS}+ body = json.dumps(routes.get(self.path)).encode() if self.path in routes else b"nope"+ self.send_response(200 if self.path in routes else 404)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ do_POST = do_DELETE = do_GET++ def log_message(self, *args):+ pass+++def stack(tmp_path):+ socket_path = str(tmp_path / "docker.sock")+ daemon = socketserver.ThreadingUnixStreamServer(socket_path, _FakeDocker)+ daemon.daemon_threads = True+ _FakeDocker.seen = []+ listen_path = str(tmp_path / "proxy" / "docker.sock")+ server = proxy.make_server(listen_path, socket_path)+ for item in (daemon, server):+ threading.Thread(target=item.serve_forever, daemon=True).start()+ yield listen_path, daemon+ for item in (server, daemon):+ item.shutdown()+ item.server_close()+++def _raw(listen_path, method, path):+ connection = collector._UnixHTTPConnection(listen_path)+ connection.request(method, path)+ response = connection.getresponse()+ return response.status, response.read()+++def test_collector_works_through_the_proxy_and_never_sees_secrets(stack):+ port, _ = stack++ [container] = collector.collect_containers(port)++ assert (container["name"], container["image"], container["status"], container["restart_count"], container["healthcheck_status"]) == ("web", "nginx:1", "running", 2, "healthy")+ assert container["ports"] == "8080->80/tcp" and container["memory_used_mb"] == 1.0 and container["pids"] == 3+ assert SECRET not in json.dumps(container)+++def test_refused_requests_never_reach_the_docker_daemon(stack):+ port, _ = stack+ attacks = [("POST", "/containers/create"), ("DELETE", f"/containers/{CID}"), ("GET", f"/containers/{CID}/archive?path=/etc"), ("GET", "/containers/../info"),+ ("GET", f"/containers/{CID}/json?size=1"), ("POST", f"/containers/{CID}/start"), ("GET", "/containers/%6Ason")]+ for method, path in attacks:+ assert _raw(port, method, path)[0] == 403, (method, path)++ assert _FakeDocker.seen == []+++def test_upstream_failures_become_a_generic_502(stack):+ port, daemon = stack+ status, body = _raw(port, "GET", f"/containers/{'b' * 64}/json") # fake daemon answers 404+ assert (status, body) == (502, b"Upstream error")++ daemon.shutdown()+ daemon.server_close()+ assert _raw(port, "GET", "/containers/json")[0] == 502+++def test_oversized_upstream_answers_are_rejected(stack, monkeypatch):+ port, _ = stack+ monkeypatch.setattr(proxy, "MAX_RESPONSE_BYTES", 10)+ assert _raw(port, "GET", "/containers/json?all=1")[0] == 502+++def test_proxy_socket_is_not_world_accessible_and_replaces_a_stale_one(tmp_path):+ listen = tmp_path / "run" / "docker.sock"+ listen.parent.mkdir()+ listen.write_text("stale")+ server = proxy.make_server(str(listen), str(tmp_path / "unused"))+ try:+ assert listen.stat().st_mode & 0o777 == 0o660+ finally:+ server.server_close()+++def test_collector_still_supports_the_plain_socket_path(tmp_path):+ assert collector.collect_containers(str(tmp_path / "missing.sock")) == []diff --git a/tests/test_pending_collectors.py b/tests/test_pending_collectors.pyindex fe90928..6580b1c 100644--- a/tests/test_pending_collectors.py+++ b/tests/test_pending_collectors.py@@ -12,7 +12,7 @@ from sqlalchemy.orm import sessionmakerfrom src.agent import collectorfrom src.backend import main-from src.backend.models import Base, PendingCollector, Server, User+from src.backend.models import Base, JoinToken, PendingCollector, Server, User@pytest.fixture@@ -98,7 +98,16 @@ def test_approving_known_machine_reuses_existing_server(db):db.add(Server(id=3, owner_id=1, hostname="nas", machine_id="machine-a"))db.commit()registration = _register(db)- main.approve_pending_collector(db.query(PendingCollector).one().id, session_id="s", db=db)+ pending_id = db.query(PendingCollector).one().id+ assert main.visible_pending_collectors(db)[0]["replaces"] == {"id": 3, "hostname": "nas"}++ # Replacing a server's credentials locks out its current collector, so it needs explicit confirmation.+ with pytest.raises(HTTPException) as exc:+ main.approve_pending_collector(pending_id, session_id="s", db=db)+ assert exc.value.status_code == 409 and "nas" in exc.value.detail+ assert db.query(PendingCollector).one().status == "pending"++ main.approve_pending_collector(pending_id, main.ApproveRequest(replace=True), session_id="s", db=db)assert _poll(db, registration["request_secret"])["server_id"] == 3assert db.query(Server).count() == 1@@ -119,10 +128,6 @@ def test_purge_removes_abandoned_requests(db):assert db.query(PendingCollector).count() == 0-def test_collector_source_is_served():- assert "def run_collector" in main.collector_source().body.decode()--def test_collector_waits_for_approval_without_restart(monkeypatch):responses = iter([{"request_secret": "secret", "pairing_code": "ABC-234", "poll_interval_seconds": 5},@@ -150,3 +155,175 @@ def test_collector_without_key_or_credentials_requests_approval(tmp_path, monkeyassert collector.resolve_credentials("http://dash/api") == (11, "approved-token")assert collector.load_credentials() == (11, "approved-token")+++def test_collector_sends_its_join_token_when_registering(monkeypatch):+ sent = []++ def fake_post(url, payload, bearer=None):+ sent.append((url, payload))+ if url.endswith("/register"):+ return {"request_secret": "s", "pairing_code": "ABC-123", "poll_interval_seconds": 2}+ return {"status": "approved", "server_id": 4, "collector_token": "tok"}++ monkeypatch.setenv("HOMELAB_JOIN_TOKEN", "hlj_abc ")+ monkeypatch.setattr(collector, "_post_json", fake_post)+ monkeypatch.setattr(collector.time, "sleep", lambda seconds: None)++ assert collector.request_approval("http://dash/api") == (4, "tok")+ assert sent[0][1]["join_token"] == "hlj_abc"+++def test_collector_without_a_join_token_sends_none(monkeypatch):+ sent = []+ monkeypatch.delenv("HOMELAB_JOIN_TOKEN", raising=False)+ monkeypatch.setattr(collector, "_post_json", lambda url, payload, bearer=None: sent.append(payload) or {"request_secret": "s", "pairing_code": "A", "poll_interval_seconds": 2} if url.endswith("/register") else {"status": "approved", "server_id": 1, "collector_token": "t"})+ monkeypatch.setattr(collector.time, "sleep", lambda seconds: None)+ collector.request_approval("http://dash/api")+ assert sent[0]["join_token"] is None+++def test_collector_explains_a_refused_registration(monkeypatch, capsys):+ import io, json, urllib.error+ calls = []++ def fake_post(url, payload, bearer=None):+ calls.append(url)+ if len(calls) == 1:+ raise urllib.error.HTTPError(url, 403, "Forbidden", {}, io.BytesIO(json.dumps({"detail": "The join token is invalid, expired or already used."}).encode()))+ if url.endswith("/register"):+ return {"request_secret": "s", "pairing_code": "A", "poll_interval_seconds": 2}+ return {"status": "approved", "server_id": 1, "collector_token": "t"}++ monkeypatch.setattr(collector, "_post_json", fake_post)+ monkeypatch.setattr(collector.time, "sleep", lambda seconds: None)+ collector.request_approval("http://dash/api")++ assert "join token is invalid, expired or already used" in capsys.readouterr().out+++# --- join tokens and the collector listener -------------------------------------------------++def _mint(db):+ return main.create_join_token(session_id="s", db=db)["join_token"]+++def _request(via="", ip="10.0.0.5"):+ return SimpleNamespace(client=SimpleNamespace(host=ip), headers={"x-collector-via": via} if via else {})+++def _register_via(db, via, token=None, machine_id="machine-a", hostname="nas", ip="10.0.0.5"):+ payload = main.CollectorRegisterRequest(hostname=hostname, machine_id=machine_id, join_token=token)+ return main.register_collector(payload, _request(via, ip), db=db)+++def strict(monkeypatch):+ monkeypatch.setattr(main, "COLLECTOR_ACCESS", "strict")+++def test_join_token_is_minted_hashed_and_expires(db):+ token = _mint(db)+ row = db.query(JoinToken).one()++ assert token.startswith("hlj_") and row.token_hash == main._hash_secret(token) and token not in row.token_hash+ expires = row.expires_at if row.expires_at.tzinfo else row.expires_at.replace(tzinfo=timezone.utc) # SQLite drops the zone+ assert 0 < (expires - datetime.now(timezone.utc)).total_seconds() <= main.JOIN_TOKEN_TTL_SECONDS+++def test_join_token_minting_requires_admin_and_is_capped(db, monkeypatch):+ for _ in range(main.MAX_OPEN_JOIN_TOKENS):+ _mint(db)+ with pytest.raises(HTTPException) as exc:+ _mint(db)+ assert exc.value.status_code == 429++ monkeypatch.setattr(main, "_current_user", lambda session_id, db: {"id": 2})+ db.add(User(id=2, name="Member", email="[email protected]", password_hash="x", role="user"))+ db.commit()+ with pytest.raises(HTTPException) as exc:+ main.create_join_token(session_id="s", db=db)+ assert exc.value.status_code == 403+++def test_registration_with_a_valid_token_is_marked_verified(db, strict):+ _register_via(db, "vpn", _mint(db))++ [item] = main.visible_pending_collectors(db)+ assert item["verified"] is True+++def test_vpn_listener_refuses_registration_without_a_token(db, strict):+ with pytest.raises(HTTPException) as exc:+ _register_via(db, "vpn")+ assert exc.value.status_code == 403 and "join token" in exc.value.detail+ assert db.query(PendingCollector).count() == 0++[email protected]("bad", ["", "hlj_nonsense", "x" * 500])+def test_wrong_tokens_are_refused(db, strict, bad):+ _mint(db)+ with pytest.raises(HTTPException) as exc:+ _register_via(db, "vpn", bad or " ")+ assert exc.value.status_code == 403+ assert db.query(PendingCollector).count() == 0+++def test_expired_token_is_refused(db, strict):+ token = _mint(db)+ db.query(JoinToken).update({"expires_at": datetime.now(timezone.utc) - timedelta(seconds=1)})+ db.commit()+ with pytest.raises(HTTPException) as exc:+ _register_via(db, "vpn", token)+ assert exc.value.status_code == 403+++def test_token_is_consumed_when_the_server_is_approved(db, strict):+ token = _mint(db)+ _register_via(db, "vpn", token)+ main.approve_pending_collector(db.query(PendingCollector).one().id, session_id="s", db=db)++ assert db.query(JoinToken).one().used_at is not None+ with pytest.raises(HTTPException) as exc:+ _register_via(db, "vpn", token, machine_id="machine-b", hostname="other")+ assert exc.value.status_code == 403+++def test_same_host_can_retry_with_its_token_before_approval(db, strict):+ token = _mint(db)+ _register_via(db, "vpn", token)+ _register_via(db, "vpn", token)++ assert db.query(PendingCollector).count() == 1+++def test_strict_mode_hides_collector_endpoints_from_the_public_listener(db, strict):+ for via in ("", "public", "something-else"):+ with pytest.raises(HTTPException) as exc:+ main.require_collector_listener(_request(via))+ assert exc.value.status_code == 404+ main.require_collector_listener(_request("vpn"))+ main.require_collector_listener(_request("internal"))+++def test_internal_listener_needs_no_token_so_the_dashboards_own_collector_can_join(db, strict):+ _register_via(db, "internal")+ assert main.visible_pending_collectors(db)[0]["verified"] is False+++def test_open_mode_keeps_working_without_tokens(db):+ _register_via(db, "")+ assert db.query(PendingCollector).count() == 1+++def test_a_bad_token_is_refused_even_in_open_mode(db):+ with pytest.raises(HTTPException) as exc:+ _register_via(db, "", "hlj_wrong")+ assert exc.value.status_code == 403+++def test_install_info_reports_the_configured_addresses(db, monkeypatch):+ monkeypatch.setattr(main, "COLLECTOR_PUBLIC_URL", "http://10.8.0.1:8081/api")+ monkeypatch.setattr(main, "COLLECTOR_IMAGE", "ghcr.io/o/c@sha256:abc")+ info = main.install_info(session_id="s", db=db)+ assert info == {"strict": False, "collector_url": "http://10.8.0.1:8081/api", "collector_image": "ghcr.io/o/c@sha256:abc", "proxy_image": None}diff --git a/tests/test_rules.py b/tests/test_rules.pyindex d508e1c..2b75d55 100644--- a/tests/test_rules.py+++ b/tests/test_rules.py@@ -167,6 +167,56 @@ def test_mount_path_with_braces_does_not_break_titles(db):assert _active(db)["host:disk:/mnt/{odd}"].title == "Disk usage on nas:/mnt/{odd} is 99%"+def _override(db, mount_thresholds):+ return main.update_admin_settings(main.AdminSettingsRequest(mount_thresholds=mount_thresholds), session_id="s", db=db)+++def test_mount_override_replaces_global_disk_thresholds_for_that_mount_only(db):+ _override(db, {"/mnt/data": {"disk_warning_pct": 50, "disk_critical_pct": 60}})+ for _ in range(3):+ _ingest(db, mounts=[_mount("/", 70), _mount("/mnt/data", 65), _mount("/mnt/other", 70)])++ active = _active(db)+ assert set(active) == {"host:disk:/mnt/data"}+ assert active["host:disk:/mnt/data"].severity == CRITICAL+++def test_mount_override_can_be_looser_than_global(db):+ _override(db, {"/mnt/backup": {"disk_warning_pct": 97, "disk_critical_pct": 99}})+ for _ in range(3):+ _ingest(db, mounts=[_mount("/mnt/backup", 92)])++ assert _active(db) == {}+++def test_mount_override_applies_to_root_for_old_collectors(db):+ _override(db, {"/": {"disk_warning_pct": 30, "disk_critical_pct": 40}})+ for _ in range(3):+ _ingest(db, disk_usage_pct=35)++ assert _active(db)["host:disk:/"].severity == WARNING+++def test_mount_thresholds_are_validated_audited_and_replaced(db):+ good = {"disk_warning_pct": 50, "disk_critical_pct": 60}+ for bad in ({"data": good}, {"/d": {"disk_warning_pct": 60, "disk_critical_pct": 60}}, {"/d": {"disk_warning_pct": 0, "disk_critical_pct": 60}},+ {"/d": {"disk_warning_pct": 50}}, {"/d": {"disk_warning_pct": "5", "disk_critical_pct": 60}}, {"/d": good | {"extra": 1}}):+ with pytest.raises(HTTPException) as exc:+ _override(db, bad)+ assert exc.value.status_code == 400++ assert _override(db, {"/mnt/data": good})["mount_thresholds"] == {"/mnt/data": good}+ assert db.query(AuditLog).one().message == "Changed per-mount disk thresholds: /mnt/data=50/60"+ assert _override(db, {})["mount_thresholds"] == {}+++def test_corrupt_mount_thresholds_are_ignored(db):+ main._set_setting(db, main.MOUNT_THRESHOLDS_KEY, "not json")+ db.commit()++ assert main.get_mount_thresholds(db) == {}++def test_overview_shows_latest_mounts_and_disk_thresholds(db, monkeypatch):monkeypatch.setattr(main, "_current_user", lambda session_id, db: {"id": 1})_ingest(db, mounts=[_mount("/", 10)])diff --git a/tests/test_system_checks.py b/tests/test_system_checks.pynew file mode 100644index 0000000..2bcee91--- /dev/null+++ b/tests/test_system_checks.py@@ -0,0 +1,150 @@+import os++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker++from src.agent import collector+from src.backend import main+from src.backend.models import Base, IncidentAlert, Server, SystemEvent, User++TOKEN = "collector-token"+BOOT_A, BOOT_B = "boot-a", "boot-b"+++def db():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))+ session.add(Server(id=5, owner_id=1, hostname="nas", check_interval_seconds=60, collector_token_hash=main._hash_secret(TOKEN)))+ session.commit()+ return session+++def _ingest(db, **system):+ host = {"boot_id": BOOT_A, "oom_kill_total": 0, "kernel_tainted": 0, "ntp_synced": True, "reboot_required": False, **system}+ main.ingest_metrics(main.MetricsIngestRequest(server_id=5, host=host), authorization=f"Bearer {TOKEN}", session_id=None, db=db)+++def _active(db):+ return {alert.rule_key: alert for alert in db.query(IncidentAlert).filter_by(status="active")}+++def _kinds(db):+ return [event.kind for event in db.query(SystemEvent).order_by(SystemEvent.id)]+++def test_first_check_in_records_no_events(db):+ _ingest(db)+ assert _kinds(db) == [] and _active(db) == {}+++def test_new_boot_id_records_a_reboot_and_raises_info_alert(db):+ _ingest(db)+ _ingest(db, boot_id=BOOT_B)++ assert _kinds(db) == ["reboot"]+ assert _active(db)["system:reboot"].severity == "INFO"+++def test_reboot_alert_clears_after_the_event_window(db):+ _ingest(db)+ _ingest(db, boot_id=BOOT_B)+ for event in db.query(SystemEvent):+ event.time = event.time.replace(year=2000)+ db.commit()+ _ingest(db, boot_id=BOOT_B)++ assert "system:reboot" not in _active(db)+++def test_oom_counter_rise_raises_alert_but_a_reboot_resetting_it_does_not(db):+ _ingest(db, oom_kill_total=4)+ _ingest(db, oom_kill_total=6)+ assert _kinds(db) == ["oom_kill"]+ assert _active(db)["system:oom"].severity == "WARNING"++ _ingest(db, boot_id=BOOT_B, oom_kill_total=0)+ assert _kinds(db) == ["oom_kill", "reboot"]+++def test_repeated_oom_kills_are_critical(db):+ for total in range(4):+ _ingest(db, oom_kill_total=total)+ assert _active(db)["system:oom"].severity == "CRITICAL"+++def test_kernel_fault_taint_alerts_until_reboot_but_module_taint_is_ignored(db):+ _ingest(db, kernel_tainted=1 | 4096) # proprietary + out-of-tree module only+ assert _active(db) == {} and _kinds(db) == []++ _ingest(db, kernel_tainted=1 | 1 << 4 | 1 << 7)+ assert _kinds(db) == ["kernel_fault"]+ assert "machine check" in db.query(SystemEvent).one().detail+ _ingest(db, kernel_tainted=1 | 1 << 4 | 1 << 7)+ assert "system:kernel" in _active(db) and _kinds(db) == ["kernel_fault"]++ _ingest(db, boot_id=BOOT_B, kernel_tainted=0)+ assert "system:kernel" not in _active(db)+++def test_ntp_alert_needs_sustained_unsynced_checks_and_resolves(db):+ _ingest(db, ntp_synced=False)+ _ingest(db, ntp_synced=False)+ assert "system:ntp" not in _active(db)+ _ingest(db, ntp_synced=False)+ assert "system:ntp" in _active(db)+ _ingest(db, ntp_synced=True)+ assert "system:ntp" not in _active(db)+++def test_unknown_values_never_alert(db):+ for _ in range(4):+ _ingest(db, boot_id=None, oom_kill_total=None, kernel_tainted=None, ntp_synced=None, reboot_required=None)+ assert _active(db) == {}+++def test_malformed_values_from_a_collector_are_dropped(db):+ _ingest(db, oom_kill_total="many", kernel_tainted=True, ntp_synced="yes", reboot_required=1, boot_id="x" * 100)+ from src.backend.models import HostMetric+ row = db.query(HostMetric).one()+ assert (row.oom_kill_total, row.kernel_tainted, row.ntp_synced, row.reboot_required) == (None, None, None, None)+ assert row.boot_id == "x" * 64+++def test_reboot_required_raises_info_alert(db):+ _ingest(db, reboot_required=True)+ assert _active(db)["system:reboot_required"].severity == "INFO"+ _ingest(db, reboot_required=False)+ assert "system:reboot_required" not in _active(db)+++def test_collector_reads_kernel_state_from_proc(tmp_path, monkeypatch):+ (tmp_path / "sys/kernel/random").mkdir(parents=True)+ (tmp_path / "sys/kernel/random/boot_id").write_text("abc-123\n")+ (tmp_path / "sys/kernel/tainted").write_text("128\n")+ (tmp_path / "vmstat").write_text("nr_free_pages 1\noom_kill 7\n")+ monkeypatch.setattr(collector, "HOST_PROC", tmp_path)+ monkeypatch.setattr(collector, "HOST_ROOT", tmp_path)+ (tmp_path / "etc").mkdir()+ (tmp_path / "etc/debian_version").write_text("12\n")++ state = collector.collect_system()+ assert (state["boot_id"], state["kernel_tainted"], state["oom_kill_total"], state["reboot_required"]) == ("abc-123", 128, 7, False)++ (tmp_path / "run").mkdir()+ (tmp_path / "run/reboot-required").touch()+ assert collector.collect_system()["reboot_required"] is True+++def test_collector_reports_unknown_when_files_are_missing(tmp_path, monkeypatch):+ monkeypatch.setattr(collector, "HOST_PROC", tmp_path)+ monkeypatch.setattr(collector, "HOST_ROOT", tmp_path)++ state = collector.collect_system()+ assert (state["boot_id"], state["kernel_tainted"], state["oom_kill_total"], state["reboot_required"]) == (None, None, None, None)diff --git a/tests/test_system_helper.py b/tests/test_system_helper.pynew file mode 100644index 0000000..95b447d--- /dev/null+++ b/tests/test_system_helper.py@@ -0,0 +1,257 @@+import json+import os+import struct+import time+from datetime import datetime, timedelta, timezone++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker++from src.agent import collector, system_helper as helper+from src.backend import main+from src.backend.models import Base, IncidentAlert, Server, SystemEvent, User++TOKEN = "collector-token"+BOOT = "boot-a"+NOW = 1_000_000.0+++def _klog(*lines):+ return "\n".join(lines) + "\n"+++def _parse(text, boot_id=BOOT):+ return helper.parse_kernel_log(text, boot_id, monotonic_now=500.0, wall_now=NOW)+++# --- kernel log parsing -------------------------------------------------------------------++def test_oom_kill_is_parsed_with_process_name_and_wall_time():+ [event] = _parse(_klog("<3>[ 400.000000] Out of memory: Killed process 4321 (java) total-vm:100kB"))++ assert (event["kind"], event["key"]) == ("oom_kill", "boot-a:oom:400.000000")+ assert event["detail"] == "The kernel OOM killer stopped java (pid 4321)"+ # 100 s before the snapshot: monotonic 500 vs 400.+ assert datetime.fromisoformat(event["time"]) == datetime.fromtimestamp(NOW - 100, timezone.utc)+++def test_cgroup_oom_is_recognised():+ [event] = _parse(_klog("<3>[ 10.500000] Memory cgroup out of memory: Killed process 77 (postgres) total-vm:1kB"))+ assert event["kind"] == "oom_kill" and "postgres" in event["detail"]+++def test_kernel_errors_are_kept_but_info_lines_are_not():+ events = _parse(_klog("<6>[ 1.000000] eth0: link up", "<4>[ 2.000000] a warning", "<3>[ 3.000000] nvme0: I/O error, dev nvme0n1", "<0>[ 4.000000] kernel panic"))+ assert [event["detail"] for event in events] == ["nvme0: I/O error, dev nvme0n1", "kernel panic"]+ assert {event["kind"] for event in events} == {"kernel_error"}+++def test_hostile_log_content_is_neutralised():+ name = "x\x1b[31m;$(reboot)\x00" + "A" * 500+ [oom] = _parse(_klog(f"<3>[ 5.000000] Out of memory: Killed process 1 ({name})"))+ [err] = _parse(_klog("<3>[ 6.000000] bad \x1b[2J\x07 line " + "B" * 1000))++ assert all(char.isprintable() for char in oom["detail"] + err["detail"])+ assert len(oom["detail"]) < 150 and len(err["detail"]) <= helper.MAX_TEXT+++def test_malformed_and_untimestamped_lines_are_ignored():+ assert _parse(_klog("garbage", "<3>no timestamp here", "<x>[1.000000] y", " continuation line", "<3>[abc] z", "")) == []+++def test_output_is_capped_to_the_newest_events():+ lines = [f"<3>[{n:>8}.000000] error number {n}" for n in range(1, 40)]+ events = _parse(_klog(*lines))+ assert len(events) == helper.MAX_ERROR_EVENTS and events[-1]["detail"] == "error number 39"+++# --- wtmp ---------------------------------------------------------------------------------++def _record(kind, user, when):+ return helper._UTMP.pack(kind, 0, b"~", b"~~", user.encode(), b"", 0, 0, 0, int(when), 0, 0, 0, 0, 0)+++def _wtmp(tmp_path, *records):+ path = tmp_path / "wtmp"+ path.write_bytes(b"".join(_record(*record) for record in records))+ return path+++def test_utmp_layout_is_384_bytes():+ assert helper._UTMP.size == 384+++def test_clean_shutdown_is_detected(tmp_path):+ path = _wtmp(tmp_path, (2, "reboot", 100), (1, "shutdown", 900), (2, "reboot", 1000))+ assert helper.classify_last_shutdown(helper.read_wtmp_records(path), boot_time=1002) == "clean"+++def test_missing_shutdown_record_is_unclean_when_shutdowns_were_recorded_before(tmp_path):+ path = _wtmp(tmp_path, (2, "reboot", 100), (1, "shutdown", 200), (2, "reboot", 300), (2, "reboot", 1000))+ assert helper.classify_last_shutdown(helper.read_wtmp_records(path), boot_time=1000) == "unclean"+++def test_never_recorded_shutdowns_are_unknown_not_unclean(tmp_path):+ path = _wtmp(tmp_path, (2, "reboot", 100), (2, "reboot", 300), (2, "reboot", 1000))+ assert helper.classify_last_shutdown(helper.read_wtmp_records(path), boot_time=1000) == "unknown"+++def test_unknown_when_wtmp_is_missing_short_or_out_of_date(tmp_path):+ assert helper.read_wtmp_records(tmp_path / "nope") == []+ assert helper.classify_last_shutdown([], 1000) == "unknown"+ one_boot = helper.read_wtmp_records(_wtmp(tmp_path, (2, "reboot", 1000)))+ assert helper.classify_last_shutdown(one_boot, 1000) == "unknown"+ stale = helper.read_wtmp_records(_wtmp(tmp_path, (1, "shutdown", 10), (2, "reboot", 100), (2, "reboot", 200)))+ assert helper.classify_last_shutdown(stale, boot_time=5000) == "unknown"+++def test_truncated_or_symlinked_wtmp_is_safe(tmp_path):+ path = _wtmp(tmp_path, (2, "reboot", 1))+ path.write_bytes(path.read_bytes() + b"junk")+ assert len(helper.read_wtmp_records(path)) == 1+ link = tmp_path / "link"+ link.symlink_to(path)+ assert helper.read_wtmp_records(link) == []+++def test_snapshot_is_written_atomically_with_a_fixed_schema(tmp_path):+ out = tmp_path / "system.json"+ helper.write_snapshot({"version": 1, "events": []}, out)+ assert json.loads(out.read_text()) == {"version": 1, "events": []}+ assert not (tmp_path / "system.json.tmp").exists() and out.stat().st_mode & 0o777 == 0o644+++# --- collector reading the helper's file --------------------------------------------------++def _helper_file(tmp_path, **overrides):+ data = {"version": 1, "generated_at": datetime.now(timezone.utc).isoformat(), "boot_id": BOOT, "last_shutdown": "clean",+ "events": [{"kind": "oom_kill"}], **overrides}+ path = tmp_path / "system.json"+ path.write_text(json.dumps(data))+ return path+++def test_collector_uses_a_fresh_file_from_the_current_boot(tmp_path):+ assert collector.collect_helper(BOOT, _helper_file(tmp_path)) == {"last_shutdown": "clean", "helper_events": [{"kind": "oom_kill"}]}++[email protected]("overrides", [{"boot_id": "other"}, {"generated_at": "2001-01-01T00:00:00+00:00"}, {"generated_at": "nonsense"}])+def test_collector_ignores_stale_or_foreign_files(tmp_path, overrides):+ assert collector.collect_helper(BOOT, _helper_file(tmp_path, **overrides)) == {}+++def test_collector_refuses_missing_symlinked_oversized_and_garbage_files(tmp_path):+ assert collector.collect_helper(BOOT, tmp_path / "missing.json") == {}+ real = _helper_file(tmp_path)+ link = tmp_path / "link.json"+ link.symlink_to(real)+ assert collector.collect_helper(BOOT, link) == {}+ big = tmp_path / "big.json"+ big.write_text(" " * (collector.HELPER_FILE_MAX_BYTES + 10))+ assert collector.collect_helper(BOOT, big) == {}+ bad = tmp_path / "bad.json"+ bad.write_text("[1,2")+ assert collector.collect_helper(BOOT, bad) == {}+ arr = tmp_path / "arr.json"+ arr.write_text("[]")+ assert collector.collect_helper(BOOT, arr) == {}+++# --- backend ingest -----------------------------------------------------------------------++def db():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))+ session.add(Server(id=5, owner_id=1, hostname="nas", check_interval_seconds=60, collector_token_hash=main._hash_secret(TOKEN)))+ session.commit()+ return session+++def _ingest(db, **host):+ host = {"boot_id": BOOT, "oom_kill_total": 0, "kernel_tainted": 0, "ntp_synced": True, "reboot_required": False, **host}+ main.ingest_metrics(main.MetricsIngestRequest(server_id=5, host=host), authorization=f"Bearer {TOKEN}", session_id=None, db=db)+++def _event(kind="oom_kill", key="boot-a:oom:1.000000", minutes_ago=5, detail="The kernel OOM killer stopped java (pid 1)"):+ return {"kind": kind, "key": key, "detail": detail, "time": (datetime.now(timezone.utc) - timedelta(minutes=minutes_ago)).isoformat()}+++def _events(db):+ return [(e.kind, e.event_key) for e in db.query(SystemEvent).order_by(SystemEvent.id)]+++def _alerts(db):+ return {a.rule_key: a for a in db.query(IncidentAlert).filter_by(status="active")}+++def test_helper_events_are_stored_once_even_when_republished(db):+ _ingest(db, helper_events=[_event()], last_shutdown="unknown")+ _ingest(db, helper_events=[_event()], last_shutdown="unknown")++ assert _events(db) == [("oom_kill", "boot-a:oom:1.000000")]+ assert "java" in _alerts(db)["system:oom"].description+++def test_with_the_helper_the_oom_counter_does_not_add_a_second_event(db):+ _ingest(db, oom_kill_total=0, helper_events=[], last_shutdown="unknown")+ _ingest(db, oom_kill_total=1, helper_events=[_event()], last_shutdown="unknown")++ assert _events(db) == [("oom_kill", "boot-a:oom:1.000000")]++[email protected]("bad", [+ {"kind": "reboot", "key": "k", "detail": "d", "time": "2026-01-01T00:00:00+00:00"}, # kind not allowed from a helper+ _event(key="has spaces"), _event(key="x" * 200), _event(minutes_ago=60 * 24 * 30), _event(minutes_ago=-600),+ {**_event(), "time": "garbage"}, {**_event(), "detail": 5}, "string", None, 12,+])+def test_malformed_helper_events_are_dropped(db, bad):+ _ingest(db, helper_events=[bad], last_shutdown="unknown")+ assert _events(db) == []+++def test_helper_event_count_and_text_are_bounded(db):+ many = [_event(key=f"boot-a:err:{n}", kind="kernel_error", detail="e" * 5000) for n in range(500)]+ _ingest(db, helper_events=many, last_shutdown="unknown")++ rows = db.query(SystemEvent).all()+ assert len(rows) == 50 and all(len(row.detail) <= 300 for row in rows)+++def test_unclean_shutdown_makes_the_reboot_critical(db):+ _ingest(db)+ _ingest(db, boot_id="boot-b", helper_events=[], last_shutdown="unclean")++ alert = _alerts(db)["system:reboot"]+ assert alert.severity == "CRITICAL" and "unclean" in alert.title+ assert db.query(SystemEvent).one().detail.startswith("The host restarted after an unclean shutdown")+++def test_clean_shutdown_keeps_the_reboot_informational(db):+ _ingest(db)+ _ingest(db, boot_id="boot-b", helper_events=[], last_shutdown="clean")++ assert _alerts(db)["system:reboot"].severity == "INFO"+ assert "clean shutdown" in db.query(SystemEvent).one().detail+++def test_helper_arriving_after_the_reboot_was_noticed_updates_the_event_and_alert(db):+ _ingest(db)+ _ingest(db, boot_id="boot-b") # collector saw the new boot before the helper had written anything+ assert _alerts(db)["system:reboot"].severity == "INFO"++ _ingest(db, boot_id="boot-b", helper_events=[], last_shutdown="unclean")++ assert _alerts(db)["system:reboot"].severity == "CRITICAL"+++def test_invalid_last_shutdown_values_are_dropped(db):+ from src.backend.models import HostMetric+ _ingest(db, last_shutdown="<script>")+ assert db.query(HostMetric).one().last_shutdown is None