Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

feat: added claude skills for easier security review

karloks200505cbda46b4cc
diff --git a/.claude/skills/security-audit/SKILL.md b/.claude/skills/security-audit/SKILL.md
new file mode 100644
index 0000000..889946f
--- /dev/null
+++ b/.claude/skills/security-audit/SKILL.md
@@ -0,0 +1,167 @@
+---
+name: security-audit
+description: Detaljan sigurnosni pregled (security review / audit) cijelog koda ili dijela koda, uz dojavu pronađenih propusta s ozbiljnošću, lokacijom, scenarijem napada i popravkom. Use when the user asks for a security review, security audit, vulnerability hunt, pentest-style code review, "provjeri sigurnost", "sigurnosni pregled", "ima li propusta", or wants a component (backend, collector, docker proxy, nginx, compose, CI, frontend, assistant) reviewed for security. For a quick check of only the pending diff, the built-in security-review skill is enough.
+argument-hint: "[full | backend | collector | infra | frontend | assistant | <path> | diff]"
+---
+
+# Security audit
+
+Cilj: pronaći **stvarne, iskoristive** sigurnosne propuste u ovom repozitoriju i dojaviti ih
+tako da ih vlasnik može odmah popraviti. Projekt je open-source, self-hosted alat koji ima
+pristup Docker socketu i hostu, pa vlasnik želi **strogu** procjenu: ne ublažavaj rizik
+radi udobnosti, ne zaokružuj "vjerojatno OK" na "OK".
+
+Ovaj skill **ne mijenja kod** i ne commitira. Samo pregledava i izvještava. Popravke radiš
+tek kad korisnik to izričito zatraži.
+
+## Argumenti
+
+`$ARGUMENTS` određuje opseg. Bez argumenta = `full`.
+
+| Argument | Opseg |
+|---|---|
+| `full` | sve dolje, redom |
+| `backend` | `src/backend/**` |
+| `collector` | `src/agent/**` (collector, docker_proxy, system_helper, assistant, health_checks) |
+| `infra` | `docker/**`, `*compose*.yml`, `example.env`, `.github/**`, `.gitignore`, `.dockerignore` |
+| `frontend` | `src/frontend/**` |
+| `assistant` | LLM asistent: `src/agent/{service,prompts,health_checks}.py`, `SYSTEM_PROMPT.md`, konverzacije u backendu |
+| `<path>` | taj put (datoteka ili direktorij) plus ono što ga poziva |
+| `diff` | samo neprihvaćene promjene (`git diff` + `git diff --staged`) pročitane u kontekstu cijelih datoteka |
+
+Ako je argument dvosmislen, odaberi najbliži opseg i na početku izvještaja reci koji si odabrao.
+
+## Pravila rada (čitaj prije početka)
+
+1. **Čitaj datoteke cijele.** `grep` služi za pronalaženje kandidata, nikad za oslobađanje
+ koda od sumnje. Funkcija koja izgleda sigurno često ima rupu u pozivatelju ili u redoslijedu
+ provjera. `src/backend/main.py` ima ~1800 redaka; pročitaj ga u cijelosti, u dijelovima.
+2. **Ne ispisuj tajne.** `.env` postoji lokalno. Ako ga treba otvoriti, ispiši samo *imena*
+ varijabli i je li vrijednost prazna/zadana, nikad vrijednosti. Isto vrijedi za tokene,
+ ključeve i lozinke pronađene u kodu ili povijesti: u izvještaju navedi lokaciju i vrstu,
+ ne sadržaj (najviše prva 4 znaka).
+3. **Samo lokalno i neškodljivo.** Ne šalji zahtjeve prema stvarnim sustavima, ne pokreći
+ Docker kontejnere ni skeniranja. Za potvrdu propusta smiješ pokrenuti postojeće testove
+ (`pytest`) i napisati kratku skriptu u scratchpad direktoriju koja poziva funkcije
+ izravno s neprijateljskim ulazom. Ništa što pokreneš ne smije dirati stvarnu bazu ili
+ mrežu.
+4. **Provjeri tvrdnje iz dokumentacije.** `SECURITY.md` i `docs/` tvrde mnogo ("rate limited",
+ "single-use", "capped", "never follows symlinks"). Za svaku tvrdnju u svom opsegu nađi kod
+ koji to provodi i provjeri da vrijedi na *svim* putevima. Tvrdnja bez pokrića u kodu je
+ nalaz (dokumentirana kontrola ne postoji), a kontrola koja postoji samo na jednom putu
+ također.
+5. **Razmišljaj kao napadač, ne kao lintar.** Za svaki ulaz pitaj: tko ga može poslati, što je
+ najgore što može sadržavati, što sustav radi s tim i što je najgore što se zbog toga može
+ dogoditi. Profile napadača vidi u [references/threat-model.md](references/threat-model.md).
+6. **Prvo provjeri je li nešto već riješeno.** Prije nego prijaviš nalaz, nađi postojeću
+ kontrolu (middleware, dependency, nginx pravilo, test) i provjeri zaobilazi li se. Nalaz
+ koji ignorira postojeću zaštitu je šum i troši korisnikovo povjerenje.
+7. **Svaki nalaz verificiraj** (vidi niže) i razvrstaj kao `POTVRĐENO` ili `TREBA PROVJERU`.
+ Ne prijavljuj ono što nisi mogao provjeriti kao potvrđeno.
+
+## Postupak
+
+### 1. Priprema (uvijek)
+
+- Pročitaj [references/threat-model.md](references/threat-model.md) i `SECURITY.md`.
+- `git status` i `git log --oneline -15`: što se nedavno mijenjalo, nove površine napada
+ najčešće su u svježem kodu.
+- Za `backend`/`full`: izlistaj sve rute (`grep -nE "^@(app|router)\.(get|post|put|patch|delete)" src/backend/main.py`)
+ i složi **inventar ruta**: metoda, put, tko smije (anonimno / bilo koji korisnik / admin /
+ kolektor s tokenom), je li iza CSRF/Origin provjere, ima li rate limit, što čita/piše.
+ Ruta bez jasnog odgovora u bilo kojem stupcu je kandidat za nalaz.
+- Pročitaj popis ovisnosti (`requirements*.txt`, `src/frontend/package.json`) radi kasnije
+ provjere.
+
+### 2. Pregled po područjima
+
+Za svako područje u opsegu pročitaj odgovarajuću kontrolnu listu i prođi je **stavku po stavku**
+nad stvarnim kodom:
+
+| Područje | Kontrolna lista |
+|---|---|
+| Backend (API, auth, sesije, 2FA, baza, pravila, alarmi) | [references/checklist-backend.md](references/checklist-backend.md) |
+| Collector, docker proxy, system helper, LLM asistent | [references/checklist-collector-agent.md](references/checklist-collector-agent.md) |
+| Docker, compose, nginx, Postgres, CI, repozitorij, ovisnosti | [references/checklist-infra.md](references/checklist-infra.md) |
+| Frontend | [references/checklist-frontend.md](references/checklist-frontend.md) |
+
+Kontrolne liste su podsjetnici, **ne granica**. Ako vidiš nešto sumnjivo što nije na listi,
+istraži i to.
+
+**Paralelizacija za `full`:** opseg je velik. Ako ti je dostupan alat `Agent`, pokreni četiri
+agenta paralelno (po jedno područje iz tablice), svakome daj samostalan prompt iz predloška
+niže, a zatim sam spoji, deduplikaj i **ponovno verificiraj** sve što su vratili prije
+izvještaja. Nalaz koji agent prijavi nije potvrđen dok ga ti nisi pročitao u kodu. Ako ne
+možeš koristiti agente, prođi područja redom sam.
+
+Predložak za agenta:
+
+> Radiš sigurnosni pregled (samo čitanje, ništa ne mijenjaj, ne commitaj, ne ispisuj vrijednosti
+> tajni) područja **<područje>** u repozitoriju <putanja>. Prvo pročitaj
+> `.claude/skills/security-audit/references/threat-model.md` i
+> `.claude/skills/security-audit/references/<checklist>.md`, zatim pročitaj **cijele** datoteke
+> tog područja i prođi kontrolnu listu stavku po stavku. Za svaki nalaz vrati: naslov,
+> datoteka:redak, što je pogrešno, konkretan scenarij napada (tko, što pošalje, što dobije),
+> koju postojeću zaštitu si provjerio i zašto ne pomaže, predloženi popravak i svoju sigurnost
+> (visoka/srednja/niska). Navedi i koje datoteke si pročitao, a koje nisi. Ne prijavljuj
+> teorijske probleme bez puta do iskorištenja.
+
+### 3. Verifikacija svakog nalaza
+
+Za svaki kandidat napiši (barem u glavi, a u izvještaju sažeto):
+
+1. **Izvor:** tko kontrolira ulaz i na kojem je povjerenju (anonimni, korisnik, admin,
+ kolektor, VPN peer, podatak s hosta)?
+2. **Put:** kojim funkcijama ulaz stiže do opasnog mjesta? Navedi datoteke i retke.
+3. **Zaštite na putu:** koje provjere, validacije, limiti, nginx pravila postoje? Zašto ne
+ pomažu (ili zašto pomažu, pa nalaza nema)?
+4. **Učinak:** što napadač dobiva (čitanje podataka, promjena stanja, preuzimanje računa,
+ izvršavanje koda, DoS, eskalacija na host)?
+5. **Dokaz:** gdje je moguće, potvrdi testom ili kratkom lokalnom skriptom. Ako ne možeš,
+ nalaz je `TREBA PROVJERU` s opisom što točno treba provjeriti.
+
+Odbaci kandidata ako: nema puta do iskorištenja bez prethodnog admin pristupa koji već daje
+isti učinak, ako je zaštićen nečim što si provjerio, ili ako je čista stilska preporuka.
+Takve stvari, ako vrijede, idu u dio "Očvršćivanje" kratkom stavkom, ne među nalaze.
+
+### 4. Ozbiljnost
+
+Ocjenjuj prema učinku **u ovom sustavu** (pristup hostu i Docker API-ju je najskuplja stvar
+koju treba braniti), uz prednost strožoj ocjeni kad si u dvojbi između dvije razine:
+
+| Razina | Značenje | Primjeri |
+|---|---|---|
+| **KRITIČNO** | Daljinsko, bez/gotovo bez autentikacije, vodi do preuzimanja dashboarda, izvršavanja koda ili izlaska na host | zaobilaženje autentikacije, preuzimanje prve admin registracije, RCE, pristup Docker socketu iz javne strane |
+| **VISOKO** | Autentificirani ili uvjetovani napad s velikim učinkom | eskalacija korisnik→admin, zaobilaženje 2FA, IDOR nad tuđim podacima, SQL injection, stored XSS u admin kontekstu, kompromitirani kolektor utječe na dashboard ili host |
+| **SREDNJE** | Stvarno iskoristivo, ograničen učinak ili traži poseban uvjet | CSRF na manje osjetljivoj akciji, rate limit koji se zaobilazi, curenje informacija, DoS jednim zahtjevom |
+| **NISKO** | Slabost bez jasnog puta do štete sam za sebe | nedostajući header, enumeracija korisnika, preslab zadani parametar |
+| **INFO** | Zapažanje ili očvršćivanje | kontrola koja radi, ali je krhka |
+
+### 5. Izvještaj
+
+Format i redoslijed: [references/report-format.md](references/report-format.md). Ukratko:
+sažetak (broj nalaza po razini, najvažniji jedan redak), **jedna tablica svih nalaza sortirana
+prema riziku od najvišeg prema najnižem** (KRITIČNO, VISOKO, SREDNJE, NISKO, INFO), detalji
+nalaza istim redom, tvrdnje iz dokumentacije koje kod ne podržava, očvršćivanje i pokrivenost
+(što je pregledano, što nije i zašto).
+
+**Tablica je obavezna.** Nalaze nikad ne prikazuj samo kao tekst niti grupirane po područjima.
+Rezultate paralelnih agenata spoji u jednu tablicu i sortiraj ih zajedno.
+
+Datoteke referenciraj kao markdown linkove (`[main.py:412](src/backend/main.py#L412)`), da su
+klikabilne u editoru.
+
+Izvještaj ispiši u razgovoru. **Ne spremaj ga u datoteku** (npr. `docs/`) dok to korisnik ne
+zatraži, jer može sadržavati neobjavljene ranjivosti.
+
+Na kraju ponudi sljedeći korak jednom rečenicom (npr. popraviti nalaze od najtežeg prema
+najlakšem, uz test za svaki) i **čekaj odgovor**.
+
+## Što ovaj skill nije
+
+- Nije dinamičko testiranje: ne skenira živi sustav, ne fuzzira, ne pokreće exploite.
+- Nije zamjena za `pip-audit` / `npm audit` ili skener slika; ako su dostupni i korisnik
+ odobri mrežu, pokreni ih, a ako nisu, to **eksplicitno navedi** u pokrivenosti kao
+ nepregledano (ovisnosti se ne ocjenjuju "iz glave" jer verzije nisu pouzdan izvor).
+- Ne potvrđuje da je sustav siguran. Izvještaj "nema nalaza" smije reći samo "u pregledanom
+ opsegu nisam našao iskoristive propuste", uz popis onoga što je pregledano.
diff --git a/.claude/skills/security-audit/references/checklist-backend.md b/.claude/skills/security-audit/references/checklist-backend.md
new file mode 100644
index 0000000..d4fa8e4
--- /dev/null
+++ b/.claude/skills/security-audit/references/checklist-backend.md
@@ -0,0 +1,140 @@
+# Kontrolna lista: backend (`src/backend/**`)
+
+Čitaj `main.py`, `auth.py`, `totp.py`, `secrets_box.py`, `rate_limit.py`, `rules.py`,
+`models.py`, `db.py`, `config.py`, `history.py`, `rollups.py`, `stats.py`, `alerts.py`,
+`seed_user.py`, `reset_2fa.py` u cijelosti. Za svaku stavku nađi kod koji je provodi.
+
+## 1. Autentikacija i autorizacija (najveći prioritet)
+
+- [ ] **Svaka ruta ima svjestan odgovor** na "tko smije?" (inventar ruta iz koraka 1). Ruta
+ koja zaboravi `Depends(...)` za auth je zadano **otvorena**: traži takve.
+- [ ] Razlika *anonimno / prijavljen / admin / kolektor* provodi se na serveru, nikad samo u
+ frontendu. Admin-only rute stvarno provjeravaju ulogu (ne samo prijavu).
+- [ ] **Ograničena sesija** (admin bez 2FA dok je politika uključena) stvarno može doći samo do
+ profila, postavljanja 2FA i odjave. Provjeri *svaku* rutu, ne samo ono što test pokriva,
+ uključujući nove rute dodane nakon te značajke.
+- [ ] **IDOR / horizontalna autorizacija:** rute s ID-evima (server, kontejner, konverzacija,
+ sesija, token, korisnik, pravilo, alarm) provjeravaju vlasništvo/ulogu. Posebno: popis
+ sesija po javnim ID-evima, brisanje tuđe sesije, konverzacije drugog korisnika.
+- [ ] **Mass assignment:** modeli zahtjeva ne dopuštaju polja kao `is_admin`, `role`,
+ `require_2fa`, `server_id`, `owner_id`. Provjeri Pydantic modele i `**body`/`setattr`.
+- [ ] Zadnji admin ne može obrisati/degradirati sam sebe do stanja bez admina (ili je to
+ namjerno i dokumentirano). Admin ne može eskalirati drugog bez provjere.
+- [ ] Prvi račun postaje admin: provjeri **sve** puteve (setup token, `ALLOW_REGISTRATION`,
+ utrka dvaju istodobnih zahtjeva, `ADMIN_BOOTSTRAP_TOKEN` usporedba, brisanje tokena nakon
+ prvog računa). Je li usporedba tokena konstantnog vremena (`hmac.compare_digest`)?
+
+## 2. Sesije, kolačići, CSRF
+
+- [ ] ID sesije: iz `secrets` (ne `random`/`uuid1`), dovoljna entropija, pohranjen hashiran ili
+ barem nikad vraćen pregledniku u tijelu odgovora. Rotacija pri promjeni privilegija
+ (prijava, 2FA, promjena lozinke).
+- [ ] Zastavice kolačića: `HttpOnly`, `Secure` (i u kojim uvjetima se gasi: provjeri da
+ zadana vrijednost nije nesigurna), `SameSite`, `Path`, istek.
+- [ ] Istek i odjava stvarno brišu sesiju na serveru; logout-everywhere radi.
+- [ ] **Provjera `Origin`/`Sec-Fetch-Site`** na POST/PUT/PATCH/DELETE: pokušaj zaobilaženja:
+ `Origin` s istim prefiksom (`https://dash.example.com.evil.tld`), različit port/shema,
+ `Origin: null`, nedostajući `Origin` + `Sec-Fetch-Site: same-site`, zaglavlje `Host`
+ pod kontrolom napadača, `CORS_ORIGINS` s džokerima. Radi li provjera za **sve** metode
+ i sve rute (middleware ili po ruti)? Postoji li izuzeće za puteve kolektora i može li ga
+ se zloupotrijebiti da bi se zaobišla provjera na ruti korisnika?
+- [ ] CORS: `allow_origins` nije `*` uz `allow_credentials=True`; nema refleksije proizvoljnog
+ `Origin`.
+
+## 3. Lozinke, 2FA, tokeni
+
+- [ ] Hashiranje lozinke: bcrypt/argon2 s razumnim troškom; granica 72 B provodi se (odbija,
+ ne skraćuje); nema lozinki u logovima; promjena lozinke traži staru lozinku (ili je
+ sesija svježa) i poništava ostale sesije.
+- [ ] **Timing/enumeracija računa:** prijava, reset, registracija vraćaju iste odgovore i
+ slično vrijeme za nepostojeći email i pogrešnu lozinku (fiktivni hash za nepostojeće
+ račune?).
+- [ ] **Rate limit** (`rate_limit.py`): ključ (IP, email) ne može se zaobići promjenom
+ zaglavlja (`X-Forwarded-For` iz nepouzdanog izvora), velikim/malim slovima ili
+ razmacima u emailu, unicode normalizacijom; memorija ograničena (napadač ne može
+ napuhati strukturu milijunima ključeva); rade li limiti pod više radnih procesa
+ (stanje u memoriji po procesu = efektivni limit × broj procesa)?
+- [ ] **TOTP** (`totp.py`): usporedba konstantnog vremena; prozor ±1; **zabrana ponovne
+ uporabe istog koda je atomarna** (jedinstveni indeks/UPDATE s uvjetom, ne
+ čitaj-pa-piši); brojač pokušaja po računu i po adresi ne resetira se ponovnim
+ početkom lozinke; međutoken (5 min) je jednokratan, vezan uz korisnika i ne vrijedi
+ kao sesija nigdje.
+- [ ] **Recovery kodovi:** entropija, hashirani, jednokratni *atomarno*, brišu se/ponovno
+ generiraju pri isključivanju 2FA.
+- [ ] `secrets_box.py`: Fernet ključ iz okoline, ponašanje bez ključa ne smije ostaviti
+ tajne u čistom tekstu; greška dekripcije ne curi detalje; ključ se ne logira.
+- [ ] **Tokeni servera i join tokeni:** generirani iz `secrets`, hashirani (SHA-256 je u redu
+ za tokene visoke entropije), usporedba konstantnog vremena ili pretraga po hashu,
+ jednokratnost i istek **atomarni**, ograničenje otvorenih tokena (20) provodi se i pod
+ paralelnim zahtjevima, brisanje servera poništava token.
+- [ ] Ponovno uparivanje postojećeg stroja (isti machine ID) traži `replace: true` na
+ serveru, ne samo u UI-ju; mogu li se vjerodajnice tuđeg servera prepisati bez toga?
+
+## 4. Ulazi koje šalje kolektor (nepouzdani)
+
+- [ ] Svaka ruta kolektora (register, poll, enroll, ingest, helper događaji) odbija se kad
+ zahtjev nije došao kroz gate (`X-Collector-Via` / `COLLECTOR_ACCESS=strict`); provjeri
+ da klijent ne može sam poslati to zaglavlje preko javne strane (nginx ga mora
+ prebrisati/ukloniti, backend ga ne smije slijepo vjerovati ako je backend dostupan
+ drugim kontejnerima).
+- [ ] Autentikacija kolektora: token po serveru; kolektor jednog servera ne može pisati
+ podatke drugog (`server_id` iz tokena, ne iz tijela zahtjeva!).
+- [ ] **Validacija:** tipovi, duljine, rasponi, strogi parsing vremena, `sha256:`+64 hex,
+ maks. 500 kontejnera, imena ≤128, ograničenja događaja po check-inu i po satu. Traži
+ polja koja se spremaju bez čišćenja (nova polja dodana kasnije često zaobiđu).
+- [ ] Veličina tijela zahtjeva ograničena (nginx `client_max_body_size` **i** Pydantic
+ `max_length`), dubina/veličina JSON-a, broj elemenata lista.
+- [ ] Idempotentnost/replay: ponovljeni check-in ne stvara duplicirane incidente ni neograničen
+ rast tablica. Hostilni podaci ne mogu "utišati" alarme drugih servera.
+
+## 5. Baza i injekcije
+
+- [ ] Svaki `text(...)`, `execute(f"...")`, `.format`, `%` ili spajanje stringova u SQL-u:
+ jesu li vrijednosti vezane parametrima? Dinamički `ORDER BY`, imena stupaca/metrika i
+ `LIMIT` iz zahtjeva idu samo kroz **bijelu listu**.
+- [ ] Imena kontejnera i putanje mountova ulaze u ključeve incidenata/cache ključeve: mogu
+ li se sudarati ili ubrizgati separator (`:`, `|`, newline) i spojiti dva incidenta?
+- [ ] LIKE upiti s korisničkim unosom (escape `%`/`_`), regex iz korisničkog unosa (ReDoS;
+ `rules.py` ima 593 retka pravila, provjeri koje izraze evaluira i je li to sigurno).
+- [ ] **`rules.py`: izvršava li se išta dinamički** (`eval`, `exec`, `compile`, `getattr` s
+ korisničkim imenom, `importlib`, šabloni stringova s korisničkim formatom)? Ako se
+ pravila/alarmi konfiguriraju korisnički, što sve konfiguracija smije izraziti?
+- [ ] Ograničenja stranica/`limit`, broj redaka, vremenski rasponi na `history`/`stats`
+ (fiksna lista raspona i metrika, ograničen cache, ključevi iz validiranih vrijednosti).
+- [ ] Transakcije: nema stanja u kojem pad usred operacije ostavlja račun bez 2FA ili sesiju
+ koja preživljava promjenu lozinke.
+
+## 6. Izlazne veze (SSRF) i integracije
+
+- [ ] Webhook/alarm URL-ovi (`ADMIN_ALERT_WEBHOOK_URL`, kanali u `alerts.py`/`rules.py`):
+ tko ih može postaviti (samo env ili i API)? Ako ih postavlja korisnik/admin preko API-ja:
+ shema samo http(s), blokiranje loopback/link-local/privatnih adresa i metadata IP-a,
+ provjera nakon DNS razrješenja, bez praćenja preusmjeravanja na unutarnje adrese, timeout.
+- [ ] Sadržaj odlazne poruke ne sadrži tajne (tokene, hashove) ni neocišćene tuđe stringove u
+ kontekstu gdje se interpretiraju (Slack/markdown/HTML/email zaglavlja → injekcija).
+- [ ] Nema `requests`/`httpx` poziva prema URL-u iz zahtjeva bez gornjih zaštita.
+
+## 7. Konfiguracija, tajne, logiranje, greške
+
+- [ ] `config.py`/okolina: nesigurne zadane vrijednosti (prazan `SECRETS_KEY`,
+ `ALLOW_REGISTRATION=true`, `COLLECTOR_ACCESS` ≠ strict, `DEBUG`, `CORS_ORIGINS=*`).
+ Zadano mora biti sigurno, a nesigurno mora zahtijevati namjernu promjenu.
+- [ ] Tajne se ne ispisuju u logove, iznimke ni odgovore (setup token se ispisuje **namjerno**
+ pri startu dok nema računa; provjeri da se ne ispisuje poslije). Nema stack traceova ni
+ SQL-a u HTTP odgovorima; nema `/docs`/`/openapi.json` izloženih na javnoj strani ako to
+ nije namjera (provjeri `FastAPI(docs_url=...)` i nginx).
+- [ ] **Log injection:** korisnički i kolektorski stringovi u logovima s novim redom/ANSI
+ sekvencama.
+- [ ] `seed_user.py` i `reset_2fa.py`: bez ugrađenih vjerodajnica, bez ispisa lozinke, ne
+ mogu se pokrenuti preko API-ja, ne ostavljaju tajne u povijesti ljuske (argumenti).
+- [ ] Pokretanje: backend se ne pokreće s `--reload`/debug u produkcijskoj slici; radi kao
+ neprivilegirani korisnik (vidi infra listu).
+
+## 8. Dostupnost (DoS jednim korisnikom)
+
+- [ ] Skupi endpointi (statistika, flota, povijest, eksport) imaju ograničenje rada i cache;
+ nema neograničenih petlji nad podacima koje kontrolira kolektor.
+- [ ] Dugotrajne operacije ne blokiraju event loop (sinkroni DB/CPU rad u `async def`
+ ruti blokira sve zahtjeve: jedan spor zahtjev = zastoj).
+- [ ] Rast podataka: tablice koje se pune iz check-inova imaju gornju granicu ili retenciju;
+ zabilježi ako je rast neograničen (poznato za sažetke povijesti).
diff --git a/.claude/skills/security-audit/references/checklist-collector-agent.md b/.claude/skills/security-audit/references/checklist-collector-agent.md
new file mode 100644
index 0000000..010943c
--- /dev/null
+++ b/.claude/skills/security-audit/references/checklist-collector-agent.md
@@ -0,0 +1,96 @@
+# Kontrolna lista: collector, docker proxy, system helper, LLM asistent (`src/agent/**`)
+
+Čitaj `collector.py`, `docker_proxy.py`, `system_helper.py`, `health_checks.py`, `service.py`,
+`prompts.py`, `main.py`, `utils.py` i `SYSTEM_PROMPT.md` u cijelosti. Docker proxy i system
+helper drže privilegije: najveća pažnja ide njima.
+
+## 1. Docker proxy (`docker_proxy.py`): drži root-ekvivalent
+
+- [ ] **Bijela lista je doslovna usporedba cijelog request targeta.** Nema dekodiranja,
+ normalizacije, `startswith`, regexa bez sidra (`re.match` bez `$`/`fullmatch`), ni
+ `in`. Pokušaji zaobilaženja: `?all=1&x=y`, `//containers/json`, `/containers/json/`,
+ `%2e`, `;`, `#`, `\`, razmak, tab, `\r\n`, velika/mala slova metode, `HEAD`/`OPTIONS`,
+ `GET` s tijelom, `Transfer-Encoding`/`Content-Length` proturječja, HTTP/1.0,
+ više zahtjeva na jednoj vezi (pipelining/smuggling), ID kontejnera koji nije hex
+ (`../`, `x/exec`).
+- [ ] ID u `/containers/<id>/json` i `/stats` strogo validiran (hex, duljina) *prije*
+ prosljeđivanja u Docker. Proxy ne gradi zahtjev prema Dockeru spajanjem neprovjerenog
+ stringa.
+- [ ] Proxy **ne prosljeđuje** zaglavlja ni metodu klijenta; zahtjev prema Dockeru gradi sam.
+- [ ] **Filtriranje odgovora:** vraća samo fiksnu listu polja; `Env`, `Labels`, `Mounts`,
+ `Config`, `HostConfig`, `Error` ne izlaze. Provjeri i za `/containers/json` i za
+ stats (stats ne sadrži tajne, ali provjeri). Nova polja = namjerna promjena + test.
+- [ ] Veličina odgovora/zahtjeva ograničena; timeouti; jedna sporo-čitajuća veza ne blokira
+ sve (DoS prema kolektoru); greške Dockera ne vraćaju sirovi tekst.
+- [ ] Unix socket: mode 0660, direktorij/volumen dijeljen **samo** s kolektorom; socket se
+ ne stvara u svijetu-zapisivom direktoriju; nema utrke pri `bind`/`chmod` (kratak
+ prozor s širim pravima; zadani `umask`).
+- [ ] Nema TCP listenera, nema mreže (`network_mode: none`, vidi infra listu).
+- [ ] Log 403 ne dopušta log injection (sanitiziran request target).
+
+## 2. System helper (`system_helper.py`): privilegiran, izlaz nepouzdan
+
+- [ ] Samo čita (dmesg *read-all*, wtmp), nikad ne piše/briše kernel log.
+- [ ] Parsiranje wtmp-a je binarno: provjeri veličinu zapisa, granice, nepotpun zapis,
+ ogroman broj zapisa; čita samo boot/shutdown zapise, ne korisnička imena ni adrese.
+- [ ] Linije kernel loga: fiksni obrasci, reducirano na printable, skraćeno, ograničen broj
+ događaja; nema `eval`/`shell`/format stringa s tim sadržajem; regexi bez katastrofalnog
+ backtrackinga na 100 kB retku.
+- [ ] Izlazna datoteka: atomarno pisanje (temp + rename), prava, samo u svom volumenu.
+- [ ] Ništa se ne izvršava iz sadržaja; nema `subprocess` s ulazom pod kontrolom napadača
+ (i ako postoji poziv, argumenti su lista, bez `shell=True`).
+
+## 3. Collector (`collector.py`, 974 retka): radi na hostu, čita host FS
+
+- [ ] **Čitanje datoteka s hosta:** fiksni popis putanja; `O_NOFOLLOW`; ograničena veličina
+ čitanja (1 MB firewall datoteke, helperov izlaz); provjera da je helper izlaz svjež i
+ iz tekućeg boota; nema putanja sastavljenih od podataka s hosta ili dashboarda
+ (`os.path.join` s apsolutnom putanjom tiho zamjenjuje bazu!). TOCTOU između `stat` i
+ `open`.
+- [ ] `/proc/1/net` i ostali parseri: nepotpuno čitanje → "unknown", nikad djelomičan
+ rezultat koji bi ports "zatvorio"; ogromne datoteke ne troše memoriju.
+- [ ] **Ono što stiže s dashboarda** (odgovor na register/poll) tretira se nepouzdano: nema
+ izvršavanja, nema putanja, nema ciljeva za sondiranje iz odgovora. Mete provjere
+ interneta samo iz okoline kolektora (`HOMELAB_CONNECTIVITY_TARGETS`, ≤5, hostname ili
+ IPv4), s tvrdim rokom.
+- [ ] Sondiranje: ne šalje podatke, samo TCP connect + DNS; ne može se usmjeriti na
+ proizvoljnu metu; ograničenje trajanja kako check-in ne bi visio.
+- [ ] **Pohrana vjerodajnica kolektora** (token servera): datoteka s modom 0600, ne u
+ slici, ne u logu, ne u argumentima procesa; kolektor ne ispisuje token pri pogrešci.
+- [ ] Transport: provjera TLS-a ako je HTTPS (ne `verify=False`); kad je običan HTTP unutar
+ VPN-a, to je dokumentirano. Nema preusmjeravanja koja bi odnijela `Authorization`
+ zaglavlje na drugi host. Timeouti na svim mrežnim pozivima.
+- [ ] Greška jednog uzorka ne ruši cijeli kolektor na način koji ga ušutkava bez traga
+ (napadač koji izaziva iznimku "gasi" nadzor?).
+- [ ] `subprocess`, `os.system`, `eval`, `exec`, `pickle`, `yaml.load`: svako pojavljivanje
+ pogledaj; argumenti kao lista; nema podataka s hosta u naredbi.
+- [ ] Ograničenja količine prijavljenih podataka (kontejneri, portovi, događaji) na strani
+ kolektora, ne samo backenda.
+
+## 4. LLM asistent (`service.py`, `prompts.py`, `SYSTEM_PROMPT.md`, `health_checks.py`)
+
+Dokumentirano: nema putanje za shell, samo tri fiksne provjere (`server_health`,
+`container_health`, `daily_checkup`), webhook samo na `ADMIN_ALERT_WEBHOOK_URL`.
+
+- [ ] **Alati su doista fiksni:** parametri alata validirani bijelom listom; nema alata s
+ proizvoljnim stringom koji završava u naredbi, putanji, URL-u ili SQL-u. Nema
+ dinamičkog dodavanja alata iz podataka.
+- [ ] **Prompt injection:** podaci koji ulaze u kontekst modela (imena kontejnera, linije
+ loga, poruke grešaka, rezultati provjera, tekst korisnika) mogu sadržavati upute.
+ Što najgore može model napraviti ako ih posluša? (Pozvati dopuštene provjere, poslati
+ webhook, otkriti sistemski prompt/konverzaciju.) Webhook i alarmi ne smiju nositi
+ tajne, a model ne smije moći promijeniti odredište.
+- [ ] **Izlaz modela se ne tretira kao pouzdan:** prikazuje se kao tekst (frontend ne radi
+ `dangerouslySetInnerHTML`/neprovjeren markdown s HTML-om), ne ulazi u SQL/naredbe/URL-ove.
+- [ ] **Konverzacije su vezane uz korisnika** (IDOR: čitanje/brisanje tuđih konverzacija
+ po ID-u); limit duljine poruke i broja poruka; limit koraka petlje alata i vremena
+ (DoS i trošak API-ja); rate limit po korisniku.
+- [ ] API ključ prema LLM pružatelju: iz okoline, ne u logovima, ne u odgovorima, ne u
+ konverzacijama koje se pohranjuju ili šalju natrag.
+- [ ] Sistemski prompt ne sadrži tajne (pretpostavi da će ga model prije ili poslije
+ ispisati).
+- [ ] Pozivi prema Docker API-ju idu samo kroz proxy (`DOCKER_SOCKET_PATH`); backend ne montira
+ `docker.sock`. Provjeri compose i kod.
+- [ ] Webhook: samo `ADMIN_ALERT_WEBHOOK_URL` iz okoline; bez toga nikakav zahtjev; payload
+ bez neocijenjenog sadržaja koji bi napravio injekciju u primatelju (Slack mrkdwn,
+ mention-i `@channel` iz imena kontejnera).
diff --git a/.claude/skills/security-audit/references/checklist-frontend.md b/.claude/skills/security-audit/references/checklist-frontend.md
new file mode 100644
index 0000000..aede1d8
--- /dev/null
+++ b/.claude/skills/security-audit/references/checklist-frontend.md
@@ -0,0 +1,82 @@
+# Kontrolna lista: frontend (`src/frontend/**`)
+
+Čitaj `src/App.jsx` (velik), `StatisticsPage.jsx`, `FleetStatistics.jsx`,
+`ContainerStatistics.jsx`, `ServerInsights.jsx`, `charts.jsx`, `distributions.jsx`,
+`*Logic.js`, `chartMath.js`, `main.jsx`, `index.html`, `vite.config.js`, `package.json`.
+
+## 1. XSS
+
+Dashboard prikazuje podatke koje kontrolira napadač (imena kontejnera, putanje mountova,
+poruke događaja, tekst asistenta, imena servera). **Stored XSS u admin kontekstu = preuzimanje
+dashboarda**, pa je ovo glavna stavka.
+
+- [ ] Pretraži: `dangerouslySetInnerHTML`, `innerHTML`, `outerHTML`, `insertAdjacentHTML`,
+ `document.write`, `eval`, `new Function`, `setTimeout("string")`, `srcdoc`, `postMessage`
+ bez provjere origina. Svako pojavljivanje prati do izvora podatka.
+- [ ] **Markdown/rich tekst** (izlaz asistenta, opisi): koristi li se biblioteka? Je li HTML
+ isključen ili sanitiziran (DOMPurify s restriktivnom konfiguracijom)? Linkovi iz
+ markdowna filtriraju `javascript:`/`data:`.
+- [ ] **Atributi s URL-om iz podataka:** `href`, `src`, `action`, `formAction`: shema samo
+ `http(s)` (i `mailto` ako treba). `target="_blank"` ima `rel="noopener noreferrer"`.
+- [ ] SVG iz podataka (grafovi koji ugrađuju stringove u SVG, `<text>`, `<title>`, `<a>`):
+ React escapea tekst, ali provjeri ručno sastavljene SVG stringove i `style` objekte
+ s podacima.
+- [ ] CSS injekcija: vrijednosti iz podataka u `style`/`className` konstruiranima iz stringova.
+- [ ] Prototype pollution: spajanje objekata iz JSON-a s kolektora (`Object.assign`, spread
+ u ključeve iz podataka, `obj[key] = ...` s ključem iz podataka, `__proto__`).
+
+## 2. Autentikacija i tajne u pregledniku
+
+- [ ] Tokeni/sesije se **ne** drže u `localStorage`/`sessionStorage`/URL-u (sesija je
+ HttpOnly kolačić). Provjeri što se sprema u `localStorage` (osjetljivi podaci, tokeni,
+ TOTP, recovery kodovi).
+- [ ] Recovery kodovi i TOTP tajna: prikazuju se samo jednom, ne spremaju se u stanje koje
+ preživljava navigaciju, ne završavaju u URL-u ni logu konzole (`console.log`).
+- [ ] Nema API ključeva, `SECRETS_KEY`-a ni internih URL-ova ugrađenih u bundle
+ (`import.meta.env.VITE_*` je **javno**). Pretraži `VITE_`.
+- [ ] Frontend ne donosi sigurnosne odluke koje backend ne provjerava (skrivanje admin
+ gumba nije kontrola pristupa; to pokriva backend lista, ovdje samo zabilježi).
+- [ ] Odjava briše stanje u memoriji; nema keširanja osjetljivih odgovora (nginx/`Cache-Control`
+ za API).
+
+## 3. Generiranje naredbi za instalaciju ("Connect a server")
+
+Panel generira naredbu koju korisnik kopira i izvršava na udaljenom stroju. Ona sadrži
+join token i digest slike.
+
+- [ ] Sve vrijednosti u naredbi (token, digest, URL, `COLLECTOR_BIND`, ime) su **validirane ili
+ citirane za ljusku**. Vrijednost s `;`, `$(...)`, `` ` ``, `'`, novim redom ili razmakom
+ ne smije izaći iz navodnika. Provjeri odakle vrijednosti dolaze (konfiguracija
+ dashboarda, odgovor API-ja) i kako se spajaju u string.
+- [ ] Naredba ne koristi `curl | sh` ni nepinirane slike; digest je obavezan, a prikaz
+ jasno razlikuje što korisnik izvršava.
+- [ ] Token se prikazuje jednom, ima vidljiv istek, a "Kopiraj" ne ostavlja ga u povijesti
+ stanja.
+
+## 4. Mrežne pozivne točke i CSRF
+
+- [ ] `fetch` pozivi za mijenjanje stanja šalju kolačić (`credentials`) samo prema vlastitom
+ originu; nema poziva prema proizvoljnim URL-ovima iz podataka.
+- [ ] Nema otvorenog preusmjeravanja (`window.location = param`), nema `redirect`/`next`
+ parametra bez provjere da je relativan put.
+- [ ] Rukovanje greškama ne ispisuje stack/tajne korisniku; odgovori API-ja koji sadrže
+ tuđe stringove prikazuju se kao tekst.
+
+## 5. CSP i build
+
+- [ ] Frontend radi pod strogim CSP-om bez `unsafe-inline` za skripte (nema inline `<script>`
+ u `index.html`, nema `eval` u bundle-u). Novi izvori koji traže proširenje CSP-a
+ moraju biti dokumentirani.
+- [ ] `vite.config.js`: dev proxy/`server.host` ne izlaže dev server; `build.sourcemap` u
+ produkciji isključen ili namjeran; nema `define` s tajnama.
+- [ ] `index.html`: bez vanjskih skripti/fontova bez SRI-ja; bez inline handlera.
+- [ ] Ovisnosti (`package.json`): neobične ili nepotrebne, `postinstall` skripte; vidi infra
+ listu za audit alate.
+
+## 6. Dostupnost kao sigurnost
+
+- [ ] Renderiranje ne pada ni ne zamrzava preglednik na hostilnim podacima: vrlo dugi
+ stringovi, tisuće kontejnera, `NaN`/`Infinity`/negativni brojevi u grafovima
+ (`chartMath.js`, `fleetLogic.js`, `containerLogic.js`), prazni nizovi, nepoznata polja.
+ Pad cijele aplikacije zbog jednog loše oblikovanog zapisa = napadač s kolektorom može
+ onesposobiti dashboard.
diff --git a/.claude/skills/security-audit/references/checklist-infra.md b/.claude/skills/security-audit/references/checklist-infra.md
new file mode 100644
index 0000000..f646351
--- /dev/null
+++ b/.claude/skills/security-audit/references/checklist-infra.md
@@ -0,0 +1,130 @@
+# Kontrolna lista: infrastruktura, repozitorij, ovisnosti
+
+Čitaj `docker/*` (Dockerfile-ove, `nginx*.conf`, `Caddyfile`, `postgres/init/*.sql`),
+`compose.yml`, `docker-compose.server.yml`, `collector-compose.yml`, `example.env`,
+`.env.example`, `.dockerignore`, `.gitignore`, `.github/workflows/*.yml`, `requirements*.txt`,
+`src/frontend/package.json` + lockfile.
+
+## 1. Dockerfile-ovi
+
+- [ ] `USER` je neprivilegiran u svakoj slici koja to može (docker-proxy i system-helper su
+ iznimke zbog socketa/capabilityja: provjeri da je to jedino mjesto i da je obrazloženo).
+- [ ] Bazne slike: pinirane (digest ili barem točna verzija), ne `latest`; minimalne.
+- [ ] Nema tajni u slojevima (`ARG`/`ENV` s tokenima, `COPY .env`, `.git`, `dev.db`, ključevi);
+ `.dockerignore` isključuje `.env*`, `.git`, `venv`, `*.db`, testove, `node_modules`.
+- [ ] Nema `curl | sh`, `pip install` bez pinova/hashova za produkcijske ovisnosti, `apt` bez
+ `--no-install-recommends` i čišćenja; build ne izvršava nepouzdani kod s mreže.
+- [ ] `HEALTHCHECK`/`CMD` ne pokreće debug način (`--reload`, `--debug`).
+- [ ] Frontend slika: izgrađena u multi-stage, u produkciji nema `node_modules`, dev servera
+ ni source mapa koje otkrivaju izvorni kod ako to nije namjera.
+
+## 2. Compose datoteke
+
+Za **svaki servis** zapiši: korisnik, `cap_drop`/`cap_add`, `read_only`,
+`security_opt: no-new-privileges`, `privileged`, `network_mode`, `pid`, `ports`, `volumes`,
+`environment`/`secrets`.
+
+- [ ] **Objavljeni portovi** vezani su na konkretnu adresu (`127.0.0.1:` ili VPN adresa), ne na
+ `0.0.0.0`, osim namjerno javnog frontenda. Docker zaobilazi `ufw`: firewall pravilo nije
+ zaštita. Backend, Postgres i docker-proxy nemaju objavljenih portova u server stogu.
+- [ ] **`/var/run/docker.sock` montiran samo u docker-proxy**, nigdje drugdje (backend,
+ collector, frontend, helper). Isto za `/`, `/etc`, `/root`, `/home`, `/var/lib/docker`.
+- [ ] Mountovi hosta su `:ro` gdje je moguće; `/hostfs` je read-only; nema `rw` mounta
+ povjerljivih direktorija.
+- [ ] Volumen s Unix socketom docker-proxyja dijele **samo** proxy i collector.
+- [ ] `cap_drop: [ALL]` kao zadano; svaki `cap_add` obrazložen (samo `CAP_SYSLOG` za helper).
+ Nema `privileged`, `pid: host`, `network_mode: host`, `ipc: host`, `devices` bez razloga.
+- [ ] `read_only: true` + `tmpfs` za potrebno; `no-new-privileges` na svemu.
+- [ ] Mrežna segmentacija: collector-gate nije na mreži reverse proxyja; backend nije na
+ mreži kojoj imaju pristup tuđe aplikacije; Postgres samo na internoj mreži;
+ `internal: true` gdje izlaz nije potreban.
+- [ ] **Zadane/placeholder tajne:** u compose/`example.env` nema stvarnih ili zadano slabih
+ lozinki (`changeme`, `postgres`, `admin`) koje bi ostale u produkciji; obavezne
+ vrijednosti koriste `${VAR:?poruka}` da stog ne krene bez njih.
+- [ ] Slike u compose-u za udaljene hostove pinirane po digestu (`COLLECTOR_IMAGE`,
+ `DOCKER_PROXY_IMAGE`), a panel ne generira naredbu bez njih.
+- [ ] Postgres: lozinke ne u compose-u u čistom tekstu ako se može `secrets:`/env iz `.env`;
+ nema objavljenog 5432.
+- [ ] Healthchecki ne otkrivaju tajne; restart politike ne skrivaju petlju rušenja.
+
+## 3. nginx (`nginx.conf`, `nginx.strict.conf`, `nginx.collector.conf`)
+
+- [ ] **Javna strana vraća 404 za sve collector endpointe** (register, poll, enroll, ingest,
+ helper). Provjeri točne `location` blokove: prefiks vs. točno, regex, razlike u
+ kosoj crti, velika/mala slova, `//`, `/./`, `%2f`, `..;/`. Najgori slučaj: put koji
+ nginx ne prepoznaje kao zabranjen, a backend ga normalizira u collector endpoint.
+- [ ] `X-Collector-Via` (i slična interna zaglavlja) **postavlja** samo gate, a javna strana
+ ga **briše/prepisuje** (`proxy_set_header X-Collector-Via "";`).
+- [ ] `real_ip` / `X-Forwarded-For`: `set_real_ip_from` samo privatni/Docker rasponi,
+ `real_ip_recursive` ispravno; nema povjerenja u XFF s javne strane; backend dobiva
+ stvarni klijentski IP za rate limit.
+- [ ] **`add_header` nasljeđivanje:** `add_header` u `location` bloku *poništava* sve
+ `add_header` iz `server`/`http` razine. Provjeri da svaki `location` koji ima vlastiti
+ `add_header` (cache, CORS) ponavlja sigurnosna zaglavlja (CSP, nosniff, frame, HSTS),
+ i da su dodana s `always` kako bi vrijedila i za greške.
+- [ ] CSP: bez `unsafe-inline`/`unsafe-eval` za `script-src`, `default-src 'self'`,
+ `frame-ancestors 'none'`, `object-src 'none'`, `base-uri 'self'`, `form-action 'self'`;
+ dopušteni `style` inline je poznata cijena (React style atributi).
+- [ ] HSTS samo na strict/javnoj konfiguraciji; `server_tokens off`.
+- [ ] `client_max_body_size` ograničen po lokaciji; `client_body_timeout`, `proxy_*_timeout`,
+ `limit_req`/`limit_conn` na osjetljivim rutama (login, enroll) kao drugi sloj.
+- [ ] `proxy_pass`: pazi na kosu crtu na kraju (promjena normalizacije puta), prosljeđuju se
+ samo potrebna zaglavlja, `proxy_set_header Host` ispravan (backend koristi `Host` za
+ provjeru Origina), WebSocket upgrade samo gdje treba.
+- [ ] Nema `autoindex on`, `alias` bez kose crte (traversal), poslužuje li se išta osim
+ statičkih datoteka SPA-a (`.map`, `.env`, `.git`)? `try_files` ne otkriva skrivene datoteke.
+- [ ] TLS konfiguracija (ako je nginx terminira): protokoli ≥ TLS1.2, bez slabih šifri.
+ Ako TLS terminira Traefik, navedi to kao pretpostavku, ne nalaz.
+- [ ] `Caddyfile`: je li to ostatak/dev konfiguracija? Ako je u produkcijskom putu, pregledaj
+ jednako strogo kao nginx.
+
+## 4. Postgres (`docker/postgres/init/*.sql`)
+
+- [ ] Aplikacija se spaja kao korisnik s **minimalnim pravima** (nije superuser, nije vlasnik
+ sheme ako migracije nisu potrebne pri radu), odvojen od administrativnog.
+- [ ] Lozinke u init skriptama nisu ugrađene u repozitorij; nema `trust` autentikacije iz
+ svih adresa; `pg_hba` ne dopušta vanjske veze.
+- [ ] Nema `CREATE EXTENSION` ni `GRANT ALL` bez razloga.
+
+## 5. GitHub Actions (`.github/workflows/images.yml`)
+
+Ovo je opskrbni lanac slika koje hostovi pokreću s root-ekvivalentnim pristupom.
+
+- [ ] `permissions:` najmanje potrebno (`contents: read`, `packages: write`, `id-token: write`
+ samo za job koji potpisuje); nema zadanih širokih dozvola.
+- [ ] **Akcije pinirane na puni commit SHA**, ne na tag (`@v4`) za akcije koje diraju tajne ili
+ potpisivanje; provjeri tko ih održava.
+- [ ] Nema `pull_request_target` s checkoutom koda iz PR-a; okidač je samo tag `v*` i/ili
+ ručni, ne svaki push/PR iz forka.
+- [ ] **Script injection:** nema `${{ github.event.* }}` / `${{ github.head_ref }}` /
+ imena grane ili poruke commita umetnutih izravno u `run:` (prenesi preko `env:`).
+- [ ] Potpisivanje cosignom: *keyless* identitet vezan uz workflow i ref; ispis digesta
+ točan; slike se grade iz tagiranog commita; build ne povlači nepinirane ovisnosti bez
+ provjere. Opiši kako domaćin **provjerava** potpis (dokumentirano ako `cosign verify`
+ nije dio generirane naredbe, to je slabost: potpis koji nitko ne provjerava ne štiti).
+- [ ] Tajne se ne ispisuju (`set -x`, `echo`); nema `secrets: inherit` bez razloga.
+
+## 6. Repozitorij i povijest
+
+- [ ] `.gitignore` pokriva `.env`, `*.db`, `*.sqlite`, ključeve, `venv`, `node_modules`.
+ `git ls-files | grep -Ei '\.env$|\.db$|\.sqlite|\.pem$|\.key$|id_rsa|secret|credential'`:
+ što je **pod gitom**? (`dev.db` je ranije bio.) `.env` i `.env.example` obje postoje:
+ provjeri da `.env.example` nema stvarne vrijednosti.
+- [ ] **Povijest:** `git log --all --diff-filter=A --name-only | grep -Ei '\.env|\.db|key|secret'`
+ i pretraga uzoraka tajni (`git log -p --all -S'SECRET' -S'PASSWORD' -S'TOKEN'` na
+ razumnom opsegu). Tajna koja je ikad bila u povijesti smatra se kompromitiranom čak i
+ ako je kasnije obrisana; to treba prijaviti uz preporuku rotacije. Ne ispisuj vrijednosti.
+- [ ] Pretraga uzoraka u radnom stablu: ključevi (`AKIA`, `ghp_`, `sk-`, `-----BEGIN`),
+ lozinke u kodu, hardkodirani tokeni, `localhost` zaobilaženja autentikacije,
+ `TODO`/`FIXME`/`HACK` uz riječi `auth`, `security`, `bypass`, `skip`.
+- [ ] `TESTS.md`, `docs/`, `README.md` ne sadrže stvarne tajne/domene/IP adrese vlasnika.
+- [ ] `SECURITY.md` ima put za prijavu ranjivosti (ima) i točne tvrdnje.
+
+## 7. Ovisnosti
+
+- [ ] `requirements*.txt` i `package-lock.json`: pinirano? Lockfile usklađen s `package.json`?
+- [ ] Ako su dostupni alati (`pip-audit`, `npm audit --omit=dev`), pokreni ih **samo uz
+ korisnikovu potvrdu** (šalju popis paketa prema registryju). Ako nisu pokrenuti,
+ napiši to u pokrivenosti: nemoj tvrditi da su ovisnosti "bez poznatih ranjivosti".
+- [ ] Nepotrebne/napuštene ovisnosti, ovisnosti iz URL-a/gita, `postinstall` skripte,
+ typosquat imena. Dev ovisnosti ne ulaze u produkcijsku sliku.
diff --git a/.claude/skills/security-audit/references/report-format.md b/.claude/skills/security-audit/references/report-format.md
new file mode 100644
index 0000000..87cf5b8
--- /dev/null
+++ b/.claude/skills/security-audit/references/report-format.md
@@ -0,0 +1,103 @@
+# Format izvještaja
+
+Piši na hrvatskom (korisnik tako komunicira), a tehničke pojmove, imena funkcija i kod ostavi
+kako jesu. Bez uvoda i bez hvaljenja. Najvažnije prvo.
+
+## Struktura
+
+### 1. Sažetak
+
+- Opseg koji je stvarno pregledan (jedna rečenica) i datum/commit (`git rev-parse --short HEAD`,
+ napomena ako ima nepohranjenih promjena).
+- Jedan redak s brojem nalaza po razini (npr. "2 visoka, 3 srednja, 4 niska"), uz koliko ih je `POTVRĐENO`, a koliko `TREBA PROVJERU`.
+- Jedna rečenica o **najvažnijem** nalazu ili "nisam našao iskoristive propuste u pregledanom
+ opsegu" ako ih nema.
+
+### 2. Tablica nalaza (obavezna)
+
+Svi nalazi **moraju** biti prikazani u jednoj tablici, **sortirani prema razini rizika, od
+najvišeg prema najnižem**: KRITIČNO, VISOKO, SREDNJE, NISKO, INFO. Unutar iste razine
+`POTVRĐENO` ide prije `TREBA PROVJERU`, a zatim po iskoristivosti (lakše iskoristivo prvo).
+Ne grupiraj po područjima (backend, infra, ...) i ne prikazuj nalaze u prozi umjesto tablice.
+
+| ID | Rizik | Nalaz | Lokacija | Napadač | Učinak | Status | Popravak (jedna rečenica) |
+|---|---|---|---|---|---|---|---|
+| S-01 | KRITIČNO | Kratak naslov | [main.py:123](src/backend/main.py#L123) | profil iz threat-modela | Pristup | POTVRĐENO | Što promijeniti |
+| S-02 | VISOKO | ... | ... | ... | DoS | TREBA PROVJERU | ... |
+
+Pravila za tablicu:
+
+- Stupac "Učinak" kaže što napadač **stvarno dobiva**, da se odmah vidi vodi li nalaz do upada.
+ Koristi samo ove vrijednosti, a za više njih odvoji ih zarezom (najteža prva):
+ `Izvršavanje koda` (kod ili naredbe na backendu, hostu ili kolektoru), `Pristup` (preuzimanje
+ računa, sesije, admin prava, zaobilaženje autentikacije ili 2FA), `Krađa podataka`
+ (čitanje podataka ili tajni koje napadač ne smije vidjeti), `Integritet` (podmetanje, mijenjanje
+ ili utišavanje podataka i alarma), `DoS` (rušenje ili usporavanje, zaključavanje računa),
+ `Izviđanje` (otkriva postojanje računa, strukturu ili konfiguraciju bez izravne štete).
+ Ocjenjuj učinak u najgorem realnom slučaju, a ne u idealnom za napadača; ako ovisi o uvjetu
+ koji nisi provjerio, napiši ga u detaljima.
+
+- Jedan redak po nalazu; ID-evi `S-01`, `S-02`, ... dodjeljuju se **nakon sortiranja**, pa
+ redni broj odgovara poretku rizika.
+- Stupac "Lokacija" sadrži klikabilne markdown linkove (najviše dva; ostale navedi u
+ detaljima).
+- Ćelije su kratke (jedan redak). Sve duže ide u detalje ispod.
+- Ako nema nalaza neke razine, ne dodaj prazne retke za nju.
+- Ista tablica (isti stupci, isti poredak) vrijedi i za rezultate paralelnih agenata nakon
+ spajanja: izvještaj je jedna tablica, ne četiri.
+
+### 3. Detalji nalaza
+
+Isti redoslijed kao u tablici. Svaki nalaz ima ovaj oblik (bez dodatnih naslova):
+
+```
+### [VISOKO] S-02 · Kratak, konkretan naslov (POTVRĐENO)
+
+**Gdje:** [datoteka.py:123](putanja/datoteka.py#L123), poziva ga [drugi.py:45](putanja/drugi.py#L45)
+**Napadač:** tko (npr. prijavljeni korisnik bez admin prava / VPN peer / kompromitirani kolektor)
+
+**Problem:** što je pogrešno, jednom-dvjema rečenicama.
+
+**Scenarij:** korak po korak što napadač pošalje i što dobije.
+
+**Zašto postojeće zaštite ne pomažu:** koje si kontrole provjerio i zašto ih ovo zaobilazi.
+
+**Popravak:** konkretna promjena (i test koji je treba pokriti), s kratkim isječkom koda ako
+pomaže.
+
+**Dokaz:** test/skripta koju si pokrenuo i rezultat, ili "nije pokrenuto: <razlog>".
+```
+
+ID je isti kao u tablici nalaza.
+Za `TREBA PROVJERU` umjesto "Dokaz" napiši **Što treba provjeriti** (točno što i kako).
+
+### 4. Tvrdnje iz dokumentacije koje kod ne podržava
+
+Tablica: tvrdnja (citat + mjesto u `SECURITY.md`/`docs/`) | što kod zapravo radi | datoteka:redak.
+Izostavi odlomak ako nema razlika.
+
+### 5. Očvršćivanje (nije ranjivost)
+
+Kratke stavke (jedan redak) za stvari koje nisu iskoristive same za sebe, ali bi smanjile
+rizik. Ne miješaj ih s nalazima.
+
+### 6. Pokrivenost
+
+- **Pregledano:** popis datoteka/područja koja si pročitao u cijelosti.
+- **Nije pregledano i zašto:** (npr. ovisnosti bez audit alata, živa konfiguracija
+ Traefika/OpenVPN-a izvan repozitorija, dinamičko testiranje).
+- **Pretpostavke** o okruženju na koje se nalazi oslanjaju.
+
+### 7. Sljedeći korak
+
+Jedna rečenica: predloži redoslijed popravljanja i pitaj želi li korisnik da počneš. Ništa
+ne mijenjaj dok ne odgovori.
+
+## Pravila stila
+
+- Nalazi su činjenice s dokazom; nagađanje označi kao `TREBA PROVJERU`.
+- Ne duplicirati: isti uzrok na više mjesta = jedan nalaz s popisom lokacija.
+- Ne pretjeruj s ozbiljnošću radi dojma, ali ni ne ublažavaj (korisnik izričito traži
+ strogu procjenu). Kad dvoji, napiši obje razine i obrazloži jednom rečenicom.
+- Ne ispisuj vrijednosti tajni; navedi samo lokaciju i vrstu.
+- Bez emojija.
diff --git a/.claude/skills/security-audit/references/threat-model.md b/.claude/skills/security-audit/references/threat-model.md
new file mode 100644
index 0000000..e20e4e6
--- /dev/null
+++ b/.claude/skills/security-audit/references/threat-model.md
@@ -0,0 +1,89 @@
+# Model prijetnji (Homelab Agent)
+
+Sažetak arhitekture i povjerenja, izveden iz `SECURITY.md` i koda. Ako se kod i ovaj dokument
+razilaze, **vjeruj kodu** i to prijavi kao nalaz o dokumentaciji.
+
+## Komponente i što dodiruju
+
+| Komponenta | Gdje | Doseže | Povjerenje |
+|---|---|---|---|
+| Frontend (nginx + React SPA) | `docker/nginx*.conf`, `src/frontend` | javni listener; proxy prema backendu | javno dostupan |
+| Backend (FastAPI) | `src/backend` | Postgres, interni mrežni segment | obrađuje sve ulaze |
+| Postgres | `docker/postgres/init` | samo backend | sadrži hashove, šifrirane TOTP tajne, podatke o hostovima |
+| Collector gate | `docker/nginx.collector.conf` | backend; jedan port na VPN adresi | jedini put do collector endpointa |
+| Collector | `src/agent/collector.py` | `/proc` hosta, host FS read-only (`/hostfs`), Docker proxy, backend preko HTTP-a | pokreće se na udaljenim hostovima, može biti kompromitiran |
+| Docker proxy | `src/agent/docker_proxy.py` | Docker socket (uid 0), Unix socket prema collectoru | drži root-ekvivalentan pristup; mora ostati minimalan |
+| System helper (opcionalno) | `src/agent/system_helper.py` | kernel log, wtmp, `CAP_SYSLOG` | privilegiran; njegov izlaz je nepouzdan |
+| LLM asistent | `src/agent/service.py`, `prompts.py`, `SYSTEM_PROMPT.md`, `health_checks.py` | fiksni read-only provjere, opcionalni webhook | ulaz uključuje podatke s hosta |
+| CI | `.github/workflows/images.yml` | GHCR, cosign | opskrbni lanac za slike koje hostovi izvršavaju |
+
+Izvan repozitorija (nisu u opsegu, ali ih treba imati na umu kao pretpostavke): Traefik pred
+frontendom, OpenVPN koji nosi promet kolektora, GHCR paketi.
+
+## Profili napadača
+
+Za svaki nalaz odredi koji ga profil može iskoristiti.
+
+1. **Anonimni korisnik interneta** na javnom listeneru. Doseže samo dashboard i njegov API.
+ Collector endpointi (register, poll, enroll, ingest) na toj strani **ne smiju postojati**.
+2. **Prijavljeni korisnik bez admin prava.** Po dizajnu vidi sve monitoring podatke svih
+ servera. Pitanje je može li *mijenjati* išta, vidjeti tuđe račune/sesije/konverzacije,
+ ili postati admin.
+3. **Susjedna web aplikacija na istoj domeni/podlozi** (sibling subdomain iza istog reverse
+ proxyja). Ista "site", pa `SameSite=Lax` ne štiti od zahtjeva koji mijenjaju stanje.
+ Zaštita je provjera `Origin` / `Sec-Fetch-Site`.
+4. **Peer na VPN-u.** Doseže collector gate. Bez valjanog join tokena ne smije ni zatražiti
+ pridruživanje; s njim čeka admin odobrenje; ingest traži token servera.
+5. **Kompromitirani kolektor.** Kontrolira sve što prijavljuje (imena kontejnera, mountovi,
+ portovi, događaji). Backend mu ne smije vjerovati: tipovi, duljine, rasponi, brojevi,
+ limiti po check-inu i po satu. Ne smije moći utjecati na druge servere ni na dashboard.
+6. **Kompromitirani ili zloćudni kontejner na hostu / susjed na Docker mreži.** Može
+ pokušati doći do backenda mimo nginxa (lažiranje `X-Collector-Via`, `X-Forwarded-For`),
+ do Unix socketa docker proxyja ili do volumena koji ne smije dijeliti.
+7. **Kompromitirani dashboard.** Ne smije moći natjerati hostove da izvrše kod (dashboard
+ ne poslužuje kod), ni usmjeriti sondiranja na proizvoljne mete (mete dolaze iz okoline
+ kolektora).
+8. **Opskrbni lanac.** Ovisnosti (pip, npm), bazne slike, GitHub Actions, workflow
+ dozvole, nepodpisane ili nepinirane slike.
+9. **Hostilni podaci s nadziranog hosta.** Imena kontejnera, putanje mountova, linije
+ kernel loga, wtmp, `/etc/os-release`. Napadač s ograničenim pristupom hostu može
+ utjecati na te stringove. Putovi: stored XSS, injekcija u ključeve/upite, prekoračenje
+ limita, razbijanje parsera, **prompt injection** prema asistentu.
+10. **Fizički/lokalni pristup hostu** (npr. `seed_user.py`, `reset_2fa.py`) je izvan modela
+ prijetnji po dizajnu: ta ovlast već daje pristup bazi. Ali skripte ne smiju imati
+ ugrađene vjerodajnice ni curiti tajne.
+
+## Najvrjednija imovina (redom)
+
+1. Pristup Docker API-ju / rootu na hostovima (preko proxyja, helpera, mountova).
+2. Admin sesija dashboarda i mogućnost odobravanja novih servera.
+3. Tokeni servera i join tokeni (hashirani u bazi).
+4. `SECRETS_KEY` (Fernet) i TOTP tajne, recovery kodovi.
+5. Monitoring podaci (otvoreni portovi, firewall hash, popis kontejnera) jer mapiraju mrežu.
+6. Integritet alarma (napadač koji može utišati ili lažirati incidente skriva napad).
+
+## Pretpostavke dizajna koje treba provjeriti, ne uzeti zdravo za gotovo
+
+- Backend vjeruje `X-Collector-Via` samo ako ništa nepouzdano ne doseže backend izravno.
+- nginx prihvaća `X-Forwarded-For` samo od privatnih/Docker adresa; rate limiti ovise o tome.
+- Collector endpointi odbijaju se i na javnoj strani (nginx 404) i u backendu
+ (`COLLECTOR_ACCESS=strict`). Provjeri **oba** sloja i da ni jedan ne ovisi o drugom.
+- "Jednokratno" (join tokeni, TOTP kodovi, recovery kodovi, setup token) mora biti atomarno
+ u bazi, ne "provjeri pa označi" (utrka).
+- "Ograničeno" (500 kontejnera, 20 otvorenih tokena, 5 pokušaja/5 min) mora vrijediti i pod
+ paralelnim zahtjevima.
+- Prvi račun postaje admin: provjeri sve puteve do tog stanja (registracija, setup token,
+ `seed_user.py`, utrka dvaju istodobnih prvih zahtjeva).
+
+## Poznata prihvaćena ograničenja (ne prijavljuj kao nova, ali provjeri da još vrijede)
+
+Iz `SECURITY.md`: svi prijavljeni računi vide sve monitoring podatke; zaključavanje 2FA koraka
+pri pogađanju kodova; sesije prije uvođenja 2FA nisu u popisu; samo TOTP; promet VPN-om je
+običan HTTP unutar tunela; sažeci povijesti se još ne brišu. Ako kod odstupa od onoga što je
+tamo opisano, to je nalaz.
+
+## Otvorene stavke iz projektne bilješke (provjeri jesu li još otvorene)
+
+Ograničenje broja čekajućih servera po IP-u; rate limit na `enroll`; `src/backend/seed_user.py`
+(ugrađena admin lozinka); `dev.db` pod gitom; zastarjeli README. Ako su još stvarne, prijavi ih
+s trenutnim stanjem u kodu.
diff --git a/README.md b/README.md
index acded9a..1274327 100644
--- a/README.md
+++ b/README.md
@@ -79,7 +79,7 @@ flowchart LR
backend --> pg[(PostgreSQL)]
backend --> redis[(Redis)]
localc["collector"] --> gate
- localc -->|"unix socket"| dproxy["docker-proxy<br/>3 read-only requests"]
+ localc -->|"unix socket"| dproxy["docker-proxy<br/>4 read-only requests"]
helper["system-helper (optional)<br/>CAP_SYSLOG, no network"] -.->|"summary file"| localc
end
browser["Browser"] -->|"https://your-domain"| traefik["Traefik"]
@@ -139,830 +139,61 @@ docker compose down # stop (data volumes are kep
To try **Connect a server** on the same machine, set `COLLECTOR_IMAGE` and
`DOCKER_PROXY_IMAGE` in `.env` to `homelab-collector:latest` and
`homelab-docker-proxy:latest` (the images Compose just built). Other machines
-need published images; see [Publishing the collector images](#publishing-the-collector-images).
+need published images; see [Publishing the collector images](docs/deployment.md#publishing-the-collector-images).
-## Deploying on a server (behind Traefik)
+## Deploying on a server
-There are two Compose files:
+The server stack (`docker-compose.server.yml`) publishes the dashboard through an existing **Traefik** and
+serves the collector endpoints only on a separate listener bound to your **VPN address**. In short:
-| File | Use | Differences |
-|---|---|---|
-| `compose.yml` | Laptop / development (the default when you run `docker compose` without `-f`) | Dashboard on port 5173 on all interfaces; plain HTTP; optional Caddy (`--profile https`). |
-| `docker-compose.server.yml` | Server behind an existing Traefik | Traefik labels and network; HTTPS-only session cookie; only the first account can register; port 5173 bound to `127.0.0.1`; no Caddy; **collector endpoints removed from the public side** and served by a separate collector gate on your VPN address. |
+1. `cp example.env .env`, then set `POSTGRES_PASSWORD`, `DASHBOARD_DOMAIN`, your Traefik names and
+ `COMPOSE_FILE=docker-compose.server.yml` (the last one matters: without it Compose uses the development stack).
+2. Generate `SECRETS_KEY` (needed for two-factor sign-in) and set `COLLECTOR_BIND` / `COLLECTOR_PUBLIC_URL` for your VPN.
+3. `docker compose -f docker-compose.server.yml up -d --build`, then register the first account with the setup token from the backend log.
+4. Click **Connect a server** to add other machines.
-### 1. Create `.env` on the server
+Every step, with the reasons and the pitfalls, is in [Deploying on a server](docs/deployment.md) and
+[Adding servers](docs/adding-servers.md). To update later: `git pull` and `docker compose -f docker-compose.server.yml up -d --build`.
-Nothing in `docker-compose.server.yml` needs editing: everything specific to you comes from
-`.env`, so `git pull` can always update the file. Start from the production template:
+## Documentation
-```bash
-cp example.env .env
-```
-
-Then set at least these:
-
-| Setting | Default | Meaning |
-|---|---|---|
-| `POSTGRES_PASSWORD` | *(required)* | A long random password for the bundled database. |
-| `DASHBOARD_DOMAIN` | *(required)* | Your domain, without `https://`. Used for the Traefik router. |
-| `TRAEFIK_ENTRYPOINT` | `websecure` | Your Traefik HTTPS entrypoint name. |
-| `TRAEFIK_CERT_RESOLVER` | `letsencrypt` | Your Traefik certificate resolver name. |
-| `TRAEFIK_NETWORK` | `traefik_default` | The Docker network Traefik is attached to. |
-| `TRAEFIK_ROUTER` | `homelab` | Name of this dashboard's Traefik router and service; change it only if it clashes with another one. |
-| `COMPOSE_FILE` | `docker-compose.server.yml` | Makes plain `docker compose ...` commands use the server file (see below). |
-
-Find the network with `docker network ls` or
-`docker inspect <traefik-container> --format '{{json .NetworkSettings.Networks}}'`.
-The entrypoint and resolver names are in Traefik's static configuration
-(`entryPoints:` / `certificatesResolvers:`).
-
-### 2. Use the right Compose file
-
-`COMPOSE_FILE=docker-compose.server.yml` in `.env` matters. Without it, Compose picks
-`compose.yml` (the development stack), and an `up` would replace the server containers
-with ones that have no Traefik labels and open registration.
-
-**Don't copy the server file under another name** (for example `docker-compose.yml`). If
-both `compose.yml` and `docker-compose.yml` exist, Compose silently uses `compose.yml`, and
-a copy won't receive updates. If you made a copy of an older version, move your values
-into `.env`, delete the copy, and continue with `docker-compose.server.yml`.
-
-### 3. Start it
-
-```bash
-docker compose -f docker-compose.server.yml up -d --build
-```
-
-Open `https://<your-domain>` and register your account right away. The first
-account needs the **setup token** (`docker compose logs backend | grep "setup token"`,
-or the value of `ADMIN_BOOTSTRAP_TOKEN` if you set one), so a stranger who finds a
-fresh deployment can't claim it; then registration closes. That first account is
-the admin. Traefik terminates HTTPS with its own certificates; the stack needs no
-certificate configuration.
-
-The public address serves only the dashboard. Collectors don't use it; set up
-their route in the next step.
-
-### 4. Let remote servers report over your VPN
-
-Collectors reach the **collector gate**, not the public domain. In `.env`:
-
-| Setting | Example | Meaning |
-|---|---|---|
-| `COLLECTOR_BIND` | `10.8.0.1` | Your VPN server's address on the dashboard machine (for OpenVPN, the `tun` address). The gate's port is published on this address only. |
-| `COLLECTOR_PORT` | `8081` | Port of the gate on that address. |
-| `COLLECTOR_PUBLIC_URL` | `http://10.8.0.1:8081/api` | What **Connect a server** tells new machines to use. |
-| `COLLECTOR_IMAGE`, `DOCKER_PROXY_IMAGE` | `ghcr.io/…@sha256:…` | The images remote machines install; see below. |
-
-Things to get right:
-
-- **Bind to the VPN address, not `0.0.0.0`.** Docker's published ports bypass
- `ufw` and similar firewalls, so a "deny from outside" rule does not protect them.
-- If Docker starts before the VPN is up, the bind fails until the address exists.
- `restart: unless-stopped` retries, or make Docker start after the VPN (for
- example a systemd drop-in with `After=openvpn-server@<name>.service`).
-- Traffic inside the VPN is plain HTTP over the encrypted tunnel; collectors
- authenticate with a per-server token that the dashboard stores only as a hash.
-
-#### Publishing the collector images
-
-Remote machines install the collector and the Docker proxy from container images,
-pinned by digest, so no code is ever downloaded from the dashboard. The
-[Publish images](.github/workflows/images.yml) workflow builds them (amd64 and
-arm64), signs them with cosign and prints their digests:
-
-```bash
-git tag v0.1.0 && git push origin v0.1.0
-```
-
-1. Open the workflow run's summary and copy the `COLLECTOR_IMAGE=…` and
- `DOCKER_PROXY_IMAGE=…` lines into `.env`.
-2. If the repository is private, make the three packages **public** (GitHub →
- Packages → *Package settings* → *Change visibility*); otherwise machines can't
- pull them without credentials.
-3. Rebuild the stack so the backend knows the images:
- `docker compose -f docker-compose.server.yml up -d --build`.
-
-**Connect a server** won't generate a command until both images are set.
-
-### Updating
-
-```bash
-git pull
-docker compose -f docker-compose.server.yml up -d --build
-```
-
-Always rebuild **all** services (don't name individual ones). The backend,
-frontend, collector, Docker proxy and system helper each have their code built
-into their image, so a service that isn't rebuilt keeps running the old version.
-The database schema is updated automatically when the backend starts.
-
-Collectors on other machines run the image digest from the command you generated.
-To update them, publish a new version tag, put the new digests in `.env`, and run a
-freshly generated command on each machine (or pull the new digests into its
-compose project). Collectors started from `collector-compose.yml` need
-`docker compose -f collector-compose.yml up -d --build` after a `git pull`.
-
-<diagram - What each step updates: git pull → source on disk; docker compose up -d --build → backend, frontend, collector, docker-proxy and gate containers rebuilt and recreated; git tag → Publish images workflow → new digests → .env and a fresh Connect a server command for remote machines; show that skipping --build leaves the old image running>
-
-### Without Traefik
-
-- **Caddy (automatic Let's Encrypt):** with `compose.yml`, set
- `DOMAIN`, `SESSION_COOKIE_SECURE=true` and `FRONTEND_BIND=127.0.0.1` in
- `.env`, open ports 80/443, and run `docker compose --profile https up -d --build`.
-- **Private, over a VPN:** install Tailscale, WireGuard or OpenVPN on the server
- and your devices, run `docker compose up -d --build`, and use `http://<vpn-ip>:5173`
- (collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly. This uses
- the development stack's single listener; for a public dashboard with collectors
- on a VPN, use the server stack above.
-
-## Adding servers
+Full documentation is in the [`docs/`](docs/README.md) folder:
-Adding and removing servers, changing check intervals and managing the
-enrollment key are admin-only.
-
-1. On the dashboard, click **Connect a server**. Check the address field: with
- `COLLECTOR_PUBLIC_URL` set it shows your VPN address; otherwise it is prefilled
- with the address you opened the dashboard at and must be reachable from the
- new machine (not `localhost`).
-2. Click **Generate command**. This creates a single-use token that expires
- (15 minutes by default) and puts it in the command.
-3. Copy the command and run it on the new machine. It needs Docker with the
- compose plugin, and a route to the address above (the VPN).
-4. Within a few seconds the machine appears under **New servers detected** with a
- **JOIN TOKEN** badge and a pairing code (also printed in
- `docker logs homelab-collector`). Click **Add**. It starts reporting on its
- next check, without a restart.
-
-<ui - The Connect a server panel: the address field, the Generate command button with its single-use / expires note, and the copyable compose command>
-
-<ui - The New servers detected list with one pending machine showing the JOIN TOKEN badge, its pairing code and the Add / Dismiss buttons>
-
-The machine running the dashboard is added the same way (its collector starts
-with the stack, over the gate's internal listener). That collector has
-`HOMELAB_DASHBOARD_HOST=true` set in the Compose files, which makes it the machine
-shown in the charts at the top of the dashboard and first in the server list.
-Collectors installed on other machines don't set it.
-
-### What the command does
-
-It starts a small compose project named `homelab-collector` from the YAML on its
-standard input (shortened here):
-
-```bash
-docker compose -p homelab-collector -f - up -d <<'EOF'
-services:
- collector:
- image: ghcr.io/…/collector@sha256:…
- environment:
- HOMELAB_API_URL: http://10.8.0.1:8081/api
- HOMELAB_JOIN_TOKEN: hlj_…
- HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sock
- volumes:
- - /proc:/host/proc:ro
- - /:/hostfs:ro
- - collector_state:/var/lib/homelab-collector
- - docker_proxy_socket:/run/docker-proxy:ro
- cap_drop: [ALL]
- read_only: true
- docker-proxy:
- image: ghcr.io/…/docker-proxy@sha256:…
- network_mode: none
- cap_drop: [ALL]
- read_only: true
- volumes:
- - /var/run/docker.sock:/var/run/docker.sock
- - docker_proxy_socket:/run/docker-proxy
-EOF
-```
-
-| Part | Purpose |
+| | |
|---|---|
-| `HOMELAB_API_URL` | Where the collector gate is. |
-| `HOMELAB_JOIN_TOKEN` | Proves an admin asked this machine to join. Single use, expires. |
-| `image: …@sha256:…` | Signed images pinned by digest. No code is downloaded from the dashboard. |
-| `/proc:/host/proc:ro` | Host CPU, memory, load, network, uptime, boot ID, OOM counter and kernel flags. |
-| `/:/hostfs:ro` | Disk usage of every mounted filesystem, hostname, OS name and `/etc/machine-id`. |
-| `docker-proxy` | The only container with the Docker socket. It answers container list, inspect (without environment variables) and stats, and nothing else. |
-| `cap_drop: [ALL]`, `read_only: true` | The collector can't read other users' private files through `/hostfs`, or write outside its state volume. |
-| `collector_state` volume | Keeps the token after approval, so restarts don't need approval again. |
-
-To remove a collector completely:
-
-```bash
-docker rm -f homelab-collector homelab-docker-proxy
-docker volume rm homelab-collector_collector_state homelab-collector_docker_proxy_socket
-```
-
-### How registration works
-
-- Over the VPN a collector can't even ask to join without a valid join token. It
- then asks, checking every 5 seconds whether it has been approved. Clicking
- **Add** creates the server and uses up the token; the collector receives its own
- token and saves it. If a token is wrong, expired or used, the collector prints
- why and you copy a fresh command.
-- Servers are matched on `/etc/machine-id`. If a new request matches a server you
- already have, the dashboard warns that approving it **replaces that server's
- credentials and signs out its current collector**, and asks you to confirm.
- Only do that when you are reinstalling that machine.
-- If you delete a server, its collector can no longer report. On a server install
- it can't ask to join again without a new command (its token was used up). In the
- development stack it asks again and only comes back if you click **Add**.
- **Dismiss** moves a request to a collapsed "Dismissed" list.
-
-<diagram - Sequence diagram of joining: admin clicks Generate command → join token created → command run on the new machine → collector POST /collectors/register with the token over the VPN listener → pending entry with pairing code → collector polls every 5 s → admin clicks Add (token consumed) → poll returns the token → collector saves it and starts POST /metrics/ingest; include the delete → 404 → needs a new command loop>
-- Requests from collectors that stopped asking disappear after 2 minutes and are
- deleted after a day.
-
-Alternatives:
-
-- **Enrollment key (no approval step):** for scripted installs, generate a key
- under **Connect a server → Advanced** and add `HOMELAB_ENROLL_KEY: <key>` to the
- collector's environment. Only a hash of the key is stored; rotating or revoking
- it does not affect servers that are already connected.
-- **Compose from the repository:** copy the repository to the machine, set
- `HOMELAB_API_URL` (and `HOMELAB_JOIN_TOKEN`) in a `.env` next to
- `collector-compose.yml`, and run
- `docker compose -f collector-compose.yml up -d --build`. This builds the images
- locally instead of pulling them.
-
-### Networking checklist
-
-- Every monitored machine must reach the collector gate (or the dashboard address
- in the development stack); the dashboard never connects to them.
-- For machines outside your LAN, put them on a VPN (OpenVPN, WireGuard, Tailscale)
- and publish the gate on the VPN address. Don't expose plain HTTP to the
- internet: collector tokens travel over this connection.
-- On a LAN, give the dashboard machine a fixed IP (e.g. a DHCP reservation).
-
-## The dashboard
-
-- **This machine** (top): live CPU, memory and root disk charts for the machine
- running the dashboard, loaded from the last 30 minutes of stored metrics.
-- **Fleet**: one card per server, with the dashboard's own machine first. Click a
- server's name to collapse its card to a single line (status, CPU, RAM, fullest
- disk, incident count); **Collapse all** / **Expand all** appear with more than
- one server. Each card shows a **Storage** bar per filesystem, colored by the
- alert thresholds (including any per-mount overrides), a **System** row and a
- **Containers** list. Both lists start closed. A filter box appears for servers
- with more than 10 containers.
-- **Network** row: internet reachability, how many ports the server listens on and how
- many are exposed, and the firewall state, opening into the listener list and the last
- 7 days of port and firewall changes.
-- **System** row: one line of status chips (when it booted, last shutdown, clock,
- OOM kills in the last 24 hours, kernel, pending updates) that opens into the
- last 7 days of reboots, OOM kills and kernel messages. A chip that says
- *unknown* means the collector couldn't read that value on this host, not that
- anything is wrong.
-- **Active incidents**: open incidents, most severe first, with the server and
- when they started.
-
-Which cards are collapsed is remembered per browser; the System and Containers
-lists always start closed.
-
-<ui - One expanded server card: CPU/RAM/Disk/Load tiles, Storage bars for several mounts (one amber or red), the System row open with a reboot and an OOM kill in its events, and the Containers list open with an unhealthy container and one container's details expanded (stat tiles and CPU/memory charts)>
-
-<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>
-
-## Statistics
-
-**Statistics** in the navigation shows how one server behaved over time. Pick a server and a
-range (1 hour, 6 hours, 24 hours, 7 days, 30 days, 90 days, 1 year or all time) and you get
-CPU, memory, swap, load and root disk charts. Each chart is the average as a line with a
-lighter band up to the highest value, so a short spike stays visible even in a yearly view.
-Hover or touch a point for the exact time, average, highest value and number of check-ins, or
-use **View as table**. Gaps (a server that was offline) are drawn as gaps. A note under the
-cards says what each point is (for example "one per hour, from hourly summaries") and when
-the history starts. The selection is kept in the address, so a view can be bookmarked, and
-the range is remembered per browser. The short ranges refresh every 30 seconds, except
-while the tab is hidden.
-
-Under the charts, **Containers** shows every container of that server as a small card (status
-dot, health, restarts, CPU, memory against its limit), with search and sorting (problems first
-by default). Select a card for that container's CPU and memory over the chosen range, with its
-memory limit drawn as a dashed line, how much of the time it was not running, and how many
-times it restarted in the range. A container only reports CPU and memory while it runs.
-
-Further down a server's page: how its CPU and memory use were **distributed** (the share of the
-time in each 10% band), **when it is busy** (average CPU or memory by hour of the day and weekday
-over the last 30 days, in your time zone), **storage** (a chart per filesystem with an estimate
-such as "full in about 3 days" when there is a clear growth trend, and none otherwise), and
-**reliability** (how long Docker takes to list containers, and how long the collector takes).
-
-Choose **All servers** for the whole fleet: containers by state and open incidents by severity
-(donuts), incidents opened over time by severity (stacked columns), what the incidents were
-about, servers by disk usage, the containers using the most memory and CPU right now, and each
-server's availability. Availability is the share of expected check-ins that arrived; it is
-approximate because it assumes each server's current check interval. Every chart can be shown
-as a table.
-
-Short ranges come from the stored check-ins, longer ones from the summaries described under
-[Database schema](#database-schema), so a year of history costs about as much to load as an
-hour.
-
-What to know about the statistics:
-
-- **Time zones.** Charts for up to a day or two use your clock; day, week (Monday to Sunday) and
- month points are aligned to UTC and labelled in UTC, because that is how they are summarised.
-- **Some figures are approximate**: distributions and the days-until-full estimate are worked out
- in the browser from the points you are looking at, availability assumes each server's current
- check interval, and the top containers show values from the latest check-in, not an average
- over the range. The page says so where it matters.
-- **Pointer-only details.** Hover and touch tooltips don't work from the keyboard yet; every chart
- has **View as table** as an alternative.
-- **Not built yet** (tracked in Milestone 8 on GitHub): CSV and JSON export, reboots per month, a
- state timeline and restarts per day for containers, rankings over the chosen range, network
- traffic charts, event markers on charts, and retention settings for the summaries.
-
-## Alerts and incidents
-
-Rules run on every collector check-in. Each problem opens at most one incident
-(per server, per rule and, for disks, per mount); while it lasts the incident is
-updated instead of duplicated, and it resolves automatically once the rule
-stops firing.
-
-| Rule | Default | Severity |
-|---|---|---|
-| Disk usage, per mount (overridable per mount) | 85% / 95% | warning / critical |
-| Inode usage, per mount | 85% / 95% | warning / critical |
-| Memory usage | 90% / 95% | warning / critical |
-| Swap usage | 50% | warning |
-| 5-minute load per CPU core | 1.5× / 2× | warning / critical |
-| Container restart loop | 3 restarts within 30 minutes | critical |
-| Container healthcheck `unhealthy` | – | critical |
-| Container exited with an error | within 60 minutes of exiting | warning |
-| Container killed by the kernel for running out of memory | within 60 minutes | critical |
-| Container stopped by a signal (exit 137 or 143, usually `docker stop`) | within 60 minutes | info |
-| Container no longer present | stays open 60 minutes | info |
-| Container running a different image build | stays open 60 minutes | info |
-| Possible memory leak (steady climb over 12 hours) | 10 MB/h and 20% growth | warning |
-| Docker unreachable (after it worked before) | 3 check-ins in a row | critical |
-| Docker slow to list containers | over 2000 ms, 3 check-ins in a row | warning |
-| Collector can't keep up with its check interval | gathering takes over 80% of the interval, 3 check-ins in a row | warning |
-| New listener reachable from other machines | stays open 60 minutes | warning |
-| Firewall configuration changed | stays open 60 minutes | info |
-| Firewall protection reduced (ufw turned off, rules removed, a file gone) | stays open 60 minutes | warning |
-| No internet (only if the check is on) | 3 check-ins in a row | warning |
-| DNS failing (only if the check is on) | 3 check-ins in a row | warning |
-| Server stopped reporting | 3 missed check-ins | critical |
-| Host rebooted | stays open 60 minutes | info (critical after an unclean shutdown) |
-| OOM killer fired | stays open 60 minutes | warning (critical at 3 or more) |
-| Kernel fault flags (machine check, bad page, oops, warning, soft lockup) | – | warning, clears on reboot |
-| Clock not synchronised | 3 check-ins in a row | warning |
-| Reboot required | – | info |
-
-New listening ports and firewall changes are noticed the same way, by comparing with
-what the previous check-in saw. The first report is a silent baseline, an unreadable
-or empty report is treated as unknown (never as "everything closed"), and short-lived
-programs using random high ports (loopback ones and all UDP ones) are ignored. A new
-port bound only to the loopback address is recorded but doesn't open an incident; one
-reachable from other machines does. Events are capped like the container ones.
-
-Container exits, removals and image changes are noticed by comparing each check-in
-with the previous one, matching containers **by name** (Compose recreates them with
-new IDs). Nothing is reported on a server's first check-in or when Docker was
-unreachable (an empty list is not proof that everything vanished), and churn is
-capped so containers that come and go constantly can't flood the database. A leak is
-only flagged when hourly memory averages rise along a straight line without a
-restart; caches and warm-up can look similar, so treat it as a prompt to look.
-
-Host rules (disk, inodes, memory, swap, load) open an incident only after
-**3 check-ins in a row** breach the threshold, so a brief spike is ignored. An
-open incident stays open while the latest value is still over the warning
-threshold, and becomes critical only when the critical threshold is breached for
-the same number of check-ins.
-
-<diagram - Incident lifecycle for a host rule as a timeline of check-ins: value crosses the warning threshold → after 3 check-ins in a row a WARNING incident opens → stays over critical for 3 check-ins → escalates to CRITICAL → drops under warning → RESOLVED; show a single spike that opens nothing>
-
-Admins can change every threshold, the number of check-ins in a row, how long
-event incidents (reboots, OOM kills, container exits, removals and image changes)
-stay open, the slow-Docker and slow-collection limits, the memory-leak window and sensitivity, and
-**per-mount disk thresholds** under
-**Admin → Alerts**. A per-mount override replaces the default disk thresholds for
-that mount path on every server. Changes apply from each server's next check-in
-and are recorded in the audit log.
-
-<ui - Admin → Alerts: the Incident thresholds form with Warning / Critical fields per rule, Check-ins in a row, and the container restart loop row>
-
-<ui - Active incidents section with a CRITICAL and a WARNING incident, each showing its server and start time>
-
-Incidents are shown on the dashboard only; there are no push notifications yet.
-(`ADMIN_ALERT_WEBHOOK_URL` is used by the assistant's `daily_checkup` only.)
-
-## System health
-
-Collectors report what the kernel exposes, without reading any logs:
-
-| Check | Source | Notes |
-|---|---|---|
-| Reboots | `/proc/sys/kernel/random/boot_id` | A new boot ID between check-ins records a **Reboot** event. |
-| OOM kills | `oom_kill` in `/proc/vmstat` | A rising counter records an **OOM kill** event. |
-| Kernel faults | `/proc/sys/kernel/tainted` | Only fault bits count (machine check, bad page, oops, warning, soft lockup); proprietary or out-of-tree module flags are ignored. |
-| Clock sync | `adjtimex` (read-only), else systemd-timesyncd's marker | Sync state only, not a measured offset. |
-| Reboot required | `/run/reboot-required` | Debian and Ubuntu only; other distros show *unknown*. |
-
-### Optional system helper
-
-Process names, kernel error lines and crash-versus-clean shutdown detection need
-the kernel log and the login records, which the collector deliberately can't
-read. A separate container does:
-
-```bash
-docker compose --profile system-events up -d # add -f docker-compose.server.yml on a server
-```
-
-It runs with a single capability (`CAP_SYSLOG`), no network and a read-only
-filesystem, sees only `/var/log/wtmp` (read-only), and writes a sanitised JSON
-summary that the collector reads from a shared volume. With it you also get:
-
-- the OOM-killed process name on each **OOM kill** event,
-- recent kernel error lines in the **System** row (informational, they don't open incidents),
-- **Last shutdown: clean / unclean**, and a **critical** incident after an unclean reboot.
-
-Without it those show as *unknown* and everything else keeps working. Read
-[SECURITY.md](SECURITY.md) before enabling it; the helper is about 200 lines of
-standard-library Python.
-
-<ui - The System row opened on a server with the helper running: status chips on top, then a list with an OOM kill (process name) and kernel error lines>
-
-## Accounts and roles
-
-- The first account created is the **admin**. It needs the install's **setup
- token** (printed in the backend log at start, or `ADMIN_BOOTSTRAP_TOKEN`). Later
- accounts are regular users (and on the server file, registration closes after
- the first account).
-- **Passwords must be at least 16 characters** (and at most 72 bytes). A few random
- words make a good passphrase. The rule applies to new accounts; existing
- accounts keep working with their current password.
-- **Account page** (sidebar → Account), for every user: change your password (needs
- the current one; signs out your other devices), turn on **two-factor sign-in**,
- and review or end your sessions (**Sign out everywhere else**). See
- [Two-factor sign-in](#two-factor-sign-in).
-- Everyone signed in can see all servers, containers and incidents.
-- Only admins can add, approve, dismiss or delete servers, change check
- intervals, manage the enrollment key and open the **Admin** page.
-- **Admin** page settings that are connected today: alert thresholds (including
- per-mount disk thresholds), how long metrics are kept (default 30 days; older
- check-in samples and system events are deleted hourly, but only after the check-ins have
- been summarised; the summaries behind the Statistics page are not deleted yet, and a
- retention setting for each level is planned), and turning
- chat with the assistant on or off. The other sections on the page show
- placeholder data and aren't saved yet.
-
-### Two-factor sign-in
-
-Two-factor uses a one-time code from any authenticator app (TOTP, the same standard
-as Google Authenticator, Aegis or a password manager).
-
-1. **Create a key once.** Secrets are stored encrypted, so the stack needs a key.
- Generate one and put it in `.env` as `SECRETS_KEY` (see `example.env`):
-
- ```bash
- docker compose run --rm --no-deps backend python -m src.backend.secrets_box
- ```
-
- Back this key up with your other secrets. **If you lose it, authenticator codes
- stop working** (recovery codes still do, and so does the reset command below).
- To rotate it, put the new key first and keep the old one after a comma.
-2. **Turn it on** under Account → Two-factor sign-in. You get a setup key and an
- `otpauth://` link to paste into your authenticator app (no QR code yet), confirm
- with a code, and are shown **10 recovery codes once**. Each works one time.
-3. **Signing in** then asks for the code after the password. A recovery code can be
- typed instead.
-
-**Admins must use it by default.** *Admin → Users → Require two-factor for admins* is
-**on**. An admin without two-factor can still sign in, but only to the Account page
-until they finish setting it up, so nobody is locked out. A member's two-factor is
-optional. The setting is only enforced when `SECRETS_KEY` is set; without it the
-backend prints a warning at start and the policy does nothing.
-
-**Locked out?** An admin who lost both the authenticator and the recovery codes can
-reset an account from the host:
-
-```bash
-docker compose exec backend python src/backend/reset_2fa.py [email protected]
-```
-
-It turns two-factor off for that account and signs it out everywhere. With the
-policy on, the account is asked to set up two-factor again at its next sign-in.
-
-## Configuration
-
-### Stack (`.env` next to the Compose file)
-
-| Variable | Default | Description |
-|---|---|---|
-| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. |
-| `GEMINI_API_KEY` | empty | Enables the Homie chat assistant. |
-| `AGENT_MODEL` | `models/gemini-3.5-flash` | Gemini model for the assistant. |
-| `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. |
-| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. |
-| `SESSION_TTL_SECONDS` | `86400` | Session lifetime. |
-| `ALLOW_REGISTRATION` | `true` (dev), `false` (server file) | Let anyone create an account. When `false`, only the first account can be created. |
-| `SECRETS_KEY` | empty | Key that encrypts two-factor secrets. Required for two-factor sign-in and for the "require two-factor for admins" policy to be enforced. Generate with `python -m src.backend.secrets_box`; comma-separated keys allow rotation. |
-| `ADMIN_BOOTSTRAP_TOKEN` | generated at start | Setup token for the first account. Leave empty to have one generated and printed in the backend log until the first account exists. |
-| `FRONTEND_BIND` | `0.0.0.0` (dev), `127.0.0.1` (server file) | Interface for the dashboard's plain-HTTP port. |
-| `BACKEND_PORT` | `8060` | Development stack only: the host port (on `127.0.0.1`) where the backend is published for debugging. The server stack publishes no backend port. |
-| `FRONTEND_PORT` | `5173` | Dashboard port on the host. |
-| `DASHBOARD_DOMAIN` | *(required on a server)* | Domain of the dashboard, without `https://` (server file). |
-| `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERT_RESOLVER` / `TRAEFIK_NETWORK` / `TRAEFIK_ROUTER` | `websecure` / `letsencrypt` / `traefik_default` / `homelab` | Your Traefik names (server file). |
-| `CORS_ORIGINS` | `http://localhost:5173` (dev), `https://<DASHBOARD_DOMAIN>` (server file) | Comma-separated origins. Also the allow-list for the cross-site request check: a state-changing request from any other origin is refused unless its `Origin` matches the `Host` the dashboard was reached on. Add your dashboard URL if something in front rewrites `Host`. |
-| `LOGIN_RATE_LIMIT` / `LOGIN_RATE_WINDOW_SECONDS` | `5` / `60` | Login attempts per IP and per email. |
-| `REGISTER_RATE_LIMIT` / `REGISTER_RATE_WINDOW_SECONDS` | `3` / `3600` | Registrations per IP. |
-| `COLLECTOR_BIND` / `COLLECTOR_PORT` | `127.0.0.1` / `8081` | Server file: address and port where the collector gate is published. Set the address to your VPN's. |
-| `COLLECTOR_PUBLIC_URL` | empty | What **Connect a server** tells new machines to use, e.g. `http://10.8.0.1:8081/api`. |
-| `COLLECTOR_IMAGE` / `DOCKER_PROXY_IMAGE` | empty | Digest-pinned images remote machines install. **Connect a server** needs both. |
-| `JOIN_TOKEN_TTL_SECONDS` | `900` | How long a generated join token stays valid. |
-| `COLLECTOR_ACCESS` | `open` (dev), `strict` (server file) | `strict` makes the backend answer collector endpoints only when they came through the collector gate. |
-| `HOMELAB_ENROLL_KEY` | empty | Lets the stack's own collector skip approval. |
-| `HOMELAB_SERVER_NAME` | host's hostname | Name shown for the stack's own machine. |
-| `DOMAIN` | `localhost` | Domain for the optional Caddy profile. |
-| `COMPOSE_FILE` | – | Set to `docker-compose.server.yml` on a server so plain `docker compose` uses it. |
-| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes (server file; applied when the frontend image is built). |
-| `HOMELAB_CHECK_INTERVAL_SECONDS` | `60` | Initial check interval of the stack's own collector (server file); the dashboard setting overrides it. |
-
-### Collector (environment of the collector container)
-
-| Variable | Default | Description |
-|---|---|---|
-| `HOMELAB_API_URL` | `http://localhost:8060` | Dashboard API address, e.g. `https://dashboard.example.com/api`. |
-| `HOMELAB_JOIN_TOKEN` | empty | Single-use token from **Connect a server**; required over the VPN listener. |
-| `HOMELAB_ENROLL_KEY` | empty | Join immediately instead of waiting for approval. |
-| `HOMELAB_SERVER_NAME` | host's `/etc/hostname` | Name shown on the dashboard. |
-| `HOMELAB_CONNECTIVITY_TARGETS` | empty (check off) | Optional internet check: up to 5 comma-separated `host:port` targets (hostnames or IPv4), e.g. `1.1.1.1:443,9.9.9.9:443`. The collector opens a TCP connection and looks the name up; it sends nothing. Set on each collector; the dashboard cannot set it. |
-| `HOMELAB_STATE_DIR` | `/var/lib/homelab-collector` | Where the token is saved (mount a volume here). |
-| `HOMELAB_MACHINE_ID` | host's `/etc/machine-id` | Identity used to recognise reinstalled machines. |
-| `HOMELAB_DASHBOARD_HOST` | empty | `true` marks this machine as the one running the dashboard (set in the Compose files for the stack's own collector; leave it unset elsewhere). |
-| `HOMELAB_SERVER_ID` / `HOMELAB_COLLECTOR_TOKEN` | empty | Explicit credentials (legacy; overrides everything else). |
-| `HOST_PROC` / `HOST_ROOT` | `/host/proc` and `/hostfs` if mounted, else `/proc` and `/` | Where the host's `/proc` and filesystem are. |
-| `HOMELAB_DOCKER_HOST` | empty | Socket of the read-only Docker proxy, e.g. `/run/docker-proxy/docker.sock`. Preferred over `DOCKER_SOCKET_PATH`. |
-| `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Direct Docker socket, used only when `HOMELAB_DOCKER_HOST` is unset (not recommended). Without any Docker access only host metrics are sent. |
-| `HOMELAB_HELPER_FILE` | `/helper/system.json` | Where the optional system helper's summary is read from. |
-
-The system helper has `HOMELAB_HELPER_INTERVAL_SECONDS` (default `30`, minimum
-`10`); the Docker proxy listens on `HOMELAB_DOCKER_PROXY_SOCKET` and forwards to
-`DOCKER_SOCKET_PATH`. Both are set in their images and rarely need changing.
-
-The check interval is set per server on the dashboard; collectors pick it up
-from the backend's response to each report.
+| [Metrics: what every number means](docs/metrics.md) | How CPU %, load, memory, swap, disk, inodes, network and container figures are measured, and their limits. |
+| [Statistics](docs/statistics.md) | History from the last hour to all time, per server, per container and for the whole fleet. |
+| [The dashboard and system health](docs/dashboard.md) | Fleet cards, storage, the Network and System rows, the optional system helper. |
+| [Alerts and incidents](docs/alerts.md) | Every rule and its default thresholds. |
+| [Accounts and two-factor sign-in](docs/accounts.md) | Roles, passwords, authenticator codes, recovery and reset. |
+| [Deploying](docs/deployment.md), [Adding servers](docs/adding-servers.md), [Configuration](docs/configuration.md), [Troubleshooting](docs/troubleshooting.md) | Installing, running and fixing it. |
+| [Development](docs/development.md), [Database schema](docs/database.md), [API overview](docs/api.md), [Homie assistant](docs/assistant.md) | For contributors. |
## Security
-[SECURITY.md](SECURITY.md) has the full picture. In short:
-
-- **Collector privileges.** In the Compose installs the collector has no Docker
- socket, drops all capabilities and has a read-only filesystem. Only the Docker
- proxy mounts the socket; it has no network, answers four read-only requests and
- strips environment variables, labels, mounts and error text from what it returns.
- Everything a collector reports about containers is validated again by the backend
- (types, ranges, lengths, counts) before it is stored.
-- **Network checks are passive.** Listening ports are read from the host's `/proc`
- (protocol, port and scope only, no process names), the firewall check reads known
- configuration files and reports only a hash and a rule count (never the rules, and not
- the live ruleset, which would need `CAP_NET_ADMIN`), and the internet check is off
- unless you configure its targets on that collector. The collector still mounts `/`
- read-only; without capabilities it can't read other users' private files through it.
-- **Collector endpoints are not public.** On the server stack the public listener
- returns 404 for register, poll, enroll and ingest; they exist only on the
- collector gate, bound to your VPN address.
-- **No code from the dashboard.** Remote machines install signed, digest-pinned
- images. The dashboard no longer serves collector code.
-- **Join tokens.** Adding a machine needs a single-use, expiring token minted by an
- admin, then an admin's click. Re-pairing a machine that matches an existing
- server needs explicit confirmation.
-- **Fresh installs can't be claimed.** The first account needs a setup token.
-- **Accounts.** Passwords must be at least 16 characters. Two-factor sign-in
- (authenticator app plus single-use recovery codes) is available to everyone and
- **required for admins by default**; secrets are encrypted with your `SECRETS_KEY`. Every logged-in account
- can see all monitoring data and use the assistant (your Gemini key) unless an
- admin turns chat off. Only admins can add, approve or remove servers. With
- `ALLOW_REGISTRATION=false` (the default in `docker-compose.server.yml`) only the
- first account can be created; keep it that way on a public domain.
-- **Browser protections.** State-changing requests from another site are refused
- (this covers other apps on sibling subdomains), and the frontend sends a strict
- Content-Security-Policy, clickjacking protection and HSTS on the public listener.
-- **Secrets at rest.** Collector tokens, join tokens, the setup token and
- enrollment keys are stored only as SHA-256 hashes; passwords are hashed with bcrypt.
-- **Client IPs.** nginx accepts `X-Forwarded-For` only from private/Docker
- network addresses (e.g. Traefik), so rate limits see the real client IP and it
- can't be spoofed from outside.
-- **Known limits.** The one-line `docker run` install is gone; machines you install
- with `collector-compose.yml` build locally. Anyone on your VPN can reach the
- collector gate (a token and an admin's click are still needed to join).
- `src/backend/seed_user.py` creates an admin straight in the database (from
- `SEED_ADMIN_EMAIL` and `SEED_ADMIN_PASSWORD`, or an interactive prompt), skipping the
- setup token; it is meant for people who already have database access.
-
-### Homie assistant (monitoring-only)
-
-The assistant has no code path for running shell commands. It may request
-exactly one fixed, read-only check at a time: `server_health`,
-`container_health`, or `daily_checkup`, implemented with Python system APIs and
-Docker's Unix API socket, never the shell or Docker CLI.
-
-The backend container does **not** mount `/var/run/docker.sock`. If you want
-the assistant's container checks, point `DOCKER_SOCKET_PATH` at a hardened
-Docker API proxy that permits only the required GET endpoints.
-
-`daily_checkup` combines host and container checks. On a critical finding it
-POSTs an alert only to `ADMIN_ALERT_WEBHOOK_URL`; without it, the result says
-`not_configured` and no request is made. Schedule it from a trusted scheduler
-(e.g. a systemd timer) rather than letting the agent create schedules.
-
-## Development
-
-### Backend without Docker
-
-Python 3.12. The backend needs Redis for sessions and rate limits; SQLite works
-in place of PostgreSQL.
-
-```bash
-python -m venv venv && source venv/bin/activate
-pip install -r requirements.txt
-docker run -d --name homelab-dev-redis -p 6379:6379 redis:7-alpine # if Redis isn't running
-DATABASE_URL=sqlite:///./dev.db REDIS_URL=redis://localhost:6379/0 \
- uvicorn src.backend.main:app --reload --host 0.0.0.0 --port 8060
-```
-
-### Frontend dev server
+[SECURITY.md](SECURITY.md) has the full picture: what each component can touch, and the limits of each
+protection. In short:
-Node 20 (`.nvmrc`). The Vite dev server proxies `/api` to `localhost:8060` (set `BACKEND_URL` to point it elsewhere),
-matching the production nginx setup.
+- **Collectors are unprivileged.** No Docker socket, no capabilities, a read-only filesystem. Only a separate
+ read-only proxy (no network, four allowed requests, stripped answers) touches the socket.
+- **Collector endpoints are not public.** On the server stack they exist only on the gate bound to your VPN
+ address, and joining needs a single-use token plus an admin's click. Remote machines install signed,
+ digest-pinned images; the dashboard never serves code that runs on your hosts.
+- **Fresh installs can't be claimed** (the first account needs a setup token), passwords need 16 characters,
+ and two-factor sign-in is **required for admins by default**.
+- **Browser protections:** cross-site requests are refused, with a strict Content-Security-Policy and HSTS.
+- **Secrets at rest:** tokens stored only as hashes, passwords with bcrypt, two-factor secrets encrypted with your `SECRETS_KEY`.
-```bash
-nvm install && nvm use
-cd src/frontend
-npm install
-npm run dev -- --host 0.0.0.0
-```
-
-Vite prints the URL (5173, or the next free port if the Compose stack already
-uses it). `./start_frontend.sh` does the same in one step.
-
-Environment variables for the frontend build:
-
-- `VITE_API_BASE_URL` (default `/api`): API base path or URL.
-- `VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` (default `3`, minimum `1`): how
- often the dashboard refreshes.
+## Development and tests
-After changing frontend or collector code, rebuild the containers:
-`docker compose up -d --build`. Otherwise the browser keeps getting the old bundle.
-
-### Tests
+Backend (Python 3.12) and frontend (Node 20) can run without Docker; see [Development](docs/development.md).
```bash
-source venv/bin/activate
-pip install -r requirements-dev.txt # the app's requirements plus pytest
-pytest
-```
-
-No Docker, PostgreSQL or Redis is needed; the suite takes about a minute and a half. The chart
-helpers have their own tests that need only Node 20 or newer: `cd src/frontend && node --test`. See
-[TESTS.md](TESTS.md) for how the tests work, what they don't cover, and what every
-test is for.
-
-### Database schema
-
-Tables are created on backend startup (`Base.metadata.create_all`), and new
-columns are added with `ALTER TABLE ... IF NOT EXISTS` in the startup hook in
-`src/backend/main.py`. Each check-in stores one row in `host_metrics` (which also
-carries the boot ID, OOM counter, kernel flags, clock state and last shutdown), one
-per filesystem in `mount_metrics` and one per container in `container_metrics` (which
-also records the exit code, out-of-memory flag, finish time and image ID), all with the
-same timestamp. `system_events` holds reboots, OOM kills, kernel messages, container
-changes (appeared, removed, image changed) and network changes (ports opened or closed,
-firewall changes); `listener_states` and `firewall_states` hold what was last seen, `join_tokens` the hashed single-use tokens, `recovery_codes` the hashed two-factor
-recovery codes (the encrypted two-factor secret lives on the `users` row), and per-mount disk
-thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises
-the `users` table on a fresh volume.
-
-**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in
-samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container
-per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.
-Each tier is built from the finer one when its period ends, a period without check-ins gets no row, and the
-purge never deletes samples that have not been summarised yet. Nothing displays these yet; they are the
-foundation for the long-range graphs and statistics.
-
-The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and
-`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are
-migrated at start: the new indexes are created and the redundant single-column ones are dropped,
-which can be repeated safely. Creating an index briefly blocks writes to that table (a few
-seconds for a table of millions of rows); on a very large install, create the indexes yourself
-first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names
-in `src/backend/models.py`, and the start-up migration will skip them.
-
-## Project layout
-
-```text
-compose.yml development stack (default for `docker compose`)
-docker-compose.server.yml server stack behind Traefik
-collector-compose.yml collector-only stack for monitored machines
-SECURITY.md what each component can touch, and its limits
-TESTS.md how to run the tests and what each one checks
-requirements-dev.txt app requirements plus pytest, for running the tests
-pytest.ini test settings (lets plain `pytest` find the `src` package)
-.github/workflows/
- images.yml builds, signs and publishes the collector images on a version tag
-docker/
- backend.Dockerfile FastAPI backend (uvicorn)
- frontend.Dockerfile Vite build served by nginx
- collector.Dockerfile standalone collector image
- docker-proxy.Dockerfile read-only Docker API proxy image
- system-helper.Dockerfile optional kernel-log / wtmp helper image
- nginx.conf development: serves the UI, proxies /api to the backend
- nginx.strict.conf server stack: public listener, collector endpoints refused
- nginx.collector.conf server stack: collector gate (VPN and internal listeners)
- Caddyfile optional HTTPS front door (dev compose, --profile https)
- postgres/init/ SQL run on a fresh PostgreSQL volume
-src/
- backend/ API: auth, monitoring, enrollment, rate limits, models
- alerts.py alert engine: one incident per rule, auto-resolve
- rules.py alert rules and default thresholds
- rollups.py summarises check-ins into hourly/daily/weekly/monthly rollups
- history.py range-aware history behind the Statistics page
- stats.py fleet statistics: incidents over time, availability, busy-hours heatmap
- totp.py one-time codes and recovery codes (standard library only)
- secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key
- reset_2fa.py command-line two-factor reset for a locked-out account
- agent/
- collector.py metrics collector (standard library only)
- docker_proxy.py read-only Docker API proxy (standard library only)
- system_helper.py optional kernel-log / wtmp helper (standard library only)
- health_checks.py read-only checks available to the assistant
- service.py, prompts.py Gemini assistant
- frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the
- charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,
- FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the page
-tests/ pytest suite
+source venv/bin/activate && pip install -r requirements-dev.txt && pytest # backend, about a minute and a half
+cd src/frontend && node --test # chart helpers, nothing to install
```
-## API overview
-
-All paths are relative to `/api` when going through the frontend/nginx.
-
-| Method & path | Auth | Purpose |
-|---|---|---|
-| `GET /health` | – | Database and Redis status. |
-| `POST /auth/register`, `POST /auth/login`, `POST /auth/logout`, `GET /auth/me` | session cookie | Accounts and sessions. |
-| `POST /auth/login/2fa` | pending login token | Second step of sign-in: authenticator or recovery code. |
-| `GET /auth/2fa/status`, `POST /auth/2fa/setup`, `POST /auth/2fa/enable` | session (also the limited setup session) | Two-factor status, start and confirm setup. |
-| `POST /auth/2fa/disable`, `POST /auth/2fa/recovery-codes` | session + password + code | Turn two-factor off; replace the recovery codes. |
-| `POST /auth/password` | session + current password | Change password; signs out other sessions. |
-| `GET /auth/sessions`, `POST /auth/sessions/revoke-others`, `DELETE /auth/sessions/{id}` | session | List and end your own sessions. |
-| `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. |
-| `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |
-| `GET /monitoring/servers/{id}/history` | session | A server's metrics over time. With `?range=` (`1h`, `6h`, `24h`, `7d`, `30d`, `90d`, `1y`, `all`) it returns at most about 500 points from the right level of detail, each with average and maximum of the requested `metrics` (`cpu,ram,swap,load,disk,inodes,docker_ms,collect_ms`). Without `range`: CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |
-| `GET /stats/fleet` | session | Incidents opened per bucket and severity, by kind, open now, mean time to resolve, and per-server availability for `?range=`. Cached for a minute. |
-| `GET /stats/servers/{id}/heatmap` | session | Average `?metric=cpu\|ram` by hour of the day and weekday over the last 30 days, for `?offset=` minutes from UTC. |
-| `GET /monitoring/servers/{id}/mounts/history`, `GET /monitoring/servers/{id}/containers/{container_id}/history` | session | The same for one filesystem (`?mount=/data`) or one container. |
-| `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). |
-| `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. |
-| `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |
-| `DELETE /monitoring/servers/{id}` | admin | Remove a server and its data. |
-| `POST /monitoring/pending-collectors/{id}/approve` / `…/dismiss` | admin | Add or dismiss a server asking to join. Approving one that matches an existing server needs `{"replace": true}`. |
-| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). |
-| `GET /monitoring/install-info` | admin | The collector address and images the panel uses. |
-| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |
-| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds. |
-| `GET /agent/config` | session | Whether chat with the assistant is enabled. |
-| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |
-| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |
-| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |
-| `POST /monitoring/metrics/ingest` | collector token | Collector reports host, container and system metrics. |
-
-The four collector endpoints above exist only on the collector gate in the server
-stack (the public listener answers 404); the development stack serves them on the
-same address as everything else.
-| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Homie assistant chats. |
-| `GET /logs`, `POST /logs`, `GET /dashboard/summary` | session | Audit log and summary counts. |
-
-## Troubleshooting
-
-| Symptom | Check |
-|---|---|
-| A new machine never shows up under **New servers detected** | `docker logs homelab-collector` on that machine. "Could not reach …" means the address in the command isn't reachable from it (firewall, `localhost`, wrong IP/domain). |
-| Collector log says "Backend no longer accepts server …" | The server was deleted on the dashboard; it is asking to join again, so click **Add** or **Dismiss**. |
-| `network traefik_default declared as external, but could not be found` | You're using `docker-compose.server.yml` on a machine without that Traefik network. Use `compose.yml` for development, or set `TRAEFIK_NETWORK` in `.env` to the network Traefik is really on. |
-| Dashboard shows old behaviour after a code change or `git pull` | Rebuild **all** services with `docker compose up -d --build` (with `-f docker-compose.server.yml` on a server unless `COMPOSE_FILE` is set) and hard-refresh the browser (Ctrl+Shift+R). |
-| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec homelab-collector grep -c dashboard_host /app/collector.py` prints `0`. Rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). |
-| A server shows only `/` and no **Storage** bars for other disks | Its collector predates per-mount reporting. Install a newer collector image (run a freshly generated command, or `docker compose -f collector-compose.yml up -d --build`). A disk mounted after the collector started appears after a collector restart. |
-| Logged out after opening the dashboard at a new address | Login cookies belong to the address they were set on; log in again. |
-| Login fails behind HTTPS with `SESSION_COOKIE_SECURE=true` | The cookie is only sent over HTTPS; open the `https://` address, not `http://…:5173`. |
-| Collector log says "Registration refused (HTTP 403): The join token is invalid, expired or already used" | The token in the command has expired, was already used, or is wrong. Click **Generate command** again and run the new command. |
-| **Connect a server** says no collector images are configured | Set `COLLECTOR_IMAGE` and `DOCKER_PROXY_IMAGE` in `.env` (see [Publishing the collector images](#publishing-the-collector-images)) and rebuild. |
-| The register page asks for a setup token | The install has no accounts yet. Use `docker compose logs backend \| grep "setup token"`, or `ADMIN_BOOTSTRAP_TOKEN`. |
-| "Password must be at least 16 characters long" | New passwords need 16 or more characters (at most 72 bytes). A few random words work well. |
-| "Cross-site request refused" (403) on a button or form | The request came from another origin. Open the dashboard at the address you normally use. If a proxy in front rewrites the `Host` header, add your dashboard URL to `CORS_ORIGINS`. |
-| The **System** row shows *unknown* for last shutdown or has no kernel messages | The optional system helper isn't running. Start it with `docker compose --profile system-events up -d`. *Unknown* for reboot-required on non-Debian hosts is expected. |
-| `Error response … cannot assign requested address` for the collector gate | `COLLECTOR_BIND` is set to an address that doesn't exist yet (VPN not up). Start the VPN first, or fix the address. |
-| The **Network** row says *no config found* for the firewall | None of the known firewall files exist on that host (`/etc/ufw`, `/etc/nftables.conf`, `/etc/iptables/rules.v4`). The check reads configuration, not the live ruleset, so a host that configures its firewall some other way shows this. |
-| The **Network** row says *not checked* for the internet | The check is opt-in. Set `HOMELAB_CONNECTIVITY_TARGETS` on that collector (for example `1.1.1.1:443`). |
-| Listeners show *unknown* or the Network row is empty | The collector couldn't read the host's network tables (older collector, or a table with more than 500,000 sockets). An unreadable report never raises a "port closed" alert. |
-| "Invalid code." although the code looks right | Codes depend on the clock: check the phone's time is automatic. A code also can't be used twice, so wait for the next one. Five wrong codes lock the code step for 5 minutes. |
-| Two-factor setup says it is not available | `SECRETS_KEY` is missing or invalid in `.env`. Generate one (see [Two-factor sign-in](#two-factor-sign-in)) and restart the backend. The backend log shows a SECURITY WARNING while the key is missing. |
-| Lost the authenticator and the recovery codes | Run `docker compose exec backend python src/backend/reset_2fa.py <email>` on the host, then sign in with the password and set it up again. |
-| Authenticator codes stopped working after a restore or `.env` change | `SECRETS_KEY` changed or was lost, so the stored secret can't be decrypted. Restore the old key (rotation: `new,old`), or use a recovery code and the reset command. |
-| Server marked offline | It missed 3 check-ins in a row. Check the machine, its collector log and its route to the dashboard. The incident resolves on the next report. |
+[TESTS.md](TESTS.md) explains how the tests work, what they don't cover and what each one is for.
## License
diff --git a/SECURITY.md b/SECURITY.md
index abd79aa..4781e2d 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -228,6 +228,15 @@ The collector's network checks are passive or opt-in, and add no privileges:
- **Visibility.** Like all monitoring data, the list of listening ports and firewall state is visible to
every signed-in account. Keep registration closed on a public dashboard.
+## Client addresses and collector installs
+
+- **Client IPs.** nginx accepts `X-Forwarded-For` only from private or Docker network addresses (for
+ example Traefik), so login and registration rate limits see the real client address and it can't be
+ spoofed from outside.
+- **No one-line installer.** The old `docker run` install is gone. Remote machines run the compose
+ command **Connect a server** generates (signed, digest-pinned images), and machines set up from
+ `collector-compose.yml` build the images locally.
+
## Statistics and history
- **Who can see it.** Every signed-in account sees the same statistics as the dashboard, for every
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..d262af9
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,33 @@
+# Documentation
+
+The [README](../README.md) is the short version: what the project is, how to start it. Everything else lives here.
+
+## Using the dashboard
+
+| Page | What is in it |
+|---|---|
+| [The dashboard and system health](dashboard.md) | Fleet cards, storage bars, the Network and System rows, active incidents, the optional system helper. |
+| [Statistics](statistics.md) | History from the last hour to all time: servers, containers and the whole fleet. |
+| [Metrics: what every number means](metrics.md) | CPU %, load, memory, swap, disk, inodes, network, container and system figures: how each is measured and its limits. |
+| [Alerts and incidents](alerts.md) | Every rule, its default thresholds, and how incidents open and resolve. |
+| [Accounts, roles and two-factor sign-in](accounts.md) | Passwords, roles, the Account page, authenticator codes, recovery codes, locked-out accounts. |
+| [Homie assistant](assistant.md) | The optional, monitoring-only AI assistant. |
+
+## Installing and running
+
+| Page | What is in it |
+|---|---|
+| [Deploying on a server](deployment.md) | The server stack behind Traefik, the VPN collector gate, publishing the collector images, updating, running without Traefik. |
+| [Adding servers](adding-servers.md) | **Connect a server**, what its command does, how registration and approval work, networking. |
+| [Configuration](configuration.md) | Every `.env` setting for the stack, the collector and the frontend build. |
+| [Troubleshooting](troubleshooting.md) | Symptoms and what to check. |
+
+## For developers
+
+| Page | What is in it |
+|---|---|
+| [Development](development.md) | Running the backend and frontend without Docker, tests, project layout. |
+| [Database schema](database.md) | Tables, summarised history, indexes and migrations. |
+| [API overview](api.md) | Every endpoint, who may call it, and what it returns. |
+| [../TESTS.md](../TESTS.md) | How the tests work, what they don't cover, and what each one checks. |
+| [../SECURITY.md](../SECURITY.md) | What every component can touch, and the limits of each protection. |
diff --git a/docs/accounts.md b/docs/accounts.md
new file mode 100644
index 0000000..35873e9
--- /dev/null
+++ b/docs/accounts.md
@@ -0,0 +1,62 @@
+[← Documentation](README.md)
+
+# Accounts and roles
+
+- The first account created is the **admin**. It needs the install's **setup
+ token** (printed in the backend log at start, or `ADMIN_BOOTSTRAP_TOKEN`). Later
+ accounts are regular users (and on the server file, registration closes after
+ the first account).
+- **Passwords must be at least 16 characters** (and at most 72 bytes). A few random
+ words make a good passphrase. The rule applies to new accounts; existing
+ accounts keep working with their current password.
+- **Account page** (sidebar → Account), for every user: change your password (needs
+ the current one; signs out your other devices), turn on **two-factor sign-in**,
+ and review or end your sessions (**Sign out everywhere else**). See
+ [Two-factor sign-in](accounts.md#two-factor-sign-in).
+- Everyone signed in can see all servers, containers and incidents.
+- Only admins can add, approve, dismiss or delete servers, change check
+ intervals, manage the enrollment key and open the **Admin** page.
+- **Admin** page settings that are connected today: alert thresholds (including
+ per-mount disk thresholds), how long metrics are kept (default 30 days; older
+ check-in samples and system events are deleted hourly, but only after the check-ins have
+ been summarised; the summaries behind the Statistics page are not deleted yet, and a
+ retention setting for each level is planned), and turning
+ chat with the assistant on or off. The other sections on the page show
+ placeholder data and aren't saved yet.
+
+## Two-factor sign-in
+
+Two-factor uses a one-time code from any authenticator app (TOTP, the same standard
+as Google Authenticator, Aegis or a password manager).
+
+1. **Create a key once.** Secrets are stored encrypted, so the stack needs a key.
+ Generate one and put it in `.env` as `SECRETS_KEY` (see `example.env`):
+
+ ```bash
+ docker compose run --rm --no-deps backend python -m src.backend.secrets_box
+ ```
+
+ Back this key up with your other secrets. **If you lose it, authenticator codes
+ stop working** (recovery codes still do, and so does the reset command below).
+ To rotate it, put the new key first and keep the old one after a comma.
+2. **Turn it on** under Account → Two-factor sign-in. You get a setup key and an
+ `otpauth://` link to paste into your authenticator app (no QR code yet), confirm
+ with a code, and are shown **10 recovery codes once**. Each works one time.
+3. **Signing in** then asks for the code after the password. A recovery code can be
+ typed instead.
+
+**Admins must use it by default.** *Admin → Users → Require two-factor for admins* is
+**on**. An admin without two-factor can still sign in, but only to the Account page
+until they finish setting it up, so nobody is locked out. A member's two-factor is
+optional. The setting is only enforced when `SECRETS_KEY` is set; without it the
+backend prints a warning at start and the policy does nothing.
+
+**Locked out?** An admin who lost both the authenticator and the recovery codes can
+reset an account from the host:
+
+```bash
+docker compose exec backend python src/backend/reset_2fa.py [email protected]
+```
+
+It turns two-factor off for that account and signs it out everywhere. With the
+policy on, the account is asked to set up two-factor again at its next sign-in.
diff --git a/docs/adding-servers.md b/docs/adding-servers.md
new file mode 100644
index 0000000..0d48c89
--- /dev/null
+++ b/docs/adding-servers.md
@@ -0,0 +1,120 @@
+[← Documentation](README.md)
+
+# Adding servers
+
+Adding and removing servers, changing check intervals and managing the
+enrollment key are admin-only.
+
+1. On the dashboard, click **Connect a server**. Check the address field: with
+ `COLLECTOR_PUBLIC_URL` set it shows your VPN address; otherwise it is prefilled
+ with the address you opened the dashboard at and must be reachable from the
+ new machine (not `localhost`).
+2. Click **Generate command**. This creates a single-use token that expires
+ (15 minutes by default) and puts it in the command.
+3. Copy the command and run it on the new machine. It needs Docker with the
+ compose plugin, and a route to the address above (the VPN).
+4. Within a few seconds the machine appears under **New servers detected** with a
+ **JOIN TOKEN** badge and a pairing code (also printed in
+ `docker logs homelab-collector`). Click **Add**. It starts reporting on its
+ next check, without a restart.
+
+<ui - The Connect a server panel: the address field, the Generate command button with its single-use / expires note, and the copyable compose command>
+
+<ui - The New servers detected list with one pending machine showing the JOIN TOKEN badge, its pairing code and the Add / Dismiss buttons>
+
+The machine running the dashboard is added the same way (its collector starts
+with the stack, over the gate's internal listener). That collector has
+`HOMELAB_DASHBOARD_HOST=true` set in the Compose files, which makes it the machine
+shown in the charts at the top of the dashboard and first in the server list.
+Collectors installed on other machines don't set it.
+
+## What the command does
+
+It starts a small compose project named `homelab-collector` from the YAML on its
+standard input (shortened here):
+
+```bash
+docker compose -p homelab-collector -f - up -d <<'EOF'
+services:
+ collector:
+ image: ghcr.io/…/collector@sha256:…
+ environment:
+ HOMELAB_API_URL: http://10.8.0.1:8081/api
+ HOMELAB_JOIN_TOKEN: hlj_…
+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sock
+ volumes:
+ - /proc:/host/proc:ro
+ - /:/hostfs:ro
+ - collector_state:/var/lib/homelab-collector
+ - docker_proxy_socket:/run/docker-proxy:ro
+ cap_drop: [ALL]
+ read_only: true
+ docker-proxy:
+ image: ghcr.io/…/docker-proxy@sha256:…
+ network_mode: none
+ cap_drop: [ALL]
+ read_only: true
+ volumes:
+ - /var/run/docker.sock:/var/run/docker.sock
+ - docker_proxy_socket:/run/docker-proxy
+EOF
+```
+
+| Part | Purpose |
+|---|---|
+| `HOMELAB_API_URL` | Where the collector gate is. |
+| `HOMELAB_JOIN_TOKEN` | Proves an admin asked this machine to join. Single use, expires. |
+| `image: …@sha256:…` | Signed images pinned by digest. No code is downloaded from the dashboard. |
+| `/proc:/host/proc:ro` | Host CPU, memory, load, network, uptime, boot ID, OOM counter and kernel flags. |
+| `/:/hostfs:ro` | Disk usage of every mounted filesystem, hostname, OS name and `/etc/machine-id`. |
+| `docker-proxy` | The only container with the Docker socket. It answers container list, inspect (without environment variables) and stats, and nothing else. |
+| `cap_drop: [ALL]`, `read_only: true` | The collector can't read other users' private files through `/hostfs`, or write outside its state volume. |
+| `collector_state` volume | Keeps the token after approval, so restarts don't need approval again. |
+
+To remove a collector completely:
+
+```bash
+docker rm -f homelab-collector homelab-docker-proxy
+docker volume rm homelab-collector_collector_state homelab-collector_docker_proxy_socket
+```
+
+## How registration works
+
+- Over the VPN a collector can't even ask to join without a valid join token. It
+ then asks, checking every 5 seconds whether it has been approved. Clicking
+ **Add** creates the server and uses up the token; the collector receives its own
+ token and saves it. If a token is wrong, expired or used, the collector prints
+ why and you copy a fresh command.
+- Servers are matched on `/etc/machine-id`. If a new request matches a server you
+ already have, the dashboard warns that approving it **replaces that server's
+ credentials and signs out its current collector**, and asks you to confirm.
+ Only do that when you are reinstalling that machine.
+- If you delete a server, its collector can no longer report. On a server install
+ it can't ask to join again without a new command (its token was used up). In the
+ development stack it asks again and only comes back if you click **Add**.
+ **Dismiss** moves a request to a collapsed "Dismissed" list.
+
+<diagram - Sequence diagram of joining: admin clicks Generate command → join token created → command run on the new machine → collector POST /collectors/register with the token over the VPN listener → pending entry with pairing code → collector polls every 5 s → admin clicks Add (token consumed) → poll returns the token → collector saves it and starts POST /metrics/ingest; include the delete → 404 → needs a new command loop>
+- Requests from collectors that stopped asking disappear after 2 minutes and are
+ deleted after a day.
+
+Alternatives:
+
+- **Enrollment key (no approval step):** for scripted installs, generate a key
+ under **Connect a server → Advanced** and add `HOMELAB_ENROLL_KEY: <key>` to the
+ collector's environment. Only a hash of the key is stored; rotating or revoking
+ it does not affect servers that are already connected.
+- **Compose from the repository:** copy the repository to the machine, set
+ `HOMELAB_API_URL` (and `HOMELAB_JOIN_TOKEN`) in a `.env` next to
+ `collector-compose.yml`, and run
+ `docker compose -f collector-compose.yml up -d --build`. This builds the images
+ locally instead of pulling them.
+
+## Networking checklist
+
+- Every monitored machine must reach the collector gate (or the dashboard address
+ in the development stack); the dashboard never connects to them.
+- For machines outside your LAN, put them on a VPN (OpenVPN, WireGuard, Tailscale)
+ and publish the gate on the VPN address. Don't expose plain HTTP to the
+ internet: collector tokens travel over this connection.
+- On a LAN, give the dashboard machine a fixed IP (e.g. a DHCP reservation).
diff --git a/docs/alerts.md b/docs/alerts.md
new file mode 100644
index 0000000..7937373
--- /dev/null
+++ b/docs/alerts.md
@@ -0,0 +1,78 @@
+[← Documentation](README.md)
+
+# Alerts and incidents
+
+The values the rules look at are explained in [Metrics](metrics.md). Note that the load rule uses the 5-minute load per CPU core.
+
+Rules run on every collector check-in. Each problem opens at most one incident
+(per server, per rule and, for disks, per mount); while it lasts the incident is
+updated instead of duplicated, and it resolves automatically once the rule
+stops firing.
+
+| Rule | Default | Severity |
+|---|---|---|
+| Disk usage, per mount (overridable per mount) | 85% / 95% | warning / critical |
+| Inode usage, per mount | 85% / 95% | warning / critical |
+| Memory usage | 90% / 95% | warning / critical |
+| Swap usage | 50% | warning |
+| 5-minute load per CPU core | 1.5× / 2× | warning / critical |
+| Container restart loop | 3 restarts within 30 minutes | critical |
+| Container healthcheck `unhealthy` | – | critical |
+| Container exited with an error | within 60 minutes of exiting | warning |
+| Container killed by the kernel for running out of memory | within 60 minutes | critical |
+| Container stopped by a signal (exit 137 or 143, usually `docker stop`) | within 60 minutes | info |
+| Container no longer present | stays open 60 minutes | info |
+| Container running a different image build | stays open 60 minutes | info |
+| Possible memory leak (steady climb over 12 hours) | 10 MB/h and 20% growth | warning |
+| Docker unreachable (after it worked before) | 3 check-ins in a row | critical |
+| Docker slow to list containers | over 2000 ms, 3 check-ins in a row | warning |
+| Collector can't keep up with its check interval | gathering takes over 80% of the interval, 3 check-ins in a row | warning |
+| New listener reachable from other machines | stays open 60 minutes | warning |
+| Firewall configuration changed | stays open 60 minutes | info |
+| Firewall protection reduced (ufw turned off, rules removed, a file gone) | stays open 60 minutes | warning |
+| No internet (only if the check is on) | 3 check-ins in a row | warning |
+| DNS failing (only if the check is on) | 3 check-ins in a row | warning |
+| Server stopped reporting | 3 missed check-ins | critical |
+| Host rebooted | stays open 60 minutes | info (critical after an unclean shutdown) |
+| OOM killer fired | stays open 60 minutes | warning (critical at 3 or more) |
+| Kernel fault flags (machine check, bad page, oops, warning, soft lockup) | – | warning, clears on reboot |
+| Clock not synchronised | 3 check-ins in a row | warning |
+| Reboot required | – | info |
+
+New listening ports and firewall changes are noticed the same way, by comparing with
+what the previous check-in saw. The first report is a silent baseline, an unreadable
+or empty report is treated as unknown (never as "everything closed"), and short-lived
+programs using random high ports (loopback ones and all UDP ones) are ignored. A new
+port bound only to the loopback address is recorded but doesn't open an incident; one
+reachable from other machines does. Events are capped like the container ones.
+
+Container exits, removals and image changes are noticed by comparing each check-in
+with the previous one, matching containers **by name** (Compose recreates them with
+new IDs). Nothing is reported on a server's first check-in or when Docker was
+unreachable (an empty list is not proof that everything vanished), and churn is
+capped so containers that come and go constantly can't flood the database. A leak is
+only flagged when hourly memory averages rise along a straight line without a
+restart; caches and warm-up can look similar, so treat it as a prompt to look.
+
+Host rules (disk, inodes, memory, swap, load) open an incident only after
+**3 check-ins in a row** breach the threshold, so a brief spike is ignored. An
+open incident stays open while the latest value is still over the warning
+threshold, and becomes critical only when the critical threshold is breached for
+the same number of check-ins.
+
+<diagram - Incident lifecycle for a host rule as a timeline of check-ins: value crosses the warning threshold → after 3 check-ins in a row a WARNING incident opens → stays over critical for 3 check-ins → escalates to CRITICAL → drops under warning → RESOLVED; show a single spike that opens nothing>
+
+Admins can change every threshold, the number of check-ins in a row, how long
+event incidents (reboots, OOM kills, container exits, removals and image changes)
+stay open, the slow-Docker and slow-collection limits, the memory-leak window and sensitivity, and
+**per-mount disk thresholds** under
+**Admin → Alerts**. A per-mount override replaces the default disk thresholds for
+that mount path on every server. Changes apply from each server's next check-in
+and are recorded in the audit log.
+
+<ui - Admin → Alerts: the Incident thresholds form with Warning / Critical fields per rule, Check-ins in a row, and the container restart loop row>
+
+<ui - Active incidents section with a CRITICAL and a WARNING incident, each showing its server and start time>
+
+Incidents are shown on the dashboard only; there are no push notifications yet.
+(`ADMIN_ALERT_WEBHOOK_URL` is used by the assistant's `daily_checkup` only.)
diff --git a/docs/api.md b/docs/api.md
new file mode 100644
index 0000000..bd07b2e
--- /dev/null
+++ b/docs/api.md
@@ -0,0 +1,41 @@
+[← Documentation](README.md)
+
+# API overview
+
+All paths are relative to `/api` when going through the frontend/nginx.
+
+| Method & path | Auth | Purpose |
+|---|---|---|
+| `GET /health` | – | Database and Redis status. |
+| `POST /auth/register`, `POST /auth/login`, `POST /auth/logout`, `GET /auth/me` | session cookie | Accounts and sessions. |
+| `POST /auth/login/2fa` | pending login token | Second step of sign-in: authenticator or recovery code. |
+| `GET /auth/2fa/status`, `POST /auth/2fa/setup`, `POST /auth/2fa/enable` | session (also the limited setup session) | Two-factor status, start and confirm setup. |
+| `POST /auth/2fa/disable`, `POST /auth/2fa/recovery-codes` | session + password + code | Turn two-factor off; replace the recovery codes. |
+| `POST /auth/password` | session + current password | Change password; signs out other sessions. |
+| `GET /auth/sessions`, `POST /auth/sessions/revoke-others`, `DELETE /auth/sessions/{id}` | session | List and end your own sessions. |
+| `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. |
+| `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |
+| `GET /monitoring/servers/{id}/history` | session | A server's metrics over time. With `?range=` (`1h`, `6h`, `24h`, `7d`, `30d`, `90d`, `1y`, `all`) it returns at most about 500 points from the right level of detail, each with average and maximum of the requested `metrics` (`cpu,ram,swap,load,disk,inodes,docker_ms,collect_ms`). Without `range`: CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |
+| `GET /stats/fleet` | session | Incidents opened per bucket and severity, by kind, open now, mean time to resolve, and per-server availability for `?range=`. Cached for a minute. |
+| `GET /stats/servers/{id}/heatmap` | session | Average `?metric=cpu\|ram` by hour of the day and weekday over the last 30 days, for `?offset=` minutes from UTC. |
+| `GET /monitoring/servers/{id}/mounts/history`, `GET /monitoring/servers/{id}/containers/{container_id}/history` | session | The same for one filesystem (`?mount=/data`) or one container. |
+| `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). |
+| `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. |
+| `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |
+| `DELETE /monitoring/servers/{id}` | admin | Remove a server and its data. |
+| `POST /monitoring/pending-collectors/{id}/approve` / `…/dismiss` | admin | Add or dismiss a server asking to join. Approving one that matches an existing server needs `{"replace": true}`. |
+| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). |
+| `GET /monitoring/install-info` | admin | The collector address and images the panel uses. |
+| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |
+| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds. |
+| `GET /agent/config` | session | Whether chat with the assistant is enabled. |
+| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |
+| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |
+| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |
+| `POST /monitoring/metrics/ingest` | collector token | Collector reports host, container and system metrics. |
+| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Homie assistant chats. |
+| `GET /logs`, `POST /logs`, `GET /dashboard/summary` | session | Audit log and summary counts. |
+
+The four collector endpoints above exist only on the collector gate in the server
+stack (the public listener answers 404); the development stack serves them on the
+same address as everything else.
diff --git a/docs/assistant.md b/docs/assistant.md
new file mode 100644
index 0000000..16a2e94
--- /dev/null
+++ b/docs/assistant.md
@@ -0,0 +1,17 @@
+[← Documentation](README.md)
+
+# Homie assistant (monitoring-only)
+
+The assistant has no code path for running shell commands. It may request
+exactly one fixed, read-only check at a time: `server_health`,
+`container_health`, or `daily_checkup`, implemented with Python system APIs and
+Docker's Unix API socket, never the shell or Docker CLI.
+
+The backend container does **not** mount `/var/run/docker.sock`. If you want
+the assistant's container checks, point `DOCKER_SOCKET_PATH` at a hardened
+Docker API proxy that permits only the required GET endpoints.
+
+`daily_checkup` combines host and container checks. On a critical finding it
+POSTs an alert only to `ADMIN_ALERT_WEBHOOK_URL`; without it, the result says
+`not_configured` and no request is made. Schedule it from a trusted scheduler
+(e.g. a systemd timer) rather than letting the agent create schedules.
diff --git a/docs/configuration.md b/docs/configuration.md
new file mode 100644
index 0000000..fdf4a45
--- /dev/null
+++ b/docs/configuration.md
@@ -0,0 +1,69 @@
+[← Documentation](README.md)
+
+# Configuration
+
+## Stack (`.env` next to the Compose file)
+
+| Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. |
+| `GEMINI_API_KEY` | empty | Enables the Homie chat assistant. |
+| `AGENT_MODEL` | `models/gemini-3.5-flash` | Gemini model for the assistant. |
+| `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. |
+| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. |
+| `SESSION_TTL_SECONDS` | `86400` | Session lifetime. |
+| `ALLOW_REGISTRATION` | `true` (dev), `false` (server file) | Let anyone create an account. When `false`, only the first account can be created. |
+| `SECRETS_KEY` | empty | Key that encrypts two-factor secrets. Required for two-factor sign-in and for the "require two-factor for admins" policy to be enforced. Generate with `python -m src.backend.secrets_box`; comma-separated keys allow rotation. |
+| `ADMIN_BOOTSTRAP_TOKEN` | generated at start | Setup token for the first account. Leave empty to have one generated and printed in the backend log until the first account exists. |
+| `FRONTEND_BIND` | `0.0.0.0` (dev), `127.0.0.1` (server file) | Interface for the dashboard's plain-HTTP port. |
+| `BACKEND_PORT` | `8060` | Development stack only: the host port (on `127.0.0.1`) where the backend is published for debugging. The server stack publishes no backend port. |
+| `FRONTEND_PORT` | `5173` | Dashboard port on the host. |
+| `DASHBOARD_DOMAIN` | *(required on a server)* | Domain of the dashboard, without `https://` (server file). |
+| `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERT_RESOLVER` / `TRAEFIK_NETWORK` / `TRAEFIK_ROUTER` | `websecure` / `letsencrypt` / `traefik_default` / `homelab` | Your Traefik names (server file). |
+| `CORS_ORIGINS` | `http://localhost:5173` (dev), `https://<DASHBOARD_DOMAIN>` (server file) | Comma-separated origins. Also the allow-list for the cross-site request check: a state-changing request from any other origin is refused unless its `Origin` matches the `Host` the dashboard was reached on. Add your dashboard URL if something in front rewrites `Host`. |
+| `LOGIN_RATE_LIMIT` / `LOGIN_RATE_WINDOW_SECONDS` | `5` / `60` | Login attempts per IP and per email. |
+| `REGISTER_RATE_LIMIT` / `REGISTER_RATE_WINDOW_SECONDS` | `3` / `3600` | Registrations per IP. |
+| `COLLECTOR_BIND` / `COLLECTOR_PORT` | `127.0.0.1` / `8081` | Server file: address and port where the collector gate is published. Set the address to your VPN's. |
+| `COLLECTOR_PUBLIC_URL` | empty | What **Connect a server** tells new machines to use, e.g. `http://10.8.0.1:8081/api`. |
+| `COLLECTOR_IMAGE` / `DOCKER_PROXY_IMAGE` | empty | Digest-pinned images remote machines install. **Connect a server** needs both. |
+| `JOIN_TOKEN_TTL_SECONDS` | `900` | How long a generated join token stays valid. |
+| `COLLECTOR_ACCESS` | `open` (dev), `strict` (server file) | `strict` makes the backend answer collector endpoints only when they came through the collector gate. |
+| `HOMELAB_ENROLL_KEY` | empty | Lets the stack's own collector skip approval. |
+| `HOMELAB_SERVER_NAME` | host's hostname | Name shown for the stack's own machine. |
+| `DOMAIN` | `localhost` | Domain for the optional Caddy profile. |
+| `COMPOSE_FILE` | – | Set to `docker-compose.server.yml` on a server so plain `docker compose` uses it. |
+| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes (server file; applied when the frontend image is built). |
+| `HOMELAB_CHECK_INTERVAL_SECONDS` | `60` | Initial check interval of the stack's own collector (server file); the dashboard setting overrides it. |
+
+## Collector (environment of the collector container)
+
+| Variable | Default | Description |
+|---|---|---|
+| `HOMELAB_API_URL` | `http://localhost:8060` | Dashboard API address, e.g. `https://dashboard.example.com/api`. |
+| `HOMELAB_JOIN_TOKEN` | empty | Single-use token from **Connect a server**; required over the VPN listener. |
+| `HOMELAB_ENROLL_KEY` | empty | Join immediately instead of waiting for approval. |
+| `HOMELAB_SERVER_NAME` | host's `/etc/hostname` | Name shown on the dashboard. |
+| `HOMELAB_CONNECTIVITY_TARGETS` | empty (check off) | Optional internet check: up to 5 comma-separated `host:port` targets (hostnames or IPv4), e.g. `1.1.1.1:443,9.9.9.9:443`. The collector opens a TCP connection and looks the name up; it sends nothing. Set on each collector; the dashboard cannot set it. |
+| `HOMELAB_STATE_DIR` | `/var/lib/homelab-collector` | Where the token is saved (mount a volume here). |
+| `HOMELAB_MACHINE_ID` | host's `/etc/machine-id` | Identity used to recognise reinstalled machines. |
+| `HOMELAB_DASHBOARD_HOST` | empty | `true` marks this machine as the one running the dashboard (set in the Compose files for the stack's own collector; leave it unset elsewhere). |
+| `HOMELAB_SERVER_ID` / `HOMELAB_COLLECTOR_TOKEN` | empty | Explicit credentials (legacy; overrides everything else). |
+| `HOST_PROC` / `HOST_ROOT` | `/host/proc` and `/hostfs` if mounted, else `/proc` and `/` | Where the host's `/proc` and filesystem are. |
+| `HOMELAB_DOCKER_HOST` | empty | Socket of the read-only Docker proxy, e.g. `/run/docker-proxy/docker.sock`. Preferred over `DOCKER_SOCKET_PATH`. |
+| `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Direct Docker socket, used only when `HOMELAB_DOCKER_HOST` is unset (not recommended). Without any Docker access only host metrics are sent. |
+| `HOMELAB_HELPER_FILE` | `/helper/system.json` | Where the optional system helper's summary is read from. |
+
+The system helper has `HOMELAB_HELPER_INTERVAL_SECONDS` (default `30`, minimum
+`10`); the Docker proxy listens on `HOMELAB_DOCKER_PROXY_SOCKET` and forwards to
+`DOCKER_SOCKET_PATH`. Both are set in their images and rarely need changing.
+
+The check interval is set per server on the dashboard; collectors pick it up
+from the backend's response to each report.
+
+## Frontend build variables
+
+Environment variables for the frontend build (set when the image is built):
+
+- `VITE_API_BASE_URL` (default `/api`): API base path or URL.
+- `VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` (default `3`, minimum `1`): how
+ often the dashboard refreshes.
diff --git a/docs/dashboard.md b/docs/dashboard.md
new file mode 100644
index 0000000..f175af4
--- /dev/null
+++ b/docs/dashboard.md
@@ -0,0 +1,68 @@
+[← Documentation](README.md)
+
+# The dashboard and system health
+
+For what each value means and how it is measured, see [Metrics](metrics.md).
+
+- **This machine** (top): live CPU, memory and root disk charts for the machine
+ running the dashboard, loaded from the last 30 minutes of stored metrics.
+- **Fleet**: one card per server, with the dashboard's own machine first. Click a
+ server's name to collapse its card to a single line (status, CPU, RAM, fullest
+ disk, incident count); **Collapse all** / **Expand all** appear with more than
+ one server. Each card shows a **Storage** bar per filesystem, colored by the
+ alert thresholds (including any per-mount overrides), a **System** row and a
+ **Containers** list. Both lists start closed. A filter box appears for servers
+ with more than 10 containers.
+- **Network** row: internet reachability, how many ports the server listens on and how
+ many are exposed, and the firewall state, opening into the listener list and the last
+ 7 days of port and firewall changes.
+- **System** row: one line of status chips (when it booted, last shutdown, clock,
+ OOM kills in the last 24 hours, kernel, pending updates) that opens into the
+ last 7 days of reboots, OOM kills and kernel messages. A chip that says
+ *unknown* means the collector couldn't read that value on this host, not that
+ anything is wrong.
+- **Active incidents**: open incidents, most severe first, with the server and
+ when they started.
+
+Which cards are collapsed is remembered per browser; the System and Containers
+lists always start closed.
+
+<ui - One expanded server card: CPU/RAM/Disk/Load tiles, Storage bars for several mounts (one amber or red), the System row open with a reboot and an OOM kill in its events, and the Containers list open with an unhealthy container and one container's details expanded (stat tiles and CPU/memory charts)>
+
+<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>
+
+## System health
+
+Collectors report what the kernel exposes, without reading any logs:
+
+| Check | Source | Notes |
+|---|---|---|
+| Reboots | `/proc/sys/kernel/random/boot_id` | A new boot ID between check-ins records a **Reboot** event. |
+| OOM kills | `oom_kill` in `/proc/vmstat` | A rising counter records an **OOM kill** event. |
+| Kernel faults | `/proc/sys/kernel/tainted` | Only fault bits count (machine check, bad page, oops, warning, soft lockup); proprietary or out-of-tree module flags are ignored. |
+| Clock sync | `adjtimex` (read-only), else systemd-timesyncd's marker | Sync state only, not a measured offset. |
+| Reboot required | `/run/reboot-required` | Debian and Ubuntu only; other distros show *unknown*. |
+
+### Optional system helper
+
+Process names, kernel error lines and crash-versus-clean shutdown detection need
+the kernel log and the login records, which the collector deliberately can't
+read. A separate container does:
+
+```bash
+docker compose --profile system-events up -d # add -f docker-compose.server.yml on a server
+```
+
+It runs with a single capability (`CAP_SYSLOG`), no network and a read-only
+filesystem, sees only `/var/log/wtmp` (read-only), and writes a sanitised JSON
+summary that the collector reads from a shared volume. With it you also get:
+
+- the OOM-killed process name on each **OOM kill** event,
+- recent kernel error lines in the **System** row (informational, they don't open incidents),
+- **Last shutdown: clean / unclean**, and a **critical** incident after an unclean reboot.
+
+Without it those show as *unknown* and everything else keeps working. Read
+[SECURITY.md](../SECURITY.md) before enabling it; the helper is about 200 lines of
+standard-library Python.
+
+<ui - The System row opened on a server with the helper running: status chips on top, then a list with an OOM kill (process name) and kernel error lines>
diff --git a/docs/database.md b/docs/database.md
new file mode 100644
index 0000000..6048c29
--- /dev/null
+++ b/docs/database.md
@@ -0,0 +1,31 @@
+[← Documentation](README.md)
+
+# Database schema
+
+Tables are created on backend startup (`Base.metadata.create_all`), and new
+columns are added with `ALTER TABLE ... IF NOT EXISTS` in the startup hook in
+`src/backend/main.py`. Each check-in stores one row in `host_metrics` (which also
+carries the boot ID, OOM counter, kernel flags, clock state and last shutdown), one
+per filesystem in `mount_metrics` and one per container in `container_metrics` (which
+also records the exit code, out-of-memory flag, finish time and image ID), all with the
+same timestamp. `system_events` holds reboots, OOM kills, kernel messages, container
+changes (appeared, removed, image changed) and network changes (ports opened or closed,
+firewall changes); `listener_states` and `firewall_states` hold what was last seen, `join_tokens` the hashed single-use tokens, `recovery_codes` the hashed two-factor
+recovery codes (the encrypted two-factor secret lives on the `users` row), and per-mount disk
+thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises
+the `users` table on a fresh volume.
+
+**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in
+samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container
+per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.
+Each tier is built from the finer one when its period ends, a period without check-ins gets no row, and the
+purge never deletes samples that have not been summarised yet. The Statistics page reads them (see [Statistics](statistics.md)). They are
+not purged yet; a retention setting for each level is planned.
+
+The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and
+`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are
+migrated at start: the new indexes are created and the redundant single-column ones are dropped,
+which can be repeated safely. Creating an index briefly blocks writes to that table (a few
+seconds for a table of millions of rows); on a very large install, create the indexes yourself
+first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names
+in `src/backend/models.py`, and the start-up migration will skip them.
diff --git a/docs/deployment.md b/docs/deployment.md
new file mode 100644
index 0000000..81ed7c3
--- /dev/null
+++ b/docs/deployment.md
@@ -0,0 +1,136 @@
+[← Documentation](README.md)
+
+# Deploying on a server (behind Traefik)
+
+There are two Compose files:
+
+| File | Use | Differences |
+|---|---|---|
+| `compose.yml` | Laptop / development (the default when you run `docker compose` without `-f`) | Dashboard on port 5173 on all interfaces; plain HTTP; optional Caddy (`--profile https`). |
+| `docker-compose.server.yml` | Server behind an existing Traefik | Traefik labels and network; HTTPS-only session cookie; only the first account can register; port 5173 bound to `127.0.0.1`; no Caddy; **collector endpoints removed from the public side** and served by a separate collector gate on your VPN address. |
+
+## 1. Create `.env` on the server
+
+Nothing in `docker-compose.server.yml` needs editing: everything specific to you comes from
+`.env`, so `git pull` can always update the file. Start from the production template:
+
+```bash
+cp example.env .env
+```
+
+Then set at least these:
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `POSTGRES_PASSWORD` | *(required)* | A long random password for the bundled database. |
+| `DASHBOARD_DOMAIN` | *(required)* | Your domain, without `https://`. Used for the Traefik router. |
+| `TRAEFIK_ENTRYPOINT` | `websecure` | Your Traefik HTTPS entrypoint name. |
+| `TRAEFIK_CERT_RESOLVER` | `letsencrypt` | Your Traefik certificate resolver name. |
+| `TRAEFIK_NETWORK` | `traefik_default` | The Docker network Traefik is attached to. |
+| `TRAEFIK_ROUTER` | `homelab` | Name of this dashboard's Traefik router and service; change it only if it clashes with another one. |
+| `COMPOSE_FILE` | `docker-compose.server.yml` | Makes plain `docker compose ...` commands use the server file (see below). |
+
+Find the network with `docker network ls` or
+`docker inspect <traefik-container> --format '{{json .NetworkSettings.Networks}}'`.
+The entrypoint and resolver names are in Traefik's static configuration
+(`entryPoints:` / `certificatesResolvers:`).
+
+## 2. Use the right Compose file
+
+`COMPOSE_FILE=docker-compose.server.yml` in `.env` matters. Without it, Compose picks
+`compose.yml` (the development stack), and an `up` would replace the server containers
+with ones that have no Traefik labels and open registration.
+
+**Don't copy the server file under another name** (for example `docker-compose.yml`). If
+both `compose.yml` and `docker-compose.yml` exist, Compose silently uses `compose.yml`, and
+a copy won't receive updates. If you made a copy of an older version, move your values
+into `.env`, delete the copy, and continue with `docker-compose.server.yml`.
+
+## 3. Start it
+
+```bash
+docker compose -f docker-compose.server.yml up -d --build
+```
+
+Open `https://<your-domain>` and register your account right away. The first
+account needs the **setup token** (`docker compose logs backend | grep "setup token"`,
+or the value of `ADMIN_BOOTSTRAP_TOKEN` if you set one), so a stranger who finds a
+fresh deployment can't claim it; then registration closes. That first account is
+the admin. Traefik terminates HTTPS with its own certificates; the stack needs no
+certificate configuration.
+
+The public address serves only the dashboard. Collectors don't use it; set up
+their route in the next step.
+
+## 4. Let remote servers report over your VPN
+
+Collectors reach the **collector gate**, not the public domain. In `.env`:
+
+| Setting | Example | Meaning |
+|---|---|---|
+| `COLLECTOR_BIND` | `10.8.0.1` | Your VPN server's address on the dashboard machine (for OpenVPN, the `tun` address). The gate's port is published on this address only. |
+| `COLLECTOR_PORT` | `8081` | Port of the gate on that address. |
+| `COLLECTOR_PUBLIC_URL` | `http://10.8.0.1:8081/api` | What **Connect a server** tells new machines to use. |
+| `COLLECTOR_IMAGE`, `DOCKER_PROXY_IMAGE` | `ghcr.io/…@sha256:…` | The images remote machines install; see below. |
+
+Things to get right:
+
+- **Bind to the VPN address, not `0.0.0.0`.** Docker's published ports bypass
+ `ufw` and similar firewalls, so a "deny from outside" rule does not protect them.
+- If Docker starts before the VPN is up, the bind fails until the address exists.
+ `restart: unless-stopped` retries, or make Docker start after the VPN (for
+ example a systemd drop-in with `After=openvpn-server@<name>.service`).
+- Traffic inside the VPN is plain HTTP over the encrypted tunnel; collectors
+ authenticate with a per-server token that the dashboard stores only as a hash.
+
+### Publishing the collector images
+
+Remote machines install the collector and the Docker proxy from container images,
+pinned by digest, so no code is ever downloaded from the dashboard. The
+[Publish images](../.github/workflows/images.yml) workflow builds them (amd64 and
+arm64), signs them with cosign and prints their digests:
+
+```bash
+git tag v0.1.0 && git push origin v0.1.0
+```
+
+1. Open the workflow run's summary and copy the `COLLECTOR_IMAGE=…` and
+ `DOCKER_PROXY_IMAGE=…` lines into `.env`.
+2. If the repository is private, make the three packages **public** (GitHub →
+ Packages → *Package settings* → *Change visibility*); otherwise machines can't
+ pull them without credentials.
+3. Rebuild the stack so the backend knows the images:
+ `docker compose -f docker-compose.server.yml up -d --build`.
+
+**Connect a server** won't generate a command until both images are set.
+
+## Updating
+
+```bash
+git pull
+docker compose -f docker-compose.server.yml up -d --build
+```
+
+Always rebuild **all** services (don't name individual ones). The backend,
+frontend, collector, Docker proxy and system helper each have their code built
+into their image, so a service that isn't rebuilt keeps running the old version.
+The database schema is updated automatically when the backend starts.
+
+Collectors on other machines run the image digest from the command you generated.
+To update them, publish a new version tag, put the new digests in `.env`, and run a
+freshly generated command on each machine (or pull the new digests into its
+compose project). Collectors started from `collector-compose.yml` need
+`docker compose -f collector-compose.yml up -d --build` after a `git pull`.
+
+<diagram - What each step updates: git pull → source on disk; docker compose up -d --build → backend, frontend, collector, docker-proxy and gate containers rebuilt and recreated; git tag → Publish images workflow → new digests → .env and a fresh Connect a server command for remote machines; show that skipping --build leaves the old image running>
+
+## Without Traefik
+
+- **Caddy (automatic Let's Encrypt):** with `compose.yml`, set
+ `DOMAIN`, `SESSION_COOKIE_SECURE=true` and `FRONTEND_BIND=127.0.0.1` in
+ `.env`, open ports 80/443, and run `docker compose --profile https up -d --build`.
+- **Private, over a VPN:** install Tailscale, WireGuard or OpenVPN on the server
+ and your devices, run `docker compose up -d --build`, and use `http://<vpn-ip>:5173`
+ (collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly. This uses
+ the development stack's single listener; for a public dashboard with collectors
+ on a VPN, use the server stack above.
diff --git a/docs/development.md b/docs/development.md
new file mode 100644
index 0000000..3c2a493
--- /dev/null
+++ b/docs/development.md
@@ -0,0 +1,97 @@
+[← Documentation](README.md)
+
+# Development
+
+# Development
+
+## Backend without Docker
+
+Python 3.12. The backend needs Redis for sessions and rate limits; SQLite works
+in place of PostgreSQL.
+
+```bash
+python -m venv venv && source venv/bin/activate
+pip install -r requirements.txt
+docker run -d --name homelab-dev-redis -p 6379:6379 redis:7-alpine # if Redis isn't running
+DATABASE_URL=sqlite:///./dev.db REDIS_URL=redis://localhost:6379/0 \
+ uvicorn src.backend.main:app --reload --host 0.0.0.0 --port 8060
+```
+
+## Frontend dev server
+
+Node 20 (`.nvmrc`). The Vite dev server proxies `/api` to `localhost:8060` (set `BACKEND_URL` to point it elsewhere),
+matching the production nginx setup.
+
+```bash
+nvm install && nvm use
+cd src/frontend
+npm install
+npm run dev -- --host 0.0.0.0
+```
+
+Vite prints the URL (5173, or the next free port if the Compose stack already
+uses it). `./start_frontend.sh` does the same in one step.
+
+
+After changing frontend or collector code, rebuild the containers:
+`docker compose up -d --build`. Otherwise the browser keeps getting the old bundle.
+
+## Tests
+
+```bash
+source venv/bin/activate
+pip install -r requirements-dev.txt # the app's requirements plus pytest
+pytest
+```
+
+No Docker, PostgreSQL or Redis is needed; the suite takes about a minute and a half. The chart
+helpers have their own tests that need only Node 20 or newer: `cd src/frontend && node --test`. See
+[TESTS.md](../TESTS.md) for how the tests work, what they don't cover, and what every
+test is for.
+
+## Project layout
+
+```text
+compose.yml development stack (default for `docker compose`)
+docker-compose.server.yml server stack behind Traefik
+collector-compose.yml collector-only stack for monitored machines
+README.md what it is and how to start it (the short version)
+docs/ the full documentation (start at docs/README.md)
+SECURITY.md what each component can touch, and its limits
+TESTS.md how to run the tests and what each one checks
+requirements-dev.txt app requirements plus pytest, for running the tests
+pytest.ini test settings (lets plain `pytest` find the `src` package)
+.github/workflows/
+ images.yml builds, signs and publishes the collector images on a version tag
+docker/
+ backend.Dockerfile FastAPI backend (uvicorn)
+ frontend.Dockerfile Vite build served by nginx
+ collector.Dockerfile standalone collector image
+ docker-proxy.Dockerfile read-only Docker API proxy image
+ system-helper.Dockerfile optional kernel-log / wtmp helper image
+ nginx.conf development: serves the UI, proxies /api to the backend
+ nginx.strict.conf server stack: public listener, collector endpoints refused
+ nginx.collector.conf server stack: collector gate (VPN and internal listeners)
+ Caddyfile optional HTTPS front door (dev compose, --profile https)
+ postgres/init/ SQL run on a fresh PostgreSQL volume
+src/
+ backend/ API: auth, monitoring, enrollment, rate limits, models
+ alerts.py alert engine: one incident per rule, auto-resolve
+ rules.py alert rules and default thresholds
+ rollups.py summarises check-ins into hourly/daily/weekly/monthly rollups
+ history.py range-aware history behind the Statistics page
+ stats.py fleet statistics: incidents over time, availability, busy-hours heatmap
+ totp.py one-time codes and recovery codes (standard library only)
+ secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key
+ reset_2fa.py command-line two-factor reset for a locked-out account
+ agent/
+ collector.py metrics collector (standard library only)
+ docker_proxy.py read-only Docker API proxy (standard library only)
+ system_helper.py optional kernel-log / wtmp helper (standard library only)
+ health_checks.py read-only checks available to the assistant
+ service.py, prompts.py Gemini assistant
+ frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the
+ charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,
+ FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the page
+tests/ pytest suite
+```
diff --git a/docs/metrics.md b/docs/metrics.md
new file mode 100644
index 0000000..38fd909
--- /dev/null
+++ b/docs/metrics.md
@@ -0,0 +1,187 @@
+[← Documentation](README.md)
+
+# Metrics: what every number means
+
+This page explains each value the dashboard and the [Statistics](statistics.md) page show: what it
+is, where it comes from, how it is calculated, and what it does **not** tell you. Where two values are
+easy to mix up (CPU % and load, memory "used" and "free") the difference is spelled out.
+
+- [How values are collected](#how-values-are-collected)
+- [Host metrics](#host-metrics): CPU, load, memory, swap, disk, inodes, network, uptime
+- [Container metrics](#container-metrics)
+- [System state](#system-state)
+- [Network checks](#network-checks)
+- [Collector self-metrics](#collector-self-metrics)
+- [How history is stored](#how-history-is-stored): what "average" and "highest" mean over a week
+- [Derived statistics](#derived-statistics): availability, distributions, busy hours, "full in about N days"
+- [Units](#units)
+
+## How values are collected
+
+Each server's collector reads the host's `/proc` and filesystem and the Docker API (through the
+read-only proxy) once per **check-in**. The default interval is 60 seconds and it can be set per server
+between 15 seconds and 15 minutes. Everything on this page is therefore a **snapshot taken at a
+check-in**, not a continuous recording. Between two check-ins nothing is measured.
+
+- The time stored with a check-in is when the dashboard received it (UTC), not the collector's clock.
+- A value the collector could not read is stored as *unknown*, never as zero. Unknown values are left
+ out of averages, never raise alerts, and show as "–" or *unknown* in the interface.
+- The backend validates every number it receives (type, range, length) before storing it.
+
+## Host metrics
+
+| Metric | What it means | How it is measured | Range |
+|---|---|---|---|
+| **CPU usage** (`cpu_utilization_pct`) | The share of CPU time the processors spent working, across **all cores** together. 100% means every core was fully busy. | Two readings of the aggregate line in `/proc/stat`, **about 0.15 seconds apart**, at each check-in. Time spent idle, and waiting for I/O, counts as not busy. | 0 – 100 % |
+| **Load average** (`load_1m`, `load_5m`, `load_15m`) | How many tasks, on average, wanted to run or were stuck waiting (see below). It measures **demand**, not utilisation. | The kernel's own load average, read with `getloadavg()`: averaged by the kernel over the last 1, 5 and 15 minutes. | 0 and up, not capped |
+| **Memory used** (`ram_pct`, `ram_used_gb`) | How much RAM programs are actually holding. | `MemTotal − MemAvailable` from `/proc/meminfo`, as a share of `MemTotal`. File caches the kernel can free at once count as **available**, so a Linux machine showing "full" RAM in other tools is usually fine here. | 0 – 100 % |
+| **Swap used** (`swap_pct`) | How much of the swap space is in use. | `SwapTotal − SwapFree` as a share of `SwapTotal`. A host without swap reports 0. | 0 – 100 % |
+| **Root disk used** (`disk_usage_pct`) | How full the root filesystem is. | `statvfs` of the host root, calculated like `df`: used ÷ (used + space available to non-root users). Reserved blocks are not counted as free. | 0 – 100 % |
+| **Inodes used** (`disk_inodes_pct`) | How many files the root filesystem can still hold. A disk can have free space and no free inodes. | `statvfs` file counts: (total − free) ÷ total. Unknown on filesystems that don't report inodes. | 0 – 100 % |
+| **Network sent / received** (`net_bytes_sent_mb`, `net_bytes_recv_mb`) | **Running totals** of bytes sent and received since the machine booted. They are not speeds. | Sum over the host's **physical** interfaces from `/proc/1/net/dev`. Virtual interfaces (loopback, Docker bridges, `veth`, VPN tunnels) are left out because their traffic is already counted on the physical one. | MiB, grows until reboot |
+| **Uptime** (`system_uptime_seconds`) | Seconds since boot. | `/proc/uptime`. | seconds |
+
+### CPU % and load are different things
+
+| | CPU usage % | Load average |
+|---|---|---|
+| Question it answers | "How busy are the processors?" | "How many tasks are waiting for them (or for I/O)?" |
+| Scale | 0–100% of the whole machine | A count of tasks; compare with the **number of cores** |
+| Counts waiting tasks | No | **Yes**: tasks that are runnable, and tasks in uninterruptible sleep (usually waiting for disk or NFS) |
+| Time window | A ~0.15 s snapshot per check-in | A kernel-maintained average over 1, 5 or 15 minutes |
+
+How to read them together:
+
+- **Load around the number of cores** is a fully used machine; **well above it** means tasks are queueing.
+ Load 4 on a 4-core machine is "full"; the same 4 on a 1-core machine is badly overloaded. That is why
+ the alert rule divides by the core count.
+- **Low CPU %, high load** usually means tasks are blocked on disk, network storage or something else
+ slow. The processors are idle while work waits.
+- **High CPU %, load near the core count** is a busy machine that is keeping up.
+
+Which load the interface shows: the **Statistics** chart draws the **1-minute** load as a raw number (not
+divided by cores), and the **alert rule** uses the **5-minute** load **per core** (warning at 1.5, critical
+at 2.0 by default). The 1-minute figure reacts quickly and is noisy; the 5-minute figure is steadier.
+
+### Reading the CPU figure honestly
+
+Because CPU % is measured over about 0.15 seconds at each check-in, one value is a snapshot of that
+moment. With a 60-second interval, each value represents roughly a quarter of a percent of the time.
+A busy period that starts and ends between two check-ins can be missed completely, while a spike that
+happens to coincide with a check-in is recorded in full. Averages over many check-ins (an hour, a day)
+are a good picture of typical usage; a single point is not. The **load average**, which the kernel keeps
+continuously, does not have this blind spot, so compare the two when something looks odd. (Measuring
+CPU over the whole interval between check-ins would remove the limitation; it is on the list of things
+to improve.)
+
+### Filesystems
+
+Disk and inode usage are reported **per filesystem**, in addition to the root figures above
+(`mount_metrics`):
+
+| Value | Meaning |
+|---|---|
+| `mount` | The path where the filesystem is mounted, for example `/` or `/media/data`. |
+| `fs_type` | `ext4`, `xfs`, `btrfs`, `zfs`, `nfs`, … |
+| `total_gb`, `used_gb` | Size and amount used, in GiB. |
+| `used_pct` | Calculated like `df` (used ÷ (used + available to non-root)), the same as the root disk figure. |
+| `inodes_pct` | Inodes in use, where the filesystem reports them. |
+
+Included: real block devices and the network or pool filesystems `zfs`, `nfs`, `nfs4`, `cifs`, `smb3`,
+`fuse.mergerfs` and `fuse.sshfs`. Left out: `tmpfs`, `devtmpfs`, `overlay`, `squashfs`, and anything under
+`/proc`, `/sys`, `/dev`, `/run`, `/snap`, `/var/snap`, `/var/lib/docker`, `/var/lib/containerd` and
+`/var/lib/kubelet`. A device mounted in several places is reported once. A disk mounted after the
+collector started appears after the collector restarts.
+
+## Container metrics
+
+Taken from Docker's stats and inspect output for every container at each check-in. Only containers that
+are **running** report usage figures; for a stopped container they are unknown, not zero.
+
+| Metric | What it means | How it is measured |
+|---|---|---|
+| **State** (`status`) | `running`, `exited`, `restarting`, `paused`, `created`, `dead`. | Docker inspect. |
+| **Health** (`healthcheck_status`) | `healthy`, `unhealthy`, `starting`, or none if the image defines no healthcheck. | Docker inspect. |
+| **Restarts** (`restart_count`) | How many times Docker has restarted it. | Docker's counter; it starts again at 0 when the container is recreated. |
+| **CPU %** (`cpu_pct`) | How much CPU the container used, where **100% is one full core**. A container using three cores' worth shows about 300%. | The change in the container's CPU time divided by the change in the whole machine's CPU time since the **previous check-in**, times the number of cores. The first check-in after the collector starts has no figure (unknown), and a container that used no CPU in the interval shows none. |
+| **Memory used** (`memory_used_mb`) | Memory the container is really holding. | Docker's usage minus its reclaimable page cache (`inactive_file`), the same way `docker stats` does it. |
+| **Memory limit / %** (`memory_limit_mb`, `memory_pct`) | The limit set for the container and how much of it is used. | No limit set means no percentage. |
+| **Network in / out** (`net_rx_mb`, `net_tx_mb`) | Running totals since the container started. Not speeds. | Sum over the container's interfaces. |
+| **Disk read / write** (`block_read_mb`, `block_write_mb`) | Running totals since the container started. | Docker's block I/O counters. |
+| **Processes** (`pids`) | How many processes are running inside it. | Docker's pids count. |
+| **Exit code, out of memory, finished at** | How a stopped container ended. Exit 137 usually means it was killed (by `docker stop`/`kill`, or by the out-of-memory killer). | Docker inspect. |
+| **Image ID** | The exact image build it is running, used to notice a changed image. | Docker inspect. |
+
+## System state
+
+Read from what the kernel exposes, without reading any log files.
+
+| Value | Meaning | Source |
+|---|---|---|
+| **Boot ID** | A new value between two check-ins means the machine rebooted. | `/proc/sys/kernel/random/boot_id` |
+| **OOM kills** | How many processes the kernel's out-of-memory killer has stopped since boot. A rising counter records an event. | `oom_kill` in `/proc/vmstat` |
+| **Kernel fault flags** | Only fault bits count (machine check, bad page, oops, warning, soft lockup). Flags for proprietary or out-of-tree modules are ignored. They stay set until reboot. | `/proc/sys/kernel/tainted` |
+| **Clock synchronised** | Whether a time service is keeping the clock disciplined. This is the sync state, **not** a measured offset. | `adjtimex` (read-only), or systemd-timesyncd's marker |
+| **Reboot required** | Updates are waiting for a restart. Debian and Ubuntu only; other systems show *unknown*. | `/run/reboot-required` |
+| **Last shutdown** (clean / unclean) | Whether the previous shutdown was orderly. Needs the optional [system helper](dashboard.md#optional-system-helper). | `wtmp` via the helper |
+| **Kernel error lines, OOM-killed process name** | Informational. Need the system helper. | Kernel log via the helper |
+
+## Network checks
+
+Passive, and only what the collector can read without privileges.
+
+| Value | Meaning |
+|---|---|
+| **Listening ports** | Which TCP/UDP ports the host listens on, and whether each is reachable from other machines or only from the machine itself (**scope**). Protocol, port and scope only, with no process names. Short-lived programs on random high ports are ignored. |
+| **Firewall** | A hash and a rule count of the known firewall configuration files, plus whether `ufw` is enabled. Never the rules themselves, and not the live ruleset. A changed hash raises an informational incident. |
+| **Internet reachability** | Off unless you set `HOMELAB_CONNECTIVITY_TARGETS` on that collector. `ok` (all targets reachable), `degraded` (some), `dns_failed` (names don't resolve and nothing else is wrong) or `down`; plus the median connection time in milliseconds. |
+
+## Collector self-metrics
+
+| Metric | Meaning |
+|---|---|
+| **Docker response time** (`docker_latency_ms`) | How long it took to list the containers. A rising value means the Docker daemon is struggling. Unknown when Docker isn't reachable. |
+| **Docker status** | `ok`, `unreachable` or `not_configured`, with a coarse reason (timeout, refused, missing, denied, error). Only a host that had Docker working before can raise "Docker unreachable". |
+| **Collection time** (`collect_duration_ms`) | How long the collector took to gather everything for one check-in. If it approaches the check interval the collector can't keep up, and a warning opens (80% by default). |
+
+## How history is stored
+
+The check-in snapshots are kept for the retention period (30 days by default). A background job also
+summarises them into one row per **hour, day, week (Monday to Sunday) and month**, all in UTC, for
+every server, filesystem and container. Each row stores, for every metric:
+
+- the **average** over the check-ins in that period,
+- the **highest value** (maximum) seen in a check-in,
+- the number of check-ins that reported it (so averages stay exact when some check-ins lack a value).
+
+What that means on the Statistics page:
+
+- A line is the **average of the snapshots** in each bucket. For CPU % that is the average of
+ ~0.15-second snapshots, as explained above.
+- The lighter band above the line reaches the **highest snapshot** in the bucket, so a spike that a
+ check-in caught stays visible even in a yearly view. It is the highest *recorded* value, not
+ necessarily the highest value that ever occurred.
+- A period with no check-ins has no point: a gap stays a gap and is never filled in.
+- Longer ranges use coarser rows, so a point can represent an hour, a day, a week or a month. The note
+ under the cards says which. Day, week and month boundaries are in UTC.
+
+See [Database schema](database.md) for the tables and [Statistics](statistics.md) for which range uses which
+level.
+
+## Derived statistics
+
+| Figure | How it is worked out | Caveats |
+|---|---|---|
+| **Availability** | The share of the check-ins a server should have made in the range that actually arrived: check-ins received ÷ (time in range ÷ check interval), capped at 100%. The range starts at the server's first check-in. | Approximate: it assumes the server's *current* check interval for the whole range. A server that never reported is unknown, not 0%. |
+| **Incidents opened / open now** | Counted from the incident log by when each incident opened, grouped by severity and by kind of problem. | Counting is capped at 20,000 incidents per request; the answer says when it was cut. |
+| **Mean time to resolve** | The average time between an incident opening and resolving, over incidents resolved in the range. | Incidents still open are not included. |
+| **Distribution (histogram)** | The share of the time a metric's average was in each 10% band, weighted by the number of check-ins behind each point. | Worked out in the browser from the points shown, so for long ranges it is a distribution of *averages* and looks calmer than the raw data. |
+| **Busy hours (heatmap)** | The average of CPU or memory for each hour of the day and weekday over the last 30 days, in your time zone (a fixed offset; daylight-saving changes inside the 30 days are ignored). | Needs a few days of history to fill in. Independent of the selected range. |
+| **"Full in about N days"** | A straight line fitted through a filesystem's usage over the range, extended to 100%. | Shown only for a **clear** rising trend: at least 6 points spanning at least a day, rising, and the line must fit the data reasonably (r² ≥ 0.5). Otherwise: "No clear growth trend". It assumes growth continues at the same rate; a cleanup or a one-off copy changes it. |
+| **Top containers / servers** | The containers using the most memory or CPU, and servers by disk usage, from the latest check-in. | Current values, not an average over the range. |
+
+## Units
+
+- **GB and GiB.** Sizes labelled "GB" are **GiB** (1 GB = 1024³ bytes), the unit `df -h` uses. "MB" values are MiB (1024² bytes).
+- **Percentages** are rounded to two decimals when stored and usually shown with one.
+- **Times** are UTC when stored. Charts for up to a day or two use your browser's clock; day, week and month points are labelled in UTC.
diff --git a/docs/statistics.md b/docs/statistics.md
new file mode 100644
index 0000000..7ef09d1
--- /dev/null
+++ b/docs/statistics.md
@@ -0,0 +1,53 @@
+[← Documentation](README.md)
+
+# Statistics
+
+For what each number means and how it is measured (including the limits of the CPU figure), see [Metrics](metrics.md).
+
+**Statistics** in the navigation shows how one server behaved over time. Pick a server and a
+range (1 hour, 6 hours, 24 hours, 7 days, 30 days, 90 days, 1 year or all time) and you get
+CPU, memory, swap, load and root disk charts. Each chart is the average as a line with a
+lighter band up to the highest value, so a short spike stays visible even in a yearly view.
+Hover or touch a point for the exact time, average, highest value and number of check-ins, or
+use **View as table**. Gaps (a server that was offline) are drawn as gaps. A note under the
+cards says what each point is (for example "one per hour, from hourly summaries") and when
+the history starts. The selection is kept in the address, so a view can be bookmarked, and
+the range is remembered per browser. The short ranges refresh every 30 seconds, except
+while the tab is hidden.
+
+Under the charts, **Containers** shows every container of that server as a small card (status
+dot, health, restarts, CPU, memory against its limit), with search and sorting (problems first
+by default). Select a card for that container's CPU and memory over the chosen range, with its
+memory limit drawn as a dashed line, how much of the time it was not running, and how many
+times it restarted in the range. A container only reports CPU and memory while it runs.
+
+Further down a server's page: how its CPU and memory use were **distributed** (the share of the
+time in each 10% band), **when it is busy** (average CPU or memory by hour of the day and weekday
+over the last 30 days, in your time zone), **storage** (a chart per filesystem with an estimate
+such as "full in about 3 days" when there is a clear growth trend, and none otherwise), and
+**reliability** (how long Docker takes to list containers, and how long the collector takes).
+
+Choose **All servers** for the whole fleet: containers by state and open incidents by severity
+(donuts), incidents opened over time by severity (stacked columns), what the incidents were
+about, servers by disk usage, the containers using the most memory and CPU right now, and each
+server's availability. Availability is the share of expected check-ins that arrived; it is
+approximate because it assumes each server's current check interval. Every chart can be shown
+as a table.
+
+Short ranges come from the stored check-ins, longer ones from the summaries described under
+[Database schema](database.md), so a year of history costs about as much to load as an
+hour.
+
+What to know about the statistics:
+
+- **Time zones.** Charts for up to a day or two use your clock; day, week (Monday to Sunday) and
+ month points are aligned to UTC and labelled in UTC, because that is how they are summarised.
+- **Some figures are approximate**: distributions and the days-until-full estimate are worked out
+ in the browser from the points you are looking at, availability assumes each server's current
+ check interval, and the top containers show values from the latest check-in, not an average
+ over the range. The page says so where it matters.
+- **Pointer-only details.** Hover and touch tooltips don't work from the keyboard yet; every chart
+ has **View as table** as an alternative.
+- **Not built yet** (tracked in Milestone 8 on GitHub): CSV and JSON export, reboots per month, a
+ state timeline and restarts per day for containers, rankings over the chosen range, network
+ traffic charts, event markers on charts, and retention settings for the summaries.
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
new file mode 100644
index 0000000..555f57a
--- /dev/null
+++ b/docs/troubleshooting.md
@@ -0,0 +1,29 @@
+[← Documentation](README.md)
+
+# Troubleshooting
+
+| Symptom | Check |
+|---|---|
+| A new machine never shows up under **New servers detected** | `docker logs homelab-collector` on that machine. "Could not reach …" means the address in the command isn't reachable from it (firewall, `localhost`, wrong IP/domain). |
+| Collector log says "Backend no longer accepts server …" | The server was deleted on the dashboard; it is asking to join again, so click **Add** or **Dismiss**. |
+| `network traefik_default declared as external, but could not be found` | You're using `docker-compose.server.yml` on a machine without that Traefik network. Use `compose.yml` for development, or set `TRAEFIK_NETWORK` in `.env` to the network Traefik is really on. |
+| Dashboard shows old behaviour after a code change or `git pull` | Rebuild **all** services with `docker compose up -d --build` (with `-f docker-compose.server.yml` on a server unless `COMPOSE_FILE` is set) and hard-refresh the browser (Ctrl+Shift+R). |
+| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec homelab-collector grep -c dashboard_host /app/collector.py` prints `0`. Rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). |
+| A server shows only `/` and no **Storage** bars for other disks | Its collector predates per-mount reporting. Install a newer collector image (run a freshly generated command, or `docker compose -f collector-compose.yml up -d --build`). A disk mounted after the collector started appears after a collector restart. |
+| Logged out after opening the dashboard at a new address | Login cookies belong to the address they were set on; log in again. |
+| Login fails behind HTTPS with `SESSION_COOKIE_SECURE=true` | The cookie is only sent over HTTPS; open the `https://` address, not `http://…:5173`. |
+| Collector log says "Registration refused (HTTP 403): The join token is invalid, expired or already used" | The token in the command has expired, was already used, or is wrong. Click **Generate command** again and run the new command. |
+| **Connect a server** says no collector images are configured | Set `COLLECTOR_IMAGE` and `DOCKER_PROXY_IMAGE` in `.env` (see [Publishing the collector images](deployment.md#publishing-the-collector-images)) and rebuild. |
+| The register page asks for a setup token | The install has no accounts yet. Use `docker compose logs backend \| grep "setup token"`, or `ADMIN_BOOTSTRAP_TOKEN`. |
+| "Password must be at least 16 characters long" | New passwords need 16 or more characters (at most 72 bytes). A few random words work well. |
+| "Cross-site request refused" (403) on a button or form | The request came from another origin. Open the dashboard at the address you normally use. If a proxy in front rewrites the `Host` header, add your dashboard URL to `CORS_ORIGINS`. |
+| The **System** row shows *unknown* for last shutdown or has no kernel messages | The optional system helper isn't running. Start it with `docker compose --profile system-events up -d`. *Unknown* for reboot-required on non-Debian hosts is expected. |
+| `Error response … cannot assign requested address` for the collector gate | `COLLECTOR_BIND` is set to an address that doesn't exist yet (VPN not up). Start the VPN first, or fix the address. |
+| The **Network** row says *no config found* for the firewall | None of the known firewall files exist on that host (`/etc/ufw`, `/etc/nftables.conf`, `/etc/iptables/rules.v4`). The check reads configuration, not the live ruleset, so a host that configures its firewall some other way shows this. |
+| The **Network** row says *not checked* for the internet | The check is opt-in. Set `HOMELAB_CONNECTIVITY_TARGETS` on that collector (for example `1.1.1.1:443`). |
+| Listeners show *unknown* or the Network row is empty | The collector couldn't read the host's network tables (older collector, or a table with more than 500,000 sockets). An unreadable report never raises a "port closed" alert. |
+| "Invalid code." although the code looks right | Codes depend on the clock: check the phone's time is automatic. A code also can't be used twice, so wait for the next one. Five wrong codes lock the code step for 5 minutes. |
+| Two-factor setup says it is not available | `SECRETS_KEY` is missing or invalid in `.env`. Generate one (see [Two-factor sign-in](accounts.md#two-factor-sign-in)) and restart the backend. The backend log shows a SECURITY WARNING while the key is missing. |
+| Lost the authenticator and the recovery codes | Run `docker compose exec backend python src/backend/reset_2fa.py <email>` on the host, then sign in with the password and set it up again. |
+| Authenticator codes stopped working after a restore or `.env` change | `SECRETS_KEY` changed or was lost, so the stored secret can't be decrypted. Restore the old key (rotation: `new,old`), or use a recovery code and the reset command. |
+| Server marked offline | It missed 3 check-ins in a row. Check the machine, its collector log and its route to the dashboard. The incident resolves on the next report. |
diff --git a/src/backend/main.py b/src/backend/main.py
index 34c9fcf..01112e8 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -507,7 +507,7 @@ def two_factor_status(session_id: Annotated[str | None, Cookie(alias=SESSION_COO
def two_factor_setup(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
_session, user = _account_session(session_id, db, allow_limited=True)
if not secrets_box.available():
- raise HTTPException(status_code=503, detail="Two-factor needs SECRETS_KEY to be configured on the server. See the README.")
+ raise HTTPException(status_code=503, detail="Two-factor needs SECRETS_KEY to be configured on the server. See docs/accounts.md (Two-factor sign-in).")
if user.totp_enabled:
raise HTTPException(status_code=409, detail="Two-factor is already turned on. Turn it off first to set it up again.")
enforce_limit(redis_client, scope="2fa-setup", subject=user.id, limit=10, window_seconds=3600)
diff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsx
index 99f9f2d..a17e05e 100644
--- a/src/frontend/src/App.jsx
+++ b/src/frontend/src/App.jsx
@@ -1744,7 +1744,7 @@ function TwoFactorPolicySetting() {
return (
<AdminSection title="Two-factor" description="Protects sign-in with a code from an authenticator app, on top of the password.">
<Toggle label="Require two-factor for administrators" description="An administrator without it can only reach the two-factor setup until it is done." checked={state?.required ?? false} onChange={change} disabled={state === null || saving} />
- {state && !state.configured && <p className="mt-3 rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-xs text-amber-200">SECRETS_KEY is not set on the server, so two-factor can't be set up and this is <strong>not being enforced</strong> yet. Set it in <code>.env</code> (see the README, Secrets at rest) and restart.</p>}
+ {state && !state.configured && <p className="mt-3 rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-xs text-amber-200">SECRETS_KEY is not set on the server, so two-factor can't be set up and this is <strong>not being enforced</strong> yet. Set it in <code>.env</code> (see docs/accounts.md, Two-factor sign-in) and restart.</p>}
{message && <p className={`mt-3 text-xs ${message.ok ? 'text-emerald-300' : 'text-amber-200'}`}>{message.text}</p>}
</AdminSection>
)
@@ -2360,7 +2360,7 @@ function TwoFactorSection({ status, reload, onEnrolled }) {
return (
<AdminSection title="Two-factor sign-in" description="A 6-digit code from an authenticator app (such as Aegis, Authy or Google Authenticator) is asked for after your password.">
- {!status.available && <p className="rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-sm text-amber-200">Two-factor can't be set up because the server has no <code>SECRETS_KEY</code>. An administrator needs to set one in <code>.env</code> (see the README, Secrets at rest).</p>}
+ {!status.available && <p className="rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-sm text-amber-200">Two-factor can't be set up because the server has no <code>SECRETS_KEY</code>. An administrator needs to set one in <code>.env</code> (see docs/accounts.md, Two-factor sign-in).</p>}
{status.available && codes && <RecoveryCodesPanel codes={codes} doneLabel={status.setup_required || onEnrolled ? 'Continue' : 'Done'} onDone={() => { setCodes(null); if (onEnrolled) onEnrolled() }} />}