Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat: added claude skills for easier security review
karloks200505cbda46b4cc
diff --git a/.claude/skills/security-audit/SKILL.md b/.claude/skills/security-audit/SKILL.mdnew file mode 100644index 0000000..889946f--- /dev/null+++ b/.claude/skills/security-audit/SKILL.md@@ -0,0 +1,167 @@+---+name: security-audit+description: Detaljan sigurnosni pregled (security review / audit) cijelog koda ili dijela koda, uz dojavu pronađenih propusta s ozbiljnošću, lokacijom, scenarijem napada i popravkom. Use when the user asks for a security review, security audit, vulnerability hunt, pentest-style code review, "provjeri sigurnost", "sigurnosni pregled", "ima li propusta", or wants a component (backend, collector, docker proxy, nginx, compose, CI, frontend, assistant) reviewed for security. For a quick check of only the pending diff, the built-in security-review skill is enough.+argument-hint: "[full | backend | collector | infra | frontend | assistant | <path> | diff]"+---++# Security audit++Cilj: pronaći **stvarne, iskoristive** sigurnosne propuste u ovom repozitoriju i dojaviti ih+tako da ih vlasnik može odmah popraviti. Projekt je open-source, self-hosted alat koji ima+pristup Docker socketu i hostu, pa vlasnik želi **strogu** procjenu: ne ublažavaj rizik+radi udobnosti, ne zaokružuj "vjerojatno OK" na "OK".++Ovaj skill **ne mijenja kod** i ne commitira. Samo pregledava i izvještava. Popravke radiš+tek kad korisnik to izričito zatraži.++## Argumenti++`$ARGUMENTS` određuje opseg. Bez argumenta = `full`.++| Argument | Opseg |+|---|---|+| `full` | sve dolje, redom |+| `backend` | `src/backend/**` |+| `collector` | `src/agent/**` (collector, docker_proxy, system_helper, assistant, health_checks) |+| `infra` | `docker/**`, `*compose*.yml`, `example.env`, `.github/**`, `.gitignore`, `.dockerignore` |+| `frontend` | `src/frontend/**` |+| `assistant` | LLM asistent: `src/agent/{service,prompts,health_checks}.py`, `SYSTEM_PROMPT.md`, konverzacije u backendu |+| `<path>` | taj put (datoteka ili direktorij) plus ono što ga poziva |+| `diff` | samo neprihvaćene promjene (`git diff` + `git diff --staged`) pročitane u kontekstu cijelih datoteka |++Ako je argument dvosmislen, odaberi najbliži opseg i na početku izvještaja reci koji si odabrao.++## Pravila rada (čitaj prije početka)++1. **Čitaj datoteke cijele.** `grep` služi za pronalaženje kandidata, nikad za oslobađanje+ koda od sumnje. Funkcija koja izgleda sigurno često ima rupu u pozivatelju ili u redoslijedu+ provjera. `src/backend/main.py` ima ~1800 redaka; pročitaj ga u cijelosti, u dijelovima.+2. **Ne ispisuj tajne.** `.env` postoji lokalno. Ako ga treba otvoriti, ispiši samo *imena*+ varijabli i je li vrijednost prazna/zadana, nikad vrijednosti. Isto vrijedi za tokene,+ ključeve i lozinke pronađene u kodu ili povijesti: u izvještaju navedi lokaciju i vrstu,+ ne sadržaj (najviše prva 4 znaka).+3. **Samo lokalno i neškodljivo.** Ne šalji zahtjeve prema stvarnim sustavima, ne pokreći+ Docker kontejnere ni skeniranja. Za potvrdu propusta smiješ pokrenuti postojeće testove+ (`pytest`) i napisati kratku skriptu u scratchpad direktoriju koja poziva funkcije+ izravno s neprijateljskim ulazom. Ništa što pokreneš ne smije dirati stvarnu bazu ili+ mrežu.+4. **Provjeri tvrdnje iz dokumentacije.** `SECURITY.md` i `docs/` tvrde mnogo ("rate limited",+ "single-use", "capped", "never follows symlinks"). Za svaku tvrdnju u svom opsegu nađi kod+ koji to provodi i provjeri da vrijedi na *svim* putevima. Tvrdnja bez pokrića u kodu je+ nalaz (dokumentirana kontrola ne postoji), a kontrola koja postoji samo na jednom putu+ također.+5. **Razmišljaj kao napadač, ne kao lintar.** Za svaki ulaz pitaj: tko ga može poslati, što je+ najgore što može sadržavati, što sustav radi s tim i što je najgore što se zbog toga može+ dogoditi. Profile napadača vidi u [references/threat-model.md](references/threat-model.md).+6. **Prvo provjeri je li nešto već riješeno.** Prije nego prijaviš nalaz, nađi postojeću+ kontrolu (middleware, dependency, nginx pravilo, test) i provjeri zaobilazi li se. Nalaz+ koji ignorira postojeću zaštitu je šum i troši korisnikovo povjerenje.+7. **Svaki nalaz verificiraj** (vidi niže) i razvrstaj kao `POTVRĐENO` ili `TREBA PROVJERU`.+ Ne prijavljuj ono što nisi mogao provjeriti kao potvrđeno.++## Postupak++### 1. Priprema (uvijek)++- Pročitaj [references/threat-model.md](references/threat-model.md) i `SECURITY.md`.+- `git status` i `git log --oneline -15`: što se nedavno mijenjalo, nove površine napada+ najčešće su u svježem kodu.+- Za `backend`/`full`: izlistaj sve rute (`grep -nE "^@(app|router)\.(get|post|put|patch|delete)" src/backend/main.py`)+ i složi **inventar ruta**: metoda, put, tko smije (anonimno / bilo koji korisnik / admin /+ kolektor s tokenom), je li iza CSRF/Origin provjere, ima li rate limit, što čita/piše.+ Ruta bez jasnog odgovora u bilo kojem stupcu je kandidat za nalaz.+- Pročitaj popis ovisnosti (`requirements*.txt`, `src/frontend/package.json`) radi kasnije+ provjere.++### 2. Pregled po područjima++Za svako područje u opsegu pročitaj odgovarajuću kontrolnu listu i prođi je **stavku po stavku**+nad stvarnim kodom:++| Područje | Kontrolna lista |+|---|---|+| Backend (API, auth, sesije, 2FA, baza, pravila, alarmi) | [references/checklist-backend.md](references/checklist-backend.md) |+| Collector, docker proxy, system helper, LLM asistent | [references/checklist-collector-agent.md](references/checklist-collector-agent.md) |+| Docker, compose, nginx, Postgres, CI, repozitorij, ovisnosti | [references/checklist-infra.md](references/checklist-infra.md) |+| Frontend | [references/checklist-frontend.md](references/checklist-frontend.md) |++Kontrolne liste su podsjetnici, **ne granica**. Ako vidiš nešto sumnjivo što nije na listi,+istraži i to.++**Paralelizacija za `full`:** opseg je velik. Ako ti je dostupan alat `Agent`, pokreni četiri+agenta paralelno (po jedno područje iz tablice), svakome daj samostalan prompt iz predloška+niže, a zatim sam spoji, deduplikaj i **ponovno verificiraj** sve što su vratili prije+izvještaja. Nalaz koji agent prijavi nije potvrđen dok ga ti nisi pročitao u kodu. Ako ne+možeš koristiti agente, prođi područja redom sam.++Predložak za agenta:++> Radiš sigurnosni pregled (samo čitanje, ništa ne mijenjaj, ne commitaj, ne ispisuj vrijednosti+> tajni) područja **<područje>** u repozitoriju <putanja>. Prvo pročitaj+> `.claude/skills/security-audit/references/threat-model.md` i+> `.claude/skills/security-audit/references/<checklist>.md`, zatim pročitaj **cijele** datoteke+> tog područja i prođi kontrolnu listu stavku po stavku. Za svaki nalaz vrati: naslov,+> datoteka:redak, što je pogrešno, konkretan scenarij napada (tko, što pošalje, što dobije),+> koju postojeću zaštitu si provjerio i zašto ne pomaže, predloženi popravak i svoju sigurnost+> (visoka/srednja/niska). Navedi i koje datoteke si pročitao, a koje nisi. Ne prijavljuj+> teorijske probleme bez puta do iskorištenja.++### 3. Verifikacija svakog nalaza++Za svaki kandidat napiši (barem u glavi, a u izvještaju sažeto):++1. **Izvor:** tko kontrolira ulaz i na kojem je povjerenju (anonimni, korisnik, admin,+ kolektor, VPN peer, podatak s hosta)?+2. **Put:** kojim funkcijama ulaz stiže do opasnog mjesta? Navedi datoteke i retke.+3. **Zaštite na putu:** koje provjere, validacije, limiti, nginx pravila postoje? Zašto ne+ pomažu (ili zašto pomažu, pa nalaza nema)?+4. **Učinak:** što napadač dobiva (čitanje podataka, promjena stanja, preuzimanje računa,+ izvršavanje koda, DoS, eskalacija na host)?+5. **Dokaz:** gdje je moguće, potvrdi testom ili kratkom lokalnom skriptom. Ako ne možeš,+ nalaz je `TREBA PROVJERU` s opisom što točno treba provjeriti.++Odbaci kandidata ako: nema puta do iskorištenja bez prethodnog admin pristupa koji već daje+isti učinak, ako je zaštićen nečim što si provjerio, ili ako je čista stilska preporuka.+Takve stvari, ako vrijede, idu u dio "Očvršćivanje" kratkom stavkom, ne među nalaze.++### 4. Ozbiljnost++Ocjenjuj prema učinku **u ovom sustavu** (pristup hostu i Docker API-ju je najskuplja stvar+koju treba braniti), uz prednost strožoj ocjeni kad si u dvojbi između dvije razine:++| Razina | Značenje | Primjeri |+|---|---|---|+| **KRITIČNO** | Daljinsko, bez/gotovo bez autentikacije, vodi do preuzimanja dashboarda, izvršavanja koda ili izlaska na host | zaobilaženje autentikacije, preuzimanje prve admin registracije, RCE, pristup Docker socketu iz javne strane |+| **VISOKO** | Autentificirani ili uvjetovani napad s velikim učinkom | eskalacija korisnik→admin, zaobilaženje 2FA, IDOR nad tuđim podacima, SQL injection, stored XSS u admin kontekstu, kompromitirani kolektor utječe na dashboard ili host |+| **SREDNJE** | Stvarno iskoristivo, ograničen učinak ili traži poseban uvjet | CSRF na manje osjetljivoj akciji, rate limit koji se zaobilazi, curenje informacija, DoS jednim zahtjevom |+| **NISKO** | Slabost bez jasnog puta do štete sam za sebe | nedostajući header, enumeracija korisnika, preslab zadani parametar |+| **INFO** | Zapažanje ili očvršćivanje | kontrola koja radi, ali je krhka |++### 5. Izvještaj++Format i redoslijed: [references/report-format.md](references/report-format.md). Ukratko:+sažetak (broj nalaza po razini, najvažniji jedan redak), **jedna tablica svih nalaza sortirana+prema riziku od najvišeg prema najnižem** (KRITIČNO, VISOKO, SREDNJE, NISKO, INFO), detalji+nalaza istim redom, tvrdnje iz dokumentacije koje kod ne podržava, očvršćivanje i pokrivenost+(što je pregledano, što nije i zašto).++**Tablica je obavezna.** Nalaze nikad ne prikazuj samo kao tekst niti grupirane po područjima.+Rezultate paralelnih agenata spoji u jednu tablicu i sortiraj ih zajedno.++Datoteke referenciraj kao markdown linkove (`[main.py:412](src/backend/main.py#L412)`), da su+klikabilne u editoru.++Izvještaj ispiši u razgovoru. **Ne spremaj ga u datoteku** (npr. `docs/`) dok to korisnik ne+zatraži, jer može sadržavati neobjavljene ranjivosti.++Na kraju ponudi sljedeći korak jednom rečenicom (npr. popraviti nalaze od najtežeg prema+najlakšem, uz test za svaki) i **čekaj odgovor**.++## Što ovaj skill nije++- Nije dinamičko testiranje: ne skenira živi sustav, ne fuzzira, ne pokreće exploite.+- Nije zamjena za `pip-audit` / `npm audit` ili skener slika; ako su dostupni i korisnik+ odobri mrežu, pokreni ih, a ako nisu, to **eksplicitno navedi** u pokrivenosti kao+ nepregledano (ovisnosti se ne ocjenjuju "iz glave" jer verzije nisu pouzdan izvor).+- Ne potvrđuje da je sustav siguran. Izvještaj "nema nalaza" smije reći samo "u pregledanom+ opsegu nisam našao iskoristive propuste", uz popis onoga što je pregledano.diff --git a/.claude/skills/security-audit/references/checklist-backend.md b/.claude/skills/security-audit/references/checklist-backend.mdnew file mode 100644index 0000000..d4fa8e4--- /dev/null+++ b/.claude/skills/security-audit/references/checklist-backend.md@@ -0,0 +1,140 @@+# Kontrolna lista: backend (`src/backend/**`)++Čitaj `main.py`, `auth.py`, `totp.py`, `secrets_box.py`, `rate_limit.py`, `rules.py`,+`models.py`, `db.py`, `config.py`, `history.py`, `rollups.py`, `stats.py`, `alerts.py`,+`seed_user.py`, `reset_2fa.py` u cijelosti. Za svaku stavku nađi kod koji je provodi.++## 1. Autentikacija i autorizacija (najveći prioritet)++- [ ] **Svaka ruta ima svjestan odgovor** na "tko smije?" (inventar ruta iz koraka 1). Ruta+ koja zaboravi `Depends(...)` za auth je zadano **otvorena**: traži takve.+- [ ] Razlika *anonimno / prijavljen / admin / kolektor* provodi se na serveru, nikad samo u+ frontendu. Admin-only rute stvarno provjeravaju ulogu (ne samo prijavu).+- [ ] **Ograničena sesija** (admin bez 2FA dok je politika uključena) stvarno može doći samo do+ profila, postavljanja 2FA i odjave. Provjeri *svaku* rutu, ne samo ono što test pokriva,+ uključujući nove rute dodane nakon te značajke.+- [ ] **IDOR / horizontalna autorizacija:** rute s ID-evima (server, kontejner, konverzacija,+ sesija, token, korisnik, pravilo, alarm) provjeravaju vlasništvo/ulogu. Posebno: popis+ sesija po javnim ID-evima, brisanje tuđe sesije, konverzacije drugog korisnika.+- [ ] **Mass assignment:** modeli zahtjeva ne dopuštaju polja kao `is_admin`, `role`,+ `require_2fa`, `server_id`, `owner_id`. Provjeri Pydantic modele i `**body`/`setattr`.+- [ ] Zadnji admin ne može obrisati/degradirati sam sebe do stanja bez admina (ili je to+ namjerno i dokumentirano). Admin ne može eskalirati drugog bez provjere.+- [ ] Prvi račun postaje admin: provjeri **sve** puteve (setup token, `ALLOW_REGISTRATION`,+ utrka dvaju istodobnih zahtjeva, `ADMIN_BOOTSTRAP_TOKEN` usporedba, brisanje tokena nakon+ prvog računa). Je li usporedba tokena konstantnog vremena (`hmac.compare_digest`)?++## 2. Sesije, kolačići, CSRF++- [ ] ID sesije: iz `secrets` (ne `random`/`uuid1`), dovoljna entropija, pohranjen hashiran ili+ barem nikad vraćen pregledniku u tijelu odgovora. Rotacija pri promjeni privilegija+ (prijava, 2FA, promjena lozinke).+- [ ] Zastavice kolačića: `HttpOnly`, `Secure` (i u kojim uvjetima se gasi: provjeri da+ zadana vrijednost nije nesigurna), `SameSite`, `Path`, istek.+- [ ] Istek i odjava stvarno brišu sesiju na serveru; logout-everywhere radi.+- [ ] **Provjera `Origin`/`Sec-Fetch-Site`** na POST/PUT/PATCH/DELETE: pokušaj zaobilaženja:+ `Origin` s istim prefiksom (`https://dash.example.com.evil.tld`), različit port/shema,+ `Origin: null`, nedostajući `Origin` + `Sec-Fetch-Site: same-site`, zaglavlje `Host`+ pod kontrolom napadača, `CORS_ORIGINS` s džokerima. Radi li provjera za **sve** metode+ i sve rute (middleware ili po ruti)? Postoji li izuzeće za puteve kolektora i može li ga+ se zloupotrijebiti da bi se zaobišla provjera na ruti korisnika?+- [ ] CORS: `allow_origins` nije `*` uz `allow_credentials=True`; nema refleksije proizvoljnog+ `Origin`.++## 3. Lozinke, 2FA, tokeni++- [ ] Hashiranje lozinke: bcrypt/argon2 s razumnim troškom; granica 72 B provodi se (odbija,+ ne skraćuje); nema lozinki u logovima; promjena lozinke traži staru lozinku (ili je+ sesija svježa) i poništava ostale sesije.+- [ ] **Timing/enumeracija računa:** prijava, reset, registracija vraćaju iste odgovore i+ slično vrijeme za nepostojeći email i pogrešnu lozinku (fiktivni hash za nepostojeće+ račune?).+- [ ] **Rate limit** (`rate_limit.py`): ključ (IP, email) ne može se zaobići promjenom+ zaglavlja (`X-Forwarded-For` iz nepouzdanog izvora), velikim/malim slovima ili+ razmacima u emailu, unicode normalizacijom; memorija ograničena (napadač ne može+ napuhati strukturu milijunima ključeva); rade li limiti pod više radnih procesa+ (stanje u memoriji po procesu = efektivni limit × broj procesa)?+- [ ] **TOTP** (`totp.py`): usporedba konstantnog vremena; prozor ±1; **zabrana ponovne+ uporabe istog koda je atomarna** (jedinstveni indeks/UPDATE s uvjetom, ne+ čitaj-pa-piši); brojač pokušaja po računu i po adresi ne resetira se ponovnim+ početkom lozinke; međutoken (5 min) je jednokratan, vezan uz korisnika i ne vrijedi+ kao sesija nigdje.+- [ ] **Recovery kodovi:** entropija, hashirani, jednokratni *atomarno*, brišu se/ponovno+ generiraju pri isključivanju 2FA.+- [ ] `secrets_box.py`: Fernet ključ iz okoline, ponašanje bez ključa ne smije ostaviti+ tajne u čistom tekstu; greška dekripcije ne curi detalje; ključ se ne logira.+- [ ] **Tokeni servera i join tokeni:** generirani iz `secrets`, hashirani (SHA-256 je u redu+ za tokene visoke entropije), usporedba konstantnog vremena ili pretraga po hashu,+ jednokratnost i istek **atomarni**, ograničenje otvorenih tokena (20) provodi se i pod+ paralelnim zahtjevima, brisanje servera poništava token.+- [ ] Ponovno uparivanje postojećeg stroja (isti machine ID) traži `replace: true` na+ serveru, ne samo u UI-ju; mogu li se vjerodajnice tuđeg servera prepisati bez toga?++## 4. Ulazi koje šalje kolektor (nepouzdani)++- [ ] Svaka ruta kolektora (register, poll, enroll, ingest, helper događaji) odbija se kad+ zahtjev nije došao kroz gate (`X-Collector-Via` / `COLLECTOR_ACCESS=strict`); provjeri+ da klijent ne može sam poslati to zaglavlje preko javne strane (nginx ga mora+ prebrisati/ukloniti, backend ga ne smije slijepo vjerovati ako je backend dostupan+ drugim kontejnerima).+- [ ] Autentikacija kolektora: token po serveru; kolektor jednog servera ne može pisati+ podatke drugog (`server_id` iz tokena, ne iz tijela zahtjeva!).+- [ ] **Validacija:** tipovi, duljine, rasponi, strogi parsing vremena, `sha256:`+64 hex,+ maks. 500 kontejnera, imena ≤128, ograničenja događaja po check-inu i po satu. Traži+ polja koja se spremaju bez čišćenja (nova polja dodana kasnije često zaobiđu).+- [ ] Veličina tijela zahtjeva ograničena (nginx `client_max_body_size` **i** Pydantic+ `max_length`), dubina/veličina JSON-a, broj elemenata lista.+- [ ] Idempotentnost/replay: ponovljeni check-in ne stvara duplicirane incidente ni neograničen+ rast tablica. Hostilni podaci ne mogu "utišati" alarme drugih servera.++## 5. Baza i injekcije++- [ ] Svaki `text(...)`, `execute(f"...")`, `.format`, `%` ili spajanje stringova u SQL-u:+ jesu li vrijednosti vezane parametrima? Dinamički `ORDER BY`, imena stupaca/metrika i+ `LIMIT` iz zahtjeva idu samo kroz **bijelu listu**.+- [ ] Imena kontejnera i putanje mountova ulaze u ključeve incidenata/cache ključeve: mogu+ li se sudarati ili ubrizgati separator (`:`, `|`, newline) i spojiti dva incidenta?+- [ ] LIKE upiti s korisničkim unosom (escape `%`/`_`), regex iz korisničkog unosa (ReDoS;+ `rules.py` ima 593 retka pravila, provjeri koje izraze evaluira i je li to sigurno).+- [ ] **`rules.py`: izvršava li se išta dinamički** (`eval`, `exec`, `compile`, `getattr` s+ korisničkim imenom, `importlib`, šabloni stringova s korisničkim formatom)? Ako se+ pravila/alarmi konfiguriraju korisnički, što sve konfiguracija smije izraziti?+- [ ] Ograničenja stranica/`limit`, broj redaka, vremenski rasponi na `history`/`stats`+ (fiksna lista raspona i metrika, ograničen cache, ključevi iz validiranih vrijednosti).+- [ ] Transakcije: nema stanja u kojem pad usred operacije ostavlja račun bez 2FA ili sesiju+ koja preživljava promjenu lozinke.++## 6. Izlazne veze (SSRF) i integracije++- [ ] Webhook/alarm URL-ovi (`ADMIN_ALERT_WEBHOOK_URL`, kanali u `alerts.py`/`rules.py`):+ tko ih može postaviti (samo env ili i API)? Ako ih postavlja korisnik/admin preko API-ja:+ shema samo http(s), blokiranje loopback/link-local/privatnih adresa i metadata IP-a,+ provjera nakon DNS razrješenja, bez praćenja preusmjeravanja na unutarnje adrese, timeout.+- [ ] Sadržaj odlazne poruke ne sadrži tajne (tokene, hashove) ni neocišćene tuđe stringove u+ kontekstu gdje se interpretiraju (Slack/markdown/HTML/email zaglavlja → injekcija).+- [ ] Nema `requests`/`httpx` poziva prema URL-u iz zahtjeva bez gornjih zaštita.++## 7. Konfiguracija, tajne, logiranje, greške++- [ ] `config.py`/okolina: nesigurne zadane vrijednosti (prazan `SECRETS_KEY`,+ `ALLOW_REGISTRATION=true`, `COLLECTOR_ACCESS` ≠ strict, `DEBUG`, `CORS_ORIGINS=*`).+ Zadano mora biti sigurno, a nesigurno mora zahtijevati namjernu promjenu.+- [ ] Tajne se ne ispisuju u logove, iznimke ni odgovore (setup token se ispisuje **namjerno**+ pri startu dok nema računa; provjeri da se ne ispisuje poslije). Nema stack traceova ni+ SQL-a u HTTP odgovorima; nema `/docs`/`/openapi.json` izloženih na javnoj strani ako to+ nije namjera (provjeri `FastAPI(docs_url=...)` i nginx).+- [ ] **Log injection:** korisnički i kolektorski stringovi u logovima s novim redom/ANSI+ sekvencama.+- [ ] `seed_user.py` i `reset_2fa.py`: bez ugrađenih vjerodajnica, bez ispisa lozinke, ne+ mogu se pokrenuti preko API-ja, ne ostavljaju tajne u povijesti ljuske (argumenti).+- [ ] Pokretanje: backend se ne pokreće s `--reload`/debug u produkcijskoj slici; radi kao+ neprivilegirani korisnik (vidi infra listu).++## 8. Dostupnost (DoS jednim korisnikom)++- [ ] Skupi endpointi (statistika, flota, povijest, eksport) imaju ograničenje rada i cache;+ nema neograničenih petlji nad podacima koje kontrolira kolektor.+- [ ] Dugotrajne operacije ne blokiraju event loop (sinkroni DB/CPU rad u `async def`+ ruti blokira sve zahtjeve: jedan spor zahtjev = zastoj).+- [ ] Rast podataka: tablice koje se pune iz check-inova imaju gornju granicu ili retenciju;+ zabilježi ako je rast neograničen (poznato za sažetke povijesti).diff --git a/.claude/skills/security-audit/references/checklist-collector-agent.md b/.claude/skills/security-audit/references/checklist-collector-agent.mdnew file mode 100644index 0000000..010943c--- /dev/null+++ b/.claude/skills/security-audit/references/checklist-collector-agent.md@@ -0,0 +1,96 @@+# Kontrolna lista: collector, docker proxy, system helper, LLM asistent (`src/agent/**`)++Čitaj `collector.py`, `docker_proxy.py`, `system_helper.py`, `health_checks.py`, `service.py`,+`prompts.py`, `main.py`, `utils.py` i `SYSTEM_PROMPT.md` u cijelosti. Docker proxy i system+helper drže privilegije: najveća pažnja ide njima.++## 1. Docker proxy (`docker_proxy.py`): drži root-ekvivalent++- [ ] **Bijela lista je doslovna usporedba cijelog request targeta.** Nema dekodiranja,+ normalizacije, `startswith`, regexa bez sidra (`re.match` bez `$`/`fullmatch`), ni+ `in`. Pokušaji zaobilaženja: `?all=1&x=y`, `//containers/json`, `/containers/json/`,+ `%2e`, `;`, `#`, `\`, razmak, tab, `\r\n`, velika/mala slova metode, `HEAD`/`OPTIONS`,+ `GET` s tijelom, `Transfer-Encoding`/`Content-Length` proturječja, HTTP/1.0,+ više zahtjeva na jednoj vezi (pipelining/smuggling), ID kontejnera koji nije hex+ (`../`, `x/exec`).+- [ ] ID u `/containers/<id>/json` i `/stats` strogo validiran (hex, duljina) *prije*+ prosljeđivanja u Docker. Proxy ne gradi zahtjev prema Dockeru spajanjem neprovjerenog+ stringa.+- [ ] Proxy **ne prosljeđuje** zaglavlja ni metodu klijenta; zahtjev prema Dockeru gradi sam.+- [ ] **Filtriranje odgovora:** vraća samo fiksnu listu polja; `Env`, `Labels`, `Mounts`,+ `Config`, `HostConfig`, `Error` ne izlaze. Provjeri i za `/containers/json` i za+ stats (stats ne sadrži tajne, ali provjeri). Nova polja = namjerna promjena + test.+- [ ] Veličina odgovora/zahtjeva ograničena; timeouti; jedna sporo-čitajuća veza ne blokira+ sve (DoS prema kolektoru); greške Dockera ne vraćaju sirovi tekst.+- [ ] Unix socket: mode 0660, direktorij/volumen dijeljen **samo** s kolektorom; socket se+ ne stvara u svijetu-zapisivom direktoriju; nema utrke pri `bind`/`chmod` (kratak+ prozor s širim pravima; zadani `umask`).+- [ ] Nema TCP listenera, nema mreže (`network_mode: none`, vidi infra listu).+- [ ] Log 403 ne dopušta log injection (sanitiziran request target).++## 2. System helper (`system_helper.py`): privilegiran, izlaz nepouzdan++- [ ] Samo čita (dmesg *read-all*, wtmp), nikad ne piše/briše kernel log.+- [ ] Parsiranje wtmp-a je binarno: provjeri veličinu zapisa, granice, nepotpun zapis,+ ogroman broj zapisa; čita samo boot/shutdown zapise, ne korisnička imena ni adrese.+- [ ] Linije kernel loga: fiksni obrasci, reducirano na printable, skraćeno, ograničen broj+ događaja; nema `eval`/`shell`/format stringa s tim sadržajem; regexi bez katastrofalnog+ backtrackinga na 100 kB retku.+- [ ] Izlazna datoteka: atomarno pisanje (temp + rename), prava, samo u svom volumenu.+- [ ] Ništa se ne izvršava iz sadržaja; nema `subprocess` s ulazom pod kontrolom napadača+ (i ako postoji poziv, argumenti su lista, bez `shell=True`).++## 3. Collector (`collector.py`, 974 retka): radi na hostu, čita host FS++- [ ] **Čitanje datoteka s hosta:** fiksni popis putanja; `O_NOFOLLOW`; ograničena veličina+ čitanja (1 MB firewall datoteke, helperov izlaz); provjera da je helper izlaz svjež i+ iz tekućeg boota; nema putanja sastavljenih od podataka s hosta ili dashboarda+ (`os.path.join` s apsolutnom putanjom tiho zamjenjuje bazu!). TOCTOU između `stat` i+ `open`.+- [ ] `/proc/1/net` i ostali parseri: nepotpuno čitanje → "unknown", nikad djelomičan+ rezultat koji bi ports "zatvorio"; ogromne datoteke ne troše memoriju.+- [ ] **Ono što stiže s dashboarda** (odgovor na register/poll) tretira se nepouzdano: nema+ izvršavanja, nema putanja, nema ciljeva za sondiranje iz odgovora. Mete provjere+ interneta samo iz okoline kolektora (`HOMELAB_CONNECTIVITY_TARGETS`, ≤5, hostname ili+ IPv4), s tvrdim rokom.+- [ ] Sondiranje: ne šalje podatke, samo TCP connect + DNS; ne može se usmjeriti na+ proizvoljnu metu; ograničenje trajanja kako check-in ne bi visio.+- [ ] **Pohrana vjerodajnica kolektora** (token servera): datoteka s modom 0600, ne u+ slici, ne u logu, ne u argumentima procesa; kolektor ne ispisuje token pri pogrešci.+- [ ] Transport: provjera TLS-a ako je HTTPS (ne `verify=False`); kad je običan HTTP unutar+ VPN-a, to je dokumentirano. Nema preusmjeravanja koja bi odnijela `Authorization`+ zaglavlje na drugi host. Timeouti na svim mrežnim pozivima.+- [ ] Greška jednog uzorka ne ruši cijeli kolektor na način koji ga ušutkava bez traga+ (napadač koji izaziva iznimku "gasi" nadzor?).+- [ ] `subprocess`, `os.system`, `eval`, `exec`, `pickle`, `yaml.load`: svako pojavljivanje+ pogledaj; argumenti kao lista; nema podataka s hosta u naredbi.+- [ ] Ograničenja količine prijavljenih podataka (kontejneri, portovi, događaji) na strani+ kolektora, ne samo backenda.++## 4. LLM asistent (`service.py`, `prompts.py`, `SYSTEM_PROMPT.md`, `health_checks.py`)++Dokumentirano: nema putanje za shell, samo tri fiksne provjere (`server_health`,+`container_health`, `daily_checkup`), webhook samo na `ADMIN_ALERT_WEBHOOK_URL`.++- [ ] **Alati su doista fiksni:** parametri alata validirani bijelom listom; nema alata s+ proizvoljnim stringom koji završava u naredbi, putanji, URL-u ili SQL-u. Nema+ dinamičkog dodavanja alata iz podataka.+- [ ] **Prompt injection:** podaci koji ulaze u kontekst modela (imena kontejnera, linije+ loga, poruke grešaka, rezultati provjera, tekst korisnika) mogu sadržavati upute.+ Što najgore može model napraviti ako ih posluša? (Pozvati dopuštene provjere, poslati+ webhook, otkriti sistemski prompt/konverzaciju.) Webhook i alarmi ne smiju nositi+ tajne, a model ne smije moći promijeniti odredište.+- [ ] **Izlaz modela se ne tretira kao pouzdan:** prikazuje se kao tekst (frontend ne radi+ `dangerouslySetInnerHTML`/neprovjeren markdown s HTML-om), ne ulazi u SQL/naredbe/URL-ove.+- [ ] **Konverzacije su vezane uz korisnika** (IDOR: čitanje/brisanje tuđih konverzacija+ po ID-u); limit duljine poruke i broja poruka; limit koraka petlje alata i vremena+ (DoS i trošak API-ja); rate limit po korisniku.+- [ ] API ključ prema LLM pružatelju: iz okoline, ne u logovima, ne u odgovorima, ne u+ konverzacijama koje se pohranjuju ili šalju natrag.+- [ ] Sistemski prompt ne sadrži tajne (pretpostavi da će ga model prije ili poslije+ ispisati).+- [ ] Pozivi prema Docker API-ju idu samo kroz proxy (`DOCKER_SOCKET_PATH`); backend ne montira+ `docker.sock`. Provjeri compose i kod.+- [ ] Webhook: samo `ADMIN_ALERT_WEBHOOK_URL` iz okoline; bez toga nikakav zahtjev; payload+ bez neocijenjenog sadržaja koji bi napravio injekciju u primatelju (Slack mrkdwn,+ mention-i `@channel` iz imena kontejnera).diff --git a/.claude/skills/security-audit/references/checklist-frontend.md b/.claude/skills/security-audit/references/checklist-frontend.mdnew file mode 100644index 0000000..aede1d8--- /dev/null+++ b/.claude/skills/security-audit/references/checklist-frontend.md@@ -0,0 +1,82 @@+# Kontrolna lista: frontend (`src/frontend/**`)++Čitaj `src/App.jsx` (velik), `StatisticsPage.jsx`, `FleetStatistics.jsx`,+`ContainerStatistics.jsx`, `ServerInsights.jsx`, `charts.jsx`, `distributions.jsx`,+`*Logic.js`, `chartMath.js`, `main.jsx`, `index.html`, `vite.config.js`, `package.json`.++## 1. XSS++Dashboard prikazuje podatke koje kontrolira napadač (imena kontejnera, putanje mountova,+poruke događaja, tekst asistenta, imena servera). **Stored XSS u admin kontekstu = preuzimanje+dashboarda**, pa je ovo glavna stavka.++- [ ] Pretraži: `dangerouslySetInnerHTML`, `innerHTML`, `outerHTML`, `insertAdjacentHTML`,+ `document.write`, `eval`, `new Function`, `setTimeout("string")`, `srcdoc`, `postMessage`+ bez provjere origina. Svako pojavljivanje prati do izvora podatka.+- [ ] **Markdown/rich tekst** (izlaz asistenta, opisi): koristi li se biblioteka? Je li HTML+ isključen ili sanitiziran (DOMPurify s restriktivnom konfiguracijom)? Linkovi iz+ markdowna filtriraju `javascript:`/`data:`.+- [ ] **Atributi s URL-om iz podataka:** `href`, `src`, `action`, `formAction`: shema samo+ `http(s)` (i `mailto` ako treba). `target="_blank"` ima `rel="noopener noreferrer"`.+- [ ] SVG iz podataka (grafovi koji ugrađuju stringove u SVG, `<text>`, `<title>`, `<a>`):+ React escapea tekst, ali provjeri ručno sastavljene SVG stringove i `style` objekte+ s podacima.+- [ ] CSS injekcija: vrijednosti iz podataka u `style`/`className` konstruiranima iz stringova.+- [ ] Prototype pollution: spajanje objekata iz JSON-a s kolektora (`Object.assign`, spread+ u ključeve iz podataka, `obj[key] = ...` s ključem iz podataka, `__proto__`).++## 2. Autentikacija i tajne u pregledniku++- [ ] Tokeni/sesije se **ne** drže u `localStorage`/`sessionStorage`/URL-u (sesija je+ HttpOnly kolačić). Provjeri što se sprema u `localStorage` (osjetljivi podaci, tokeni,+ TOTP, recovery kodovi).+- [ ] Recovery kodovi i TOTP tajna: prikazuju se samo jednom, ne spremaju se u stanje koje+ preživljava navigaciju, ne završavaju u URL-u ni logu konzole (`console.log`).+- [ ] Nema API ključeva, `SECRETS_KEY`-a ni internih URL-ova ugrađenih u bundle+ (`import.meta.env.VITE_*` je **javno**). Pretraži `VITE_`.+- [ ] Frontend ne donosi sigurnosne odluke koje backend ne provjerava (skrivanje admin+ gumba nije kontrola pristupa; to pokriva backend lista, ovdje samo zabilježi).+- [ ] Odjava briše stanje u memoriji; nema keširanja osjetljivih odgovora (nginx/`Cache-Control`+ za API).++## 3. Generiranje naredbi za instalaciju ("Connect a server")++Panel generira naredbu koju korisnik kopira i izvršava na udaljenom stroju. Ona sadrži+join token i digest slike.++- [ ] Sve vrijednosti u naredbi (token, digest, URL, `COLLECTOR_BIND`, ime) su **validirane ili+ citirane za ljusku**. Vrijednost s `;`, `$(...)`, `` ` ``, `'`, novim redom ili razmakom+ ne smije izaći iz navodnika. Provjeri odakle vrijednosti dolaze (konfiguracija+ dashboarda, odgovor API-ja) i kako se spajaju u string.+- [ ] Naredba ne koristi `curl | sh` ni nepinirane slike; digest je obavezan, a prikaz+ jasno razlikuje što korisnik izvršava.+- [ ] Token se prikazuje jednom, ima vidljiv istek, a "Kopiraj" ne ostavlja ga u povijesti+ stanja.++## 4. Mrežne pozivne točke i CSRF++- [ ] `fetch` pozivi za mijenjanje stanja šalju kolačić (`credentials`) samo prema vlastitom+ originu; nema poziva prema proizvoljnim URL-ovima iz podataka.+- [ ] Nema otvorenog preusmjeravanja (`window.location = param`), nema `redirect`/`next`+ parametra bez provjere da je relativan put.+- [ ] Rukovanje greškama ne ispisuje stack/tajne korisniku; odgovori API-ja koji sadrže+ tuđe stringove prikazuju se kao tekst.++## 5. CSP i build++- [ ] Frontend radi pod strogim CSP-om bez `unsafe-inline` za skripte (nema inline `<script>`+ u `index.html`, nema `eval` u bundle-u). Novi izvori koji traže proširenje CSP-a+ moraju biti dokumentirani.+- [ ] `vite.config.js`: dev proxy/`server.host` ne izlaže dev server; `build.sourcemap` u+ produkciji isključen ili namjeran; nema `define` s tajnama.+- [ ] `index.html`: bez vanjskih skripti/fontova bez SRI-ja; bez inline handlera.+- [ ] Ovisnosti (`package.json`): neobične ili nepotrebne, `postinstall` skripte; vidi infra+ listu za audit alate.++## 6. Dostupnost kao sigurnost++- [ ] Renderiranje ne pada ni ne zamrzava preglednik na hostilnim podacima: vrlo dugi+ stringovi, tisuće kontejnera, `NaN`/`Infinity`/negativni brojevi u grafovima+ (`chartMath.js`, `fleetLogic.js`, `containerLogic.js`), prazni nizovi, nepoznata polja.+ Pad cijele aplikacije zbog jednog loše oblikovanog zapisa = napadač s kolektorom može+ onesposobiti dashboard.diff --git a/.claude/skills/security-audit/references/checklist-infra.md b/.claude/skills/security-audit/references/checklist-infra.mdnew file mode 100644index 0000000..f646351--- /dev/null+++ b/.claude/skills/security-audit/references/checklist-infra.md@@ -0,0 +1,130 @@+# Kontrolna lista: infrastruktura, repozitorij, ovisnosti++Čitaj `docker/*` (Dockerfile-ove, `nginx*.conf`, `Caddyfile`, `postgres/init/*.sql`),+`compose.yml`, `docker-compose.server.yml`, `collector-compose.yml`, `example.env`,+`.env.example`, `.dockerignore`, `.gitignore`, `.github/workflows/*.yml`, `requirements*.txt`,+`src/frontend/package.json` + lockfile.++## 1. Dockerfile-ovi++- [ ] `USER` je neprivilegiran u svakoj slici koja to može (docker-proxy i system-helper su+ iznimke zbog socketa/capabilityja: provjeri da je to jedino mjesto i da je obrazloženo).+- [ ] Bazne slike: pinirane (digest ili barem točna verzija), ne `latest`; minimalne.+- [ ] Nema tajni u slojevima (`ARG`/`ENV` s tokenima, `COPY .env`, `.git`, `dev.db`, ključevi);+ `.dockerignore` isključuje `.env*`, `.git`, `venv`, `*.db`, testove, `node_modules`.+- [ ] Nema `curl | sh`, `pip install` bez pinova/hashova za produkcijske ovisnosti, `apt` bez+ `--no-install-recommends` i čišćenja; build ne izvršava nepouzdani kod s mreže.+- [ ] `HEALTHCHECK`/`CMD` ne pokreće debug način (`--reload`, `--debug`).+- [ ] Frontend slika: izgrađena u multi-stage, u produkciji nema `node_modules`, dev servera+ ni source mapa koje otkrivaju izvorni kod ako to nije namjera.++## 2. Compose datoteke++Za **svaki servis** zapiši: korisnik, `cap_drop`/`cap_add`, `read_only`,+`security_opt: no-new-privileges`, `privileged`, `network_mode`, `pid`, `ports`, `volumes`,+`environment`/`secrets`.++- [ ] **Objavljeni portovi** vezani su na konkretnu adresu (`127.0.0.1:` ili VPN adresa), ne na+ `0.0.0.0`, osim namjerno javnog frontenda. Docker zaobilazi `ufw`: firewall pravilo nije+ zaštita. Backend, Postgres i docker-proxy nemaju objavljenih portova u server stogu.+- [ ] **`/var/run/docker.sock` montiran samo u docker-proxy**, nigdje drugdje (backend,+ collector, frontend, helper). Isto za `/`, `/etc`, `/root`, `/home`, `/var/lib/docker`.+- [ ] Mountovi hosta su `:ro` gdje je moguće; `/hostfs` je read-only; nema `rw` mounta+ povjerljivih direktorija.+- [ ] Volumen s Unix socketom docker-proxyja dijele **samo** proxy i collector.+- [ ] `cap_drop: [ALL]` kao zadano; svaki `cap_add` obrazložen (samo `CAP_SYSLOG` za helper).+ Nema `privileged`, `pid: host`, `network_mode: host`, `ipc: host`, `devices` bez razloga.+- [ ] `read_only: true` + `tmpfs` za potrebno; `no-new-privileges` na svemu.+- [ ] Mrežna segmentacija: collector-gate nije na mreži reverse proxyja; backend nije na+ mreži kojoj imaju pristup tuđe aplikacije; Postgres samo na internoj mreži;+ `internal: true` gdje izlaz nije potreban.+- [ ] **Zadane/placeholder tajne:** u compose/`example.env` nema stvarnih ili zadano slabih+ lozinki (`changeme`, `postgres`, `admin`) koje bi ostale u produkciji; obavezne+ vrijednosti koriste `${VAR:?poruka}` da stog ne krene bez njih.+- [ ] Slike u compose-u za udaljene hostove pinirane po digestu (`COLLECTOR_IMAGE`,+ `DOCKER_PROXY_IMAGE`), a panel ne generira naredbu bez njih.+- [ ] Postgres: lozinke ne u compose-u u čistom tekstu ako se može `secrets:`/env iz `.env`;+ nema objavljenog 5432.+- [ ] Healthchecki ne otkrivaju tajne; restart politike ne skrivaju petlju rušenja.++## 3. nginx (`nginx.conf`, `nginx.strict.conf`, `nginx.collector.conf`)++- [ ] **Javna strana vraća 404 za sve collector endpointe** (register, poll, enroll, ingest,+ helper). Provjeri točne `location` blokove: prefiks vs. točno, regex, razlike u+ kosoj crti, velika/mala slova, `//`, `/./`, `%2f`, `..;/`. Najgori slučaj: put koji+ nginx ne prepoznaje kao zabranjen, a backend ga normalizira u collector endpoint.+- [ ] `X-Collector-Via` (i slična interna zaglavlja) **postavlja** samo gate, a javna strana+ ga **briše/prepisuje** (`proxy_set_header X-Collector-Via "";`).+- [ ] `real_ip` / `X-Forwarded-For`: `set_real_ip_from` samo privatni/Docker rasponi,+ `real_ip_recursive` ispravno; nema povjerenja u XFF s javne strane; backend dobiva+ stvarni klijentski IP za rate limit.+- [ ] **`add_header` nasljeđivanje:** `add_header` u `location` bloku *poništava* sve+ `add_header` iz `server`/`http` razine. Provjeri da svaki `location` koji ima vlastiti+ `add_header` (cache, CORS) ponavlja sigurnosna zaglavlja (CSP, nosniff, frame, HSTS),+ i da su dodana s `always` kako bi vrijedila i za greške.+- [ ] CSP: bez `unsafe-inline`/`unsafe-eval` za `script-src`, `default-src 'self'`,+ `frame-ancestors 'none'`, `object-src 'none'`, `base-uri 'self'`, `form-action 'self'`;+ dopušteni `style` inline je poznata cijena (React style atributi).+- [ ] HSTS samo na strict/javnoj konfiguraciji; `server_tokens off`.+- [ ] `client_max_body_size` ograničen po lokaciji; `client_body_timeout`, `proxy_*_timeout`,+ `limit_req`/`limit_conn` na osjetljivim rutama (login, enroll) kao drugi sloj.+- [ ] `proxy_pass`: pazi na kosu crtu na kraju (promjena normalizacije puta), prosljeđuju se+ samo potrebna zaglavlja, `proxy_set_header Host` ispravan (backend koristi `Host` za+ provjeru Origina), WebSocket upgrade samo gdje treba.+- [ ] Nema `autoindex on`, `alias` bez kose crte (traversal), poslužuje li se išta osim+ statičkih datoteka SPA-a (`.map`, `.env`, `.git`)? `try_files` ne otkriva skrivene datoteke.+- [ ] TLS konfiguracija (ako je nginx terminira): protokoli ≥ TLS1.2, bez slabih šifri.+ Ako TLS terminira Traefik, navedi to kao pretpostavku, ne nalaz.+- [ ] `Caddyfile`: je li to ostatak/dev konfiguracija? Ako je u produkcijskom putu, pregledaj+ jednako strogo kao nginx.++## 4. Postgres (`docker/postgres/init/*.sql`)++- [ ] Aplikacija se spaja kao korisnik s **minimalnim pravima** (nije superuser, nije vlasnik+ sheme ako migracije nisu potrebne pri radu), odvojen od administrativnog.+- [ ] Lozinke u init skriptama nisu ugrađene u repozitorij; nema `trust` autentikacije iz+ svih adresa; `pg_hba` ne dopušta vanjske veze.+- [ ] Nema `CREATE EXTENSION` ni `GRANT ALL` bez razloga.++## 5. GitHub Actions (`.github/workflows/images.yml`)++Ovo je opskrbni lanac slika koje hostovi pokreću s root-ekvivalentnim pristupom.++- [ ] `permissions:` najmanje potrebno (`contents: read`, `packages: write`, `id-token: write`+ samo za job koji potpisuje); nema zadanih širokih dozvola.+- [ ] **Akcije pinirane na puni commit SHA**, ne na tag (`@v4`) za akcije koje diraju tajne ili+ potpisivanje; provjeri tko ih održava.+- [ ] Nema `pull_request_target` s checkoutom koda iz PR-a; okidač je samo tag `v*` i/ili+ ručni, ne svaki push/PR iz forka.+- [ ] **Script injection:** nema `${{ github.event.* }}` / `${{ github.head_ref }}` /+ imena grane ili poruke commita umetnutih izravno u `run:` (prenesi preko `env:`).+- [ ] Potpisivanje cosignom: *keyless* identitet vezan uz workflow i ref; ispis digesta+ točan; slike se grade iz tagiranog commita; build ne povlači nepinirane ovisnosti bez+ provjere. Opiši kako domaćin **provjerava** potpis (dokumentirano ako `cosign verify`+ nije dio generirane naredbe, to je slabost: potpis koji nitko ne provjerava ne štiti).+- [ ] Tajne se ne ispisuju (`set -x`, `echo`); nema `secrets: inherit` bez razloga.++## 6. Repozitorij i povijest++- [ ] `.gitignore` pokriva `.env`, `*.db`, `*.sqlite`, ključeve, `venv`, `node_modules`.+ `git ls-files | grep -Ei '\.env$|\.db$|\.sqlite|\.pem$|\.key$|id_rsa|secret|credential'`:+ što je **pod gitom**? (`dev.db` je ranije bio.) `.env` i `.env.example` obje postoje:+ provjeri da `.env.example` nema stvarne vrijednosti.+- [ ] **Povijest:** `git log --all --diff-filter=A --name-only | grep -Ei '\.env|\.db|key|secret'`+ i pretraga uzoraka tajni (`git log -p --all -S'SECRET' -S'PASSWORD' -S'TOKEN'` na+ razumnom opsegu). Tajna koja je ikad bila u povijesti smatra se kompromitiranom čak i+ ako je kasnije obrisana; to treba prijaviti uz preporuku rotacije. Ne ispisuj vrijednosti.+- [ ] Pretraga uzoraka u radnom stablu: ključevi (`AKIA`, `ghp_`, `sk-`, `-----BEGIN`),+ lozinke u kodu, hardkodirani tokeni, `localhost` zaobilaženja autentikacije,+ `TODO`/`FIXME`/`HACK` uz riječi `auth`, `security`, `bypass`, `skip`.+- [ ] `TESTS.md`, `docs/`, `README.md` ne sadrže stvarne tajne/domene/IP adrese vlasnika.+- [ ] `SECURITY.md` ima put za prijavu ranjivosti (ima) i točne tvrdnje.++## 7. Ovisnosti++- [ ] `requirements*.txt` i `package-lock.json`: pinirano? Lockfile usklađen s `package.json`?+- [ ] Ako su dostupni alati (`pip-audit`, `npm audit --omit=dev`), pokreni ih **samo uz+ korisnikovu potvrdu** (šalju popis paketa prema registryju). Ako nisu pokrenuti,+ napiši to u pokrivenosti: nemoj tvrditi da su ovisnosti "bez poznatih ranjivosti".+- [ ] Nepotrebne/napuštene ovisnosti, ovisnosti iz URL-a/gita, `postinstall` skripte,+ typosquat imena. Dev ovisnosti ne ulaze u produkcijsku sliku.diff --git a/.claude/skills/security-audit/references/report-format.md b/.claude/skills/security-audit/references/report-format.mdnew file mode 100644index 0000000..87cf5b8--- /dev/null+++ b/.claude/skills/security-audit/references/report-format.md@@ -0,0 +1,103 @@+# Format izvještaja++Piši na hrvatskom (korisnik tako komunicira), a tehničke pojmove, imena funkcija i kod ostavi+kako jesu. Bez uvoda i bez hvaljenja. Najvažnije prvo.++## Struktura++### 1. Sažetak++- Opseg koji je stvarno pregledan (jedna rečenica) i datum/commit (`git rev-parse --short HEAD`,+ napomena ako ima nepohranjenih promjena).+- Jedan redak s brojem nalaza po razini (npr. "2 visoka, 3 srednja, 4 niska"), uz koliko ih je `POTVRĐENO`, a koliko `TREBA PROVJERU`.+- Jedna rečenica o **najvažnijem** nalazu ili "nisam našao iskoristive propuste u pregledanom+ opsegu" ako ih nema.++### 2. Tablica nalaza (obavezna)++Svi nalazi **moraju** biti prikazani u jednoj tablici, **sortirani prema razini rizika, od+najvišeg prema najnižem**: KRITIČNO, VISOKO, SREDNJE, NISKO, INFO. Unutar iste razine+`POTVRĐENO` ide prije `TREBA PROVJERU`, a zatim po iskoristivosti (lakše iskoristivo prvo).+Ne grupiraj po područjima (backend, infra, ...) i ne prikazuj nalaze u prozi umjesto tablice.++| ID | Rizik | Nalaz | Lokacija | Napadač | Učinak | Status | Popravak (jedna rečenica) |+|---|---|---|---|---|---|---|---|+| S-01 | KRITIČNO | Kratak naslov | [main.py:123](src/backend/main.py#L123) | profil iz threat-modela | Pristup | POTVRĐENO | Što promijeniti |+| S-02 | VISOKO | ... | ... | ... | DoS | TREBA PROVJERU | ... |++Pravila za tablicu:++- Stupac "Učinak" kaže što napadač **stvarno dobiva**, da se odmah vidi vodi li nalaz do upada.+ Koristi samo ove vrijednosti, a za više njih odvoji ih zarezom (najteža prva):+ `Izvršavanje koda` (kod ili naredbe na backendu, hostu ili kolektoru), `Pristup` (preuzimanje+ računa, sesije, admin prava, zaobilaženje autentikacije ili 2FA), `Krađa podataka`+ (čitanje podataka ili tajni koje napadač ne smije vidjeti), `Integritet` (podmetanje, mijenjanje+ ili utišavanje podataka i alarma), `DoS` (rušenje ili usporavanje, zaključavanje računa),+ `Izviđanje` (otkriva postojanje računa, strukturu ili konfiguraciju bez izravne štete).+ Ocjenjuj učinak u najgorem realnom slučaju, a ne u idealnom za napadača; ako ovisi o uvjetu+ koji nisi provjerio, napiši ga u detaljima.++- Jedan redak po nalazu; ID-evi `S-01`, `S-02`, ... dodjeljuju se **nakon sortiranja**, pa+ redni broj odgovara poretku rizika.+- Stupac "Lokacija" sadrži klikabilne markdown linkove (najviše dva; ostale navedi u+ detaljima).+- Ćelije su kratke (jedan redak). Sve duže ide u detalje ispod.+- Ako nema nalaza neke razine, ne dodaj prazne retke za nju.+- Ista tablica (isti stupci, isti poredak) vrijedi i za rezultate paralelnih agenata nakon+ spajanja: izvještaj je jedna tablica, ne četiri.++### 3. Detalji nalaza++Isti redoslijed kao u tablici. Svaki nalaz ima ovaj oblik (bez dodatnih naslova):++```+### [VISOKO] S-02 · Kratak, konkretan naslov (POTVRĐENO)++**Gdje:** [datoteka.py:123](putanja/datoteka.py#L123), poziva ga [drugi.py:45](putanja/drugi.py#L45)+**Napadač:** tko (npr. prijavljeni korisnik bez admin prava / VPN peer / kompromitirani kolektor)++**Problem:** što je pogrešno, jednom-dvjema rečenicama.++**Scenarij:** korak po korak što napadač pošalje i što dobije.++**Zašto postojeće zaštite ne pomažu:** koje si kontrole provjerio i zašto ih ovo zaobilazi.++**Popravak:** konkretna promjena (i test koji je treba pokriti), s kratkim isječkom koda ako+pomaže.++**Dokaz:** test/skripta koju si pokrenuo i rezultat, ili "nije pokrenuto: <razlog>".+```++ID je isti kao u tablici nalaza.+Za `TREBA PROVJERU` umjesto "Dokaz" napiši **Što treba provjeriti** (točno što i kako).++### 4. Tvrdnje iz dokumentacije koje kod ne podržava++Tablica: tvrdnja (citat + mjesto u `SECURITY.md`/`docs/`) | što kod zapravo radi | datoteka:redak.+Izostavi odlomak ako nema razlika.++### 5. Očvršćivanje (nije ranjivost)++Kratke stavke (jedan redak) za stvari koje nisu iskoristive same za sebe, ali bi smanjile+rizik. Ne miješaj ih s nalazima.++### 6. Pokrivenost++- **Pregledano:** popis datoteka/područja koja si pročitao u cijelosti.+- **Nije pregledano i zašto:** (npr. ovisnosti bez audit alata, živa konfiguracija+ Traefika/OpenVPN-a izvan repozitorija, dinamičko testiranje).+- **Pretpostavke** o okruženju na koje se nalazi oslanjaju.++### 7. Sljedeći korak++Jedna rečenica: predloži redoslijed popravljanja i pitaj želi li korisnik da počneš. Ništa+ne mijenjaj dok ne odgovori.++## Pravila stila++- Nalazi su činjenice s dokazom; nagađanje označi kao `TREBA PROVJERU`.+- Ne duplicirati: isti uzrok na više mjesta = jedan nalaz s popisom lokacija.+- Ne pretjeruj s ozbiljnošću radi dojma, ali ni ne ublažavaj (korisnik izričito traži+ strogu procjenu). Kad dvoji, napiši obje razine i obrazloži jednom rečenicom.+- Ne ispisuj vrijednosti tajni; navedi samo lokaciju i vrstu.+- Bez emojija.diff --git a/.claude/skills/security-audit/references/threat-model.md b/.claude/skills/security-audit/references/threat-model.mdnew file mode 100644index 0000000..e20e4e6--- /dev/null+++ b/.claude/skills/security-audit/references/threat-model.md@@ -0,0 +1,89 @@+# Model prijetnji (Homelab Agent)++Sažetak arhitekture i povjerenja, izveden iz `SECURITY.md` i koda. Ako se kod i ovaj dokument+razilaze, **vjeruj kodu** i to prijavi kao nalaz o dokumentaciji.++## Komponente i što dodiruju++| Komponenta | Gdje | Doseže | Povjerenje |+|---|---|---|---|+| Frontend (nginx + React SPA) | `docker/nginx*.conf`, `src/frontend` | javni listener; proxy prema backendu | javno dostupan |+| Backend (FastAPI) | `src/backend` | Postgres, interni mrežni segment | obrađuje sve ulaze |+| Postgres | `docker/postgres/init` | samo backend | sadrži hashove, šifrirane TOTP tajne, podatke o hostovima |+| Collector gate | `docker/nginx.collector.conf` | backend; jedan port na VPN adresi | jedini put do collector endpointa |+| Collector | `src/agent/collector.py` | `/proc` hosta, host FS read-only (`/hostfs`), Docker proxy, backend preko HTTP-a | pokreće se na udaljenim hostovima, može biti kompromitiran |+| Docker proxy | `src/agent/docker_proxy.py` | Docker socket (uid 0), Unix socket prema collectoru | drži root-ekvivalentan pristup; mora ostati minimalan |+| System helper (opcionalno) | `src/agent/system_helper.py` | kernel log, wtmp, `CAP_SYSLOG` | privilegiran; njegov izlaz je nepouzdan |+| LLM asistent | `src/agent/service.py`, `prompts.py`, `SYSTEM_PROMPT.md`, `health_checks.py` | fiksni read-only provjere, opcionalni webhook | ulaz uključuje podatke s hosta |+| CI | `.github/workflows/images.yml` | GHCR, cosign | opskrbni lanac za slike koje hostovi izvršavaju |++Izvan repozitorija (nisu u opsegu, ali ih treba imati na umu kao pretpostavke): Traefik pred+frontendom, OpenVPN koji nosi promet kolektora, GHCR paketi.++## Profili napadača++Za svaki nalaz odredi koji ga profil može iskoristiti.++1. **Anonimni korisnik interneta** na javnom listeneru. Doseže samo dashboard i njegov API.+ Collector endpointi (register, poll, enroll, ingest) na toj strani **ne smiju postojati**.+2. **Prijavljeni korisnik bez admin prava.** Po dizajnu vidi sve monitoring podatke svih+ servera. Pitanje je može li *mijenjati* išta, vidjeti tuđe račune/sesije/konverzacije,+ ili postati admin.+3. **Susjedna web aplikacija na istoj domeni/podlozi** (sibling subdomain iza istog reverse+ proxyja). Ista "site", pa `SameSite=Lax` ne štiti od zahtjeva koji mijenjaju stanje.+ Zaštita je provjera `Origin` / `Sec-Fetch-Site`.+4. **Peer na VPN-u.** Doseže collector gate. Bez valjanog join tokena ne smije ni zatražiti+ pridruživanje; s njim čeka admin odobrenje; ingest traži token servera.+5. **Kompromitirani kolektor.** Kontrolira sve što prijavljuje (imena kontejnera, mountovi,+ portovi, događaji). Backend mu ne smije vjerovati: tipovi, duljine, rasponi, brojevi,+ limiti po check-inu i po satu. Ne smije moći utjecati na druge servere ni na dashboard.+6. **Kompromitirani ili zloćudni kontejner na hostu / susjed na Docker mreži.** Može+ pokušati doći do backenda mimo nginxa (lažiranje `X-Collector-Via`, `X-Forwarded-For`),+ do Unix socketa docker proxyja ili do volumena koji ne smije dijeliti.+7. **Kompromitirani dashboard.** Ne smije moći natjerati hostove da izvrše kod (dashboard+ ne poslužuje kod), ni usmjeriti sondiranja na proizvoljne mete (mete dolaze iz okoline+ kolektora).+8. **Opskrbni lanac.** Ovisnosti (pip, npm), bazne slike, GitHub Actions, workflow+ dozvole, nepodpisane ili nepinirane slike.+9. **Hostilni podaci s nadziranog hosta.** Imena kontejnera, putanje mountova, linije+ kernel loga, wtmp, `/etc/os-release`. Napadač s ograničenim pristupom hostu može+ utjecati na te stringove. Putovi: stored XSS, injekcija u ključeve/upite, prekoračenje+ limita, razbijanje parsera, **prompt injection** prema asistentu.+10. **Fizički/lokalni pristup hostu** (npr. `seed_user.py`, `reset_2fa.py`) je izvan modela+ prijetnji po dizajnu: ta ovlast već daje pristup bazi. Ali skripte ne smiju imati+ ugrađene vjerodajnice ni curiti tajne.++## Najvrjednija imovina (redom)++1. Pristup Docker API-ju / rootu na hostovima (preko proxyja, helpera, mountova).+2. Admin sesija dashboarda i mogućnost odobravanja novih servera.+3. Tokeni servera i join tokeni (hashirani u bazi).+4. `SECRETS_KEY` (Fernet) i TOTP tajne, recovery kodovi.+5. Monitoring podaci (otvoreni portovi, firewall hash, popis kontejnera) jer mapiraju mrežu.+6. Integritet alarma (napadač koji može utišati ili lažirati incidente skriva napad).++## Pretpostavke dizajna koje treba provjeriti, ne uzeti zdravo za gotovo++- Backend vjeruje `X-Collector-Via` samo ako ništa nepouzdano ne doseže backend izravno.+- nginx prihvaća `X-Forwarded-For` samo od privatnih/Docker adresa; rate limiti ovise o tome.+- Collector endpointi odbijaju se i na javnoj strani (nginx 404) i u backendu+ (`COLLECTOR_ACCESS=strict`). Provjeri **oba** sloja i da ni jedan ne ovisi o drugom.+- "Jednokratno" (join tokeni, TOTP kodovi, recovery kodovi, setup token) mora biti atomarno+ u bazi, ne "provjeri pa označi" (utrka).+- "Ograničeno" (500 kontejnera, 20 otvorenih tokena, 5 pokušaja/5 min) mora vrijediti i pod+ paralelnim zahtjevima.+- Prvi račun postaje admin: provjeri sve puteve do tog stanja (registracija, setup token,+ `seed_user.py`, utrka dvaju istodobnih prvih zahtjeva).++## Poznata prihvaćena ograničenja (ne prijavljuj kao nova, ali provjeri da još vrijede)++Iz `SECURITY.md`: svi prijavljeni računi vide sve monitoring podatke; zaključavanje 2FA koraka+pri pogađanju kodova; sesije prije uvođenja 2FA nisu u popisu; samo TOTP; promet VPN-om je+običan HTTP unutar tunela; sažeci povijesti se još ne brišu. Ako kod odstupa od onoga što je+tamo opisano, to je nalaz.++## Otvorene stavke iz projektne bilješke (provjeri jesu li još otvorene)++Ograničenje broja čekajućih servera po IP-u; rate limit na `enroll`; `src/backend/seed_user.py`+(ugrađena admin lozinka); `dev.db` pod gitom; zastarjeli README. Ako su još stvarne, prijavi ih+s trenutnim stanjem u kodu.diff --git a/README.md b/README.mdindex acded9a..1274327 100644--- a/README.md+++ b/README.md@@ -79,7 +79,7 @@ flowchart LRbackend --> pg[(PostgreSQL)]backend --> redis[(Redis)]localc["collector"] --> gate- localc -->|"unix socket"| dproxy["docker-proxy<br/>3 read-only requests"]+ localc -->|"unix socket"| dproxy["docker-proxy<br/>4 read-only requests"]helper["system-helper (optional)<br/>CAP_SYSLOG, no network"] -.->|"summary file"| localcendbrowser["Browser"] -->|"https://your-domain"| traefik["Traefik"]@@ -139,830 +139,61 @@ docker compose down # stop (data volumes are kepTo try **Connect a server** on the same machine, set `COLLECTOR_IMAGE` and`DOCKER_PROXY_IMAGE` in `.env` to `homelab-collector:latest` and`homelab-docker-proxy:latest` (the images Compose just built). Other machines-need published images; see [Publishing the collector images](#publishing-the-collector-images).+need published images; see [Publishing the collector images](docs/deployment.md#publishing-the-collector-images).-## Deploying on a server (behind Traefik)+## Deploying on a server-There are two Compose files:+The server stack (`docker-compose.server.yml`) publishes the dashboard through an existing **Traefik** and+serves the collector endpoints only on a separate listener bound to your **VPN address**. In short:-| File | Use | Differences |-|---|---|---|-| `compose.yml` | Laptop / development (the default when you run `docker compose` without `-f`) | Dashboard on port 5173 on all interfaces; plain HTTP; optional Caddy (`--profile https`). |-| `docker-compose.server.yml` | Server behind an existing Traefik | Traefik labels and network; HTTPS-only session cookie; only the first account can register; port 5173 bound to `127.0.0.1`; no Caddy; **collector endpoints removed from the public side** and served by a separate collector gate on your VPN address. |+1. `cp example.env .env`, then set `POSTGRES_PASSWORD`, `DASHBOARD_DOMAIN`, your Traefik names and+ `COMPOSE_FILE=docker-compose.server.yml` (the last one matters: without it Compose uses the development stack).+2. Generate `SECRETS_KEY` (needed for two-factor sign-in) and set `COLLECTOR_BIND` / `COLLECTOR_PUBLIC_URL` for your VPN.+3. `docker compose -f docker-compose.server.yml up -d --build`, then register the first account with the setup token from the backend log.+4. Click **Connect a server** to add other machines.-### 1. Create `.env` on the server+Every step, with the reasons and the pitfalls, is in [Deploying on a server](docs/deployment.md) and+[Adding servers](docs/adding-servers.md). To update later: `git pull` and `docker compose -f docker-compose.server.yml up -d --build`.-Nothing in `docker-compose.server.yml` needs editing: everything specific to you comes from-`.env`, so `git pull` can always update the file. Start from the production template:+## Documentation-```bash-cp example.env .env-```--Then set at least these:--| Setting | Default | Meaning |-|---|---|---|-| `POSTGRES_PASSWORD` | *(required)* | A long random password for the bundled database. |-| `DASHBOARD_DOMAIN` | *(required)* | Your domain, without `https://`. Used for the Traefik router. |-| `TRAEFIK_ENTRYPOINT` | `websecure` | Your Traefik HTTPS entrypoint name. |-| `TRAEFIK_CERT_RESOLVER` | `letsencrypt` | Your Traefik certificate resolver name. |-| `TRAEFIK_NETWORK` | `traefik_default` | The Docker network Traefik is attached to. |-| `TRAEFIK_ROUTER` | `homelab` | Name of this dashboard's Traefik router and service; change it only if it clashes with another one. |-| `COMPOSE_FILE` | `docker-compose.server.yml` | Makes plain `docker compose ...` commands use the server file (see below). |--Find the network with `docker network ls` or-`docker inspect <traefik-container> --format '{{json .NetworkSettings.Networks}}'`.-The entrypoint and resolver names are in Traefik's static configuration-(`entryPoints:` / `certificatesResolvers:`).--### 2. Use the right Compose file--`COMPOSE_FILE=docker-compose.server.yml` in `.env` matters. Without it, Compose picks-`compose.yml` (the development stack), and an `up` would replace the server containers-with ones that have no Traefik labels and open registration.--**Don't copy the server file under another name** (for example `docker-compose.yml`). If-both `compose.yml` and `docker-compose.yml` exist, Compose silently uses `compose.yml`, and-a copy won't receive updates. If you made a copy of an older version, move your values-into `.env`, delete the copy, and continue with `docker-compose.server.yml`.--### 3. Start it--```bash-docker compose -f docker-compose.server.yml up -d --build-```--Open `https://<your-domain>` and register your account right away. The first-account needs the **setup token** (`docker compose logs backend | grep "setup token"`,-or the value of `ADMIN_BOOTSTRAP_TOKEN` if you set one), so a stranger who finds a-fresh deployment can't claim it; then registration closes. That first account is-the admin. Traefik terminates HTTPS with its own certificates; the stack needs no-certificate configuration.--The public address serves only the dashboard. Collectors don't use it; set up-their route in the next step.--### 4. Let remote servers report over your VPN--Collectors reach the **collector gate**, not the public domain. In `.env`:--| Setting | Example | Meaning |-|---|---|---|-| `COLLECTOR_BIND` | `10.8.0.1` | Your VPN server's address on the dashboard machine (for OpenVPN, the `tun` address). The gate's port is published on this address only. |-| `COLLECTOR_PORT` | `8081` | Port of the gate on that address. |-| `COLLECTOR_PUBLIC_URL` | `http://10.8.0.1:8081/api` | What **Connect a server** tells new machines to use. |-| `COLLECTOR_IMAGE`, `DOCKER_PROXY_IMAGE` | `ghcr.io/…@sha256:…` | The images remote machines install; see below. |--Things to get right:--- **Bind to the VPN address, not `0.0.0.0`.** Docker's published ports bypass- `ufw` and similar firewalls, so a "deny from outside" rule does not protect them.-- If Docker starts before the VPN is up, the bind fails until the address exists.- `restart: unless-stopped` retries, or make Docker start after the VPN (for- example a systemd drop-in with `After=openvpn-server@<name>.service`).-- Traffic inside the VPN is plain HTTP over the encrypted tunnel; collectors- authenticate with a per-server token that the dashboard stores only as a hash.--#### Publishing the collector images--Remote machines install the collector and the Docker proxy from container images,-pinned by digest, so no code is ever downloaded from the dashboard. The-[Publish images](.github/workflows/images.yml) workflow builds them (amd64 and-arm64), signs them with cosign and prints their digests:--```bash-git tag v0.1.0 && git push origin v0.1.0-```--1. Open the workflow run's summary and copy the `COLLECTOR_IMAGE=…` and- `DOCKER_PROXY_IMAGE=…` lines into `.env`.-2. If the repository is private, make the three packages **public** (GitHub →- Packages → *Package settings* → *Change visibility*); otherwise machines can't- pull them without credentials.-3. Rebuild the stack so the backend knows the images:- `docker compose -f docker-compose.server.yml up -d --build`.--**Connect a server** won't generate a command until both images are set.--### Updating--```bash-git pull-docker compose -f docker-compose.server.yml up -d --build-```--Always rebuild **all** services (don't name individual ones). The backend,-frontend, collector, Docker proxy and system helper each have their code built-into their image, so a service that isn't rebuilt keeps running the old version.-The database schema is updated automatically when the backend starts.--Collectors on other machines run the image digest from the command you generated.-To update them, publish a new version tag, put the new digests in `.env`, and run a-freshly generated command on each machine (or pull the new digests into its-compose project). Collectors started from `collector-compose.yml` need-`docker compose -f collector-compose.yml up -d --build` after a `git pull`.--<diagram - What each step updates: git pull → source on disk; docker compose up -d --build → backend, frontend, collector, docker-proxy and gate containers rebuilt and recreated; git tag → Publish images workflow → new digests → .env and a fresh Connect a server command for remote machines; show that skipping --build leaves the old image running>--### Without Traefik--- **Caddy (automatic Let's Encrypt):** with `compose.yml`, set- `DOMAIN`, `SESSION_COOKIE_SECURE=true` and `FRONTEND_BIND=127.0.0.1` in- `.env`, open ports 80/443, and run `docker compose --profile https up -d --build`.-- **Private, over a VPN:** install Tailscale, WireGuard or OpenVPN on the server- and your devices, run `docker compose up -d --build`, and use `http://<vpn-ip>:5173`- (collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly. This uses- the development stack's single listener; for a public dashboard with collectors- on a VPN, use the server stack above.--## Adding servers+Full documentation is in the [`docs/`](docs/README.md) folder:-Adding and removing servers, changing check intervals and managing the-enrollment key are admin-only.--1. On the dashboard, click **Connect a server**. Check the address field: with- `COLLECTOR_PUBLIC_URL` set it shows your VPN address; otherwise it is prefilled- with the address you opened the dashboard at and must be reachable from the- new machine (not `localhost`).-2. Click **Generate command**. This creates a single-use token that expires- (15 minutes by default) and puts it in the command.-3. Copy the command and run it on the new machine. It needs Docker with the- compose plugin, and a route to the address above (the VPN).-4. Within a few seconds the machine appears under **New servers detected** with a- **JOIN TOKEN** badge and a pairing code (also printed in- `docker logs homelab-collector`). Click **Add**. It starts reporting on its- next check, without a restart.--<ui - The Connect a server panel: the address field, the Generate command button with its single-use / expires note, and the copyable compose command>--<ui - The New servers detected list with one pending machine showing the JOIN TOKEN badge, its pairing code and the Add / Dismiss buttons>--The machine running the dashboard is added the same way (its collector starts-with the stack, over the gate's internal listener). That collector has-`HOMELAB_DASHBOARD_HOST=true` set in the Compose files, which makes it the machine-shown in the charts at the top of the dashboard and first in the server list.-Collectors installed on other machines don't set it.--### What the command does--It starts a small compose project named `homelab-collector` from the YAML on its-standard input (shortened here):--```bash-docker compose -p homelab-collector -f - up -d <<'EOF'-services:- collector:- image: ghcr.io/…/collector@sha256:…- environment:- HOMELAB_API_URL: http://10.8.0.1:8081/api- HOMELAB_JOIN_TOKEN: hlj_…- HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sock- volumes:- - /proc:/host/proc:ro- - /:/hostfs:ro- - collector_state:/var/lib/homelab-collector- - docker_proxy_socket:/run/docker-proxy:ro- cap_drop: [ALL]- read_only: true- docker-proxy:- image: ghcr.io/…/docker-proxy@sha256:…- network_mode: none- cap_drop: [ALL]- read_only: true- volumes:- - /var/run/docker.sock:/var/run/docker.sock- - docker_proxy_socket:/run/docker-proxy-EOF-```--| Part | Purpose |+| | ||---|---|-| `HOMELAB_API_URL` | Where the collector gate is. |-| `HOMELAB_JOIN_TOKEN` | Proves an admin asked this machine to join. Single use, expires. |-| `image: …@sha256:…` | Signed images pinned by digest. No code is downloaded from the dashboard. |-| `/proc:/host/proc:ro` | Host CPU, memory, load, network, uptime, boot ID, OOM counter and kernel flags. |-| `/:/hostfs:ro` | Disk usage of every mounted filesystem, hostname, OS name and `/etc/machine-id`. |-| `docker-proxy` | The only container with the Docker socket. It answers container list, inspect (without environment variables) and stats, and nothing else. |-| `cap_drop: [ALL]`, `read_only: true` | The collector can't read other users' private files through `/hostfs`, or write outside its state volume. |-| `collector_state` volume | Keeps the token after approval, so restarts don't need approval again. |--To remove a collector completely:--```bash-docker rm -f homelab-collector homelab-docker-proxy-docker volume rm homelab-collector_collector_state homelab-collector_docker_proxy_socket-```--### How registration works--- Over the VPN a collector can't even ask to join without a valid join token. It- then asks, checking every 5 seconds whether it has been approved. Clicking- **Add** creates the server and uses up the token; the collector receives its own- token and saves it. If a token is wrong, expired or used, the collector prints- why and you copy a fresh command.-- Servers are matched on `/etc/machine-id`. If a new request matches a server you- already have, the dashboard warns that approving it **replaces that server's- credentials and signs out its current collector**, and asks you to confirm.- Only do that when you are reinstalling that machine.-- If you delete a server, its collector can no longer report. On a server install- it can't ask to join again without a new command (its token was used up). In the- development stack it asks again and only comes back if you click **Add**.- **Dismiss** moves a request to a collapsed "Dismissed" list.--<diagram - Sequence diagram of joining: admin clicks Generate command → join token created → command run on the new machine → collector POST /collectors/register with the token over the VPN listener → pending entry with pairing code → collector polls every 5 s → admin clicks Add (token consumed) → poll returns the token → collector saves it and starts POST /metrics/ingest; include the delete → 404 → needs a new command loop>-- Requests from collectors that stopped asking disappear after 2 minutes and are- deleted after a day.--Alternatives:--- **Enrollment key (no approval step):** for scripted installs, generate a key- under **Connect a server → Advanced** and add `HOMELAB_ENROLL_KEY: <key>` to the- collector's environment. Only a hash of the key is stored; rotating or revoking- it does not affect servers that are already connected.-- **Compose from the repository:** copy the repository to the machine, set- `HOMELAB_API_URL` (and `HOMELAB_JOIN_TOKEN`) in a `.env` next to- `collector-compose.yml`, and run- `docker compose -f collector-compose.yml up -d --build`. This builds the images- locally instead of pulling them.--### Networking checklist--- Every monitored machine must reach the collector gate (or the dashboard address- in the development stack); the dashboard never connects to them.-- For machines outside your LAN, put them on a VPN (OpenVPN, WireGuard, Tailscale)- and publish the gate on the VPN address. Don't expose plain HTTP to the- internet: collector tokens travel over this connection.-- On a LAN, give the dashboard machine a fixed IP (e.g. a DHCP reservation).--## The dashboard--- **This machine** (top): live CPU, memory and root disk charts for the machine- running the dashboard, loaded from the last 30 minutes of stored metrics.-- **Fleet**: one card per server, with the dashboard's own machine first. Click a- server's name to collapse its card to a single line (status, CPU, RAM, fullest- disk, incident count); **Collapse all** / **Expand all** appear with more than- one server. Each card shows a **Storage** bar per filesystem, colored by the- alert thresholds (including any per-mount overrides), a **System** row and a- **Containers** list. Both lists start closed. A filter box appears for servers- with more than 10 containers.-- **Network** row: internet reachability, how many ports the server listens on and how- many are exposed, and the firewall state, opening into the listener list and the last- 7 days of port and firewall changes.-- **System** row: one line of status chips (when it booted, last shutdown, clock,- OOM kills in the last 24 hours, kernel, pending updates) that opens into the- last 7 days of reboots, OOM kills and kernel messages. A chip that says- *unknown* means the collector couldn't read that value on this host, not that- anything is wrong.-- **Active incidents**: open incidents, most severe first, with the server and- when they started.--Which cards are collapsed is remembered per browser; the System and Containers-lists always start closed.--<ui - One expanded server card: CPU/RAM/Disk/Load tiles, Storage bars for several mounts (one amber or red), the System row open with a reboot and an OOM kill in its events, and the Containers list open with an unhealthy container and one container's details expanded (stat tiles and CPU/memory charts)>--<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>--## Statistics--**Statistics** in the navigation shows how one server behaved over time. Pick a server and a-range (1 hour, 6 hours, 24 hours, 7 days, 30 days, 90 days, 1 year or all time) and you get-CPU, memory, swap, load and root disk charts. Each chart is the average as a line with a-lighter band up to the highest value, so a short spike stays visible even in a yearly view.-Hover or touch a point for the exact time, average, highest value and number of check-ins, or-use **View as table**. Gaps (a server that was offline) are drawn as gaps. A note under the-cards says what each point is (for example "one per hour, from hourly summaries") and when-the history starts. The selection is kept in the address, so a view can be bookmarked, and-the range is remembered per browser. The short ranges refresh every 30 seconds, except-while the tab is hidden.--Under the charts, **Containers** shows every container of that server as a small card (status-dot, health, restarts, CPU, memory against its limit), with search and sorting (problems first-by default). Select a card for that container's CPU and memory over the chosen range, with its-memory limit drawn as a dashed line, how much of the time it was not running, and how many-times it restarted in the range. A container only reports CPU and memory while it runs.--Further down a server's page: how its CPU and memory use were **distributed** (the share of the-time in each 10% band), **when it is busy** (average CPU or memory by hour of the day and weekday-over the last 30 days, in your time zone), **storage** (a chart per filesystem with an estimate-such as "full in about 3 days" when there is a clear growth trend, and none otherwise), and-**reliability** (how long Docker takes to list containers, and how long the collector takes).--Choose **All servers** for the whole fleet: containers by state and open incidents by severity-(donuts), incidents opened over time by severity (stacked columns), what the incidents were-about, servers by disk usage, the containers using the most memory and CPU right now, and each-server's availability. Availability is the share of expected check-ins that arrived; it is-approximate because it assumes each server's current check interval. Every chart can be shown-as a table.--Short ranges come from the stored check-ins, longer ones from the summaries described under-[Database schema](#database-schema), so a year of history costs about as much to load as an-hour.--What to know about the statistics:--- **Time zones.** Charts for up to a day or two use your clock; day, week (Monday to Sunday) and- month points are aligned to UTC and labelled in UTC, because that is how they are summarised.-- **Some figures are approximate**: distributions and the days-until-full estimate are worked out- in the browser from the points you are looking at, availability assumes each server's current- check interval, and the top containers show values from the latest check-in, not an average- over the range. The page says so where it matters.-- **Pointer-only details.** Hover and touch tooltips don't work from the keyboard yet; every chart- has **View as table** as an alternative.-- **Not built yet** (tracked in Milestone 8 on GitHub): CSV and JSON export, reboots per month, a- state timeline and restarts per day for containers, rankings over the chosen range, network- traffic charts, event markers on charts, and retention settings for the summaries.--## Alerts and incidents--Rules run on every collector check-in. Each problem opens at most one incident-(per server, per rule and, for disks, per mount); while it lasts the incident is-updated instead of duplicated, and it resolves automatically once the rule-stops firing.--| Rule | Default | Severity |-|---|---|---|-| Disk usage, per mount (overridable per mount) | 85% / 95% | warning / critical |-| Inode usage, per mount | 85% / 95% | warning / critical |-| Memory usage | 90% / 95% | warning / critical |-| Swap usage | 50% | warning |-| 5-minute load per CPU core | 1.5× / 2× | warning / critical |-| Container restart loop | 3 restarts within 30 minutes | critical |-| Container healthcheck `unhealthy` | – | critical |-| Container exited with an error | within 60 minutes of exiting | warning |-| Container killed by the kernel for running out of memory | within 60 minutes | critical |-| Container stopped by a signal (exit 137 or 143, usually `docker stop`) | within 60 minutes | info |-| Container no longer present | stays open 60 minutes | info |-| Container running a different image build | stays open 60 minutes | info |-| Possible memory leak (steady climb over 12 hours) | 10 MB/h and 20% growth | warning |-| Docker unreachable (after it worked before) | 3 check-ins in a row | critical |-| Docker slow to list containers | over 2000 ms, 3 check-ins in a row | warning |-| Collector can't keep up with its check interval | gathering takes over 80% of the interval, 3 check-ins in a row | warning |-| New listener reachable from other machines | stays open 60 minutes | warning |-| Firewall configuration changed | stays open 60 minutes | info |-| Firewall protection reduced (ufw turned off, rules removed, a file gone) | stays open 60 minutes | warning |-| No internet (only if the check is on) | 3 check-ins in a row | warning |-| DNS failing (only if the check is on) | 3 check-ins in a row | warning |-| Server stopped reporting | 3 missed check-ins | critical |-| Host rebooted | stays open 60 minutes | info (critical after an unclean shutdown) |-| OOM killer fired | stays open 60 minutes | warning (critical at 3 or more) |-| Kernel fault flags (machine check, bad page, oops, warning, soft lockup) | – | warning, clears on reboot |-| Clock not synchronised | 3 check-ins in a row | warning |-| Reboot required | – | info |--New listening ports and firewall changes are noticed the same way, by comparing with-what the previous check-in saw. The first report is a silent baseline, an unreadable-or empty report is treated as unknown (never as "everything closed"), and short-lived-programs using random high ports (loopback ones and all UDP ones) are ignored. A new-port bound only to the loopback address is recorded but doesn't open an incident; one-reachable from other machines does. Events are capped like the container ones.--Container exits, removals and image changes are noticed by comparing each check-in-with the previous one, matching containers **by name** (Compose recreates them with-new IDs). Nothing is reported on a server's first check-in or when Docker was-unreachable (an empty list is not proof that everything vanished), and churn is-capped so containers that come and go constantly can't flood the database. A leak is-only flagged when hourly memory averages rise along a straight line without a-restart; caches and warm-up can look similar, so treat it as a prompt to look.--Host rules (disk, inodes, memory, swap, load) open an incident only after-**3 check-ins in a row** breach the threshold, so a brief spike is ignored. An-open incident stays open while the latest value is still over the warning-threshold, and becomes critical only when the critical threshold is breached for-the same number of check-ins.--<diagram - Incident lifecycle for a host rule as a timeline of check-ins: value crosses the warning threshold → after 3 check-ins in a row a WARNING incident opens → stays over critical for 3 check-ins → escalates to CRITICAL → drops under warning → RESOLVED; show a single spike that opens nothing>--Admins can change every threshold, the number of check-ins in a row, how long-event incidents (reboots, OOM kills, container exits, removals and image changes)-stay open, the slow-Docker and slow-collection limits, the memory-leak window and sensitivity, and-**per-mount disk thresholds** under-**Admin → Alerts**. A per-mount override replaces the default disk thresholds for-that mount path on every server. Changes apply from each server's next check-in-and are recorded in the audit log.--<ui - Admin → Alerts: the Incident thresholds form with Warning / Critical fields per rule, Check-ins in a row, and the container restart loop row>--<ui - Active incidents section with a CRITICAL and a WARNING incident, each showing its server and start time>--Incidents are shown on the dashboard only; there are no push notifications yet.-(`ADMIN_ALERT_WEBHOOK_URL` is used by the assistant's `daily_checkup` only.)--## System health--Collectors report what the kernel exposes, without reading any logs:--| Check | Source | Notes |-|---|---|---|-| Reboots | `/proc/sys/kernel/random/boot_id` | A new boot ID between check-ins records a **Reboot** event. |-| OOM kills | `oom_kill` in `/proc/vmstat` | A rising counter records an **OOM kill** event. |-| Kernel faults | `/proc/sys/kernel/tainted` | Only fault bits count (machine check, bad page, oops, warning, soft lockup); proprietary or out-of-tree module flags are ignored. |-| Clock sync | `adjtimex` (read-only), else systemd-timesyncd's marker | Sync state only, not a measured offset. |-| Reboot required | `/run/reboot-required` | Debian and Ubuntu only; other distros show *unknown*. |--### Optional system helper--Process names, kernel error lines and crash-versus-clean shutdown detection need-the kernel log and the login records, which the collector deliberately can't-read. A separate container does:--```bash-docker compose --profile system-events up -d # add -f docker-compose.server.yml on a server-```--It runs with a single capability (`CAP_SYSLOG`), no network and a read-only-filesystem, sees only `/var/log/wtmp` (read-only), and writes a sanitised JSON-summary that the collector reads from a shared volume. With it you also get:--- the OOM-killed process name on each **OOM kill** event,-- recent kernel error lines in the **System** row (informational, they don't open incidents),-- **Last shutdown: clean / unclean**, and a **critical** incident after an unclean reboot.--Without it those show as *unknown* and everything else keeps working. Read-[SECURITY.md](SECURITY.md) before enabling it; the helper is about 200 lines of-standard-library Python.--<ui - The System row opened on a server with the helper running: status chips on top, then a list with an OOM kill (process name) and kernel error lines>--## Accounts and roles--- The first account created is the **admin**. It needs the install's **setup- token** (printed in the backend log at start, or `ADMIN_BOOTSTRAP_TOKEN`). Later- accounts are regular users (and on the server file, registration closes after- the first account).-- **Passwords must be at least 16 characters** (and at most 72 bytes). A few random- words make a good passphrase. The rule applies to new accounts; existing- accounts keep working with their current password.-- **Account page** (sidebar → Account), for every user: change your password (needs- the current one; signs out your other devices), turn on **two-factor sign-in**,- and review or end your sessions (**Sign out everywhere else**). See- [Two-factor sign-in](#two-factor-sign-in).-- Everyone signed in can see all servers, containers and incidents.-- Only admins can add, approve, dismiss or delete servers, change check- intervals, manage the enrollment key and open the **Admin** page.-- **Admin** page settings that are connected today: alert thresholds (including- per-mount disk thresholds), how long metrics are kept (default 30 days; older- check-in samples and system events are deleted hourly, but only after the check-ins have- been summarised; the summaries behind the Statistics page are not deleted yet, and a- retention setting for each level is planned), and turning- chat with the assistant on or off. The other sections on the page show- placeholder data and aren't saved yet.--### Two-factor sign-in--Two-factor uses a one-time code from any authenticator app (TOTP, the same standard-as Google Authenticator, Aegis or a password manager).--1. **Create a key once.** Secrets are stored encrypted, so the stack needs a key.- Generate one and put it in `.env` as `SECRETS_KEY` (see `example.env`):-- ```bash- docker compose run --rm --no-deps backend python -m src.backend.secrets_box- ```-- Back this key up with your other secrets. **If you lose it, authenticator codes- stop working** (recovery codes still do, and so does the reset command below).- To rotate it, put the new key first and keep the old one after a comma.-2. **Turn it on** under Account → Two-factor sign-in. You get a setup key and an- `otpauth://` link to paste into your authenticator app (no QR code yet), confirm- with a code, and are shown **10 recovery codes once**. Each works one time.-3. **Signing in** then asks for the code after the password. A recovery code can be- typed instead.--**Admins must use it by default.** *Admin → Users → Require two-factor for admins* is-**on**. An admin without two-factor can still sign in, but only to the Account page-until they finish setting it up, so nobody is locked out. A member's two-factor is-optional. The setting is only enforced when `SECRETS_KEY` is set; without it the-backend prints a warning at start and the policy does nothing.--**Locked out?** An admin who lost both the authenticator and the recovery codes can-reset an account from the host:--```bash-docker compose exec backend python src/backend/reset_2fa.py [email protected]-```--It turns two-factor off for that account and signs it out everywhere. With the-policy on, the account is asked to set up two-factor again at its next sign-in.--## Configuration--### Stack (`.env` next to the Compose file)--| Variable | Default | Description |-|---|---|---|-| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. |-| `GEMINI_API_KEY` | empty | Enables the Homie chat assistant. |-| `AGENT_MODEL` | `models/gemini-3.5-flash` | Gemini model for the assistant. |-| `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. |-| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. |-| `SESSION_TTL_SECONDS` | `86400` | Session lifetime. |-| `ALLOW_REGISTRATION` | `true` (dev), `false` (server file) | Let anyone create an account. When `false`, only the first account can be created. |-| `SECRETS_KEY` | empty | Key that encrypts two-factor secrets. Required for two-factor sign-in and for the "require two-factor for admins" policy to be enforced. Generate with `python -m src.backend.secrets_box`; comma-separated keys allow rotation. |-| `ADMIN_BOOTSTRAP_TOKEN` | generated at start | Setup token for the first account. Leave empty to have one generated and printed in the backend log until the first account exists. |-| `FRONTEND_BIND` | `0.0.0.0` (dev), `127.0.0.1` (server file) | Interface for the dashboard's plain-HTTP port. |-| `BACKEND_PORT` | `8060` | Development stack only: the host port (on `127.0.0.1`) where the backend is published for debugging. The server stack publishes no backend port. |-| `FRONTEND_PORT` | `5173` | Dashboard port on the host. |-| `DASHBOARD_DOMAIN` | *(required on a server)* | Domain of the dashboard, without `https://` (server file). |-| `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERT_RESOLVER` / `TRAEFIK_NETWORK` / `TRAEFIK_ROUTER` | `websecure` / `letsencrypt` / `traefik_default` / `homelab` | Your Traefik names (server file). |-| `CORS_ORIGINS` | `http://localhost:5173` (dev), `https://<DASHBOARD_DOMAIN>` (server file) | Comma-separated origins. Also the allow-list for the cross-site request check: a state-changing request from any other origin is refused unless its `Origin` matches the `Host` the dashboard was reached on. Add your dashboard URL if something in front rewrites `Host`. |-| `LOGIN_RATE_LIMIT` / `LOGIN_RATE_WINDOW_SECONDS` | `5` / `60` | Login attempts per IP and per email. |-| `REGISTER_RATE_LIMIT` / `REGISTER_RATE_WINDOW_SECONDS` | `3` / `3600` | Registrations per IP. |-| `COLLECTOR_BIND` / `COLLECTOR_PORT` | `127.0.0.1` / `8081` | Server file: address and port where the collector gate is published. Set the address to your VPN's. |-| `COLLECTOR_PUBLIC_URL` | empty | What **Connect a server** tells new machines to use, e.g. `http://10.8.0.1:8081/api`. |-| `COLLECTOR_IMAGE` / `DOCKER_PROXY_IMAGE` | empty | Digest-pinned images remote machines install. **Connect a server** needs both. |-| `JOIN_TOKEN_TTL_SECONDS` | `900` | How long a generated join token stays valid. |-| `COLLECTOR_ACCESS` | `open` (dev), `strict` (server file) | `strict` makes the backend answer collector endpoints only when they came through the collector gate. |-| `HOMELAB_ENROLL_KEY` | empty | Lets the stack's own collector skip approval. |-| `HOMELAB_SERVER_NAME` | host's hostname | Name shown for the stack's own machine. |-| `DOMAIN` | `localhost` | Domain for the optional Caddy profile. |-| `COMPOSE_FILE` | – | Set to `docker-compose.server.yml` on a server so plain `docker compose` uses it. |-| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes (server file; applied when the frontend image is built). |-| `HOMELAB_CHECK_INTERVAL_SECONDS` | `60` | Initial check interval of the stack's own collector (server file); the dashboard setting overrides it. |--### Collector (environment of the collector container)--| Variable | Default | Description |-|---|---|---|-| `HOMELAB_API_URL` | `http://localhost:8060` | Dashboard API address, e.g. `https://dashboard.example.com/api`. |-| `HOMELAB_JOIN_TOKEN` | empty | Single-use token from **Connect a server**; required over the VPN listener. |-| `HOMELAB_ENROLL_KEY` | empty | Join immediately instead of waiting for approval. |-| `HOMELAB_SERVER_NAME` | host's `/etc/hostname` | Name shown on the dashboard. |-| `HOMELAB_CONNECTIVITY_TARGETS` | empty (check off) | Optional internet check: up to 5 comma-separated `host:port` targets (hostnames or IPv4), e.g. `1.1.1.1:443,9.9.9.9:443`. The collector opens a TCP connection and looks the name up; it sends nothing. Set on each collector; the dashboard cannot set it. |-| `HOMELAB_STATE_DIR` | `/var/lib/homelab-collector` | Where the token is saved (mount a volume here). |-| `HOMELAB_MACHINE_ID` | host's `/etc/machine-id` | Identity used to recognise reinstalled machines. |-| `HOMELAB_DASHBOARD_HOST` | empty | `true` marks this machine as the one running the dashboard (set in the Compose files for the stack's own collector; leave it unset elsewhere). |-| `HOMELAB_SERVER_ID` / `HOMELAB_COLLECTOR_TOKEN` | empty | Explicit credentials (legacy; overrides everything else). |-| `HOST_PROC` / `HOST_ROOT` | `/host/proc` and `/hostfs` if mounted, else `/proc` and `/` | Where the host's `/proc` and filesystem are. |-| `HOMELAB_DOCKER_HOST` | empty | Socket of the read-only Docker proxy, e.g. `/run/docker-proxy/docker.sock`. Preferred over `DOCKER_SOCKET_PATH`. |-| `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Direct Docker socket, used only when `HOMELAB_DOCKER_HOST` is unset (not recommended). Without any Docker access only host metrics are sent. |-| `HOMELAB_HELPER_FILE` | `/helper/system.json` | Where the optional system helper's summary is read from. |--The system helper has `HOMELAB_HELPER_INTERVAL_SECONDS` (default `30`, minimum-`10`); the Docker proxy listens on `HOMELAB_DOCKER_PROXY_SOCKET` and forwards to-`DOCKER_SOCKET_PATH`. Both are set in their images and rarely need changing.--The check interval is set per server on the dashboard; collectors pick it up-from the backend's response to each report.+| [Metrics: what every number means](docs/metrics.md) | How CPU %, load, memory, swap, disk, inodes, network and container figures are measured, and their limits. |+| [Statistics](docs/statistics.md) | History from the last hour to all time, per server, per container and for the whole fleet. |+| [The dashboard and system health](docs/dashboard.md) | Fleet cards, storage, the Network and System rows, the optional system helper. |+| [Alerts and incidents](docs/alerts.md) | Every rule and its default thresholds. |+| [Accounts and two-factor sign-in](docs/accounts.md) | Roles, passwords, authenticator codes, recovery and reset. |+| [Deploying](docs/deployment.md), [Adding servers](docs/adding-servers.md), [Configuration](docs/configuration.md), [Troubleshooting](docs/troubleshooting.md) | Installing, running and fixing it. |+| [Development](docs/development.md), [Database schema](docs/database.md), [API overview](docs/api.md), [Homie assistant](docs/assistant.md) | For contributors. |## Security-[SECURITY.md](SECURITY.md) has the full picture. In short:--- **Collector privileges.** In the Compose installs the collector has no Docker- socket, drops all capabilities and has a read-only filesystem. Only the Docker- proxy mounts the socket; it has no network, answers four read-only requests and- strips environment variables, labels, mounts and error text from what it returns.- Everything a collector reports about containers is validated again by the backend- (types, ranges, lengths, counts) before it is stored.-- **Network checks are passive.** Listening ports are read from the host's `/proc`- (protocol, port and scope only, no process names), the firewall check reads known- configuration files and reports only a hash and a rule count (never the rules, and not- the live ruleset, which would need `CAP_NET_ADMIN`), and the internet check is off- unless you configure its targets on that collector. The collector still mounts `/`- read-only; without capabilities it can't read other users' private files through it.-- **Collector endpoints are not public.** On the server stack the public listener- returns 404 for register, poll, enroll and ingest; they exist only on the- collector gate, bound to your VPN address.-- **No code from the dashboard.** Remote machines install signed, digest-pinned- images. The dashboard no longer serves collector code.-- **Join tokens.** Adding a machine needs a single-use, expiring token minted by an- admin, then an admin's click. Re-pairing a machine that matches an existing- server needs explicit confirmation.-- **Fresh installs can't be claimed.** The first account needs a setup token.-- **Accounts.** Passwords must be at least 16 characters. Two-factor sign-in- (authenticator app plus single-use recovery codes) is available to everyone and- **required for admins by default**; secrets are encrypted with your `SECRETS_KEY`. Every logged-in account- can see all monitoring data and use the assistant (your Gemini key) unless an- admin turns chat off. Only admins can add, approve or remove servers. With- `ALLOW_REGISTRATION=false` (the default in `docker-compose.server.yml`) only the- first account can be created; keep it that way on a public domain.-- **Browser protections.** State-changing requests from another site are refused- (this covers other apps on sibling subdomains), and the frontend sends a strict- Content-Security-Policy, clickjacking protection and HSTS on the public listener.-- **Secrets at rest.** Collector tokens, join tokens, the setup token and- enrollment keys are stored only as SHA-256 hashes; passwords are hashed with bcrypt.-- **Client IPs.** nginx accepts `X-Forwarded-For` only from private/Docker- network addresses (e.g. Traefik), so rate limits see the real client IP and it- can't be spoofed from outside.-- **Known limits.** The one-line `docker run` install is gone; machines you install- with `collector-compose.yml` build locally. Anyone on your VPN can reach the- collector gate (a token and an admin's click are still needed to join).- `src/backend/seed_user.py` creates an admin straight in the database (from- `SEED_ADMIN_EMAIL` and `SEED_ADMIN_PASSWORD`, or an interactive prompt), skipping the- setup token; it is meant for people who already have database access.--### Homie assistant (monitoring-only)--The assistant has no code path for running shell commands. It may request-exactly one fixed, read-only check at a time: `server_health`,-`container_health`, or `daily_checkup`, implemented with Python system APIs and-Docker's Unix API socket, never the shell or Docker CLI.--The backend container does **not** mount `/var/run/docker.sock`. If you want-the assistant's container checks, point `DOCKER_SOCKET_PATH` at a hardened-Docker API proxy that permits only the required GET endpoints.--`daily_checkup` combines host and container checks. On a critical finding it-POSTs an alert only to `ADMIN_ALERT_WEBHOOK_URL`; without it, the result says-`not_configured` and no request is made. Schedule it from a trusted scheduler-(e.g. a systemd timer) rather than letting the agent create schedules.--## Development--### Backend without Docker--Python 3.12. The backend needs Redis for sessions and rate limits; SQLite works-in place of PostgreSQL.--```bash-python -m venv venv && source venv/bin/activate-pip install -r requirements.txt-docker run -d --name homelab-dev-redis -p 6379:6379 redis:7-alpine # if Redis isn't running-DATABASE_URL=sqlite:///./dev.db REDIS_URL=redis://localhost:6379/0 \- uvicorn src.backend.main:app --reload --host 0.0.0.0 --port 8060-```--### Frontend dev server+[SECURITY.md](SECURITY.md) has the full picture: what each component can touch, and the limits of each+protection. In short:-Node 20 (`.nvmrc`). The Vite dev server proxies `/api` to `localhost:8060` (set `BACKEND_URL` to point it elsewhere),-matching the production nginx setup.+- **Collectors are unprivileged.** No Docker socket, no capabilities, a read-only filesystem. Only a separate+ read-only proxy (no network, four allowed requests, stripped answers) touches the socket.+- **Collector endpoints are not public.** On the server stack they exist only on the gate bound to your VPN+ address, and joining needs a single-use token plus an admin's click. Remote machines install signed,+ digest-pinned images; the dashboard never serves code that runs on your hosts.+- **Fresh installs can't be claimed** (the first account needs a setup token), passwords need 16 characters,+ and two-factor sign-in is **required for admins by default**.+- **Browser protections:** cross-site requests are refused, with a strict Content-Security-Policy and HSTS.+- **Secrets at rest:** tokens stored only as hashes, passwords with bcrypt, two-factor secrets encrypted with your `SECRETS_KEY`.-```bash-nvm install && nvm use-cd src/frontend-npm install-npm run dev -- --host 0.0.0.0-```--Vite prints the URL (5173, or the next free port if the Compose stack already-uses it). `./start_frontend.sh` does the same in one step.--Environment variables for the frontend build:--- `VITE_API_BASE_URL` (default `/api`): API base path or URL.-- `VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` (default `3`, minimum `1`): how- often the dashboard refreshes.+## Development and tests-After changing frontend or collector code, rebuild the containers:-`docker compose up -d --build`. Otherwise the browser keeps getting the old bundle.--### Tests+Backend (Python 3.12) and frontend (Node 20) can run without Docker; see [Development](docs/development.md).```bash-source venv/bin/activate-pip install -r requirements-dev.txt # the app's requirements plus pytest-pytest-```--No Docker, PostgreSQL or Redis is needed; the suite takes about a minute and a half. The chart-helpers have their own tests that need only Node 20 or newer: `cd src/frontend && node --test`. See-[TESTS.md](TESTS.md) for how the tests work, what they don't cover, and what every-test is for.--### Database schema--Tables are created on backend startup (`Base.metadata.create_all`), and new-columns are added with `ALTER TABLE ... IF NOT EXISTS` in the startup hook in-`src/backend/main.py`. Each check-in stores one row in `host_metrics` (which also-carries the boot ID, OOM counter, kernel flags, clock state and last shutdown), one-per filesystem in `mount_metrics` and one per container in `container_metrics` (which-also records the exit code, out-of-memory flag, finish time and image ID), all with the-same timestamp. `system_events` holds reboots, OOM kills, kernel messages, container-changes (appeared, removed, image changed) and network changes (ports opened or closed,-firewall changes); `listener_states` and `firewall_states` hold what was last seen, `join_tokens` the hashed single-use tokens, `recovery_codes` the hashed two-factor-recovery codes (the encrypted two-factor secret lives on the `users` row), and per-mount disk-thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises-the `users` table on a fresh volume.--**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in-samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container-per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.-Each tier is built from the finer one when its period ends, a period without check-ins gets no row, and the-purge never deletes samples that have not been summarised yet. Nothing displays these yet; they are the-foundation for the long-range graphs and statistics.--The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and-`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are-migrated at start: the new indexes are created and the redundant single-column ones are dropped,-which can be repeated safely. Creating an index briefly blocks writes to that table (a few-seconds for a table of millions of rows); on a very large install, create the indexes yourself-first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names-in `src/backend/models.py`, and the start-up migration will skip them.--## Project layout--```text-compose.yml development stack (default for `docker compose`)-docker-compose.server.yml server stack behind Traefik-collector-compose.yml collector-only stack for monitored machines-SECURITY.md what each component can touch, and its limits-TESTS.md how to run the tests and what each one checks-requirements-dev.txt app requirements plus pytest, for running the tests-pytest.ini test settings (lets plain `pytest` find the `src` package)-.github/workflows/- images.yml builds, signs and publishes the collector images on a version tag-docker/- backend.Dockerfile FastAPI backend (uvicorn)- frontend.Dockerfile Vite build served by nginx- collector.Dockerfile standalone collector image- docker-proxy.Dockerfile read-only Docker API proxy image- system-helper.Dockerfile optional kernel-log / wtmp helper image- nginx.conf development: serves the UI, proxies /api to the backend- nginx.strict.conf server stack: public listener, collector endpoints refused- nginx.collector.conf server stack: collector gate (VPN and internal listeners)- Caddyfile optional HTTPS front door (dev compose, --profile https)- postgres/init/ SQL run on a fresh PostgreSQL volume-src/- backend/ API: auth, monitoring, enrollment, rate limits, models- alerts.py alert engine: one incident per rule, auto-resolve- rules.py alert rules and default thresholds- rollups.py summarises check-ins into hourly/daily/weekly/monthly rollups- history.py range-aware history behind the Statistics page- stats.py fleet statistics: incidents over time, availability, busy-hours heatmap- totp.py one-time codes and recovery codes (standard library only)- secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key- reset_2fa.py command-line two-factor reset for a locked-out account- agent/- collector.py metrics collector (standard library only)- docker_proxy.py read-only Docker API proxy (standard library only)- system_helper.py optional kernel-log / wtmp helper (standard library only)- health_checks.py read-only checks available to the assistant- service.py, prompts.py Gemini assistant- frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the- charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,- FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the page-tests/ pytest suite+source venv/bin/activate && pip install -r requirements-dev.txt && pytest # backend, about a minute and a half+cd src/frontend && node --test # chart helpers, nothing to install```-## API overview--All paths are relative to `/api` when going through the frontend/nginx.--| Method & path | Auth | Purpose |-|---|---|---|-| `GET /health` | – | Database and Redis status. |-| `POST /auth/register`, `POST /auth/login`, `POST /auth/logout`, `GET /auth/me` | session cookie | Accounts and sessions. |-| `POST /auth/login/2fa` | pending login token | Second step of sign-in: authenticator or recovery code. |-| `GET /auth/2fa/status`, `POST /auth/2fa/setup`, `POST /auth/2fa/enable` | session (also the limited setup session) | Two-factor status, start and confirm setup. |-| `POST /auth/2fa/disable`, `POST /auth/2fa/recovery-codes` | session + password + code | Turn two-factor off; replace the recovery codes. |-| `POST /auth/password` | session + current password | Change password; signs out other sessions. |-| `GET /auth/sessions`, `POST /auth/sessions/revoke-others`, `DELETE /auth/sessions/{id}` | session | List and end your own sessions. |-| `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. |-| `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |-| `GET /monitoring/servers/{id}/history` | session | A server's metrics over time. With `?range=` (`1h`, `6h`, `24h`, `7d`, `30d`, `90d`, `1y`, `all`) it returns at most about 500 points from the right level of detail, each with average and maximum of the requested `metrics` (`cpu,ram,swap,load,disk,inodes,docker_ms,collect_ms`). Without `range`: CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |-| `GET /stats/fleet` | session | Incidents opened per bucket and severity, by kind, open now, mean time to resolve, and per-server availability for `?range=`. Cached for a minute. |-| `GET /stats/servers/{id}/heatmap` | session | Average `?metric=cpu\|ram` by hour of the day and weekday over the last 30 days, for `?offset=` minutes from UTC. |-| `GET /monitoring/servers/{id}/mounts/history`, `GET /monitoring/servers/{id}/containers/{container_id}/history` | session | The same for one filesystem (`?mount=/data`) or one container. |-| `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). |-| `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. |-| `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |-| `DELETE /monitoring/servers/{id}` | admin | Remove a server and its data. |-| `POST /monitoring/pending-collectors/{id}/approve` / `…/dismiss` | admin | Add or dismiss a server asking to join. Approving one that matches an existing server needs `{"replace": true}`. |-| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). |-| `GET /monitoring/install-info` | admin | The collector address and images the panel uses. |-| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |-| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds. |-| `GET /agent/config` | session | Whether chat with the assistant is enabled. |-| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |-| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |-| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |-| `POST /monitoring/metrics/ingest` | collector token | Collector reports host, container and system metrics. |--The four collector endpoints above exist only on the collector gate in the server-stack (the public listener answers 404); the development stack serves them on the-same address as everything else.-| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Homie assistant chats. |-| `GET /logs`, `POST /logs`, `GET /dashboard/summary` | session | Audit log and summary counts. |--## Troubleshooting--| Symptom | Check |-|---|---|-| A new machine never shows up under **New servers detected** | `docker logs homelab-collector` on that machine. "Could not reach …" means the address in the command isn't reachable from it (firewall, `localhost`, wrong IP/domain). |-| Collector log says "Backend no longer accepts server …" | The server was deleted on the dashboard; it is asking to join again, so click **Add** or **Dismiss**. |-| `network traefik_default declared as external, but could not be found` | You're using `docker-compose.server.yml` on a machine without that Traefik network. Use `compose.yml` for development, or set `TRAEFIK_NETWORK` in `.env` to the network Traefik is really on. |-| Dashboard shows old behaviour after a code change or `git pull` | Rebuild **all** services with `docker compose up -d --build` (with `-f docker-compose.server.yml` on a server unless `COMPOSE_FILE` is set) and hard-refresh the browser (Ctrl+Shift+R). |-| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec homelab-collector grep -c dashboard_host /app/collector.py` prints `0`. Rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). |-| A server shows only `/` and no **Storage** bars for other disks | Its collector predates per-mount reporting. Install a newer collector image (run a freshly generated command, or `docker compose -f collector-compose.yml up -d --build`). A disk mounted after the collector started appears after a collector restart. |-| Logged out after opening the dashboard at a new address | Login cookies belong to the address they were set on; log in again. |-| Login fails behind HTTPS with `SESSION_COOKIE_SECURE=true` | The cookie is only sent over HTTPS; open the `https://` address, not `http://…:5173`. |-| Collector log says "Registration refused (HTTP 403): The join token is invalid, expired or already used" | The token in the command has expired, was already used, or is wrong. Click **Generate command** again and run the new command. |-| **Connect a server** says no collector images are configured | Set `COLLECTOR_IMAGE` and `DOCKER_PROXY_IMAGE` in `.env` (see [Publishing the collector images](#publishing-the-collector-images)) and rebuild. |-| The register page asks for a setup token | The install has no accounts yet. Use `docker compose logs backend \| grep "setup token"`, or `ADMIN_BOOTSTRAP_TOKEN`. |-| "Password must be at least 16 characters long" | New passwords need 16 or more characters (at most 72 bytes). A few random words work well. |-| "Cross-site request refused" (403) on a button or form | The request came from another origin. Open the dashboard at the address you normally use. If a proxy in front rewrites the `Host` header, add your dashboard URL to `CORS_ORIGINS`. |-| The **System** row shows *unknown* for last shutdown or has no kernel messages | The optional system helper isn't running. Start it with `docker compose --profile system-events up -d`. *Unknown* for reboot-required on non-Debian hosts is expected. |-| `Error response … cannot assign requested address` for the collector gate | `COLLECTOR_BIND` is set to an address that doesn't exist yet (VPN not up). Start the VPN first, or fix the address. |-| The **Network** row says *no config found* for the firewall | None of the known firewall files exist on that host (`/etc/ufw`, `/etc/nftables.conf`, `/etc/iptables/rules.v4`). The check reads configuration, not the live ruleset, so a host that configures its firewall some other way shows this. |-| The **Network** row says *not checked* for the internet | The check is opt-in. Set `HOMELAB_CONNECTIVITY_TARGETS` on that collector (for example `1.1.1.1:443`). |-| Listeners show *unknown* or the Network row is empty | The collector couldn't read the host's network tables (older collector, or a table with more than 500,000 sockets). An unreadable report never raises a "port closed" alert. |-| "Invalid code." although the code looks right | Codes depend on the clock: check the phone's time is automatic. A code also can't be used twice, so wait for the next one. Five wrong codes lock the code step for 5 minutes. |-| Two-factor setup says it is not available | `SECRETS_KEY` is missing or invalid in `.env`. Generate one (see [Two-factor sign-in](#two-factor-sign-in)) and restart the backend. The backend log shows a SECURITY WARNING while the key is missing. |-| Lost the authenticator and the recovery codes | Run `docker compose exec backend python src/backend/reset_2fa.py <email>` on the host, then sign in with the password and set it up again. |-| Authenticator codes stopped working after a restore or `.env` change | `SECRETS_KEY` changed or was lost, so the stored secret can't be decrypted. Restore the old key (rotation: `new,old`), or use a recovery code and the reset command. |-| Server marked offline | It missed 3 check-ins in a row. Check the machine, its collector log and its route to the dashboard. The incident resolves on the next report. |+[TESTS.md](TESTS.md) explains how the tests work, what they don't cover and what each one is for.## Licensediff --git a/SECURITY.md b/SECURITY.mdindex abd79aa..4781e2d 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -228,6 +228,15 @@ The collector's network checks are passive or opt-in, and add no privileges:- **Visibility.** Like all monitoring data, the list of listening ports and firewall state is visible toevery signed-in account. Keep registration closed on a public dashboard.+## Client addresses and collector installs++- **Client IPs.** nginx accepts `X-Forwarded-For` only from private or Docker network addresses (for+ example Traefik), so login and registration rate limits see the real client address and it can't be+ spoofed from outside.+- **No one-line installer.** The old `docker run` install is gone. Remote machines run the compose+ command **Connect a server** generates (signed, digest-pinned images), and machines set up from+ `collector-compose.yml` build the images locally.+## Statistics and history- **Who can see it.** Every signed-in account sees the same statistics as the dashboard, for everydiff --git a/docs/README.md b/docs/README.mdnew file mode 100644index 0000000..d262af9--- /dev/null+++ b/docs/README.md@@ -0,0 +1,33 @@+# Documentation++The [README](../README.md) is the short version: what the project is, how to start it. Everything else lives here.++## Using the dashboard++| Page | What is in it |+|---|---|+| [The dashboard and system health](dashboard.md) | Fleet cards, storage bars, the Network and System rows, active incidents, the optional system helper. |+| [Statistics](statistics.md) | History from the last hour to all time: servers, containers and the whole fleet. |+| [Metrics: what every number means](metrics.md) | CPU %, load, memory, swap, disk, inodes, network, container and system figures: how each is measured and its limits. |+| [Alerts and incidents](alerts.md) | Every rule, its default thresholds, and how incidents open and resolve. |+| [Accounts, roles and two-factor sign-in](accounts.md) | Passwords, roles, the Account page, authenticator codes, recovery codes, locked-out accounts. |+| [Homie assistant](assistant.md) | The optional, monitoring-only AI assistant. |++## Installing and running++| Page | What is in it |+|---|---|+| [Deploying on a server](deployment.md) | The server stack behind Traefik, the VPN collector gate, publishing the collector images, updating, running without Traefik. |+| [Adding servers](adding-servers.md) | **Connect a server**, what its command does, how registration and approval work, networking. |+| [Configuration](configuration.md) | Every `.env` setting for the stack, the collector and the frontend build. |+| [Troubleshooting](troubleshooting.md) | Symptoms and what to check. |++## For developers++| Page | What is in it |+|---|---|+| [Development](development.md) | Running the backend and frontend without Docker, tests, project layout. |+| [Database schema](database.md) | Tables, summarised history, indexes and migrations. |+| [API overview](api.md) | Every endpoint, who may call it, and what it returns. |+| [../TESTS.md](../TESTS.md) | How the tests work, what they don't cover, and what each one checks. |+| [../SECURITY.md](../SECURITY.md) | What every component can touch, and the limits of each protection. |diff --git a/docs/accounts.md b/docs/accounts.mdnew file mode 100644index 0000000..35873e9--- /dev/null+++ b/docs/accounts.md@@ -0,0 +1,62 @@+[← Documentation](README.md)++# Accounts and roles++- The first account created is the **admin**. It needs the install's **setup+ token** (printed in the backend log at start, or `ADMIN_BOOTSTRAP_TOKEN`). Later+ accounts are regular users (and on the server file, registration closes after+ the first account).+- **Passwords must be at least 16 characters** (and at most 72 bytes). A few random+ words make a good passphrase. The rule applies to new accounts; existing+ accounts keep working with their current password.+- **Account page** (sidebar → Account), for every user: change your password (needs+ the current one; signs out your other devices), turn on **two-factor sign-in**,+ and review or end your sessions (**Sign out everywhere else**). See+ [Two-factor sign-in](accounts.md#two-factor-sign-in).+- Everyone signed in can see all servers, containers and incidents.+- Only admins can add, approve, dismiss or delete servers, change check+ intervals, manage the enrollment key and open the **Admin** page.+- **Admin** page settings that are connected today: alert thresholds (including+ per-mount disk thresholds), how long metrics are kept (default 30 days; older+ check-in samples and system events are deleted hourly, but only after the check-ins have+ been summarised; the summaries behind the Statistics page are not deleted yet, and a+ retention setting for each level is planned), and turning+ chat with the assistant on or off. The other sections on the page show+ placeholder data and aren't saved yet.++## Two-factor sign-in++Two-factor uses a one-time code from any authenticator app (TOTP, the same standard+as Google Authenticator, Aegis or a password manager).++1. **Create a key once.** Secrets are stored encrypted, so the stack needs a key.+ Generate one and put it in `.env` as `SECRETS_KEY` (see `example.env`):++ ```bash+ docker compose run --rm --no-deps backend python -m src.backend.secrets_box+ ```++ Back this key up with your other secrets. **If you lose it, authenticator codes+ stop working** (recovery codes still do, and so does the reset command below).+ To rotate it, put the new key first and keep the old one after a comma.+2. **Turn it on** under Account → Two-factor sign-in. You get a setup key and an+ `otpauth://` link to paste into your authenticator app (no QR code yet), confirm+ with a code, and are shown **10 recovery codes once**. Each works one time.+3. **Signing in** then asks for the code after the password. A recovery code can be+ typed instead.++**Admins must use it by default.** *Admin → Users → Require two-factor for admins* is+**on**. An admin without two-factor can still sign in, but only to the Account page+until they finish setting it up, so nobody is locked out. A member's two-factor is+optional. The setting is only enforced when `SECRETS_KEY` is set; without it the+backend prints a warning at start and the policy does nothing.++**Locked out?** An admin who lost both the authenticator and the recovery codes can+reset an account from the host:++```bash+docker compose exec backend python src/backend/reset_2fa.py [email protected]+```++It turns two-factor off for that account and signs it out everywhere. With the+policy on, the account is asked to set up two-factor again at its next sign-in.diff --git a/docs/adding-servers.md b/docs/adding-servers.mdnew file mode 100644index 0000000..0d48c89--- /dev/null+++ b/docs/adding-servers.md@@ -0,0 +1,120 @@+[← Documentation](README.md)++# Adding servers++Adding and removing servers, changing check intervals and managing the+enrollment key are admin-only.++1. On the dashboard, click **Connect a server**. Check the address field: with+ `COLLECTOR_PUBLIC_URL` set it shows your VPN address; otherwise it is prefilled+ with the address you opened the dashboard at and must be reachable from the+ new machine (not `localhost`).+2. Click **Generate command**. This creates a single-use token that expires+ (15 minutes by default) and puts it in the command.+3. Copy the command and run it on the new machine. It needs Docker with the+ compose plugin, and a route to the address above (the VPN).+4. Within a few seconds the machine appears under **New servers detected** with a+ **JOIN TOKEN** badge and a pairing code (also printed in+ `docker logs homelab-collector`). Click **Add**. It starts reporting on its+ next check, without a restart.++<ui - The Connect a server panel: the address field, the Generate command button with its single-use / expires note, and the copyable compose command>++<ui - The New servers detected list with one pending machine showing the JOIN TOKEN badge, its pairing code and the Add / Dismiss buttons>++The machine running the dashboard is added the same way (its collector starts+with the stack, over the gate's internal listener). That collector has+`HOMELAB_DASHBOARD_HOST=true` set in the Compose files, which makes it the machine+shown in the charts at the top of the dashboard and first in the server list.+Collectors installed on other machines don't set it.++## What the command does++It starts a small compose project named `homelab-collector` from the YAML on its+standard input (shortened here):++```bash+docker compose -p homelab-collector -f - up -d <<'EOF'+services:+ collector:+ image: ghcr.io/…/collector@sha256:…+ environment:+ HOMELAB_API_URL: http://10.8.0.1:8081/api+ HOMELAB_JOIN_TOKEN: hlj_…+ HOMELAB_DOCKER_HOST: /run/docker-proxy/docker.sock+ volumes:+ - /proc:/host/proc:ro+ - /:/hostfs:ro+ - collector_state:/var/lib/homelab-collector+ - docker_proxy_socket:/run/docker-proxy:ro+ cap_drop: [ALL]+ read_only: true+ docker-proxy:+ image: ghcr.io/…/docker-proxy@sha256:…+ network_mode: none+ cap_drop: [ALL]+ read_only: true+ volumes:+ - /var/run/docker.sock:/var/run/docker.sock+ - docker_proxy_socket:/run/docker-proxy+EOF+```++| Part | Purpose |+|---|---|+| `HOMELAB_API_URL` | Where the collector gate is. |+| `HOMELAB_JOIN_TOKEN` | Proves an admin asked this machine to join. Single use, expires. |+| `image: …@sha256:…` | Signed images pinned by digest. No code is downloaded from the dashboard. |+| `/proc:/host/proc:ro` | Host CPU, memory, load, network, uptime, boot ID, OOM counter and kernel flags. |+| `/:/hostfs:ro` | Disk usage of every mounted filesystem, hostname, OS name and `/etc/machine-id`. |+| `docker-proxy` | The only container with the Docker socket. It answers container list, inspect (without environment variables) and stats, and nothing else. |+| `cap_drop: [ALL]`, `read_only: true` | The collector can't read other users' private files through `/hostfs`, or write outside its state volume. |+| `collector_state` volume | Keeps the token after approval, so restarts don't need approval again. |++To remove a collector completely:++```bash+docker rm -f homelab-collector homelab-docker-proxy+docker volume rm homelab-collector_collector_state homelab-collector_docker_proxy_socket+```++## How registration works++- Over the VPN a collector can't even ask to join without a valid join token. It+ then asks, checking every 5 seconds whether it has been approved. Clicking+ **Add** creates the server and uses up the token; the collector receives its own+ token and saves it. If a token is wrong, expired or used, the collector prints+ why and you copy a fresh command.+- Servers are matched on `/etc/machine-id`. If a new request matches a server you+ already have, the dashboard warns that approving it **replaces that server's+ credentials and signs out its current collector**, and asks you to confirm.+ Only do that when you are reinstalling that machine.+- If you delete a server, its collector can no longer report. On a server install+ it can't ask to join again without a new command (its token was used up). In the+ development stack it asks again and only comes back if you click **Add**.+ **Dismiss** moves a request to a collapsed "Dismissed" list.++<diagram - Sequence diagram of joining: admin clicks Generate command → join token created → command run on the new machine → collector POST /collectors/register with the token over the VPN listener → pending entry with pairing code → collector polls every 5 s → admin clicks Add (token consumed) → poll returns the token → collector saves it and starts POST /metrics/ingest; include the delete → 404 → needs a new command loop>+- Requests from collectors that stopped asking disappear after 2 minutes and are+ deleted after a day.++Alternatives:++- **Enrollment key (no approval step):** for scripted installs, generate a key+ under **Connect a server → Advanced** and add `HOMELAB_ENROLL_KEY: <key>` to the+ collector's environment. Only a hash of the key is stored; rotating or revoking+ it does not affect servers that are already connected.+- **Compose from the repository:** copy the repository to the machine, set+ `HOMELAB_API_URL` (and `HOMELAB_JOIN_TOKEN`) in a `.env` next to+ `collector-compose.yml`, and run+ `docker compose -f collector-compose.yml up -d --build`. This builds the images+ locally instead of pulling them.++## Networking checklist++- Every monitored machine must reach the collector gate (or the dashboard address+ in the development stack); the dashboard never connects to them.+- For machines outside your LAN, put them on a VPN (OpenVPN, WireGuard, Tailscale)+ and publish the gate on the VPN address. Don't expose plain HTTP to the+ internet: collector tokens travel over this connection.+- On a LAN, give the dashboard machine a fixed IP (e.g. a DHCP reservation).diff --git a/docs/alerts.md b/docs/alerts.mdnew file mode 100644index 0000000..7937373--- /dev/null+++ b/docs/alerts.md@@ -0,0 +1,78 @@+[← Documentation](README.md)++# Alerts and incidents++The values the rules look at are explained in [Metrics](metrics.md). Note that the load rule uses the 5-minute load per CPU core.++Rules run on every collector check-in. Each problem opens at most one incident+(per server, per rule and, for disks, per mount); while it lasts the incident is+updated instead of duplicated, and it resolves automatically once the rule+stops firing.++| Rule | Default | Severity |+|---|---|---|+| Disk usage, per mount (overridable per mount) | 85% / 95% | warning / critical |+| Inode usage, per mount | 85% / 95% | warning / critical |+| Memory usage | 90% / 95% | warning / critical |+| Swap usage | 50% | warning |+| 5-minute load per CPU core | 1.5× / 2× | warning / critical |+| Container restart loop | 3 restarts within 30 minutes | critical |+| Container healthcheck `unhealthy` | – | critical |+| Container exited with an error | within 60 minutes of exiting | warning |+| Container killed by the kernel for running out of memory | within 60 minutes | critical |+| Container stopped by a signal (exit 137 or 143, usually `docker stop`) | within 60 minutes | info |+| Container no longer present | stays open 60 minutes | info |+| Container running a different image build | stays open 60 minutes | info |+| Possible memory leak (steady climb over 12 hours) | 10 MB/h and 20% growth | warning |+| Docker unreachable (after it worked before) | 3 check-ins in a row | critical |+| Docker slow to list containers | over 2000 ms, 3 check-ins in a row | warning |+| Collector can't keep up with its check interval | gathering takes over 80% of the interval, 3 check-ins in a row | warning |+| New listener reachable from other machines | stays open 60 minutes | warning |+| Firewall configuration changed | stays open 60 minutes | info |+| Firewall protection reduced (ufw turned off, rules removed, a file gone) | stays open 60 minutes | warning |+| No internet (only if the check is on) | 3 check-ins in a row | warning |+| DNS failing (only if the check is on) | 3 check-ins in a row | warning |+| Server stopped reporting | 3 missed check-ins | critical |+| Host rebooted | stays open 60 minutes | info (critical after an unclean shutdown) |+| OOM killer fired | stays open 60 minutes | warning (critical at 3 or more) |+| Kernel fault flags (machine check, bad page, oops, warning, soft lockup) | – | warning, clears on reboot |+| Clock not synchronised | 3 check-ins in a row | warning |+| Reboot required | – | info |++New listening ports and firewall changes are noticed the same way, by comparing with+what the previous check-in saw. The first report is a silent baseline, an unreadable+or empty report is treated as unknown (never as "everything closed"), and short-lived+programs using random high ports (loopback ones and all UDP ones) are ignored. A new+port bound only to the loopback address is recorded but doesn't open an incident; one+reachable from other machines does. Events are capped like the container ones.++Container exits, removals and image changes are noticed by comparing each check-in+with the previous one, matching containers **by name** (Compose recreates them with+new IDs). Nothing is reported on a server's first check-in or when Docker was+unreachable (an empty list is not proof that everything vanished), and churn is+capped so containers that come and go constantly can't flood the database. A leak is+only flagged when hourly memory averages rise along a straight line without a+restart; caches and warm-up can look similar, so treat it as a prompt to look.++Host rules (disk, inodes, memory, swap, load) open an incident only after+**3 check-ins in a row** breach the threshold, so a brief spike is ignored. An+open incident stays open while the latest value is still over the warning+threshold, and becomes critical only when the critical threshold is breached for+the same number of check-ins.++<diagram - Incident lifecycle for a host rule as a timeline of check-ins: value crosses the warning threshold → after 3 check-ins in a row a WARNING incident opens → stays over critical for 3 check-ins → escalates to CRITICAL → drops under warning → RESOLVED; show a single spike that opens nothing>++Admins can change every threshold, the number of check-ins in a row, how long+event incidents (reboots, OOM kills, container exits, removals and image changes)+stay open, the slow-Docker and slow-collection limits, the memory-leak window and sensitivity, and+**per-mount disk thresholds** under+**Admin → Alerts**. A per-mount override replaces the default disk thresholds for+that mount path on every server. Changes apply from each server's next check-in+and are recorded in the audit log.++<ui - Admin → Alerts: the Incident thresholds form with Warning / Critical fields per rule, Check-ins in a row, and the container restart loop row>++<ui - Active incidents section with a CRITICAL and a WARNING incident, each showing its server and start time>++Incidents are shown on the dashboard only; there are no push notifications yet.+(`ADMIN_ALERT_WEBHOOK_URL` is used by the assistant's `daily_checkup` only.)diff --git a/docs/api.md b/docs/api.mdnew file mode 100644index 0000000..bd07b2e--- /dev/null+++ b/docs/api.md@@ -0,0 +1,41 @@+[← Documentation](README.md)++# API overview++All paths are relative to `/api` when going through the frontend/nginx.++| Method & path | Auth | Purpose |+|---|---|---|+| `GET /health` | – | Database and Redis status. |+| `POST /auth/register`, `POST /auth/login`, `POST /auth/logout`, `GET /auth/me` | session cookie | Accounts and sessions. |+| `POST /auth/login/2fa` | pending login token | Second step of sign-in: authenticator or recovery code. |+| `GET /auth/2fa/status`, `POST /auth/2fa/setup`, `POST /auth/2fa/enable` | session (also the limited setup session) | Two-factor status, start and confirm setup. |+| `POST /auth/2fa/disable`, `POST /auth/2fa/recovery-codes` | session + password + code | Turn two-factor off; replace the recovery codes. |+| `POST /auth/password` | session + current password | Change password; signs out other sessions. |+| `GET /auth/sessions`, `POST /auth/sessions/revoke-others`, `DELETE /auth/sessions/{id}` | session | List and end your own sessions. |+| `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. |+| `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |+| `GET /monitoring/servers/{id}/history` | session | A server's metrics over time. With `?range=` (`1h`, `6h`, `24h`, `7d`, `30d`, `90d`, `1y`, `all`) it returns at most about 500 points from the right level of detail, each with average and maximum of the requested `metrics` (`cpu,ram,swap,load,disk,inodes,docker_ms,collect_ms`). Without `range`: CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |+| `GET /stats/fleet` | session | Incidents opened per bucket and severity, by kind, open now, mean time to resolve, and per-server availability for `?range=`. Cached for a minute. |+| `GET /stats/servers/{id}/heatmap` | session | Average `?metric=cpu\|ram` by hour of the day and weekday over the last 30 days, for `?offset=` minutes from UTC. |+| `GET /monitoring/servers/{id}/mounts/history`, `GET /monitoring/servers/{id}/containers/{container_id}/history` | session | The same for one filesystem (`?mount=/data`) or one container. |+| `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). |+| `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. |+| `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |+| `DELETE /monitoring/servers/{id}` | admin | Remove a server and its data. |+| `POST /monitoring/pending-collectors/{id}/approve` / `…/dismiss` | admin | Add or dismiss a server asking to join. Approving one that matches an existing server needs `{"replace": true}`. |+| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). |+| `GET /monitoring/install-info` | admin | The collector address and images the panel uses. |+| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |+| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds. |+| `GET /agent/config` | session | Whether chat with the assistant is enabled. |+| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |+| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |+| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |+| `POST /monitoring/metrics/ingest` | collector token | Collector reports host, container and system metrics. |+| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Homie assistant chats. |+| `GET /logs`, `POST /logs`, `GET /dashboard/summary` | session | Audit log and summary counts. |++The four collector endpoints above exist only on the collector gate in the server+stack (the public listener answers 404); the development stack serves them on the+same address as everything else.diff --git a/docs/assistant.md b/docs/assistant.mdnew file mode 100644index 0000000..16a2e94--- /dev/null+++ b/docs/assistant.md@@ -0,0 +1,17 @@+[← Documentation](README.md)++# Homie assistant (monitoring-only)++The assistant has no code path for running shell commands. It may request+exactly one fixed, read-only check at a time: `server_health`,+`container_health`, or `daily_checkup`, implemented with Python system APIs and+Docker's Unix API socket, never the shell or Docker CLI.++The backend container does **not** mount `/var/run/docker.sock`. If you want+the assistant's container checks, point `DOCKER_SOCKET_PATH` at a hardened+Docker API proxy that permits only the required GET endpoints.++`daily_checkup` combines host and container checks. On a critical finding it+POSTs an alert only to `ADMIN_ALERT_WEBHOOK_URL`; without it, the result says+`not_configured` and no request is made. Schedule it from a trusted scheduler+(e.g. a systemd timer) rather than letting the agent create schedules.diff --git a/docs/configuration.md b/docs/configuration.mdnew file mode 100644index 0000000..fdf4a45--- /dev/null+++ b/docs/configuration.md@@ -0,0 +1,69 @@+[← Documentation](README.md)++# Configuration++## Stack (`.env` next to the Compose file)++| Variable | Default | Description |+|---|---|---|+| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. |+| `GEMINI_API_KEY` | empty | Enables the Homie chat assistant. |+| `AGENT_MODEL` | `models/gemini-3.5-flash` | Gemini model for the assistant. |+| `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. |+| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. |+| `SESSION_TTL_SECONDS` | `86400` | Session lifetime. |+| `ALLOW_REGISTRATION` | `true` (dev), `false` (server file) | Let anyone create an account. When `false`, only the first account can be created. |+| `SECRETS_KEY` | empty | Key that encrypts two-factor secrets. Required for two-factor sign-in and for the "require two-factor for admins" policy to be enforced. Generate with `python -m src.backend.secrets_box`; comma-separated keys allow rotation. |+| `ADMIN_BOOTSTRAP_TOKEN` | generated at start | Setup token for the first account. Leave empty to have one generated and printed in the backend log until the first account exists. |+| `FRONTEND_BIND` | `0.0.0.0` (dev), `127.0.0.1` (server file) | Interface for the dashboard's plain-HTTP port. |+| `BACKEND_PORT` | `8060` | Development stack only: the host port (on `127.0.0.1`) where the backend is published for debugging. The server stack publishes no backend port. |+| `FRONTEND_PORT` | `5173` | Dashboard port on the host. |+| `DASHBOARD_DOMAIN` | *(required on a server)* | Domain of the dashboard, without `https://` (server file). |+| `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERT_RESOLVER` / `TRAEFIK_NETWORK` / `TRAEFIK_ROUTER` | `websecure` / `letsencrypt` / `traefik_default` / `homelab` | Your Traefik names (server file). |+| `CORS_ORIGINS` | `http://localhost:5173` (dev), `https://<DASHBOARD_DOMAIN>` (server file) | Comma-separated origins. Also the allow-list for the cross-site request check: a state-changing request from any other origin is refused unless its `Origin` matches the `Host` the dashboard was reached on. Add your dashboard URL if something in front rewrites `Host`. |+| `LOGIN_RATE_LIMIT` / `LOGIN_RATE_WINDOW_SECONDS` | `5` / `60` | Login attempts per IP and per email. |+| `REGISTER_RATE_LIMIT` / `REGISTER_RATE_WINDOW_SECONDS` | `3` / `3600` | Registrations per IP. |+| `COLLECTOR_BIND` / `COLLECTOR_PORT` | `127.0.0.1` / `8081` | Server file: address and port where the collector gate is published. Set the address to your VPN's. |+| `COLLECTOR_PUBLIC_URL` | empty | What **Connect a server** tells new machines to use, e.g. `http://10.8.0.1:8081/api`. |+| `COLLECTOR_IMAGE` / `DOCKER_PROXY_IMAGE` | empty | Digest-pinned images remote machines install. **Connect a server** needs both. |+| `JOIN_TOKEN_TTL_SECONDS` | `900` | How long a generated join token stays valid. |+| `COLLECTOR_ACCESS` | `open` (dev), `strict` (server file) | `strict` makes the backend answer collector endpoints only when they came through the collector gate. |+| `HOMELAB_ENROLL_KEY` | empty | Lets the stack's own collector skip approval. |+| `HOMELAB_SERVER_NAME` | host's hostname | Name shown for the stack's own machine. |+| `DOMAIN` | `localhost` | Domain for the optional Caddy profile. |+| `COMPOSE_FILE` | – | Set to `docker-compose.server.yml` on a server so plain `docker compose` uses it. |+| `LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` | `3` | How often the dashboard refreshes (server file; applied when the frontend image is built). |+| `HOMELAB_CHECK_INTERVAL_SECONDS` | `60` | Initial check interval of the stack's own collector (server file); the dashboard setting overrides it. |++## Collector (environment of the collector container)++| Variable | Default | Description |+|---|---|---|+| `HOMELAB_API_URL` | `http://localhost:8060` | Dashboard API address, e.g. `https://dashboard.example.com/api`. |+| `HOMELAB_JOIN_TOKEN` | empty | Single-use token from **Connect a server**; required over the VPN listener. |+| `HOMELAB_ENROLL_KEY` | empty | Join immediately instead of waiting for approval. |+| `HOMELAB_SERVER_NAME` | host's `/etc/hostname` | Name shown on the dashboard. |+| `HOMELAB_CONNECTIVITY_TARGETS` | empty (check off) | Optional internet check: up to 5 comma-separated `host:port` targets (hostnames or IPv4), e.g. `1.1.1.1:443,9.9.9.9:443`. The collector opens a TCP connection and looks the name up; it sends nothing. Set on each collector; the dashboard cannot set it. |+| `HOMELAB_STATE_DIR` | `/var/lib/homelab-collector` | Where the token is saved (mount a volume here). |+| `HOMELAB_MACHINE_ID` | host's `/etc/machine-id` | Identity used to recognise reinstalled machines. |+| `HOMELAB_DASHBOARD_HOST` | empty | `true` marks this machine as the one running the dashboard (set in the Compose files for the stack's own collector; leave it unset elsewhere). |+| `HOMELAB_SERVER_ID` / `HOMELAB_COLLECTOR_TOKEN` | empty | Explicit credentials (legacy; overrides everything else). |+| `HOST_PROC` / `HOST_ROOT` | `/host/proc` and `/hostfs` if mounted, else `/proc` and `/` | Where the host's `/proc` and filesystem are. |+| `HOMELAB_DOCKER_HOST` | empty | Socket of the read-only Docker proxy, e.g. `/run/docker-proxy/docker.sock`. Preferred over `DOCKER_SOCKET_PATH`. |+| `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Direct Docker socket, used only when `HOMELAB_DOCKER_HOST` is unset (not recommended). Without any Docker access only host metrics are sent. |+| `HOMELAB_HELPER_FILE` | `/helper/system.json` | Where the optional system helper's summary is read from. |++The system helper has `HOMELAB_HELPER_INTERVAL_SECONDS` (default `30`, minimum+`10`); the Docker proxy listens on `HOMELAB_DOCKER_PROXY_SOCKET` and forwards to+`DOCKER_SOCKET_PATH`. Both are set in their images and rarely need changing.++The check interval is set per server on the dashboard; collectors pick it up+from the backend's response to each report.++## Frontend build variables++Environment variables for the frontend build (set when the image is built):++- `VITE_API_BASE_URL` (default `/api`): API base path or URL.+- `VITE_LIVE_GRAPH_REFRESH_INTERVAL_SECONDS` (default `3`, minimum `1`): how+ often the dashboard refreshes.diff --git a/docs/dashboard.md b/docs/dashboard.mdnew file mode 100644index 0000000..f175af4--- /dev/null+++ b/docs/dashboard.md@@ -0,0 +1,68 @@+[← Documentation](README.md)++# The dashboard and system health++For what each value means and how it is measured, see [Metrics](metrics.md).++- **This machine** (top): live CPU, memory and root disk charts for the machine+ running the dashboard, loaded from the last 30 minutes of stored metrics.+- **Fleet**: one card per server, with the dashboard's own machine first. Click a+ server's name to collapse its card to a single line (status, CPU, RAM, fullest+ disk, incident count); **Collapse all** / **Expand all** appear with more than+ one server. Each card shows a **Storage** bar per filesystem, colored by the+ alert thresholds (including any per-mount overrides), a **System** row and a+ **Containers** list. Both lists start closed. A filter box appears for servers+ with more than 10 containers.+- **Network** row: internet reachability, how many ports the server listens on and how+ many are exposed, and the firewall state, opening into the listener list and the last+ 7 days of port and firewall changes.+- **System** row: one line of status chips (when it booted, last shutdown, clock,+ OOM kills in the last 24 hours, kernel, pending updates) that opens into the+ last 7 days of reboots, OOM kills and kernel messages. A chip that says+ *unknown* means the collector couldn't read that value on this host, not that+ anything is wrong.+- **Active incidents**: open incidents, most severe first, with the server and+ when they started.++Which cards are collapsed is remembered per browser; the System and Containers+lists always start closed.++<ui - One expanded server card: CPU/RAM/Disk/Load tiles, Storage bars for several mounts (one amber or red), the System row open with a reboot and an OOM kill in its events, and the Containers list open with an unhealthy container and one container's details expanded (stat tiles and CPU/memory charts)>++<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>++## System health++Collectors report what the kernel exposes, without reading any logs:++| Check | Source | Notes |+|---|---|---|+| Reboots | `/proc/sys/kernel/random/boot_id` | A new boot ID between check-ins records a **Reboot** event. |+| OOM kills | `oom_kill` in `/proc/vmstat` | A rising counter records an **OOM kill** event. |+| Kernel faults | `/proc/sys/kernel/tainted` | Only fault bits count (machine check, bad page, oops, warning, soft lockup); proprietary or out-of-tree module flags are ignored. |+| Clock sync | `adjtimex` (read-only), else systemd-timesyncd's marker | Sync state only, not a measured offset. |+| Reboot required | `/run/reboot-required` | Debian and Ubuntu only; other distros show *unknown*. |++### Optional system helper++Process names, kernel error lines and crash-versus-clean shutdown detection need+the kernel log and the login records, which the collector deliberately can't+read. A separate container does:++```bash+docker compose --profile system-events up -d # add -f docker-compose.server.yml on a server+```++It runs with a single capability (`CAP_SYSLOG`), no network and a read-only+filesystem, sees only `/var/log/wtmp` (read-only), and writes a sanitised JSON+summary that the collector reads from a shared volume. With it you also get:++- the OOM-killed process name on each **OOM kill** event,+- recent kernel error lines in the **System** row (informational, they don't open incidents),+- **Last shutdown: clean / unclean**, and a **critical** incident after an unclean reboot.++Without it those show as *unknown* and everything else keeps working. Read+[SECURITY.md](../SECURITY.md) before enabling it; the helper is about 200 lines of+standard-library Python.++<ui - The System row opened on a server with the helper running: status chips on top, then a list with an OOM kill (process name) and kernel error lines>diff --git a/docs/database.md b/docs/database.mdnew file mode 100644index 0000000..6048c29--- /dev/null+++ b/docs/database.md@@ -0,0 +1,31 @@+[← Documentation](README.md)++# Database schema++Tables are created on backend startup (`Base.metadata.create_all`), and new+columns are added with `ALTER TABLE ... IF NOT EXISTS` in the startup hook in+`src/backend/main.py`. Each check-in stores one row in `host_metrics` (which also+carries the boot ID, OOM counter, kernel flags, clock state and last shutdown), one+per filesystem in `mount_metrics` and one per container in `container_metrics` (which+also records the exit code, out-of-memory flag, finish time and image ID), all with the+same timestamp. `system_events` holds reboots, OOM kills, kernel messages, container+changes (appeared, removed, image changed) and network changes (ports opened or closed,+firewall changes); `listener_states` and `firewall_states` hold what was last seen, `join_tokens` the hashed single-use tokens, `recovery_codes` the hashed two-factor+recovery codes (the encrypted two-factor secret lives on the `users` row), and per-mount disk+thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises+the `users` table on a fresh volume.++**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in+samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container+per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.+Each tier is built from the finer one when its period ends, a period without check-ins gets no row, and the+purge never deletes samples that have not been summarised yet. The Statistics page reads them (see [Statistics](statistics.md)). They are+not purged yet; a retention setting for each level is planned.++The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and+`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are+migrated at start: the new indexes are created and the redundant single-column ones are dropped,+which can be repeated safely. Creating an index briefly blocks writes to that table (a few+seconds for a table of millions of rows); on a very large install, create the indexes yourself+first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names+in `src/backend/models.py`, and the start-up migration will skip them.diff --git a/docs/deployment.md b/docs/deployment.mdnew file mode 100644index 0000000..81ed7c3--- /dev/null+++ b/docs/deployment.md@@ -0,0 +1,136 @@+[← Documentation](README.md)++# Deploying on a server (behind Traefik)++There are two Compose files:++| File | Use | Differences |+|---|---|---|+| `compose.yml` | Laptop / development (the default when you run `docker compose` without `-f`) | Dashboard on port 5173 on all interfaces; plain HTTP; optional Caddy (`--profile https`). |+| `docker-compose.server.yml` | Server behind an existing Traefik | Traefik labels and network; HTTPS-only session cookie; only the first account can register; port 5173 bound to `127.0.0.1`; no Caddy; **collector endpoints removed from the public side** and served by a separate collector gate on your VPN address. |++## 1. Create `.env` on the server++Nothing in `docker-compose.server.yml` needs editing: everything specific to you comes from+`.env`, so `git pull` can always update the file. Start from the production template:++```bash+cp example.env .env+```++Then set at least these:++| Setting | Default | Meaning |+|---|---|---|+| `POSTGRES_PASSWORD` | *(required)* | A long random password for the bundled database. |+| `DASHBOARD_DOMAIN` | *(required)* | Your domain, without `https://`. Used for the Traefik router. |+| `TRAEFIK_ENTRYPOINT` | `websecure` | Your Traefik HTTPS entrypoint name. |+| `TRAEFIK_CERT_RESOLVER` | `letsencrypt` | Your Traefik certificate resolver name. |+| `TRAEFIK_NETWORK` | `traefik_default` | The Docker network Traefik is attached to. |+| `TRAEFIK_ROUTER` | `homelab` | Name of this dashboard's Traefik router and service; change it only if it clashes with another one. |+| `COMPOSE_FILE` | `docker-compose.server.yml` | Makes plain `docker compose ...` commands use the server file (see below). |++Find the network with `docker network ls` or+`docker inspect <traefik-container> --format '{{json .NetworkSettings.Networks}}'`.+The entrypoint and resolver names are in Traefik's static configuration+(`entryPoints:` / `certificatesResolvers:`).++## 2. Use the right Compose file++`COMPOSE_FILE=docker-compose.server.yml` in `.env` matters. Without it, Compose picks+`compose.yml` (the development stack), and an `up` would replace the server containers+with ones that have no Traefik labels and open registration.++**Don't copy the server file under another name** (for example `docker-compose.yml`). If+both `compose.yml` and `docker-compose.yml` exist, Compose silently uses `compose.yml`, and+a copy won't receive updates. If you made a copy of an older version, move your values+into `.env`, delete the copy, and continue with `docker-compose.server.yml`.++## 3. Start it++```bash+docker compose -f docker-compose.server.yml up -d --build+```++Open `https://<your-domain>` and register your account right away. The first+account needs the **setup token** (`docker compose logs backend | grep "setup token"`,+or the value of `ADMIN_BOOTSTRAP_TOKEN` if you set one), so a stranger who finds a+fresh deployment can't claim it; then registration closes. That first account is+the admin. Traefik terminates HTTPS with its own certificates; the stack needs no+certificate configuration.++The public address serves only the dashboard. Collectors don't use it; set up+their route in the next step.++## 4. Let remote servers report over your VPN++Collectors reach the **collector gate**, not the public domain. In `.env`:++| Setting | Example | Meaning |+|---|---|---|+| `COLLECTOR_BIND` | `10.8.0.1` | Your VPN server's address on the dashboard machine (for OpenVPN, the `tun` address). The gate's port is published on this address only. |+| `COLLECTOR_PORT` | `8081` | Port of the gate on that address. |+| `COLLECTOR_PUBLIC_URL` | `http://10.8.0.1:8081/api` | What **Connect a server** tells new machines to use. |+| `COLLECTOR_IMAGE`, `DOCKER_PROXY_IMAGE` | `ghcr.io/…@sha256:…` | The images remote machines install; see below. |++Things to get right:++- **Bind to the VPN address, not `0.0.0.0`.** Docker's published ports bypass+ `ufw` and similar firewalls, so a "deny from outside" rule does not protect them.+- If Docker starts before the VPN is up, the bind fails until the address exists.+ `restart: unless-stopped` retries, or make Docker start after the VPN (for+ example a systemd drop-in with `After=openvpn-server@<name>.service`).+- Traffic inside the VPN is plain HTTP over the encrypted tunnel; collectors+ authenticate with a per-server token that the dashboard stores only as a hash.++### Publishing the collector images++Remote machines install the collector and the Docker proxy from container images,+pinned by digest, so no code is ever downloaded from the dashboard. The+[Publish images](../.github/workflows/images.yml) workflow builds them (amd64 and+arm64), signs them with cosign and prints their digests:++```bash+git tag v0.1.0 && git push origin v0.1.0+```++1. Open the workflow run's summary and copy the `COLLECTOR_IMAGE=…` and+ `DOCKER_PROXY_IMAGE=…` lines into `.env`.+2. If the repository is private, make the three packages **public** (GitHub →+ Packages → *Package settings* → *Change visibility*); otherwise machines can't+ pull them without credentials.+3. Rebuild the stack so the backend knows the images:+ `docker compose -f docker-compose.server.yml up -d --build`.++**Connect a server** won't generate a command until both images are set.++## Updating++```bash+git pull+docker compose -f docker-compose.server.yml up -d --build+```++Always rebuild **all** services (don't name individual ones). The backend,+frontend, collector, Docker proxy and system helper each have their code built+into their image, so a service that isn't rebuilt keeps running the old version.+The database schema is updated automatically when the backend starts.++Collectors on other machines run the image digest from the command you generated.+To update them, publish a new version tag, put the new digests in `.env`, and run a+freshly generated command on each machine (or pull the new digests into its+compose project). Collectors started from `collector-compose.yml` need+`docker compose -f collector-compose.yml up -d --build` after a `git pull`.++<diagram - What each step updates: git pull → source on disk; docker compose up -d --build → backend, frontend, collector, docker-proxy and gate containers rebuilt and recreated; git tag → Publish images workflow → new digests → .env and a fresh Connect a server command for remote machines; show that skipping --build leaves the old image running>++## Without Traefik++- **Caddy (automatic Let's Encrypt):** with `compose.yml`, set+ `DOMAIN`, `SESSION_COOKIE_SECURE=true` and `FRONTEND_BIND=127.0.0.1` in+ `.env`, open ports 80/443, and run `docker compose --profile https up -d --build`.+- **Private, over a VPN:** install Tailscale, WireGuard or OpenVPN on the server+ and your devices, run `docker compose up -d --build`, and use `http://<vpn-ip>:5173`+ (collectors: `http://<vpn-ip>:5173/api`). Nothing is exposed publicly. This uses+ the development stack's single listener; for a public dashboard with collectors+ on a VPN, use the server stack above.diff --git a/docs/development.md b/docs/development.mdnew file mode 100644index 0000000..3c2a493--- /dev/null+++ b/docs/development.md@@ -0,0 +1,97 @@+[← Documentation](README.md)++# Development++# Development++## Backend without Docker++Python 3.12. The backend needs Redis for sessions and rate limits; SQLite works+in place of PostgreSQL.++```bash+python -m venv venv && source venv/bin/activate+pip install -r requirements.txt+docker run -d --name homelab-dev-redis -p 6379:6379 redis:7-alpine # if Redis isn't running+DATABASE_URL=sqlite:///./dev.db REDIS_URL=redis://localhost:6379/0 \+ uvicorn src.backend.main:app --reload --host 0.0.0.0 --port 8060+```++## Frontend dev server++Node 20 (`.nvmrc`). The Vite dev server proxies `/api` to `localhost:8060` (set `BACKEND_URL` to point it elsewhere),+matching the production nginx setup.++```bash+nvm install && nvm use+cd src/frontend+npm install+npm run dev -- --host 0.0.0.0+```++Vite prints the URL (5173, or the next free port if the Compose stack already+uses it). `./start_frontend.sh` does the same in one step.+++After changing frontend or collector code, rebuild the containers:+`docker compose up -d --build`. Otherwise the browser keeps getting the old bundle.++## Tests++```bash+source venv/bin/activate+pip install -r requirements-dev.txt # the app's requirements plus pytest+pytest+```++No Docker, PostgreSQL or Redis is needed; the suite takes about a minute and a half. The chart+helpers have their own tests that need only Node 20 or newer: `cd src/frontend && node --test`. See+[TESTS.md](../TESTS.md) for how the tests work, what they don't cover, and what every+test is for.++## Project layout++```text+compose.yml development stack (default for `docker compose`)+docker-compose.server.yml server stack behind Traefik+collector-compose.yml collector-only stack for monitored machines+README.md what it is and how to start it (the short version)+docs/ the full documentation (start at docs/README.md)+SECURITY.md what each component can touch, and its limits+TESTS.md how to run the tests and what each one checks+requirements-dev.txt app requirements plus pytest, for running the tests+pytest.ini test settings (lets plain `pytest` find the `src` package)+.github/workflows/+ images.yml builds, signs and publishes the collector images on a version tag+docker/+ backend.Dockerfile FastAPI backend (uvicorn)+ frontend.Dockerfile Vite build served by nginx+ collector.Dockerfile standalone collector image+ docker-proxy.Dockerfile read-only Docker API proxy image+ system-helper.Dockerfile optional kernel-log / wtmp helper image+ nginx.conf development: serves the UI, proxies /api to the backend+ nginx.strict.conf server stack: public listener, collector endpoints refused+ nginx.collector.conf server stack: collector gate (VPN and internal listeners)+ Caddyfile optional HTTPS front door (dev compose, --profile https)+ postgres/init/ SQL run on a fresh PostgreSQL volume+src/+ backend/ API: auth, monitoring, enrollment, rate limits, models+ alerts.py alert engine: one incident per rule, auto-resolve+ rules.py alert rules and default thresholds+ rollups.py summarises check-ins into hourly/daily/weekly/monthly rollups+ history.py range-aware history behind the Statistics page+ stats.py fleet statistics: incidents over time, availability, busy-hours heatmap+ totp.py one-time codes and recovery codes (standard library only)+ secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key+ reset_2fa.py command-line two-factor reset for a locked-out account+ agent/+ collector.py metrics collector (standard library only)+ docker_proxy.py read-only Docker API proxy (standard library only)+ system_helper.py optional kernel-log / wtmp helper (standard library only)+ health_checks.py read-only checks available to the assistant+ service.py, prompts.py Gemini assistant+ frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the+ charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,+ FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the page+tests/ pytest suite+```diff --git a/docs/metrics.md b/docs/metrics.mdnew file mode 100644index 0000000..38fd909--- /dev/null+++ b/docs/metrics.md@@ -0,0 +1,187 @@+[← Documentation](README.md)++# Metrics: what every number means++This page explains each value the dashboard and the [Statistics](statistics.md) page show: what it+is, where it comes from, how it is calculated, and what it does **not** tell you. Where two values are+easy to mix up (CPU % and load, memory "used" and "free") the difference is spelled out.++- [How values are collected](#how-values-are-collected)+- [Host metrics](#host-metrics): CPU, load, memory, swap, disk, inodes, network, uptime+- [Container metrics](#container-metrics)+- [System state](#system-state)+- [Network checks](#network-checks)+- [Collector self-metrics](#collector-self-metrics)+- [How history is stored](#how-history-is-stored): what "average" and "highest" mean over a week+- [Derived statistics](#derived-statistics): availability, distributions, busy hours, "full in about N days"+- [Units](#units)++## How values are collected++Each server's collector reads the host's `/proc` and filesystem and the Docker API (through the+read-only proxy) once per **check-in**. The default interval is 60 seconds and it can be set per server+between 15 seconds and 15 minutes. Everything on this page is therefore a **snapshot taken at a+check-in**, not a continuous recording. Between two check-ins nothing is measured.++- The time stored with a check-in is when the dashboard received it (UTC), not the collector's clock.+- A value the collector could not read is stored as *unknown*, never as zero. Unknown values are left+ out of averages, never raise alerts, and show as "–" or *unknown* in the interface.+- The backend validates every number it receives (type, range, length) before storing it.++## Host metrics++| Metric | What it means | How it is measured | Range |+|---|---|---|---|+| **CPU usage** (`cpu_utilization_pct`) | The share of CPU time the processors spent working, across **all cores** together. 100% means every core was fully busy. | Two readings of the aggregate line in `/proc/stat`, **about 0.15 seconds apart**, at each check-in. Time spent idle, and waiting for I/O, counts as not busy. | 0 – 100 % |+| **Load average** (`load_1m`, `load_5m`, `load_15m`) | How many tasks, on average, wanted to run or were stuck waiting (see below). It measures **demand**, not utilisation. | The kernel's own load average, read with `getloadavg()`: averaged by the kernel over the last 1, 5 and 15 minutes. | 0 and up, not capped |+| **Memory used** (`ram_pct`, `ram_used_gb`) | How much RAM programs are actually holding. | `MemTotal − MemAvailable` from `/proc/meminfo`, as a share of `MemTotal`. File caches the kernel can free at once count as **available**, so a Linux machine showing "full" RAM in other tools is usually fine here. | 0 – 100 % |+| **Swap used** (`swap_pct`) | How much of the swap space is in use. | `SwapTotal − SwapFree` as a share of `SwapTotal`. A host without swap reports 0. | 0 – 100 % |+| **Root disk used** (`disk_usage_pct`) | How full the root filesystem is. | `statvfs` of the host root, calculated like `df`: used ÷ (used + space available to non-root users). Reserved blocks are not counted as free. | 0 – 100 % |+| **Inodes used** (`disk_inodes_pct`) | How many files the root filesystem can still hold. A disk can have free space and no free inodes. | `statvfs` file counts: (total − free) ÷ total. Unknown on filesystems that don't report inodes. | 0 – 100 % |+| **Network sent / received** (`net_bytes_sent_mb`, `net_bytes_recv_mb`) | **Running totals** of bytes sent and received since the machine booted. They are not speeds. | Sum over the host's **physical** interfaces from `/proc/1/net/dev`. Virtual interfaces (loopback, Docker bridges, `veth`, VPN tunnels) are left out because their traffic is already counted on the physical one. | MiB, grows until reboot |+| **Uptime** (`system_uptime_seconds`) | Seconds since boot. | `/proc/uptime`. | seconds |++### CPU % and load are different things++| | CPU usage % | Load average |+|---|---|---|+| Question it answers | "How busy are the processors?" | "How many tasks are waiting for them (or for I/O)?" |+| Scale | 0–100% of the whole machine | A count of tasks; compare with the **number of cores** |+| Counts waiting tasks | No | **Yes**: tasks that are runnable, and tasks in uninterruptible sleep (usually waiting for disk or NFS) |+| Time window | A ~0.15 s snapshot per check-in | A kernel-maintained average over 1, 5 or 15 minutes |++How to read them together:++- **Load around the number of cores** is a fully used machine; **well above it** means tasks are queueing.+ Load 4 on a 4-core machine is "full"; the same 4 on a 1-core machine is badly overloaded. That is why+ the alert rule divides by the core count.+- **Low CPU %, high load** usually means tasks are blocked on disk, network storage or something else+ slow. The processors are idle while work waits.+- **High CPU %, load near the core count** is a busy machine that is keeping up.++Which load the interface shows: the **Statistics** chart draws the **1-minute** load as a raw number (not+divided by cores), and the **alert rule** uses the **5-minute** load **per core** (warning at 1.5, critical+at 2.0 by default). The 1-minute figure reacts quickly and is noisy; the 5-minute figure is steadier.++### Reading the CPU figure honestly++Because CPU % is measured over about 0.15 seconds at each check-in, one value is a snapshot of that+moment. With a 60-second interval, each value represents roughly a quarter of a percent of the time.+A busy period that starts and ends between two check-ins can be missed completely, while a spike that+happens to coincide with a check-in is recorded in full. Averages over many check-ins (an hour, a day)+are a good picture of typical usage; a single point is not. The **load average**, which the kernel keeps+continuously, does not have this blind spot, so compare the two when something looks odd. (Measuring+CPU over the whole interval between check-ins would remove the limitation; it is on the list of things+to improve.)++### Filesystems++Disk and inode usage are reported **per filesystem**, in addition to the root figures above+(`mount_metrics`):++| Value | Meaning |+|---|---|+| `mount` | The path where the filesystem is mounted, for example `/` or `/media/data`. |+| `fs_type` | `ext4`, `xfs`, `btrfs`, `zfs`, `nfs`, … |+| `total_gb`, `used_gb` | Size and amount used, in GiB. |+| `used_pct` | Calculated like `df` (used ÷ (used + available to non-root)), the same as the root disk figure. |+| `inodes_pct` | Inodes in use, where the filesystem reports them. |++Included: real block devices and the network or pool filesystems `zfs`, `nfs`, `nfs4`, `cifs`, `smb3`,+`fuse.mergerfs` and `fuse.sshfs`. Left out: `tmpfs`, `devtmpfs`, `overlay`, `squashfs`, and anything under+`/proc`, `/sys`, `/dev`, `/run`, `/snap`, `/var/snap`, `/var/lib/docker`, `/var/lib/containerd` and+`/var/lib/kubelet`. A device mounted in several places is reported once. A disk mounted after the+collector started appears after the collector restarts.++## Container metrics++Taken from Docker's stats and inspect output for every container at each check-in. Only containers that+are **running** report usage figures; for a stopped container they are unknown, not zero.++| Metric | What it means | How it is measured |+|---|---|---|+| **State** (`status`) | `running`, `exited`, `restarting`, `paused`, `created`, `dead`. | Docker inspect. |+| **Health** (`healthcheck_status`) | `healthy`, `unhealthy`, `starting`, or none if the image defines no healthcheck. | Docker inspect. |+| **Restarts** (`restart_count`) | How many times Docker has restarted it. | Docker's counter; it starts again at 0 when the container is recreated. |+| **CPU %** (`cpu_pct`) | How much CPU the container used, where **100% is one full core**. A container using three cores' worth shows about 300%. | The change in the container's CPU time divided by the change in the whole machine's CPU time since the **previous check-in**, times the number of cores. The first check-in after the collector starts has no figure (unknown), and a container that used no CPU in the interval shows none. |+| **Memory used** (`memory_used_mb`) | Memory the container is really holding. | Docker's usage minus its reclaimable page cache (`inactive_file`), the same way `docker stats` does it. |+| **Memory limit / %** (`memory_limit_mb`, `memory_pct`) | The limit set for the container and how much of it is used. | No limit set means no percentage. |+| **Network in / out** (`net_rx_mb`, `net_tx_mb`) | Running totals since the container started. Not speeds. | Sum over the container's interfaces. |+| **Disk read / write** (`block_read_mb`, `block_write_mb`) | Running totals since the container started. | Docker's block I/O counters. |+| **Processes** (`pids`) | How many processes are running inside it. | Docker's pids count. |+| **Exit code, out of memory, finished at** | How a stopped container ended. Exit 137 usually means it was killed (by `docker stop`/`kill`, or by the out-of-memory killer). | Docker inspect. |+| **Image ID** | The exact image build it is running, used to notice a changed image. | Docker inspect. |++## System state++Read from what the kernel exposes, without reading any log files.++| Value | Meaning | Source |+|---|---|---|+| **Boot ID** | A new value between two check-ins means the machine rebooted. | `/proc/sys/kernel/random/boot_id` |+| **OOM kills** | How many processes the kernel's out-of-memory killer has stopped since boot. A rising counter records an event. | `oom_kill` in `/proc/vmstat` |+| **Kernel fault flags** | Only fault bits count (machine check, bad page, oops, warning, soft lockup). Flags for proprietary or out-of-tree modules are ignored. They stay set until reboot. | `/proc/sys/kernel/tainted` |+| **Clock synchronised** | Whether a time service is keeping the clock disciplined. This is the sync state, **not** a measured offset. | `adjtimex` (read-only), or systemd-timesyncd's marker |+| **Reboot required** | Updates are waiting for a restart. Debian and Ubuntu only; other systems show *unknown*. | `/run/reboot-required` |+| **Last shutdown** (clean / unclean) | Whether the previous shutdown was orderly. Needs the optional [system helper](dashboard.md#optional-system-helper). | `wtmp` via the helper |+| **Kernel error lines, OOM-killed process name** | Informational. Need the system helper. | Kernel log via the helper |++## Network checks++Passive, and only what the collector can read without privileges.++| Value | Meaning |+|---|---|+| **Listening ports** | Which TCP/UDP ports the host listens on, and whether each is reachable from other machines or only from the machine itself (**scope**). Protocol, port and scope only, with no process names. Short-lived programs on random high ports are ignored. |+| **Firewall** | A hash and a rule count of the known firewall configuration files, plus whether `ufw` is enabled. Never the rules themselves, and not the live ruleset. A changed hash raises an informational incident. |+| **Internet reachability** | Off unless you set `HOMELAB_CONNECTIVITY_TARGETS` on that collector. `ok` (all targets reachable), `degraded` (some), `dns_failed` (names don't resolve and nothing else is wrong) or `down`; plus the median connection time in milliseconds. |++## Collector self-metrics++| Metric | Meaning |+|---|---|+| **Docker response time** (`docker_latency_ms`) | How long it took to list the containers. A rising value means the Docker daemon is struggling. Unknown when Docker isn't reachable. |+| **Docker status** | `ok`, `unreachable` or `not_configured`, with a coarse reason (timeout, refused, missing, denied, error). Only a host that had Docker working before can raise "Docker unreachable". |+| **Collection time** (`collect_duration_ms`) | How long the collector took to gather everything for one check-in. If it approaches the check interval the collector can't keep up, and a warning opens (80% by default). |++## How history is stored++The check-in snapshots are kept for the retention period (30 days by default). A background job also+summarises them into one row per **hour, day, week (Monday to Sunday) and month**, all in UTC, for+every server, filesystem and container. Each row stores, for every metric:++- the **average** over the check-ins in that period,+- the **highest value** (maximum) seen in a check-in,+- the number of check-ins that reported it (so averages stay exact when some check-ins lack a value).++What that means on the Statistics page:++- A line is the **average of the snapshots** in each bucket. For CPU % that is the average of+ ~0.15-second snapshots, as explained above.+- The lighter band above the line reaches the **highest snapshot** in the bucket, so a spike that a+ check-in caught stays visible even in a yearly view. It is the highest *recorded* value, not+ necessarily the highest value that ever occurred.+- A period with no check-ins has no point: a gap stays a gap and is never filled in.+- Longer ranges use coarser rows, so a point can represent an hour, a day, a week or a month. The note+ under the cards says which. Day, week and month boundaries are in UTC.++See [Database schema](database.md) for the tables and [Statistics](statistics.md) for which range uses which+level.++## Derived statistics++| Figure | How it is worked out | Caveats |+|---|---|---|+| **Availability** | The share of the check-ins a server should have made in the range that actually arrived: check-ins received ÷ (time in range ÷ check interval), capped at 100%. The range starts at the server's first check-in. | Approximate: it assumes the server's *current* check interval for the whole range. A server that never reported is unknown, not 0%. |+| **Incidents opened / open now** | Counted from the incident log by when each incident opened, grouped by severity and by kind of problem. | Counting is capped at 20,000 incidents per request; the answer says when it was cut. |+| **Mean time to resolve** | The average time between an incident opening and resolving, over incidents resolved in the range. | Incidents still open are not included. |+| **Distribution (histogram)** | The share of the time a metric's average was in each 10% band, weighted by the number of check-ins behind each point. | Worked out in the browser from the points shown, so for long ranges it is a distribution of *averages* and looks calmer than the raw data. |+| **Busy hours (heatmap)** | The average of CPU or memory for each hour of the day and weekday over the last 30 days, in your time zone (a fixed offset; daylight-saving changes inside the 30 days are ignored). | Needs a few days of history to fill in. Independent of the selected range. |+| **"Full in about N days"** | A straight line fitted through a filesystem's usage over the range, extended to 100%. | Shown only for a **clear** rising trend: at least 6 points spanning at least a day, rising, and the line must fit the data reasonably (r² ≥ 0.5). Otherwise: "No clear growth trend". It assumes growth continues at the same rate; a cleanup or a one-off copy changes it. |+| **Top containers / servers** | The containers using the most memory or CPU, and servers by disk usage, from the latest check-in. | Current values, not an average over the range. |++## Units++- **GB and GiB.** Sizes labelled "GB" are **GiB** (1 GB = 1024³ bytes), the unit `df -h` uses. "MB" values are MiB (1024² bytes).+- **Percentages** are rounded to two decimals when stored and usually shown with one.+- **Times** are UTC when stored. Charts for up to a day or two use your browser's clock; day, week and month points are labelled in UTC.diff --git a/docs/statistics.md b/docs/statistics.mdnew file mode 100644index 0000000..7ef09d1--- /dev/null+++ b/docs/statistics.md@@ -0,0 +1,53 @@+[← Documentation](README.md)++# Statistics++For what each number means and how it is measured (including the limits of the CPU figure), see [Metrics](metrics.md).++**Statistics** in the navigation shows how one server behaved over time. Pick a server and a+range (1 hour, 6 hours, 24 hours, 7 days, 30 days, 90 days, 1 year or all time) and you get+CPU, memory, swap, load and root disk charts. Each chart is the average as a line with a+lighter band up to the highest value, so a short spike stays visible even in a yearly view.+Hover or touch a point for the exact time, average, highest value and number of check-ins, or+use **View as table**. Gaps (a server that was offline) are drawn as gaps. A note under the+cards says what each point is (for example "one per hour, from hourly summaries") and when+the history starts. The selection is kept in the address, so a view can be bookmarked, and+the range is remembered per browser. The short ranges refresh every 30 seconds, except+while the tab is hidden.++Under the charts, **Containers** shows every container of that server as a small card (status+dot, health, restarts, CPU, memory against its limit), with search and sorting (problems first+by default). Select a card for that container's CPU and memory over the chosen range, with its+memory limit drawn as a dashed line, how much of the time it was not running, and how many+times it restarted in the range. A container only reports CPU and memory while it runs.++Further down a server's page: how its CPU and memory use were **distributed** (the share of the+time in each 10% band), **when it is busy** (average CPU or memory by hour of the day and weekday+over the last 30 days, in your time zone), **storage** (a chart per filesystem with an estimate+such as "full in about 3 days" when there is a clear growth trend, and none otherwise), and+**reliability** (how long Docker takes to list containers, and how long the collector takes).++Choose **All servers** for the whole fleet: containers by state and open incidents by severity+(donuts), incidents opened over time by severity (stacked columns), what the incidents were+about, servers by disk usage, the containers using the most memory and CPU right now, and each+server's availability. Availability is the share of expected check-ins that arrived; it is+approximate because it assumes each server's current check interval. Every chart can be shown+as a table.++Short ranges come from the stored check-ins, longer ones from the summaries described under+[Database schema](database.md), so a year of history costs about as much to load as an+hour.++What to know about the statistics:++- **Time zones.** Charts for up to a day or two use your clock; day, week (Monday to Sunday) and+ month points are aligned to UTC and labelled in UTC, because that is how they are summarised.+- **Some figures are approximate**: distributions and the days-until-full estimate are worked out+ in the browser from the points you are looking at, availability assumes each server's current+ check interval, and the top containers show values from the latest check-in, not an average+ over the range. The page says so where it matters.+- **Pointer-only details.** Hover and touch tooltips don't work from the keyboard yet; every chart+ has **View as table** as an alternative.+- **Not built yet** (tracked in Milestone 8 on GitHub): CSV and JSON export, reboots per month, a+ state timeline and restarts per day for containers, rankings over the chosen range, network+ traffic charts, event markers on charts, and retention settings for the summaries.diff --git a/docs/troubleshooting.md b/docs/troubleshooting.mdnew file mode 100644index 0000000..555f57a--- /dev/null+++ b/docs/troubleshooting.md@@ -0,0 +1,29 @@+[← Documentation](README.md)++# Troubleshooting++| Symptom | Check |+|---|---|+| A new machine never shows up under **New servers detected** | `docker logs homelab-collector` on that machine. "Could not reach …" means the address in the command isn't reachable from it (firewall, `localhost`, wrong IP/domain). |+| Collector log says "Backend no longer accepts server …" | The server was deleted on the dashboard; it is asking to join again, so click **Add** or **Dismiss**. |+| `network traefik_default declared as external, but could not be found` | You're using `docker-compose.server.yml` on a machine without that Traefik network. Use `compose.yml` for development, or set `TRAEFIK_NETWORK` in `.env` to the network Traefik is really on. |+| Dashboard shows old behaviour after a code change or `git pull` | Rebuild **all** services with `docker compose up -d --build` (with `-f docker-compose.server.yml` on a server unless `COMPOSE_FILE` is set) and hard-refresh the browser (Ctrl+Shift+R). |+| "The machine running the dashboard isn't reporting yet" at the top | The stack's own collector hasn't checked in: add it under **New servers detected**, then wait one check interval. If it is reporting, its image is outdated: `docker exec homelab-collector grep -c dashboard_host /app/collector.py` prints `0`. Rebuild it (`docker compose build --no-cache collector && docker compose up -d collector`). |+| A server shows only `/` and no **Storage** bars for other disks | Its collector predates per-mount reporting. Install a newer collector image (run a freshly generated command, or `docker compose -f collector-compose.yml up -d --build`). A disk mounted after the collector started appears after a collector restart. |+| Logged out after opening the dashboard at a new address | Login cookies belong to the address they were set on; log in again. |+| Login fails behind HTTPS with `SESSION_COOKIE_SECURE=true` | The cookie is only sent over HTTPS; open the `https://` address, not `http://…:5173`. |+| Collector log says "Registration refused (HTTP 403): The join token is invalid, expired or already used" | The token in the command has expired, was already used, or is wrong. Click **Generate command** again and run the new command. |+| **Connect a server** says no collector images are configured | Set `COLLECTOR_IMAGE` and `DOCKER_PROXY_IMAGE` in `.env` (see [Publishing the collector images](deployment.md#publishing-the-collector-images)) and rebuild. |+| The register page asks for a setup token | The install has no accounts yet. Use `docker compose logs backend \| grep "setup token"`, or `ADMIN_BOOTSTRAP_TOKEN`. |+| "Password must be at least 16 characters long" | New passwords need 16 or more characters (at most 72 bytes). A few random words work well. |+| "Cross-site request refused" (403) on a button or form | The request came from another origin. Open the dashboard at the address you normally use. If a proxy in front rewrites the `Host` header, add your dashboard URL to `CORS_ORIGINS`. |+| The **System** row shows *unknown* for last shutdown or has no kernel messages | The optional system helper isn't running. Start it with `docker compose --profile system-events up -d`. *Unknown* for reboot-required on non-Debian hosts is expected. |+| `Error response … cannot assign requested address` for the collector gate | `COLLECTOR_BIND` is set to an address that doesn't exist yet (VPN not up). Start the VPN first, or fix the address. |+| The **Network** row says *no config found* for the firewall | None of the known firewall files exist on that host (`/etc/ufw`, `/etc/nftables.conf`, `/etc/iptables/rules.v4`). The check reads configuration, not the live ruleset, so a host that configures its firewall some other way shows this. |+| The **Network** row says *not checked* for the internet | The check is opt-in. Set `HOMELAB_CONNECTIVITY_TARGETS` on that collector (for example `1.1.1.1:443`). |+| Listeners show *unknown* or the Network row is empty | The collector couldn't read the host's network tables (older collector, or a table with more than 500,000 sockets). An unreadable report never raises a "port closed" alert. |+| "Invalid code." although the code looks right | Codes depend on the clock: check the phone's time is automatic. A code also can't be used twice, so wait for the next one. Five wrong codes lock the code step for 5 minutes. |+| Two-factor setup says it is not available | `SECRETS_KEY` is missing or invalid in `.env`. Generate one (see [Two-factor sign-in](accounts.md#two-factor-sign-in)) and restart the backend. The backend log shows a SECURITY WARNING while the key is missing. |+| Lost the authenticator and the recovery codes | Run `docker compose exec backend python src/backend/reset_2fa.py <email>` on the host, then sign in with the password and set it up again. |+| Authenticator codes stopped working after a restore or `.env` change | `SECRETS_KEY` changed or was lost, so the stored secret can't be decrypted. Restore the old key (rotation: `new,old`), or use a recovery code and the reset command. |+| Server marked offline | It missed 3 check-ins in a row. Check the machine, its collector log and its route to the dashboard. The incident resolves on the next report. |diff --git a/src/backend/main.py b/src/backend/main.pyindex 34c9fcf..01112e8 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -507,7 +507,7 @@ def two_factor_status(session_id: Annotated[str | None, Cookie(alias=SESSION_COOdef two_factor_setup(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):_session, user = _account_session(session_id, db, allow_limited=True)if not secrets_box.available():- raise HTTPException(status_code=503, detail="Two-factor needs SECRETS_KEY to be configured on the server. See the README.")+ raise HTTPException(status_code=503, detail="Two-factor needs SECRETS_KEY to be configured on the server. See docs/accounts.md (Two-factor sign-in).")if user.totp_enabled:raise HTTPException(status_code=409, detail="Two-factor is already turned on. Turn it off first to set it up again.")enforce_limit(redis_client, scope="2fa-setup", subject=user.id, limit=10, window_seconds=3600)diff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsxindex 99f9f2d..a17e05e 100644--- a/src/frontend/src/App.jsx+++ b/src/frontend/src/App.jsx@@ -1744,7 +1744,7 @@ function TwoFactorPolicySetting() {return (<AdminSection title="Two-factor" description="Protects sign-in with a code from an authenticator app, on top of the password."><Toggle label="Require two-factor for administrators" description="An administrator without it can only reach the two-factor setup until it is done." checked={state?.required ?? false} onChange={change} disabled={state === null || saving} />- {state && !state.configured && <p className="mt-3 rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-xs text-amber-200">SECRETS_KEY is not set on the server, so two-factor can't be set up and this is <strong>not being enforced</strong> yet. Set it in <code>.env</code> (see the README, Secrets at rest) and restart.</p>}+ {state && !state.configured && <p className="mt-3 rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-xs text-amber-200">SECRETS_KEY is not set on the server, so two-factor can't be set up and this is <strong>not being enforced</strong> yet. Set it in <code>.env</code> (see docs/accounts.md, Two-factor sign-in) and restart.</p>}{message && <p className={`mt-3 text-xs ${message.ok ? 'text-emerald-300' : 'text-amber-200'}`}>{message.text}</p>}</AdminSection>)@@ -2360,7 +2360,7 @@ function TwoFactorSection({ status, reload, onEnrolled }) {return (<AdminSection title="Two-factor sign-in" description="A 6-digit code from an authenticator app (such as Aegis, Authy or Google Authenticator) is asked for after your password.">- {!status.available && <p className="rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-sm text-amber-200">Two-factor can't be set up because the server has no <code>SECRETS_KEY</code>. An administrator needs to set one in <code>.env</code> (see the README, Secrets at rest).</p>}+ {!status.available && <p className="rounded-xl border border-amber-400/30 bg-amber-400/5 p-3 text-sm text-amber-200">Two-factor can't be set up because the server has no <code>SECRETS_KEY</code>. An administrator needs to set one in <code>.env</code> (see docs/accounts.md, Two-factor sign-in).</p>}{status.available && codes && <RecoveryCodesPanel codes={codes} doneLabel={status.setup_required || onEnrolled ? 'Continue' : 'Done'} onDone={() => { setCodes(null); if (onEnrolled) onEnrolled() }} />}