Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

docs: say what retention never deletes, and small clean-ups from the review (#222)

Karlo Kajba Šimanić8bdcfafa48da
diff --git a/ODSTUPANJA.md b/ODSTUPANJA.md
index 8598eba..4a9e10d 100644
--- a/ODSTUPANJA.md
+++ b/ODSTUPANJA.md
@@ -1,7 +1,8 @@
# Odstupanja od dizajna
Popis svega što se u implementaciji redizajna razlikuje od PDF-ova u `design/web/` i `design/mobile/`,
-s razlogom i, gdje postoji, GitHub issueom. Stanje: 14. 10. 2026.
+s razlogom i, gdje postoji, GitHub issueom. Stanje: 14. 10. 2026. Retci o incident agentu usklađeni su 10. 11. 2026.
+(automatska dijagnoza, ključ u Adminu i praćenje potrošnje sada postoje).
Mobilni prikazi (`design/mobile/`) napravljeni su 14. 10. 2026; njihova odstupanja su u odjeljku „Mobitel i tablet".
@@ -21,7 +22,7 @@ su anotacije dizajna i namjerno nisu ugrađeni.
| Dizajn | Implementacija | Razlog | Issue |
| --- | --- | --- | --- |
-| Agentova bilješka na svakom incidentu („Agent: limit is 4 GB…") | Bilješka dolazi iz lokalne dijagnoze po pravilima, ne od jezičnog modela | Automatska LLM dijagnoza ne postoji | #93 |
+| Agentova bilješka na svakom incidentu („Agent: limit is 4 GB…") | Bilješka je prva rečenica agentove dijagnoze kad ona postoji; inače dolazi iz lokalne dijagnoze po pravilima | Bez API ključa, ili dok dijagnoza ne stigne, nema teksta modela | – |
| „Ask the agent" otvara razgovor o tom kontejneru | Vodi na `/agent` bez konteksta kontejnera („Open diagnosis" otvara točno taj incident) | Chat još ne prima kontekst kontejnera | – |
| Zadnji incident: „Resolved 2 h ago · memory limit raised to 6 GB" | Piše samo kada je riješen i na kojem serveru | Nitko ne upisuje kako je incident riješen | – |
| „You'll get a push on ntfy if that changes" | Rečenica je izostavljena | Obavijesti ovise o kanalima koje je svaki korisnik postavio, pa rečenica ne bi vrijedila za svakoga | – |
@@ -31,7 +32,7 @@ su anotacije dizajna i namjerno nisu ugrađeni.
| „Another machine": polje samo za ime | Dodano je i polje za adresu na koju se novi stroj javlja | Bez ispravne adrese naredba ne radi (localhost, VPN) | – |
| Brisanje servera nije na Overviewu | „Remove this server" je i u System tabu, uz Admin → Servers | Zgodno je imati ga uz podatke servera | – |
| Kartica „What you'll get": „Kept for a year as 30-minute points" | „Kept as rolled-up points for as long as you set in Admin" | Zadržavanje je postavka, a točke nisu uvijek 30-minutne | – |
-| „The agent only sends data to your model provider when it diagnoses an incident" | „…when you ask it something" | Automatska dijagnoza ne postoji | #93 |
+| „The agent only sends data to your model provider when it diagnoses an incident" | „…when it diagnoses an incident or you ask it something, and only once an admin has saved an API key" | Podaci idu i kod pitanja u chatu, a bez ključa ne ide ništa | – |
## Tabovi servera (`08-server-tabs.pdf`)
@@ -86,9 +87,9 @@ su anotacije dizajna i namjerno nisu ugrađeni.
| Dizajn | Implementacija | Razlog | Issue |
| --- | --- | --- | --- |
-| Dijagnozu piše jezični model („watchtower updated the image to 17.4.1…") | Dijagnoza se gradi fiksnim pravilima iz spremljenih metrika i događaja; sažetak je opis pravila | Automatska LLM dijagnoza ne postoji; ovako ne košta ništa i ništa ne šalje van | #93 |
+| Dijagnozu piše jezični model („watchtower updated the image to 17.4.1…") | Kad postoji API ključ, dijagnozu piše model i ona se provjerava prema podacima; bez ključa, ili dok ne stigne, prikazuje se dijagnoza po fiksnim pravilima | Agent je opcionalan; dijagnoza po pravilima ne košta ništa i ništa ne šalje van | – |
| Evidence: retci iz loga kontejnera i `docker events` | Događaji koje je Infradeck zabilježio na tom serveru | Docker proxy ne može čitati logove | #100 |
-| „Confidence: high · 3 of 3 signals agree" | „3 of 3 checks against the data hold" | Nema modela koji bi procijenio pouzdanost | #93 |
+| „Confidence: high · 3 of 3 signals agree" | Agentova dijagnoza: oznaka „high confidence" i broj izostavljenih tvrdnji; dijagnoza po pravilima: „3 of 3 checks against the data hold" | Pouzdanost procjenjuje samo model; pravila je nemaju | – |
| Koraci s točnom putanjom (`/opt/gitlab/docker-compose.yml`) i brojkama iz povijesti verzija | Koraci iz fiksnog priručnika po vrsti pravila, s imenom kontejnera i izračunatim limitom | Infradeck ne zna gdje je compose datoteka ni koju verziju slike je kontejner imao prije | – |
| „Notified ntfy, email" | „sent to the channels people chose" ili „nobody yet" | Ne bilježi se po kanalu | – |
| Chat: odgovori s oznakama izvora („memory 12:30–13:55", „docker events") | Običan tekst odgovora; ispod njega sklopljeni „Looked up": pretrage koje je agent napravio, s njegovim pitanjem, što je tražio i redak činjenica iz rezultata | Izvori su pretrage alata, a ne oznake u tekstu; retke činjenica piše Infradeck, ne model | #158 |
@@ -128,12 +129,10 @@ su anotacije dizajna i namjerno nisu ugrađeni.
| Alert rules: stupac Servers s izborom servera | Uvijek „All" | Pravila vrijede za sve servere | – |
| Alert rules: „New rule" | Gumb je onemogućen, s oznakom „Coming soon" | Vlastita pravila ne postoje | – |
| Alert rules: bilješka zašto je pravilo isključeno | Samo broj isključenih pravila | Razlog se ne upisuje | – |
-| Incident agent: „Anthropic API · key ending 4f2a", „Change" | OpenAI, ključ i model iz `.env`; „Change" je onemogućen, „Test connection" provjerava ključ | Ključ se zasad postavlja u `.env`; spremanje u Adminu je dio #109 | #109 |
-| „Diagnose new incidents automatically" prekidač | Uvijek uključeno i onemogućeno | Dijagnoza je lokalna i ne može se isključiti | #93 |
+| Incident agent: „Anthropic API · key ending 4f2a", „Change" | Provider, model i ključ biraju se u Adminu; prikazuje se samo da je ključ spremljen, bez njegovih zadnjih znakova | Ključ, ni djelomično, ne napušta backend | – |
| „Read container logs" s iznimkama | Onemogućeno | Proxy ne čita logove | #100 |
| „Redact IPs and hostnames" | Onemogućeno | Nije napravljeno | – |
| Incident agent: nema retka za broj pretraga | Redak „Look-ups": koliko alata agent smije pozvati po dijagnozi (8) i po poruci u chatu (4) | Budžet poziva alata, podizanje s 0 traži lozinku i 2FA | #157 |
-| „$1.84 estimated provider cost · cap $10" | „— · not tracked" | Potrošnja se ne mjeri | #91 |
| Retention: tri zasebna roka | Jedan rok za sažetke; javljanja se čuvaju tjedan dana | Tako radi postojeće zadržavanje | #58 |
| Database: putanja, rast po tjednu, „Last backup", „Export", „Back up now" | Veličina baze i broj zapisa; gumbi su onemogućeni, s oznakom „Coming soon" | Backup ne postoji | – |
| About: „built 2 Oct 2026" | Samo verzija | Datum builda se ne bilježi | #106 |
@@ -151,14 +150,13 @@ su anotacije dizajna i namjerno nisu ugrađeni.
| Statistics: legenda „This week / Last week" | „Last 7 days / The previous 7 days" | Isti tekst kao na desktopu | – |
| Agent: „Show 5 log lines" | „Show N recorded events" | Infradeck ne čita logove kontejnera, nego bilježi događaje | #100 |
| Agent: „Push alerts are on · Critical only between 23:00 and 07:00" | Redak „Where alerts go" vodi u Account | Infradeck nema push obavijesti na mobitel (nema aplikacije ni Web Pusha) | #50 |
-| Agent chat: oznake izvora ispod odgovora | Običan tekst | Model ne vraća izvore | #93 |
+| Agent chat: oznake izvora ispod odgovora | Ispod odgovora su koraci istrage (što je agent provjerio i što je našao), ne oznake izvora | Odgovor u chatu ne citira činjenice; citati postoje samo u dijagnozi | – |
| More: „Owner" | Owner za admine, Member za ostale | Ne postoji uloga vlasnika | #106 |
| Security: redovi s strelicom i prekidačem za 2FA | Isti redovi kao na desktopu (gumbi „Change password", „Set up") | Postupci traže lozinku i kod, pa ostaju gumbi | – |
| Sessions: lokacija i „via WireGuard" | Adresa i vrijeme prijave | Nema geolokacije | – |
| Notifications: kanal „This iPhone · Push" | Nema | Nema push obavijesti | #50 |
| Notifications: redovi „Weekly report" i „Email digest 08:00" | Jedna rečenica da još ne postoje | Ne postoje | #33 |
| Admin: „Infradeck 0.9.3 is out" | Nema; hub prikazuje strojeve koji čekaju | Infradeck ne zove van | – |
-| Admin: „Incident agent · $1.84 of $10" | „chat on/off" ili „no key" | Potrošnja se ne mjeri | #91 |
| Admin: Connect a server u tri numerirana koraka | Ista kartica kao na desktopu, uz Copy i Share ispod naredbe | Jedna implementacija | – |
| Admin: „You'll get a push when it connects" | Nema | Nema push obavijesti | #50 |
| Alert rules: „+" za novo pravilo | Nema | Vlastita pravila ne postoje | – |
diff --git a/SECURITY.md b/SECURITY.md
index 4064e34..863fe86 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -504,6 +504,18 @@ URL, a keyword or a value in a JSON reply, a TCP port, a TLS certificate, a DNS
retention* (1 year by default, 2 years or forever; nothing shorter can be chosen); daily, weekly and
monthly summaries are kept forever. Nothing is deleted before the coarser level that replaces it exists,
so long-range history stays available while the bulky raw rows don't grow without bound.
+- **What the retention setting does not cover.** These are kept until you remove them, and no setting
+ deletes them by age:
+ - **incidents**, with the agent's reports on them (each report keeps the masked evidence it was based
+ on): until their server is removed in Admin, which deletes them with it;
+ - **the audit log** (who did what and when: sign-ins, admin changes, the agent's look-ups; an entry
+ about replaced collector credentials names the address it came from): kept, except that deleting a
+ chat conversation deletes its own entries;
+ - **chat conversations**: until the person who had them deletes them;
+ - **the agent's usage records** (model, tokens, cost, a masked error): kept.
+
+ They are small next to the metrics, but they grow, and the audit log and the chat hold personal data.
+ If you need them gone after some time, delete them in the database; nothing else depends on old rows.
## Container privileges
diff --git a/TESTS.md b/TESTS.md
index 7572d67..aec2646 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -150,7 +150,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
| [`test_agent_service.py`](#test_agent_servicepy) | 11 | The chat's reply is one message; secrets in the question and incident context are masked; Test connection trying a real request; the Chat Completions provider and conversation history. |
| [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |
| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. |
-| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 55 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
+| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 56 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
| [`test_agent_investigation.py`](#test_agent_investigationpy) | 27 | The agent looking things up: the budget of tool calls per diagnosis and chat message, forced final answers, errors, the audit log, citations from tool results. |
| [`test_agent_steps.py`](#test_agent_stepspy) | 34 | The agent's look-ups in plain words: what each tool asked and found, no raw results or secrets, windows left for the page, odd entries never failing. |
| [`test_agent_tools.py`](#test_agent_toolspy) | 52 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. |
@@ -392,6 +392,7 @@ request.
| `test_a_verify_condition_on_an_unknown_metric_becomes_manual` | A condition on a metric that isn't in the pack is not watched. |
| `test_an_unusable_answer_is_rejected` | Answers that aren't JSON, aren't an object or have no summary are rejected (3 cases). |
| `test_a_diagnosis_is_saved_as_a_new_version_and_counted` | Each diagnosis is a new version, each model call counts as `diagnosis`, the evidence is sent as untrusted and masked. |
+| `test_a_watched_metric_keeps_its_unit_from_the_evidence` | A condition on a metric is saved with that metric's unit from the evidence pack, so the page says "under 3500 MB" (#213). |
| `test_an_unusable_answer_is_asked_for_once_more` | An unusable answer is asked for once more, saying why. |
| `test_two_unusable_answers_give_a_failed_report` | After two unusable answers the report is saved as failed, with the reason and the evidence. |
| `test_a_provider_error_is_counted_and_raised_so_the_job_is_retried` | A provider error is recorded and raised so the queue retries the job; no report is saved. |
@@ -494,7 +495,7 @@ and a firewall.
| `test_a_container_that_ran_long_enough_without_restarting_resolves_the_incident` | A container running past the condition without a restart resolves the incident, with the condition as its resolution, and starts the hold. |
| `test_a_recent_restart_keeps_it_open_and_shows_progress` | A recent restart keeps the incident open and reports how long the condition has held. |
| `test_old_data_is_not_proof` | Samples older than 5 minutes never resolve an incident. |
-| `test_a_metric_under_its_threshold_long_enough_holds` | A metric under its threshold for long enough holds, described by the metric's name. |
+| `test_a_metric_under_its_threshold_long_enough_holds` | A metric under its threshold for long enough holds, described by the metric's name and, when the report has it, its unit ("under 80%", "under 3500 MB"). |
| `test_the_rule_does_not_reopen_what_the_agent_closed_until_the_hold_ends_or_it_breaks` | The rule stays quiet after the agent resolved its incident, until the problem comes back. |
| `test_manual_conditions_are_not_watched` | Conditions that need a person are never acted on. |
diff --git a/docs/database.md b/docs/database.md
index 781e821..4d192b1 100644
--- a/docs/database.md
+++ b/docs/database.md
@@ -46,6 +46,11 @@ only deleted once summarised. Hourly rollups, `check_rollups` and `system_events
once its day is summarised. Daily, weekly and monthly rollups are never deleted. Deletes run in chunks of
10,000 rows.
+**Not covered by retention.** `incident_alerts` and their `agent_reports` stay until their server is removed
+(which deletes them with it). `audit_logs` and `agent_runs` are never deleted by age; deleting a chat
+conversation deletes its `chat_messages` and its own audit entries (the agent's look-ups stay, unlinked). See
+[SECURITY.md](../SECURITY.md#statistics-and-history).
+
The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and
`(server_id, time)`, or `(server_id, container_id, time)` for containers. A database from before
migrations gets them from the baseline migration, which also drops the redundant single-column
diff --git a/src/agent/diagnosis.py b/src/agent/diagnosis.py
index f4cbb3a..f944c2f 100644
--- a/src/agent/diagnosis.py
+++ b/src/agent/diagnosis.py
@@ -184,6 +184,8 @@ def diagnose(db, incident: IncidentAlert, provider: Provider, trigger: str = "in
for attempt in range(MAX_ATTEMPTS):
try:
answer = check_answer(completion.text, labels, pack_labels, container)
+ # The unit of the watched metric, so the condition reads "under 80%" and not "under 80".
+ answer["verify"]["unit"] = next((metric["unit"] for metric in pack["metrics"] if metric["id"] == answer["verify"]["target"]), None)
except InvalidAnswer as exc:
reason = str(exc)
if attempt + 1 < MAX_ATTEMPTS:
diff --git a/src/agent/verify.py b/src/agent/verify.py
index 67cf8d1..0418d3e 100644
--- a/src/agent/verify.py
+++ b/src/agent/verify.py
@@ -35,7 +35,8 @@ def describe(verify: dict | None) -> str | None:
minutes = int(verify["minutes"])
if verify["type"] == "container_running_for":
return f"{verify.get('target') or 'the container'} runs {minutes} min without a restart"
- return f"{verify.get('label') or verify.get('target')} stays under {verify.get('threshold'):g} for {minutes} min"
+ unit = verify.get("unit") or "" # from the evidence pack (%, MB, GB); reports from before it was kept have none
+ return f"{verify.get('label') or verify.get('target')} stays under {verify.get('threshold'):g}{unit if unit == '%' else f' {unit}' if unit else ''} for {minutes} min"
def _metric_rows(db, incident: IncidentAlert, metric: str, since: datetime) -> list[tuple[datetime, float | None]]:
diff --git a/src/backend/routes/admin.py b/src/backend/routes/admin.py
index 166f989..d4a0156 100644
--- a/src/backend/routes/admin.py
+++ b/src/backend/routes/admin.py
@@ -15,7 +15,7 @@ from ...agent import jobs as agent_jobs, settings as agent_settings, usage as ag
from ...agent import providers as agent_providers
from ..config import APP_VERSION, SESSION_COOKIE_NAME
from ..db import engine, get_db
-from ..models import ChatMessage, HostMetric, IncidentAlert, Invite, Server, User
+from ..models import ChatMessage, HostMetric, IncidentAlert, Invite, User
router = APIRouter()
diff --git a/src/frontend/src/dashboard/EmptyFleet.jsx b/src/frontend/src/dashboard/EmptyFleet.jsx
index 122cd75..28f4fed 100644
--- a/src/frontend/src/dashboard/EmptyFleet.jsx
+++ b/src/frontend/src/dashboard/EmptyFleet.jsx
@@ -314,7 +314,7 @@ export function EmptyFleet({ overview, isAdmin, onChanged }) {
<div className="card grid divide-y divide-line sm:grid-cols-2 sm:divide-x xl:grid-cols-4 xl:divide-y-0">
{PROMISES.map(([title, text]) => <div key={title} className="px-[18px] py-4"><h3 className="m-0 text-[13px] font-semibold text-paper">{title}</h3><p className="mb-0 mt-2 text-xs leading-[19px] text-muted">{text}</p></div>)}
</div>
- <p className="mb-0 mt-3.5 text-xs text-muted">Everything stays on this machine. The agent only sends data to your model provider when you ask it something, and you can turn that off in Admin.</p>
+ <p className="mb-0 mt-3.5 text-xs text-muted">Everything stays on this machine. The agent only sends data to your model provider when it diagnoses an incident or you ask it something, and only once an admin has saved an API key. Both can be turned off in Admin.</p>
</section>
</div>
)
diff --git a/tests/assistant/test_agent_diagnosis.py b/tests/assistant/test_agent_diagnosis.py
index fc9e84e..e6d5479 100644
--- a/tests/assistant/test_agent_diagnosis.py
+++ b/tests/assistant/test_agent_diagnosis.py
@@ -205,6 +205,15 @@ def test_a_diagnosis_is_saved_as_a_new_version_and_counted(db):
assert shown["version"] == 2 and shown["signals"][0]["evidence"][1]["id"] == "check:1"
+def test_a_watched_metric_keeps_its_unit_from_the_evidence(db):
+ watched = {"type": "metric_below_for", "target": "metric:main", "threshold": 3500, "minutes": 10}
+ provider = FakeProvider(answer(signals=[{"text": "Memory hits the limit.", "evidence": ["metric:main"], "holds": True}], verify=watched))
+
+ report = diagnosis.diagnose(db, db.get(IncidentAlert, 42), provider)
+
+ assert json.loads(report.body)["verify"]["unit"] == "MB" # so the page says "under 3500 MB", not "under 3500"
+
+
def test_an_unusable_answer_is_asked_for_once_more(db):
provider = FakeProvider("not json", valid_answer())
diff --git a/tests/assistant/test_agent_watch.py b/tests/assistant/test_agent_watch.py
index b7689b0..88c1efe 100644
--- a/tests/assistant/test_agent_watch.py
+++ b/tests/assistant/test_agent_watch.py
@@ -132,7 +132,9 @@ def test_a_metric_under_its_threshold_long_enough_holds(db):
condition = {"type": "metric_below_for", "target": "metric:main", "threshold": 80, "minutes": 10, "label": "memory in use"}
result = verify.evaluate(db, db.get(IncidentAlert, 60), condition, NOW)
- assert result["holds"] and result["text"] == "memory in use stays under 80 for 10 min"
+ assert result["holds"] and result["text"] == "memory in use stays under 80 for 10 min" # a report from before units were kept
+ assert verify.describe({**condition, "unit": "%"}) == "memory in use stays under 80% for 10 min"
+ assert verify.describe({**condition, "unit": "MB", "threshold": 3500}) == "memory in use stays under 3500 MB for 10 min"
def test_the_rule_does_not_reopen_what_the_agent_closed_until_the_hold_ends_or_it_breaks(db):
diff --git a/tests/collectors/test_collector_checks.py b/tests/collectors/test_collector_checks.py
index 29cb030..0dfa97b 100644
--- a/tests/collectors/test_collector_checks.py
+++ b/tests/collectors/test_collector_checks.py
@@ -325,6 +325,12 @@ def test_validate_check_rejects_credentials_in_a_url():
# --- calls to the dashboard (#205) ------------------------------------------------------------------------------
+def _read_request(handler):
+ """Read the request's body. A server that closes with it unread makes the kernel reset the connection, and the
+ collector then sees "Connection reset" instead of the answer."""
+ handler.rfile.read(int(handler.headers.get("Content-Length") or 0))
+
+
def _serve(handler):
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
@@ -349,6 +355,7 @@ def test_a_call_to_the_dashboard_never_follows_a_redirect_with_the_token():
class SignInRedirect(http.server.BaseHTTPRequestHandler):
def do_POST(self): # noqa: N802
+ _read_request(self)
self.send_response(302)
self.send_header("Location", f"{elsewhere_url}/login?rd=secret-return-path")
self.end_headers()
@@ -370,6 +377,7 @@ def test_a_call_to_the_dashboard_never_follows_a_redirect_with_the_token():
def test_a_huge_answer_from_the_dashboard_is_refused(monkeypatch):
class Huge(http.server.BaseHTTPRequestHandler):
def do_POST(self): # noqa: N802
+ _read_request(self)
body = b"[" + b"0," * collector.MAX_BODY_BYTES + b"0]"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))