Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
fix(agent): give a diagnosis a deadline its job outlasts and mask provider errors (#219)
Karlo Kajba Šimanić58c3d6e95f05
diff --git a/TESTS.md b/TESTS.mdindex 270f63f..7572d67 100644--- a/TESTS.md+++ b/TESTS.md@@ -150,12 +150,12 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_agent_service.py`](#test_agent_servicepy) | 11 | The chat's reply is one message; secrets in the question and incident context are masked; Test connection trying a real request; the Chat Completions provider and conversation history. || [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. || [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. |-| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 53 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |+| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 55 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. || [`test_agent_investigation.py`](#test_agent_investigationpy) | 27 | The agent looking things up: the budget of tool calls per diagnosis and chat message, forced final answers, errors, the audit log, citations from tool results. || [`test_agent_steps.py`](#test_agent_stepspy) | 34 | The agent's look-ups in plain words: what each tool asked and found, no raw results or secrets, windows left for the page, odd entries never failing. || [`test_agent_tools.py`](#test_agent_toolspy) | 52 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. || [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. |-| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |+| [`test_agent_worker.py`](#test_agent_workerpy) | 13 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. || [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. || [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. || [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |@@ -395,6 +395,8 @@ request.| `test_an_unusable_answer_is_asked_for_once_more` | An unusable answer is asked for once more, saying why. || `test_two_unusable_answers_give_a_failed_report` | After two unusable answers the report is saved as failed, with the reason and the evidence. || `test_a_provider_error_is_counted_and_raised_so_the_job_is_retried` | A provider error is recorded and raised so the queue retries the job; no report is saved. |+| `test_a_provider_error_is_stored_without_the_key_it_repeats` | A provider error that repeats the API key is stored masked in the usage records (#209). |+| `test_after_its_deadline_a_diagnosis_must_answer_and_its_job_has_time_for_that` | Past its deadline a diagnosis gets no more look-ups, and the worker's job time covers the answer after it, so a diagnosis never runs twice at once (#209). || `test_the_worker_diagnoses_and_reports_back` | The worker's job ends as "diagnosed" with the confidence and version. || `test_without_a_key_the_worker_leaves_the_rule_based_diagnosis` | Without an API key the job is skipped and says why. || `test_with_automatic_diagnosis_off_only_a_request_is_diagnosed` | With automatic diagnosis off, only a requested diagnosis runs. |@@ -506,6 +508,7 @@ and a firewall.| `test_the_incident_is_saved_even_when_redis_is_down` | A Redis failure loses the job, not the incident. || `test_a_finished_job_leaves_the_processing_list_and_shows_in_recent` | A finished job leaves the processing list and shows in the recent results. || `test_a_failing_job_is_retried_later_and_given_up_after_three_attempts` | A failing job is tried three times in all and then marked failed with its error. |+| `test_a_failed_jobs_note_does_not_show_a_key_the_provider_repeated` | The error Admin shows for a recent job is masked, so a key the provider repeated doesn't show (#209). || `test_a_retry_waits_before_it_runs_again` | A retry waits 30 seconds before it can run. || `test_a_job_that_hangs_is_abandoned_after_its_time_limit` | A job past its time limit is abandoned and retried. || `test_jobs_interrupted_by_a_restart_go_back_to_the_queue` | Jobs a worker took but never finished go back to the queue when it starts again. |diff --git a/docs/agent-investigation.md b/docs/agent-investigation.mdindex 2258e21..2ce91b9 100644--- a/docs/agent-investigation.md+++ b/docs/agent-investigation.md@@ -151,8 +151,10 @@ The budget is a number of tool calls per answer, set in **Admin › Incident age- The model then gets one more step with tools not allowed (`allow_tools=False`), so it has to answer. The tool listitself stays the same for the whole conversation, because changing it would make the provider's cache and Claude'searlier thinking invalid.-- The same happens when the month's spending cap is reached during an investigation, or (chat only) after 45- seconds, so the reply comes back before the web server gives up waiting (120 seconds).+- The same happens when the month's spending cap is reached during an investigation, after 45 seconds in the chat,+ so the reply comes back before the web server gives up waiting (120 seconds), and after 5 minutes in a diagnosis.+ The worker gives a diagnosis job enough time for the answer on top of that (`worker.JOB_TIMEOUT_SECONDS`, 14+ minutes), so a slow diagnosis is never started a second time while the first one still runs.- A model that still asks for tools after being told to answer is stopped and the answer fails, so a loop can't runon. In the worst case a diagnosis makes its budget plus two model calls (plus one for an answer that has to beasked for again).@@ -249,8 +251,9 @@ with a real result from the tools' test database.**Wording.** Change the functions in `steps.py`; reports pick it up at once, chat replies saved before keep theirold wording. Keep each line to facts the tool returned, never an interpretation.-**The budget, the deadline, the extra steps.** `settings.TOOL_BUDGETS`, `chat.DEADLINE_SECONDS` and-`investigate.EXTRA_STEPS`. The deadline must leave room for one more model call inside nginx's 120 seconds.+**The budget, the deadline, the extra steps.** `settings.TOOL_BUDGETS`, `chat.DEADLINE_SECONDS`,+`diagnosis.DEADLINE_SECONDS` and `investigate.EXTRA_STEPS`. The chat's deadline must leave room for one more model+call inside nginx's 120 seconds; the worker's job time follows the diagnosis deadline by itself.**The prompts.** `DIAGNOSIS_PROMPT.md` and `SYSTEM_PROMPT.md`, sections `[TOOLS]`. They tell the model to startwide, cite tool IDs, give a specific reason, and never claim a look-up it didn't make.diff --git a/src/agent/diagnosis.py b/src/agent/diagnosis.pyindex 8775b3d..f4cbb3a 100644--- a/src/agent/diagnosis.py+++ b/src/agent/diagnosis.py@@ -35,6 +35,10 @@ MAX_SIGNALS = 6MAX_STEPS = 5MAX_ATTEMPTS = 2MIN_WATCH_MINUTES = 5+# After this long the agent stops looking things up and has to answer (#209). The worker gives a job enough time+# for that answer on top of it (worker.JOB_TIMEOUT_SECONDS), so a slow diagnosis is never started a second time+# while the first one still runs.+DEADLINE_SECONDS = 300# A diagnosis of the same rule on the same server (so the same container, mount or check) this recent is reused for a# new incident instead of asking the model again.REUSE_WITHIN = timedelta(hours=24)@@ -174,7 +178,7 @@ def diagnose(db, incident: IncidentAlert, provider: Provider, trigger: str = "instarted = time.monotonic()try:found = investigate(db, provider, _messages(pack, budget), SCHEMA, "incident_diagnosis", budget=budget, effort="high", record=record,- audit=audit_log(db, title=f"Diagnosis of INC-{incident.id:04d}"))+ audit=audit_log(db, title=f"Diagnosis of INC-{incident.id:04d}"), deadline_seconds=DEADLINE_SECONDS)labels = {**found.labels, **pack_labels}completion = found.completionfor attempt in range(MAX_ATTEMPTS):diff --git a/src/agent/usage.py b/src/agent/usage.pyindex 1c8cc0f..b8cd466 100644--- a/src/agent/usage.py+++ b/src/agent/usage.py@@ -13,6 +13,7 @@ from datetime import datetime, timedelta, timezonefrom sqlalchemy import funcfrom ..backend.models import AgentRun, AppSetting+from .masking import mask_textPRICES_KEY = "agent_prices"CAP_KEY = "agent_monthly_cap_usd"@@ -113,7 +114,7 @@ def record(db, *, kind: str, model: str, status: str = "ok", input_tokens: int =user_id: int | None = None, conversation_id: str | None = None, incident_id: int | None = None, error: str | None = None) -> AgentRun:run = AgentRun(kind=kind, status=status, model=model[:100], input_tokens=max(0, int(input_tokens or 0)), output_tokens=max(0, int(output_tokens or 0)),cost_usd=cost_of(get_prices(db), model, input_tokens or 0, output_tokens or 0), duration_ms=max(0, int(duration_ms or 0)),- user_id=user_id, conversation_id=conversation_id, incident_id=incident_id, error=(error or None) and error[:300])+ user_id=user_id, conversation_id=conversation_id, incident_id=incident_id, error=(error or None) and mask_text(error)[:300])db.add(run)db.commit()return rundiff --git a/src/agent/worker.py b/src/agent/worker.pyindex 3ee3b06..009d3e7 100644--- a/src/agent/worker.py+++ b/src/agent/worker.py@@ -20,12 +20,17 @@ import timefrom src.agent import diagnosis, jobs, settings, usage, verifyfrom src.agent.providers import get_provider, key_configured+from src.agent.providers.base import MAX_RETRIES, TIMEOUT_SECONDS+from src.agent.masking import mask_textfrom src.backend.config import REDIS_URLfrom src.backend.db import SessionLocalfrom src.backend.models import IncidentAlert-# A diagnosis can take two model calls, each with its own retries; past this the job is given up and retried later.-JOB_TIMEOUT_SECONDS = 300+# Past this a job is given up and retried later. A diagnosis stops looking things up at its deadline; after that+# come at most three model calls (one still running, the answer, and one more if the answer was unusable), each+# with its own retries. The job's time covers all of that, so a slow diagnosis is never run twice at once (#209).+MODEL_CALL_SECONDS = TIMEOUT_SECONDS * (MAX_RETRIES + 1)+JOB_TIMEOUT_SECONDS = diagnosis.DEADLINE_SECONDS + 3 * MODEL_CALL_SECONDSPOLL_SECONDS = 5# How often the conditions the agent gave are checked against the data (see verify.py).WATCH_SECONDS = 60@@ -98,7 +103,8 @@ def process(redis_client, raw: str, handlers=None, timeout: float = JOB_TIMEOUT_try:status_text, note_text = run_with_timeout(handler, job, timeout)except Exception as exc:- return jobs.retry_or_fail(redis_client, raw, job, f"{type(exc).__name__}: {exc}", round((time.monotonic() - started) * 1000))+ # The provider's own words, which Admin shows in the list of recent jobs: masked like the usage records.+ return jobs.retry_or_fail(redis_client, raw, job, mask_text(f"{type(exc).__name__}: {exc}"), round((time.monotonic() - started) * 1000))jobs.finish(redis_client, raw, job, status_text, note_text, round((time.monotonic() - started) * 1000))return status_textdiff --git a/tests/assistant/test_agent_diagnosis.py b/tests/assistant/test_agent_diagnosis.pyindex 0d53db8..fc9e84e 100644--- a/tests/assistant/test_agent_diagnosis.py+++ b/tests/assistant/test_agent_diagnosis.py@@ -227,6 +227,26 @@ def test_a_provider_error_is_counted_and_raised_so_the_job_is_retried(db):assert run.status == "error" and db.query(AgentReport).count() == 0+def test_a_provider_error_is_stored_without_the_key_it_repeats(db):+ leaked = RuntimeError("Error code: 401 - Incorrect API key provided: sk-proj-" + "a" * 40)+ with pytest.raises(RuntimeError):+ diagnosis.diagnose(db, db.get(IncidentAlert, 42), FakeProvider(leaked))++ error = db.query(AgentRun).one().error+ assert "Incorrect API key" in error and "a" * 40 not in error # admins see what went wrong, not the key (#209)+++def test_after_its_deadline_a_diagnosis_must_answer_and_its_job_has_time_for_that(db, monkeypatch):+ monkeypatch.setattr(diagnosis, "DEADLINE_SECONDS", 0)+ provider = FakeProvider(valid_answer())++ assert diagnosis.diagnose(db, db.get(IncidentAlert, 42), provider).status == "ok"+ assert provider.started[0]["tools"] and provider.allowed == [False] # tools listed, but no more look-ups+ # The answer after the deadline (and one still running, and one retry) fits in the job's time, so a slow diagnosis+ # isn't started a second time while the first one runs (#209).+ assert worker.JOB_TIMEOUT_SECONDS >= 300 + 3 * worker.MODEL_CALL_SECONDS++def test_the_worker_diagnoses_and_reports_back(db, monkeypatch):monkeypatch.setattr(worker, "SessionLocal", lambda: db)monkeypatch.setattr(db, "close", lambda: None)diff --git a/tests/assistant/test_agent_worker.py b/tests/assistant/test_agent_worker.pyindex bc9d9cd..9b0586a 100644--- a/tests/assistant/test_agent_worker.py+++ b/tests/assistant/test_agent_worker.py@@ -112,6 +112,19 @@ def test_a_failing_job_is_retried_later_and_given_up_after_three_attempts(redis)assert "did not answer" in agent_jobs.status(redis)["recent"][0]["note"]+def test_a_failed_jobs_note_does_not_show_a_key_the_provider_repeated(redis):+ def refused(job):+ raise RuntimeError("Incorrect API key provided: sk-proj-" + "b" * 40)+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")++ for _ in range(agent_jobs.MAX_ATTEMPTS):+ agent_jobs.promote_due(redis, now=time.time() + 3600)+ worker.process(redis, _take(redis), {agent_jobs.OPENED: refused})++ note = agent_jobs.status(redis)["recent"][0]["note"]+ assert "Incorrect API key" in note and "b" * 40 not in note # Admin shows the recent jobs (#209)++def test_a_retry_waits_before_it_runs_again(redis):agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")worker.process(redis, _take(redis), {agent_jobs.OPENED: lambda job: 1 / 0})