Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

fix(agent): give a diagnosis a deadline its job outlasts and mask provider errors (#219)

Karlo Kajba Šimanić58c3d6e95f05
diff --git a/TESTS.md b/TESTS.md
index 270f63f..7572d67 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -150,12 +150,12 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
| [`test_agent_service.py`](#test_agent_servicepy) | 11 | The chat's reply is one message; secrets in the question and incident context are masked; Test connection trying a real request; the Chat Completions provider and conversation history. |
| [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |
| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. |
-| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 53 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
+| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 55 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
| [`test_agent_investigation.py`](#test_agent_investigationpy) | 27 | The agent looking things up: the budget of tool calls per diagnosis and chat message, forced final answers, errors, the audit log, citations from tool results. |
| [`test_agent_steps.py`](#test_agent_stepspy) | 34 | The agent's look-ups in plain words: what each tool asked and found, no raw results or secrets, windows left for the page, odd entries never failing. |
| [`test_agent_tools.py`](#test_agent_toolspy) | 52 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. |
| [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. |
-| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |
+| [`test_agent_worker.py`](#test_agent_workerpy) | 13 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |
| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |
| [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. |
| [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |
@@ -395,6 +395,8 @@ request.
| `test_an_unusable_answer_is_asked_for_once_more` | An unusable answer is asked for once more, saying why. |
| `test_two_unusable_answers_give_a_failed_report` | After two unusable answers the report is saved as failed, with the reason and the evidence. |
| `test_a_provider_error_is_counted_and_raised_so_the_job_is_retried` | A provider error is recorded and raised so the queue retries the job; no report is saved. |
+| `test_a_provider_error_is_stored_without_the_key_it_repeats` | A provider error that repeats the API key is stored masked in the usage records (#209). |
+| `test_after_its_deadline_a_diagnosis_must_answer_and_its_job_has_time_for_that` | Past its deadline a diagnosis gets no more look-ups, and the worker's job time covers the answer after it, so a diagnosis never runs twice at once (#209). |
| `test_the_worker_diagnoses_and_reports_back` | The worker's job ends as "diagnosed" with the confidence and version. |
| `test_without_a_key_the_worker_leaves_the_rule_based_diagnosis` | Without an API key the job is skipped and says why. |
| `test_with_automatic_diagnosis_off_only_a_request_is_diagnosed` | With automatic diagnosis off, only a requested diagnosis runs. |
@@ -506,6 +508,7 @@ and a firewall.
| `test_the_incident_is_saved_even_when_redis_is_down` | A Redis failure loses the job, not the incident. |
| `test_a_finished_job_leaves_the_processing_list_and_shows_in_recent` | A finished job leaves the processing list and shows in the recent results. |
| `test_a_failing_job_is_retried_later_and_given_up_after_three_attempts` | A failing job is tried three times in all and then marked failed with its error. |
+| `test_a_failed_jobs_note_does_not_show_a_key_the_provider_repeated` | The error Admin shows for a recent job is masked, so a key the provider repeated doesn't show (#209). |
| `test_a_retry_waits_before_it_runs_again` | A retry waits 30 seconds before it can run. |
| `test_a_job_that_hangs_is_abandoned_after_its_time_limit` | A job past its time limit is abandoned and retried. |
| `test_jobs_interrupted_by_a_restart_go_back_to_the_queue` | Jobs a worker took but never finished go back to the queue when it starts again. |
diff --git a/docs/agent-investigation.md b/docs/agent-investigation.md
index 2258e21..2ce91b9 100644
--- a/docs/agent-investigation.md
+++ b/docs/agent-investigation.md
@@ -151,8 +151,10 @@ The budget is a number of tool calls per answer, set in **Admin › Incident age
- The model then gets one more step with tools not allowed (`allow_tools=False`), so it has to answer. The tool list
itself stays the same for the whole conversation, because changing it would make the provider's cache and Claude's
earlier thinking invalid.
-- The same happens when the month's spending cap is reached during an investigation, or (chat only) after 45
- seconds, so the reply comes back before the web server gives up waiting (120 seconds).
+- The same happens when the month's spending cap is reached during an investigation, after 45 seconds in the chat,
+ so the reply comes back before the web server gives up waiting (120 seconds), and after 5 minutes in a diagnosis.
+ The worker gives a diagnosis job enough time for the answer on top of that (`worker.JOB_TIMEOUT_SECONDS`, 14
+ minutes), so a slow diagnosis is never started a second time while the first one still runs.
- A model that still asks for tools after being told to answer is stopped and the answer fails, so a loop can't run
on. In the worst case a diagnosis makes its budget plus two model calls (plus one for an answer that has to be
asked for again).
@@ -249,8 +251,9 @@ with a real result from the tools' test database.
**Wording.** Change the functions in `steps.py`; reports pick it up at once, chat replies saved before keep their
old wording. Keep each line to facts the tool returned, never an interpretation.
-**The budget, the deadline, the extra steps.** `settings.TOOL_BUDGETS`, `chat.DEADLINE_SECONDS` and
-`investigate.EXTRA_STEPS`. The deadline must leave room for one more model call inside nginx's 120 seconds.
+**The budget, the deadline, the extra steps.** `settings.TOOL_BUDGETS`, `chat.DEADLINE_SECONDS`,
+`diagnosis.DEADLINE_SECONDS` and `investigate.EXTRA_STEPS`. The chat's deadline must leave room for one more model
+call inside nginx's 120 seconds; the worker's job time follows the diagnosis deadline by itself.
**The prompts.** `DIAGNOSIS_PROMPT.md` and `SYSTEM_PROMPT.md`, sections `[TOOLS]`. They tell the model to start
wide, cite tool IDs, give a specific reason, and never claim a look-up it didn't make.
diff --git a/src/agent/diagnosis.py b/src/agent/diagnosis.py
index 8775b3d..f4cbb3a 100644
--- a/src/agent/diagnosis.py
+++ b/src/agent/diagnosis.py
@@ -35,6 +35,10 @@ MAX_SIGNALS = 6
MAX_STEPS = 5
MAX_ATTEMPTS = 2
MIN_WATCH_MINUTES = 5
+# After this long the agent stops looking things up and has to answer (#209). The worker gives a job enough time
+# for that answer on top of it (worker.JOB_TIMEOUT_SECONDS), so a slow diagnosis is never started a second time
+# while the first one still runs.
+DEADLINE_SECONDS = 300
# A diagnosis of the same rule on the same server (so the same container, mount or check) this recent is reused for a
# new incident instead of asking the model again.
REUSE_WITHIN = timedelta(hours=24)
@@ -174,7 +178,7 @@ def diagnose(db, incident: IncidentAlert, provider: Provider, trigger: str = "in
started = time.monotonic()
try:
found = investigate(db, provider, _messages(pack, budget), SCHEMA, "incident_diagnosis", budget=budget, effort="high", record=record,
- audit=audit_log(db, title=f"Diagnosis of INC-{incident.id:04d}"))
+ audit=audit_log(db, title=f"Diagnosis of INC-{incident.id:04d}"), deadline_seconds=DEADLINE_SECONDS)
labels = {**found.labels, **pack_labels}
completion = found.completion
for attempt in range(MAX_ATTEMPTS):
diff --git a/src/agent/usage.py b/src/agent/usage.py
index 1c8cc0f..b8cd466 100644
--- a/src/agent/usage.py
+++ b/src/agent/usage.py
@@ -13,6 +13,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import func
from ..backend.models import AgentRun, AppSetting
+from .masking import mask_text
PRICES_KEY = "agent_prices"
CAP_KEY = "agent_monthly_cap_usd"
@@ -113,7 +114,7 @@ def record(db, *, kind: str, model: str, status: str = "ok", input_tokens: int =
user_id: int | None = None, conversation_id: str | None = None, incident_id: int | None = None, error: str | None = None) -> AgentRun:
run = AgentRun(kind=kind, status=status, model=model[:100], input_tokens=max(0, int(input_tokens or 0)), output_tokens=max(0, int(output_tokens or 0)),
cost_usd=cost_of(get_prices(db), model, input_tokens or 0, output_tokens or 0), duration_ms=max(0, int(duration_ms or 0)),
- user_id=user_id, conversation_id=conversation_id, incident_id=incident_id, error=(error or None) and error[:300])
+ user_id=user_id, conversation_id=conversation_id, incident_id=incident_id, error=(error or None) and mask_text(error)[:300])
db.add(run)
db.commit()
return run
diff --git a/src/agent/worker.py b/src/agent/worker.py
index 3ee3b06..009d3e7 100644
--- a/src/agent/worker.py
+++ b/src/agent/worker.py
@@ -20,12 +20,17 @@ import time
from src.agent import diagnosis, jobs, settings, usage, verify
from src.agent.providers import get_provider, key_configured
+from src.agent.providers.base import MAX_RETRIES, TIMEOUT_SECONDS
+from src.agent.masking import mask_text
from src.backend.config import REDIS_URL
from src.backend.db import SessionLocal
from src.backend.models import IncidentAlert
-# A diagnosis can take two model calls, each with its own retries; past this the job is given up and retried later.
-JOB_TIMEOUT_SECONDS = 300
+# Past this a job is given up and retried later. A diagnosis stops looking things up at its deadline; after that
+# come at most three model calls (one still running, the answer, and one more if the answer was unusable), each
+# with its own retries. The job's time covers all of that, so a slow diagnosis is never run twice at once (#209).
+MODEL_CALL_SECONDS = TIMEOUT_SECONDS * (MAX_RETRIES + 1)
+JOB_TIMEOUT_SECONDS = diagnosis.DEADLINE_SECONDS + 3 * MODEL_CALL_SECONDS
POLL_SECONDS = 5
# How often the conditions the agent gave are checked against the data (see verify.py).
WATCH_SECONDS = 60
@@ -98,7 +103,8 @@ def process(redis_client, raw: str, handlers=None, timeout: float = JOB_TIMEOUT_
try:
status_text, note_text = run_with_timeout(handler, job, timeout)
except Exception as exc:
- return jobs.retry_or_fail(redis_client, raw, job, f"{type(exc).__name__}: {exc}", round((time.monotonic() - started) * 1000))
+ # The provider's own words, which Admin shows in the list of recent jobs: masked like the usage records.
+ return jobs.retry_or_fail(redis_client, raw, job, mask_text(f"{type(exc).__name__}: {exc}"), round((time.monotonic() - started) * 1000))
jobs.finish(redis_client, raw, job, status_text, note_text, round((time.monotonic() - started) * 1000))
return status_text
diff --git a/tests/assistant/test_agent_diagnosis.py b/tests/assistant/test_agent_diagnosis.py
index 0d53db8..fc9e84e 100644
--- a/tests/assistant/test_agent_diagnosis.py
+++ b/tests/assistant/test_agent_diagnosis.py
@@ -227,6 +227,26 @@ def test_a_provider_error_is_counted_and_raised_so_the_job_is_retried(db):
assert run.status == "error" and db.query(AgentReport).count() == 0
+def test_a_provider_error_is_stored_without_the_key_it_repeats(db):
+ leaked = RuntimeError("Error code: 401 - Incorrect API key provided: sk-proj-" + "a" * 40)
+ with pytest.raises(RuntimeError):
+ diagnosis.diagnose(db, db.get(IncidentAlert, 42), FakeProvider(leaked))
+
+ error = db.query(AgentRun).one().error
+ assert "Incorrect API key" in error and "a" * 40 not in error # admins see what went wrong, not the key (#209)
+
+
+def test_after_its_deadline_a_diagnosis_must_answer_and_its_job_has_time_for_that(db, monkeypatch):
+ monkeypatch.setattr(diagnosis, "DEADLINE_SECONDS", 0)
+ provider = FakeProvider(valid_answer())
+
+ assert diagnosis.diagnose(db, db.get(IncidentAlert, 42), provider).status == "ok"
+ assert provider.started[0]["tools"] and provider.allowed == [False] # tools listed, but no more look-ups
+ # The answer after the deadline (and one still running, and one retry) fits in the job's time, so a slow diagnosis
+ # isn't started a second time while the first one runs (#209).
+ assert worker.JOB_TIMEOUT_SECONDS >= 300 + 3 * worker.MODEL_CALL_SECONDS
+
+
def test_the_worker_diagnoses_and_reports_back(db, monkeypatch):
monkeypatch.setattr(worker, "SessionLocal", lambda: db)
monkeypatch.setattr(db, "close", lambda: None)
diff --git a/tests/assistant/test_agent_worker.py b/tests/assistant/test_agent_worker.py
index bc9d9cd..9b0586a 100644
--- a/tests/assistant/test_agent_worker.py
+++ b/tests/assistant/test_agent_worker.py
@@ -112,6 +112,19 @@ def test_a_failing_job_is_retried_later_and_given_up_after_three_attempts(redis)
assert "did not answer" in agent_jobs.status(redis)["recent"][0]["note"]
+def test_a_failed_jobs_note_does_not_show_a_key_the_provider_repeated(redis):
+ def refused(job):
+ raise RuntimeError("Incorrect API key provided: sk-proj-" + "b" * 40)
+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
+
+ for _ in range(agent_jobs.MAX_ATTEMPTS):
+ agent_jobs.promote_due(redis, now=time.time() + 3600)
+ worker.process(redis, _take(redis), {agent_jobs.OPENED: refused})
+
+ note = agent_jobs.status(redis)["recent"][0]["note"]
+ assert "Incorrect API key" in note and "b" * 40 not in note # Admin shows the recent jobs (#209)
+
+
def test_a_retry_waits_before_it_runs_again(redis):
agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
worker.process(redis, _take(redis), {agent_jobs.OPENED: lambda job: 1 / 0})