Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat(agent): worker container and job queue for incident diagnoses
karloks200559efff9724c1
diff --git a/SECURITY.md b/SECURITY.mdindex cdca572..236705e 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -10,6 +10,7 @@ This page says what each component can reach, so you can decide what to run.| Collector | container | host `/proc`, the host filesystem (read-only), the Docker proxy, the backend over HTTP | No Docker socket in the compose installs. || Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows four read-only requests. See below. || System helper (optional) | container, uid 0 with **one** capability | kernel log, `/var/log/wtmp` (read-only), one output volume | Off by default. Details below. |+| Agent worker | container, no capabilities, read-only filesystem | PostgreSQL, Redis, the model provider on the internet | Runs the agent's jobs from a queue in Redis; no published port. || Collector gate (server stack) | nginx container, no capabilities | the backend; publishes one port on your VPN address | The only way to reach collector endpoints. See below. || Backend / frontend / database / Redis | containers | each other, over separate networks | Only the frontend port is published. Redis and PostgreSQL are on a network with no route out and are reachable only by the backend. Redis needs a password. |diff --git a/TESTS.md b/TESTS.mdindex 8a840f3..61930b6 100644--- a/TESTS.md+++ b/TESTS.md@@ -148,6 +148,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. || [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. || [`test_agent_service.py`](#test_agent_servicepy) | 9 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. |+| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. || [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. || [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. || [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |@@ -317,6 +318,23 @@ shouldn't, or an admin might be locked out.| `test_only_recent_turns_go_back_to_the_model` | Long conversations send only the most recent turns. || `test_a_failed_turn_is_not_kept_in_the_conversation` | A message that failed (timeout, provider error) is not remembered as part of the conversation. |+### test_agent_worker.py++| Test | What it checks |+|---|---|+| `test_opening_an_incident_queues_exactly_one_job_after_the_commit` | A new WARNING or CRITICAL incident queues one job, only once the database commit is done, and refreshing it queues no more. |+| `test_getting_worse_queues_one_more_job_and_getting_better_does_not` | Going from warning to critical queues one `incident.worsened` job; repeating it or going back down queues nothing. |+| `test_info_incidents_and_rolled_back_ones_queue_nothing` | INFO notes and incidents whose transaction was rolled back never reach the queue. |+| `test_the_incident_is_saved_even_when_redis_is_down` | A Redis failure loses the job, not the incident. |+| `test_a_finished_job_leaves_the_processing_list_and_shows_in_recent` | A finished job leaves the processing list and shows in the recent results. |+| `test_a_failing_job_is_retried_later_and_given_up_after_three_attempts` | A failing job is tried three times in all and then marked failed with its error. |+| `test_a_retry_waits_before_it_runs_again` | A retry waits 30 seconds before it can run. |+| `test_a_job_that_hangs_is_abandoned_after_its_time_limit` | A job past its time limit is abandoned and retried. |+| `test_jobs_interrupted_by_a_restart_go_back_to_the_queue` | Jobs a worker took but never finished go back to the queue when it starts again. |+| `test_garbage_in_the_queue_is_dropped_not_retried` | Something in the queue that isn't a job is dropped, not retried forever. |+| `test_a_job_checks_that_its_incident_still_needs_a_diagnosis` | A job skips incidents that are gone or resolved. |+| `test_status_says_whether_a_worker_is_alive` | The worker's heartbeat is what Admin shows as online. |+### test_alerts.pyThe incident engine and the container rules.diff --git a/compose.yml b/compose.ymlindex c454399..13469aa 100644--- a/compose.yml+++ b/compose.yml@@ -99,6 +99,39 @@ services:timeout: 5sretries: 5+ # Runs the incident agent's jobs (diagnoses) from the queue in Redis, apart from the backend, so a slow or+ # stuck model never holds up the API. Stopping it only delays diagnoses; everything else keeps working.+ # It needs the database, Redis and the internet (the model provider), and nothing else. See docs/containers.md.+ agent-worker:+ build:+ context: .+ dockerfile: docker/backend.Dockerfile+ image: infradeck-backend:latest+ container_name: infradeck-agent-worker+ restart: unless-stopped+ command: ["python", "-m", "src.agent.worker"]+ networks: [data, front]+ environment:+ DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}+ REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}+ PYTHONDONTWRITEBYTECODE: "1"+ cap_drop: [ALL]+ read_only: true+ tmpfs:+ - /tmp+ security_opt:+ - no-new-privileges:true+ healthcheck:+ test: ["CMD", "python", "-m", "src.agent.worker", "--health"]+ interval: 30s+ timeout: 5s+ retries: 3+ depends_on:+ backend:+ condition: service_healthy+# Monitors the machine running this stack. Other servers use collector-compose.yml.collector:build:diff --git a/docker-compose.server.yml b/docker-compose.server.ymlindex 3995d54..153c347 100644--- a/docker-compose.server.yml+++ b/docker-compose.server.yml@@ -105,6 +105,39 @@ services:timeout: 5sretries: 5+ # Runs the incident agent's jobs (diagnoses) from the queue in Redis, apart from the backend, so a slow or+ # stuck model never holds up the API. Stopping it only delays diagnoses; everything else keeps working.+ # It needs the database, Redis and the internet (the model provider), and nothing else. See docs/containers.md.+ agent-worker:+ build:+ context: .+ dockerfile: docker/backend.Dockerfile+ image: infradeck-backend:latest+ container_name: infradeck-agent-worker+ restart: unless-stopped+ command: ["python", "-m", "src.agent.worker"]+ networks: [data, front]+ environment:+ DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}+ REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}+ PYTHONDONTWRITEBYTECODE: "1"+ cap_drop: [ALL]+ read_only: true+ tmpfs:+ - /tmp+ security_opt:+ - no-new-privileges:true+ healthcheck:+ test: ["CMD", "python", "-m", "src.agent.worker", "--health"]+ interval: 30s+ timeout: 5s+ retries: 3+ depends_on:+ backend:+ condition: service_healthy+# Monitors the machine running this stack. Other servers use collector-compose.yml.collector:build:diff --git a/docs/api.md b/docs/api.mdindex 9b5e232..169e579 100644--- a/docs/api.md+++ b/docs/api.md@@ -30,6 +30,7 @@ All paths are relative to `/api` when going through the frontend/nginx.| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds; the agent's model prices, monthly budget and daily chat limit. || `GET /agent/config` | session | Whether chat with the agent can be used: switched on, a key configured, and the monthly budget not used up. || `GET /admin/agent/usage` | admin | This month's model calls, tokens and estimated cost against the budget, per model and per day, with the prices and limits. |+| `GET /admin/agent/worker` | admin | Whether the agent worker is running, how many jobs wait, are running or wait to be retried, and the latest results. || `POST /admin/agent/test` | admin | Checks the OpenAI key and that the configured model exists, without spending tokens. || `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. || `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |diff --git a/docs/containers.md b/docs/containers.mdindex 36961e4..ac72668 100644--- a/docs/containers.md+++ b/docs/containers.md@@ -16,7 +16,8 @@ For the settings, see [Configuration](configuration.md).| `infradeck-frontend` | built from `docker/frontend.Dockerfile` (nginx) | dashboard machine | yes | Serves the web app and passes `/api` to the backend. || `infradeck-backend` | built from `docker/backend.Dockerfile` (Python, FastAPI) | dashboard machine | yes | The API: sign-in, metrics intake, alerts, statistics, notifications, the assistant. || `infradeck-postgres` | `postgres:16-alpine` | dashboard machine | yes | Everything that must be kept: accounts, servers, metric history, incidents, settings. |-| `infradeck-redis` | `redis:7-alpine` | dashboard machine | yes | Short-lived state: sign-in sessions, rate-limit counters, pending two-factor logins, a small cache. |+| `infradeck-redis` | `redis:7-alpine` | dashboard machine | yes | Short-lived state: sign-in sessions, rate-limit counters, pending two-factor logins, a small cache, the agent's job queue. |+| `infradeck-agent-worker` | the backend image | dashboard machine | for the incident agent | Runs the agent's jobs (diagnoses) from the queue in Redis, apart from the API. || `infradeck-collector` | built from `docker/collector.Dockerfile` | every monitored machine | yes, one per machine | Reads the machine's CPU, memory, disks, network and containers and sends them to the backend. || `infradeck-docker-proxy` | built from `docker/docker-proxy.Dockerfile` | every monitored machine | yes, next to each collector | The only container with the Docker socket; answers six read-only questions. || `infradeck-collector-gate` | `nginx:alpine` | dashboard machine, server stack only | yes in `docker-compose.server.yml` | The only door to the collector endpoints, published on your VPN address. |@@ -170,6 +171,7 @@ fails until it is back. Nothing already stored is lost: it is all in the volume.| `2fa-pending:…` and its attempt counter | A password check that still waits for the authenticator code. | 5 minutes || `rate-limit:…` | Attempt counters for sign-in, registration, two-factor, password change, collector registration, invite checks, the assistant chat, notification tests and more. | the length of each window (e.g. 60 s) || `stats:…` | Cached fleet statistics, so the Statistics page doesn't recompute them on every load. | 60 seconds |+| `agent:queue`, `agent:processing`, `agent:delayed`, `agent:recent`, `agent:worker` | The agent worker's jobs: waiting, being run, waiting to be retried, the last 50 results, and the worker's heartbeat. | until done; the heartbeat 30 seconds |Session IDs and the values limited per address or email are stored only as SHA-256 digests, so a copy ofRedis doesn't hand out working sessions or a list of email addresses.@@ -198,6 +200,32 @@ checked, so users appear signed out. `/health` shows `redis: down`. Collectors kauthenticate with their own credentials stored in PostgreSQL), so no monitoring data is lost. When Rediscomes back, sessions saved in its volume work again.+## infradeck-agent-worker++**What it does.** Runs the incident agent's jobs. When an incident opens, or gets more severe, the backend puts one+job in a queue in Redis; the worker takes jobs one at a time and works on them. Automatic diagnosis (#116) is not+built yet, so for now a job only checks that its incident still needs a diagnosis and notes that; the chat runs in+the backend.++**How it works.** It is the backend image started with `python -m src.agent.worker`. Jobs are published only after+the database commit that opened the incident, and each event once. A job the worker takes moves to a processing+list and leaves it only when it is done, so a job interrupted by a restart goes back to the queue. A failed job is+tried again after 30 and 60 seconds and then given up; a job running longer than 2 minutes is abandoned. The+worker writes a heartbeat every few seconds, which **Admin → Incident agent → Worker** and the container's+healthcheck read. It runs with no capabilities, a read-only filesystem and no published port.++**Why a separate container.** A diagnosis means calls to a language model that can take tens of seconds or hang.+In the backend, that would tie up the threads that serve the dashboard and take check-ins. In its own container+it can be slow, restarted or stopped without the rest noticing. Redis is already part of the stack, so the queue+needed nothing new; a dedicated queue system (Celery, RabbitMQ) would add containers for one job type.++**Networks.** `data` (PostgreSQL and Redis) and `front`, because `data` has no route out and the worker has to+reach the model provider on the internet.++**If it stops.** New incidents wait in the queue for their diagnosis and are worked on when it starts again (each+waiting job is kept). Incidents, notifications, the chat and everything else keep working. Admin shows the+worker as offline.+## infradeck-collector**What it does.** Measures the machine it runs on and sends one check-in to the backend every 60 secondsdiff --git a/src/agent/worker.py b/src/agent/worker.pynew file mode 100644index 0000000..4274585--- /dev/null+++ b/src/agent/worker.py@@ -0,0 +1,117 @@+"""The agent worker: takes jobs from the queue in Redis (see src/backend/agent_jobs.py) and runs them one at a time.++It runs in its own container, so a slow or stuck model never holds up the backend, and stopping it only means+incidents wait for their diagnosis; everything else keeps working. Run with ``python -m src.agent.worker``;+``--health`` exits 0 while a worker is alive, for the container's healthcheck.++Automatic diagnosis (#116) is not built yet, so a job only checks that its incident still needs one and notes+that. The plumbing (queue, retries, time limit, status in Admin) is what this file provides.+"""+from __future__ import annotations++import json+import signal+import sys+import threading+import time++from src.backend import agent_jobs, agent_usage+from src.backend.config import REDIS_URL+from src.backend.db import SessionLocal+from src.backend.models import IncidentAlert++JOB_TIMEOUT_SECONDS = 120+POLL_SECONDS = 5+++def handle_incident(job: dict) -> tuple[str, str]:+ """``incident.opened`` and ``incident.worsened``. Returns (status, note) for the recent-jobs list."""+ db = SessionLocal()+ try:+ incident = db.get(IncidentAlert, job["incident_id"])+ if incident is None:+ return "skipped", "The incident no longer exists."+ if incident.status != "active":+ return "skipped", "The incident was already resolved."+ if agent_usage.budget(db)["state"] == "paused":+ return "skipped", "This month's budget for the agent is used up."+ return "noted", "Automatic diagnosis is not built yet (#116); the incident keeps its rule-based diagnosis."+ finally:+ db.close()+++HANDLERS = {agent_jobs.OPENED: handle_incident, agent_jobs.WORSENED: handle_incident}+++def run_with_timeout(handler, job: dict, seconds: float) -> tuple[str, str]:+ """Run ``handler(job)`` in a thread and stop waiting after ``seconds``. Raises TimeoutError, or what the handler raised.++ A thread can't be killed, so a handler past its time is abandoned (it is a daemon and ends with the process);+ the model client's own timeout keeps that from piling up."""+ outcome: dict = {}++ def target():+ try:+ outcome["result"] = handler(job)+ except BaseException as exc: # handed back to the caller below+ outcome["error"] = exc++ thread = threading.Thread(target=target, name=f"agent-job-{job.get('id', '')[:8]}", daemon=True)+ thread.start()+ thread.join(seconds)+ if thread.is_alive():+ raise TimeoutError(f"The job took longer than {seconds:g} s.")+ if "error" in outcome:+ raise outcome["error"]+ return outcome["result"]+++def process(redis_client, raw: str, handlers=None, timeout: float = JOB_TIMEOUT_SECONDS) -> str:+ """One job from the processing list: run it, then mark it finished, retry it later or give up. Returns the outcome."""+ handlers = handlers or HANDLERS+ started = time.monotonic()+ try:+ job = json.loads(raw)+ handler = handlers[job["type"]]+ except (ValueError, KeyError, TypeError):+ agent_jobs.finish(redis_client, raw, {"type": "unknown"}, "failed", "Not a job this worker knows.")+ return "failed"+ try:+ status_text, note_text = run_with_timeout(handler, job, timeout)+ except Exception as exc:+ return agent_jobs.retry_or_fail(redis_client, raw, job, f"{type(exc).__name__}: {exc}", round((time.monotonic() - started) * 1000))+ agent_jobs.finish(redis_client, raw, job, status_text, note_text, round((time.monotonic() - started) * 1000))+ return status_text+++def main() -> int:+ import redis++ # Waiting for a job blocks for POLL_SECONDS, so reads must be allowed to take longer than that.+ client = redis.Redis.from_url(REDIS_URL, decode_responses=True, socket_timeout=POLL_SECONDS + 10, socket_connect_timeout=5)+ if "--health" in sys.argv:+ return 0 if client.get(agent_jobs.HEARTBEAT) else 1++ stopping = threading.Event()+ for name in (signal.SIGTERM, signal.SIGINT):+ signal.signal(name, lambda *_: stopping.set())+ started_at = agent_jobs.now_iso()+ recovered = agent_jobs.recover(client)+ print(f"Agent worker started{f'; {recovered} unfinished jobs back in the queue' if recovered else ''}.", flush=True)+ while not stopping.is_set():+ try:+ agent_jobs.beat(client, started_at)+ agent_jobs.promote_due(client)+ raw = agent_jobs.take(client, POLL_SECONDS)+ if raw is not None:+ outcome = process(client, raw)+ print(f"Agent job {outcome}: {raw[:200]}", flush=True)+ except redis.RedisError as exc: # Redis restarting: wait and carry on+ print(f"Agent worker can't reach Redis: {exc}", flush=True)+ stopping.wait(POLL_SECONDS)+ print("Agent worker stopped.", flush=True)+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/src/backend/agent_jobs.py b/src/backend/agent_jobs.pynew file mode 100644index 0000000..b7705f2--- /dev/null+++ b/src/backend/agent_jobs.py@@ -0,0 +1,155 @@+"""The agent's job queue in Redis: the backend publishes, the agent worker (src/agent/worker.py) consumes.++When an incident opens, or gets more severe, the backend queues one job for the worker. Jobs are published only+after the database commit that created or changed the incident, so the worker never looks for an incident that+was rolled back, and each event is queued once (``incident.opened`` once per incident, ``incident.worsened`` once+per incident and severity).++The queue is reliable for one worker: a job the worker takes moves to a processing list and leaves it only when+it is finished, so a job interrupted by a crash or a restart goes back to the queue when the worker starts again.+A job that fails is tried again later, up to MAX_ATTEMPTS times. If Redis is down, the incident is still saved;+only the job is lost, and that is logged.+"""+from __future__ import annotations++import json+import time+import uuid+from collections.abc import Callable+from datetime import datetime, timezone++from sqlalchemy import event+from sqlalchemy.orm import Session++QUEUE = "infradeck:agent:queue"+PROCESSING = "infradeck:agent:processing"+DELAYED = "infradeck:agent:delayed"+RECENT = "infradeck:agent:recent"+HEARTBEAT = "infradeck:agent:worker"+ONCE = "infradeck:agent:once:"++OPENED, WORSENED = "incident.opened", "incident.worsened"+JOB_TYPES = (OPENED, WORSENED)+# INFO incidents (an image changed, a container went away) are notes, not problems to diagnose.+DIAGNOSED_SEVERITIES = ("WARNING", "CRITICAL")+SEVERITY_RANK = {"INFO": 0, "WARNING": 1, "CRITICAL": 2}+MAX_ATTEMPTS = 3+RETRY_BASE_SECONDS = 30+ONCE_SECONDS = 30 * 86400+RECENT_KEPT = 50+HEARTBEAT_SECONDS = 30++# Set by the backend (and the worker) to how they reach Redis; read when a job is published.+redis_getter: Callable[[], object] | None = None+++def now_iso() -> str:+ return datetime.now(timezone.utc).isoformat()+++def new_job(job_type: str, incident_id: int, severity: str | None = None) -> dict:+ return {"id": uuid.uuid4().hex, "type": job_type, "incident_id": int(incident_id), "severity": severity, "attempts": 0, "enqueued_at": now_iso()}+++def publish(redis_client, job_type: str, incident_id: int, severity: str | None = None) -> bool:+ """Queue one job, unless the same event was already queued. Returns True if it was queued."""+ if job_type not in JOB_TYPES:+ raise ValueError(f"Unknown agent job: {job_type}")+ once = f"{ONCE}{job_type}:{incident_id}" + (f":{severity}" if job_type == WORSENED else "")+ if not redis_client.set(once, "1", nx=True, ex=ONCE_SECONDS):+ return False+ redis_client.lpush(QUEUE, json.dumps(new_job(job_type, incident_id, severity)))+ return True+++# ---------- the backend side: noting events in a database session and publishing them after commit ----------++def note(db, job_type: str, incident, severity: str) -> None:+ """Called where incidents are opened or escalated; the job is published once the session commits."""+ if severity in DIAGNOSED_SEVERITIES:+ db.info.setdefault("agent_pending", []).append((job_type, incident, severity))++[email protected]_for(Session, "after_flush")+def _collect_ids(session, _context):+ pending = session.info.pop("agent_pending", None)+ if pending:+ session.info.setdefault("agent_ready", []).extend((job_type, incident.id, severity) for job_type, incident, severity in pending if incident.id is not None)++[email protected]_for(Session, "after_commit")+def _publish_after_commit(session):+ ready = session.info.pop("agent_ready", None)+ if not ready or redis_getter is None:+ return+ try:+ client = redis_getter()+ for job_type, incident_id, severity in ready:+ publish(client, job_type, incident_id, severity)+ except Exception as exc: # the incident is saved either way; only its diagnosis job is lost+ print(f"Could not queue agent jobs: {type(exc).__name__}: {exc}", flush=True)++[email protected]_for(Session, "after_rollback")+def _forget_after_rollback(session):+ session.info.pop("agent_pending", None)+ session.info.pop("agent_ready", None)+++# ---------- the worker side ----------++def recover(redis_client) -> int:+ """Put jobs a previous worker took but never finished back in the queue. Returns how many."""+ moved = 0+ while redis_client.lmove(PROCESSING, QUEUE, "LEFT", "RIGHT") is not None:+ moved += 1+ return moved+++def promote_due(redis_client, now: float | None = None) -> int:+ """Move retries whose wait is over from the delayed set back to the queue."""+ moved = 0+ for raw in redis_client.zrangebyscore(DELAYED, "-inf", now if now is not None else time.time()):+ if redis_client.zrem(DELAYED, raw):+ redis_client.rpush(QUEUE, raw)+ moved += 1+ return moved+++def take(redis_client, timeout: int = 5) -> str | None:+ """The oldest job, moved to the processing list; None when nothing arrived within ``timeout`` seconds."""+ return redis_client.blmove(QUEUE, PROCESSING, timeout, "RIGHT", "LEFT")+++def finish(redis_client, raw: str, job: dict, status: str, note_text: str = "", duration_ms: int = 0) -> None:+ redis_client.lrem(PROCESSING, 1, raw)+ redis_client.lpush(RECENT, json.dumps({"id": job.get("id"), "type": job.get("type"), "incident_id": job.get("incident_id"), "status": status,+ "note": note_text[:300], "attempts": job.get("attempts", 0) + 1, "duration_ms": duration_ms, "finished_at": now_iso()}))+ redis_client.ltrim(RECENT, 0, RECENT_KEPT - 1)+++def retry_or_fail(redis_client, raw: str, job: dict, error: str, duration_ms: int = 0, now: float | None = None) -> str:+ """After a failure: wait and try again (30 s, then 60 s), or give up after MAX_ATTEMPTS. Returns what happened."""+ attempts = job.get("attempts", 0) + 1+ if attempts >= MAX_ATTEMPTS:+ finish(redis_client, raw, job, "failed", error, duration_ms)+ return "failed"+ redis_client.lrem(PROCESSING, 1, raw)+ later = {**job, "attempts": attempts, "last_error": error[:300]}+ redis_client.zadd(DELAYED, {json.dumps(later): (now if now is not None else time.time()) + RETRY_BASE_SECONDS * 2 ** (attempts - 1)})+ return "retry"+++def beat(redis_client, started_at: str) -> None:+ redis_client.set(HEARTBEAT, json.dumps({"started_at": started_at, "last_seen": now_iso()}), ex=HEARTBEAT_SECONDS)+++def status(redis_client) -> dict:+ """For Admin: is a worker running, how much is waiting, and the latest results."""+ heartbeat = redis_client.get(HEARTBEAT)+ recent = [json.loads(raw) for raw in redis_client.lrange(RECENT, 0, 9)]+ return {+ "online": heartbeat is not None, "worker": json.loads(heartbeat) if heartbeat else None,+ "queued": redis_client.llen(QUEUE), "processing": redis_client.llen(PROCESSING), "retrying": redis_client.zcard(DELAYED),+ "recent": recent,+ }diff --git a/src/backend/alerts.py b/src/backend/alerts.pyindex f651a94..7609f24 100644--- a/src/backend/alerts.py+++ b/src/backend/alerts.py@@ -9,6 +9,7 @@ from __future__ import annotationsfrom collections.abc import Iterablefrom datetime import datetime, timezone+from . import agent_jobsfrom .models import IncidentAlertINFO, WARNING, CRITICAL = "INFO", "WARNING", "CRITICAL"@@ -32,9 +33,14 @@ def raise_alert(db, server_id: int, rule_key: str, severity: str, title: str, deraise ValueError(f"Unknown severity: {severity}")existing = _active(db, server_id).filter(IncidentAlert.rule_key == rule_key).first()if existing:+ # A more severe state is worth a new look by the agent; the same or a milder one is not.+ if agent_jobs.SEVERITY_RANK.get(severity, 0) > agent_jobs.SEVERITY_RANK.get(existing.severity, 0):+ agent_jobs.note(db, agent_jobs.WORSENED, existing, severity)existing.severity, existing.title, existing.description = severity, title[:255], descriptionreturn False- db.add(IncidentAlert(server_id=server_id, rule_key=rule_key, severity=severity, title=title[:255], description=description))+ alert = IncidentAlert(server_id=server_id, rule_key=rule_key, severity=severity, title=title[:255], description=description)+ db.add(alert)+ agent_jobs.note(db, agent_jobs.OPENED, alert, severity)return Truediff --git a/src/backend/main.py b/src/backend/main.pyindex 371614c..ed492a9 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -27,7 +27,7 @@ from sqlalchemy import case, func, inspect, textfrom .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alertsimport contextvars-from . import agent_usage, auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totp+from . import agent_jobs, agent_usage, auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totpfrom .auth import (STAGE_ENROLLMENT, STAGE_FULL, count_pending_attempt, create_pending_two_factor, create_session, delete_pending_two_factor, delete_session, get_recent_logs,get_session, get_user_from_session, hash_password, issue_recovery_codes, list_user_sessions, pending_two_factor_remember, read_pending_two_factor, session_lifetime, recovery_codes_left, record_audit_log,@@ -231,6 +231,8 @@ def agent_error_message(exc: Exception) -> str:redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)+# Jobs for the agent worker are published after the commit that opened or escalated an incident (see agent_jobs).+agent_jobs.redis_getter = lambda: redis_client# The interactive API documentation is a map of every endpoint, so it is off unless ENABLE_API_DOCS=true.app = FastAPI(title="Infradeck API", docs_url="/docs" if ENABLE_API_DOCS else None, redoc_url=None, openapi_url="/openapi.json" if ENABLE_API_DOCS else None)@@ -3235,6 +3237,16 @@ def admin_agent_usage(session_id: Annotated[str | None, Cookie(alias=SESSION_COOreturn {**agent_usage.month_summary(db), "model": AGENT_MODEL}[email protected]("/admin/agent/worker")+def admin_agent_worker(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+ """Whether the agent worker is running, how many jobs wait, and the latest results."""+ _require_admin(session_id, db)+ try:+ return agent_jobs.status(redis_client)+ except Exception as exc:+ raise HTTPException(status_code=503, detail="Redis is not answering, so the agent's queue can't be read.") from exc++@app.post("/admin/agent/test")def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):"""Admin's "Test connection": the API key works and the configured model exists. Spends no tokens."""diff --git a/src/frontend/src/admin/AdminPage.jsx b/src/frontend/src/admin/AdminPage.jsxindex a15f2ec..82e7a24 100644--- a/src/frontend/src/admin/AdminPage.jsx+++ b/src/frontend/src/admin/AdminPage.jsx@@ -7,6 +7,7 @@ import { AnotherMachineCard, ThisMachineCard } from '../dashboard/EmptyFleet.jsximport { INTERVALS, intervalLabel } from '../dashboard/ServerPanel.jsx'import { serverState } from '../dashboard/logic.js'import { Sparkline } from '../dashboard/charts.jsx'+import { usePolled } from '../dashboard/data.js'import { Icon } from '../icons.jsx'import { refreshOverview, useOverview } from '../overviewStore.js'import { AppShell, serverLink } from '../shell.jsx'@@ -291,10 +292,39 @@ function Agent({ settings, system, onSaved }) {<Row title="Redact IPs and hostnames before sending" detail="Not available yet. Until then, turn Chat off if names must not leave this machine."><Switch label="Redact before sending" checked={false} disabled onChange={() => {}} /></Row></div><Outcome state={state} />+ <AgentWorker /><AgentSpending settings={settings} onSaved={onSaved} /></>}+const JOB_TONE = { failed: 'chip-warn', noted: '', skipped: '', done: 'chip-ok' }++// The worker that runs the agent's jobs: is it alive, what is waiting, and what it did last.+function AgentWorker() {+ const { data, error } = usePolled('/admin/agent/worker', 15)+ if (!data) return <Notice tone="plain">{error || 'Checking the agent worker…'}</Notice>+ const waiting = [data.queued ? `${data.queued} queued` : null, data.processing ? `${data.processing} running` : null, data.retrying ? `${data.retrying} waiting to retry` : null].filter(Boolean).join(' · ') || 'nothing waiting'+ return (+ <div className="card overflow-hidden">+ <Row icon="Q" title="Worker" tone={data.online ? undefined : 'warn'}+ detail={data.online ? `Running since ${ago(data.worker.started_at).replace(' ago', '')} · ${waiting}` : `Not running, so new incidents wait for their diagnosis · ${waiting}. Start it with docker compose up -d agent-worker.`}>+ <span className={`chip ${data.online ? 'chip-ok' : 'chip-warn'}`}>{data.online ? 'Online' : 'Offline'}</span>+ </Row>+ {data.recent.length > 0 && <ul className="m-0 list-none divide-y divide-line border-t border-line p-0">+ {data.recent.slice(0, 5).map((job) => (+ <li key={job.id} className="flex flex-wrap items-baseline gap-x-3 gap-y-1 px-[18px] py-2.5 text-xs">+ <span className="w-16 shrink-0 text-muted">{ago(job.finished_at)}</span>+ <span className="mono shrink-0 text-soft">INC-{String(job.incident_id).padStart(4, '0')}</span>+ <span className="shrink-0 text-muted">{job.type === 'incident.worsened' ? 'got worse' : 'opened'}</span>+ <span className={`chip ${JOB_TONE[job.status] ?? ''}`}>{job.status}</span>+ <span className="min-w-0 flex-1 basis-60 text-muted">{job.note}</span>+ </li>+ ))}+ </ul>}+ </div>+ )+}+// "$0.0031" for small amounts, "$1.84" from a cent up.const dollars = (value) => (value == null ? '—' : `$${value > 0 && value < 0.01 ? value.toFixed(4) : value.toFixed(2)}`)const thousands = (value) => (value >= 1e6 ? `${(value / 1e6).toFixed(1)} M` : value >= 1e4 ? `${Math.round(value / 1e3)} k` : value.toLocaleString())diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.pyindex ae1a603..b8da65a 100644--- a/tests/accounts/test_new_pages.py+++ b/tests/accounts/test_new_pages.py@@ -465,3 +465,9 @@ def test_a_failed_call_is_recorded_without_tokens(app, monkeypatch):])def test_spending_settings_are_checked(app, body):assert signed_in(app, 1).put("/admin/settings", json=body).status_code == 400+++def test_admins_see_the_agent_workers_status(app):+ status = signed_in(app, 1).get("/admin/agent/worker")+ assert status.status_code == 200 and status.json()["online"] is False and status.json()["queued"] >= 0+ assert signed_in(app, 3).get("/admin/agent/worker").status_code == 403diff --git a/tests/assistant/test_agent_worker.py b/tests/assistant/test_agent_worker.pynew file mode 100644index 0000000..ecbaa62--- /dev/null+++ b/tests/assistant/test_agent_worker.py@@ -0,0 +1,162 @@+"""The agent's job queue and worker (#110): one job per opened or worsened incident, published only after the+commit, and a worker that retries, gives up, survives restarts and never waits forever on one job."""+import json+import time++import pytest+from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker+from sqlalchemy.pool import StaticPool++from src.agent import worker+from src.backend import agent_jobs, alerts+from src.backend.models import Base, IncidentAlert, Server+from tests.fakes import FakeRedis+++def redis(monkeypatch):+ client = FakeRedis()+ monkeypatch.setattr(agent_jobs, "redis_getter", lambda: client)+ return client+++def db():+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(Server(id=1, owner_id=1, hostname="alpha", check_interval_seconds=60))+ session.commit()+ session.bind_engine = engine+ yield session+ session.close()+++def queued(redis):+ return [json.loads(raw) for raw in reversed(redis.lrange(agent_jobs.QUEUE, 0, -1))]+++def test_opening_an_incident_queues_exactly_one_job_after_the_commit(db, redis):+ alerts.raise_alert(db, 1, "host:memory", "WARNING", "Memory", "high")+ assert queued(redis) == [] # nothing before the commit+ db.commit()+ alerts.raise_alert(db, 1, "host:memory", "WARNING", "Memory", "still high") # the same condition, refreshed+ db.commit()++ jobs = queued(redis)+ incident = db.query(IncidentAlert).one()+ assert [(job["type"], job["incident_id"], job["severity"]) for job in jobs] == [("incident.opened", incident.id, "WARNING")]+++def test_getting_worse_queues_one_more_job_and_getting_better_does_not(db, redis):+ alerts.raise_alert(db, 1, "host:memory", "WARNING", "Memory", "high")+ db.commit()+ for severity in ("CRITICAL", "CRITICAL", "WARNING", "CRITICAL"):+ alerts.raise_alert(db, 1, "host:memory", severity, "Memory", "x")+ db.commit()++ assert [job["type"] for job in queued(redis)] == ["incident.opened", "incident.worsened"]+++def test_info_incidents_and_rolled_back_ones_queue_nothing(db, redis):+ alerts.raise_alert(db, 1, "container:web:image", "INFO", "Image", "changed")+ db.commit()+ alerts.raise_alert(db, 1, "host:disk:/", "CRITICAL", "Disk", "full")+ db.flush()+ db.rollback()+ db.commit()++ assert queued(redis) == []+++def test_the_incident_is_saved_even_when_redis_is_down(db, monkeypatch):+ def broken():+ raise ConnectionError("redis is down")+ monkeypatch.setattr(agent_jobs, "redis_getter", broken)++ alerts.raise_alert(db, 1, "host:disk:/", "CRITICAL", "Disk", "full")+ db.commit()++ assert db.query(IncidentAlert).count() == 1+++def _take(redis):+ return agent_jobs.take(redis)+++def test_a_finished_job_leaves_the_processing_list_and_shows_in_recent(redis):+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")+ raw = _take(redis)++ assert worker.process(redis, raw, {agent_jobs.OPENED: lambda job: ("noted", f"incident {job['incident_id']}")}) == "noted"++ assert redis.llen(agent_jobs.PROCESSING) == 0+ recent = agent_jobs.status(redis)["recent"]+ assert (recent[0]["status"], recent[0]["note"], recent[0]["incident_id"]) == ("noted", "incident 7", 7)+++def test_a_failing_job_is_retried_later_and_given_up_after_three_attempts(redis):+ def broken(job):+ raise RuntimeError("the model did not answer")+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")++ outcomes = []+ for _ in range(agent_jobs.MAX_ATTEMPTS):+ agent_jobs.promote_due(redis, now=time.time() + 3600) # pretend the wait is over+ outcomes.append(worker.process(redis, _take(redis), {agent_jobs.OPENED: broken}))++ assert outcomes == ["retry", "retry", "failed"]+ assert redis.llen(agent_jobs.QUEUE) == 0 and redis.zcard(agent_jobs.DELAYED) == 0 and redis.llen(agent_jobs.PROCESSING) == 0+ assert "did not answer" in agent_jobs.status(redis)["recent"][0]["note"]+++def test_a_retry_waits_before_it_runs_again(redis):+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")+ worker.process(redis, _take(redis), {agent_jobs.OPENED: lambda job: 1 / 0})++ assert agent_jobs.promote_due(redis, now=time.time()) == 0 # 30 s haven't passed+ assert agent_jobs.promote_due(redis, now=time.time() + 31) == 1+++def test_a_job_that_hangs_is_abandoned_after_its_time_limit(redis):+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")++ assert worker.process(redis, _take(redis), {agent_jobs.OPENED: lambda job: time.sleep(5)}, timeout=0.05) == "retry"+++def test_jobs_interrupted_by_a_restart_go_back_to_the_queue(redis):+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")+ _take(redis) # taken, then the worker died++ assert agent_jobs.recover(redis) == 1+ assert redis.llen(agent_jobs.QUEUE) == 1 and redis.llen(agent_jobs.PROCESSING) == 0+++def test_garbage_in_the_queue_is_dropped_not_retried(redis):+ redis.lpush(agent_jobs.QUEUE, "not json")++ assert worker.process(redis, _take(redis)) == "failed"+ assert redis.zcard(agent_jobs.DELAYED) == 0+++def test_a_job_checks_that_its_incident_still_needs_a_diagnosis(db, redis, monkeypatch):+ monkeypatch.setattr(agent_jobs, "redis_getter", lambda: redis)+ monkeypatch.setattr(worker, "SessionLocal", sessionmaker(bind=db.bind_engine))+ alerts.raise_alert(db, 1, "host:disk:/", "CRITICAL", "Disk", "full")+ db.commit()+ incident = db.query(IncidentAlert).one()++ assert worker.handle_incident({"incident_id": incident.id})[0] == "noted"+ incident.status = "resolved"+ db.commit()+ assert worker.handle_incident({"incident_id": incident.id}) == ("skipped", "The incident was already resolved.")+ assert worker.handle_incident({"incident_id": 999})[0] == "skipped"+++def test_status_says_whether_a_worker_is_alive(redis):+ assert agent_jobs.status(redis)["online"] is False++ agent_jobs.beat(redis, agent_jobs.now_iso())++ assert agent_jobs.status(redis)["online"] is Truediff --git a/tests/fakes.py b/tests/fakes.pyindex 587ccc5..a74492e 100644--- a/tests/fakes.py+++ b/tests/fakes.py@@ -80,6 +80,66 @@ class FakeRedis:entry = self.hashes.get(key, {})return sum(int(entry.pop(field, None) is not None) for field in fields)+ # lists and sorted sets, for the agent's job queue (left is the head, as in Redis)+ def _list(self, key):+ return self.store.setdefault(key, [])++ def lpush(self, key, *values):+ for value in values:+ self._list(key).insert(0, value)+ return len(self.store[key])++ def rpush(self, key, *values):+ self._list(key).extend(values)+ return len(self.store[key])++ def llen(self, key):+ return len(self.store.get(key) or [])++ def lrange(self, key, start, end):+ items = self.store.get(key) or []+ return items[start:] if end == -1 else items[start:end + 1]++ def ltrim(self, key, start, end):+ self.store[key] = self.lrange(key, start, end)+ return True++ def lrem(self, key, count, value):+ items = self.store.get(key) or []+ if value in items:+ items.remove(value)+ return 1+ return 0++ def lmove(self, source, destination, src="LEFT", dest="RIGHT"):+ items = self.store.get(source) or []+ if not items:+ return None+ value = items.pop(0 if src == "LEFT" else -1)+ target = self._list(destination)+ target.insert(0, value) if dest == "LEFT" else target.append(value)+ return value++ def blmove(self, source, destination, timeout, src="LEFT", dest="RIGHT"):+ return self.lmove(source, destination, src, dest)++ def zadd(self, key, mapping):+ self.hashes.setdefault(key, {}).update(mapping)+ return len(mapping)++ def zrangebyscore(self, key, low, high):+ entries = self.hashes.get(key, {})+ low = float("-inf") if low == "-inf" else float(low)+ high = float("inf") if high == "+inf" else float(high)+ return [member for member, score in sorted(entries.items(), key=lambda item: item[1]) if low <= score <= high]++ def zrem(self, key, *members):+ entries = self.hashes.get(key, {})+ return sum(int(entries.pop(member, None) is not None) for member in members)++ def zcard(self, key):+ return len(self.hashes.get(key, {}))+# the rate limiter pipelines an increment with an expirydef pipeline(self):parent = self