Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

feat(agent): worker container and job queue for incident diagnoses

karloks200559efff9724c1
diff --git a/SECURITY.md b/SECURITY.md
index cdca572..236705e 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -10,6 +10,7 @@ This page says what each component can reach, so you can decide what to run.
| Collector | container | host `/proc`, the host filesystem (read-only), the Docker proxy, the backend over HTTP | No Docker socket in the compose installs. |
| Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows four read-only requests. See below. |
| System helper (optional) | container, uid 0 with **one** capability | kernel log, `/var/log/wtmp` (read-only), one output volume | Off by default. Details below. |
+| Agent worker | container, no capabilities, read-only filesystem | PostgreSQL, Redis, the model provider on the internet | Runs the agent's jobs from a queue in Redis; no published port. |
| Collector gate (server stack) | nginx container, no capabilities | the backend; publishes one port on your VPN address | The only way to reach collector endpoints. See below. |
| Backend / frontend / database / Redis | containers | each other, over separate networks | Only the frontend port is published. Redis and PostgreSQL are on a network with no route out and are reachable only by the backend. Redis needs a password. |
diff --git a/TESTS.md b/TESTS.md
index 8a840f3..61930b6 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -148,6 +148,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
| [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |
| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |
| [`test_agent_service.py`](#test_agent_servicepy) | 9 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. |
+| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |
| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |
| [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. |
| [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |
@@ -317,6 +318,23 @@ shouldn't, or an admin might be locked out.
| `test_only_recent_turns_go_back_to_the_model` | Long conversations send only the most recent turns. |
| `test_a_failed_turn_is_not_kept_in_the_conversation` | A message that failed (timeout, provider error) is not remembered as part of the conversation. |
+### test_agent_worker.py
+
+| Test | What it checks |
+|---|---|
+| `test_opening_an_incident_queues_exactly_one_job_after_the_commit` | A new WARNING or CRITICAL incident queues one job, only once the database commit is done, and refreshing it queues no more. |
+| `test_getting_worse_queues_one_more_job_and_getting_better_does_not` | Going from warning to critical queues one `incident.worsened` job; repeating it or going back down queues nothing. |
+| `test_info_incidents_and_rolled_back_ones_queue_nothing` | INFO notes and incidents whose transaction was rolled back never reach the queue. |
+| `test_the_incident_is_saved_even_when_redis_is_down` | A Redis failure loses the job, not the incident. |
+| `test_a_finished_job_leaves_the_processing_list_and_shows_in_recent` | A finished job leaves the processing list and shows in the recent results. |
+| `test_a_failing_job_is_retried_later_and_given_up_after_three_attempts` | A failing job is tried three times in all and then marked failed with its error. |
+| `test_a_retry_waits_before_it_runs_again` | A retry waits 30 seconds before it can run. |
+| `test_a_job_that_hangs_is_abandoned_after_its_time_limit` | A job past its time limit is abandoned and retried. |
+| `test_jobs_interrupted_by_a_restart_go_back_to_the_queue` | Jobs a worker took but never finished go back to the queue when it starts again. |
+| `test_garbage_in_the_queue_is_dropped_not_retried` | Something in the queue that isn't a job is dropped, not retried forever. |
+| `test_a_job_checks_that_its_incident_still_needs_a_diagnosis` | A job skips incidents that are gone or resolved. |
+| `test_status_says_whether_a_worker_is_alive` | The worker's heartbeat is what Admin shows as online. |
+
### test_alerts.py
The incident engine and the container rules.
diff --git a/compose.yml b/compose.yml
index c454399..13469aa 100644
--- a/compose.yml
+++ b/compose.yml
@@ -99,6 +99,39 @@ services:
timeout: 5s
retries: 5
+ # Runs the incident agent's jobs (diagnoses) from the queue in Redis, apart from the backend, so a slow or
+ # stuck model never holds up the API. Stopping it only delays diagnoses; everything else keeps working.
+ # It needs the database, Redis and the internet (the model provider), and nothing else. See docs/containers.md.
+ agent-worker:
+ build:
+ context: .
+ dockerfile: docker/backend.Dockerfile
+ image: infradeck-backend:latest
+ container_name: infradeck-agent-worker
+ restart: unless-stopped
+ command: ["python", "-m", "src.agent.worker"]
+ networks: [data, front]
+ environment:
+ DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
+ REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
+ PYTHONDONTWRITEBYTECODE: "1"
+ cap_drop: [ALL]
+ read_only: true
+ tmpfs:
+ - /tmp
+ security_opt:
+ - no-new-privileges:true
+ healthcheck:
+ test: ["CMD", "python", "-m", "src.agent.worker", "--health"]
+ interval: 30s
+ timeout: 5s
+ retries: 3
+ depends_on:
+ backend:
+ condition: service_healthy
+
# Monitors the machine running this stack. Other servers use collector-compose.yml.
collector:
build:
diff --git a/docker-compose.server.yml b/docker-compose.server.yml
index 3995d54..153c347 100644
--- a/docker-compose.server.yml
+++ b/docker-compose.server.yml
@@ -105,6 +105,39 @@ services:
timeout: 5s
retries: 5
+ # Runs the incident agent's jobs (diagnoses) from the queue in Redis, apart from the backend, so a slow or
+ # stuck model never holds up the API. Stopping it only delays diagnoses; everything else keeps working.
+ # It needs the database, Redis and the internet (the model provider), and nothing else. See docs/containers.md.
+ agent-worker:
+ build:
+ context: .
+ dockerfile: docker/backend.Dockerfile
+ image: infradeck-backend:latest
+ container_name: infradeck-agent-worker
+ restart: unless-stopped
+ command: ["python", "-m", "src.agent.worker"]
+ networks: [data, front]
+ environment:
+ DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
+ REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
+ PYTHONDONTWRITEBYTECODE: "1"
+ cap_drop: [ALL]
+ read_only: true
+ tmpfs:
+ - /tmp
+ security_opt:
+ - no-new-privileges:true
+ healthcheck:
+ test: ["CMD", "python", "-m", "src.agent.worker", "--health"]
+ interval: 30s
+ timeout: 5s
+ retries: 3
+ depends_on:
+ backend:
+ condition: service_healthy
+
# Monitors the machine running this stack. Other servers use collector-compose.yml.
collector:
build:
diff --git a/docs/api.md b/docs/api.md
index 9b5e232..169e579 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -30,6 +30,7 @@ All paths are relative to `/api` when going through the frontend/nginx.
| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds; the agent's model prices, monthly budget and daily chat limit. |
| `GET /agent/config` | session | Whether chat with the agent can be used: switched on, a key configured, and the monthly budget not used up. |
| `GET /admin/agent/usage` | admin | This month's model calls, tokens and estimated cost against the budget, per model and per day, with the prices and limits. |
+| `GET /admin/agent/worker` | admin | Whether the agent worker is running, how many jobs wait, are running or wait to be retried, and the latest results. |
| `POST /admin/agent/test` | admin | Checks the OpenAI key and that the configured model exists, without spending tokens. |
| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |
| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |
diff --git a/docs/containers.md b/docs/containers.md
index 36961e4..ac72668 100644
--- a/docs/containers.md
+++ b/docs/containers.md
@@ -16,7 +16,8 @@ For the settings, see [Configuration](configuration.md).
| `infradeck-frontend` | built from `docker/frontend.Dockerfile` (nginx) | dashboard machine | yes | Serves the web app and passes `/api` to the backend. |
| `infradeck-backend` | built from `docker/backend.Dockerfile` (Python, FastAPI) | dashboard machine | yes | The API: sign-in, metrics intake, alerts, statistics, notifications, the assistant. |
| `infradeck-postgres` | `postgres:16-alpine` | dashboard machine | yes | Everything that must be kept: accounts, servers, metric history, incidents, settings. |
-| `infradeck-redis` | `redis:7-alpine` | dashboard machine | yes | Short-lived state: sign-in sessions, rate-limit counters, pending two-factor logins, a small cache. |
+| `infradeck-redis` | `redis:7-alpine` | dashboard machine | yes | Short-lived state: sign-in sessions, rate-limit counters, pending two-factor logins, a small cache, the agent's job queue. |
+| `infradeck-agent-worker` | the backend image | dashboard machine | for the incident agent | Runs the agent's jobs (diagnoses) from the queue in Redis, apart from the API. |
| `infradeck-collector` | built from `docker/collector.Dockerfile` | every monitored machine | yes, one per machine | Reads the machine's CPU, memory, disks, network and containers and sends them to the backend. |
| `infradeck-docker-proxy` | built from `docker/docker-proxy.Dockerfile` | every monitored machine | yes, next to each collector | The only container with the Docker socket; answers six read-only questions. |
| `infradeck-collector-gate` | `nginx:alpine` | dashboard machine, server stack only | yes in `docker-compose.server.yml` | The only door to the collector endpoints, published on your VPN address. |
@@ -170,6 +171,7 @@ fails until it is back. Nothing already stored is lost: it is all in the volume.
| `2fa-pending:…` and its attempt counter | A password check that still waits for the authenticator code. | 5 minutes |
| `rate-limit:…` | Attempt counters for sign-in, registration, two-factor, password change, collector registration, invite checks, the assistant chat, notification tests and more. | the length of each window (e.g. 60 s) |
| `stats:…` | Cached fleet statistics, so the Statistics page doesn't recompute them on every load. | 60 seconds |
+| `agent:queue`, `agent:processing`, `agent:delayed`, `agent:recent`, `agent:worker` | The agent worker's jobs: waiting, being run, waiting to be retried, the last 50 results, and the worker's heartbeat. | until done; the heartbeat 30 seconds |
Session IDs and the values limited per address or email are stored only as SHA-256 digests, so a copy of
Redis doesn't hand out working sessions or a list of email addresses.
@@ -198,6 +200,32 @@ checked, so users appear signed out. `/health` shows `redis: down`. Collectors k
authenticate with their own credentials stored in PostgreSQL), so no monitoring data is lost. When Redis
comes back, sessions saved in its volume work again.
+## infradeck-agent-worker
+
+**What it does.** Runs the incident agent's jobs. When an incident opens, or gets more severe, the backend puts one
+job in a queue in Redis; the worker takes jobs one at a time and works on them. Automatic diagnosis (#116) is not
+built yet, so for now a job only checks that its incident still needs a diagnosis and notes that; the chat runs in
+the backend.
+
+**How it works.** It is the backend image started with `python -m src.agent.worker`. Jobs are published only after
+the database commit that opened the incident, and each event once. A job the worker takes moves to a processing
+list and leaves it only when it is done, so a job interrupted by a restart goes back to the queue. A failed job is
+tried again after 30 and 60 seconds and then given up; a job running longer than 2 minutes is abandoned. The
+worker writes a heartbeat every few seconds, which **Admin → Incident agent → Worker** and the container's
+healthcheck read. It runs with no capabilities, a read-only filesystem and no published port.
+
+**Why a separate container.** A diagnosis means calls to a language model that can take tens of seconds or hang.
+In the backend, that would tie up the threads that serve the dashboard and take check-ins. In its own container
+it can be slow, restarted or stopped without the rest noticing. Redis is already part of the stack, so the queue
+needed nothing new; a dedicated queue system (Celery, RabbitMQ) would add containers for one job type.
+
+**Networks.** `data` (PostgreSQL and Redis) and `front`, because `data` has no route out and the worker has to
+reach the model provider on the internet.
+
+**If it stops.** New incidents wait in the queue for their diagnosis and are worked on when it starts again (each
+waiting job is kept). Incidents, notifications, the chat and everything else keep working. Admin shows the
+worker as offline.
+
## infradeck-collector
**What it does.** Measures the machine it runs on and sends one check-in to the backend every 60 seconds
diff --git a/src/agent/worker.py b/src/agent/worker.py
new file mode 100644
index 0000000..4274585
--- /dev/null
+++ b/src/agent/worker.py
@@ -0,0 +1,117 @@
+"""The agent worker: takes jobs from the queue in Redis (see src/backend/agent_jobs.py) and runs them one at a time.
+
+It runs in its own container, so a slow or stuck model never holds up the backend, and stopping it only means
+incidents wait for their diagnosis; everything else keeps working. Run with ``python -m src.agent.worker``;
+``--health`` exits 0 while a worker is alive, for the container's healthcheck.
+
+Automatic diagnosis (#116) is not built yet, so a job only checks that its incident still needs one and notes
+that. The plumbing (queue, retries, time limit, status in Admin) is what this file provides.
+"""
+from __future__ import annotations
+
+import json
+import signal
+import sys
+import threading
+import time
+
+from src.backend import agent_jobs, agent_usage
+from src.backend.config import REDIS_URL
+from src.backend.db import SessionLocal
+from src.backend.models import IncidentAlert
+
+JOB_TIMEOUT_SECONDS = 120
+POLL_SECONDS = 5
+
+
+def handle_incident(job: dict) -> tuple[str, str]:
+ """``incident.opened`` and ``incident.worsened``. Returns (status, note) for the recent-jobs list."""
+ db = SessionLocal()
+ try:
+ incident = db.get(IncidentAlert, job["incident_id"])
+ if incident is None:
+ return "skipped", "The incident no longer exists."
+ if incident.status != "active":
+ return "skipped", "The incident was already resolved."
+ if agent_usage.budget(db)["state"] == "paused":
+ return "skipped", "This month's budget for the agent is used up."
+ return "noted", "Automatic diagnosis is not built yet (#116); the incident keeps its rule-based diagnosis."
+ finally:
+ db.close()
+
+
+HANDLERS = {agent_jobs.OPENED: handle_incident, agent_jobs.WORSENED: handle_incident}
+
+
+def run_with_timeout(handler, job: dict, seconds: float) -> tuple[str, str]:
+ """Run ``handler(job)`` in a thread and stop waiting after ``seconds``. Raises TimeoutError, or what the handler raised.
+
+ A thread can't be killed, so a handler past its time is abandoned (it is a daemon and ends with the process);
+ the model client's own timeout keeps that from piling up."""
+ outcome: dict = {}
+
+ def target():
+ try:
+ outcome["result"] = handler(job)
+ except BaseException as exc: # handed back to the caller below
+ outcome["error"] = exc
+
+ thread = threading.Thread(target=target, name=f"agent-job-{job.get('id', '')[:8]}", daemon=True)
+ thread.start()
+ thread.join(seconds)
+ if thread.is_alive():
+ raise TimeoutError(f"The job took longer than {seconds:g} s.")
+ if "error" in outcome:
+ raise outcome["error"]
+ return outcome["result"]
+
+
+def process(redis_client, raw: str, handlers=None, timeout: float = JOB_TIMEOUT_SECONDS) -> str:
+ """One job from the processing list: run it, then mark it finished, retry it later or give up. Returns the outcome."""
+ handlers = handlers or HANDLERS
+ started = time.monotonic()
+ try:
+ job = json.loads(raw)
+ handler = handlers[job["type"]]
+ except (ValueError, KeyError, TypeError):
+ agent_jobs.finish(redis_client, raw, {"type": "unknown"}, "failed", "Not a job this worker knows.")
+ return "failed"
+ try:
+ status_text, note_text = run_with_timeout(handler, job, timeout)
+ except Exception as exc:
+ return agent_jobs.retry_or_fail(redis_client, raw, job, f"{type(exc).__name__}: {exc}", round((time.monotonic() - started) * 1000))
+ agent_jobs.finish(redis_client, raw, job, status_text, note_text, round((time.monotonic() - started) * 1000))
+ return status_text
+
+
+def main() -> int:
+ import redis
+
+ # Waiting for a job blocks for POLL_SECONDS, so reads must be allowed to take longer than that.
+ client = redis.Redis.from_url(REDIS_URL, decode_responses=True, socket_timeout=POLL_SECONDS + 10, socket_connect_timeout=5)
+ if "--health" in sys.argv:
+ return 0 if client.get(agent_jobs.HEARTBEAT) else 1
+
+ stopping = threading.Event()
+ for name in (signal.SIGTERM, signal.SIGINT):
+ signal.signal(name, lambda *_: stopping.set())
+ started_at = agent_jobs.now_iso()
+ recovered = agent_jobs.recover(client)
+ print(f"Agent worker started{f'; {recovered} unfinished jobs back in the queue' if recovered else ''}.", flush=True)
+ while not stopping.is_set():
+ try:
+ agent_jobs.beat(client, started_at)
+ agent_jobs.promote_due(client)
+ raw = agent_jobs.take(client, POLL_SECONDS)
+ if raw is not None:
+ outcome = process(client, raw)
+ print(f"Agent job {outcome}: {raw[:200]}", flush=True)
+ except redis.RedisError as exc: # Redis restarting: wait and carry on
+ print(f"Agent worker can't reach Redis: {exc}", flush=True)
+ stopping.wait(POLL_SECONDS)
+ print("Agent worker stopped.", flush=True)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/src/backend/agent_jobs.py b/src/backend/agent_jobs.py
new file mode 100644
index 0000000..b7705f2
--- /dev/null
+++ b/src/backend/agent_jobs.py
@@ -0,0 +1,155 @@
+"""The agent's job queue in Redis: the backend publishes, the agent worker (src/agent/worker.py) consumes.
+
+When an incident opens, or gets more severe, the backend queues one job for the worker. Jobs are published only
+after the database commit that created or changed the incident, so the worker never looks for an incident that
+was rolled back, and each event is queued once (``incident.opened`` once per incident, ``incident.worsened`` once
+per incident and severity).
+
+The queue is reliable for one worker: a job the worker takes moves to a processing list and leaves it only when
+it is finished, so a job interrupted by a crash or a restart goes back to the queue when the worker starts again.
+A job that fails is tried again later, up to MAX_ATTEMPTS times. If Redis is down, the incident is still saved;
+only the job is lost, and that is logged.
+"""
+from __future__ import annotations
+
+import json
+import time
+import uuid
+from collections.abc import Callable
+from datetime import datetime, timezone
+
+from sqlalchemy import event
+from sqlalchemy.orm import Session
+
+QUEUE = "infradeck:agent:queue"
+PROCESSING = "infradeck:agent:processing"
+DELAYED = "infradeck:agent:delayed"
+RECENT = "infradeck:agent:recent"
+HEARTBEAT = "infradeck:agent:worker"
+ONCE = "infradeck:agent:once:"
+
+OPENED, WORSENED = "incident.opened", "incident.worsened"
+JOB_TYPES = (OPENED, WORSENED)
+# INFO incidents (an image changed, a container went away) are notes, not problems to diagnose.
+DIAGNOSED_SEVERITIES = ("WARNING", "CRITICAL")
+SEVERITY_RANK = {"INFO": 0, "WARNING": 1, "CRITICAL": 2}
+MAX_ATTEMPTS = 3
+RETRY_BASE_SECONDS = 30
+ONCE_SECONDS = 30 * 86400
+RECENT_KEPT = 50
+HEARTBEAT_SECONDS = 30
+
+# Set by the backend (and the worker) to how they reach Redis; read when a job is published.
+redis_getter: Callable[[], object] | None = None
+
+
+def now_iso() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def new_job(job_type: str, incident_id: int, severity: str | None = None) -> dict:
+ return {"id": uuid.uuid4().hex, "type": job_type, "incident_id": int(incident_id), "severity": severity, "attempts": 0, "enqueued_at": now_iso()}
+
+
+def publish(redis_client, job_type: str, incident_id: int, severity: str | None = None) -> bool:
+ """Queue one job, unless the same event was already queued. Returns True if it was queued."""
+ if job_type not in JOB_TYPES:
+ raise ValueError(f"Unknown agent job: {job_type}")
+ once = f"{ONCE}{job_type}:{incident_id}" + (f":{severity}" if job_type == WORSENED else "")
+ if not redis_client.set(once, "1", nx=True, ex=ONCE_SECONDS):
+ return False
+ redis_client.lpush(QUEUE, json.dumps(new_job(job_type, incident_id, severity)))
+ return True
+
+
+# ---------- the backend side: noting events in a database session and publishing them after commit ----------
+
+def note(db, job_type: str, incident, severity: str) -> None:
+ """Called where incidents are opened or escalated; the job is published once the session commits."""
+ if severity in DIAGNOSED_SEVERITIES:
+ db.info.setdefault("agent_pending", []).append((job_type, incident, severity))
+
+
[email protected]_for(Session, "after_flush")
+def _collect_ids(session, _context):
+ pending = session.info.pop("agent_pending", None)
+ if pending:
+ session.info.setdefault("agent_ready", []).extend((job_type, incident.id, severity) for job_type, incident, severity in pending if incident.id is not None)
+
+
[email protected]_for(Session, "after_commit")
+def _publish_after_commit(session):
+ ready = session.info.pop("agent_ready", None)
+ if not ready or redis_getter is None:
+ return
+ try:
+ client = redis_getter()
+ for job_type, incident_id, severity in ready:
+ publish(client, job_type, incident_id, severity)
+ except Exception as exc: # the incident is saved either way; only its diagnosis job is lost
+ print(f"Could not queue agent jobs: {type(exc).__name__}: {exc}", flush=True)
+
+
[email protected]_for(Session, "after_rollback")
+def _forget_after_rollback(session):
+ session.info.pop("agent_pending", None)
+ session.info.pop("agent_ready", None)
+
+
+# ---------- the worker side ----------
+
+def recover(redis_client) -> int:
+ """Put jobs a previous worker took but never finished back in the queue. Returns how many."""
+ moved = 0
+ while redis_client.lmove(PROCESSING, QUEUE, "LEFT", "RIGHT") is not None:
+ moved += 1
+ return moved
+
+
+def promote_due(redis_client, now: float | None = None) -> int:
+ """Move retries whose wait is over from the delayed set back to the queue."""
+ moved = 0
+ for raw in redis_client.zrangebyscore(DELAYED, "-inf", now if now is not None else time.time()):
+ if redis_client.zrem(DELAYED, raw):
+ redis_client.rpush(QUEUE, raw)
+ moved += 1
+ return moved
+
+
+def take(redis_client, timeout: int = 5) -> str | None:
+ """The oldest job, moved to the processing list; None when nothing arrived within ``timeout`` seconds."""
+ return redis_client.blmove(QUEUE, PROCESSING, timeout, "RIGHT", "LEFT")
+
+
+def finish(redis_client, raw: str, job: dict, status: str, note_text: str = "", duration_ms: int = 0) -> None:
+ redis_client.lrem(PROCESSING, 1, raw)
+ redis_client.lpush(RECENT, json.dumps({"id": job.get("id"), "type": job.get("type"), "incident_id": job.get("incident_id"), "status": status,
+ "note": note_text[:300], "attempts": job.get("attempts", 0) + 1, "duration_ms": duration_ms, "finished_at": now_iso()}))
+ redis_client.ltrim(RECENT, 0, RECENT_KEPT - 1)
+
+
+def retry_or_fail(redis_client, raw: str, job: dict, error: str, duration_ms: int = 0, now: float | None = None) -> str:
+ """After a failure: wait and try again (30 s, then 60 s), or give up after MAX_ATTEMPTS. Returns what happened."""
+ attempts = job.get("attempts", 0) + 1
+ if attempts >= MAX_ATTEMPTS:
+ finish(redis_client, raw, job, "failed", error, duration_ms)
+ return "failed"
+ redis_client.lrem(PROCESSING, 1, raw)
+ later = {**job, "attempts": attempts, "last_error": error[:300]}
+ redis_client.zadd(DELAYED, {json.dumps(later): (now if now is not None else time.time()) + RETRY_BASE_SECONDS * 2 ** (attempts - 1)})
+ return "retry"
+
+
+def beat(redis_client, started_at: str) -> None:
+ redis_client.set(HEARTBEAT, json.dumps({"started_at": started_at, "last_seen": now_iso()}), ex=HEARTBEAT_SECONDS)
+
+
+def status(redis_client) -> dict:
+ """For Admin: is a worker running, how much is waiting, and the latest results."""
+ heartbeat = redis_client.get(HEARTBEAT)
+ recent = [json.loads(raw) for raw in redis_client.lrange(RECENT, 0, 9)]
+ return {
+ "online": heartbeat is not None, "worker": json.loads(heartbeat) if heartbeat else None,
+ "queued": redis_client.llen(QUEUE), "processing": redis_client.llen(PROCESSING), "retrying": redis_client.zcard(DELAYED),
+ "recent": recent,
+ }
diff --git a/src/backend/alerts.py b/src/backend/alerts.py
index f651a94..7609f24 100644
--- a/src/backend/alerts.py
+++ b/src/backend/alerts.py
@@ -9,6 +9,7 @@ from __future__ import annotations
from collections.abc import Iterable
from datetime import datetime, timezone
+from . import agent_jobs
from .models import IncidentAlert
INFO, WARNING, CRITICAL = "INFO", "WARNING", "CRITICAL"
@@ -32,9 +33,14 @@ def raise_alert(db, server_id: int, rule_key: str, severity: str, title: str, de
raise ValueError(f"Unknown severity: {severity}")
existing = _active(db, server_id).filter(IncidentAlert.rule_key == rule_key).first()
if existing:
+ # A more severe state is worth a new look by the agent; the same or a milder one is not.
+ if agent_jobs.SEVERITY_RANK.get(severity, 0) > agent_jobs.SEVERITY_RANK.get(existing.severity, 0):
+ agent_jobs.note(db, agent_jobs.WORSENED, existing, severity)
existing.severity, existing.title, existing.description = severity, title[:255], description
return False
- db.add(IncidentAlert(server_id=server_id, rule_key=rule_key, severity=severity, title=title[:255], description=description))
+ alert = IncidentAlert(server_id=server_id, rule_key=rule_key, severity=severity, title=title[:255], description=description)
+ db.add(alert)
+ agent_jobs.note(db, agent_jobs.OPENED, alert, severity)
return True
diff --git a/src/backend/main.py b/src/backend/main.py
index 371614c..ed492a9 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -27,7 +27,7 @@ from sqlalchemy import case, func, inspect, text
from .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alerts
import contextvars
-from . import agent_usage, auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totp
+from . import agent_jobs, agent_usage, auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totp
from .auth import (
STAGE_ENROLLMENT, STAGE_FULL, count_pending_attempt, create_pending_two_factor, create_session, delete_pending_two_factor, delete_session, get_recent_logs,
get_session, get_user_from_session, hash_password, issue_recovery_codes, list_user_sessions, pending_two_factor_remember, read_pending_two_factor, session_lifetime, recovery_codes_left, record_audit_log,
@@ -231,6 +231,8 @@ def agent_error_message(exc: Exception) -> str:
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
+# Jobs for the agent worker are published after the commit that opened or escalated an incident (see agent_jobs).
+agent_jobs.redis_getter = lambda: redis_client
# The interactive API documentation is a map of every endpoint, so it is off unless ENABLE_API_DOCS=true.
app = FastAPI(title="Infradeck API", docs_url="/docs" if ENABLE_API_DOCS else None, redoc_url=None, openapi_url="/openapi.json" if ENABLE_API_DOCS else None)
@@ -3235,6 +3237,16 @@ def admin_agent_usage(session_id: Annotated[str | None, Cookie(alias=SESSION_COO
return {**agent_usage.month_summary(db), "model": AGENT_MODEL}
[email protected]("/admin/agent/worker")
+def admin_agent_worker(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
+ """Whether the agent worker is running, how many jobs wait, and the latest results."""
+ _require_admin(session_id, db)
+ try:
+ return agent_jobs.status(redis_client)
+ except Exception as exc:
+ raise HTTPException(status_code=503, detail="Redis is not answering, so the agent's queue can't be read.") from exc
+
+
@app.post("/admin/agent/test")
def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
"""Admin's "Test connection": the API key works and the configured model exists. Spends no tokens."""
diff --git a/src/frontend/src/admin/AdminPage.jsx b/src/frontend/src/admin/AdminPage.jsx
index a15f2ec..82e7a24 100644
--- a/src/frontend/src/admin/AdminPage.jsx
+++ b/src/frontend/src/admin/AdminPage.jsx
@@ -7,6 +7,7 @@ import { AnotherMachineCard, ThisMachineCard } from '../dashboard/EmptyFleet.jsx
import { INTERVALS, intervalLabel } from '../dashboard/ServerPanel.jsx'
import { serverState } from '../dashboard/logic.js'
import { Sparkline } from '../dashboard/charts.jsx'
+import { usePolled } from '../dashboard/data.js'
import { Icon } from '../icons.jsx'
import { refreshOverview, useOverview } from '../overviewStore.js'
import { AppShell, serverLink } from '../shell.jsx'
@@ -291,10 +292,39 @@ function Agent({ settings, system, onSaved }) {
<Row title="Redact IPs and hostnames before sending" detail="Not available yet. Until then, turn Chat off if names must not leave this machine."><Switch label="Redact before sending" checked={false} disabled onChange={() => {}} /></Row>
</div>
<Outcome state={state} />
+ <AgentWorker />
<AgentSpending settings={settings} onSaved={onSaved} />
</>
}
+const JOB_TONE = { failed: 'chip-warn', noted: '', skipped: '', done: 'chip-ok' }
+
+// The worker that runs the agent's jobs: is it alive, what is waiting, and what it did last.
+function AgentWorker() {
+ const { data, error } = usePolled('/admin/agent/worker', 15)
+ if (!data) return <Notice tone="plain">{error || 'Checking the agent worker…'}</Notice>
+ const waiting = [data.queued ? `${data.queued} queued` : null, data.processing ? `${data.processing} running` : null, data.retrying ? `${data.retrying} waiting to retry` : null].filter(Boolean).join(' · ') || 'nothing waiting'
+ return (
+ <div className="card overflow-hidden">
+ <Row icon="Q" title="Worker" tone={data.online ? undefined : 'warn'}
+ detail={data.online ? `Running since ${ago(data.worker.started_at).replace(' ago', '')} · ${waiting}` : `Not running, so new incidents wait for their diagnosis · ${waiting}. Start it with docker compose up -d agent-worker.`}>
+ <span className={`chip ${data.online ? 'chip-ok' : 'chip-warn'}`}>{data.online ? 'Online' : 'Offline'}</span>
+ </Row>
+ {data.recent.length > 0 && <ul className="m-0 list-none divide-y divide-line border-t border-line p-0">
+ {data.recent.slice(0, 5).map((job) => (
+ <li key={job.id} className="flex flex-wrap items-baseline gap-x-3 gap-y-1 px-[18px] py-2.5 text-xs">
+ <span className="w-16 shrink-0 text-muted">{ago(job.finished_at)}</span>
+ <span className="mono shrink-0 text-soft">INC-{String(job.incident_id).padStart(4, '0')}</span>
+ <span className="shrink-0 text-muted">{job.type === 'incident.worsened' ? 'got worse' : 'opened'}</span>
+ <span className={`chip ${JOB_TONE[job.status] ?? ''}`}>{job.status}</span>
+ <span className="min-w-0 flex-1 basis-60 text-muted">{job.note}</span>
+ </li>
+ ))}
+ </ul>}
+ </div>
+ )
+}
+
// "$0.0031" for small amounts, "$1.84" from a cent up.
const dollars = (value) => (value == null ? '—' : `$${value > 0 && value < 0.01 ? value.toFixed(4) : value.toFixed(2)}`)
const thousands = (value) => (value >= 1e6 ? `${(value / 1e6).toFixed(1)} M` : value >= 1e4 ? `${Math.round(value / 1e3)} k` : value.toLocaleString())
diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.py
index ae1a603..b8da65a 100644
--- a/tests/accounts/test_new_pages.py
+++ b/tests/accounts/test_new_pages.py
@@ -465,3 +465,9 @@ def test_a_failed_call_is_recorded_without_tokens(app, monkeypatch):
])
def test_spending_settings_are_checked(app, body):
assert signed_in(app, 1).put("/admin/settings", json=body).status_code == 400
+
+
+def test_admins_see_the_agent_workers_status(app):
+ status = signed_in(app, 1).get("/admin/agent/worker")
+ assert status.status_code == 200 and status.json()["online"] is False and status.json()["queued"] >= 0
+ assert signed_in(app, 3).get("/admin/agent/worker").status_code == 403
diff --git a/tests/assistant/test_agent_worker.py b/tests/assistant/test_agent_worker.py
new file mode 100644
index 0000000..ecbaa62
--- /dev/null
+++ b/tests/assistant/test_agent_worker.py
@@ -0,0 +1,162 @@
+"""The agent's job queue and worker (#110): one job per opened or worsened incident, published only after the
+commit, and a worker that retries, gives up, survives restarts and never waits forever on one job."""
+import json
+import time
+
+import pytest
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+from sqlalchemy.pool import StaticPool
+
+from src.agent import worker
+from src.backend import agent_jobs, alerts
+from src.backend.models import Base, IncidentAlert, Server
+from tests.fakes import FakeRedis
+
+
+def redis(monkeypatch):
+ client = FakeRedis()
+ monkeypatch.setattr(agent_jobs, "redis_getter", lambda: client)
+ return client
+
+
+def db():
+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
+ Base.metadata.create_all(engine)
+ session = sessionmaker(bind=engine)()
+ session.add(Server(id=1, owner_id=1, hostname="alpha", check_interval_seconds=60))
+ session.commit()
+ session.bind_engine = engine
+ yield session
+ session.close()
+
+
+def queued(redis):
+ return [json.loads(raw) for raw in reversed(redis.lrange(agent_jobs.QUEUE, 0, -1))]
+
+
+def test_opening_an_incident_queues_exactly_one_job_after_the_commit(db, redis):
+ alerts.raise_alert(db, 1, "host:memory", "WARNING", "Memory", "high")
+ assert queued(redis) == [] # nothing before the commit
+ db.commit()
+ alerts.raise_alert(db, 1, "host:memory", "WARNING", "Memory", "still high") # the same condition, refreshed
+ db.commit()
+
+ jobs = queued(redis)
+ incident = db.query(IncidentAlert).one()
+ assert [(job["type"], job["incident_id"], job["severity"]) for job in jobs] == [("incident.opened", incident.id, "WARNING")]
+
+
+def test_getting_worse_queues_one_more_job_and_getting_better_does_not(db, redis):
+ alerts.raise_alert(db, 1, "host:memory", "WARNING", "Memory", "high")
+ db.commit()
+ for severity in ("CRITICAL", "CRITICAL", "WARNING", "CRITICAL"):
+ alerts.raise_alert(db, 1, "host:memory", severity, "Memory", "x")
+ db.commit()
+
+ assert [job["type"] for job in queued(redis)] == ["incident.opened", "incident.worsened"]
+
+
+def test_info_incidents_and_rolled_back_ones_queue_nothing(db, redis):
+ alerts.raise_alert(db, 1, "container:web:image", "INFO", "Image", "changed")
+ db.commit()
+ alerts.raise_alert(db, 1, "host:disk:/", "CRITICAL", "Disk", "full")
+ db.flush()
+ db.rollback()
+ db.commit()
+
+ assert queued(redis) == []
+
+
+def test_the_incident_is_saved_even_when_redis_is_down(db, monkeypatch):
+ def broken():
+ raise ConnectionError("redis is down")
+ monkeypatch.setattr(agent_jobs, "redis_getter", broken)
+
+ alerts.raise_alert(db, 1, "host:disk:/", "CRITICAL", "Disk", "full")
+ db.commit()
+
+ assert db.query(IncidentAlert).count() == 1
+
+
+def _take(redis):
+ return agent_jobs.take(redis)
+
+
+def test_a_finished_job_leaves_the_processing_list_and_shows_in_recent(redis):
+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
+ raw = _take(redis)
+
+ assert worker.process(redis, raw, {agent_jobs.OPENED: lambda job: ("noted", f"incident {job['incident_id']}")}) == "noted"
+
+ assert redis.llen(agent_jobs.PROCESSING) == 0
+ recent = agent_jobs.status(redis)["recent"]
+ assert (recent[0]["status"], recent[0]["note"], recent[0]["incident_id"]) == ("noted", "incident 7", 7)
+
+
+def test_a_failing_job_is_retried_later_and_given_up_after_three_attempts(redis):
+ def broken(job):
+ raise RuntimeError("the model did not answer")
+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
+
+ outcomes = []
+ for _ in range(agent_jobs.MAX_ATTEMPTS):
+ agent_jobs.promote_due(redis, now=time.time() + 3600) # pretend the wait is over
+ outcomes.append(worker.process(redis, _take(redis), {agent_jobs.OPENED: broken}))
+
+ assert outcomes == ["retry", "retry", "failed"]
+ assert redis.llen(agent_jobs.QUEUE) == 0 and redis.zcard(agent_jobs.DELAYED) == 0 and redis.llen(agent_jobs.PROCESSING) == 0
+ assert "did not answer" in agent_jobs.status(redis)["recent"][0]["note"]
+
+
+def test_a_retry_waits_before_it_runs_again(redis):
+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
+ worker.process(redis, _take(redis), {agent_jobs.OPENED: lambda job: 1 / 0})
+
+ assert agent_jobs.promote_due(redis, now=time.time()) == 0 # 30 s haven't passed
+ assert agent_jobs.promote_due(redis, now=time.time() + 31) == 1
+
+
+def test_a_job_that_hangs_is_abandoned_after_its_time_limit(redis):
+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
+
+ assert worker.process(redis, _take(redis), {agent_jobs.OPENED: lambda job: time.sleep(5)}, timeout=0.05) == "retry"
+
+
+def test_jobs_interrupted_by_a_restart_go_back_to_the_queue(redis):
+ agent_jobs.publish(redis, agent_jobs.OPENED, 7, "CRITICAL")
+ _take(redis) # taken, then the worker died
+
+ assert agent_jobs.recover(redis) == 1
+ assert redis.llen(agent_jobs.QUEUE) == 1 and redis.llen(agent_jobs.PROCESSING) == 0
+
+
+def test_garbage_in_the_queue_is_dropped_not_retried(redis):
+ redis.lpush(agent_jobs.QUEUE, "not json")
+
+ assert worker.process(redis, _take(redis)) == "failed"
+ assert redis.zcard(agent_jobs.DELAYED) == 0
+
+
+def test_a_job_checks_that_its_incident_still_needs_a_diagnosis(db, redis, monkeypatch):
+ monkeypatch.setattr(agent_jobs, "redis_getter", lambda: redis)
+ monkeypatch.setattr(worker, "SessionLocal", sessionmaker(bind=db.bind_engine))
+ alerts.raise_alert(db, 1, "host:disk:/", "CRITICAL", "Disk", "full")
+ db.commit()
+ incident = db.query(IncidentAlert).one()
+
+ assert worker.handle_incident({"incident_id": incident.id})[0] == "noted"
+ incident.status = "resolved"
+ db.commit()
+ assert worker.handle_incident({"incident_id": incident.id}) == ("skipped", "The incident was already resolved.")
+ assert worker.handle_incident({"incident_id": 999})[0] == "skipped"
+
+
+def test_status_says_whether_a_worker_is_alive(redis):
+ assert agent_jobs.status(redis)["online"] is False
+
+ agent_jobs.beat(redis, agent_jobs.now_iso())
+
+ assert agent_jobs.status(redis)["online"] is True
diff --git a/tests/fakes.py b/tests/fakes.py
index 587ccc5..a74492e 100644
--- a/tests/fakes.py
+++ b/tests/fakes.py
@@ -80,6 +80,66 @@ class FakeRedis:
entry = self.hashes.get(key, {})
return sum(int(entry.pop(field, None) is not None) for field in fields)
+ # lists and sorted sets, for the agent's job queue (left is the head, as in Redis)
+ def _list(self, key):
+ return self.store.setdefault(key, [])
+
+ def lpush(self, key, *values):
+ for value in values:
+ self._list(key).insert(0, value)
+ return len(self.store[key])
+
+ def rpush(self, key, *values):
+ self._list(key).extend(values)
+ return len(self.store[key])
+
+ def llen(self, key):
+ return len(self.store.get(key) or [])
+
+ def lrange(self, key, start, end):
+ items = self.store.get(key) or []
+ return items[start:] if end == -1 else items[start:end + 1]
+
+ def ltrim(self, key, start, end):
+ self.store[key] = self.lrange(key, start, end)
+ return True
+
+ def lrem(self, key, count, value):
+ items = self.store.get(key) or []
+ if value in items:
+ items.remove(value)
+ return 1
+ return 0
+
+ def lmove(self, source, destination, src="LEFT", dest="RIGHT"):
+ items = self.store.get(source) or []
+ if not items:
+ return None
+ value = items.pop(0 if src == "LEFT" else -1)
+ target = self._list(destination)
+ target.insert(0, value) if dest == "LEFT" else target.append(value)
+ return value
+
+ def blmove(self, source, destination, timeout, src="LEFT", dest="RIGHT"):
+ return self.lmove(source, destination, src, dest)
+
+ def zadd(self, key, mapping):
+ self.hashes.setdefault(key, {}).update(mapping)
+ return len(mapping)
+
+ def zrangebyscore(self, key, low, high):
+ entries = self.hashes.get(key, {})
+ low = float("-inf") if low == "-inf" else float(low)
+ high = float("inf") if high == "+inf" else float(high)
+ return [member for member, score in sorted(entries.items(), key=lambda item: item[1]) if low <= score <= high]
+
+ def zrem(self, key, *members):
+ entries = self.hashes.get(key, {})
+ return sum(int(entries.pop(member, None) is not None) for member in members)
+
+ def zcard(self, key):
+ return len(self.hashes.get(key, {}))
+
# the rate limiter pipelines an increment with an expiry
def pipeline(self):
parent = self