Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

feat(agent): choose the provider, model and API key in Admin, add Anthropic and Gemini (#185)

Karlo Kajba Šimanićd3f9286cc45b
diff --git a/README.md b/README.md
index c6258fd..fc6dcee 100644
--- a/README.md
+++ b/README.md
@@ -114,7 +114,7 @@ Rules for disk (with per-mount overrides), inodes, memory, swap, load per core,
<img src="docs/images/infradeck-agent.svg" alt="The incident agent reads masked evidence, diagnoses the incident and watches that the fix holds" width="100%">
-Set `OPENAI_API_KEY` and the agent diagnoses every warning and critical incident from **masked** evidence, ties each claim to the data, proposes a fix, watches whether it holds and adds the diagnosis to your notifications. You can also chat about any incident. Monthly budget, per-model prices and who sees what are set in Admin. **It reads; it never runs commands.** See [Incident agent](docs/assistant.md).
+Add an Anthropic, OpenAI or Gemini API key in Admin and the agent diagnoses every warning and critical incident from **masked** evidence, ties each claim to the data, proposes a fix, watches whether it holds and adds the diagnosis to your notifications. You can also chat about any incident. Monthly budget, per-model prices and who sees what are set in Admin. **It reads; it never runs commands.** See [Incident agent](docs/assistant.md).
<img width="2879" height="1499" alt="Infradeck dashboard" src="https://github.com/user-attachments/assets/4331b563-9537-4393-882d-f0dbe5a1aedd" />
diff --git a/SECURITY.md b/SECURITY.md
index edcdd40..7b79d50 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -72,7 +72,7 @@ rule. An admin can also switch a rule off entirely in Admin › Alert rules, whi
and the facts Infradeck worked out (signals and recorded events, which contain container and host
names) to the model provider. Turn Chat off in Admin › Incident agent if that must not happen.
-**The agent's diagnosis** (when `OPENAI_API_KEY` is set) sends an evidence pack to the model provider: the
+**The agent's diagnosis** (when an API key is saved in Admin › Incident agent) sends an evidence pack to the model provider: the
incident, the server's name, OS, cores and memory, metrics around the incident, recorded events, image changes,
the containers' names, images and figures, and similar incidents before it. Before it leaves, every text in it is
masked for secrets (tokens, keys, passwords, credentials in URLs, private keys; `src/agent/masking.py`), and the
@@ -436,7 +436,7 @@ URL, a keyword or a value in a JSON reply, a TCP port, a TLS certificate, a DNS
a session for any user, administrator included, with no password or two-factor.
- **Limits on what a signed-in user can run up.** Messages to the assistant (30 an hour per user), new
conversations (30 an hour, 100 in total) and audit-log entries (120 an hour) are limited, since each can cost
- the administrator's OpenAI quota or database space. Setting up two-factor from a normal session needs the
+ the administrator's AI provider quota or database space. Setting up two-factor from a normal session needs the
password again, and every try counts toward the lockout, the right password included. A limited
(set-up-only) session lasts 30 minutes, not a day.
- **Sessions are stored hashed.** Redis holds only the SHA-256 of the cookie value, in the session key, the
diff --git a/TESTS.md b/TESTS.md
index 1b545b7..04f7276 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,
covers (and what it doesn't), and what every single test is for, so you can decide
whether a failure matters before you upgrade or change your setup.
-- **1438 test cases** from 776 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
+- **1458 test cases** from 788 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.
- **Safe to run next to a live install.** The tests use a throwaway in-memory database
and fakes; they never touch your running containers or your data. (Two collector
@@ -134,7 +134,7 @@ side and the backend's side in one file, because the feature only works when bot
| `tests/collectors/` | Adding and trusting collectors (enrollment, approval, pending servers, the dashboard's own machine), what they report and how it is validated, the filesystems they report, the read-only Docker proxy, and the version they report. |
| `tests/monitoring/` | What is watched and how it turns into incidents: the alert engine and its rules, system, container and network checks, the system helper and the container details. |
| `tests/history/` | Long-term history: rollups, the history API, statistics and the database indexes behind them. |
-| `tests/assistant/` | The AI assistant: its fixed health checks, error messages and conversations. |
+| `tests/assistant/` | The AI assistant: its providers, fixed health checks, error messages and conversations. |
| `tests/security/` | Tests written from security reviews that cut across areas: request size limits, the second review's findings and the nginx security headers. |
`tests/conftest.py` (settings every test shares) and `tests/fakes.py` (the fake Redis) stay at the top.
@@ -147,7 +147,8 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
| [`test_account_security.py`](#test_account_securitypy) | 146 | Two-factor sign-in, recovery codes, encrypted secrets, sessions you can review and end, password change, and the password check before turning two-factor on. |
| [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |
| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |
-| [`test_agent_service.py`](#test_agent_servicepy) | 9 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. |
+| [`test_agent_service.py`](#test_agent_servicepy) | 8 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. |
+| [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |
| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
| [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. |
| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |
@@ -311,8 +312,8 @@ shouldn't, or an admin might be locked out.
| `test_overloaded_provider_error_is_explained_to_the_user` | When the AI provider reports it is overloaded, the user gets a readable explanation. |
| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the chat or logs; it is replaced with `[redacted]`. |
| `test_openai_keys_are_redacted_too` | An OpenAI key (`sk-…`) in an error is redacted, and a rejected key is explained as such. |
-| `test_a_missing_key_says_where_to_set_it` | Without a key the chat says to set `OPENAI_API_KEY`. |
-| `test_an_unknown_model_points_at_agent_model` | A model the key can't use points the admin at `AGENT_MODEL`. |
+| `test_a_missing_key_says_where_to_set_it` | Without a key the chat says to add one in Admin › Incident agent. |
+| `test_an_unknown_model_points_at_the_model_setting` | A model the key can't use points the admin at the model in Admin › Incident agent. |
### test_agent_service.py
@@ -322,12 +323,25 @@ shouldn't, or an admin might be locked out.
| `test_arbitrary_command_is_rejected_before_execution` | If the assistant ever asks for anything outside the allow-list (such as `rm -rf /`), it is refused before anything runs. |
| `test_openai_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests use Structured Outputs in strict mode, the schema only allows the fixed checks, and token usage is read from the response. |
| `test_a_refusal_is_an_error_not_an_empty_reply` | When the model refuses, the chat gets an error instead of an empty answer. |
-| `test_the_provider_needs_a_key` | Without `OPENAI_API_KEY` the provider is not created. |
| `test_test_connection_spends_no_tokens` | Admin's Test connection only looks the model up; it sends no prompt. |
| `test_a_conversation_sends_the_system_prompt_and_its_history` | Each request carries the system prompt and the conversation so far, in order. |
| `test_only_recent_turns_go_back_to_the_model` | Long conversations send only the most recent turns. |
| `test_a_failed_turn_is_not_kept_in_the_conversation` | A message that failed (timeout, provider error) is not remembered as part of the conversation. |
+### test_agent_providers.py
+
+| Test | What it checks |
+|---|---|
+| `test_claude_gets_the_system_prompt_apart_and_must_answer_in_the_schema` | Claude gets the system prompt separately, the turns in order, the JSON schema, adaptive thinking with the requested effort and prompt caching; cached input counts as input. |
+| `test_a_refused_or_cut_off_claude_answer_is_an_error` *(2 inputs)* | A refusal or an answer cut off at the token limit is an error, not a half answer. |
+| `test_claude_test_connection_spends_no_tokens` | Test connection only looks the Claude model up; it sends no prompt. |
+| `test_without_a_choice_the_agent_has_no_provider` | Until an admin picks a provider there is no key and no provider. |
+| `test_each_provider_starts_with_its_own_default_model` *(3 inputs)* | Anthropic, OpenAI and Gemini each get the right client and their own default model. |
+| `test_gemini_goes_to_googles_address` | Gemini requests go to Google's OpenAI-compatible endpoint. |
+| `test_the_key_is_stored_encrypted_and_each_provider_keeps_its_own` | No key is stored in plain text, each provider keeps its own key, and saving without a key keeps the stored one. |
+| `test_a_key_that_cant_be_decrypted_counts_as_no_key` | After `SECRETS_KEY` changes, an unreadable key turns the agent off instead of failing. |
+| `test_a_removed_key_turns_the_agent_off` | Removing the key leaves the provider chosen but turns the agent off. |
+
### test_agent_diagnosis.py
| Test | What it checks |
diff --git a/compose.yml b/compose.yml
index e87f96b..17063b2 100644
--- a/compose.yml
+++ b/compose.yml
@@ -70,8 +70,6 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
- OPENAI_API_KEY: ${OPENAI_API_KEY:-}
- AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
# Set to true when the dashboard is served over HTTPS.
SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-false}
ALLOW_REGISTRATION: ${ALLOW_REGISTRATION:-false}
@@ -137,8 +135,8 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
- OPENAI_API_KEY: ${OPENAI_API_KEY:-}
- AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
+ # Decrypts the agent's API key, which Admin › Incident agent stores encrypted.
+ SECRETS_KEY: ${SECRETS_KEY:-}
PYTHONDONTWRITEBYTECODE: "1"
cap_drop: [ALL]
read_only: true
diff --git a/docker-compose.server.yml b/docker-compose.server.yml
index e27393c..6769b78 100644
--- a/docker-compose.server.yml
+++ b/docker-compose.server.yml
@@ -78,8 +78,6 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
- OPENAI_API_KEY: ${OPENAI_API_KEY:-}
- AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
# Served over HTTPS by your reverse proxy, so the login cookie is HTTPS-only.
SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-true}
# Only the first account can register; set to true to let others sign up.
@@ -146,8 +144,8 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
- OPENAI_API_KEY: ${OPENAI_API_KEY:-}
- AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
+ # Decrypts the agent's API key, which Admin › Incident agent stores encrypted.
+ SECRETS_KEY: ${SECRETS_KEY:-}
PYTHONDONTWRITEBYTECODE: "1"
cap_drop: [ALL]
read_only: true
diff --git a/docs/api.md b/docs/api.md
index f02e53e..81e922a 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -35,7 +35,9 @@ All paths are relative to `/api` when going through the frontend/nginx.
| `POST /incidents/{id}/diagnose` | admin | Queue a new diagnosis by the agent (409 without a key, 429 while the budget is used up). |
| `GET /admin/agent/usage` | admin | This month's model calls, tokens and estimated cost against the budget, per model and per day, with the prices and limits. |
| `GET /admin/agent/worker` | admin | Whether the agent worker is running, how many jobs wait, are running or wait to be retried, and the latest results. |
-| `POST /admin/agent/test` | admin | Checks the OpenAI key and that the configured model exists, without spending tokens. |
+| `PUT /admin/agent/provider` | admin (rate limited) | Choose the agent's provider (`anthropic`, `openai`, `gemini`) and model, and optionally save a new API key, stored encrypted (409 without `SECRETS_KEY`). Returns the choice and whether a key is saved, never the key. |
+| `DELETE /admin/agent/key` | admin | Remove the chosen provider's API key; the agent stops until a new one is saved. |
+| `POST /admin/agent/test` | admin | Checks the saved key and that the configured model exists, without spending tokens. |
| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |
| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |
| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |
diff --git a/docs/assistant.md b/docs/assistant.md
index 19a2ed4..f4f9289 100644
--- a/docs/assistant.md
+++ b/docs/assistant.md
@@ -8,19 +8,31 @@ change files on your servers. Everything about it is in `src/agent/` and in **Ad
## Setting it up
-1. Create an API key at <https://platform.openai.com/api-keys> and put it in `.env`:
- ```
- OPENAI_API_KEY=sk-…
- AGENT_MODEL=gpt-5-mini # optional; any OpenAI model with Structured Outputs
- ```
-2. `docker compose up -d backend agent-worker` (Compose recreates both with the new values).
-3. **Admin → Incident agent → Test connection** checks the key and that the model exists, without spending tokens.
-4. Enter the model's price per million tokens under **Prices** (from the provider's pricing page), so its calls
+1. Make sure `SECRETS_KEY` is set in `.env` (see [Configuration](configuration.md)): the API key is stored
+ encrypted with it, so it can't be saved without one.
+2. Create an API key with one of the providers: Anthropic (Claude), OpenAI or Google Gemini.
+3. In **Admin → Incident agent**, pick the provider, keep or change the model, paste the key and click **Save**.
+ Models used when the field is left empty:
+
+ | Provider | Default model |
+ |---|---|
+ | Anthropic | `claude-haiku-5-5` |
+ | OpenAI | `gpt-6-luna` |
+ | Google Gemini | `gemini-3.1-flash-lite` |
+
+ Another model works if it supports structured (JSON schema) output.
+4. **Test connection** checks the key and that the model exists, without spending tokens.
+5. Enter the model's price per million tokens under **Prices** (from the provider's pricing page), so its calls
count towards the budget, and set a **Monthly budget** if you want one.
+The key never comes back from the backend: the page only shows that one is saved. Each provider keeps its own
+key, so switching back and forth doesn't lose them; **Remove key** deletes the chosen provider's key. Every change
+is written to the audit log, without the key. The provider addresses are fixed in the code
+(`src/agent/providers/`), so the agent's data can only go to one of these three APIs.
+
Without a key nothing is sent anywhere: incidents keep the rule-based diagnosis (worked out locally from the same
-data), notifications go out as before, and the chat is off. Change the model by changing `AGENT_MODEL` and running
-step 2 again.
+data), notifications go out as before, and the chat is off. If `SECRETS_KEY` changes and the stored key can no
+longer be decrypted, the agent behaves as if there were no key until it is saved again.
## Diagnoses
@@ -87,9 +99,10 @@ off, the budget used up, or the worker not running. Resolved notifications never
## Chat
The chat answers questions about an incident or any server. Each message goes to the model with the incident's
-facts, and the conversation so far (its latest 40 turns). Every reply is held to a JSON schema with OpenAI's
-Structured Outputs: a message, and at most one check from a fixed list. The provider sits behind a small interface
-(`providers.py`), so other providers can be added later.
+facts, and the conversation so far (its latest 40 turns). Every reply is held to a JSON schema with the
+provider's structured output: a message, and at most one check from a fixed list. All three providers sit behind
+one small interface (`src/agent/providers/`). With Claude, the chat thinks briefly (effort `low`) and a diagnosis
+thinks harder (effort `high`); the other providers ignore the setting.
The chat has no code path for running shell commands. It may request exactly one fixed, read-only check at a time:
`server_health`, `container_health`, or `daily_checkup`, implemented with Python system APIs and Docker's Unix API
diff --git a/docs/configuration.md b/docs/configuration.md
index b956752..8bac009 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -11,8 +11,6 @@ Every setting below is listed, with no values, in [`example.env`](../example.env
| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. |
| `POSTGRES_DB` / `POSTGRES_USER` | `infradeck` / `infradeck` | Name of the database and its user. Installs from before the rename can keep `homelab` (see [SETUP_DATABASES.md](../SETUP_DATABASES.md)). |
| `REDIS_PASSWORD` | *(required)* | Redis password. Redis holds the login sessions, so nothing that can merely reach it may use it. Use letters and digits only (`openssl rand -hex 24`) because it is put into a URL. |
-| `OPENAI_API_KEY` | empty | OpenAI API key for the incident agent: its diagnoses and the chat. Empty: both are off, and incidents keep the rule-based diagnosis. **Admin → Incident agent → Test connection** checks it. |
-| `AGENT_MODEL` | `gpt-5-mini` | OpenAI model for the agent. It must support Structured Outputs. |
| `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. |
| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. |
| `SESSION_TTL_SECONDS` | `86400` | Session lifetime without *Remember me*. |
diff --git a/docs/containers.md b/docs/containers.md
index a720610..f13c143 100644
--- a/docs/containers.md
+++ b/docs/containers.md
@@ -107,7 +107,7 @@ through the gate); on the development stack the local collector talks to the bac
- registering collectors and receiving their check-ins (`/monitoring/metrics/ingest`);
- the alert rules, incidents and notification channels (email/webhook, quiet hours);
- statistics, rollups (summarised history) and retention;
-- the optional incident agent chat, which talks to the OpenAI API when `OPENAI_API_KEY` is set.
+- the optional incident agent chat, which talks to the chosen AI provider (Anthropic, OpenAI or Google Gemini) once an admin saves an API key.
**How it works.** A FastAPI app run by uvicorn on port 8000. At startup it creates any missing tables and
columns in PostgreSQL, moves old Redis keys to the current prefix, and prints the first-run setup token
diff --git a/docs/deployment.md b/docs/deployment.md
index c87b0fd..9a8884f 100644
--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -39,8 +39,7 @@ Then set at least these:
| `TRAEFIK_NETWORK` | `traefik_default` | Traefik only: the Docker network Traefik is attached to. |
| `TRAEFIK_ROUTER` | `infradeck` | Traefik only: name of this dashboard's router and service; change it only if it clashes with another one. |
| `COMPOSE_FILE` | `docker-compose.server.yml`, with Traefik `docker-compose.server.yml:deploy/traefik.yml` | Makes plain `docker compose ...` commands use the server file and your proxy's file (see below). |
-| `SECRETS_KEY` | *(strongly recommended)* | Encrypts two-factor secrets and notification addresses. Two-factor is required for admins by default and can't be set up without it. Generate with `docker compose run --rm backend python -m src.backend.secrets_box`. |
-| `OPENAI_API_KEY`, `AGENT_MODEL` | empty, `gpt-5-mini` | Optional: the incident agent's diagnoses and chat. See [Incident agent](assistant.md#setting-it-up). |
+| `SECRETS_KEY` | *(strongly recommended)* | Encrypts two-factor secrets, notification addresses and the incident agent's API key. Two-factor is required for admins by default and can't be set up without it. Generate with `docker compose run --rm backend python -m src.backend.secrets_box`. |
With Traefik, find its network with `docker network ls` or
`docker inspect <traefik-container> --format '{{json .NetworkSettings.Networks}}'`.
@@ -84,8 +83,8 @@ certificate configuration.
The public address serves only the dashboard. Collectors don't use it; set up
their route in the next step.
-The stack also starts **`agent-worker`**, which runs the incident agent's diagnoses. Without `OPENAI_API_KEY` it
-only notes that there is no key; with one, enter the model's price in **Admin → Incident agent** so its calls count
+The stack also starts **`agent-worker`**, which runs the incident agent's diagnoses. Until an admin saves an API key in
+**Admin → Incident agent** (see [Incident agent](assistant.md#setting-it-up)) it only notes that there is no key; with one, enter the model's price in **Admin → Incident agent** so its calls count
towards the budget.
## 4. Let remote servers report over your VPN
diff --git a/docs/development.md b/docs/development.md
index 5ad4628..4740d29 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -104,7 +104,7 @@ src/
secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key
reset_2fa.py command-line two-factor reset for a locked-out account
agent/ the incident agent; runs in the backend (chat) and the agent-worker container
- providers.py the language model behind one interface (OpenAI)
+ providers/ the language model behind one interface: Claude, OpenAI, Gemini; the choice and its encrypted key
chat.py, prompts.py, SYSTEM_PROMPT.md the incident chat and its instructions
health_checks.py the fixed, read-only checks the agent may ask for
jobs.py job queue in Redis: published after an incident opens or worsens
diff --git a/example.env b/example.env
index a6ea815..2304628 100644
--- a/example.env
+++ b/example.env
@@ -51,12 +51,12 @@ POSTGRES_DB=
POSTGRES_USER=
# =============================================================================
-# Two-factor sign-in
+# Stored secrets (two-factor, notifications, incident agent)
# =============================================================================
-# (both) Key that encrypts secrets the app must read back, such as two-factor secrets and notification
-# addresses. Without it two-factor sign-in can't be set up and is not enforced. Keep it out of database
-# backups. Generate one with:
+# (both) Key that encrypts secrets the app must read back, such as two-factor secrets, notification addresses
+# and the incident agent's API key. Without it two-factor sign-in can't be set up and is not enforced, and the
+# agent's API key can't be saved. Keep it out of database backups. Generate one with:
# docker compose run --rm backend python -m src.backend.secrets_box
# To rotate, put the new key first and keep the old one after a comma: <new>,<old>
SECRETS_KEY=
@@ -238,18 +238,6 @@ SMTP_FROM=
# (both) URL that receives a JSON POST when the daily health check finds a critical problem.
ADMIN_ALERT_WEBHOOK_URL=
-# =============================================================================
-# Incident agent (optional)
-# =============================================================================
-
-# (both) OpenAI API key for the incident agent's chat. Empty: chat is off, monitoring and the rule-based
-# diagnosis work without it. Create one at https://platform.openai.com/api-keys
-# Admin → Incident agent → Test connection checks it without spending tokens.
-OPENAI_API_KEY=
-
-# (both) OpenAI model the agent uses. It must support Structured Outputs. Default: gpt-5-mini
-AGENT_MODEL=
-
# =============================================================================
# Optional, rarely needed
# =============================================================================
diff --git a/requirements.txt b/requirements.txt
index 2bd2d00..9c7d0ac 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,5 @@
alembic
+anthropic
bcrypt
cryptography
email-validator
diff --git a/src/agent/chat.py b/src/agent/chat.py
index e3f8c46..c4a2a0f 100644
--- a/src/agent/chat.py
+++ b/src/agent/chat.py
@@ -39,15 +39,14 @@ class Reply:
class Chat:
"""One conversation's turns, sent in full (up to MAX_HISTORY_MESSAGES) with every new message."""
- def __init__(self, provider: Provider):
- self.provider = provider
+ def __init__(self):
self.history: list[dict] = []
- def send_message(self, message: str) -> Reply:
+ def send_message(self, provider: Provider, message: str) -> Reply:
self.history.append({"role": "user", "content": message})
messages = [{"role": "system", "content": SYSTEM_PROMPT}, *self.history[-MAX_HISTORY_MESSAGES:]]
try:
- completion = self.provider.complete(messages, REPLY_SCHEMA, "agent_reply")
+ completion = provider.complete(messages, REPLY_SCHEMA, "agent_reply", effort="low")
except Exception:
self.history.pop() # a failed turn is not part of the conversation
raise
@@ -58,8 +57,7 @@ class Chat:
class AgentService:
"""LLM interface restricted to a small, fixed health-check capability set."""
- def __init__(self, health_checks: HealthCheckRunner | None = None, provider: Provider | None = None):
- self.provider = provider or get_provider()
+ def __init__(self, health_checks: HealthCheckRunner | None = None):
self.health_checks = health_checks or HealthCheckRunner()
# Least recently used first (a dict keeps insertion order). A chat only lives in memory so the assistant
# remembers the conversation; the cache is bounded so endless new conversations can't grow it without limit.
@@ -67,7 +65,7 @@ class AgentService:
self.lock = threading.Lock()
def _new_chat(self) -> Chat:
- return Chat(self.provider)
+ return Chat()
def _get_chat(self, user_id: int, conversation_id: str):
chat_key = (user_id, conversation_id)
@@ -81,19 +79,19 @@ class AgentService:
return chat
@staticmethod
- def _ask(chat, message: str, usage_logger: Callable[[Completion, int], None] | None):
+ def _ask(chat, provider: Provider, message: str, usage_logger: Callable[[Completion, int], None] | None):
"""One call to the model; ``usage_logger`` gets its tokens and how long it took, in milliseconds."""
started = time.monotonic()
- reply = chat.send_message(message)
+ reply = chat.send_message(provider, message)
completion = getattr(reply, "completion", None)
if usage_logger and completion is not None:
usage_logger(completion, round((time.monotonic() - started) * 1000))
return reply
- def send(self, user_id: int, conversation_id: str, message: str, command_logger: Callable[[dict], None] | None = None,
+ def send(self, user_id: int, conversation_id: str, message: str, provider: Provider, command_logger: Callable[[dict], None] | None = None,
usage_logger: Callable[[Completion, int], None] | None = None) -> str:
chat = self._get_chat(user_id, conversation_id)
- response_text, action = self._parse_agent_response(self._response_text(self._ask(chat, message, usage_logger)))
+ response_text, action = self._parse_agent_response(self._response_text(self._ask(chat, provider, message, usage_logger)))
if not action:
return response_text
@@ -110,7 +108,7 @@ class AgentService:
"check": check_name, "level": level,
"steps": [f"Completed predefined check '{check_name}' with status {result['status']}."],
})
- final_text, final_action = self._parse_agent_response(self._response_text(self._ask(chat, "[HEALTH-CHECK-RESULT]\n" + json.dumps(result), usage_logger)))
+ final_text, final_action = self._parse_agent_response(self._response_text(self._ask(chat, provider, "[HEALTH-CHECK-RESULT]\n" + json.dumps(result), usage_logger)))
if final_action:
raise ValueError("Agent must summarize a health-check result without requesting another check.")
return final_text
@@ -148,11 +146,11 @@ class AgentService:
_agent_service = None
-def send_agent_message(user_id: int, conversation_id: str, message: str, command_logger=None, usage_logger=None) -> str:
+def send_agent_message(user_id: int, conversation_id: str, message: str, provider: Provider, command_logger=None, usage_logger=None) -> str:
global _agent_service
if _agent_service is None:
_agent_service = AgentService()
- return _agent_service.send(user_id, conversation_id, message, command_logger=command_logger, usage_logger=usage_logger)
+ return _agent_service.send(user_id, conversation_id, message, provider, command_logger=command_logger, usage_logger=usage_logger)
def delete_agent_conversation(user_id: int, conversation_id: str) -> None:
@@ -160,6 +158,6 @@ def delete_agent_conversation(user_id: int, conversation_id: str) -> None:
_agent_service.delete_conversation(user_id, conversation_id)
-def check_provider_connection() -> dict:
+def check_provider_connection(db) -> dict:
"""For Admin's "Test connection": the key works and the configured model exists. Spends no tokens."""
- return get_provider().check()
+ return get_provider(db).check()
diff --git a/src/agent/diagnosis.py b/src/agent/diagnosis.py
index 93fc20a..70fd47c 100644
--- a/src/agent/diagnosis.py
+++ b/src/agent/diagnosis.py
@@ -144,7 +144,7 @@ def diagnose(db, incident: IncidentAlert, provider: Provider, trigger: str = "in
for _attempt in range(MAX_ATTEMPTS):
started = time.monotonic()
try:
- completion = provider.complete(_messages(pack, reason), SCHEMA, "incident_diagnosis")
+ completion = provider.complete(_messages(pack, reason), SCHEMA, "incident_diagnosis", effort="high")
except Exception as exc:
usage.record(db, kind="diagnosis", status="error", model=model, incident_id=incident.id, duration_ms=round((time.monotonic() - started) * 1000), error=f"{type(exc).__name__}: {exc}")
raise
diff --git a/src/agent/providers.py b/src/agent/providers.py
deleted file mode 100644
index 2c1d19f..0000000
--- a/src/agent/providers.py
+++ /dev/null
@@ -1,81 +0,0 @@
-"""The language model behind the agent, behind one small interface so another provider can be added later.
-
-A provider turns a list of chat messages into one reply that matches a JSON schema, and reports how many tokens
-that took. It never sees the database or the servers; the agent decides what goes into the messages.
-"""
-from __future__ import annotations
-
-import os
-from dataclasses import dataclass
-from typing import Protocol
-
-DEFAULT_MODEL = "gpt-5-mini"
-TIMEOUT_SECONDS = 60
-MAX_RETRIES = 2
-
-
-@dataclass
-class Completion:
- text: str
- model: str
- input_tokens: int = 0
- output_tokens: int = 0
-
-
-class Provider(Protocol):
- name: str
-
- def complete(self, messages: list[dict], schema: dict, schema_name: str) -> Completion:
- """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. Returns JSON text matching ``schema``."""
-
- def check(self) -> dict:
- """Confirms the key works and the model exists, without spending tokens."""
-
-
-def configured_key() -> str:
- return os.getenv("OPENAI_API_KEY", "").strip()
-
-
-def configured_model() -> str:
- return os.getenv("AGENT_MODEL", "").strip() or DEFAULT_MODEL
-
-
-class OpenAIProvider:
- """The OpenAI API through Chat Completions, with Structured Outputs so a reply always parses."""
-
- name = "OpenAI"
-
- def __init__(self, api_key: str | None = None, model: str | None = None, client=None):
- key = api_key or configured_key()
- if not key and client is None:
- raise RuntimeError("OPENAI_API_KEY is not configured.")
- self.model = model or configured_model()
- if client is None:
- from openai import OpenAI # imported here so the rest of the app runs without the package
- client = OpenAI(api_key=key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
- self.client = client
-
- def complete(self, messages: list[dict], schema: dict, schema_name: str) -> Completion:
- response = self.client.chat.completions.create(
- model=self.model,
- messages=messages,
- response_format={"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}},
- )
- choice = response.choices[0]
- if getattr(choice.message, "refusal", None):
- raise ValueError(f"The model refused to answer: {choice.message.refusal}")
- usage = getattr(response, "usage", None)
- return Completion(
- text=choice.message.content or "",
- model=getattr(response, "model", self.model),
- input_tokens=getattr(usage, "prompt_tokens", 0) or 0,
- output_tokens=getattr(usage, "completion_tokens", 0) or 0,
- )
-
- def check(self) -> dict:
- model = self.client.models.retrieve(self.model)
- return {"provider": self.name, "model": getattr(model, "id", self.model)}
-
-
-def get_provider() -> Provider:
- return OpenAIProvider()
diff --git a/src/agent/providers/__init__.py b/src/agent/providers/__init__.py
new file mode 100644
index 0000000..24fab9b
--- /dev/null
+++ b/src/agent/providers/__init__.py
@@ -0,0 +1,101 @@
+"""The language model behind the agent: which provider and model, chosen in Admin › Incident agent and stored in
+app_settings. The API key is stored encrypted with SECRETS_KEY (secrets_box) and never leaves the backend again.
+
+Each provider keeps its own key, so switching between them doesn't lose the others. Provider addresses are fixed
+here, not configurable, so the agent's data can only ever go to one of these three APIs.
+"""
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+
+from src.backend import secrets_box
+from src.backend.models import AppSetting
+
+from .base import Completion, Provider
+from .claude import ClaudeProvider
+from .openai_compatible import GEMINI_BASE_URL, OpenAIProvider
+
+__all__ = ["Completion", "Provider", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]
+
+logger = logging.getLogger(__name__)
+
+# id: (name shown in Admin, model used when none is set)
+PROVIDERS = {
+ "anthropic": ("Anthropic", "claude-haiku-5-5"),
+ "openai": ("OpenAI", "gpt-6-luna"),
+ "gemini": ("Google Gemini", "gemini-3.1-flash-lite"),
+}
+PROVIDER_KEY = "agent_provider"
+MODEL_KEY = "agent_model"
+API_KEY_PREFIX = "agent_api_key_"
+
+
+class NoApiKey(RuntimeError):
+ """No provider is chosen, or it has no usable key."""
+
+
+@dataclass
+class Choice:
+ provider: str | None # None until an admin picks one
+ model: str
+ key: str
+
+
+def _get(db, key: str) -> str | None:
+ setting = db.get(AppSetting, key)
+ return setting.value if setting else None
+
+
+def _set(db, key: str, value: str | None) -> None:
+ setting = db.get(AppSetting, key)
+ if value is None:
+ if setting is not None:
+ db.delete(setting)
+ elif setting is None:
+ db.add(AppSetting(key=key, value=value))
+ else:
+ setting.value = value
+
+
+def current(db) -> Choice:
+ provider = _get(db, PROVIDER_KEY)
+ if provider not in PROVIDERS:
+ return Choice(None, "", "")
+ key = ""
+ stored = _get(db, API_KEY_PREFIX + provider)
+ if stored:
+ try:
+ key = secrets_box.decrypt(stored)
+ except secrets_box.SecretsUnavailable:
+ # SECRETS_KEY was changed or removed: the key is unreadable, so the agent behaves as if there were none.
+ logger.warning("The stored API key for %s can't be decrypted with the configured SECRETS_KEY.", provider)
+ return Choice(provider, _get(db, MODEL_KEY) or PROVIDERS[provider][1], key)
+
+
+def key_configured(db) -> bool:
+ return bool(current(db).key)
+
+
+def get_provider(db) -> Provider:
+ choice = current(db)
+ if not choice.key:
+ raise NoApiKey("The agent has no API key.")
+ if choice.provider == "anthropic":
+ return ClaudeProvider(choice.key, choice.model)
+ if choice.provider == "gemini":
+ return OpenAIProvider(choice.key, choice.model, name="Google Gemini", base_url=GEMINI_BASE_URL)
+ return OpenAIProvider(choice.key, choice.model)
+
+
+def save(db, provider: str, model: str | None, api_key: str | None) -> None:
+ """Choose ``provider`` and ``model`` (None: the provider's default). A new ``api_key`` replaces the stored one;
+ None keeps it. Raises SecretsUnavailable when a key is given but SECRETS_KEY isn't set. The caller commits."""
+ if api_key:
+ _set(db, API_KEY_PREFIX + provider, secrets_box.encrypt(api_key))
+ _set(db, PROVIDER_KEY, provider)
+ _set(db, MODEL_KEY, model or None)
+
+
+def remove_key(db, provider: str) -> None:
+ _set(db, API_KEY_PREFIX + provider, None)
diff --git a/src/agent/providers/base.py b/src/agent/providers/base.py
new file mode 100644
index 0000000..d0fcd2c
--- /dev/null
+++ b/src/agent/providers/base.py
@@ -0,0 +1,29 @@
+"""What every provider returns and offers. A provider never sees the database or the servers; the agent decides
+what goes into the messages, and masks them first (masking.py)."""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol
+
+TIMEOUT_SECONDS = 60
+MAX_RETRIES = 2
+
+
+@dataclass
+class Completion:
+ text: str
+ model: str
+ input_tokens: int = 0
+ output_tokens: int = 0
+
+
+class Provider(Protocol):
+ name: str
+ model: str
+
+ def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
+ """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. Returns JSON text matching ``schema``.
+ ``effort`` ("low" | "medium" | "high") is how hard the model thinks, where the provider supports it."""
+
+ def check(self) -> dict:
+ """Confirms the key works and the model exists, without spending tokens."""
diff --git a/src/agent/providers/claude.py b/src/agent/providers/claude.py
new file mode 100644
index 0000000..e6c09d8
--- /dev/null
+++ b/src/agent/providers/claude.py
@@ -0,0 +1,47 @@
+"""Claude through the Anthropic API, with structured outputs so a reply always parses."""
+from __future__ import annotations
+
+from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion
+
+# Enough for adaptive thinking plus the longest answer the schemas allow, and small enough for a non-streaming request.
+MAX_TOKENS = 16000
+
+
+class ClaudeProvider:
+ name = "Anthropic"
+
+ def __init__(self, api_key: str, model: str, client=None):
+ self.model = model
+ if client is None:
+ import anthropic # imported here so the rest of the app runs without the package
+ client = anthropic.Anthropic(api_key=api_key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
+ self.client = client
+
+ def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
+ system = "\n\n".join(message["content"] for message in messages if message["role"] == "system")
+ turns = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]
+ response = self.client.messages.create(
+ model=self.model,
+ max_tokens=MAX_TOKENS,
+ system=system,
+ messages=turns,
+ thinking={"type": "adaptive"},
+ output_config={"effort": effort, "format": {"type": "json_schema", "schema": schema}},
+ cache_control={"type": "ephemeral"}, # the system prompt and earlier turns repeat on every call
+ )
+ if response.stop_reason == "refusal":
+ raise ValueError("The model refused to answer.")
+ if response.stop_reason == "max_tokens":
+ raise ValueError("The model's answer was cut off before it was complete.")
+ usage = response.usage
+ return Completion(
+ text="".join(block.text for block in response.content if block.type == "text"),
+ model=getattr(response, "model", self.model),
+ # Cached input is billed differently, but it is still input: counting it keeps the budget on the safe side.
+ input_tokens=(usage.input_tokens or 0) + (getattr(usage, "cache_creation_input_tokens", 0) or 0) + (getattr(usage, "cache_read_input_tokens", 0) or 0),
+ output_tokens=usage.output_tokens or 0,
+ )
+
+ def check(self) -> dict:
+ model = self.client.models.retrieve(self.model)
+ return {"provider": self.name, "model": getattr(model, "id", self.model)}
diff --git a/src/agent/providers/openai_compatible.py b/src/agent/providers/openai_compatible.py
new file mode 100644
index 0000000..e46df4c
--- /dev/null
+++ b/src/agent/providers/openai_compatible.py
@@ -0,0 +1,38 @@
+"""OpenAI, and Google Gemini through its OpenAI-compatible endpoint: both take the same Chat Completions request."""
+from __future__ import annotations
+
+from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion
+
+GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/"
+
+
+class OpenAIProvider:
+ """Chat Completions with Structured Outputs, so a reply always parses."""
+
+ def __init__(self, api_key: str, model: str, name: str = "OpenAI", base_url: str | None = None, client=None):
+ self.name, self.model = name, model
+ if client is None:
+ from openai import OpenAI # imported here so the rest of the app runs without the package
+ client = OpenAI(api_key=api_key, base_url=base_url, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
+ self.client = client
+
+ def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
+ response = self.client.chat.completions.create(
+ model=self.model,
+ messages=messages,
+ response_format={"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}},
+ )
+ choice = response.choices[0]
+ if getattr(choice.message, "refusal", None):
+ raise ValueError(f"The model refused to answer: {choice.message.refusal}")
+ usage = getattr(response, "usage", None)
+ return Completion(
+ text=choice.message.content or "",
+ model=getattr(response, "model", self.model),
+ input_tokens=getattr(usage, "prompt_tokens", 0) or 0,
+ output_tokens=getattr(usage, "completion_tokens", 0) or 0,
+ )
+
+ def check(self) -> dict:
+ model = self.client.models.retrieve(self.model)
+ return {"provider": self.name, "model": getattr(model, "id", self.model)}
diff --git a/src/agent/worker.py b/src/agent/worker.py
index bf891d3..3ee3b06 100644
--- a/src/agent/worker.py
+++ b/src/agent/worker.py
@@ -19,7 +19,7 @@ import threading
import time
from src.agent import diagnosis, jobs, settings, usage, verify
-from src.agent.providers import configured_key, get_provider
+from src.agent.providers import get_provider, key_configured
from src.backend.config import REDIS_URL
from src.backend.db import SessionLocal
from src.backend.models import IncidentAlert
@@ -44,14 +44,14 @@ def handle_incident(job: dict) -> tuple[str, str]:
return "skipped", "This month's budget for the agent is used up."
if job.get("type") != jobs.REQUESTED and not settings.auto_diagnose(db):
return "skipped", "Automatic diagnosis is switched off in Admin; it runs when someone asks."
- if not configured_key():
- return "skipped", "No OPENAI_API_KEY is configured; the incident keeps its rule-based diagnosis."
+ if not key_configured(db):
+ return "skipped", "The agent has no API key (Admin › Incident agent); the incident keeps its rule-based diagnosis."
if job.get("type", jobs.OPENED) == jobs.OPENED:
# The same problem again within a day: its diagnosis still applies, and asking again would cost tokens.
reused = diagnosis.reuse_recent(db, incident)
if reused is not None:
return "reused", f"Same diagnosis as INC-{json.loads(reused.body)['reused_from']['incident_id']:04d}; no tokens spent"
- report = diagnosis.diagnose(db, incident, get_provider(), trigger=job.get("type", jobs.OPENED))
+ report = diagnosis.diagnose(db, incident, get_provider(db), trigger=job.get("type", jobs.OPENED))
if report.status != "ok":
return "failed", report.error or "The model's answer could not be used."
return "diagnosed", f"{report.confidence} confidence · version {report.version}"
diff --git a/src/backend/config.py b/src/backend/config.py
index 2b5e1f0..15385a2 100644
--- a/src/backend/config.py
+++ b/src/backend/config.py
@@ -20,8 +20,6 @@ DATABASE_URL = os.getenv(
)
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
SESSION_TTL_SECONDS = int(os.getenv("SESSION_TTL_SECONDS", "86400"))
-# The OpenAI model the agent uses (see src/agent/providers.py).
-AGENT_MODEL = os.getenv("AGENT_MODEL", "").strip() or "gpt-5-mini"
SESSION_COOKIE_NAME = os.getenv("SESSION_COOKIE_NAME", "homie_session")
SESSION_COOKIE_SECURE = os.getenv("SESSION_COOKIE_SECURE", "false").lower() == "true"
# When false (the default), only the first account can be created, so a fresh install can still be set up and
diff --git a/src/backend/main.py b/src/backend/main.py
index ac583a4..fe85d07 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -39,7 +39,6 @@ from .models import ApiToken, Invite, NotificationChannel, AppSetting, AuditLog,
from .db import engine
from .migrate import upgrade_database
from .config import (
- AGENT_MODEL,
APP_VERSION,
LOGIN_RATE_LIMIT,
LOGIN_RATE_WINDOW_SECONDS,
@@ -67,7 +66,7 @@ from .rules import HELPER_EVENT_KINDS, DEFAULT_THRESHOLDS, HOST_RULE_PREFIX, con
from .rate_limit import enforce_ip_limit, enforce_limit, enforce_login_limit, enforce_register_limit, refund_attempt, reset_limits
from ..agent import jobs as agent_jobs, settings as agent_settings, usage as agent_usage, verify as agent_verify
from ..agent.diagnosis import latest as latest_report, report_json
-from ..agent.providers import configured_key
+from ..agent.providers import NoApiKey, get_provider, key_configured
from ..agent.chat import check_provider_connection, delete_agent_conversation, send_agent_message
@@ -210,22 +209,22 @@ class MetricsIngestRequest(BaseModel):
def agent_error_message(exc: Exception) -> str:
"""Turn provider failures into useful, non-sensitive chat text."""
+ if isinstance(exc, NoApiKey):
+ return "The agent has no API key. An admin adds one in Admin › Incident agent."
raw_message = str(exc).strip()
normalized = raw_message.lower()
- if "openai_api_key is not configured" in normalized:
- return "The agent has no API key. An admin sets OPENAI_API_KEY in .env and restarts the backend."
- if any(marker in normalized for marker in ("401", "invalid_api_key", "incorrect api key", "authentication")):
- prefix = "OpenAI rejected the API key. Check OPENAI_API_KEY in .env."
- elif any(marker in normalized for marker in ("model_not_found", "does not exist", "404")):
- prefix = "OpenAI doesn't offer the configured model to this key. Check AGENT_MODEL in .env."
- elif any(marker in normalized for marker in ("resourceexhausted", "overloaded", "capacity", " 503", "status 503", " 500", " 502")):
- prefix = "OpenAI is currently overloaded or unavailable. Please try again shortly."
- elif any(marker in normalized for marker in ("rate limit", "quota", " 429", "status 429", "insufficient_quota")):
- prefix = "OpenAI rate limit or quota was reached. Please try again later."
+ if any(marker in normalized for marker in ("401", "invalid_api_key", "incorrect api key", "api key not valid", "api_key_invalid", "authentication")):
+ prefix = "The AI provider rejected the API key. Check it in Admin › Incident agent."
+ elif any(marker in normalized for marker in ("model_not_found", "does not exist", "not_found_error", "404")):
+ prefix = "The AI provider doesn't offer the configured model to this key. Check the model in Admin › Incident agent."
+ elif any(marker in normalized for marker in ("resourceexhausted", "overloaded", "capacity", " 503", "status 503", " 500", " 502", " 529")):
+ prefix = "The AI provider is currently overloaded or unavailable. Please try again shortly."
+ elif any(marker in normalized for marker in ("rate limit", "rate_limit", "quota", " 429", "status 429", "insufficient_quota")):
+ prefix = "The AI provider's rate limit or quota was reached. Please try again later."
elif any(marker in normalized for marker in ("timeout", "timed out", "deadline exceeded")):
- prefix = "OpenAI did not respond before the request timed out. Please try again."
+ prefix = "The AI provider did not respond before the request timed out. Please try again."
else:
- prefix = "OpenAI could not complete this request."
+ prefix = "The AI provider could not complete this request."
if not raw_message:
return prefix
@@ -712,7 +711,7 @@ def agent_config(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_N
user = _current_user(session_id, db)
spend = agent_usage.budget(db)
account = db.get(User, user["id"])
- return {"chat_enabled": is_agent_chat_enabled(db), "may_chat": bool(account) and agent_settings.may_chat(db, account), "key_configured": bool(configured_key()), "budget_paused": spend["state"] == "paused"}
+ return {"chat_enabled": is_agent_chat_enabled(db), "may_chat": bool(account) and agent_settings.may_chat(db, account), "key_configured": key_configured(db), "budget_paused": spend["state"] == "paused"}
OVERVIEW_ALERT_LIMIT = 50
@@ -1607,7 +1606,7 @@ DIAGNOSIS_WAIT = timedelta(minutes=3)
def _diagnosis_coming(db) -> bool:
"""Whether a diagnosis can be expected for a new incident: there is a key, automatic diagnosis is on, the budget
isn't used up and a worker is running. When not, notifications don't wait for one."""
- if not configured_key() or not agent_settings.auto_diagnose(db) or agent_usage.budget(db)["state"] == "paused":
+ if not key_configured(db) or not agent_settings.auto_diagnose(db) or agent_usage.budget(db)["state"] == "paused":
return False
try:
return bool(redis_client.get(agent_jobs.HEARTBEAT))
diff --git a/src/backend/routes/admin.py b/src/backend/routes/admin.py
index a9c4851..aef1c8b 100644
--- a/src/backend/routes/admin.py
+++ b/src/backend/routes/admin.py
@@ -4,15 +4,16 @@ from __future__ import annotations
import secrets
from datetime import datetime, timedelta, timezone
-from typing import Annotated
+from typing import Annotated, Literal
from fastapi import APIRouter, Cookie, Depends, HTTPException
+from pydantic import BaseModel, Field
from sqlalchemy import func, text
from .. import incidents, main, notify, secrets_box
from ...agent import jobs as agent_jobs, settings as agent_settings, usage as agent_usage
-from ...agent.providers import configured_key
-from ..config import AGENT_MODEL, APP_VERSION, SESSION_COOKIE_NAME
+from ...agent import providers as agent_providers
+from ..config import APP_VERSION, SESSION_COOKIE_NAME
from ..db import engine, get_db
from ..models import ChatMessage, HostMetric, IncidentAlert, Invite, Server, User
@@ -117,6 +118,12 @@ def admin_rules(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NA
"fired": fired.get(row["kind"], 0), "noisy": fired.get(row["kind"], 0) >= main.NOISY_FIRED} for row in main.RULE_ROWS],
"thresholds": thresholds}
+def _agent_provider(db) -> dict:
+ """The chosen provider and model, whether it has a key, and the choices. The key itself never leaves the backend."""
+ choice = agent_providers.current(db)
+ return {"provider": choice.provider, "model": choice.model, "key_configured": bool(choice.key),
+ "providers": [{"id": key, "name": name, "default_model": model} for key, (name, model) in agent_providers.PROVIDERS.items()]}
+
@router.get("/admin/system")
def admin_system(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
"""Read-only facts about this install for the Admin page: version, database size, what the agent was used for."""
@@ -131,7 +138,7 @@ def admin_system(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_N
"version": APP_VERSION, "database": {"engine": engine.dialect.name, "bytes": size,
"host_rows": db.query(func.count(HostMetric.id)).scalar(), "host_rows_week": db.query(func.count(HostMetric.id)).filter(HostMetric.time >= week).scalar()},
"history_retention_days": main.retention.history_days(db),
- "agent": {"provider": "OpenAI", "model": AGENT_MODEL, "key_configured": bool(configured_key()), "budget": agent_usage.budget(db), "chat_enabled": main.is_agent_chat_enabled(db),
+ "agent": {**_agent_provider(db), "budget": agent_usage.budget(db), "chat_enabled": main.is_agent_chat_enabled(db),
"incidents_this_month": db.query(IncidentAlert).filter(IncidentAlert.created_at >= month).count(),
"chat_messages_this_month": db.query(ChatMessage).filter(ChatMessage.role == "user", ChatMessage.created_at >= month).count()},
"smtp_configured": notify.smtp_configured(), "secrets_configured": secrets_box.available(),
@@ -141,7 +148,7 @@ def admin_system(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_N
def admin_agent_usage(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
"""This month's model calls, tokens and estimated cost against the budget, per model and per day."""
main._require_admin(session_id, db)
- return {**agent_usage.month_summary(db), "model": AGENT_MODEL}
+ return {**agent_usage.month_summary(db), "model": agent_providers.current(db).model}
@router.get("/admin/agent/worker")
def admin_agent_worker(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
@@ -158,7 +165,41 @@ def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOK
admin = main._require_admin(session_id, db)
main.enforce_limit(main.redis_client, scope="agent-test", subject=admin["id"], limit=10, window_seconds=600)
try:
- result = main.check_provider_connection()
+ result = main.check_provider_connection(db)
except Exception as exc:
raise HTTPException(status_code=502, detail=main.agent_error_message(exc)) from exc
return {"ok": True, **result}
+
+
+class AgentProviderRequest(BaseModel):
+ provider: Literal["anthropic", "openai", "gemini"]
+ # Empty: the provider's default model. Same names the price list accepts.
+ model: str | None = Field(default=None, max_length=100, pattern=r"^([A-Za-z0-9][A-Za-z0-9._:/-]{0,99})?$")
+ # Empty: keep the stored key. Keys are printable ASCII without spaces.
+ api_key: str | None = Field(default=None, max_length=512, pattern=r"^([\x21-\x7e]{8,512})?$")
+
[email protected]("/admin/agent/provider")
+def admin_agent_provider(payload: AgentProviderRequest, session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
+ """Choose the agent's provider and model, and optionally store a new API key for it (encrypted with SECRETS_KEY)."""
+ admin = main._require_admin(session_id, db)
+ main.enforce_limit(main.redis_client, scope="agent-provider", subject=admin["id"], limit=20, window_seconds=600)
+ if payload.api_key and not secrets_box.available():
+ raise HTTPException(status_code=409, detail="Set SECRETS_KEY in .env first: the API key is stored encrypted with it.")
+ agent_providers.save(db, payload.provider, payload.model, payload.api_key)
+ db.commit()
+ choice = agent_providers.current(db)
+ name = agent_providers.PROVIDERS[payload.provider][0]
+ main.record_audit_log(db, user_id=admin["id"], level="WARNING" if payload.api_key else "INFO", source="admin",
+ message=f"Agent provider set to {name} ({choice.model}){'; a new API key was saved' if payload.api_key else ''}")
+ return _agent_provider(db)
+
[email protected]("/admin/agent/key")
+def admin_agent_remove_key(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
+ """Remove the stored API key of the chosen provider; the agent stops until a new one is saved."""
+ admin = main._require_admin(session_id, db)
+ provider = agent_providers.current(db).provider
+ if provider:
+ agent_providers.remove_key(db, provider)
+ db.commit()
+ main.record_audit_log(db, user_id=admin["id"], level="WARNING", source="admin", message=f"The agent's API key for {agent_providers.PROVIDERS[provider][0]} was removed")
+ return _agent_provider(db)
diff --git a/src/backend/routes/chat.py b/src/backend/routes/chat.py
index a8276a0..3e6e9a7 100644
--- a/src/backend/routes/chat.py
+++ b/src/backend/routes/chat.py
@@ -10,7 +10,7 @@ from fastapi import APIRouter, Cookie, Depends, HTTPException, Query
from sqlalchemy import func
from .. import main
-from ...agent import settings as agent_settings, usage as agent_usage
+from ...agent import providers as agent_providers, settings as agent_settings, usage as agent_usage
from ..config import SESSION_COOKIE_NAME
from ..db import get_db
from ..models import AuditLog, ChatMessage, Conversation, User
@@ -137,6 +137,7 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,
user["id"],
payload.conversation_id,
main._incident_context(db, payload.incident_id) + message,
+ main.get_provider(db),
usage_logger=log_usage,
command_logger=lambda event: main.record_audit_log(
db,
@@ -152,7 +153,7 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,
)
except Exception as exc:
user_message = main.agent_error_message(exc)
- agent_usage.record(db, kind="chat", status="error", model=main.AGENT_MODEL, user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id, error=user_message)
+ agent_usage.record(db, kind="chat", status="error", model=agent_providers.current(db).model or "unknown", user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id, error=user_message)
main.record_audit_log(
db,
user_id=user["id"],
diff --git a/src/backend/routes/incidents.py b/src/backend/routes/incidents.py
index f3f3baa..ac25c08 100644
--- a/src/backend/routes/incidents.py
+++ b/src/backend/routes/incidents.py
@@ -13,7 +13,7 @@ from pydantic import BaseModel, Field
from .. import incidents, main
from ...agent import jobs as agent_jobs, settings as agent_settings, usage as agent_usage, verify as agent_verify
from ...agent.diagnosis import latest as latest_report, report_json
-from ...agent.providers import configured_key
+from ...agent.providers import key_configured
from ..config import SESSION_COOKIE_NAME
from ..db import get_db
from ..models import IncidentAlert, Server, User
@@ -50,7 +50,7 @@ def get_incident(incident_id: main.PathId, session_id: Annotated[str | None, Coo
if report and report["status"] == "ok" and alert.status == "active":
report["watch"] = agent_verify.evaluate(db, alert, report["verify"])
return {**incidents.summary(alert, server.hostname, datetime.now(timezone.utc), users), "diagnosis": incidents.diagnose(db, alert, server), "cpu_cores": server.cpu_cores,
- "agent": {"report": report, "hidden": not visible, "key_configured": bool(configured_key()), "budget_paused": agent_usage.budget(db)["state"] == "paused"}}
+ "agent": {"report": report, "hidden": not visible, "key_configured": key_configured(db), "budget_paused": agent_usage.budget(db)["state"] == "paused"}}
@router.post("/incidents/{incident_id}/diagnose")
@@ -59,8 +59,8 @@ def request_diagnosis(incident_id: main.PathId, session_id: Annotated[str | None
on the incident when the worker gets to it."""
user = main._require_operator(session_id, db)
alert, _server = _incident_or_404(db, incident_id)
- if not configured_key():
- raise HTTPException(status_code=409, detail="The agent has no API key. An admin sets OPENAI_API_KEY in .env.")
+ if not key_configured(db):
+ raise HTTPException(status_code=409, detail="The agent has no API key. An admin adds one in Admin › Incident agent.")
if agent_usage.budget(db)["state"] == "paused":
raise HTTPException(status_code=429, detail="This month's budget for the agent is used up. An admin can raise it in Admin › Incident agent.")
main.enforce_limit(main.redis_client, scope="diagnose-user", subject=user["id"], limit=10, window_seconds=600)
diff --git a/src/frontend/src/admin/AdminPage.jsx b/src/frontend/src/admin/AdminPage.jsx
index 0a8529b..e0dbff9 100644
--- a/src/frontend/src/admin/AdminPage.jsx
+++ b/src/frontend/src/admin/AdminPage.jsx
@@ -12,6 +12,7 @@ import { Caret, Icon } from '../icons.jsx'
import { refreshOverview, useOverview } from '../overviewStore.js'
import { AppShell, serverLink } from '../shell.jsx'
import { ADMIN_GROUPS } from '../more/groups.js'
+import AgentProvider from './AgentProvider.jsx'
import { Field, Notice, Outcome, Row, Section, SettingsLayout, ShownOnce, Soon, Switch, ago, keptFor, useAction, useLoaded, usePhone } from '../ui.jsx'
const REPOSITORY = 'https://github.com/karloks2005/Infradeck'
@@ -275,18 +276,11 @@ function MountOverrides({ settings, onSaved }) {
)
}
-function Agent({ settings, system, onSaved }) {
+function Agent({ settings, system, onSaved, onSystemChanged }) {
const [state, run] = useAction()
- const [test, runTestAction] = useAction()
- const [tested, setTested] = useState(null)
- const runTest = (work) => { setTested(null); return runTestAction(async () => setTested(await work())) }
- const agent = system.agent
return <>
<div className="card divide-y divide-line">
- <Row icon="AI" title="Model provider" detail={agent.key_configured ? `${agent.provider || 'OpenAI'} · ${agent.model} · key configured` : 'No API key configured. Set OPENAI_API_KEY in .env and restart the backend to turn the chat on.'} tone={agent.key_configured ? undefined : 'warn'}>
- <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key and the model without spending tokens.' : 'Set OPENAI_API_KEY first.'} onClick={() => runTest(() => apiRequest('/admin/agent/test', { method: 'POST' }))}>{test.busy ? 'Testing…' : 'Test connection'}</button>
- </Row>
- {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key and offers {tested.model}.</p>}</div>}
+ <AgentProvider agent={system.agent} secretsConfigured={system.secrets_configured} onChanged={onSystemChanged} />
<Row title="Diagnose new incidents automatically" detail="When a warning or critical incident opens or gets worse, the agent worker sends its evidence (masked) to the model and writes a diagnosis. Off: incidents keep the rule-based diagnosis, and anyone who may act on incidents can still ask for one."><Switch label="Diagnose automatically" checked={settings.agent_auto_diagnose !== false} disabled={state.busy} onChange={(on) => run(async () => onSaved(await put({ agent_auto_diagnose: on })))} /></Row>
<Row title="Read container logs" detail="Not available: Infradeck's Docker proxy can't read logs. See SECURITY.md."><Switch label="Read container logs" checked={false} disabled onChange={() => {}} /></Row>
<Row title="Chat" detail="The main switch for the chat; who may use it is set under Chat access. Questions and the incident's facts are sent to the provider."><Switch label="Chat with the agent" checked={settings.agent_chat_enabled} disabled={state.busy} onChange={(on) => run(async () => onSaved(await put({ agent_chat_enabled: on })))} /></Row>
@@ -544,7 +538,7 @@ export default function AdminPage({ user, onLogout }) {
</Section>
<Section id="users" title="Users" description="Who can sign in, what they may do, and whether they must use two-factor."><Users me={user} /></Section>
<Section id="rules" title="Alert rules" description="What opens an incident. Rules apply to every server. The “Fired” column shows how noisy each rule has been."><Rules onCount={setRuleCount} /><MountOverrides settings={settings.data} onSaved={settings.setData} /></Section>
- <Section id="agent" title="Incident agent" description="Diagnoses incidents and answers questions using the data Infradeck already has. It is read-only: it can't run commands, restart containers or change files on your servers."><Agent settings={settings.data} system={system.data} onSaved={settings.setData} /></Section>
+ <Section id="agent" title="Incident agent" description="Diagnoses incidents and answers questions using the data Infradeck already has. It is read-only: it can't run commands, restart containers or change files on your servers."><Agent settings={settings.data} system={system.data} onSaved={settings.setData} onSystemChanged={system.reload} /></Section>
<Section id="retention" title="Data & retention" description="How long history is kept and how big it is."><Retention settings={settings.data} system={system.data} onSaved={settings.setData} /></Section>
<Section id="about" title="About"><About settings={settings.data} system={system.data} onSaved={settings.setData} /></Section>
</>}
diff --git a/src/frontend/src/admin/AgentProvider.jsx b/src/frontend/src/admin/AgentProvider.jsx
new file mode 100644
index 0000000..4bd777c
--- /dev/null
+++ b/src/frontend/src/admin/AgentProvider.jsx
@@ -0,0 +1,65 @@
+// Which AI provider and model the incident agent uses, and its API key. The key is sent once and stored encrypted;
+// the backend never sends it back, so the field only ever shows whether one is saved.
+import { useState } from 'react'
+import { apiRequest } from '../api.js'
+import { Field, Outcome, Row, useAction } from '../ui.jsx'
+
+export default function AgentProvider({ agent, secretsConfigured, onChanged }) {
+ const providers = agent.providers
+ const defaultModel = (id) => providers.find((item) => item.id === id)?.default_model || ''
+ const [provider, setProvider] = useState(agent.provider || providers[0].id)
+ const [model, setModel] = useState(agent.model || defaultModel(provider))
+ const [apiKey, setApiKey] = useState('')
+ const [state, run] = useAction()
+ const [test, runTestAction] = useAction()
+ const [tested, setTested] = useState(null)
+ const chosen = providers.find((item) => item.id === agent.provider)
+ const switching = Boolean(agent.provider) && provider !== agent.provider
+
+ const choose = (id) => { setProvider(id); setModel(defaultModel(id)) }
+ const save = (event) => {
+ event.preventDefault()
+ setTested(null)
+ run(async () => {
+ await apiRequest('/admin/agent/provider', { method: 'PUT', body: JSON.stringify({ provider, model: model.trim() || null, api_key: apiKey.trim() || null }) })
+ setApiKey('')
+ await onChanged()
+ }, 'Saved.')
+ }
+ const removeKey = () => {
+ if (!window.confirm(`Remove the ${chosen.name} API key? The agent stops until a new key is saved.`)) return
+ setTested(null)
+ run(async () => { await apiRequest('/admin/agent/key', { method: 'DELETE' }); await onChanged() }, 'Key removed.')
+ }
+ const runTest = () => { setTested(null); return runTestAction(async () => setTested(await apiRequest('/admin/agent/test', { method: 'POST' }))) }
+
+ const detail = agent.key_configured ? `${chosen.name} · ${agent.model} · key saved`
+ : chosen ? `${chosen.name} has no API key yet. The agent stays off until one is saved.` : 'No provider chosen yet. Pick one and paste its API key to turn the agent on.'
+ return (
+ <>
+ <Row icon="AI" title="Model provider" detail={detail} tone={agent.key_configured ? undefined : 'warn'}>
+ <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key and the model without spending tokens.' : 'Save an API key first.'} onClick={runTest}>{test.busy ? 'Testing…' : 'Test connection'}</button>
+ </Row>
+ {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key and offers {tested.model}.</p>}</div>}
+ <form className="grid items-start gap-3 px-[18px] py-3.5 md:grid-cols-[10rem_minmax(0,1fr)_minmax(0,1.5fr)_auto]" onSubmit={save}>
+ <Field label="Provider">
+ <select className="select h-9 w-full" value={provider} disabled={state.busy} onChange={(event) => choose(event.target.value)}>
+ {providers.map((item) => <option key={item.id} value={item.id}>{item.name}</option>)}
+ </select>
+ </Field>
+ <Field label="Model" hint={`Empty: ${defaultModel(provider)}`}>
+ <input className="field mono h-9 w-full" maxLength={100} value={model} disabled={state.busy} onChange={(event) => setModel(event.target.value)} />
+ </Field>
+ <Field label="API key" hint={!secretsConfigured ? 'Set SECRETS_KEY in .env first: the key is stored encrypted with it.' : switching ? 'Each provider keeps its own key. Leave empty to keep a key saved earlier.' : agent.key_configured ? 'Saved. Paste a new key to replace it.' : 'Stored encrypted; never shown again.'}>
+ <input type="password" className="field mono h-9 w-full" autoComplete="off" spellCheck={false} maxLength={512} value={apiKey} disabled={state.busy || !secretsConfigured}
+ placeholder={agent.key_configured && !switching ? '••••••••' : ''} onChange={(event) => setApiKey(event.target.value)} />
+ </Field>
+ <div className="flex h-9 items-center gap-3 md:mt-[22px]">
+ <button type="submit" className="btn btn-primary h-9" disabled={state.busy}>Save</button>
+ {agent.key_configured && <button type="button" className="link text-xs" disabled={state.busy} onClick={removeKey}>Remove key</button>}
+ </div>
+ <div className="empty:hidden md:col-span-4"><Outcome state={state} /></div>
+ </form>
+ </>
+ )
+}
diff --git a/src/frontend/src/agent/AgentPage.jsx b/src/frontend/src/agent/AgentPage.jsx
index d33c3a5..0cedf94 100644
--- a/src/frontend/src/agent/AgentPage.jsx
+++ b/src/frontend/src/agent/AgentPage.jsx
@@ -113,7 +113,7 @@ function AgentLine({ incident, canAct, onChanged, compact = false }) {
useEffect(() => { if (arrived) setWaiting(null) }, [arrived])
const resolved = incident.status === 'resolved'
const canAsk = canAct && agent.key_configured && !agent.budget_paused
- const why = !agent.key_configured ? 'Set OPENAI_API_KEY in .env for the agent to diagnose incidents.' : agent.budget_paused ? 'This month\'s budget for the agent is used up.' : !canAct ? 'Only owners and admins can ask for a diagnosis.' : undefined
+ const why = !agent.key_configured ? 'The agent has no API key yet. An admin adds one in Admin › Incident agent.' : agent.budget_paused ? 'This month\'s budget for the agent is used up.' : !canAct ? 'Only owners and admins can ask for a diagnosis.' : undefined
const ask = () => run(async () => { await post(`/incidents/${incident.id}/diagnose`); setWaiting({ version, at: Date.now(), late: false }) })
let text
if (slow) text = 'Still waiting for the agent. It is taking longer than usual; the worker may be busy or stopped (Admin › Incident agent › Worker).'
@@ -484,7 +484,7 @@ export default function AgentPage({ user, onLogout }) {
// Why the chat can't be used right now, or null when it can.
const chatOff = !config ? null : !config.chat_enabled ? 'Chat with the agent has been turned off by an admin.'
: config.may_chat === false ? 'Chat with the agent is for admins. An admin can allow it for you in Admin › Users.'
- : config.key_configured === false ? 'The agent has no API key yet. An admin sets OPENAI_API_KEY in .env.'
+ : config.key_configured === false ? 'The agent has no API key yet. An admin adds one in Admin › Incident agent.'
: config.budget_paused ? 'This month\'s budget for the agent is used up. An admin can raise it in Admin › Incident agent.' : null
if (phone) {
diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.py
index e902f4e..39f8eab 100644
--- a/tests/accounts/test_new_pages.py
+++ b/tests/accounts/test_new_pages.py
@@ -342,21 +342,21 @@ def test_a_server_is_renamed_and_keeps_the_name_and_the_install_describes_itself
def test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monkeypatch):
owner = signed_in(app, 1)
- monkeypatch.setenv("OPENAI_API_KEY", "sk-test-secret-value")
- monkeypatch.setattr(main, "check_provider_connection", lambda: {"provider": "OpenAI", "model": "gpt-test"})
+ monkeypatch.setattr(main, "check_provider_connection", lambda db: {"provider": "Anthropic", "model": "claude-test"})
+ saved = owner.put("/admin/agent/provider", json={"provider": "anthropic", "model": "claude-test", "api_key": "sk-ant-test-secret-value"})
system = owner.get("/admin/system")
- assert system.json()["agent"]["key_configured"] is True and system.json()["agent"]["provider"] == "OpenAI"
+ assert system.json()["agent"]["key_configured"] is True and system.json()["agent"]["provider"] == "anthropic"
result = owner.post("/admin/agent/test")
- assert result.json() == {"ok": True, "provider": "OpenAI", "model": "gpt-test"}
- assert "sk-test-secret-value" not in system.text + result.text
+ assert result.json() == {"ok": True, "provider": "Anthropic", "model": "claude-test"}
+ assert "sk-ant-test-secret-value" not in saved.text + system.text + result.text
assert signed_in(app, 3).post("/admin/agent/test").status_code == 403
def test_a_failed_agent_connection_test_explains_itself_without_the_key(app, monkeypatch):
owner = signed_in(app, 1)
- def rejected():
+ def rejected(db):
raise RuntimeError("Error code: 401 - Incorrect API key provided: sk-test-secret-value")
monkeypatch.setattr(main, "check_provider_connection", rejected)
@@ -365,6 +365,36 @@ def test_a_failed_agent_connection_test_explains_itself_without_the_key(app, mon
assert "sk-test-secret-value" not in response.text
[email protected]("body", [{"provider": "somewhere-else"}, {"provider": "openai", "model": "gpt test; drop"}, {"provider": "openai", "api_key": "has spaces in it"},
+ {"provider": "openai", "api_key": "short"}, {"provider": "openai", "api_key": "x" * 513}, {}])
+def test_the_agent_provider_refuses_odd_input(app, body):
+ assert signed_in(app, 1).put("/admin/agent/provider", json=body).status_code == 422
+
+
+def test_only_admins_choose_the_agent_provider_and_remove_its_key(app):
+ viewer = signed_in(app, 3)
+ assert viewer.put("/admin/agent/provider", json={"provider": "openai", "api_key": "sk-test-not-real"}).status_code == 403
+ assert viewer.delete("/admin/agent/key").status_code == 403
+
+
+def test_without_secrets_key_no_api_key_is_stored(app, monkeypatch):
+ monkeypatch.delenv("SECRETS_KEY")
+ response = signed_in(app, 1).put("/admin/agent/provider", json={"provider": "openai", "api_key": "sk-test-not-real"})
+
+ assert response.status_code == 409 and "SECRETS_KEY" in response.json()["detail"]
+ assert app.db.get(main.AppSetting, "agent_api_key_openai") is None
+
+
+def test_saving_and_removing_the_agent_key_is_audited_without_the_key(app):
+ owner = signed_in(app, 1)
+ saved = owner.put("/admin/agent/provider", json={"provider": "gemini", "api_key": "AIza-test-secret-value"}).json()
+ assert saved["provider"] == "gemini" and saved["model"] == "gemini-3.1-flash-lite" and saved["key_configured"] is True
+
+ assert owner.delete("/admin/agent/key").json()["key_configured"] is False
+ logged = " ".join(entry.message for entry in app.db.query(main.AuditLog))
+ assert "Google Gemini" in logged and "removed" in logged and "AIza-test-secret-value" not in logged
+
+
# --- the agent's spending (#112) ---------------------------------------------------------------------------------
from src.agent.providers import Completion
@@ -372,13 +402,14 @@ from src.backend.models import AgentRun
def _fake_agent(monkeypatch, tokens=(1000, 200), fail=None):
- def fake(user_id, conversation_id, message, command_logger=None, usage_logger=None):
+ def fake(user_id, conversation_id, message, provider, command_logger=None, usage_logger=None):
if fail:
raise fail
# A dated model name, as the API reports it: it must still use the price set for "gpt-test".
usage_logger(Completion(text="{}", model="gpt-test-2026-01-01", input_tokens=tokens[0], output_tokens=tokens[1]), 1500)
return "Here is the answer."
monkeypatch.setattr(main, "send_agent_message", fake)
+ monkeypatch.setattr(main, "get_provider", lambda db: object())
monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)
@@ -497,7 +528,7 @@ def test_asking_for_a_new_diagnosis_queues_a_job(app, monkeypatch):
owner = signed_in(app, 1)
assert owner.post("/incidents/42/diagnose").status_code == 409 # no key configured
- monkeypatch.setenv("OPENAI_API_KEY", "sk-test-not-real")
+ assert owner.put("/admin/agent/provider", json={"provider": "openai", "api_key": "sk-test-not-real"}).status_code == 200
assert owner.post("/incidents/42/diagnose").json() == {"queued": True}
assert owner.post("/incidents/42/diagnose").status_code == 200 # asking again is allowed
queued = [json.loads(raw) for raw in app.redis.lrange(jobs.QUEUE, 0, -1)]
@@ -562,6 +593,7 @@ def test_hidden_diagnoses_stay_hidden_from_users_in_the_page_the_overview_and_no
def test_chat_is_for_admins_unless_allowed_per_user(app, monkeypatch):
monkeypatch.setattr(main, "send_agent_message", lambda *args, **kwargs: "ok")
+ monkeypatch.setattr(main, "get_provider", lambda db: object())
monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)
owner, viewer = signed_in(app, 1), signed_in(app, 3)
viewer.post("/conversations", json={"conversation_id": "v1"})
diff --git a/tests/assistant/test_agent_diagnosis.py b/tests/assistant/test_agent_diagnosis.py
index eec9217..b41baf7 100644
--- a/tests/assistant/test_agent_diagnosis.py
+++ b/tests/assistant/test_agent_diagnosis.py
@@ -163,7 +163,7 @@ class FakeProvider:
self.answers = list(answers)
self.calls = []
- def complete(self, messages, schema, schema_name):
+ def complete(self, messages, schema, schema_name, effort="medium"):
self.calls.append(messages)
reply = self.answers.pop(0)
if isinstance(reply, Exception):
@@ -215,8 +215,8 @@ def test_a_provider_error_is_counted_and_raised_so_the_job_is_retried(db):
def test_the_worker_diagnoses_and_reports_back(db, monkeypatch):
monkeypatch.setattr(worker, "SessionLocal", lambda: db)
monkeypatch.setattr(db, "close", lambda: None)
- monkeypatch.setattr(worker, "configured_key", lambda: "sk-test")
- monkeypatch.setattr(worker, "get_provider", lambda: FakeProvider(valid_answer()))
+ monkeypatch.setattr(worker, "key_configured", lambda db: True)
+ monkeypatch.setattr(worker, "get_provider", lambda db: FakeProvider(valid_answer()))
assert worker.handle_incident({"incident_id": 42, "type": "incident.opened"}) == ("diagnosed", "high confidence · version 1")
@@ -224,10 +224,10 @@ def test_the_worker_diagnoses_and_reports_back(db, monkeypatch):
def test_without_a_key_the_worker_leaves_the_rule_based_diagnosis(db, monkeypatch):
monkeypatch.setattr(worker, "SessionLocal", lambda: db)
monkeypatch.setattr(db, "close", lambda: None)
- monkeypatch.setattr(worker, "configured_key", lambda: "")
+ monkeypatch.setattr(worker, "key_configured", lambda db: False)
status, note = worker.handle_incident({"incident_id": 42, "type": "incident.opened"})
- assert status == "skipped" and "OPENAI_API_KEY" in note
+ assert status == "skipped" and "no API key" in note
def test_with_automatic_diagnosis_off_only_a_request_is_diagnosed(db, monkeypatch):
@@ -236,8 +236,8 @@ def test_with_automatic_diagnosis_off_only_a_request_is_diagnosed(db, monkeypatc
db.commit()
monkeypatch.setattr(worker, "SessionLocal", lambda: db)
monkeypatch.setattr(db, "close", lambda: None)
- monkeypatch.setattr(worker, "configured_key", lambda: "sk-test")
- monkeypatch.setattr(worker, "get_provider", lambda: FakeProvider(valid_answer()))
+ monkeypatch.setattr(worker, "key_configured", lambda db: True)
+ monkeypatch.setattr(worker, "get_provider", lambda db: FakeProvider(valid_answer()))
assert worker.handle_incident({"incident_id": 42, "type": "incident.opened"})[0] == "skipped"
assert worker.handle_incident({"incident_id": 42, "type": "incident.requested"})[0] == "diagnosed"
diff --git a/tests/assistant/test_agent_errors.py b/tests/assistant/test_agent_errors.py
index f7a8f04..2cbad8e 100644
--- a/tests/assistant/test_agent_errors.py
+++ b/tests/assistant/test_agent_errors.py
@@ -1,3 +1,4 @@
+from src.agent.providers import NoApiKey
from src.backend.main import agent_error_message
@@ -23,8 +24,8 @@ def test_openai_keys_are_redacted_too():
def test_a_missing_key_says_where_to_set_it():
- assert "OPENAI_API_KEY" in agent_error_message(RuntimeError("OPENAI_API_KEY is not configured."))
+ assert "Admin › Incident agent" in agent_error_message(NoApiKey("The agent has no API key."))
-def test_an_unknown_model_points_at_agent_model():
- assert "AGENT_MODEL" in agent_error_message(Exception("Error code: 404 - The model `gpt-nope` does not exist"))
+def test_an_unknown_model_points_at_the_model_setting():
+ assert "Check the model in Admin" in agent_error_message(Exception("Error code: 404 - The model `gpt-nope` does not exist"))
diff --git a/tests/assistant/test_agent_providers.py b/tests/assistant/test_agent_providers.py
new file mode 100644
index 0000000..05f8032
--- /dev/null
+++ b/tests/assistant/test_agent_providers.py
@@ -0,0 +1,114 @@
+"""The agent's providers (Claude, OpenAI, Gemini) with fake clients, and how the chosen provider and its encrypted
+API key are stored. No test reaches a real API."""
+from types import SimpleNamespace
+
+import pytest
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+from sqlalchemy.pool import StaticPool
+
+from src.agent import chat as agent_service, providers
+from src.agent.providers import Completion, NoApiKey
+from src.agent.providers.claude import ClaudeProvider
+from src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider
+from src.backend import secrets_box
+from src.backend.models import AppSetting, Base
+
+
+def db(monkeypatch):
+ monkeypatch.setenv("SECRETS_KEY", secrets_box.generate_key())
+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
+ Base.metadata.create_all(engine)
+ session = sessionmaker(bind=engine)()
+ yield session
+ session.close()
+
+
+# --- Claude ------------------------------------------------------------------------------------------------------
+
+class FakeMessages:
+ def __init__(self, text='{"message":"All good.","action":null}', stop_reason="end_turn"):
+ self.calls, self.text, self.stop_reason = [], text, stop_reason
+
+ def create(self, **kwargs):
+ self.calls.append(kwargs)
+ usage = SimpleNamespace(input_tokens=100, output_tokens=8, cache_creation_input_tokens=20, cache_read_input_tokens=5)
+ content = [SimpleNamespace(type="thinking", thinking=""), SimpleNamespace(type="text", text=self.text)]
+ return SimpleNamespace(content=content, stop_reason=self.stop_reason, model="claude-test", usage=usage)
+
+
+def claude(messages):
+ client = SimpleNamespace(messages=messages, models=SimpleNamespace(retrieve=lambda model: SimpleNamespace(id=model)))
+ return ClaudeProvider("sk-ant-test", "claude-test", client=client)
+
+
+def test_claude_gets_the_system_prompt_apart_and_must_answer_in_the_schema():
+ messages = FakeMessages()
+ turns = [{"role": "system", "content": "You are the agent."}, {"role": "user", "content": "hi"}, {"role": "assistant", "content": "{}"}, {"role": "user", "content": "again"}]
+
+ result = claude(messages).complete(turns, agent_service.REPLY_SCHEMA, "agent_reply", effort="low")
+
+ assert result == Completion(text='{"message":"All good.","action":null}', model="claude-test", input_tokens=125, output_tokens=8)
+ sent = messages.calls[0]
+ assert sent["system"] == "You are the agent." and [turn["role"] for turn in sent["messages"]] == ["user", "assistant", "user"]
+ assert sent["output_config"] == {"effort": "low", "format": {"type": "json_schema", "schema": agent_service.REPLY_SCHEMA}}
+ assert sent["thinking"] == {"type": "adaptive"} and sent["cache_control"] == {"type": "ephemeral"}
+
+
[email protected]("stop_reason, error", [("refusal", "refused"), ("max_tokens", "cut off")])
+def test_a_refused_or_cut_off_claude_answer_is_an_error(stop_reason, error):
+ with pytest.raises(ValueError, match=error):
+ claude(FakeMessages(text="{", stop_reason=stop_reason)).complete([{"role": "user", "content": "hi"}], agent_service.REPLY_SCHEMA, "agent_reply")
+
+
+def test_claude_test_connection_spends_no_tokens():
+ messages = FakeMessages()
+
+ assert claude(messages).check() == {"provider": "Anthropic", "model": "claude-test"}
+ assert messages.calls == []
+
+
+# --- which provider, and its key ---------------------------------------------------------------------------------
+
+def test_without_a_choice_the_agent_has_no_provider(db):
+ assert providers.current(db).provider is None and not providers.key_configured(db)
+ with pytest.raises(NoApiKey):
+ providers.get_provider(db)
+
+
[email protected]("provider, cls, model", [("anthropic", ClaudeProvider, "claude-haiku-5-5"), ("openai", OpenAIProvider, "gpt-6-luna"), ("gemini", OpenAIProvider, "gemini-3.1-flash-lite")])
+def test_each_provider_starts_with_its_own_default_model(db, provider, cls, model):
+ providers.save(db, provider, None, "test-key-123")
+
+ built = providers.get_provider(db)
+ assert isinstance(built, cls) and built.model == model
+
+
+def test_gemini_goes_to_googles_address(db):
+ providers.save(db, "gemini", None, "AIza-test-key")
+
+ assert str(providers.get_provider(db).client.base_url) == GEMINI_BASE_URL
+
+
+def test_the_key_is_stored_encrypted_and_each_provider_keeps_its_own(db):
+ providers.save(db, "anthropic", "claude-test", "sk-ant-first-key")
+ providers.save(db, "openai", None, "sk-openai-key")
+ assert all("sk-" not in setting.value for setting in db.query(AppSetting))
+
+ providers.save(db, "anthropic", None, None) # no new key: the stored one stays
+ assert providers.current(db).key == "sk-ant-first-key"
+
+
+def test_a_key_that_cant_be_decrypted_counts_as_no_key(db, monkeypatch):
+ providers.save(db, "anthropic", None, "sk-ant-key")
+ monkeypatch.setenv("SECRETS_KEY", secrets_box.generate_key())
+
+ assert not providers.key_configured(db)
+
+
+def test_a_removed_key_turns_the_agent_off(db):
+ providers.save(db, "openai", None, "sk-openai-key")
+ providers.remove_key(db, "openai")
+
+ assert not providers.key_configured(db) and providers.current(db).provider == "openai"
diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.py
index 313c174..6005a8f 100644
--- a/tests/assistant/test_agent_service.py
+++ b/tests/assistant/test_agent_service.py
@@ -15,7 +15,7 @@ class FakeChat:
self.responses = iter(responses)
self.messages = []
- def send_message(self, message):
+ def send_message(self, provider, message):
self.messages.append(message)
return FakeResponse(next(self.responses))
@@ -46,7 +46,7 @@ def test_predefined_health_check_runs_and_is_audited():
service = make_service(chat, checks)
events = []
- result = service.send(1, "conversation", "Check the host.", events.append)
+ result = service.send(1, "conversation", "Check the host.", object(), events.append)
assert result == "Server health is OK."
assert checks.requested == ["server_health"]
@@ -59,7 +59,7 @@ def test_arbitrary_command_is_rejected_before_execution():
service = make_service(chat)
with pytest.raises(ValueError, match="outside the allow-list"):
- service.send(1, "conversation", "Delete everything")
+ service.send(1, "conversation", "Delete everything", object())
# --- the OpenAI provider ----------------------------------------------------------------------------------------
@@ -89,7 +89,7 @@ def fake_client(completions):
def test_openai_provider_asks_for_the_reply_schema_and_reports_tokens():
completions = FakeCompletions()
- provider = OpenAIProvider(model="gpt-test", client=fake_client(completions))
+ provider = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions))
result = provider.complete([{"role": "user", "content": "hi"}], agent_service.REPLY_SCHEMA, "agent_reply")
@@ -102,22 +102,15 @@ def test_openai_provider_asks_for_the_reply_schema_and_reports_tokens():
def test_a_refusal_is_an_error_not_an_empty_reply():
- provider = OpenAIProvider(model="gpt-test", client=fake_client(FakeCompletions(content=None, refusal="I can't help with that.")))
+ provider = OpenAIProvider("sk-test", "gpt-test", client=fake_client(FakeCompletions(content=None, refusal="I can't help with that.")))
with pytest.raises(ValueError, match="refused"):
provider.complete([], agent_service.REPLY_SCHEMA, "agent_reply")
-def test_the_provider_needs_a_key(monkeypatch):
- monkeypatch.delenv("OPENAI_API_KEY", raising=False)
-
- with pytest.raises(RuntimeError, match="OPENAI_API_KEY"):
- OpenAIProvider()
-
-
def test_test_connection_spends_no_tokens():
completions = FakeCompletions()
- provider = OpenAIProvider(model="gpt-test", client=fake_client(completions))
+ provider = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions))
assert provider.check() == {"provider": "OpenAI", "model": "gpt-test"}
assert completions.calls == []
@@ -125,10 +118,10 @@ def test_test_connection_spends_no_tokens():
def test_a_conversation_sends_the_system_prompt_and_its_history():
completions = FakeCompletions()
- chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))
+ provider, chat = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions)), agent_service.Chat()
- chat.send_message("first")
- chat.send_message("second")
+ chat.send_message(provider, "first")
+ chat.send_message(provider, "second")
sent = completions.calls[-1]["messages"]
assert sent[0]["role"] == "system" and [message["content"] for message in sent[1:] if message["role"] == "user"] == ["first", "second"]
@@ -138,19 +131,19 @@ def test_a_conversation_sends_the_system_prompt_and_its_history():
def test_only_recent_turns_go_back_to_the_model(monkeypatch):
monkeypatch.setattr(agent_service, "MAX_HISTORY_MESSAGES", 4)
completions = FakeCompletions()
- chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))
+ provider, chat = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions)), agent_service.Chat()
for number in range(6):
- chat.send_message(f"message {number}")
+ chat.send_message(provider, f"message {number}")
assert len(completions.calls[-1]["messages"]) == 1 + 4
def test_a_failed_turn_is_not_kept_in_the_conversation():
completions = FakeCompletions(fail=TimeoutError("timed out"))
- chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))
+ provider, chat = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions)), agent_service.Chat()
with pytest.raises(TimeoutError):
- chat.send_message("hello")
+ chat.send_message(provider, "hello")
assert chat.history == []
diff --git a/tests/assistant/test_agent_watch.py b/tests/assistant/test_agent_watch.py
index 202b440..b7689b0 100644
--- a/tests/assistant/test_agent_watch.py
+++ b/tests/assistant/test_agent_watch.py
@@ -69,8 +69,8 @@ def test_the_worker_reuses_before_asking_the_model(db, monkeypatch):
add_report(db, 31, created_at=NOW - timedelta(hours=1))
monkeypatch.setattr(worker, "SessionLocal", lambda: db)
monkeypatch.setattr(db, "close", lambda: None)
- monkeypatch.setattr(worker, "configured_key", lambda: "sk-test")
- monkeypatch.setattr(worker, "get_provider", lambda: pytest.fail("the model must not be asked"))
+ monkeypatch.setattr(worker, "key_configured", lambda db: True)
+ monkeypatch.setattr(worker, "get_provider", lambda db: pytest.fail("the model must not be asked"))
assert worker.handle_incident({"incident_id": 42, "type": "incident.opened"}) == ("reused", "Same diagnosis as INC-0031; no tokens spent")
@@ -80,9 +80,9 @@ def test_getting_worse_or_asking_again_is_never_reused(db, monkeypatch):
asked = []
monkeypatch.setattr(worker, "SessionLocal", lambda: db)
monkeypatch.setattr(db, "close", lambda: None)
- monkeypatch.setattr(worker, "configured_key", lambda: "sk-test")
+ monkeypatch.setattr(worker, "key_configured", lambda db: True)
monkeypatch.setattr(worker.diagnosis, "diagnose", lambda db, incident, provider, trigger: asked.append(trigger) or AgentReport(status="failed", error="x"))
- monkeypatch.setattr(worker, "get_provider", lambda: object())
+ monkeypatch.setattr(worker, "get_provider", lambda db: object())
worker.handle_incident({"incident_id": 42, "type": "incident.worsened"})
worker.handle_incident({"incident_id": 42, "type": "incident.requested"})
diff --git a/tests/conftest.py b/tests/conftest.py
index 4195a5d..8d4c9fe 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -13,10 +13,3 @@ from src.backend import main
def _no_ingest_spacing(monkeypatch):
"""Tests send check-ins back to back, so the minimum spacing between check-ins is off unless a test turns it on."""
monkeypatch.setattr(main, "INGEST_MIN_SPACING_RATIO", 0)
-
-
[email protected](autouse=True)
-def _no_real_model_calls(monkeypatch):
- """The backend reads .env, which may hold a real OPENAI_API_KEY. No test may reach the provider with it: tests
- that need a key set a fake one and replace the provider."""
- monkeypatch.delenv("OPENAI_API_KEY", raising=False)
diff --git a/tests/security/test_second_review.py b/tests/security/test_second_review.py
index b7f455e..f917d08 100644
--- a/tests/security/test_second_review.py
+++ b/tests/security/test_second_review.py
@@ -45,6 +45,7 @@ def app(monkeypatch):
monkeypatch.setattr(main, "SESSION_COOKIE_SECURE", False)
monkeypatch.setenv("SECRETS_KEY", Fernet.generate_key().decode())
monkeypatch.setattr(main, "send_agent_message", lambda *args, **kwargs: "ok")
+ monkeypatch.setattr(main, "get_provider", lambda db: object())
monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)
monkeypatch.setattr(main.agent_settings, "may_chat", lambda db, user: True) # chat access has its own tests
client = TestClient(main.app, raise_server_exceptions=False)