Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

feat(agent): usage tracking, monthly budget and daily chat limit

karloks2005922c358863c7
diff --git a/ODSTUPANJA.md b/ODSTUPANJA.md
index 30bbe74..1a2dd26 100644
--- a/ODSTUPANJA.md
+++ b/ODSTUPANJA.md
@@ -127,7 +127,7 @@ su anotacije dizajna i namjerno nisu ugrađeni.
| Alert rules: stupac Servers s izborom servera | Uvijek „All" | Pravila vrijede za sve servere | – |
| Alert rules: „New rule" | Gumb je onemogućen | Vlastita pravila ne postoje | – |
| Alert rules: bilješka zašto je pravilo isključeno | Samo broj isključenih pravila | Razlog se ne upisuje | – |
-| Incident agent: „Anthropic API · key ending 4f2a", „Change" | Google Gemini, model iz `.env`; „Change" je onemogućen | Pružatelj i ključ se postavljaju u `.env` | #91 |
+| Incident agent: „Anthropic API · key ending 4f2a", „Change" | OpenAI, ključ i model iz `.env`; „Change" je onemogućen, „Test connection" provjerava ključ | Ključ se zasad postavlja u `.env`; spremanje u Adminu je dio #109 | #109 |
| „Diagnose new incidents automatically" prekidač | Uvijek uključeno i onemogućeno | Dijagnoza je lokalna i ne može se isključiti | #93 |
| „Read container logs" s iznimkama | Onemogućeno | Proxy ne čita logove | #100 |
| „Redact IPs and hostnames" | Onemogućeno | Nije napravljeno | – |
diff --git a/README.md b/README.md
index cfadfe3..9cc6dab 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@ One machine runs the dashboard; every other machine runs a small collector that
reports host and Docker container metrics back to it. It works the same whether
the machines share a LAN or are spread across sites and VPSes.
-It also includes **VertexAI**, an optional monitoring-only AI assistant (Gemini)
+It also includes an optional, monitoring-only incident agent (OpenAI API, set `OPENAI_API_KEY`)
that can run a fixed set of read-only health checks and never executes commands.
It's main purpose is to report errors, diagnose errors and propose optimal fixes.
diff --git a/SECURITY.md b/SECURITY.md
index d9215e6..cdca572 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -372,7 +372,7 @@ The collector's network checks are passive or opt-in, and add no privileges:
a session for any user, administrator included, with no password or two-factor.
- **Limits on what a signed-in user can run up.** Messages to the assistant (30 an hour per user), new
conversations (30 an hour, 100 in total) and audit-log entries (120 an hour) are limited, since each can cost
- the administrator's Gemini quota or database space. Setting up two-factor from a normal session needs the
+ the administrator's OpenAI quota or database space. Setting up two-factor from a normal session needs the
password again, and every try counts toward the lockout, the right password included. A limited
(set-up-only) session lasts 30 minutes, not a day.
- **Sessions are stored hashed.** Redis holds only the SHA-256 of the cookie value, in the session key, the
diff --git a/TESTS.md b/TESTS.md
index fd2f135..8a840f3 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -146,8 +146,8 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
|---|---|---|
| [`test_account_security.py`](#test_account_securitypy) | 136 | Two-factor sign-in, recovery codes, encrypted secrets, sessions you can review and end, password change, and the password check before turning two-factor on. |
| [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |
-| [`test_agent_errors.py`](#test_agent_errorspy) | 2 | Helpful, non-leaking error messages from the AI assistant. |
-| [`test_agent_service.py`](#test_agent_servicepy) | 2 | The assistant can only run the fixed read-only checks. |
+| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |
+| [`test_agent_service.py`](#test_agent_servicepy) | 9 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. |
| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |
| [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. |
| [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |
@@ -299,6 +299,9 @@ shouldn't, or an admin might be locked out.
|---|---|
| `test_overloaded_provider_error_is_explained_to_the_user` | When the AI provider reports it is overloaded, the user gets a readable explanation. |
| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the chat or logs; it is replaced with `[redacted]`. |
+| `test_openai_keys_are_redacted_too` | An OpenAI key (`sk-…`) in an error is redacted, and a rejected key is explained as such. |
+| `test_a_missing_key_says_where_to_set_it` | Without a key the chat says to set `OPENAI_API_KEY`. |
+| `test_an_unknown_model_points_at_agent_model` | A model the key can't use points the admin at `AGENT_MODEL`. |
### test_agent_service.py
@@ -306,6 +309,13 @@ shouldn't, or an admin might be locked out.
|---|---|
| `test_predefined_health_check_runs_and_is_audited` | When the assistant asks for one of the fixed checks (such as `server_health`), the check runs, the result is fed back to it, and both the request and the result are written to the audit log. |
| `test_arbitrary_command_is_rejected_before_execution` | If the assistant ever asks for anything outside the allow-list (such as `rm -rf /`), it is refused before anything runs. |
+| `test_openai_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests use Structured Outputs in strict mode, the schema only allows the fixed checks, and token usage is read from the response. |
+| `test_a_refusal_is_an_error_not_an_empty_reply` | When the model refuses, the chat gets an error instead of an empty answer. |
+| `test_the_provider_needs_a_key` | Without `OPENAI_API_KEY` the provider is not created. |
+| `test_test_connection_spends_no_tokens` | Admin's Test connection only looks the model up; it sends no prompt. |
+| `test_a_conversation_sends_the_system_prompt_and_its_history` | Each request carries the system prompt and the conversation so far, in order. |
+| `test_only_recent_turns_go_back_to_the_model` | Long conversations send only the most recent turns. |
+| `test_a_failed_turn_is_not_kept_in_the_conversation` | A message that failed (timeout, provider error) is not remembered as part of the conversation. |
### test_alerts.py
diff --git a/compose.yml b/compose.yml
index e0d7f0d..c454399 100644
--- a/compose.yml
+++ b/compose.yml
@@ -47,8 +47,8 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
- GEMINI_API_KEY: ${GEMINI_API_KEY:-}
- AGENT_MODEL: ${AGENT_MODEL:-models/gemini-3.5-flash}
+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
# Set to true when the dashboard is served over HTTPS.
SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-false}
ALLOW_REGISTRATION: ${ALLOW_REGISTRATION:-false}
diff --git a/docker-compose.server.yml b/docker-compose.server.yml
index 7de7a5b..3995d54 100644
--- a/docker-compose.server.yml
+++ b/docker-compose.server.yml
@@ -52,8 +52,8 @@ services:
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}
REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0
- GEMINI_API_KEY: ${GEMINI_API_KEY:-}
- AGENT_MODEL: ${AGENT_MODEL:-models/gemini-3.5-flash}
+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}
# Served over HTTPS via Traefik, so the login cookie is HTTPS-only.
SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-true}
# Only the first account can register; set to true to let others sign up.
diff --git a/docs/api.md b/docs/api.md
index 962ccc3..9b5e232 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -27,8 +27,10 @@ All paths are relative to `/api` when going through the frontend/nginx.
| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). |
| `GET /monitoring/install-info` | admin | The collector address and images the panel uses. |
| `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |
-| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds. |
-| `GET /agent/config` | session | Whether chat with the assistant is enabled. |
+| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds; the agent's model prices, monthly budget and daily chat limit. |
+| `GET /agent/config` | session | Whether chat with the agent can be used: switched on, a key configured, and the monthly budget not used up. |
+| `GET /admin/agent/usage` | admin | This month's model calls, tokens and estimated cost against the budget, per model and per day, with the prices and limits. |
+| `POST /admin/agent/test` | admin | Checks the OpenAI key and that the configured model exists, without spending tokens. |
| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |
| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |
| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |
diff --git a/docs/assistant.md b/docs/assistant.md
index 16a2e94..41c398f 100644
--- a/docs/assistant.md
+++ b/docs/assistant.md
@@ -1,6 +1,22 @@
[← Documentation](README.md)
-# Homie assistant (monitoring-only)
+# Incident agent chat (monitoring-only)
+
+The chat runs on the OpenAI API. Set `OPENAI_API_KEY` in `.env` (and optionally `AGENT_MODEL`, default
+`gpt-5-mini`), then restart the backend. **Admin → Incident agent → Test connection** confirms the key works and
+the model exists without spending tokens. Without a key the chat is off; incidents, notifications and the
+rule-based diagnosis work the same.
+
+**What it costs.** Every call to the model is recorded with its tokens (see `agent_runs` in
+[Database schema](database.md)). In **Admin → Incident agent** you enter each model's price per million tokens (copy it
+from the provider's pricing page; prices aren't fetched automatically), and the page shows the month's estimated cost,
+calls, tokens and cost per model. With a **monthly budget** set, admins get a notification on their channels at 80 %,
+and at 100 % the chat pauses until the 1st of next month (incidents and notifications keep working). Each person may
+also send at most a set number of chat messages in 24 hours (100 by default), on top of 30 an hour.
+
+Every reply is held to a JSON schema with OpenAI's Structured Outputs: a message, and at most one check from
+the fixed list below. The provider sits behind a small interface (`src/agent/providers.py`), so other providers
+can be added later.
The assistant has no code path for running shell commands. It may request
exactly one fixed, read-only check at a time: `server_health`,
diff --git a/docs/configuration.md b/docs/configuration.md
index b711d05..6975e3d 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -11,8 +11,8 @@ Every setting below is listed, with no values, in [`example.env`](../example.env
| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. |
| `POSTGRES_DB` / `POSTGRES_USER` | `infradeck` / `infradeck` | Name of the database and its user. Installs from before the rename can keep `homelab` (see [SETUP_DATABASES.md](../SETUP_DATABASES.md)). |
| `REDIS_PASSWORD` | *(required)* | Redis password. Redis holds the login sessions, so nothing that can merely reach it may use it. Use letters and digits only (`openssl rand -hex 24`) because it is put into a URL. |
-| `GEMINI_API_KEY` | empty | Enables the Homie chat assistant. |
-| `AGENT_MODEL` | `models/gemini-3.5-flash` | Gemini model for the assistant. |
+| `OPENAI_API_KEY` | empty | OpenAI API key for the incident agent's chat. Empty: chat is off; the rule-based diagnosis works without it. **Admin → Incident agent → Test connection** checks it. |
+| `AGENT_MODEL` | `gpt-5-mini` | OpenAI model for the agent. It must support Structured Outputs. |
| `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. |
| `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. |
| `SESSION_TTL_SECONDS` | `86400` | Session lifetime without *Remember me*. |
diff --git a/docs/containers.md b/docs/containers.md
index bb1c20e..36961e4 100644
--- a/docs/containers.md
+++ b/docs/containers.md
@@ -105,7 +105,7 @@ through the gate); on the development stack the local collector talks to the bac
- registering collectors and receiving their check-ins (`/monitoring/metrics/ingest`);
- the alert rules, incidents and notification channels (email/webhook, quiet hours);
- statistics, rollups (summarised history) and retention;
-- the optional Homie assistant, which talks to Gemini when `GEMINI_API_KEY` is set.
+- the optional incident agent chat, which talks to the OpenAI API when `OPENAI_API_KEY` is set.
**How it works.** A FastAPI app run by uvicorn on port 8000. At startup it creates any missing tables and
columns in PostgreSQL, moves old Redis keys to the current prefix, and prints the first-run setup token
diff --git a/docs/database.md b/docs/database.md
index 6048c29..426d227 100644
--- a/docs/database.md
+++ b/docs/database.md
@@ -15,6 +15,12 @@ recovery codes (the encrypted two-factor secret lives on the `users` row), and p
thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises
the `users` table on a fresh volume.
+**The agent's spending.** `agent_runs` has one row per call to the language model: what it was for (`chat`, later
+`diagnosis`), whether it worked, the model, input and output tokens, the estimated cost in USD, how long it took, and
+who asked about which incident. The month's cost and the budget check add these rows up; a call made before its model
+had a price is priced with the current table when it is counted. The prices, the monthly budget and the daily chat
+limit are rows in `app_settings` (`agent_prices`, `agent_monthly_cap_usd`, `agent_daily_chat_limit`).
+
**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in
samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container
per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.
diff --git a/docs/development.md b/docs/development.md
index 1df8782..cfbce47 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -89,7 +89,7 @@ src/
docker_proxy.py read-only Docker API proxy (standard library only)
system_helper.py optional kernel-log / wtmp helper (standard library only)
health_checks.py read-only checks available to the assistant
- service.py, prompts.py Gemini assistant
+ service.py, prompts.py, providers.py incident agent chat (OpenAI)
frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the
charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,
FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the page
diff --git a/example.env b/example.env
index 7b4f8e3..791d57a 100644
--- a/example.env
+++ b/example.env
@@ -220,14 +220,15 @@ SMTP_FROM=
ADMIN_ALERT_WEBHOOK_URL=
# =============================================================================
-# AI assistant (optional)
+# Incident agent (optional)
# =============================================================================
-# (both) Google Gemini API key for the Homie assistant. Empty: chat is off, monitoring works without it.
-# Get one at https://aistudio.google.com/apikey
-GEMINI_API_KEY=
+# (both) OpenAI API key for the incident agent's chat. Empty: chat is off, monitoring and the rule-based
+# diagnosis work without it. Create one at https://platform.openai.com/api-keys
+# Admin → Incident agent → Test connection checks it without spending tokens.
+OPENAI_API_KEY=
-# (both) Gemini model the assistant uses. Default: models/gemini-3.5-flash
+# (both) OpenAI model the agent uses. It must support Structured Outputs. Default: gpt-5-mini
AGENT_MODEL=
# =============================================================================
diff --git a/requirements.txt b/requirements.txt
index 0df689c..a09676d 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -2,7 +2,7 @@ bcrypt
cryptography
email-validator
fastapi
-google-genai
+openai
psycopg[binary]
python-dotenv
redis
diff --git a/src/agent/providers.py b/src/agent/providers.py
new file mode 100644
index 0000000..2c1d19f
--- /dev/null
+++ b/src/agent/providers.py
@@ -0,0 +1,81 @@
+"""The language model behind the agent, behind one small interface so another provider can be added later.
+
+A provider turns a list of chat messages into one reply that matches a JSON schema, and reports how many tokens
+that took. It never sees the database or the servers; the agent decides what goes into the messages.
+"""
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+from typing import Protocol
+
+DEFAULT_MODEL = "gpt-5-mini"
+TIMEOUT_SECONDS = 60
+MAX_RETRIES = 2
+
+
+@dataclass
+class Completion:
+ text: str
+ model: str
+ input_tokens: int = 0
+ output_tokens: int = 0
+
+
+class Provider(Protocol):
+ name: str
+
+ def complete(self, messages: list[dict], schema: dict, schema_name: str) -> Completion:
+ """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. Returns JSON text matching ``schema``."""
+
+ def check(self) -> dict:
+ """Confirms the key works and the model exists, without spending tokens."""
+
+
+def configured_key() -> str:
+ return os.getenv("OPENAI_API_KEY", "").strip()
+
+
+def configured_model() -> str:
+ return os.getenv("AGENT_MODEL", "").strip() or DEFAULT_MODEL
+
+
+class OpenAIProvider:
+ """The OpenAI API through Chat Completions, with Structured Outputs so a reply always parses."""
+
+ name = "OpenAI"
+
+ def __init__(self, api_key: str | None = None, model: str | None = None, client=None):
+ key = api_key or configured_key()
+ if not key and client is None:
+ raise RuntimeError("OPENAI_API_KEY is not configured.")
+ self.model = model or configured_model()
+ if client is None:
+ from openai import OpenAI # imported here so the rest of the app runs without the package
+ client = OpenAI(api_key=key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
+ self.client = client
+
+ def complete(self, messages: list[dict], schema: dict, schema_name: str) -> Completion:
+ response = self.client.chat.completions.create(
+ model=self.model,
+ messages=messages,
+ response_format={"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}},
+ )
+ choice = response.choices[0]
+ if getattr(choice.message, "refusal", None):
+ raise ValueError(f"The model refused to answer: {choice.message.refusal}")
+ usage = getattr(response, "usage", None)
+ return Completion(
+ text=choice.message.content or "",
+ model=getattr(response, "model", self.model),
+ input_tokens=getattr(usage, "prompt_tokens", 0) or 0,
+ output_tokens=getattr(usage, "completion_tokens", 0) or 0,
+ )
+
+ def check(self) -> dict:
+ model = self.client.models.retrieve(self.model)
+ return {"provider": self.name, "model": getattr(model, "id", self.model)}
+
+
+def get_provider() -> Provider:
+ return OpenAIProvider()
diff --git a/src/agent/service.py b/src/agent/service.py
index b79a2d0..e3f8c46 100644
--- a/src/agent/service.py
+++ b/src/agent/service.py
@@ -1,69 +1,99 @@
from __future__ import annotations
import json
-import os
import threading
+import time
from collections.abc import Callable
-from dotenv import load_dotenv
-from google import genai
-from google.genai import types
-
from .health_checks import HealthCheckRunner
from .prompts import SYSTEM_PROMPT
+from .providers import Completion, Provider, get_provider
MAX_CACHED_CHATS = 200
-
-load_dotenv()
+# Only the most recent turns go back to the model, so a long conversation can't grow every request without limit.
+MAX_HISTORY_MESSAGES = 40
+
+# Every reply is a message and at most one predefined check; Structured Outputs makes the model keep to it.
+REPLY_SCHEMA = {
+ "type": "object",
+ "properties": {
+ "message": {"type": "string"},
+ "action": {
+ "anyOf": [
+ {"type": "object", "properties": {"check": {"type": "string", "enum": sorted(HealthCheckRunner.ALLOWED_CHECKS)}}, "required": ["check"], "additionalProperties": False},
+ {"type": "null"},
+ ],
+ },
+ },
+ "required": ["message", "action"],
+ "additionalProperties": False,
+}
+
+
+class Reply:
+ def __init__(self, text: str, completion: Completion | None = None):
+ self.text = text
+ self.completion = completion
+
+
+class Chat:
+ """One conversation's turns, sent in full (up to MAX_HISTORY_MESSAGES) with every new message."""
+
+ def __init__(self, provider: Provider):
+ self.provider = provider
+ self.history: list[dict] = []
+
+ def send_message(self, message: str) -> Reply:
+ self.history.append({"role": "user", "content": message})
+ messages = [{"role": "system", "content": SYSTEM_PROMPT}, *self.history[-MAX_HISTORY_MESSAGES:]]
+ try:
+ completion = self.provider.complete(messages, REPLY_SCHEMA, "agent_reply")
+ except Exception:
+ self.history.pop() # a failed turn is not part of the conversation
+ raise
+ self.history.append({"role": "assistant", "content": completion.text})
+ return Reply(completion.text, completion)
class AgentService:
"""LLM interface restricted to a small, fixed health-check capability set."""
- def __init__(self, health_checks: HealthCheckRunner | None = None):
- api_key = os.getenv("GEMINI_API_KEY")
- if not api_key:
- raise RuntimeError("GEMINI_API_KEY is not configured.")
- self.client = genai.Client(api_key=api_key)
+ def __init__(self, health_checks: HealthCheckRunner | None = None, provider: Provider | None = None):
+ self.provider = provider or get_provider()
self.health_checks = health_checks or HealthCheckRunner()
# Least recently used first (a dict keeps insertion order). A chat only lives in memory so the assistant
# remembers the conversation; the cache is bounded so endless new conversations can't grow it without limit.
self.chats = {}
self.lock = threading.Lock()
+ def _new_chat(self) -> Chat:
+ return Chat(self.provider)
+
def _get_chat(self, user_id: int, conversation_id: str):
chat_key = (user_id, conversation_id)
with self.lock:
if chat_key not in self.chats:
- self.chats[chat_key] = self.client.chats.create(
- model=os.getenv("AGENT_MODEL", "models/gemini-3.5-flash"),
- config=types.GenerateContentConfig(
- system_instruction=SYSTEM_PROMPT,
- temperature=0.2,
- response_mime_type="application/json",
- response_schema={
- "type": "OBJECT",
- "properties": {
- "message": {"type": "STRING"},
- "action": {
- "type": "OBJECT", "nullable": True,
- "properties": {"check": {"type": "STRING"}},
- "required": ["check"],
- },
- },
- "required": ["message", "action"],
- },
- ),
- )
+ self.chats[chat_key] = self._new_chat()
chat = self.chats.pop(chat_key)
self.chats[chat_key] = chat # now the most recently used
while len(self.chats) > MAX_CACHED_CHATS:
del self.chats[next(iter(self.chats))]
return chat
- def send(self, user_id: int, conversation_id: str, message: str, command_logger: Callable[[dict], None] | None = None) -> str:
+ @staticmethod
+ def _ask(chat, message: str, usage_logger: Callable[[Completion, int], None] | None):
+ """One call to the model; ``usage_logger`` gets its tokens and how long it took, in milliseconds."""
+ started = time.monotonic()
+ reply = chat.send_message(message)
+ completion = getattr(reply, "completion", None)
+ if usage_logger and completion is not None:
+ usage_logger(completion, round((time.monotonic() - started) * 1000))
+ return reply
+
+ def send(self, user_id: int, conversation_id: str, message: str, command_logger: Callable[[dict], None] | None = None,
+ usage_logger: Callable[[Completion, int], None] | None = None) -> str:
chat = self._get_chat(user_id, conversation_id)
- response_text, action = self._parse_agent_response(self._response_text(chat.send_message(message)))
+ response_text, action = self._parse_agent_response(self._response_text(self._ask(chat, message, usage_logger)))
if not action:
return response_text
@@ -80,13 +110,13 @@ class AgentService:
"check": check_name, "level": level,
"steps": [f"Completed predefined check '{check_name}' with status {result['status']}."],
})
- final_text, final_action = self._parse_agent_response(self._response_text(chat.send_message("[HEALTH-CHECK-RESULT]\n" + json.dumps(result))))
+ final_text, final_action = self._parse_agent_response(self._response_text(self._ask(chat, "[HEALTH-CHECK-RESULT]\n" + json.dumps(result), usage_logger)))
if final_action:
raise ValueError("Agent must summarize a health-check result without requesting another check.")
return final_text
def delete_conversation(self, user_id: int, conversation_id: str) -> None:
- """Discard the provider's in-memory context after its persisted chat is deleted."""
+ """Discard the in-memory context after its persisted chat is deleted."""
with self.lock:
self.chats.pop((user_id, conversation_id), None)
@@ -118,13 +148,18 @@ class AgentService:
_agent_service = None
-def send_agent_message(user_id: int, conversation_id: str, message: str, command_logger=None) -> str:
+def send_agent_message(user_id: int, conversation_id: str, message: str, command_logger=None, usage_logger=None) -> str:
global _agent_service
if _agent_service is None:
_agent_service = AgentService()
- return _agent_service.send(user_id, conversation_id, message, command_logger=command_logger)
+ return _agent_service.send(user_id, conversation_id, message, command_logger=command_logger, usage_logger=usage_logger)
def delete_agent_conversation(user_id: int, conversation_id: str) -> None:
if _agent_service is not None:
_agent_service.delete_conversation(user_id, conversation_id)
+
+
+def check_provider_connection() -> dict:
+ """For Admin's "Test connection": the key works and the configured model exists. Spends no tokens."""
+ return get_provider().check()
diff --git a/src/backend/agent_usage.py b/src/backend/agent_usage.py
new file mode 100644
index 0000000..14f757d
--- /dev/null
+++ b/src/backend/agent_usage.py
@@ -0,0 +1,176 @@
+"""What the agent's model calls cost: one row per call (AgentRun), prices per model that an admin enters, a monthly
+cap that pauses the agent, and a daily limit on chat messages per person.
+
+Prices are not fetched from the provider: they change, differ per account and aren't in the API, so the admin
+copies them from the provider's pricing page. A call to a model without a price still counts its tokens.
+"""
+from __future__ import annotations
+
+import json
+import re
+from datetime import datetime, timedelta, timezone
+
+from sqlalchemy import func
+
+from .models import AgentRun, AppSetting
+
+PRICES_KEY = "agent_prices"
+CAP_KEY = "agent_monthly_cap_usd"
+DAILY_LIMIT_KEY = "agent_daily_chat_limit"
+DEFAULT_DAILY_CHAT_LIMIT = 100
+MAX_DAILY_CHAT_LIMIT = 10000
+WARN_SHARE = 0.8
+MAX_CAP_USD = 100000
+MAX_PRICE_PER_MILLION = 10000
+MAX_PRICED_MODELS = 30
+MODEL_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]{0,99}")
+
+
+def month_start(now: datetime) -> datetime:
+ return now.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
+
+
+def _setting(db, key: str):
+ setting = db.get(AppSetting, key)
+ return setting.value if setting else None
+
+
+def get_prices(db) -> dict[str, dict[str, float]]:
+ try:
+ return validate_prices(json.loads(_setting(db, PRICES_KEY) or "{}"))
+ except ValueError:
+ return {}
+
+
+def validate_prices(raw) -> dict[str, dict[str, float]]:
+ """{model: {"input": USD per million tokens, "output": USD per million tokens}}. Raises ValueError."""
+ if not isinstance(raw, dict):
+ raise ValueError("Prices must be an object of models.")
+ if len(raw) > MAX_PRICED_MODELS:
+ raise ValueError(f"At most {MAX_PRICED_MODELS} models can have a price.")
+ prices = {}
+ for model, price in raw.items():
+ if not isinstance(model, str) or not MODEL_NAME.fullmatch(model.strip()):
+ raise ValueError("A model name may only contain letters, digits and . _ : / -")
+ if not isinstance(price, dict) or set(price) != {"input", "output"}:
+ raise ValueError(f"The price of {model} needs an input and an output price.")
+ values = {}
+ for side in ("input", "output"):
+ value = price[side]
+ if isinstance(value, bool) or not isinstance(value, (int, float)) or not 0 <= value <= MAX_PRICE_PER_MILLION:
+ raise ValueError(f"The {side} price of {model} must be between 0 and {MAX_PRICE_PER_MILLION} USD per million tokens.")
+ values[side] = float(value)
+ prices[model.strip()] = values
+ return prices
+
+
+def get_cap(db) -> float | None:
+ try:
+ value = float(_setting(db, CAP_KEY) or "")
+ except ValueError:
+ return None
+ return value if value > 0 else None
+
+
+def validate_cap(value) -> float | None:
+ """None or 0 means no cap."""
+ if value is None:
+ return None
+ if isinstance(value, bool) or not isinstance(value, (int, float)) or not 0 <= value <= MAX_CAP_USD:
+ raise ValueError(f"The monthly budget must be between 0 and {MAX_CAP_USD} USD.")
+ return float(value) or None
+
+
+def get_daily_limit(db) -> int:
+ try:
+ return max(1, min(MAX_DAILY_CHAT_LIMIT, int(_setting(db, DAILY_LIMIT_KEY) or DEFAULT_DAILY_CHAT_LIMIT)))
+ except ValueError:
+ return DEFAULT_DAILY_CHAT_LIMIT
+
+
+def validate_daily_limit(value) -> int:
+ if isinstance(value, bool) or not isinstance(value, int) or not 1 <= value <= MAX_DAILY_CHAT_LIMIT:
+ raise ValueError(f"The daily chat limit must be a whole number between 1 and {MAX_DAILY_CHAT_LIMIT}.")
+ return value
+
+
+def price_for(prices: dict, model: str) -> dict | None:
+ """The price of ``model``; a dated name such as gpt-5-mini-2026-01-01 uses the price of gpt-5-mini."""
+ if model in prices:
+ return prices[model]
+ matches = [name for name in prices if model.startswith(f"{name}-")]
+ return prices[max(matches, key=len)] if matches else None
+
+
+def cost_of(prices: dict, model: str, input_tokens: int, output_tokens: int) -> float | None:
+ price = price_for(prices, model)
+ if price is None:
+ return None
+ return (input_tokens * price["input"] + output_tokens * price["output"]) / 1_000_000
+
+
+def record(db, *, kind: str, model: str, status: str = "ok", input_tokens: int = 0, output_tokens: int = 0, duration_ms: int = 0,
+ user_id: int | None = None, conversation_id: str | None = None, incident_id: int | None = None, error: str | None = None) -> AgentRun:
+ run = AgentRun(kind=kind, status=status, model=model[:100], input_tokens=max(0, int(input_tokens or 0)), output_tokens=max(0, int(output_tokens or 0)),
+ cost_usd=cost_of(get_prices(db), model, input_tokens or 0, output_tokens or 0), duration_ms=max(0, int(duration_ms or 0)),
+ user_id=user_id, conversation_id=conversation_id, incident_id=incident_id, error=(error or None) and error[:300])
+ db.add(run)
+ db.commit()
+ return run
+
+
+def month_cost(db, now: datetime | None = None) -> tuple[float, int]:
+ """This month's estimated cost so far, and how many calls had no price to count."""
+ now = now or datetime.now(timezone.utc)
+ prices = get_prices(db)
+ since = AgentRun.created_at >= month_start(now)
+ total = float(db.query(func.coalesce(func.sum(AgentRun.cost_usd), 0)).filter(since, AgentRun.cost_usd.isnot(None)).scalar() or 0)
+ unpriced = 0
+ rows = db.query(AgentRun.model, func.sum(AgentRun.input_tokens), func.sum(AgentRun.output_tokens), func.count(AgentRun.id)) \
+ .filter(since, AgentRun.cost_usd.is_(None)).group_by(AgentRun.model).all()
+ for model, input_tokens, output_tokens, count in rows:
+ later = cost_of(prices, model, input_tokens or 0, output_tokens or 0)
+ if later is None:
+ unpriced += count if (input_tokens or output_tokens) else 0
+ else:
+ total += later
+ return total, unpriced
+
+
+def budget(db, now: datetime | None = None) -> dict:
+ """{"cost", "cap", "share", "state"}: state is ok, warn (from 80 % of the cap) or paused (the cap is reached)."""
+ cost, unpriced = month_cost(db, now)
+ cap = get_cap(db)
+ share = cost / cap if cap else None
+ state = "paused" if share is not None and share >= 1 else "warn" if share is not None and share >= WARN_SHARE else "ok"
+ return {"cost": round(cost, 6), "cap": cap, "share": share, "state": state, "unpriced_runs": unpriced}
+
+
+def month_summary(db, now: datetime | None = None) -> dict:
+ """Admin's usage card: totals, per model, and cost per day of this month."""
+ now = now or datetime.now(timezone.utc)
+ start = month_start(now)
+ prices = get_prices(db)
+ runs = db.query(AgentRun).filter(AgentRun.created_at >= start).all()
+ by_model: dict[str, dict] = {}
+ days: dict[str, float] = {}
+ totals = {"calls": 0, "errors": 0, "chat_calls": 0, "diagnoses": 0, "input_tokens": 0, "output_tokens": 0}
+ for run in runs:
+ cost = run.cost_usd if run.cost_usd is not None else cost_of(prices, run.model, run.input_tokens, run.output_tokens)
+ totals["calls"] += 1
+ totals["errors"] += run.status != "ok"
+ totals["chat_calls"] += run.kind == "chat"
+ totals["diagnoses"] += run.kind == "diagnosis" and run.status == "ok"
+ totals["input_tokens"] += run.input_tokens
+ totals["output_tokens"] += run.output_tokens
+ entry = by_model.setdefault(run.model, {"model": run.model, "calls": 0, "input_tokens": 0, "output_tokens": 0, "cost": 0.0, "priced": price_for(prices, run.model) is not None})
+ entry["calls"] += 1
+ entry["input_tokens"] += run.input_tokens
+ entry["output_tokens"] += run.output_tokens
+ entry["cost"] += cost or 0
+ day = run.created_at.astimezone(timezone.utc).date().isoformat()
+ days[day] = days.get(day, 0) + (cost or 0)
+ span = (now.date() - start.date()).days + 1
+ daily = [{"day": (start + timedelta(days=offset)).date().isoformat(), "cost": round(days.get((start + timedelta(days=offset)).date().isoformat(), 0), 6)} for offset in range(span)]
+ return {"month": start.date().isoformat()[:7], **totals, **budget(db, now), "by_model": sorted(by_model.values(), key=lambda entry: -entry["cost"]), "daily": daily,
+ "prices": prices, "daily_chat_limit": get_daily_limit(db), "warn_share": WARN_SHARE}
diff --git a/src/backend/config.py b/src/backend/config.py
index 5d3816d..8a2e637 100644
--- a/src/backend/config.py
+++ b/src/backend/config.py
@@ -19,7 +19,8 @@ DATABASE_URL = os.getenv(
)
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
SESSION_TTL_SECONDS = int(os.getenv("SESSION_TTL_SECONDS", "86400"))
-AGENT_MODEL = os.getenv("AGENT_MODEL", "models/gemini-3.5-flash")
+# The OpenAI model the agent uses (see src/agent/providers.py).
+AGENT_MODEL = os.getenv("AGENT_MODEL", "").strip() or "gpt-5-mini"
SESSION_COOKIE_NAME = os.getenv("SESSION_COOKIE_NAME", "homie_session")
SESSION_COOKIE_SECURE = os.getenv("SESSION_COOKIE_SECURE", "false").lower() == "true"
# When false (the default), only the first account can be created, so a fresh install can still be set up and
diff --git a/src/backend/main.py b/src/backend/main.py
index 64883be..371614c 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -27,7 +27,7 @@ from sqlalchemy import case, func, inspect, text
from .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alerts
import contextvars
-from . import auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totp
+from . import agent_usage, auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totp
from .auth import (
STAGE_ENROLLMENT, STAGE_FULL, count_pending_attempt, create_pending_two_factor, create_session, delete_pending_two_factor, delete_session, get_recent_logs,
get_session, get_user_from_session, hash_password, issue_recovery_codes, list_user_sessions, pending_two_factor_remember, read_pending_two_factor, session_lifetime, recovery_codes_left, record_audit_log,
@@ -64,7 +64,8 @@ from .config import (
)
from .rules import HELPER_EVENT_KINDS, DEFAULT_THRESHOLDS, HOST_RULE_PREFIX, container_rules, host_rules, validate_thresholds, detect_system_events, system_rules, SYSTEM_RULE_PREFIX, validate_mount_thresholds, validate_helper_events, REBOOT_DETAILS, REBOOT_KEY_PREFIX, detect_container_events, memory_trend_rules, CONTAINER_EVENT_KINDS, MAX_CONTAINER_EVENTS_PER_CHECKIN, MAX_CONTAINER_EVENTS_PER_HOUR, TREND_RULE_PREFIX, NETWORK_EVENT_KINDS, NETWORK_RULE_PREFIX, MAX_NETWORK_EVENTS_PER_CHECKIN, MAX_NETWORK_EVENTS_PER_HOUR, INTERNET_STATUSES, LISTENER_SCOPES, LISTENER_PROTOCOLS, MAX_LISTENERS, FIREWALL_LABELS, detect_listener_changes, detect_firewall_changes, network_rules, validate_listeners, validate_firewall
from .rate_limit import enforce_ip_limit, enforce_limit, enforce_login_limit, enforce_register_limit, refund_attempt, reset_limits
-from ..agent.service import delete_agent_conversation, send_agent_message
+from ..agent.providers import configured_key
+from ..agent.service import check_provider_connection, delete_agent_conversation, send_agent_message
class RegisterRequest(BaseModel):
@@ -187,6 +188,11 @@ class AdminSettingsRequest(BaseModel):
disabled_rules: list[str] | None = Field(default=None, max_length=100)
# What this install calls itself on the sign-in page and in the sidebar; an empty string goes back to the default.
instance_name: str | None = Field(default=None, max_length=200)
+ # The agent's spending: USD per million tokens per model (replacing the whole table), a monthly budget in USD
+ # (0 removes it) and how many chat messages one person may send in 24 hours.
+ agent_prices: dict | None = None
+ agent_monthly_cap_usd: float | None = None
+ agent_daily_chat_limit: int | None = None
class MetricsIngestRequest(BaseModel):
@@ -200,21 +206,27 @@ def agent_error_message(exc: Exception) -> str:
"""Turn provider failures into useful, non-sensitive chat text."""
raw_message = str(exc).strip()
normalized = raw_message.lower()
- if any(marker in normalized for marker in ("resourceexhausted", "overloaded", "capacity", " 503", "status 503")):
- prefix = "Gemini is currently overloaded or unavailable. Please try again shortly."
- elif any(marker in normalized for marker in ("rate limit", "quota", " 429", "status 429")):
- prefix = "Gemini rate limit or quota was reached. Please try again later."
+ if "openai_api_key is not configured" in normalized:
+ return "The agent has no API key. An admin sets OPENAI_API_KEY in .env and restarts the backend."
+ if any(marker in normalized for marker in ("401", "invalid_api_key", "incorrect api key", "authentication")):
+ prefix = "OpenAI rejected the API key. Check OPENAI_API_KEY in .env."
+ elif any(marker in normalized for marker in ("model_not_found", "does not exist", "404")):
+ prefix = "OpenAI doesn't offer the configured model to this key. Check AGENT_MODEL in .env."
+ elif any(marker in normalized for marker in ("resourceexhausted", "overloaded", "capacity", " 503", "status 503", " 500", " 502")):
+ prefix = "OpenAI is currently overloaded or unavailable. Please try again shortly."
+ elif any(marker in normalized for marker in ("rate limit", "quota", " 429", "status 429", "insufficient_quota")):
+ prefix = "OpenAI rate limit or quota was reached. Please try again later."
elif any(marker in normalized for marker in ("timeout", "timed out", "deadline exceeded")):
- prefix = "Gemini did not respond before the request timed out. Please try again."
+ prefix = "OpenAI did not respond before the request timed out. Please try again."
else:
- prefix = "Gemini could not complete this request."
+ prefix = "OpenAI could not complete this request."
if not raw_message:
return prefix
# Errors can include request headers or provider diagnostics: keep a short,
# redacted detail for the administrator without leaking credentials.
detail = re.sub(r"(?i)(api[_ -]?key|authorization|token)\s*[:=]\s*\S+", r"\1=[redacted]", raw_message)
- detail = re.sub(r"AIza[\w-]+", "[redacted]", detail)[:300]
+ detail = re.sub(r"(AIza|sk-)[\w-]+", "[redacted]", detail)[:300]
return f"{prefix} Provider error: {detail}"
@@ -987,7 +999,8 @@ def get_mount_thresholds(db) -> dict:
def _admin_settings(db) -> dict:
return {"instance_name": get_instance_name(db), "metrics_retention_days": get_metrics_retention_days(db), "agent_chat_enabled": is_agent_chat_enabled(db), "alert_thresholds": get_alert_thresholds(db), "mount_thresholds": get_mount_thresholds(db), "require_2fa_for_admins": require_admin_two_factor(db), "secrets_configured": secrets_box.available(),
- "require_2fa_for_everyone": require_two_factor_for_everyone(db), "disabled_rules": sorted(get_disabled_rules(db))}
+ "require_2fa_for_everyone": require_two_factor_for_everyone(db), "disabled_rules": sorted(get_disabled_rules(db)),
+ "agent_prices": agent_usage.get_prices(db), "agent_monthly_cap_usd": agent_usage.get_cap(db), "agent_daily_chat_limit": agent_usage.get_daily_limit(db)}
def purge_old_metrics(db, now: datetime | None = None) -> int:
@@ -1040,6 +1053,24 @@ def update_admin_settings(payload: AdminSettingsRequest, session_id: Annotated[s
if set(wanted) != get_disabled_rules(db):
_set_setting(db, DISABLED_RULES_KEY, json.dumps(wanted))
changes.append("Switched off alert rules: " + (", ".join(wanted) or "none"))
+ try:
+ if payload.agent_prices is not None:
+ prices = agent_usage.validate_prices(payload.agent_prices)
+ if prices != agent_usage.get_prices(db):
+ _set_setting(db, agent_usage.PRICES_KEY, json.dumps(prices))
+ changes.append("Changed the agent's model prices: " + (", ".join(f"{model} {price['input']:g}/{price['output']:g}" for model, price in prices.items()) or "none"))
+ if payload.agent_monthly_cap_usd is not None:
+ cap = agent_usage.validate_cap(payload.agent_monthly_cap_usd)
+ if cap != agent_usage.get_cap(db):
+ _set_setting(db, agent_usage.CAP_KEY, str(cap or 0))
+ changes.append(f"Set the agent's monthly budget to ${cap:g}" if cap else "Removed the agent's monthly budget")
+ if payload.agent_daily_chat_limit is not None:
+ limit = agent_usage.validate_daily_limit(payload.agent_daily_chat_limit)
+ if limit != agent_usage.get_daily_limit(db):
+ _set_setting(db, agent_usage.DAILY_LIMIT_KEY, str(limit))
+ changes.append(f"Set the agent chat limit to {limit} messages per person per day")
+ except ValueError as exc:
+ raise HTTPException(status_code=400, detail=str(exc)) from exc
if payload.agent_chat_enabled is not None:
_set_setting(db, AGENT_CHAT_ENABLED_KEY, "true" if payload.agent_chat_enabled else "false")
changes.append(f"{'Enabled' if payload.agent_chat_enabled else 'Disabled'} chat with the agent")
@@ -1071,7 +1102,8 @@ def update_admin_settings(payload: AdminSettingsRequest, session_id: Annotated[s
def agent_config(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
"""What the Agent page needs to know about the agent; readable by every signed-in user."""
_current_user(session_id, db)
- return {"chat_enabled": is_agent_chat_enabled(db)}
+ spend = agent_usage.budget(db)
+ return {"chat_enabled": is_agent_chat_enabled(db), "key_configured": bool(configured_key()), "budget_paused": spend["state"] == "paused"}
OVERVIEW_ALERT_LIMIT = 50
@@ -2490,7 +2522,7 @@ def delete_conversation(conversation_id: str, session_id: Annotated[str | None,
LOG_WRITE_LIMIT = 120
-CHAT_MESSAGE_LIMIT = 30 # messages to the assistant per user per hour (each can cost the admin's Gemini quota)
+CHAT_MESSAGE_LIMIT = 30 # messages to the assistant per user per hour (each can cost the admin's OpenAI quota)
CONVERSATION_CREATE_LIMIT = 30 # new conversations per user per hour
MAX_CONVERSATIONS_PER_USER = 100
@@ -2527,6 +2559,10 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cook
if not is_agent_chat_enabled(db):
raise HTTPException(status_code=403, detail="Chat with the agent has been turned off by an admin.")
enforce_limit(redis_client, scope="chat-user", subject=user["id"], limit=CHAT_MESSAGE_LIMIT, window_seconds=3600)
+ spend = agent_usage.budget(db)
+ if spend["state"] == "paused":
+ raise HTTPException(status_code=429, detail=f"The agent's budget for this month is used up (${spend['cost']:.2f} of ${spend['cap']:.2f}). An admin can raise it in Admin › Incident agent.")
+ enforce_limit(redis_client, scope="chat-user-day", subject=user["id"], limit=agent_usage.get_daily_limit(db), window_seconds=86400)
message = payload.message.strip()
if not message:
@@ -2539,11 +2575,16 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cook
conversation.title = message[:255]
db.commit()
+ def log_usage(completion, duration_ms: int) -> None:
+ agent_usage.record(db, kind="chat", model=completion.model, input_tokens=completion.input_tokens, output_tokens=completion.output_tokens, duration_ms=duration_ms,
+ user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id)
+
try:
response_text = send_agent_message(
user["id"],
payload.conversation_id,
_incident_context(db, payload.incident_id) + message,
+ usage_logger=log_usage,
command_logger=lambda event: record_audit_log(
db,
user_id=user["id"],
@@ -2558,6 +2599,7 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cook
)
except Exception as exc:
user_message = agent_error_message(exc)
+ agent_usage.record(db, kind="chat", status="error", model=AGENT_MODEL, user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id, error=user_message)
record_audit_log(
db,
user_id=user["id"],
@@ -2574,6 +2616,7 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cook
raise HTTPException(status_code=503, detail={"message": user_message}) from exc
steps = ["Received the task and evaluated the request.", "Generated a structured response for the requested task."]
+ _warn_about_budget(db, spend)
record_audit_log(
db,
@@ -2962,6 +3005,26 @@ def dispatch_notifications(db, now: datetime | None = None) -> int:
return sent
+def _warn_about_budget(db, before: dict) -> None:
+ """Tell the admins once a month when the agent passes 80 % of its budget, and again when it pauses at 100 %."""
+ after = agent_usage.budget(db)
+ rank = {"ok": 0, "warn": 1, "paused": 2}
+ if rank[after["state"]] <= rank[before["state"]] or not after["cap"]:
+ return
+ try:
+ if not redis_client.set(f"infradeck:agent-budget:{datetime.now(timezone.utc):%Y-%m}:{after['state']}", "1", nx=True, ex=40 * 86400):
+ return
+ except Exception:
+ return
+ paused = after["state"] == "paused"
+ title = "The incident agent paused: its monthly budget is used up" if paused else f"The incident agent has used {round(after['share'] * 100)}% of its monthly budget"
+ body = f"${after['cost']:.2f} of ${after['cap']:.2f} this month. {'Chat stays off until next month or until an admin raises the budget' if paused else 'At 100% the agent pauses until next month'}. Incidents and notifications keep working. Admin › Incident agent."
+ record_audit_log(db, user_id=None, level="WARN", message=title[:255], source="agent", event_type="agent_budget")
+ admins = {user.id: user for user in db.query(User).filter(User.role == "admin", User.is_active.is_(True))}
+ for channel in db.query(NotificationChannel).filter(NotificationChannel.user_id.in_(list(admins))).order_by(NotificationChannel.id):
+ _deliver(db, channel, admins[channel.user_id], title, body, "WARNING")
+
+
def _notify_loop() -> None:
while True:
time.sleep(15)
@@ -3158,13 +3221,32 @@ def admin_system(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_N
"version": APP_VERSION, "database": {"engine": engine.dialect.name, "bytes": size,
"host_rows": db.query(func.count(HostMetric.id)).scalar(), "host_rows_week": db.query(func.count(HostMetric.id)).filter(HostMetric.time >= week).scalar()},
"retention_days": get_metrics_retention_days(db),
- "agent": {"model": AGENT_MODEL, "key_configured": bool(os.getenv("GEMINI_API_KEY", "").strip()), "chat_enabled": is_agent_chat_enabled(db),
+ "agent": {"provider": "OpenAI", "model": AGENT_MODEL, "key_configured": bool(configured_key()), "budget": agent_usage.budget(db), "chat_enabled": is_agent_chat_enabled(db),
"incidents_this_month": db.query(IncidentAlert).filter(IncidentAlert.created_at >= month).count(),
"chat_messages_this_month": db.query(ChatMessage).filter(ChatMessage.role == "user", ChatMessage.created_at >= month).count()},
"smtp_configured": notify.smtp_configured(), "secrets_configured": secrets_box.available(),
}
[email protected]("/admin/agent/usage")
+def admin_agent_usage(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
+ """This month's model calls, tokens and estimated cost against the budget, per model and per day."""
+ _require_admin(session_id, db)
+ return {**agent_usage.month_summary(db), "model": AGENT_MODEL}
+
+
[email protected]("/admin/agent/test")
+def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
+ """Admin's "Test connection": the API key works and the configured model exists. Spends no tokens."""
+ admin = _require_admin(session_id, db)
+ enforce_limit(redis_client, scope="agent-test", subject=admin["id"], limit=10, window_seconds=600)
+ try:
+ result = check_provider_connection()
+ except Exception as exc:
+ raise HTTPException(status_code=502, detail=agent_error_message(exc)) from exc
+ return {"ok": True, **result}
+
+
@app.get("/")
def root():
return {"message": "Infradeck API"}
diff --git a/src/backend/models.py b/src/backend/models.py
index e247ad3..f4574d3 100644
--- a/src/backend/models.py
+++ b/src/backend/models.py
@@ -492,3 +492,27 @@ class Invite(Base):
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
used_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
+
+
+class AgentRun(Base):
+ """One call to the agent's language model: what it was for, which model, how many tokens and what that cost.
+
+ The month's spending and the cap come from these rows. ``cost_usd`` is set when the model had a price at the
+ time; rows without one are priced with the current table when they are added up."""
+
+ __tablename__ = "agent_runs"
+ __table_args__ = (Index("ix_agent_runs_created", "created_at"),)
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ kind: Mapped[str] = mapped_column(String(16), nullable=False) # chat | diagnosis
+ status: Mapped[str] = mapped_column(String(16), nullable=False) # ok | error
+ model: Mapped[str] = mapped_column(String(100), nullable=False)
+ input_tokens: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+ output_tokens: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+ cost_usd: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ duration_ms: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+ user_id: Mapped[Optional[int]] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)
+ conversation_id: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
+ incident_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
+ error: Mapped[Optional[str]] = mapped_column(String(300), nullable=True)
+ created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)
diff --git a/src/frontend/src/admin/AdminPage.jsx b/src/frontend/src/admin/AdminPage.jsx
index 97fd0af..a15f2ec 100644
--- a/src/frontend/src/admin/AdminPage.jsx
+++ b/src/frontend/src/admin/AdminPage.jsx
@@ -6,6 +6,7 @@ import { apiRequest } from '../api.js'
import { AnotherMachineCard, ThisMachineCard } from '../dashboard/EmptyFleet.jsx'
import { INTERVALS, intervalLabel } from '../dashboard/ServerPanel.jsx'
import { serverState } from '../dashboard/logic.js'
+import { Sparkline } from '../dashboard/charts.jsx'
import { Icon } from '../icons.jsx'
import { refreshOverview, useOverview } from '../overviewStore.js'
import { AppShell, serverLink } from '../shell.jsx'
@@ -274,19 +275,113 @@ function MountOverrides({ settings, onSaved }) {
function Agent({ settings, system, onSaved }) {
const [state, run] = useAction()
+ const [test, runTestAction] = useAction()
+ const [tested, setTested] = useState(null)
+ const runTest = (work) => { setTested(null); return runTestAction(async () => setTested(await work())) }
const agent = system.agent
return <>
<div className="card divide-y divide-line">
- <Row icon="AI" title="Model provider" detail={agent.key_configured ? `Google Gemini · ${agent.model.replace('models/', '')} · key configured` : 'No API key configured. Set GEMINI_API_KEY in .env to turn the chat on.'} tone={agent.key_configured ? undefined : 'warn'}><button type="button" className="btn btn-sm" disabled title="The provider and key are set in .env (AGENT_MODEL, GEMINI_API_KEY) and need a restart.">Change</button></Row>
+ <Row icon="AI" title="Model provider" detail={agent.key_configured ? `${agent.provider || 'OpenAI'} · ${agent.model} · key configured` : 'No API key configured. Set OPENAI_API_KEY in .env and restart the backend to turn the chat on.'} tone={agent.key_configured ? undefined : 'warn'}>
+ <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key and the model without spending tokens.' : 'Set OPENAI_API_KEY first.'} onClick={() => runTest(() => apiRequest('/admin/agent/test', { method: 'POST' }))}>{test.busy ? 'Testing…' : 'Test connection'}</button>
+ </Row>
+ {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key and offers {tested.model}.</p>}</div>}
<Row title="Diagnose new incidents automatically" detail="Always on. The diagnosis is built from stored metrics and events by fixed rules, so it costs nothing and sends nothing to the provider."><Switch label="Diagnose automatically" checked disabled onChange={() => {}} /></Row>
<Row title="Read container logs" detail="Not available: Infradeck's Docker proxy can't read logs. See SECURITY.md."><Switch label="Read container logs" checked={false} disabled onChange={() => {}} /></Row>
<Row title="Chat" detail="Available to every signed-in user while it is on. Questions and the incident's facts are sent to the provider."><Switch label="Chat with the agent" checked={settings.agent_chat_enabled} disabled={state.busy} onChange={(on) => run(async () => onSaved(await put({ agent_chat_enabled: on })))} /></Row>
<Row title="Redact IPs and hostnames before sending" detail="Not available yet. Until then, turn Chat off if names must not leave this machine."><Switch label="Redact before sending" checked={false} disabled onChange={() => {}} /></Row>
</div>
+ <Outcome state={state} />
+ <AgentSpending settings={settings} onSaved={onSaved} />
+ </>
+}
+
+// "$0.0031" for small amounts, "$1.84" from a cent up.
+const dollars = (value) => (value == null ? '—' : `$${value > 0 && value < 0.01 ? value.toFixed(4) : value.toFixed(2)}`)
+const thousands = (value) => (value >= 1e6 ? `${(value / 1e6).toFixed(1)} M` : value >= 1e4 ? `${Math.round(value / 1e3)} k` : value.toLocaleString())
+
+// What the model calls cost this month, against the budget, and the settings that limit it.
+function AgentSpending({ settings, onSaved }) {
+ const { data, error, reload } = useLoaded('/admin/agent/usage')
+ const [state, run] = useAction()
+ const [cap, setCap] = useState(settings.agent_monthly_cap_usd ?? '')
+ const [limit, setLimit] = useState(String(settings.agent_daily_chat_limit ?? 100))
+ const [prices, setPrices] = useState(() => Object.entries(settings.agent_prices || {}).map(([model, price]) => ({ model, input: String(price.input), output: String(price.output) })))
+ if (!data) return <Notice tone="plain">{error || 'Loading the agent\'s usage…'}</Notice>
+ const save = (body, done) => run(async () => { onSaved(await put(body)); reload() }, done)
+ const savePrices = (event) => {
+ event.preventDefault()
+ save({ agent_prices: Object.fromEntries(prices.filter((row) => row.model.trim()).map((row) => [row.model.trim(), { input: Number(row.input), output: Number(row.output) }])) }, 'Prices saved.')
+ }
+ const setRow = (index, key, value) => setPrices((rows) => rows.map((row, position) => (position === index ? { ...row, [key]: value } : row)))
+ const share = data.share == null ? null : Math.min(1, data.share)
+ const daily = data.daily.map((day) => day.cost)
+ const configuredPriced = prices.some((row) => row.model.trim() && (data.model === row.model.trim() || data.model.startsWith(`${row.model.trim()}-`)))
+ return <>
+ {data.state === 'paused' && <Notice>The agent is paused: this month's budget is used up ({dollars(data.cost)} of {dollars(data.cap)}). Chat stays off until next month or until you raise the budget. Incidents and notifications keep working.</Notice>}
+ {data.state === 'warn' && <Notice>The agent has used {Math.round(data.share * 100)}% of this month's budget ({dollars(data.cost)} of {dollars(data.cap)}). At 100% it pauses until next month.</Notice>}
<div className="card grid divide-y divide-line sm:grid-cols-3 sm:divide-x sm:divide-y-0">
- {[[agent.incidents_this_month, 'incidents diagnosed this month'], [agent.chat_messages_this_month, 'chat messages this month'], ['—', 'provider cost · not tracked']].map(([value, label]) => <div key={label} className="px-[18px] py-4"><div className="text-2xl font-semibold leading-none text-paper">{value}</div><div className="mt-2 text-xs text-muted">{label}</div></div>)}
+ <div className="px-[18px] py-4">
+ <div className="text-2xl font-semibold leading-none text-paper">{dollars(data.cost)}{data.cap && <span className="ml-1.5 text-sm font-normal text-muted">of {dollars(data.cap)}</span>}</div>
+ <div className="mt-2 text-xs text-muted">estimated cost this month{data.unpriced_runs ? ` · ${data.unpriced_runs} ${data.unpriced_runs === 1 ? 'call' : 'calls'} without a price` : ''}</div>
+ {share != null && <span className={`meter mt-3 ${data.state === 'paused' ? 'meter-signal' : data.state === 'warn' ? 'meter-warn' : ''}`} role="meter" aria-label="Share of the monthly budget" aria-valuemin={0} aria-valuemax={100} aria-valuenow={Math.round(share * 100)}><i style={{ width: `${share * 100}%`, minWidth: share > 0 ? 3 : 0 }} /></span>}
+ </div>
+ <div className="px-[18px] py-4">
+ <div className="text-2xl font-semibold leading-none text-paper">{data.calls}</div>
+ <div className="mt-2 text-xs text-muted">calls to the model this month{data.errors ? ` · ${data.errors} failed` : ''}{data.diagnoses ? ` · ${data.diagnoses} diagnoses` : ''}</div>
+ {daily.some((value) => value > 0) && <Sparkline className="mt-3" values={daily} width={140} height={18} max={Math.max(...daily)} />}
+ </div>
+ <div className="px-[18px] py-4">
+ <div className="text-2xl font-semibold leading-none text-paper">{thousands(data.input_tokens + data.output_tokens)}</div>
+ <div className="mt-2 text-xs text-muted">tokens · {thousands(data.input_tokens)} in, {thousands(data.output_tokens)} out</div>
+ </div>
+ </div>
+
+ {data.by_model.length > 0 && <div className="card overflow-hidden"><div className="custom-scroll overflow-x-auto"><table className="w-full min-w-[460px] border-collapse text-left text-[13px]">
+ <thead><tr className="grid-head h-10"><th className="pl-[18px] font-medium">Model</th><th className="text-right font-medium">Calls</th><th className="text-right font-medium">Tokens in / out</th><th className="pr-[18px] text-right font-medium">Cost</th></tr></thead>
+ <tbody>{data.by_model.map((row) => (
+ <tr key={row.model} className="h-10 border-t border-line">
+ <td className="mono pl-[18px] text-paper">{row.model}</td>
+ <td className="text-right text-soft">{row.calls}</td>
+ <td className="text-right text-soft">{thousands(row.input_tokens)} / {thousands(row.output_tokens)}</td>
+ <td className={`pr-[18px] text-right ${row.priced ? 'text-paper' : 'text-warn'}`}>{row.priced ? dollars(row.cost) : 'no price'}</td>
+ </tr>
+ ))}</tbody>
+ </table></div></div>}
+
+ <div className="card divide-y divide-line">
+ <Row title="Monthly budget" detail="At 80% admins get a notification; at 100% the chat pauses until the 1st of next month. Empty: no budget.">
+ <form className="flex items-center gap-2" onSubmit={(event) => { event.preventDefault(); save({ agent_monthly_cap_usd: cap === '' ? 0 : Number(cap) }, 'Budget saved.') }}>
+ <span className="text-[13px] text-muted">$</span>
+ <input type="number" min="0" step="0.01" inputMode="decimal" className="field h-9 w-24" aria-label="Monthly budget in US dollars" value={cap} placeholder="none" onChange={(event) => setCap(event.target.value)} />
+ <button type="submit" className="btn btn-sm h-9" disabled={state.busy || String(cap) === String(settings.agent_monthly_cap_usd ?? '')}>Save</button>
+ </form>
+ </Row>
+ <Row title="Chat messages per person" detail="In any 24 hours, on top of 30 an hour. Admins included.">
+ <form className="flex items-center gap-2" onSubmit={(event) => { event.preventDefault(); save({ agent_daily_chat_limit: Number(limit) }, 'Limit saved.') }}>
+ <input type="number" min="1" step="1" inputMode="numeric" className="field h-9 w-24" aria-label="Chat messages per person per day" value={limit} onChange={(event) => setLimit(event.target.value)} />
+ <button type="submit" className="btn btn-sm h-9" disabled={state.busy || limit === String(settings.agent_daily_chat_limit)}>Save</button>
+ </form>
+ </Row>
+ <form onSubmit={savePrices} className="px-[18px] py-3.5">
+ <div className="text-[13px] font-medium text-paper">Prices per million tokens</div>
+ <p className="mb-0 mt-0.5 text-xs leading-5 text-muted">Copy them from your provider's pricing page; they aren't fetched automatically. A price for <span className="mono">gpt-5-mini</span> also covers dated names such as <span className="mono">gpt-5-mini-2026-01-01</span>.{!configuredPriced && <span className="text-warn"> The model in use, <span className="mono">{data.model}</span>, has no price, so its calls aren't counted in the budget.</span>}</p>
+ <div className="mt-3 space-y-2">
+ {prices.map((row, index) => (
+ <div key={index} className="flex flex-wrap items-end gap-2">
+ <Field label="Model" className="min-w-0 flex-1 basis-40"><input className="field mono h-9" required maxLength={100} value={row.model} onChange={(event) => setRow(index, 'model', event.target.value)} /></Field>
+ <Field label="Input $"><input type="number" min="0" step="any" required inputMode="decimal" className="field h-9 w-24" value={row.input} onChange={(event) => setRow(index, 'input', event.target.value)} /></Field>
+ <Field label="Output $"><input type="number" min="0" step="any" required inputMode="decimal" className="field h-9 w-24" value={row.output} onChange={(event) => setRow(index, 'output', event.target.value)} /></Field>
+ <button type="button" className="btn btn-ghost h-9" aria-label={`Remove the price of ${row.model || 'this model'}`} onClick={() => setPrices((rows) => rows.filter((_, position) => position !== index))}><Icon name="close" size={13} /></button>
+ </div>
+ ))}
+ </div>
+ <div className="mt-3 flex flex-wrap items-center gap-2.5">
+ <button type="button" className="btn btn-sm" onClick={() => setPrices((rows) => [...rows, { model: rows.some((item) => item.model === data.model) ? '' : data.model, input: '', output: '' }])}><Icon name="plus" size={13} />Add a model</button>
+ <button type="submit" className="btn btn-sm btn-primary" disabled={state.busy}>Save prices</button>
+ <Outcome state={state} />
+ </div>
+ </form>
</div>
- <Outcome state={state} />
</>
}
@@ -365,7 +460,7 @@ function PhoneHub({ overview, settings, system }) {
<div className="divide-y divide-line overflow-hidden rounded-md border border-line bg-ink">
{row('/admin?section=users', 'Users', null, users ? `${users.users.length}${invited ? ` · ${invited} invited` : ''}` : null)}
{row('/admin?section=rules', 'Alert rules', null, rules ? `${on} on · ${rules.rules.length - on} off` : null)}
- {row('/admin?section=agent', 'Incident agent', null, system.agent.key_configured ? (settings.agent_chat_enabled ? 'chat on' : 'chat off') : 'no key', undefined, system.agent.key_configured ? undefined : 'warn')}
+ {row('/admin?section=agent', 'Incident agent', null, !system.agent.key_configured ? 'no key' : system.agent.budget.cap ? `${dollars(system.agent.budget.cost)} of ${dollars(system.agent.budget.cap)}` : `${dollars(system.agent.budget.cost)} this month`, undefined, !system.agent.key_configured || system.agent.budget.state !== 'ok' ? 'warn' : undefined)}
{row('/admin?section=retention', 'Data and retention', null, megabytes(system.database.bytes))}
{row('/admin?section=about', 'About', null, system.version)}
</div>
diff --git a/src/frontend/src/agent/AgentPage.jsx b/src/frontend/src/agent/AgentPage.jsx
index 8f1b7e8..2b2a44c 100644
--- a/src/frontend/src/agent/AgentPage.jsx
+++ b/src/frontend/src/agent/AgentPage.jsx
@@ -160,7 +160,7 @@ function Incident({ incident, canAct, onChanged }) {
const SUGGESTIONS = { default: ['Is anything else close to its limit?', 'What changed on this server today?', 'Explain this incident in one paragraph'], none: ['Is anything close to its limit?', 'Which server needs attention first?', 'What happened overnight?'] }
-function Chat({ user, incident, enabled, phone = false, back }) {
+function Chat({ user, incident, enabled, offReason, phone = false, back }) {
const conversationId = `${incident ? `incident-${incident.id}` : 'general'}-u${user.id}`
const [messages, setMessages] = useState(null)
const [draft, setDraft] = useState('')
@@ -204,7 +204,7 @@ function Chat({ user, incident, enabled, phone = false, back }) {
return <>
<PhoneHeader back={back} title="Ask the agent" action={incident ? <span className="chip mono shrink-0 text-paper">{incident.reference}</span> : <button type="button" className="h-11 border-0 bg-transparent px-1 text-[13px] text-muted" onClick={restart}>New chat</button>} />
<div className="space-y-5 px-4 pb-[190px] pt-4" aria-live="polite">
- {!enabled && <Notice tone="plain">Chat with the agent has been turned off by an admin.</Notice>}
+ {!enabled && <Notice tone="plain">{offReason}</Notice>}
{enabled && messages === null && <p className="m-0 text-xs text-muted">Loading…</p>}
{enabled && messages?.length === 0 && <p className="m-0 text-[13px] leading-5 text-muted">{incident ? 'The agent is given this incident\'s facts with your question.' : 'Ask about any server.'}</p>}
{enabled && (messages || []).map((message) => (message.role === 'user'
@@ -232,7 +232,7 @@ function Chat({ user, incident, enabled, phone = false, back }) {
<div className="mt-2 flex items-center gap-2 text-xs text-muted">About <span className="chip max-w-full truncate text-paper">{incident ? `${incident.reference} · ${incident.hostname}` : 'any server'}</span></div>
</div>
<div className="custom-scroll min-h-0 flex-1 space-y-4 overflow-y-auto px-5 py-4">
- {!enabled && <Notice tone="plain">Chat with the agent has been turned off by an admin. The diagnosis on the left doesn't need it.</Notice>}
+ {!enabled && <Notice tone="plain">{offReason} The diagnosis on the left doesn't need it.</Notice>}
{enabled && messages === null && <p className="m-0 text-xs text-muted">Loading…</p>}
{enabled && messages?.length === 0 && <div className="space-y-2">
<p className="m-0 text-xs text-muted">{incident ? 'The agent is given this incident\'s facts with your question.' : 'Ask about any server.'} Suggestions:</p>
@@ -401,13 +401,17 @@ export default function AgentPage({ user, onLogout }) {
useEffect(() => { if (selected) setTab(selected.status === 'resolved' ? 'resolved' : selected.muted_until ? 'muted' : 'open') }, [selected?.id]) // eslint-disable-line react-hooks/exhaustive-deps
const list = inbox ? inbox[tab] : []
const crumbs = selected ? [{ label: 'Agent', to: '/agent' }, { label: selected.reference }] : [{ label: 'Agent' }]
+ // Why the chat can't be used right now, or null when it can.
+ const chatOff = !config ? null : !config.chat_enabled ? 'Chat with the agent has been turned off by an admin.'
+ : config.key_configured === false ? 'The agent has no API key yet. An admin sets OPENAI_API_KEY in .env.'
+ : config.budget_paused ? 'This month\'s budget for the agent is used up. An admin can raise it in Admin › Incident agent.' : null
if (phone) {
const view = params.get('view')
const incident = detail && detail.id === selectedId ? detail : null
const chatOpen = view === 'chat' && (!requested || incident)
let body
- if (chatOpen) body = <Chat key={selectedId ?? 'none'} phone user={user} incident={incident} enabled={config ? config.chat_enabled : true} back={incident ? { label: incident.title, to: `/agent?incident=${incident.id}` } : { label: 'Incidents', to: '/agent' }} />
+ if (chatOpen) body = <Chat key={selectedId ?? 'none'} phone user={user} incident={incident} enabled={!chatOff} offReason={chatOff} back={incident ? { label: incident.title, to: `/agent?incident=${incident.id}` } : { label: 'Incidents', to: '/agent' }} />
else if (incident) body = <PhoneIncident incident={incident} canAct={Boolean(inbox?.can_act)} onChanged={() => { reload(); reloadDetail() }} onAsk={() => setParams({ incident: String(incident.id), view: 'chat' })} />
else if (selectedId || (requested && !inbox)) body = <><PhoneHeader back={{ label: 'Incidents', to: '/agent' }} title="Incident" live={false} /><p className="m-0 px-4 py-16 text-center text-[13px] text-muted">{detailError || 'Loading the incident…'}</p></>
else body = <PhoneInbox inbox={inbox} error={error} tab={tab} setTab={setTab} onSelect={(id) => setParams({ incident: String(id) })} />
@@ -435,7 +439,7 @@ export default function AgentPage({ user, onLogout }) {
: selectedId ? <p className="m-0 px-7 py-16 text-center text-[13px] text-muted">{detailError || 'Loading the incident…'}</p>
: <div className="px-7 py-16 text-center"><p className="m-0 text-sm font-medium text-paper">{inbox && !all.length ? 'No incidents yet.' : 'Nothing needs you right now.'}</p><p className="mb-0 mt-2 text-[13px] text-muted">{inbox && !all.length ? 'When a rule fires on one of your servers, the incident shows up here with what was found and how to fix it.' : 'Pick a resolved incident on the left to see how it went.'}</p></div>}
</div>
- <div className="min-h-0 border-t border-line xl:border-t-0"><Chat key={selectedId ?? 'none'} user={user} incident={detail && detail.id === selectedId ? detail : null} enabled={config ? config.chat_enabled : true} /></div>
+ <div className="min-h-0 border-t border-line xl:border-t-0"><Chat key={selectedId ?? 'none'} user={user} incident={detail && detail.id === selectedId ? detail : null} enabled={!chatOff} offReason={chatOff} /></div>
</div>
</AppShell>
)
diff --git a/tests/accounts/test_admin.py b/tests/accounts/test_admin.py
index b4b9bbc..2fa8364 100644
--- a/tests/accounts/test_admin.py
+++ b/tests/accounts/test_admin.py
@@ -154,9 +154,11 @@ def test_admin_can_turn_agent_chat_off_and_on(fleet, monkeypatch):
result = main.update_admin_settings(main.AdminSettingsRequest(agent_chat_enabled=False), session_id="s", db=fleet)
- assert result == {"instance_name": main.get_instance_name(fleet), "metrics_retention_days": main.DEFAULT_METRICS_RETENTION_DAYS, "agent_chat_enabled": False, "alert_thresholds": rules.DEFAULT_THRESHOLDS, "mount_thresholds": {}, "require_2fa_for_admins": True, "secrets_configured": main.secrets_box.available(), "require_2fa_for_everyone": False, "disabled_rules": []}
+ assert result == {"instance_name": main.get_instance_name(fleet), "metrics_retention_days": main.DEFAULT_METRICS_RETENTION_DAYS, "agent_chat_enabled": False, "alert_thresholds": rules.DEFAULT_THRESHOLDS, "mount_thresholds": {}, "require_2fa_for_admins": True, "secrets_configured": main.secrets_box.available(), "require_2fa_for_everyone": False, "disabled_rules": [],
+ "agent_prices": {}, "agent_monthly_cap_usd": None, "agent_daily_chat_limit": main.agent_usage.DEFAULT_DAILY_CHAT_LIMIT}
_login_as(monkeypatch, 2)
- assert main.agent_config(session_id="s", db=fleet) == {"chat_enabled": False}
+ config = main.agent_config(session_id="s", db=fleet)
+ assert config["chat_enabled"] is False and config["budget_paused"] is False
def test_agent_chat_is_rejected_while_disabled(fleet, monkeypatch):
diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.py
index e6ec5cf..ae1a603 100644
--- a/tests/accounts/test_new_pages.py
+++ b/tests/accounts/test_new_pages.py
@@ -337,3 +337,131 @@ def test_a_server_is_renamed_and_keeps_the_name_and_the_install_describes_itself
system = owner.get("/admin/system").json()
assert system["version"] == main.APP_VERSION and system["database"]["host_rows"] == 30 and system["agent"]["incidents_this_month"] >= 1
+
+
+def test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monkeypatch):
+ owner = signed_in(app, 1)
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-test-secret-value")
+ monkeypatch.setattr(main, "check_provider_connection", lambda: {"provider": "OpenAI", "model": "gpt-test"})
+
+ system = owner.get("/admin/system")
+ assert system.json()["agent"]["key_configured"] is True and system.json()["agent"]["provider"] == "OpenAI"
+ result = owner.post("/admin/agent/test")
+ assert result.json() == {"ok": True, "provider": "OpenAI", "model": "gpt-test"}
+ assert "sk-test-secret-value" not in system.text + result.text
+ assert signed_in(app, 3).post("/admin/agent/test").status_code == 403
+
+
+def test_a_failed_agent_connection_test_explains_itself_without_the_key(app, monkeypatch):
+ owner = signed_in(app, 1)
+
+ def rejected():
+ raise RuntimeError("Error code: 401 - Incorrect API key provided: sk-test-secret-value")
+ monkeypatch.setattr(main, "check_provider_connection", rejected)
+
+ response = owner.post("/admin/agent/test")
+ assert response.status_code == 502 and "rejected the API key" in response.json()["detail"]
+ assert "sk-test-secret-value" not in response.text
+
+
+# --- the agent's spending (#112) ---------------------------------------------------------------------------------
+
+from src.agent.providers import Completion
+from src.backend.models import AgentRun
+
+
+def _fake_agent(monkeypatch, tokens=(1000, 200), fail=None):
+ def fake(user_id, conversation_id, message, command_logger=None, usage_logger=None):
+ if fail:
+ raise fail
+ # A dated model name, as the API reports it: it must still use the price set for "gpt-test".
+ usage_logger(Completion(text="{}", model="gpt-test-2026-01-01", input_tokens=tokens[0], output_tokens=tokens[1]), 1500)
+ return "Here is the answer."
+ monkeypatch.setattr(main, "send_agent_message", fake)
+ monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)
+
+
+def _ask(client, conversation="c1"):
+ client.post("/conversations", json={"conversation_id": conversation, "title": "t"})
+ return client.post("/agent/chat", json={"conversation_id": conversation, "message": "Why did it restart?", "incident_id": 42})
+
+
+def test_every_chat_call_is_counted_with_its_tokens_and_cost(app, monkeypatch):
+ _fake_agent(monkeypatch)
+ owner = signed_in(app, 1)
+ assert owner.put("/admin/settings", json={"agent_prices": {"gpt-test": {"input": 1, "output": 10}}}).status_code == 200
+
+ assert _ask(owner).status_code == 200
+
+ run = app.db.query(AgentRun).one()
+ assert (run.kind, run.status, run.input_tokens, run.output_tokens, run.duration_ms, run.user_id, run.incident_id) == ("chat", "ok", 1000, 200, 1500, 1, 42)
+ assert run.cost_usd == pytest.approx(0.003) # 1000 × $1 + 200 × $10 per million tokens
+ usage = owner.get("/admin/agent/usage").json()
+ assert usage["calls"] == 1 and usage["cost"] == pytest.approx(0.003) and usage["by_model"][0]["model"] == "gpt-test-2026-01-01"
+ assert usage["daily"][-1]["cost"] == pytest.approx(0.003)
+ assert signed_in(app, 3).get("/admin/agent/usage").status_code == 403
+
+
+def test_calls_made_before_a_price_was_entered_are_priced_later(app, monkeypatch):
+ _fake_agent(monkeypatch)
+ owner = signed_in(app, 1)
+ _ask(owner)
+ assert owner.get("/admin/agent/usage").json()["unpriced_runs"] == 1
+
+ owner.put("/admin/settings", json={"agent_prices": {"gpt-test": {"input": 1, "output": 10}}})
+
+ usage = owner.get("/admin/agent/usage").json()
+ assert usage["cost"] == pytest.approx(0.003) and usage["unpriced_runs"] == 0
+
+
+def test_the_monthly_budget_pauses_the_chat_and_tells_the_admins_once(app, monkeypatch):
+ _fake_agent(monkeypatch)
+ sent = []
+ monkeypatch.setattr(main, "_deliver", lambda db, channel, owner, title, body, severity: sent.append((owner.id, title)) or True)
+ app.db.add(NotificationChannel(user_id=1, kind="ntfy", target_enc=secrets_box.encrypt("https://ntfy.sh/x"), display="ntfy.sh/x"))
+ app.db.add(NotificationChannel(user_id=3, kind="ntfy", target_enc=secrets_box.encrypt("https://ntfy.sh/y"), display="ntfy.sh/y"))
+ app.db.commit()
+ owner = signed_in(app, 1)
+ owner.put("/admin/settings", json={"agent_prices": {"gpt-test": {"input": 1, "output": 10}}, "agent_monthly_cap_usd": 0.005})
+
+ assert _ask(owner).status_code == 200 # $0.003 of $0.005
+ assert _ask(owner).status_code == 200 # $0.006: over the budget
+ paused = _ask(owner)
+
+ assert paused.status_code == 429 and "budget" in paused.json()["detail"]
+ assert app.db.query(AgentRun).count() == 2 # the paused request never reached the model
+ assert sent == [(1, "The incident agent paused: its monthly budget is used up")] # admins only, once
+ assert owner.get("/agent/config").json()["budget_paused"] is True
+
+ owner.put("/admin/settings", json={"agent_monthly_cap_usd": 0}) # removing the budget lets it run again
+ assert _ask(owner).status_code == 200
+
+
+def test_the_daily_limit_counts_messages_per_person(app, monkeypatch):
+ _fake_agent(monkeypatch)
+ owner = signed_in(app, 1)
+ owner.put("/admin/settings", json={"agent_daily_chat_limit": 2})
+
+ assert [_ask(owner).status_code for _ in range(3)] == [200, 200, 429]
+ assert _ask(signed_in(app, 2), conversation="c2").status_code == 200 # someone else still can
+
+
+def test_a_failed_call_is_recorded_without_tokens(app, monkeypatch):
+ _fake_agent(monkeypatch, fail=TimeoutError("timed out"))
+ owner = signed_in(app, 1)
+
+ assert _ask(owner).status_code == 503
+
+ run = app.db.query(AgentRun).one()
+ assert (run.status, run.input_tokens, run.cost_usd) == ("error", 0, None) and "timed out" in run.error
+
+
+ {"agent_prices": {"gpt test": {"input": 1, "output": 2}}},
+ {"agent_prices": {"gpt-test": {"input": -1, "output": 2}}},
+ {"agent_prices": {"gpt-test": {"input": 1}}},
+ {"agent_monthly_cap_usd": -5},
+ {"agent_daily_chat_limit": 0},
+])
+def test_spending_settings_are_checked(app, body):
+ assert signed_in(app, 1).put("/admin/settings", json=body).status_code == 400
diff --git a/tests/assistant/test_agent_errors.py b/tests/assistant/test_agent_errors.py
index 372f219..f7a8f04 100644
--- a/tests/assistant/test_agent_errors.py
+++ b/tests/assistant/test_agent_errors.py
@@ -13,3 +13,18 @@ def test_provider_error_redacts_api_keys():
assert "[redacted]" in message
assert "AIzaSecretValue" not in message
+
+
+def test_openai_keys_are_redacted_too():
+ message = agent_error_message(Exception("Error code: 401 - Incorrect API key provided: sk-proj-abc123SECRET"))
+
+ assert "sk-proj-abc123SECRET" not in message
+ assert "rejected the API key" in message
+
+
+def test_a_missing_key_says_where_to_set_it():
+ assert "OPENAI_API_KEY" in agent_error_message(RuntimeError("OPENAI_API_KEY is not configured."))
+
+
+def test_an_unknown_model_points_at_agent_model():
+ assert "AGENT_MODEL" in agent_error_message(Exception("Error code: 404 - The model `gpt-nope` does not exist"))
diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.py
index baafced..a98c6b6 100644
--- a/tests/assistant/test_agent_service.py
+++ b/tests/assistant/test_agent_service.py
@@ -60,3 +60,97 @@ def test_arbitrary_command_is_rejected_before_execution():
with pytest.raises(ValueError, match="outside the allow-list"):
service.send(1, "conversation", "Delete everything")
+
+
+# --- the OpenAI provider ----------------------------------------------------------------------------------------
+
+from types import SimpleNamespace
+
+from src.agent import service as agent_service
+from src.agent.providers import Completion, OpenAIProvider
+
+
+class FakeCompletions:
+ def __init__(self, content='{"message":"All good.","action":null}', refusal=None, fail=None):
+ self.calls = []
+ self.content, self.refusal, self.fail = content, refusal, fail
+
+ def create(self, **kwargs):
+ self.calls.append(kwargs)
+ if self.fail:
+ raise self.fail
+ message = SimpleNamespace(content=self.content, refusal=self.refusal)
+ return SimpleNamespace(choices=[SimpleNamespace(message=message)], model="gpt-test", usage=SimpleNamespace(prompt_tokens=120, completion_tokens=8))
+
+
+def fake_client(completions):
+ return SimpleNamespace(chat=SimpleNamespace(completions=completions), models=SimpleNamespace(retrieve=lambda model: SimpleNamespace(id=model)))
+
+
+def test_openai_provider_asks_for_the_reply_schema_and_reports_tokens():
+ completions = FakeCompletions()
+ provider = OpenAIProvider(model="gpt-test", client=fake_client(completions))
+
+ result = provider.complete([{"role": "user", "content": "hi"}], agent_service.REPLY_SCHEMA, "agent_reply")
+
+ assert result == Completion(text='{"message":"All good.","action":null}', model="gpt-test", input_tokens=120, output_tokens=8)
+ format_ = completions.calls[0]["response_format"]
+ assert format_["type"] == "json_schema" and format_["json_schema"]["strict"] is True
+ # The model can only pick a check from the allow-list; anything else is impossible, not just rejected later.
+ check = format_["json_schema"]["schema"]["properties"]["action"]["anyOf"][0]["properties"]["check"]
+ assert set(check["enum"]) == set(agent_service.HealthCheckRunner.ALLOWED_CHECKS)
+
+
+def test_a_refusal_is_an_error_not_an_empty_reply():
+ provider = OpenAIProvider(model="gpt-test", client=fake_client(FakeCompletions(content=None, refusal="I can't help with that.")))
+
+ with pytest.raises(ValueError, match="refused"):
+ provider.complete([], agent_service.REPLY_SCHEMA, "agent_reply")
+
+
+def test_the_provider_needs_a_key(monkeypatch):
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+
+ with pytest.raises(RuntimeError, match="OPENAI_API_KEY"):
+ OpenAIProvider()
+
+
+def test_test_connection_spends_no_tokens():
+ completions = FakeCompletions()
+ provider = OpenAIProvider(model="gpt-test", client=fake_client(completions))
+
+ assert provider.check() == {"provider": "OpenAI", "model": "gpt-test"}
+ assert completions.calls == []
+
+
+def test_a_conversation_sends_the_system_prompt_and_its_history():
+ completions = FakeCompletions()
+ chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))
+
+ chat.send_message("first")
+ chat.send_message("second")
+
+ sent = completions.calls[-1]["messages"]
+ assert sent[0]["role"] == "system" and [message["content"] for message in sent[1:] if message["role"] == "user"] == ["first", "second"]
+ assert [message["role"] for message in sent[1:]] == ["user", "assistant", "user"]
+
+
+def test_only_recent_turns_go_back_to_the_model(monkeypatch):
+ monkeypatch.setattr(agent_service, "MAX_HISTORY_MESSAGES", 4)
+ completions = FakeCompletions()
+ chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))
+
+ for number in range(6):
+ chat.send_message(f"message {number}")
+
+ assert len(completions.calls[-1]["messages"]) == 1 + 4
+
+
+def test_a_failed_turn_is_not_kept_in_the_conversation():
+ completions = FakeCompletions(fail=TimeoutError("timed out"))
+ chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))
+
+ with pytest.raises(TimeoutError):
+ chat.send_message("hello")
+
+ assert chat.history == []
diff --git a/tests/security/test_second_review.py b/tests/security/test_second_review.py
index 95090a0..57f794d 100644
--- a/tests/security/test_second_review.py
+++ b/tests/security/test_second_review.py
@@ -297,14 +297,10 @@ def test_server_ids_in_the_address_must_fit_too(app, path):
def test_the_assistants_memory_of_chats_is_bounded_and_forgets_the_least_recently_used(monkeypatch):
from src.agent import service
- class FakeChats:
- def create(self, **kwargs):
- return object()
-
instance = object.__new__(service.AgentService) if hasattr(service, "AgentService") else None
if instance is None:
pytest.skip("the assistant service class has another name")
- instance.client = type("C", (), {"chats": FakeChats()})()
+ instance.provider = object()
instance.chats, instance.lock = {}, __import__("threading").Lock()
monkeypatch.setattr(service, "MAX_CACHED_CHATS", 3)
for number in range(5):