Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat(agent): usage tracking, monthly budget and daily chat limit
karloks2005922c358863c7
diff --git a/ODSTUPANJA.md b/ODSTUPANJA.mdindex 30bbe74..1a2dd26 100644--- a/ODSTUPANJA.md+++ b/ODSTUPANJA.md@@ -127,7 +127,7 @@ su anotacije dizajna i namjerno nisu ugrađeni.| Alert rules: stupac Servers s izborom servera | Uvijek „All" | Pravila vrijede za sve servere | – || Alert rules: „New rule" | Gumb je onemogućen | Vlastita pravila ne postoje | – || Alert rules: bilješka zašto je pravilo isključeno | Samo broj isključenih pravila | Razlog se ne upisuje | – |-| Incident agent: „Anthropic API · key ending 4f2a", „Change" | Google Gemini, model iz `.env`; „Change" je onemogućen | Pružatelj i ključ se postavljaju u `.env` | #91 |+| Incident agent: „Anthropic API · key ending 4f2a", „Change" | OpenAI, ključ i model iz `.env`; „Change" je onemogućen, „Test connection" provjerava ključ | Ključ se zasad postavlja u `.env`; spremanje u Adminu je dio #109 | #109 || „Diagnose new incidents automatically" prekidač | Uvijek uključeno i onemogućeno | Dijagnoza je lokalna i ne može se isključiti | #93 || „Read container logs" s iznimkama | Onemogućeno | Proxy ne čita logove | #100 || „Redact IPs and hostnames" | Onemogućeno | Nije napravljeno | – |diff --git a/README.md b/README.mdindex cfadfe3..9cc6dab 100644--- a/README.md+++ b/README.md@@ -7,7 +7,7 @@ One machine runs the dashboard; every other machine runs a small collector thatreports host and Docker container metrics back to it. It works the same whetherthe machines share a LAN or are spread across sites and VPSes.-It also includes **VertexAI**, an optional monitoring-only AI assistant (Gemini)+It also includes an optional, monitoring-only incident agent (OpenAI API, set `OPENAI_API_KEY`)that can run a fixed set of read-only health checks and never executes commands.It's main purpose is to report errors, diagnose errors and propose optimal fixes.diff --git a/SECURITY.md b/SECURITY.mdindex d9215e6..cdca572 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -372,7 +372,7 @@ The collector's network checks are passive or opt-in, and add no privileges:a session for any user, administrator included, with no password or two-factor.- **Limits on what a signed-in user can run up.** Messages to the assistant (30 an hour per user), newconversations (30 an hour, 100 in total) and audit-log entries (120 an hour) are limited, since each can cost- the administrator's Gemini quota or database space. Setting up two-factor from a normal session needs the+ the administrator's OpenAI quota or database space. Setting up two-factor from a normal session needs thepassword again, and every try counts toward the lockout, the right password included. A limited(set-up-only) session lasts 30 minutes, not a day.- **Sessions are stored hashed.** Redis holds only the SHA-256 of the cookie value, in the session key, thediff --git a/TESTS.md b/TESTS.mdindex fd2f135..8a840f3 100644--- a/TESTS.md+++ b/TESTS.md@@ -146,8 +146,8 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.|---|---|---|| [`test_account_security.py`](#test_account_securitypy) | 136 | Two-factor sign-in, recovery codes, encrypted secrets, sessions you can review and end, password change, and the password check before turning two-factor on. || [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |-| [`test_agent_errors.py`](#test_agent_errorspy) | 2 | Helpful, non-leaking error messages from the AI assistant. |-| [`test_agent_service.py`](#test_agent_servicepy) | 2 | The assistant can only run the fixed read-only checks. |+| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |+| [`test_agent_service.py`](#test_agent_servicepy) | 9 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. || [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. || [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. || [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |@@ -299,6 +299,9 @@ shouldn't, or an admin might be locked out.|---|---|| `test_overloaded_provider_error_is_explained_to_the_user` | When the AI provider reports it is overloaded, the user gets a readable explanation. || `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the chat or logs; it is replaced with `[redacted]`. |+| `test_openai_keys_are_redacted_too` | An OpenAI key (`sk-…`) in an error is redacted, and a rejected key is explained as such. |+| `test_a_missing_key_says_where_to_set_it` | Without a key the chat says to set `OPENAI_API_KEY`. |+| `test_an_unknown_model_points_at_agent_model` | A model the key can't use points the admin at `AGENT_MODEL`. |### test_agent_service.py@@ -306,6 +309,13 @@ shouldn't, or an admin might be locked out.|---|---|| `test_predefined_health_check_runs_and_is_audited` | When the assistant asks for one of the fixed checks (such as `server_health`), the check runs, the result is fed back to it, and both the request and the result are written to the audit log. || `test_arbitrary_command_is_rejected_before_execution` | If the assistant ever asks for anything outside the allow-list (such as `rm -rf /`), it is refused before anything runs. |+| `test_openai_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests use Structured Outputs in strict mode, the schema only allows the fixed checks, and token usage is read from the response. |+| `test_a_refusal_is_an_error_not_an_empty_reply` | When the model refuses, the chat gets an error instead of an empty answer. |+| `test_the_provider_needs_a_key` | Without `OPENAI_API_KEY` the provider is not created. |+| `test_test_connection_spends_no_tokens` | Admin's Test connection only looks the model up; it sends no prompt. |+| `test_a_conversation_sends_the_system_prompt_and_its_history` | Each request carries the system prompt and the conversation so far, in order. |+| `test_only_recent_turns_go_back_to_the_model` | Long conversations send only the most recent turns. |+| `test_a_failed_turn_is_not_kept_in_the_conversation` | A message that failed (timeout, provider error) is not remembered as part of the conversation. |### test_alerts.pydiff --git a/compose.yml b/compose.ymlindex e0d7f0d..c454399 100644--- a/compose.yml+++ b/compose.yml@@ -47,8 +47,8 @@ services:environment:DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0- GEMINI_API_KEY: ${GEMINI_API_KEY:-}- AGENT_MODEL: ${AGENT_MODEL:-models/gemini-3.5-flash}+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}# Set to true when the dashboard is served over HTTPS.SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-false}ALLOW_REGISTRATION: ${ALLOW_REGISTRATION:-false}diff --git a/docker-compose.server.yml b/docker-compose.server.ymlindex 7de7a5b..3995d54 100644--- a/docker-compose.server.yml+++ b/docker-compose.server.yml@@ -52,8 +52,8 @@ services:environment:DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-infradeck}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-infradeck}REDIS_URL: redis://:${REDIS_PASSWORD:?Set REDIS_PASSWORD in .env (generate one with openssl rand -hex 24)}@redis:6379/0- GEMINI_API_KEY: ${GEMINI_API_KEY:-}- AGENT_MODEL: ${AGENT_MODEL:-models/gemini-3.5-flash}+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}+ AGENT_MODEL: ${AGENT_MODEL:-gpt-5-mini}# Served over HTTPS via Traefik, so the login cookie is HTTPS-only.SESSION_COOKIE_SECURE: ${SESSION_COOKIE_SECURE:-true}# Only the first account can register; set to true to let others sign up.diff --git a/docs/api.md b/docs/api.mdindex 962ccc3..9b5e232 100644--- a/docs/api.md+++ b/docs/api.md@@ -27,8 +27,10 @@ All paths are relative to `/api` when going through the frontend/nginx.| `POST /monitoring/join-tokens` | admin | Create the single-use join token for **Connect a server** (shown once). || `GET /monitoring/install-info` | admin | The collector address and images the panel uses. || `GET` / `POST` / `DELETE /monitoring/enrollment-key` | admin | Show status / create or rotate / revoke the enrollment key. |-| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds. |-| `GET /agent/config` | session | Whether chat with the assistant is enabled. |+| `GET` / `PUT /admin/settings` | admin | Metrics retention, chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds; the agent's model prices, monthly budget and daily chat limit. |+| `GET /agent/config` | session | Whether chat with the agent can be used: switched on, a key configured, and the monthly budget not used up. |+| `GET /admin/agent/usage` | admin | This month's model calls, tokens and estimated cost against the budget, per model and per day, with the prices and limits. |+| `POST /admin/agent/test` | admin | Checks the OpenAI key and that the configured model exists, without spending tokens. || `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. || `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. || `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |diff --git a/docs/assistant.md b/docs/assistant.mdindex 16a2e94..41c398f 100644--- a/docs/assistant.md+++ b/docs/assistant.md@@ -1,6 +1,22 @@[← Documentation](README.md)-# Homie assistant (monitoring-only)+# Incident agent chat (monitoring-only)++The chat runs on the OpenAI API. Set `OPENAI_API_KEY` in `.env` (and optionally `AGENT_MODEL`, default+`gpt-5-mini`), then restart the backend. **Admin → Incident agent → Test connection** confirms the key works and+the model exists without spending tokens. Without a key the chat is off; incidents, notifications and the+rule-based diagnosis work the same.++**What it costs.** Every call to the model is recorded with its tokens (see `agent_runs` in+[Database schema](database.md)). In **Admin → Incident agent** you enter each model's price per million tokens (copy it+from the provider's pricing page; prices aren't fetched automatically), and the page shows the month's estimated cost,+calls, tokens and cost per model. With a **monthly budget** set, admins get a notification on their channels at 80 %,+and at 100 % the chat pauses until the 1st of next month (incidents and notifications keep working). Each person may+also send at most a set number of chat messages in 24 hours (100 by default), on top of 30 an hour.++Every reply is held to a JSON schema with OpenAI's Structured Outputs: a message, and at most one check from+the fixed list below. The provider sits behind a small interface (`src/agent/providers.py`), so other providers+can be added later.The assistant has no code path for running shell commands. It may requestexactly one fixed, read-only check at a time: `server_health`,diff --git a/docs/configuration.md b/docs/configuration.mdindex b711d05..6975e3d 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -11,8 +11,8 @@ Every setting below is listed, with no values, in [`example.env`](../example.env| `POSTGRES_PASSWORD` | *(required)* | PostgreSQL password. || `POSTGRES_DB` / `POSTGRES_USER` | `infradeck` / `infradeck` | Name of the database and its user. Installs from before the rename can keep `homelab` (see [SETUP_DATABASES.md](../SETUP_DATABASES.md)). || `REDIS_PASSWORD` | *(required)* | Redis password. Redis holds the login sessions, so nothing that can merely reach it may use it. Use letters and digits only (`openssl rand -hex 24`) because it is put into a URL. |-| `GEMINI_API_KEY` | empty | Enables the Homie chat assistant. |-| `AGENT_MODEL` | `models/gemini-3.5-flash` | Gemini model for the assistant. |+| `OPENAI_API_KEY` | empty | OpenAI API key for the incident agent's chat. Empty: chat is off; the rule-based diagnosis works without it. **Admin → Incident agent → Test connection** checks it. |+| `AGENT_MODEL` | `gpt-5-mini` | OpenAI model for the agent. It must support Structured Outputs. || `ADMIN_ALERT_WEBHOOK_URL` | empty | Receives JSON alerts for critical `daily_checkup` findings. || `SESSION_COOKIE_SECURE` | `false` (dev), `true` (server file) | Send the login cookie over HTTPS only. || `SESSION_TTL_SECONDS` | `86400` | Session lifetime without *Remember me*. |diff --git a/docs/containers.md b/docs/containers.mdindex bb1c20e..36961e4 100644--- a/docs/containers.md+++ b/docs/containers.md@@ -105,7 +105,7 @@ through the gate); on the development stack the local collector talks to the bac- registering collectors and receiving their check-ins (`/monitoring/metrics/ingest`);- the alert rules, incidents and notification channels (email/webhook, quiet hours);- statistics, rollups (summarised history) and retention;-- the optional Homie assistant, which talks to Gemini when `GEMINI_API_KEY` is set.+- the optional incident agent chat, which talks to the OpenAI API when `OPENAI_API_KEY` is set.**How it works.** A FastAPI app run by uvicorn on port 8000. At startup it creates any missing tables andcolumns in PostgreSQL, moves old Redis keys to the current prefix, and prints the first-run setup tokendiff --git a/docs/database.md b/docs/database.mdindex 6048c29..426d227 100644--- a/docs/database.md+++ b/docs/database.md@@ -15,6 +15,12 @@ recovery codes (the encrypted two-factor secret lives on the `users` row), and pthresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialisesthe `users` table on a fresh volume.+**The agent's spending.** `agent_runs` has one row per call to the language model: what it was for (`chat`, later+`diagnosis`), whether it worked, the model, input and output tokens, the estimated cost in USD, how long it took, and+who asked about which incident. The month's cost and the budget check add these rows up; a call made before its model+had a price is priced with the current table when it is counted. The prices, the monthly budget and the daily chat+limit are rows in `app_settings` (`agent_prices`, `agent_monthly_cap_usd`, `agent_daily_chat_limit`).+**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-insamples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or containerper hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.diff --git a/docs/development.md b/docs/development.mdindex 1df8782..cfbce47 100644--- a/docs/development.md+++ b/docs/development.md@@ -89,7 +89,7 @@ src/docker_proxy.py read-only Docker API proxy (standard library only)system_helper.py optional kernel-log / wtmp helper (standard library only)health_checks.py read-only checks available to the assistant- service.py, prompts.py Gemini assistant+ service.py, prompts.py, providers.py incident agent chat (OpenAI)frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are thecharts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the pagediff --git a/example.env b/example.envindex 7b4f8e3..791d57a 100644--- a/example.env+++ b/example.env@@ -220,14 +220,15 @@ SMTP_FROM=ADMIN_ALERT_WEBHOOK_URL=# =============================================================================-# AI assistant (optional)+# Incident agent (optional)# =============================================================================-# (both) Google Gemini API key for the Homie assistant. Empty: chat is off, monitoring works without it.-# Get one at https://aistudio.google.com/apikey-GEMINI_API_KEY=+# (both) OpenAI API key for the incident agent's chat. Empty: chat is off, monitoring and the rule-based+# diagnosis work without it. Create one at https://platform.openai.com/api-keys+# Admin → Incident agent → Test connection checks it without spending tokens.+OPENAI_API_KEY=-# (both) Gemini model the assistant uses. Default: models/gemini-3.5-flash+# (both) OpenAI model the agent uses. It must support Structured Outputs. Default: gpt-5-miniAGENT_MODEL=# =============================================================================diff --git a/requirements.txt b/requirements.txtindex 0df689c..a09676d 100644--- a/requirements.txt+++ b/requirements.txt@@ -2,7 +2,7 @@ bcryptcryptographyemail-validatorfastapi-google-genai+openaipsycopg[binary]python-dotenvredisdiff --git a/src/agent/providers.py b/src/agent/providers.pynew file mode 100644index 0000000..2c1d19f--- /dev/null+++ b/src/agent/providers.py@@ -0,0 +1,81 @@+"""The language model behind the agent, behind one small interface so another provider can be added later.++A provider turns a list of chat messages into one reply that matches a JSON schema, and reports how many tokens+that took. It never sees the database or the servers; the agent decides what goes into the messages.+"""+from __future__ import annotations++import os+from dataclasses import dataclass+from typing import Protocol++DEFAULT_MODEL = "gpt-5-mini"+TIMEOUT_SECONDS = 60+MAX_RETRIES = 2+++@dataclass+class Completion:+ text: str+ model: str+ input_tokens: int = 0+ output_tokens: int = 0+++class Provider(Protocol):+ name: str++ def complete(self, messages: list[dict], schema: dict, schema_name: str) -> Completion:+ """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. Returns JSON text matching ``schema``."""++ def check(self) -> dict:+ """Confirms the key works and the model exists, without spending tokens."""+++def configured_key() -> str:+ return os.getenv("OPENAI_API_KEY", "").strip()+++def configured_model() -> str:+ return os.getenv("AGENT_MODEL", "").strip() or DEFAULT_MODEL+++class OpenAIProvider:+ """The OpenAI API through Chat Completions, with Structured Outputs so a reply always parses."""++ name = "OpenAI"++ def __init__(self, api_key: str | None = None, model: str | None = None, client=None):+ key = api_key or configured_key()+ if not key and client is None:+ raise RuntimeError("OPENAI_API_KEY is not configured.")+ self.model = model or configured_model()+ if client is None:+ from openai import OpenAI # imported here so the rest of the app runs without the package+ client = OpenAI(api_key=key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)+ self.client = client++ def complete(self, messages: list[dict], schema: dict, schema_name: str) -> Completion:+ response = self.client.chat.completions.create(+ model=self.model,+ messages=messages,+ response_format={"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}},+ )+ choice = response.choices[0]+ if getattr(choice.message, "refusal", None):+ raise ValueError(f"The model refused to answer: {choice.message.refusal}")+ usage = getattr(response, "usage", None)+ return Completion(+ text=choice.message.content or "",+ model=getattr(response, "model", self.model),+ input_tokens=getattr(usage, "prompt_tokens", 0) or 0,+ output_tokens=getattr(usage, "completion_tokens", 0) or 0,+ )++ def check(self) -> dict:+ model = self.client.models.retrieve(self.model)+ return {"provider": self.name, "model": getattr(model, "id", self.model)}+++def get_provider() -> Provider:+ return OpenAIProvider()diff --git a/src/agent/service.py b/src/agent/service.pyindex b79a2d0..e3f8c46 100644--- a/src/agent/service.py+++ b/src/agent/service.py@@ -1,69 +1,99 @@from __future__ import annotationsimport json-import osimport threading+import timefrom collections.abc import Callable-from dotenv import load_dotenv-from google import genai-from google.genai import types-from .health_checks import HealthCheckRunnerfrom .prompts import SYSTEM_PROMPT+from .providers import Completion, Provider, get_providerMAX_CACHED_CHATS = 200--load_dotenv()+# Only the most recent turns go back to the model, so a long conversation can't grow every request without limit.+MAX_HISTORY_MESSAGES = 40++# Every reply is a message and at most one predefined check; Structured Outputs makes the model keep to it.+REPLY_SCHEMA = {+ "type": "object",+ "properties": {+ "message": {"type": "string"},+ "action": {+ "anyOf": [+ {"type": "object", "properties": {"check": {"type": "string", "enum": sorted(HealthCheckRunner.ALLOWED_CHECKS)}}, "required": ["check"], "additionalProperties": False},+ {"type": "null"},+ ],+ },+ },+ "required": ["message", "action"],+ "additionalProperties": False,+}+++class Reply:+ def __init__(self, text: str, completion: Completion | None = None):+ self.text = text+ self.completion = completion+++class Chat:+ """One conversation's turns, sent in full (up to MAX_HISTORY_MESSAGES) with every new message."""++ def __init__(self, provider: Provider):+ self.provider = provider+ self.history: list[dict] = []++ def send_message(self, message: str) -> Reply:+ self.history.append({"role": "user", "content": message})+ messages = [{"role": "system", "content": SYSTEM_PROMPT}, *self.history[-MAX_HISTORY_MESSAGES:]]+ try:+ completion = self.provider.complete(messages, REPLY_SCHEMA, "agent_reply")+ except Exception:+ self.history.pop() # a failed turn is not part of the conversation+ raise+ self.history.append({"role": "assistant", "content": completion.text})+ return Reply(completion.text, completion)class AgentService:"""LLM interface restricted to a small, fixed health-check capability set."""- def __init__(self, health_checks: HealthCheckRunner | None = None):- api_key = os.getenv("GEMINI_API_KEY")- if not api_key:- raise RuntimeError("GEMINI_API_KEY is not configured.")- self.client = genai.Client(api_key=api_key)+ def __init__(self, health_checks: HealthCheckRunner | None = None, provider: Provider | None = None):+ self.provider = provider or get_provider()self.health_checks = health_checks or HealthCheckRunner()# Least recently used first (a dict keeps insertion order). A chat only lives in memory so the assistant# remembers the conversation; the cache is bounded so endless new conversations can't grow it without limit.self.chats = {}self.lock = threading.Lock()+ def _new_chat(self) -> Chat:+ return Chat(self.provider)+def _get_chat(self, user_id: int, conversation_id: str):chat_key = (user_id, conversation_id)with self.lock:if chat_key not in self.chats:- self.chats[chat_key] = self.client.chats.create(- model=os.getenv("AGENT_MODEL", "models/gemini-3.5-flash"),- config=types.GenerateContentConfig(- system_instruction=SYSTEM_PROMPT,- temperature=0.2,- response_mime_type="application/json",- response_schema={- "type": "OBJECT",- "properties": {- "message": {"type": "STRING"},- "action": {- "type": "OBJECT", "nullable": True,- "properties": {"check": {"type": "STRING"}},- "required": ["check"],- },- },- "required": ["message", "action"],- },- ),- )+ self.chats[chat_key] = self._new_chat()chat = self.chats.pop(chat_key)self.chats[chat_key] = chat # now the most recently usedwhile len(self.chats) > MAX_CACHED_CHATS:del self.chats[next(iter(self.chats))]return chat- def send(self, user_id: int, conversation_id: str, message: str, command_logger: Callable[[dict], None] | None = None) -> str:+ @staticmethod+ def _ask(chat, message: str, usage_logger: Callable[[Completion, int], None] | None):+ """One call to the model; ``usage_logger`` gets its tokens and how long it took, in milliseconds."""+ started = time.monotonic()+ reply = chat.send_message(message)+ completion = getattr(reply, "completion", None)+ if usage_logger and completion is not None:+ usage_logger(completion, round((time.monotonic() - started) * 1000))+ return reply++ def send(self, user_id: int, conversation_id: str, message: str, command_logger: Callable[[dict], None] | None = None,+ usage_logger: Callable[[Completion, int], None] | None = None) -> str:chat = self._get_chat(user_id, conversation_id)- response_text, action = self._parse_agent_response(self._response_text(chat.send_message(message)))+ response_text, action = self._parse_agent_response(self._response_text(self._ask(chat, message, usage_logger)))if not action:return response_text@@ -80,13 +110,13 @@ class AgentService:"check": check_name, "level": level,"steps": [f"Completed predefined check '{check_name}' with status {result['status']}."],})- final_text, final_action = self._parse_agent_response(self._response_text(chat.send_message("[HEALTH-CHECK-RESULT]\n" + json.dumps(result))))+ final_text, final_action = self._parse_agent_response(self._response_text(self._ask(chat, "[HEALTH-CHECK-RESULT]\n" + json.dumps(result), usage_logger)))if final_action:raise ValueError("Agent must summarize a health-check result without requesting another check.")return final_textdef delete_conversation(self, user_id: int, conversation_id: str) -> None:- """Discard the provider's in-memory context after its persisted chat is deleted."""+ """Discard the in-memory context after its persisted chat is deleted."""with self.lock:self.chats.pop((user_id, conversation_id), None)@@ -118,13 +148,18 @@ class AgentService:_agent_service = None-def send_agent_message(user_id: int, conversation_id: str, message: str, command_logger=None) -> str:+def send_agent_message(user_id: int, conversation_id: str, message: str, command_logger=None, usage_logger=None) -> str:global _agent_serviceif _agent_service is None:_agent_service = AgentService()- return _agent_service.send(user_id, conversation_id, message, command_logger=command_logger)+ return _agent_service.send(user_id, conversation_id, message, command_logger=command_logger, usage_logger=usage_logger)def delete_agent_conversation(user_id: int, conversation_id: str) -> None:if _agent_service is not None:_agent_service.delete_conversation(user_id, conversation_id)+++def check_provider_connection() -> dict:+ """For Admin's "Test connection": the key works and the configured model exists. Spends no tokens."""+ return get_provider().check()diff --git a/src/backend/agent_usage.py b/src/backend/agent_usage.pynew file mode 100644index 0000000..14f757d--- /dev/null+++ b/src/backend/agent_usage.py@@ -0,0 +1,176 @@+"""What the agent's model calls cost: one row per call (AgentRun), prices per model that an admin enters, a monthly+cap that pauses the agent, and a daily limit on chat messages per person.++Prices are not fetched from the provider: they change, differ per account and aren't in the API, so the admin+copies them from the provider's pricing page. A call to a model without a price still counts its tokens.+"""+from __future__ import annotations++import json+import re+from datetime import datetime, timedelta, timezone++from sqlalchemy import func++from .models import AgentRun, AppSetting++PRICES_KEY = "agent_prices"+CAP_KEY = "agent_monthly_cap_usd"+DAILY_LIMIT_KEY = "agent_daily_chat_limit"+DEFAULT_DAILY_CHAT_LIMIT = 100+MAX_DAILY_CHAT_LIMIT = 10000+WARN_SHARE = 0.8+MAX_CAP_USD = 100000+MAX_PRICE_PER_MILLION = 10000+MAX_PRICED_MODELS = 30+MODEL_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]{0,99}")+++def month_start(now: datetime) -> datetime:+ return now.replace(day=1, hour=0, minute=0, second=0, microsecond=0)+++def _setting(db, key: str):+ setting = db.get(AppSetting, key)+ return setting.value if setting else None+++def get_prices(db) -> dict[str, dict[str, float]]:+ try:+ return validate_prices(json.loads(_setting(db, PRICES_KEY) or "{}"))+ except ValueError:+ return {}+++def validate_prices(raw) -> dict[str, dict[str, float]]:+ """{model: {"input": USD per million tokens, "output": USD per million tokens}}. Raises ValueError."""+ if not isinstance(raw, dict):+ raise ValueError("Prices must be an object of models.")+ if len(raw) > MAX_PRICED_MODELS:+ raise ValueError(f"At most {MAX_PRICED_MODELS} models can have a price.")+ prices = {}+ for model, price in raw.items():+ if not isinstance(model, str) or not MODEL_NAME.fullmatch(model.strip()):+ raise ValueError("A model name may only contain letters, digits and . _ : / -")+ if not isinstance(price, dict) or set(price) != {"input", "output"}:+ raise ValueError(f"The price of {model} needs an input and an output price.")+ values = {}+ for side in ("input", "output"):+ value = price[side]+ if isinstance(value, bool) or not isinstance(value, (int, float)) or not 0 <= value <= MAX_PRICE_PER_MILLION:+ raise ValueError(f"The {side} price of {model} must be between 0 and {MAX_PRICE_PER_MILLION} USD per million tokens.")+ values[side] = float(value)+ prices[model.strip()] = values+ return prices+++def get_cap(db) -> float | None:+ try:+ value = float(_setting(db, CAP_KEY) or "")+ except ValueError:+ return None+ return value if value > 0 else None+++def validate_cap(value) -> float | None:+ """None or 0 means no cap."""+ if value is None:+ return None+ if isinstance(value, bool) or not isinstance(value, (int, float)) or not 0 <= value <= MAX_CAP_USD:+ raise ValueError(f"The monthly budget must be between 0 and {MAX_CAP_USD} USD.")+ return float(value) or None+++def get_daily_limit(db) -> int:+ try:+ return max(1, min(MAX_DAILY_CHAT_LIMIT, int(_setting(db, DAILY_LIMIT_KEY) or DEFAULT_DAILY_CHAT_LIMIT)))+ except ValueError:+ return DEFAULT_DAILY_CHAT_LIMIT+++def validate_daily_limit(value) -> int:+ if isinstance(value, bool) or not isinstance(value, int) or not 1 <= value <= MAX_DAILY_CHAT_LIMIT:+ raise ValueError(f"The daily chat limit must be a whole number between 1 and {MAX_DAILY_CHAT_LIMIT}.")+ return value+++def price_for(prices: dict, model: str) -> dict | None:+ """The price of ``model``; a dated name such as gpt-5-mini-2026-01-01 uses the price of gpt-5-mini."""+ if model in prices:+ return prices[model]+ matches = [name for name in prices if model.startswith(f"{name}-")]+ return prices[max(matches, key=len)] if matches else None+++def cost_of(prices: dict, model: str, input_tokens: int, output_tokens: int) -> float | None:+ price = price_for(prices, model)+ if price is None:+ return None+ return (input_tokens * price["input"] + output_tokens * price["output"]) / 1_000_000+++def record(db, *, kind: str, model: str, status: str = "ok", input_tokens: int = 0, output_tokens: int = 0, duration_ms: int = 0,+ user_id: int | None = None, conversation_id: str | None = None, incident_id: int | None = None, error: str | None = None) -> AgentRun:+ run = AgentRun(kind=kind, status=status, model=model[:100], input_tokens=max(0, int(input_tokens or 0)), output_tokens=max(0, int(output_tokens or 0)),+ cost_usd=cost_of(get_prices(db), model, input_tokens or 0, output_tokens or 0), duration_ms=max(0, int(duration_ms or 0)),+ user_id=user_id, conversation_id=conversation_id, incident_id=incident_id, error=(error or None) and error[:300])+ db.add(run)+ db.commit()+ return run+++def month_cost(db, now: datetime | None = None) -> tuple[float, int]:+ """This month's estimated cost so far, and how many calls had no price to count."""+ now = now or datetime.now(timezone.utc)+ prices = get_prices(db)+ since = AgentRun.created_at >= month_start(now)+ total = float(db.query(func.coalesce(func.sum(AgentRun.cost_usd), 0)).filter(since, AgentRun.cost_usd.isnot(None)).scalar() or 0)+ unpriced = 0+ rows = db.query(AgentRun.model, func.sum(AgentRun.input_tokens), func.sum(AgentRun.output_tokens), func.count(AgentRun.id)) \+ .filter(since, AgentRun.cost_usd.is_(None)).group_by(AgentRun.model).all()+ for model, input_tokens, output_tokens, count in rows:+ later = cost_of(prices, model, input_tokens or 0, output_tokens or 0)+ if later is None:+ unpriced += count if (input_tokens or output_tokens) else 0+ else:+ total += later+ return total, unpriced+++def budget(db, now: datetime | None = None) -> dict:+ """{"cost", "cap", "share", "state"}: state is ok, warn (from 80 % of the cap) or paused (the cap is reached)."""+ cost, unpriced = month_cost(db, now)+ cap = get_cap(db)+ share = cost / cap if cap else None+ state = "paused" if share is not None and share >= 1 else "warn" if share is not None and share >= WARN_SHARE else "ok"+ return {"cost": round(cost, 6), "cap": cap, "share": share, "state": state, "unpriced_runs": unpriced}+++def month_summary(db, now: datetime | None = None) -> dict:+ """Admin's usage card: totals, per model, and cost per day of this month."""+ now = now or datetime.now(timezone.utc)+ start = month_start(now)+ prices = get_prices(db)+ runs = db.query(AgentRun).filter(AgentRun.created_at >= start).all()+ by_model: dict[str, dict] = {}+ days: dict[str, float] = {}+ totals = {"calls": 0, "errors": 0, "chat_calls": 0, "diagnoses": 0, "input_tokens": 0, "output_tokens": 0}+ for run in runs:+ cost = run.cost_usd if run.cost_usd is not None else cost_of(prices, run.model, run.input_tokens, run.output_tokens)+ totals["calls"] += 1+ totals["errors"] += run.status != "ok"+ totals["chat_calls"] += run.kind == "chat"+ totals["diagnoses"] += run.kind == "diagnosis" and run.status == "ok"+ totals["input_tokens"] += run.input_tokens+ totals["output_tokens"] += run.output_tokens+ entry = by_model.setdefault(run.model, {"model": run.model, "calls": 0, "input_tokens": 0, "output_tokens": 0, "cost": 0.0, "priced": price_for(prices, run.model) is not None})+ entry["calls"] += 1+ entry["input_tokens"] += run.input_tokens+ entry["output_tokens"] += run.output_tokens+ entry["cost"] += cost or 0+ day = run.created_at.astimezone(timezone.utc).date().isoformat()+ days[day] = days.get(day, 0) + (cost or 0)+ span = (now.date() - start.date()).days + 1+ daily = [{"day": (start + timedelta(days=offset)).date().isoformat(), "cost": round(days.get((start + timedelta(days=offset)).date().isoformat(), 0), 6)} for offset in range(span)]+ return {"month": start.date().isoformat()[:7], **totals, **budget(db, now), "by_model": sorted(by_model.values(), key=lambda entry: -entry["cost"]), "daily": daily,+ "prices": prices, "daily_chat_limit": get_daily_limit(db), "warn_share": WARN_SHARE}diff --git a/src/backend/config.py b/src/backend/config.pyindex 5d3816d..8a2e637 100644--- a/src/backend/config.py+++ b/src/backend/config.py@@ -19,7 +19,8 @@ DATABASE_URL = os.getenv()REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")SESSION_TTL_SECONDS = int(os.getenv("SESSION_TTL_SECONDS", "86400"))-AGENT_MODEL = os.getenv("AGENT_MODEL", "models/gemini-3.5-flash")+# The OpenAI model the agent uses (see src/agent/providers.py).+AGENT_MODEL = os.getenv("AGENT_MODEL", "").strip() or "gpt-5-mini"SESSION_COOKIE_NAME = os.getenv("SESSION_COOKIE_NAME", "homie_session")SESSION_COOKIE_SECURE = os.getenv("SESSION_COOKIE_SECURE", "false").lower() == "true"# When false (the default), only the first account can be created, so a fresh install can still be set up anddiff --git a/src/backend/main.py b/src/backend/main.pyindex 64883be..371614c 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -27,7 +27,7 @@ from sqlalchemy import case, func, inspect, textfrom .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alertsimport contextvars-from . import auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totp+from . import agent_usage, auth, dashboard, history as history_service, incidents, notify, rollups, secrets_box, stats, totpfrom .auth import (STAGE_ENROLLMENT, STAGE_FULL, count_pending_attempt, create_pending_two_factor, create_session, delete_pending_two_factor, delete_session, get_recent_logs,get_session, get_user_from_session, hash_password, issue_recovery_codes, list_user_sessions, pending_two_factor_remember, read_pending_two_factor, session_lifetime, recovery_codes_left, record_audit_log,@@ -64,7 +64,8 @@ from .config import ()from .rules import HELPER_EVENT_KINDS, DEFAULT_THRESHOLDS, HOST_RULE_PREFIX, container_rules, host_rules, validate_thresholds, detect_system_events, system_rules, SYSTEM_RULE_PREFIX, validate_mount_thresholds, validate_helper_events, REBOOT_DETAILS, REBOOT_KEY_PREFIX, detect_container_events, memory_trend_rules, CONTAINER_EVENT_KINDS, MAX_CONTAINER_EVENTS_PER_CHECKIN, MAX_CONTAINER_EVENTS_PER_HOUR, TREND_RULE_PREFIX, NETWORK_EVENT_KINDS, NETWORK_RULE_PREFIX, MAX_NETWORK_EVENTS_PER_CHECKIN, MAX_NETWORK_EVENTS_PER_HOUR, INTERNET_STATUSES, LISTENER_SCOPES, LISTENER_PROTOCOLS, MAX_LISTENERS, FIREWALL_LABELS, detect_listener_changes, detect_firewall_changes, network_rules, validate_listeners, validate_firewallfrom .rate_limit import enforce_ip_limit, enforce_limit, enforce_login_limit, enforce_register_limit, refund_attempt, reset_limits-from ..agent.service import delete_agent_conversation, send_agent_message+from ..agent.providers import configured_key+from ..agent.service import check_provider_connection, delete_agent_conversation, send_agent_messageclass RegisterRequest(BaseModel):@@ -187,6 +188,11 @@ class AdminSettingsRequest(BaseModel):disabled_rules: list[str] | None = Field(default=None, max_length=100)# What this install calls itself on the sign-in page and in the sidebar; an empty string goes back to the default.instance_name: str | None = Field(default=None, max_length=200)+ # The agent's spending: USD per million tokens per model (replacing the whole table), a monthly budget in USD+ # (0 removes it) and how many chat messages one person may send in 24 hours.+ agent_prices: dict | None = None+ agent_monthly_cap_usd: float | None = None+ agent_daily_chat_limit: int | None = Noneclass MetricsIngestRequest(BaseModel):@@ -200,21 +206,27 @@ def agent_error_message(exc: Exception) -> str:"""Turn provider failures into useful, non-sensitive chat text."""raw_message = str(exc).strip()normalized = raw_message.lower()- if any(marker in normalized for marker in ("resourceexhausted", "overloaded", "capacity", " 503", "status 503")):- prefix = "Gemini is currently overloaded or unavailable. Please try again shortly."- elif any(marker in normalized for marker in ("rate limit", "quota", " 429", "status 429")):- prefix = "Gemini rate limit or quota was reached. Please try again later."+ if "openai_api_key is not configured" in normalized:+ return "The agent has no API key. An admin sets OPENAI_API_KEY in .env and restarts the backend."+ if any(marker in normalized for marker in ("401", "invalid_api_key", "incorrect api key", "authentication")):+ prefix = "OpenAI rejected the API key. Check OPENAI_API_KEY in .env."+ elif any(marker in normalized for marker in ("model_not_found", "does not exist", "404")):+ prefix = "OpenAI doesn't offer the configured model to this key. Check AGENT_MODEL in .env."+ elif any(marker in normalized for marker in ("resourceexhausted", "overloaded", "capacity", " 503", "status 503", " 500", " 502")):+ prefix = "OpenAI is currently overloaded or unavailable. Please try again shortly."+ elif any(marker in normalized for marker in ("rate limit", "quota", " 429", "status 429", "insufficient_quota")):+ prefix = "OpenAI rate limit or quota was reached. Please try again later."elif any(marker in normalized for marker in ("timeout", "timed out", "deadline exceeded")):- prefix = "Gemini did not respond before the request timed out. Please try again."+ prefix = "OpenAI did not respond before the request timed out. Please try again."else:- prefix = "Gemini could not complete this request."+ prefix = "OpenAI could not complete this request."if not raw_message:return prefix# Errors can include request headers or provider diagnostics: keep a short,# redacted detail for the administrator without leaking credentials.detail = re.sub(r"(?i)(api[_ -]?key|authorization|token)\s*[:=]\s*\S+", r"\1=[redacted]", raw_message)- detail = re.sub(r"AIza[\w-]+", "[redacted]", detail)[:300]+ detail = re.sub(r"(AIza|sk-)[\w-]+", "[redacted]", detail)[:300]return f"{prefix} Provider error: {detail}"@@ -987,7 +999,8 @@ def get_mount_thresholds(db) -> dict:def _admin_settings(db) -> dict:return {"instance_name": get_instance_name(db), "metrics_retention_days": get_metrics_retention_days(db), "agent_chat_enabled": is_agent_chat_enabled(db), "alert_thresholds": get_alert_thresholds(db), "mount_thresholds": get_mount_thresholds(db), "require_2fa_for_admins": require_admin_two_factor(db), "secrets_configured": secrets_box.available(),- "require_2fa_for_everyone": require_two_factor_for_everyone(db), "disabled_rules": sorted(get_disabled_rules(db))}+ "require_2fa_for_everyone": require_two_factor_for_everyone(db), "disabled_rules": sorted(get_disabled_rules(db)),+ "agent_prices": agent_usage.get_prices(db), "agent_monthly_cap_usd": agent_usage.get_cap(db), "agent_daily_chat_limit": agent_usage.get_daily_limit(db)}def purge_old_metrics(db, now: datetime | None = None) -> int:@@ -1040,6 +1053,24 @@ def update_admin_settings(payload: AdminSettingsRequest, session_id: Annotated[sif set(wanted) != get_disabled_rules(db):_set_setting(db, DISABLED_RULES_KEY, json.dumps(wanted))changes.append("Switched off alert rules: " + (", ".join(wanted) or "none"))+ try:+ if payload.agent_prices is not None:+ prices = agent_usage.validate_prices(payload.agent_prices)+ if prices != agent_usage.get_prices(db):+ _set_setting(db, agent_usage.PRICES_KEY, json.dumps(prices))+ changes.append("Changed the agent's model prices: " + (", ".join(f"{model} {price['input']:g}/{price['output']:g}" for model, price in prices.items()) or "none"))+ if payload.agent_monthly_cap_usd is not None:+ cap = agent_usage.validate_cap(payload.agent_monthly_cap_usd)+ if cap != agent_usage.get_cap(db):+ _set_setting(db, agent_usage.CAP_KEY, str(cap or 0))+ changes.append(f"Set the agent's monthly budget to ${cap:g}" if cap else "Removed the agent's monthly budget")+ if payload.agent_daily_chat_limit is not None:+ limit = agent_usage.validate_daily_limit(payload.agent_daily_chat_limit)+ if limit != agent_usage.get_daily_limit(db):+ _set_setting(db, agent_usage.DAILY_LIMIT_KEY, str(limit))+ changes.append(f"Set the agent chat limit to {limit} messages per person per day")+ except ValueError as exc:+ raise HTTPException(status_code=400, detail=str(exc)) from excif payload.agent_chat_enabled is not None:_set_setting(db, AGENT_CHAT_ENABLED_KEY, "true" if payload.agent_chat_enabled else "false")changes.append(f"{'Enabled' if payload.agent_chat_enabled else 'Disabled'} chat with the agent")@@ -1071,7 +1102,8 @@ def update_admin_settings(payload: AdminSettingsRequest, session_id: Annotated[sdef agent_config(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):"""What the Agent page needs to know about the agent; readable by every signed-in user."""_current_user(session_id, db)- return {"chat_enabled": is_agent_chat_enabled(db)}+ spend = agent_usage.budget(db)+ return {"chat_enabled": is_agent_chat_enabled(db), "key_configured": bool(configured_key()), "budget_paused": spend["state"] == "paused"}OVERVIEW_ALERT_LIMIT = 50@@ -2490,7 +2522,7 @@ def delete_conversation(conversation_id: str, session_id: Annotated[str | None,LOG_WRITE_LIMIT = 120-CHAT_MESSAGE_LIMIT = 30 # messages to the assistant per user per hour (each can cost the admin's Gemini quota)+CHAT_MESSAGE_LIMIT = 30 # messages to the assistant per user per hour (each can cost the admin's OpenAI quota)CONVERSATION_CREATE_LIMIT = 30 # new conversations per user per hourMAX_CONVERSATIONS_PER_USER = 100@@ -2527,6 +2559,10 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cookif not is_agent_chat_enabled(db):raise HTTPException(status_code=403, detail="Chat with the agent has been turned off by an admin.")enforce_limit(redis_client, scope="chat-user", subject=user["id"], limit=CHAT_MESSAGE_LIMIT, window_seconds=3600)+ spend = agent_usage.budget(db)+ if spend["state"] == "paused":+ raise HTTPException(status_code=429, detail=f"The agent's budget for this month is used up (${spend['cost']:.2f} of ${spend['cap']:.2f}). An admin can raise it in Admin › Incident agent.")+ enforce_limit(redis_client, scope="chat-user-day", subject=user["id"], limit=agent_usage.get_daily_limit(db), window_seconds=86400)message = payload.message.strip()if not message:@@ -2539,11 +2575,16 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cookconversation.title = message[:255]db.commit()+ def log_usage(completion, duration_ms: int) -> None:+ agent_usage.record(db, kind="chat", model=completion.model, input_tokens=completion.input_tokens, output_tokens=completion.output_tokens, duration_ms=duration_ms,+ user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id)+try:response_text = send_agent_message(user["id"],payload.conversation_id,_incident_context(db, payload.incident_id) + message,+ usage_logger=log_usage,command_logger=lambda event: record_audit_log(db,user_id=user["id"],@@ -2558,6 +2599,7 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cook)except Exception as exc:user_message = agent_error_message(exc)+ agent_usage.record(db, kind="chat", status="error", model=AGENT_MODEL, user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id, error=user_message)record_audit_log(db,user_id=user["id"],@@ -2574,6 +2616,7 @@ def agent_chat(payload: AgentChatRequest, session_id: Annotated[str | None, Cookraise HTTPException(status_code=503, detail={"message": user_message}) from excsteps = ["Received the task and evaluated the request.", "Generated a structured response for the requested task."]+ _warn_about_budget(db, spend)record_audit_log(db,@@ -2962,6 +3005,26 @@ def dispatch_notifications(db, now: datetime | None = None) -> int:return sent+def _warn_about_budget(db, before: dict) -> None:+ """Tell the admins once a month when the agent passes 80 % of its budget, and again when it pauses at 100 %."""+ after = agent_usage.budget(db)+ rank = {"ok": 0, "warn": 1, "paused": 2}+ if rank[after["state"]] <= rank[before["state"]] or not after["cap"]:+ return+ try:+ if not redis_client.set(f"infradeck:agent-budget:{datetime.now(timezone.utc):%Y-%m}:{after['state']}", "1", nx=True, ex=40 * 86400):+ return+ except Exception:+ return+ paused = after["state"] == "paused"+ title = "The incident agent paused: its monthly budget is used up" if paused else f"The incident agent has used {round(after['share'] * 100)}% of its monthly budget"+ body = f"${after['cost']:.2f} of ${after['cap']:.2f} this month. {'Chat stays off until next month or until an admin raises the budget' if paused else 'At 100% the agent pauses until next month'}. Incidents and notifications keep working. Admin › Incident agent."+ record_audit_log(db, user_id=None, level="WARN", message=title[:255], source="agent", event_type="agent_budget")+ admins = {user.id: user for user in db.query(User).filter(User.role == "admin", User.is_active.is_(True))}+ for channel in db.query(NotificationChannel).filter(NotificationChannel.user_id.in_(list(admins))).order_by(NotificationChannel.id):+ _deliver(db, channel, admins[channel.user_id], title, body, "WARNING")++def _notify_loop() -> None:while True:time.sleep(15)@@ -3158,13 +3221,32 @@ def admin_system(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_N"version": APP_VERSION, "database": {"engine": engine.dialect.name, "bytes": size,"host_rows": db.query(func.count(HostMetric.id)).scalar(), "host_rows_week": db.query(func.count(HostMetric.id)).filter(HostMetric.time >= week).scalar()},"retention_days": get_metrics_retention_days(db),- "agent": {"model": AGENT_MODEL, "key_configured": bool(os.getenv("GEMINI_API_KEY", "").strip()), "chat_enabled": is_agent_chat_enabled(db),+ "agent": {"provider": "OpenAI", "model": AGENT_MODEL, "key_configured": bool(configured_key()), "budget": agent_usage.budget(db), "chat_enabled": is_agent_chat_enabled(db),"incidents_this_month": db.query(IncidentAlert).filter(IncidentAlert.created_at >= month).count(),"chat_messages_this_month": db.query(ChatMessage).filter(ChatMessage.role == "user", ChatMessage.created_at >= month).count()},"smtp_configured": notify.smtp_configured(), "secrets_configured": secrets_box.available(),}[email protected]("/admin/agent/usage")+def admin_agent_usage(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+ """This month's model calls, tokens and estimated cost against the budget, per model and per day."""+ _require_admin(session_id, db)+ return {**agent_usage.month_summary(db), "model": AGENT_MODEL}++[email protected]("/admin/agent/test")+def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+ """Admin's "Test connection": the API key works and the configured model exists. Spends no tokens."""+ admin = _require_admin(session_id, db)+ enforce_limit(redis_client, scope="agent-test", subject=admin["id"], limit=10, window_seconds=600)+ try:+ result = check_provider_connection()+ except Exception as exc:+ raise HTTPException(status_code=502, detail=agent_error_message(exc)) from exc+ return {"ok": True, **result}++@app.get("/")def root():return {"message": "Infradeck API"}diff --git a/src/backend/models.py b/src/backend/models.pyindex e247ad3..f4574d3 100644--- a/src/backend/models.py+++ b/src/backend/models.py@@ -492,3 +492,27 @@ class Invite(Base):created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)used_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)+++class AgentRun(Base):+ """One call to the agent's language model: what it was for, which model, how many tokens and what that cost.++ The month's spending and the cap come from these rows. ``cost_usd`` is set when the model had a price at the+ time; rows without one are priced with the current table when they are added up."""++ __tablename__ = "agent_runs"+ __table_args__ = (Index("ix_agent_runs_created", "created_at"),)++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ kind: Mapped[str] = mapped_column(String(16), nullable=False) # chat | diagnosis+ status: Mapped[str] = mapped_column(String(16), nullable=False) # ok | error+ model: Mapped[str] = mapped_column(String(100), nullable=False)+ input_tokens: Mapped[int] = mapped_column(Integer, default=0, nullable=False)+ output_tokens: Mapped[int] = mapped_column(Integer, default=0, nullable=False)+ cost_usd: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ duration_ms: Mapped[int] = mapped_column(Integer, default=0, nullable=False)+ user_id: Mapped[Optional[int]] = mapped_column(ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True)+ conversation_id: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)+ incident_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)+ error: Mapped[Optional[str]] = mapped_column(String(300), nullable=True)+ created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)diff --git a/src/frontend/src/admin/AdminPage.jsx b/src/frontend/src/admin/AdminPage.jsxindex 97fd0af..a15f2ec 100644--- a/src/frontend/src/admin/AdminPage.jsx+++ b/src/frontend/src/admin/AdminPage.jsx@@ -6,6 +6,7 @@ import { apiRequest } from '../api.js'import { AnotherMachineCard, ThisMachineCard } from '../dashboard/EmptyFleet.jsx'import { INTERVALS, intervalLabel } from '../dashboard/ServerPanel.jsx'import { serverState } from '../dashboard/logic.js'+import { Sparkline } from '../dashboard/charts.jsx'import { Icon } from '../icons.jsx'import { refreshOverview, useOverview } from '../overviewStore.js'import { AppShell, serverLink } from '../shell.jsx'@@ -274,19 +275,113 @@ function MountOverrides({ settings, onSaved }) {function Agent({ settings, system, onSaved }) {const [state, run] = useAction()+ const [test, runTestAction] = useAction()+ const [tested, setTested] = useState(null)+ const runTest = (work) => { setTested(null); return runTestAction(async () => setTested(await work())) }const agent = system.agentreturn <><div className="card divide-y divide-line">- <Row icon="AI" title="Model provider" detail={agent.key_configured ? `Google Gemini · ${agent.model.replace('models/', '')} · key configured` : 'No API key configured. Set GEMINI_API_KEY in .env to turn the chat on.'} tone={agent.key_configured ? undefined : 'warn'}><button type="button" className="btn btn-sm" disabled title="The provider and key are set in .env (AGENT_MODEL, GEMINI_API_KEY) and need a restart.">Change</button></Row>+ <Row icon="AI" title="Model provider" detail={agent.key_configured ? `${agent.provider || 'OpenAI'} · ${agent.model} · key configured` : 'No API key configured. Set OPENAI_API_KEY in .env and restart the backend to turn the chat on.'} tone={agent.key_configured ? undefined : 'warn'}>+ <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key and the model without spending tokens.' : 'Set OPENAI_API_KEY first.'} onClick={() => runTest(() => apiRequest('/admin/agent/test', { method: 'POST' }))}>{test.busy ? 'Testing…' : 'Test connection'}</button>+ </Row>+ {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key and offers {tested.model}.</p>}</div>}<Row title="Diagnose new incidents automatically" detail="Always on. The diagnosis is built from stored metrics and events by fixed rules, so it costs nothing and sends nothing to the provider."><Switch label="Diagnose automatically" checked disabled onChange={() => {}} /></Row><Row title="Read container logs" detail="Not available: Infradeck's Docker proxy can't read logs. See SECURITY.md."><Switch label="Read container logs" checked={false} disabled onChange={() => {}} /></Row><Row title="Chat" detail="Available to every signed-in user while it is on. Questions and the incident's facts are sent to the provider."><Switch label="Chat with the agent" checked={settings.agent_chat_enabled} disabled={state.busy} onChange={(on) => run(async () => onSaved(await put({ agent_chat_enabled: on })))} /></Row><Row title="Redact IPs and hostnames before sending" detail="Not available yet. Until then, turn Chat off if names must not leave this machine."><Switch label="Redact before sending" checked={false} disabled onChange={() => {}} /></Row></div>+ <Outcome state={state} />+ <AgentSpending settings={settings} onSaved={onSaved} />+ </>+}++// "$0.0031" for small amounts, "$1.84" from a cent up.+const dollars = (value) => (value == null ? '—' : `$${value > 0 && value < 0.01 ? value.toFixed(4) : value.toFixed(2)}`)+const thousands = (value) => (value >= 1e6 ? `${(value / 1e6).toFixed(1)} M` : value >= 1e4 ? `${Math.round(value / 1e3)} k` : value.toLocaleString())++// What the model calls cost this month, against the budget, and the settings that limit it.+function AgentSpending({ settings, onSaved }) {+ const { data, error, reload } = useLoaded('/admin/agent/usage')+ const [state, run] = useAction()+ const [cap, setCap] = useState(settings.agent_monthly_cap_usd ?? '')+ const [limit, setLimit] = useState(String(settings.agent_daily_chat_limit ?? 100))+ const [prices, setPrices] = useState(() => Object.entries(settings.agent_prices || {}).map(([model, price]) => ({ model, input: String(price.input), output: String(price.output) })))+ if (!data) return <Notice tone="plain">{error || 'Loading the agent\'s usage…'}</Notice>+ const save = (body, done) => run(async () => { onSaved(await put(body)); reload() }, done)+ const savePrices = (event) => {+ event.preventDefault()+ save({ agent_prices: Object.fromEntries(prices.filter((row) => row.model.trim()).map((row) => [row.model.trim(), { input: Number(row.input), output: Number(row.output) }])) }, 'Prices saved.')+ }+ const setRow = (index, key, value) => setPrices((rows) => rows.map((row, position) => (position === index ? { ...row, [key]: value } : row)))+ const share = data.share == null ? null : Math.min(1, data.share)+ const daily = data.daily.map((day) => day.cost)+ const configuredPriced = prices.some((row) => row.model.trim() && (data.model === row.model.trim() || data.model.startsWith(`${row.model.trim()}-`)))+ return <>+ {data.state === 'paused' && <Notice>The agent is paused: this month's budget is used up ({dollars(data.cost)} of {dollars(data.cap)}). Chat stays off until next month or until you raise the budget. Incidents and notifications keep working.</Notice>}+ {data.state === 'warn' && <Notice>The agent has used {Math.round(data.share * 100)}% of this month's budget ({dollars(data.cost)} of {dollars(data.cap)}). At 100% it pauses until next month.</Notice>}<div className="card grid divide-y divide-line sm:grid-cols-3 sm:divide-x sm:divide-y-0">- {[[agent.incidents_this_month, 'incidents diagnosed this month'], [agent.chat_messages_this_month, 'chat messages this month'], ['—', 'provider cost · not tracked']].map(([value, label]) => <div key={label} className="px-[18px] py-4"><div className="text-2xl font-semibold leading-none text-paper">{value}</div><div className="mt-2 text-xs text-muted">{label}</div></div>)}+ <div className="px-[18px] py-4">+ <div className="text-2xl font-semibold leading-none text-paper">{dollars(data.cost)}{data.cap && <span className="ml-1.5 text-sm font-normal text-muted">of {dollars(data.cap)}</span>}</div>+ <div className="mt-2 text-xs text-muted">estimated cost this month{data.unpriced_runs ? ` · ${data.unpriced_runs} ${data.unpriced_runs === 1 ? 'call' : 'calls'} without a price` : ''}</div>+ {share != null && <span className={`meter mt-3 ${data.state === 'paused' ? 'meter-signal' : data.state === 'warn' ? 'meter-warn' : ''}`} role="meter" aria-label="Share of the monthly budget" aria-valuemin={0} aria-valuemax={100} aria-valuenow={Math.round(share * 100)}><i style={{ width: `${share * 100}%`, minWidth: share > 0 ? 3 : 0 }} /></span>}+ </div>+ <div className="px-[18px] py-4">+ <div className="text-2xl font-semibold leading-none text-paper">{data.calls}</div>+ <div className="mt-2 text-xs text-muted">calls to the model this month{data.errors ? ` · ${data.errors} failed` : ''}{data.diagnoses ? ` · ${data.diagnoses} diagnoses` : ''}</div>+ {daily.some((value) => value > 0) && <Sparkline className="mt-3" values={daily} width={140} height={18} max={Math.max(...daily)} />}+ </div>+ <div className="px-[18px] py-4">+ <div className="text-2xl font-semibold leading-none text-paper">{thousands(data.input_tokens + data.output_tokens)}</div>+ <div className="mt-2 text-xs text-muted">tokens · {thousands(data.input_tokens)} in, {thousands(data.output_tokens)} out</div>+ </div>+ </div>++ {data.by_model.length > 0 && <div className="card overflow-hidden"><div className="custom-scroll overflow-x-auto"><table className="w-full min-w-[460px] border-collapse text-left text-[13px]">+ <thead><tr className="grid-head h-10"><th className="pl-[18px] font-medium">Model</th><th className="text-right font-medium">Calls</th><th className="text-right font-medium">Tokens in / out</th><th className="pr-[18px] text-right font-medium">Cost</th></tr></thead>+ <tbody>{data.by_model.map((row) => (+ <tr key={row.model} className="h-10 border-t border-line">+ <td className="mono pl-[18px] text-paper">{row.model}</td>+ <td className="text-right text-soft">{row.calls}</td>+ <td className="text-right text-soft">{thousands(row.input_tokens)} / {thousands(row.output_tokens)}</td>+ <td className={`pr-[18px] text-right ${row.priced ? 'text-paper' : 'text-warn'}`}>{row.priced ? dollars(row.cost) : 'no price'}</td>+ </tr>+ ))}</tbody>+ </table></div></div>}++ <div className="card divide-y divide-line">+ <Row title="Monthly budget" detail="At 80% admins get a notification; at 100% the chat pauses until the 1st of next month. Empty: no budget.">+ <form className="flex items-center gap-2" onSubmit={(event) => { event.preventDefault(); save({ agent_monthly_cap_usd: cap === '' ? 0 : Number(cap) }, 'Budget saved.') }}>+ <span className="text-[13px] text-muted">$</span>+ <input type="number" min="0" step="0.01" inputMode="decimal" className="field h-9 w-24" aria-label="Monthly budget in US dollars" value={cap} placeholder="none" onChange={(event) => setCap(event.target.value)} />+ <button type="submit" className="btn btn-sm h-9" disabled={state.busy || String(cap) === String(settings.agent_monthly_cap_usd ?? '')}>Save</button>+ </form>+ </Row>+ <Row title="Chat messages per person" detail="In any 24 hours, on top of 30 an hour. Admins included.">+ <form className="flex items-center gap-2" onSubmit={(event) => { event.preventDefault(); save({ agent_daily_chat_limit: Number(limit) }, 'Limit saved.') }}>+ <input type="number" min="1" step="1" inputMode="numeric" className="field h-9 w-24" aria-label="Chat messages per person per day" value={limit} onChange={(event) => setLimit(event.target.value)} />+ <button type="submit" className="btn btn-sm h-9" disabled={state.busy || limit === String(settings.agent_daily_chat_limit)}>Save</button>+ </form>+ </Row>+ <form onSubmit={savePrices} className="px-[18px] py-3.5">+ <div className="text-[13px] font-medium text-paper">Prices per million tokens</div>+ <p className="mb-0 mt-0.5 text-xs leading-5 text-muted">Copy them from your provider's pricing page; they aren't fetched automatically. A price for <span className="mono">gpt-5-mini</span> also covers dated names such as <span className="mono">gpt-5-mini-2026-01-01</span>.{!configuredPriced && <span className="text-warn"> The model in use, <span className="mono">{data.model}</span>, has no price, so its calls aren't counted in the budget.</span>}</p>+ <div className="mt-3 space-y-2">+ {prices.map((row, index) => (+ <div key={index} className="flex flex-wrap items-end gap-2">+ <Field label="Model" className="min-w-0 flex-1 basis-40"><input className="field mono h-9" required maxLength={100} value={row.model} onChange={(event) => setRow(index, 'model', event.target.value)} /></Field>+ <Field label="Input $"><input type="number" min="0" step="any" required inputMode="decimal" className="field h-9 w-24" value={row.input} onChange={(event) => setRow(index, 'input', event.target.value)} /></Field>+ <Field label="Output $"><input type="number" min="0" step="any" required inputMode="decimal" className="field h-9 w-24" value={row.output} onChange={(event) => setRow(index, 'output', event.target.value)} /></Field>+ <button type="button" className="btn btn-ghost h-9" aria-label={`Remove the price of ${row.model || 'this model'}`} onClick={() => setPrices((rows) => rows.filter((_, position) => position !== index))}><Icon name="close" size={13} /></button>+ </div>+ ))}+ </div>+ <div className="mt-3 flex flex-wrap items-center gap-2.5">+ <button type="button" className="btn btn-sm" onClick={() => setPrices((rows) => [...rows, { model: rows.some((item) => item.model === data.model) ? '' : data.model, input: '', output: '' }])}><Icon name="plus" size={13} />Add a model</button>+ <button type="submit" className="btn btn-sm btn-primary" disabled={state.busy}>Save prices</button>+ <Outcome state={state} />+ </div>+ </form></div>- <Outcome state={state} /></>}@@ -365,7 +460,7 @@ function PhoneHub({ overview, settings, system }) {<div className="divide-y divide-line overflow-hidden rounded-md border border-line bg-ink">{row('/admin?section=users', 'Users', null, users ? `${users.users.length}${invited ? ` · ${invited} invited` : ''}` : null)}{row('/admin?section=rules', 'Alert rules', null, rules ? `${on} on · ${rules.rules.length - on} off` : null)}- {row('/admin?section=agent', 'Incident agent', null, system.agent.key_configured ? (settings.agent_chat_enabled ? 'chat on' : 'chat off') : 'no key', undefined, system.agent.key_configured ? undefined : 'warn')}+ {row('/admin?section=agent', 'Incident agent', null, !system.agent.key_configured ? 'no key' : system.agent.budget.cap ? `${dollars(system.agent.budget.cost)} of ${dollars(system.agent.budget.cap)}` : `${dollars(system.agent.budget.cost)} this month`, undefined, !system.agent.key_configured || system.agent.budget.state !== 'ok' ? 'warn' : undefined)}{row('/admin?section=retention', 'Data and retention', null, megabytes(system.database.bytes))}{row('/admin?section=about', 'About', null, system.version)}</div>diff --git a/src/frontend/src/agent/AgentPage.jsx b/src/frontend/src/agent/AgentPage.jsxindex 8f1b7e8..2b2a44c 100644--- a/src/frontend/src/agent/AgentPage.jsx+++ b/src/frontend/src/agent/AgentPage.jsx@@ -160,7 +160,7 @@ function Incident({ incident, canAct, onChanged }) {const SUGGESTIONS = { default: ['Is anything else close to its limit?', 'What changed on this server today?', 'Explain this incident in one paragraph'], none: ['Is anything close to its limit?', 'Which server needs attention first?', 'What happened overnight?'] }-function Chat({ user, incident, enabled, phone = false, back }) {+function Chat({ user, incident, enabled, offReason, phone = false, back }) {const conversationId = `${incident ? `incident-${incident.id}` : 'general'}-u${user.id}`const [messages, setMessages] = useState(null)const [draft, setDraft] = useState('')@@ -204,7 +204,7 @@ function Chat({ user, incident, enabled, phone = false, back }) {return <><PhoneHeader back={back} title="Ask the agent" action={incident ? <span className="chip mono shrink-0 text-paper">{incident.reference}</span> : <button type="button" className="h-11 border-0 bg-transparent px-1 text-[13px] text-muted" onClick={restart}>New chat</button>} /><div className="space-y-5 px-4 pb-[190px] pt-4" aria-live="polite">- {!enabled && <Notice tone="plain">Chat with the agent has been turned off by an admin.</Notice>}+ {!enabled && <Notice tone="plain">{offReason}</Notice>}{enabled && messages === null && <p className="m-0 text-xs text-muted">Loading…</p>}{enabled && messages?.length === 0 && <p className="m-0 text-[13px] leading-5 text-muted">{incident ? 'The agent is given this incident\'s facts with your question.' : 'Ask about any server.'}</p>}{enabled && (messages || []).map((message) => (message.role === 'user'@@ -232,7 +232,7 @@ function Chat({ user, incident, enabled, phone = false, back }) {<div className="mt-2 flex items-center gap-2 text-xs text-muted">About <span className="chip max-w-full truncate text-paper">{incident ? `${incident.reference} · ${incident.hostname}` : 'any server'}</span></div></div><div className="custom-scroll min-h-0 flex-1 space-y-4 overflow-y-auto px-5 py-4">- {!enabled && <Notice tone="plain">Chat with the agent has been turned off by an admin. The diagnosis on the left doesn't need it.</Notice>}+ {!enabled && <Notice tone="plain">{offReason} The diagnosis on the left doesn't need it.</Notice>}{enabled && messages === null && <p className="m-0 text-xs text-muted">Loading…</p>}{enabled && messages?.length === 0 && <div className="space-y-2"><p className="m-0 text-xs text-muted">{incident ? 'The agent is given this incident\'s facts with your question.' : 'Ask about any server.'} Suggestions:</p>@@ -401,13 +401,17 @@ export default function AgentPage({ user, onLogout }) {useEffect(() => { if (selected) setTab(selected.status === 'resolved' ? 'resolved' : selected.muted_until ? 'muted' : 'open') }, [selected?.id]) // eslint-disable-line react-hooks/exhaustive-depsconst list = inbox ? inbox[tab] : []const crumbs = selected ? [{ label: 'Agent', to: '/agent' }, { label: selected.reference }] : [{ label: 'Agent' }]+ // Why the chat can't be used right now, or null when it can.+ const chatOff = !config ? null : !config.chat_enabled ? 'Chat with the agent has been turned off by an admin.'+ : config.key_configured === false ? 'The agent has no API key yet. An admin sets OPENAI_API_KEY in .env.'+ : config.budget_paused ? 'This month\'s budget for the agent is used up. An admin can raise it in Admin › Incident agent.' : nullif (phone) {const view = params.get('view')const incident = detail && detail.id === selectedId ? detail : nullconst chatOpen = view === 'chat' && (!requested || incident)let body- if (chatOpen) body = <Chat key={selectedId ?? 'none'} phone user={user} incident={incident} enabled={config ? config.chat_enabled : true} back={incident ? { label: incident.title, to: `/agent?incident=${incident.id}` } : { label: 'Incidents', to: '/agent' }} />+ if (chatOpen) body = <Chat key={selectedId ?? 'none'} phone user={user} incident={incident} enabled={!chatOff} offReason={chatOff} back={incident ? { label: incident.title, to: `/agent?incident=${incident.id}` } : { label: 'Incidents', to: '/agent' }} />else if (incident) body = <PhoneIncident incident={incident} canAct={Boolean(inbox?.can_act)} onChanged={() => { reload(); reloadDetail() }} onAsk={() => setParams({ incident: String(incident.id), view: 'chat' })} />else if (selectedId || (requested && !inbox)) body = <><PhoneHeader back={{ label: 'Incidents', to: '/agent' }} title="Incident" live={false} /><p className="m-0 px-4 py-16 text-center text-[13px] text-muted">{detailError || 'Loading the incident…'}</p></>else body = <PhoneInbox inbox={inbox} error={error} tab={tab} setTab={setTab} onSelect={(id) => setParams({ incident: String(id) })} />@@ -435,7 +439,7 @@ export default function AgentPage({ user, onLogout }) {: selectedId ? <p className="m-0 px-7 py-16 text-center text-[13px] text-muted">{detailError || 'Loading the incident…'}</p>: <div className="px-7 py-16 text-center"><p className="m-0 text-sm font-medium text-paper">{inbox && !all.length ? 'No incidents yet.' : 'Nothing needs you right now.'}</p><p className="mb-0 mt-2 text-[13px] text-muted">{inbox && !all.length ? 'When a rule fires on one of your servers, the incident shows up here with what was found and how to fix it.' : 'Pick a resolved incident on the left to see how it went.'}</p></div>}</div>- <div className="min-h-0 border-t border-line xl:border-t-0"><Chat key={selectedId ?? 'none'} user={user} incident={detail && detail.id === selectedId ? detail : null} enabled={config ? config.chat_enabled : true} /></div>+ <div className="min-h-0 border-t border-line xl:border-t-0"><Chat key={selectedId ?? 'none'} user={user} incident={detail && detail.id === selectedId ? detail : null} enabled={!chatOff} offReason={chatOff} /></div></div></AppShell>)diff --git a/tests/accounts/test_admin.py b/tests/accounts/test_admin.pyindex b4b9bbc..2fa8364 100644--- a/tests/accounts/test_admin.py+++ b/tests/accounts/test_admin.py@@ -154,9 +154,11 @@ def test_admin_can_turn_agent_chat_off_and_on(fleet, monkeypatch):result = main.update_admin_settings(main.AdminSettingsRequest(agent_chat_enabled=False), session_id="s", db=fleet)- assert result == {"instance_name": main.get_instance_name(fleet), "metrics_retention_days": main.DEFAULT_METRICS_RETENTION_DAYS, "agent_chat_enabled": False, "alert_thresholds": rules.DEFAULT_THRESHOLDS, "mount_thresholds": {}, "require_2fa_for_admins": True, "secrets_configured": main.secrets_box.available(), "require_2fa_for_everyone": False, "disabled_rules": []}+ assert result == {"instance_name": main.get_instance_name(fleet), "metrics_retention_days": main.DEFAULT_METRICS_RETENTION_DAYS, "agent_chat_enabled": False, "alert_thresholds": rules.DEFAULT_THRESHOLDS, "mount_thresholds": {}, "require_2fa_for_admins": True, "secrets_configured": main.secrets_box.available(), "require_2fa_for_everyone": False, "disabled_rules": [],+ "agent_prices": {}, "agent_monthly_cap_usd": None, "agent_daily_chat_limit": main.agent_usage.DEFAULT_DAILY_CHAT_LIMIT}_login_as(monkeypatch, 2)- assert main.agent_config(session_id="s", db=fleet) == {"chat_enabled": False}+ config = main.agent_config(session_id="s", db=fleet)+ assert config["chat_enabled"] is False and config["budget_paused"] is Falsedef test_agent_chat_is_rejected_while_disabled(fleet, monkeypatch):diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.pyindex e6ec5cf..ae1a603 100644--- a/tests/accounts/test_new_pages.py+++ b/tests/accounts/test_new_pages.py@@ -337,3 +337,131 @@ def test_a_server_is_renamed_and_keeps_the_name_and_the_install_describes_itselfsystem = owner.get("/admin/system").json()assert system["version"] == main.APP_VERSION and system["database"]["host_rows"] == 30 and system["agent"]["incidents_this_month"] >= 1+++def test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monkeypatch):+ owner = signed_in(app, 1)+ monkeypatch.setenv("OPENAI_API_KEY", "sk-test-secret-value")+ monkeypatch.setattr(main, "check_provider_connection", lambda: {"provider": "OpenAI", "model": "gpt-test"})++ system = owner.get("/admin/system")+ assert system.json()["agent"]["key_configured"] is True and system.json()["agent"]["provider"] == "OpenAI"+ result = owner.post("/admin/agent/test")+ assert result.json() == {"ok": True, "provider": "OpenAI", "model": "gpt-test"}+ assert "sk-test-secret-value" not in system.text + result.text+ assert signed_in(app, 3).post("/admin/agent/test").status_code == 403+++def test_a_failed_agent_connection_test_explains_itself_without_the_key(app, monkeypatch):+ owner = signed_in(app, 1)++ def rejected():+ raise RuntimeError("Error code: 401 - Incorrect API key provided: sk-test-secret-value")+ monkeypatch.setattr(main, "check_provider_connection", rejected)++ response = owner.post("/admin/agent/test")+ assert response.status_code == 502 and "rejected the API key" in response.json()["detail"]+ assert "sk-test-secret-value" not in response.text+++# --- the agent's spending (#112) ---------------------------------------------------------------------------------++from src.agent.providers import Completion+from src.backend.models import AgentRun+++def _fake_agent(monkeypatch, tokens=(1000, 200), fail=None):+ def fake(user_id, conversation_id, message, command_logger=None, usage_logger=None):+ if fail:+ raise fail+ # A dated model name, as the API reports it: it must still use the price set for "gpt-test".+ usage_logger(Completion(text="{}", model="gpt-test-2026-01-01", input_tokens=tokens[0], output_tokens=tokens[1]), 1500)+ return "Here is the answer."+ monkeypatch.setattr(main, "send_agent_message", fake)+ monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)+++def _ask(client, conversation="c1"):+ client.post("/conversations", json={"conversation_id": conversation, "title": "t"})+ return client.post("/agent/chat", json={"conversation_id": conversation, "message": "Why did it restart?", "incident_id": 42})+++def test_every_chat_call_is_counted_with_its_tokens_and_cost(app, monkeypatch):+ _fake_agent(monkeypatch)+ owner = signed_in(app, 1)+ assert owner.put("/admin/settings", json={"agent_prices": {"gpt-test": {"input": 1, "output": 10}}}).status_code == 200++ assert _ask(owner).status_code == 200++ run = app.db.query(AgentRun).one()+ assert (run.kind, run.status, run.input_tokens, run.output_tokens, run.duration_ms, run.user_id, run.incident_id) == ("chat", "ok", 1000, 200, 1500, 1, 42)+ assert run.cost_usd == pytest.approx(0.003) # 1000 × $1 + 200 × $10 per million tokens+ usage = owner.get("/admin/agent/usage").json()+ assert usage["calls"] == 1 and usage["cost"] == pytest.approx(0.003) and usage["by_model"][0]["model"] == "gpt-test-2026-01-01"+ assert usage["daily"][-1]["cost"] == pytest.approx(0.003)+ assert signed_in(app, 3).get("/admin/agent/usage").status_code == 403+++def test_calls_made_before_a_price_was_entered_are_priced_later(app, monkeypatch):+ _fake_agent(monkeypatch)+ owner = signed_in(app, 1)+ _ask(owner)+ assert owner.get("/admin/agent/usage").json()["unpriced_runs"] == 1++ owner.put("/admin/settings", json={"agent_prices": {"gpt-test": {"input": 1, "output": 10}}})++ usage = owner.get("/admin/agent/usage").json()+ assert usage["cost"] == pytest.approx(0.003) and usage["unpriced_runs"] == 0+++def test_the_monthly_budget_pauses_the_chat_and_tells_the_admins_once(app, monkeypatch):+ _fake_agent(monkeypatch)+ sent = []+ monkeypatch.setattr(main, "_deliver", lambda db, channel, owner, title, body, severity: sent.append((owner.id, title)) or True)+ app.db.add(NotificationChannel(user_id=1, kind="ntfy", target_enc=secrets_box.encrypt("https://ntfy.sh/x"), display="ntfy.sh/x"))+ app.db.add(NotificationChannel(user_id=3, kind="ntfy", target_enc=secrets_box.encrypt("https://ntfy.sh/y"), display="ntfy.sh/y"))+ app.db.commit()+ owner = signed_in(app, 1)+ owner.put("/admin/settings", json={"agent_prices": {"gpt-test": {"input": 1, "output": 10}}, "agent_monthly_cap_usd": 0.005})++ assert _ask(owner).status_code == 200 # $0.003 of $0.005+ assert _ask(owner).status_code == 200 # $0.006: over the budget+ paused = _ask(owner)++ assert paused.status_code == 429 and "budget" in paused.json()["detail"]+ assert app.db.query(AgentRun).count() == 2 # the paused request never reached the model+ assert sent == [(1, "The incident agent paused: its monthly budget is used up")] # admins only, once+ assert owner.get("/agent/config").json()["budget_paused"] is True++ owner.put("/admin/settings", json={"agent_monthly_cap_usd": 0}) # removing the budget lets it run again+ assert _ask(owner).status_code == 200+++def test_the_daily_limit_counts_messages_per_person(app, monkeypatch):+ _fake_agent(monkeypatch)+ owner = signed_in(app, 1)+ owner.put("/admin/settings", json={"agent_daily_chat_limit": 2})++ assert [_ask(owner).status_code for _ in range(3)] == [200, 200, 429]+ assert _ask(signed_in(app, 2), conversation="c2").status_code == 200 # someone else still can+++def test_a_failed_call_is_recorded_without_tokens(app, monkeypatch):+ _fake_agent(monkeypatch, fail=TimeoutError("timed out"))+ owner = signed_in(app, 1)++ assert _ask(owner).status_code == 503++ run = app.db.query(AgentRun).one()+ assert (run.status, run.input_tokens, run.cost_usd) == ("error", 0, None) and "timed out" in run.error++[email protected]("body", [+ {"agent_prices": {"gpt test": {"input": 1, "output": 2}}},+ {"agent_prices": {"gpt-test": {"input": -1, "output": 2}}},+ {"agent_prices": {"gpt-test": {"input": 1}}},+ {"agent_monthly_cap_usd": -5},+ {"agent_daily_chat_limit": 0},+])+def test_spending_settings_are_checked(app, body):+ assert signed_in(app, 1).put("/admin/settings", json=body).status_code == 400diff --git a/tests/assistant/test_agent_errors.py b/tests/assistant/test_agent_errors.pyindex 372f219..f7a8f04 100644--- a/tests/assistant/test_agent_errors.py+++ b/tests/assistant/test_agent_errors.py@@ -13,3 +13,18 @@ def test_provider_error_redacts_api_keys():assert "[redacted]" in messageassert "AIzaSecretValue" not in message+++def test_openai_keys_are_redacted_too():+ message = agent_error_message(Exception("Error code: 401 - Incorrect API key provided: sk-proj-abc123SECRET"))++ assert "sk-proj-abc123SECRET" not in message+ assert "rejected the API key" in message+++def test_a_missing_key_says_where_to_set_it():+ assert "OPENAI_API_KEY" in agent_error_message(RuntimeError("OPENAI_API_KEY is not configured."))+++def test_an_unknown_model_points_at_agent_model():+ assert "AGENT_MODEL" in agent_error_message(Exception("Error code: 404 - The model `gpt-nope` does not exist"))diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.pyindex baafced..a98c6b6 100644--- a/tests/assistant/test_agent_service.py+++ b/tests/assistant/test_agent_service.py@@ -60,3 +60,97 @@ def test_arbitrary_command_is_rejected_before_execution():with pytest.raises(ValueError, match="outside the allow-list"):service.send(1, "conversation", "Delete everything")+++# --- the OpenAI provider ----------------------------------------------------------------------------------------++from types import SimpleNamespace++from src.agent import service as agent_service+from src.agent.providers import Completion, OpenAIProvider+++class FakeCompletions:+ def __init__(self, content='{"message":"All good.","action":null}', refusal=None, fail=None):+ self.calls = []+ self.content, self.refusal, self.fail = content, refusal, fail++ def create(self, **kwargs):+ self.calls.append(kwargs)+ if self.fail:+ raise self.fail+ message = SimpleNamespace(content=self.content, refusal=self.refusal)+ return SimpleNamespace(choices=[SimpleNamespace(message=message)], model="gpt-test", usage=SimpleNamespace(prompt_tokens=120, completion_tokens=8))+++def fake_client(completions):+ return SimpleNamespace(chat=SimpleNamespace(completions=completions), models=SimpleNamespace(retrieve=lambda model: SimpleNamespace(id=model)))+++def test_openai_provider_asks_for_the_reply_schema_and_reports_tokens():+ completions = FakeCompletions()+ provider = OpenAIProvider(model="gpt-test", client=fake_client(completions))++ result = provider.complete([{"role": "user", "content": "hi"}], agent_service.REPLY_SCHEMA, "agent_reply")++ assert result == Completion(text='{"message":"All good.","action":null}', model="gpt-test", input_tokens=120, output_tokens=8)+ format_ = completions.calls[0]["response_format"]+ assert format_["type"] == "json_schema" and format_["json_schema"]["strict"] is True+ # The model can only pick a check from the allow-list; anything else is impossible, not just rejected later.+ check = format_["json_schema"]["schema"]["properties"]["action"]["anyOf"][0]["properties"]["check"]+ assert set(check["enum"]) == set(agent_service.HealthCheckRunner.ALLOWED_CHECKS)+++def test_a_refusal_is_an_error_not_an_empty_reply():+ provider = OpenAIProvider(model="gpt-test", client=fake_client(FakeCompletions(content=None, refusal="I can't help with that.")))++ with pytest.raises(ValueError, match="refused"):+ provider.complete([], agent_service.REPLY_SCHEMA, "agent_reply")+++def test_the_provider_needs_a_key(monkeypatch):+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)++ with pytest.raises(RuntimeError, match="OPENAI_API_KEY"):+ OpenAIProvider()+++def test_test_connection_spends_no_tokens():+ completions = FakeCompletions()+ provider = OpenAIProvider(model="gpt-test", client=fake_client(completions))++ assert provider.check() == {"provider": "OpenAI", "model": "gpt-test"}+ assert completions.calls == []+++def test_a_conversation_sends_the_system_prompt_and_its_history():+ completions = FakeCompletions()+ chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))++ chat.send_message("first")+ chat.send_message("second")++ sent = completions.calls[-1]["messages"]+ assert sent[0]["role"] == "system" and [message["content"] for message in sent[1:] if message["role"] == "user"] == ["first", "second"]+ assert [message["role"] for message in sent[1:]] == ["user", "assistant", "user"]+++def test_only_recent_turns_go_back_to_the_model(monkeypatch):+ monkeypatch.setattr(agent_service, "MAX_HISTORY_MESSAGES", 4)+ completions = FakeCompletions()+ chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))++ for number in range(6):+ chat.send_message(f"message {number}")++ assert len(completions.calls[-1]["messages"]) == 1 + 4+++def test_a_failed_turn_is_not_kept_in_the_conversation():+ completions = FakeCompletions(fail=TimeoutError("timed out"))+ chat = agent_service.Chat(OpenAIProvider(model="gpt-test", client=fake_client(completions)))++ with pytest.raises(TimeoutError):+ chat.send_message("hello")++ assert chat.history == []diff --git a/tests/security/test_second_review.py b/tests/security/test_second_review.pyindex 95090a0..57f794d 100644--- a/tests/security/test_second_review.py+++ b/tests/security/test_second_review.py@@ -297,14 +297,10 @@ def test_server_ids_in_the_address_must_fit_too(app, path):def test_the_assistants_memory_of_chats_is_bounded_and_forgets_the_least_recently_used(monkeypatch):from src.agent import service- class FakeChats:- def create(self, **kwargs):- return object()-instance = object.__new__(service.AgentService) if hasattr(service, "AgentService") else Noneif instance is None:pytest.skip("the assistant service class has another name")- instance.client = type("C", (), {"chats": FakeChats()})()+ instance.provider = object()instance.chats, instance.lock = {}, __import__("threading").Lock()monkeypatch.setattr(service, "MAX_CACHED_CHATS", 3)for number in range(5):