Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
fix(agent): make tool requests acceptable to Anthropic and OpenAI (#197)
Karlo Kajba Šimanić205c2aa6874f
diff --git a/SECURITY.md b/SECURITY.mdindex 4ca3fc7..f68508f 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -78,7 +78,9 @@ the containers' names, images and figures, and similar incidents before it. Befomasked for secrets (tokens, keys, passwords, credentials in URLs, private keys; `src/agent/masking.py`), and themodel is told the pack is untrusted data. Its answer is checked before anyone sees it: claims that cite no factfrom the pack are dropped, and fix commands that could destroy data or open the machine up are flagged and never-marked recommended (`src/agent/safety.py`). Nothing the model writes is run. Turn off "Diagnose new incidents+marked recommended (`src/agent/safety.py`). Nothing the model writes is run. Requests to OpenAI are sent with+`store: false`, so OpenAI keeps no copy of the conversation for later retrieval (its own abuse-monitoring retention+still applies). Turn off "Diagnose new incidentsautomatically" in Admin › Incident agent to send nothing unless someone asks; without a key nothing is sentand incidents keep the rule-based diagnosis, which is worked out locally.diff --git a/TESTS.md b/TESTS.mdindex ad4fe44..80d3cc8 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,covers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **1563 test cases** from 849 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).+- **1574 test cases** from 857 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collector@@ -146,13 +146,13 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.|---|---|---|| [`test_account_security.py`](#test_account_securitypy) | 146 | Two-factor sign-in, recovery codes, encrypted secrets, sessions you can review and end, password change, and the password check before turning two-factor on. || [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |-| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |-| [`test_agent_service.py`](#test_agent_servicepy) | 9 | The chat's reply is one message; the OpenAI provider and conversation history. |+| [`test_agent_errors.py`](#test_agent_errorspy) | 6 | Helpful, non-leaking error messages from the AI assistant: a plain sentence in the chat, the provider's own words only for admins. |+| [`test_agent_service.py`](#test_agent_servicepy) | 10 | The chat's reply is one message; Test connection trying a real request; the Chat Completions provider and conversation history. || [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |-| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 24 | Tool calling with Claude, OpenAI and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer. |+| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. || [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. || [`test_agent_investigation.py`](#test_agent_investigationpy) | 27 | The agent looking things up: the budget of tool calls per diagnosis and chat message, forced final answers, errors, the audit log, citations from tool results. |-| [`test_agent_tools.py`](#test_agent_toolspy) | 49 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. |+| [`test_agent_tools.py`](#test_agent_toolspy) | 52 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. || [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. || [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. || [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |@@ -312,7 +312,8 @@ shouldn't, or an admin might be locked out.| Test | What it checks ||---|---|| `test_overloaded_provider_error_is_explained_to_the_user` | When the AI provider reports it is overloaded, the user gets a readable explanation. |-| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the chat or logs; it is replaced with `[redacted]`. |+| `test_the_chat_gets_a_plain_sentence_and_only_admins_the_providers_words` | A request the provider rejects is a plain sentence in the chat; only the admin's Test connection adds the provider's own error (#196). |+| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the admin or the logs; it is replaced with `[redacted]`. || `test_openai_keys_are_redacted_too` | An OpenAI key (`sk-…`) in an error is redacted, and a rejected key is explained as such. || `test_a_missing_key_says_where_to_set_it` | Without a key the chat says to add one in Admin › Incident agent. || `test_an_unknown_model_points_at_the_model_setting` | A model the key can't use points the admin at the model in Admin › Incident agent. |@@ -323,8 +324,9 @@ shouldn't, or an admin might be locked out.|---|---|| `test_without_a_tool_budget_the_chat_is_one_call_without_tools` | With the chat's look-up budget at 0, a message is one call to the model, with no tools and effort `low`. || `test_a_reply_without_a_message_is_an_error` | A reply whose message is missing or empty is an error, not a blank answer. |+| `test_test_connection_tries_one_request_shaped_like_a_diagnosis_and_counts_it` | Test connection checks the key, then sends one request with every tool and the diagnosis schema (tools not allowed), counts it as a `test` call, and raises the provider's refusal (#196). || `test_the_reply_can_only_be_a_message` | The reply schema has one field, the message, and nothing else is allowed. |-| `test_openai_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests use Structured Outputs in strict mode with the reply schema, and token usage is read from the response. |+| `test_the_chat_completions_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests to the Chat Completions provider (Gemini) use Structured Outputs in strict mode with the reply schema, and token usage is read from the response. || `test_a_refusal_is_an_error_not_an_empty_reply` | When the model refuses, the chat gets an error instead of an empty answer. || `test_test_connection_spends_no_tokens` | Admin's Test connection only looks the model up; it sends no prompt. || `test_a_conversation_sends_the_system_prompt_and_its_history` | Each request carries the system prompt and the conversation so far, in order. |@@ -347,21 +349,27 @@ shouldn't, or an admin might be locked out.### test_agent_tool_calls.py-Most tests run once per provider (Claude, OpenAI, Gemini), with fake clients.+Most tests run once per provider (Claude, OpenAI through the Responses API, Gemini through Chat Completions), with+fake clients. The fakes can't know each provider's own limits (#196), which is why Test connection sends one real+request.| Test | What it checks ||---|---|| `test_with_tools_the_model_can_still_answer_at_once` *(3 inputs)* | With tools offered, a model that answers straight away gives the final answer and its token usage. || `test_a_tool_call_is_answered_and_the_history_goes_back_unchanged` *(3 inputs)* | A tool call comes back parsed; after its result, the next request repeats every earlier turn exactly as sent and adds the result. || `test_claude_gets_its_own_answer_back_untouched_thinking_included` | Claude's own turn, thinking included, goes back unchanged; its tools are strict and the system prompt is sent apart. |-| `test_several_tool_calls_in_one_turn_get_all_their_results` *(3 inputs)* | Several calls in one turn are all answered: in one message for Claude, one tool message each for OpenAI and Gemini. |+| `test_several_tool_calls_in_one_turn_get_all_their_results` *(3 inputs)* | Several calls in one turn are all answered: in one message for Claude, one `function_call_output` item each for OpenAI, one tool message each for Gemini. || `test_a_failed_tool_is_reported_as_an_error` *(3 inputs)* | A failed tool is marked as an error for the model. || `test_tool_results_are_masked_before_they_leave` *(3 inputs)* | A password in a tool result is masked before it is sent to the provider. || `test_without_tools_allowed_the_model_must_answer_but_the_tools_stay_listed` *(3 inputs)* | When the budget is used up the model is told to answer (`tool_choice` none), while the tool list stays the same. || `test_claude_refusing_midway_is_an_error` | A refusal from Claude in the middle of an investigation is an error. || `test_openai_refusing_is_an_error` | A refusal from OpenAI with tools offered is an error. |-| `test_openai_tool_arguments_must_be_a_json_object` *(2 inputs)* | Tool arguments that aren't valid JSON, or aren't an object, are refused. |-| `test_openai_tools_are_strict_functions` | OpenAI and Gemini get strict function tools and a strict answer schema. |+| `test_gemini_refusing_is_an_error` | A refusal from Gemini with tools offered is an error. |+| `test_openai_tool_arguments_must_be_a_json_object` *(4 inputs)* | For OpenAI and Gemini, tool arguments that aren't valid JSON, or aren't an object, are refused. |+| `test_openai_uses_the_responses_api_with_strict_tools_reasoning_and_nothing_stored` | OpenAI gets strict function tools, a strict answer schema, the reasoning effort, the system prompt as instructions, `store: false` and the encrypted reasoning back. |+| `test_openai_gets_its_reasoning_back_unchanged_with_the_tool_results` | The next request repeats OpenAI's reasoning item (encrypted) and its tool call unchanged, followed by the result. |+| `test_an_openai_answer_cut_off_is_an_error` | A response stopped at the output limit is an error, not a half answer. |+| `test_gemini_tools_are_strict_functions` | Gemini gets strict function tools and a strict answer schema. |### test_agent_diagnosis.py@@ -425,6 +433,8 @@ and a firewall.| Test | What it checks ||---|---|+| `test_no_request_has_more_union_typed_parameters_than_the_provider_accepts` *(2 inputs)* | The tools have no nullable or union-typed parameter, and with the diagnosis or chat schema a request stays within Anthropic's limit of 16 (#196). |+| `test_an_optional_argument_left_empty_or_null_means_not_given` | An optional argument sent as an empty string or null gives the same result (not given); a required one may not be empty. || `test_every_tool_is_listed_once_with_a_strict_schema` | The eleven tools are each listed once, every schema is strict (all fields required, nothing extra allowed), and every host metric offered can be read. || `test_an_unknown_tool_is_refused` | A call to a tool that isn't on the list (`run_shell`) is refused. || `test_bad_arguments_are_refused_with_a_reason_the_model_can_act_on` *(17 inputs)* | Missing or extra arguments, wrong types, values outside a list, oversized text, bad or reversed times, windows older than 90 days, container snapshots older than 7 days, and unknown checks or containers are refused before anything is read, with a reason (and the names that do exist). |diff --git a/docs/api.md b/docs/api.mdindex 0217320..69be2b6 100644--- a/docs/api.md+++ b/docs/api.md@@ -38,7 +38,7 @@ All paths are relative to `/api` when going through the frontend/nginx.| `GET /admin/agent/worker` | admin | Whether the agent worker is running, how many jobs wait, are running or wait to be retried, and the latest results. || `PUT /admin/agent/provider` | admin (rate limited) + `reauth_token` and `code` | Choose the agent's provider (`anthropic`, `openai`, `gemini`) and model, and optionally save a new API key, stored encrypted (409 without `SECRETS_KEY`). Returns the choice and whether a key is saved, never the key. || `DELETE /admin/agent/key` | admin | Remove the chosen provider's API key; the agent stops until a new one is saved. |-| `POST /admin/agent/test` | admin | Checks the saved key and that the configured model exists, without spending tokens. |+| `POST /admin/agent/test` | admin | Checks the saved key, that the configured model exists, and that the provider accepts a request shaped like a diagnosis (every tool and the answer schema, tools not allowed). That request costs a few thousand tokens and is counted in the usage as kind `test`; the reply adds `tokens`. At most 10 in 10 minutes. || `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. || `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. || `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |diff --git a/docs/assistant.md b/docs/assistant.mdindex 6e2dc5a..2cf8684 100644--- a/docs/assistant.md+++ b/docs/assistant.md@@ -23,8 +23,12 @@ change files on your servers. Everything about it is in `src/agent/` and in **Ad| OpenAI | `gpt-6-luna` || Google Gemini | `gemini-3.1-flash-lite` |- Another model works if it supports structured (JSON schema) output.-4. **Test connection** checks the key and that the model exists, without spending tokens.+ Another model works if it supports structured (JSON schema) output and tool calls. With OpenAI it must be a+ reasoning model (the gpt-5 family and later): the agent sets how hard it thinks.+4. **Test connection** checks the key, that the model exists, and that the provider accepts the agent's real+ requests: one small request with every tool and the diagnosis schema. It costs a few thousand tokens (about+ 2 000 to 4 500 in our tests), counted in the usage like any other call. Run it after changing the provider or+ the model; a provider that refuses the request says why here, instead of every diagnosis failing later.5. Enter the model's price per million tokens under **Prices** (from the provider's pricing page), so its callscount towards the budget, and set a **Monthly budget** if you want one.@@ -112,7 +116,10 @@ facts and the conversation so far (its latest 40 turns), and the model may look[read-only tools](#what-the-agent-can-look-up) as a diagnosis, up to 4 calls per message by default. Only thequestions and replies stay in the conversation; the look-ups of an earlier message aren't sent again. Every replyis held to a JSON schema with the provider's structured output: one message. All three providers sit behind one-small interface (`src/agent/providers/`).+small interface (`src/agent/providers/`): Claude through Anthropic's Messages API; OpenAI through its Responses API+(the only one where its reasoning models can use tools), sent with `store: false` so nothing is kept at OpenAI, with+the model's reasoning carried from one step to the next in encrypted form; Gemini through Google's OpenAI-compatible+Chat Completions endpoint.With Claude, an answer that may use tools thinks harder (effort `high`); a chat message without tools thinksbriefly (`low`). The other providers ignore the setting.@@ -179,7 +186,10 @@ different windows, the label keeps both.**Adding a tool.** Write the function in the module for its area (`servers.py`, `containers.py`, `activity.py`):it gets the database session, the checked arguments, the time and a `Facts` collector, and returns a dict whosefacts each carry an `id` added with `facts.add`. Register it in `tools/__init__.py` with a description and a-strict schema (every property required, `additionalProperties: false`, null for "not given"), and add tests to+strict schema (every property required, `additionalProperties: false`, and an optional text argument made with+`_optional`, a plain string where empty means "not given", never a nullable type: Anthropic refuses strict schemas+with more than 16 nullable or union-typed parameters per request, counted over every tool and the answer's schema,+and `test_no_request_has_more_union_typed_parameters_than_the_provider_accepts` checks it), and add tests to`tests/assistant/test_agent_tools.py`: the shared tests then check that it stays bounded, masked, labelled andread-only.diff --git a/src/agent/chat.py b/src/agent/chat.pyindex bc446ba..085c527 100644--- a/src/agent/chat.py+++ b/src/agent/chat.py@@ -5,8 +5,10 @@ from __future__ import annotationsimport jsonimport threading+import timefrom collections.abc import Callable+from . import diagnosis, tools, usagefrom .investigate import Auditor, Recorder, investigatefrom .prompts import SYSTEM_PROMPTfrom .providers import Completion, Provider, get_provider@@ -108,6 +110,23 @@ def delete_agent_conversation(user_id: int, conversation_id: str) -> None:_agent_service.delete_conversation(user_id, conversation_id)-def check_provider_connection(db) -> dict:- """For Admin's "Test connection": the key works and the configured model exists. Spends no tokens."""- return get_provider(db).check()+TRIAL_MESSAGES = [{"role": "system", "content": "This is a connection test. Do not call any tool."},+ {"role": "user", "content": "Answer with the shortest values the schema allows."}]+++def check_provider_connection(db, user_id: int | None = None) -> dict:+ """For Admin's "Test connection": the key works, the configured model exists, and the provider accepts a+ request shaped like a diagnosis: every tool and the diagnosis schema, tools not allowed (#196). That one request+ is the only thing here that costs tokens (a few thousand, mostly the tools' definitions), and it is+ counted in the usage like any other (kind ``test``). Raises the provider's error when it is refused."""+ provider = get_provider(db)+ found = provider.check()+ started = time.monotonic()+ try:+ completion = provider.start(TRIAL_MESSAGES, tools.TOOLS, diagnosis.SCHEMA, "incident_diagnosis", effort="low").step(allow_tools=False).completion+ except Exception as exc:+ usage.record(db, kind="test", status="error", model=provider.model, user_id=user_id, duration_ms=round((time.monotonic() - started) * 1000), error=f"{type(exc).__name__}: {exc}")+ raise+ usage.record(db, kind="test", model=completion.model or provider.model, input_tokens=completion.input_tokens, output_tokens=completion.output_tokens,+ duration_ms=round((time.monotonic() - started) * 1000), user_id=user_id)+ return {**found, "tokens": completion.input_tokens + completion.output_tokens}diff --git a/src/agent/providers/__init__.py b/src/agent/providers/__init__.pyindex 159c386..df14d6b 100644--- a/src/agent/providers/__init__.py+++ b/src/agent/providers/__init__.py@@ -15,6 +15,7 @@ from src.backend.models import AppSettingfrom .base import Completion, Conversation, Provider, Tool, ToolCall, ToolResult, Turnfrom .claude import ClaudeProviderfrom .openai_compatible import GEMINI_BASE_URL, OpenAIProvider+from .openai_responses import OpenAIResponsesProvider__all__ = ["Completion", "Conversation", "Provider", "Tool", "ToolCall", "ToolResult", "Turn", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]@@ -85,7 +86,7 @@ def get_provider(db) -> Provider:return ClaudeProvider(choice.key, choice.model)if choice.provider == "gemini":return OpenAIProvider(choice.key, choice.model, name="Google Gemini", base_url=GEMINI_BASE_URL)- return OpenAIProvider(choice.key, choice.model)+ return OpenAIResponsesProvider(choice.key, choice.model)def save(db, provider: str, model: str | None, api_key: str | None) -> None:diff --git a/src/agent/providers/openai_compatible.py b/src/agent/providers/openai_compatible.pyindex 8ca87ad..f13b28a 100644--- a/src/agent/providers/openai_compatible.py+++ b/src/agent/providers/openai_compatible.py@@ -1,4 +1,5 @@-"""OpenAI, and Google Gemini through its OpenAI-compatible endpoint: both take the same Chat Completions request."""+"""Google Gemini through its OpenAI-compatible endpoint, which takes a Chat Completions request. OpenAI itself uses+the Responses API (openai_responses.py), since its reasoning models can't use tools through Chat Completions."""from __future__ import annotationsimport jsondiff --git a/src/agent/providers/openai_responses.py b/src/agent/providers/openai_responses.pynew file mode 100644index 0000000..b80901c--- /dev/null+++ b/src/agent/providers/openai_responses.py@@ -0,0 +1,81 @@+"""OpenAI through the Responses API: tools, reasoning and a strict JSON answer in one request (#196).++OpenAI's reasoning models (the default, gpt-6-luna, among them) can't use tools through Chat Completions, so OpenAI+has its own provider here; Gemini keeps the OpenAI-compatible Chat Completions endpoint (openai_compatible.py).++Nothing is kept at OpenAI: every request is sent with ``store=False``. The model's reasoning still carries over+from one step to the next, because each response's items, including the reasoning in encrypted form+(``reasoning.encrypted_content``), go back unchanged with the next request.+"""+from __future__ import annotations++import json++from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion, Conversation, Tool, ToolCall, ToolResult, Turn+++class OpenAIResponsesProvider:+ name = "OpenAI"++ def __init__(self, api_key: str, model: str, client=None):+ self.model = model+ if client is None:+ from openai import OpenAI # imported here so the rest of the app runs without the package+ client = OpenAI(api_key=api_key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)+ self.client = client++ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> ResponsesConversation:+ return ResponsesConversation(self, messages, tools, schema, schema_name, effort)++ def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:+ return self.start(messages, [], schema, schema_name, effort).step().completion++ def check(self) -> dict:+ model = self.client.models.retrieve(self.model)+ return {"provider": self.name, "model": getattr(model, "id", self.model)}+++def _item(item) -> dict:+ """An output item as the next request's input, unchanged."""+ return item.model_dump(mode="json", exclude_none=True) if hasattr(item, "model_dump") else dict(item)+++class ResponsesConversation(Conversation):+ def __init__(self, provider: OpenAIResponsesProvider, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str):+ self.provider, self.effort = provider, effort+ self.instructions = "\n\n".join(message["content"] for message in messages if message["role"] == "system")+ self.history = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]+ self.text = {"format": {"type": "json_schema", "name": schema_name, "schema": schema, "strict": True}}+ self.tools = [{"type": "function", "name": tool.name, "description": tool.description, "parameters": tool.schema, "strict": True} for tool in tools]++ def step(self, allow_tools: bool = True) -> Turn:+ extra = {"tools": self.tools, "tool_choice": "auto" if allow_tools else "none"} if self.tools else {}+ response = self.provider.client.responses.create(+ model=self.provider.model, instructions=self.instructions or None, input=list(self.history), text=self.text,+ reasoning={"effort": self.effort}, store=False, include=["reasoning.encrypted_content"], **extra,+ )+ if getattr(response, "status", None) == "incomplete":+ reason = getattr(getattr(response, "incomplete_details", None), "reason", None)+ raise ValueError("The model's answer was cut off before it was complete." if reason == "max_output_tokens" else f"The model stopped before answering ({reason or 'incomplete'}).")+ output = list(response.output or [])+ for item in output:+ if item.type == "message" and any(getattr(part, "type", None) == "refusal" for part in item.content or []):+ raise ValueError("The model refused to answer.")+ try:+ calls = [ToolCall(item.call_id, item.name, json.loads(item.arguments)) for item in output if item.type == "function_call"]+ except (json.JSONDecodeError, TypeError) as exc:+ raise ValueError("The model sent tool arguments that aren't valid JSON.") from exc+ if any(not isinstance(call.arguments, dict) for call in calls):+ raise ValueError("The model sent tool arguments that aren't a JSON object.")+ self.history.extend(_item(item) for item in output) # reasoning included, so the next step goes on from it+ usage = getattr(response, "usage", None)+ return Turn(Completion(+ text="" if calls else response.output_text or "",+ model=getattr(response, "model", self.provider.model),+ input_tokens=getattr(usage, "input_tokens", 0) or 0,+ output_tokens=getattr(usage, "output_tokens", 0) or 0, # reasoning tokens included+ ), calls)++ def _append_results(self, results: list[ToolResult]) -> None:+ for result in results:+ self.history.append({"type": "function_call_output", "call_id": result.call_id, "output": f"Error: {result.content}" if result.is_error else result.content})diff --git a/src/agent/tools/__init__.py b/src/agent/tools/__init__.pyindex 2c12eef..cb9a1a4 100644--- a/src/agent/tools/__init__.py+++ b/src/agent/tools/__init__.py@@ -33,14 +33,20 @@ def _object(**properties) -> dict:return {"type": "object", "properties": properties, "required": list(properties), "additionalProperties": False}-def _nullable(kind: str, description: str) -> dict:- return {"anyOf": [{"type": kind}, {"type": "null"}], "description": description}+class _Optional(dict):+ """A text argument that may be left empty ("" or null), which the tool reads as "not given". It is a plain+ string in the schema because strict schemas allow at most 16 nullable or union-typed parameters per request,+ counted over every tool and the answer's schema (Anthropic rejects more); see docs/assistant.md."""+++def _optional(description: str) -> dict:+ return _Optional(type="string", description=description)SERVER = {"type": "string", "description": "The server's hostname, as the evidence names it."}-FROM = _nullable("string", "Start of the window, ISO 8601 in UTC (2026-11-07T10:00). null: 24 hours before `to`.")-TO = _nullable("string", "End of the window, ISO 8601 in UTC. null: now.")-AT = _nullable("string", "The moment to look at, ISO 8601 in UTC. null: now.")+FROM = _optional("Start of the window, ISO 8601 in UTC (2026-11-07T10:00). Empty: 24 hours before `to`.")+TO = _optional("End of the window, ISO 8601 in UTC. Empty: now.")+AT = _optional("The moment to look at, ISO 8601 in UTC. Empty: now.")WINDOW_NOTE = (f" Any window within the last {MAX_DAYS} days: every check-in for the last 7 days, hourly averages and peaks before that."" Start wide, then narrow down to the interesting hours.")@@ -66,7 +72,7 @@ _REGISTRY: dict[str, tuple[Tool, Handler]] = {tool.name: (tool, handler) for too"and (for the last 7 days) each change of status, health, exit code, OOM kill or image." + WINDOW_NOTE,_object(server=SERVER, container={"type": "string", "description": "The container's name."}, **{"from": FROM, "to": TO})), containers.container_history),(Tool("disk_usage", "Every filesystem of a server (or one): used space and inodes now, usage over the last `days`, growth in GB per day and, when it grows, "- "in about how many days it fills up.", _object(server=SERVER, mount=_nullable("string", "One mount point such as /var. null: all of them."),+ "in about how many days it fills up.", _object(server=SERVER, mount=_optional("One mount point such as /var. Empty: all of them."),days={"type": "integer", "description": f"How many days back to look at, 1 to {MAX_DAYS}."})), servers.disk_usage),(Tool("events", "What the server recorded in a window: reboots, OOM kills, kernel faults and errors, image changes, containers added or removed, ports ""opened or closed, firewall changes, package updates. The newest 30, oldest first." + WINDOW_NOTE,@@ -77,35 +83,32 @@ _REGISTRY: dict[str, tuple[Tool, Handler]] = {tool.name: (tool, handler) for tooservers.compare_to_baseline),(Tool("find_incidents", "Earlier incidents on one server or all of them: title, kind, severity, how long they lasted and how they were resolved. Newest ""first. The window defaults to the last 30 days.",- _object(server=_nullable("string", "A hostname. null: every server."),- kind=_nullable("string", "Only this kind, as in the evidence's incident `kind` (host:disk, container:restart_loop, …). null: every kind."),- **{"from": _nullable("string", "Start of the window, ISO 8601 in UTC. null: 30 days before `to`."), "to": TO})), activity.find_incidents),+ _object(server=_optional("A hostname. Empty: every server."),+ kind=_optional("Only this kind, as in the evidence's incident `kind` (host:disk, container:restart_loop, …). Empty: every kind."),+ **{"from": _optional("Start of the window, ISO 8601 in UTC. Empty: 30 days before `to`."), "to": TO})), activity.find_incidents),(Tool("fleet_snapshot", "Every server at one moment: online or not, open incidents and the worst of them, availability checks failing from it. ""Shows whether a problem is on one machine or many.", _object(at=AT)), activity.fleet_snapshot),(Tool("check_results", "Availability checks (HTTP, TCP, TLS, DNS, ping …) over a window: uptime, when they were down, the errors they got and response ""times. Give one check by name, or a server for every check that runs from it." + WINDOW_NOTE,- _object(check=_nullable("string", "The check's name."), server=_nullable("string", "A hostname: every check run from it."), **{"from": FROM, "to": TO})),+ _object(check=_optional("The check's name."), server=_optional("A hostname: every check run from it."), **{"from": FROM, "to": TO})),activity.check_results),(Tool("network_state", "A server's listening ports (and whether other machines can reach them), its firewall configuration, and port and firewall changes ""in the last 7 days.", _object(server=SERVER)), servers.network_state),]}-TOOLS: list[Tool] = [tool for tool, _ in _REGISTRY.values()]+# What the provider gets: the same schemas as plain JSON (the _Optional marker is only for checking arguments here).+TOOLS: list[Tool] = [Tool(tool.name, tool.description, json.loads(json.dumps(tool.schema))) for tool, _ in _REGISTRY.values()]+++def _blank(value) -> bool:+ return value is None or (isinstance(value, str) and not value.strip())def _check(schema: dict, value, path: str) -> None:"""``value`` against the small part of JSON Schema the tools use. Raises ToolError naming the argument."""- if "anyOf" in schema:- for option in schema["anyOf"]:- try:- return _check(option, value, path)- except ToolError:- continue- raise ToolError(f"{path} has the wrong type.")+ if isinstance(schema, _Optional) and _blank(value):+ returnkind = schema["type"]- if kind == "null":- if value is not None:- raise ToolError(f"{path} must be null.")- elif kind == "object":+ if kind == "object":if not isinstance(value, dict) or set(value) != set(schema["properties"]):raise ToolError(f"Give exactly these arguments: {', '.join(schema['properties'])}.")for key, sub in schema["properties"].items():@@ -165,6 +168,8 @@ def run(db, name: str, arguments, now: datetime | None = None) -> Output:raise ToolError(f"There is no tool named {str(name)[:60]!r}.")tool, handler = _REGISTRY[name]_check(tool.schema, arguments, "arguments")+ # An optional argument left empty reaches the tool as None, whether the model sent "" or null.+ arguments = {key: None if isinstance(tool.schema["properties"][key], _Optional) and _blank(value) else value for key, value in arguments.items()}now = now or datetime.now(timezone.utc)facts = Facts()text = _fit(mask(handler(db, arguments, now, facts)))diff --git a/src/backend/main.py b/src/backend/main.pyindex fd9c9c5..b7dd55d 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -222,8 +222,10 @@ class MetricsIngestRequest(BaseModel):containers: list[dict] = []-def agent_error_message(exc: Exception) -> str:- """Turn provider failures into useful, non-sensitive chat text."""+def agent_error_message(exc: Exception, detail: bool = False) -> str:+ """Turn provider failures into useful, non-sensitive text. Only ``detail=True`` (for admins, such as Test+ connection) adds the provider's own words, redacted; the chat gets the plain sentence and the details go to the+ server log and the agent's usage record (agent_error_detail)."""if isinstance(exc, NoApiKey):return "The agent has no API key. An admin adds one in Admin › Incident agent."raw_message = str(exc).strip()@@ -238,16 +240,17 @@ def agent_error_message(exc: Exception) -> str:prefix = "The AI provider's rate limit or quota was reached. Please try again later."elif any(marker in normalized for marker in ("timeout", "timed out", "deadline exceeded")):prefix = "The AI provider did not respond before the request timed out. Please try again."+ elif any(marker in normalized for marker in (" 400", "status 400", "invalid_request")):+ prefix = "The AI provider rejected the request. An admin finds the reason in the backend's log, or with Test connection in Admin › Incident agent."else:prefix = "The AI provider could not complete this request."+ return f"{prefix} Provider error: {agent_error_detail(exc)}" if detail and raw_message else prefix- if not raw_message:- return prefix- # Errors can include request headers or provider diagnostics: keep a short,- # redacted detail for the administrator without leaking credentials.- detail = re.sub(r"(?i)(api[_ -]?key|authorization|token)\s*[:=]\s*\S+", r"\1=[redacted]", raw_message)- detail = re.sub(r"(AIza|sk-)[\w-]+", "[redacted]", detail)[:300]- return f"{prefix} Provider error: {detail}"++def agent_error_detail(exc: Exception) -> str:+ """The provider's own error, short and with keys redacted: errors can include request headers or diagnostics."""+ text = re.sub(r"(?i)(api[_ -]?key|authorization|token)\s*[:=]\s*\S+", r"\1=[redacted]", str(exc).strip())+ return re.sub(r"(AIza|sk-)[\w-]+", "[redacted]", text)[:300]redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)diff --git a/src/backend/models.py b/src/backend/models.pyindex 561e5d8..034d2d8 100644--- a/src/backend/models.py+++ b/src/backend/models.py@@ -509,7 +509,7 @@ class AgentRun(Base):__table_args__ = (Index("ix_agent_runs_created", "created_at"),)id: Mapped[int] = mapped_column(Integer, primary_key=True)- kind: Mapped[str] = mapped_column(String(16), nullable=False) # chat | diagnosis+ kind: Mapped[str] = mapped_column(String(16), nullable=False) # chat | diagnosis | test (Admin's Test connection)status: Mapped[str] = mapped_column(String(16), nullable=False) # ok | errormodel: Mapped[str] = mapped_column(String(100), nullable=False)input_tokens: Mapped[int] = mapped_column(Integer, default=0, nullable=False)diff --git a/src/backend/routes/admin.py b/src/backend/routes/admin.pyindex 87663f1..166f989 100644--- a/src/backend/routes/admin.py+++ b/src/backend/routes/admin.py@@ -161,13 +161,14 @@ def admin_agent_worker(session_id: Annotated[str | None, Cookie(alias=SESSION_CO@router.post("/admin/agent/test")def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):- """Admin's "Test connection": the API key works and the configured model exists. Spends no tokens."""+ """Admin's "Test connection": the API key works, the model exists, and the provider accepts the agent's tools+ and answer schema (one small request of a few thousand tokens; see check_provider_connection)."""admin = main._require_admin(session_id, db)main.enforce_limit(main.redis_client, scope="agent-test", subject=admin["id"], limit=10, window_seconds=600)try:- result = main.check_provider_connection(db)+ result = main.check_provider_connection(db, user_id=admin["id"])except Exception as exc:- raise HTTPException(status_code=502, detail=main.agent_error_message(exc)) from exc+ raise HTTPException(status_code=502, detail=main.agent_error_message(exc, detail=True)) from excreturn {"ok": True, **result}diff --git a/src/backend/routes/chat.py b/src/backend/routes/chat.pyindex e00c534..3b05db5 100644--- a/src/backend/routes/chat.py+++ b/src/backend/routes/chat.py@@ -4,6 +4,7 @@ tune them) and the shared helpers (`_incident_context`, `send_agent_message`, `_this module is the HTTP layer."""from __future__ import annotations+import loggingfrom typing import Annotatedfrom fastapi import APIRouter, Cookie, Depends, HTTPException, Query@@ -15,6 +16,7 @@ from ..config import SESSION_COOKIE_NAMEfrom ..db import get_dbfrom ..models import AuditLog, ChatMessage, Conversation, User+logger = logging.getLogger(__name__)router = APIRouter()@@ -148,7 +150,11 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,)except Exception as exc:user_message = main.agent_error_message(exc)- agent_usage.record(db, kind="chat", status="error", model=agent_providers.current(db).model or "unknown", user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id, error=user_message)+ # The person chatting gets a plain sentence; the provider's own words go to the log and the usage record, which+ # only admins see.+ logger.warning("The agent's chat request failed: %s", main.agent_error_detail(exc))+ agent_usage.record(db, kind="chat", status="error", model=agent_providers.current(db).model or "unknown", user_id=user["id"], conversation_id=payload.conversation_id,+ incident_id=payload.incident_id, error=f"{type(exc).__name__}: {main.agent_error_detail(exc)}")main.record_audit_log(db,user_id=user["id"],diff --git a/src/frontend/src/admin/AgentProvider.jsx b/src/frontend/src/admin/AgentProvider.jsxindex 32b9129..b6ced4c 100644--- a/src/frontend/src/admin/AgentProvider.jsx+++ b/src/frontend/src/admin/AgentProvider.jsx@@ -50,9 +50,9 @@ export default function AgentProvider({ agent, user, secretsConfigured, onChangereturn (<><Row icon="AI" title="Model provider" detail={detail} tone={agent.key_configured ? undefined : 'warn'}>- <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key and the model without spending tokens.' : 'Save an API key first.'} onClick={runTest}>{test.busy ? 'Testing…' : 'Test connection'}</button>+ <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key, the model, and one small request with the agent\'s tools (a few thousand tokens).' : 'Save an API key first.'} onClick={runTest}>{test.busy ? 'Testing…' : 'Test connection'}</button></Row>- {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key and offers {tested.model}.</p>}</div>}+ {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key, offers {tested.model} and takes the agent's tools{tested.tokens ? ` (${tested.tokens.toLocaleString()} tokens)` : ''}.</p>}</div>}<form className="grid items-start gap-3 px-[18px] py-3.5 md:grid-cols-[10rem_minmax(0,1fr)_minmax(0,1.5fr)_auto]" onSubmit={save}><Field label="Provider"><select className="select h-9 w-full" value={provider} disabled={state.busy} onChange={(event) => choose(event.target.value)}>diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.pyindex 40877c8..dff2cf4 100644--- a/tests/accounts/test_new_pages.py+++ b/tests/accounts/test_new_pages.py@@ -351,7 +351,7 @@ def test_a_server_is_renamed_and_keeps_the_name_and_the_install_describes_itselfdef test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monkeypatch):owner = signed_in(app, 1)- monkeypatch.setattr(main, "check_provider_connection", lambda db: {"provider": "Anthropic", "model": "claude-test"})+ monkeypatch.setattr(main, "check_provider_connection", lambda db, user_id=None: {"provider": "Anthropic", "model": "claude-test"})saved = owner.put("/admin/agent/provider", json={"provider": "anthropic", "model": "claude-test", "api_key": "sk-ant-test-secret-value", **confirmed(app, owner)})system = owner.get("/admin/system")@@ -365,7 +365,7 @@ def test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monkdef test_a_failed_agent_connection_test_explains_itself_without_the_key(app, monkeypatch):owner = signed_in(app, 1)- def rejected(db):+ def rejected(db, user_id=None):raise RuntimeError("Error code: 401 - Incorrect API key provided: sk-test-secret-value")monkeypatch.setattr(main, "check_provider_connection", rejected)diff --git a/tests/assistant/test_agent_errors.py b/tests/assistant/test_agent_errors.pyindex 2cbad8e..2293274 100644--- a/tests/assistant/test_agent_errors.py+++ b/tests/assistant/test_agent_errors.py@@ -1,23 +1,31 @@from src.agent.providers import NoApiKey-from src.backend.main import agent_error_message+from src.backend.main import agent_error_detail, agent_error_messagedef test_overloaded_provider_error_is_explained_to_the_user():message = agent_error_message(Exception("503 RESOURCE_EXHAUSTED: model overloaded"))assert "overloaded" in message.lower()- assert "Provider error" in message+++def test_the_chat_gets_a_plain_sentence_and_only_admins_the_providers_words():+ error = Exception("Error code: 400 - {'type': 'invalid_request_error', 'message': 'Schemas contains too many parameters with union types'}")++ plain, detailed = agent_error_message(error), agent_error_message(error, detail=True)++ assert plain.startswith("The AI provider rejected the request") and "Provider error" not in plain and "Schemas" not in plain+ assert "Provider error" in detailed and "too many parameters" in detaileddef test_provider_error_redacts_api_keys():- message = agent_error_message(Exception("status 429 api_key=AIzaSecretValue"))+ message = agent_error_message(Exception("status 429 api_key=AIzaSecretValue"), detail=True)assert "[redacted]" in message- assert "AIzaSecretValue" not in message+ assert "AIzaSecretValue" not in message and "AIzaSecretValue" not in agent_error_detail(Exception("api_key=AIzaSecretValue"))def test_openai_keys_are_redacted_too():- message = agent_error_message(Exception("Error code: 401 - Incorrect API key provided: sk-proj-abc123SECRET"))+ message = agent_error_message(Exception("Error code: 401 - Incorrect API key provided: sk-proj-abc123SECRET"), detail=True)assert "sk-proj-abc123SECRET" not in messageassert "rejected the API key" in messagediff --git a/tests/assistant/test_agent_providers.py b/tests/assistant/test_agent_providers.pyindex 05f8032..c19bb88 100644--- a/tests/assistant/test_agent_providers.py+++ b/tests/assistant/test_agent_providers.py@@ -11,6 +11,7 @@ from src.agent import chat as agent_service, providersfrom src.agent.providers import Completion, NoApiKeyfrom src.agent.providers.claude import ClaudeProviderfrom src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider+from src.agent.providers.openai_responses import OpenAIResponsesProviderfrom src.backend import secrets_boxfrom src.backend.models import AppSetting, Base@@ -77,7 +78,7 @@ def test_without_a_choice_the_agent_has_no_provider(db):providers.get_provider(db)[email protected]("provider, cls, model", [("anthropic", ClaudeProvider, "claude-haiku-5-5"), ("openai", OpenAIProvider, "gpt-6-luna"), ("gemini", OpenAIProvider, "gemini-3.1-flash-lite")])[email protected]("provider, cls, model", [("anthropic", ClaudeProvider, "claude-haiku-5-5"), ("openai", OpenAIResponsesProvider, "gpt-6-luna"), ("gemini", OpenAIProvider, "gemini-3.1-flash-lite")])def test_each_provider_starts_with_its_own_default_model(db, provider, cls, model):providers.save(db, provider, None, "test-key-123")diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.pyindex 7a81a23..d52a1f5 100644--- a/tests/assistant/test_agent_service.py+++ b/tests/assistant/test_agent_service.py@@ -25,11 +25,42 @@ def test_a_reply_without_a_message_is_an_error():make_service().send(1, "conversation", "Hello", ScriptedProvider('{"message":" "}'))+def test_test_connection_tries_one_request_shaped_like_a_diagnosis_and_counts_it(monkeypatch):+ from sqlalchemy import create_engine+ from sqlalchemy.orm import sessionmaker+ from sqlalchemy.pool import StaticPool++ from src.agent import diagnosis, tools+ from src.backend.models import AgentRun, Base+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)+ Base.metadata.create_all(engine)+ db = sessionmaker(bind=engine)()+ provider = ScriptedProvider('{"summary":"x"}', tokens=(3100, 40))+ provider.check = lambda: {"provider": "Anthropic", "model": "claude-test"}+ monkeypatch.setattr(agent_service, "get_provider", lambda db: provider)++ result = agent_service.check_provider_connection(db, user_id=1)++ assert result == {"provider": "Anthropic", "model": "claude-test", "tokens": 3140}+ assert provider.started[0]["tools"] == [tool.name for tool in tools.TOOLS] and provider.started[0]["schema_name"] == "incident_diagnosis"+ assert provider.allowed == [False] # the tools are listed, as in a diagnosis, but not called+ run = db.query(AgentRun).one()+ assert (run.kind, run.status, run.input_tokens, run.user_id) == ("test", "ok", 3100, 1)++ refused = ScriptedProvider(ValueError("Schemas contains too many parameters with union types"))+ refused.check = provider.check+ monkeypatch.setattr(agent_service, "get_provider", lambda db: refused)+ with pytest.raises(ValueError, match="union types"):+ agent_service.check_provider_connection(db)+ assert [run.status for run in db.query(AgentRun).order_by(AgentRun.id)] == ["ok", "error"]+ assert diagnosis.SCHEMA # the same schema a diagnosis uses++def test_the_reply_can_only_be_a_message():assert agent_service.REPLY_SCHEMA["properties"].keys() == {"message"} and agent_service.REPLY_SCHEMA["additionalProperties"] is False-# --- the OpenAI provider ----------------------------------------------------------------------------------------+# --- the Chat Completions provider (Gemini's OpenAI-compatible endpoint) ----------------------------------------from types import SimpleNamespace@@ -53,7 +84,7 @@ def fake_client(completions):return SimpleNamespace(chat=SimpleNamespace(completions=completions), models=SimpleNamespace(retrieve=lambda model: SimpleNamespace(id=model)))-def test_openai_provider_asks_for_the_reply_schema_and_reports_tokens():+def test_the_chat_completions_provider_asks_for_the_reply_schema_and_reports_tokens():completions = FakeCompletions()provider = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions))diff --git a/tests/assistant/test_agent_tool_calls.py b/tests/assistant/test_agent_tool_calls.pyindex fb18609..5f11bae 100644--- a/tests/assistant/test_agent_tool_calls.py+++ b/tests/assistant/test_agent_tool_calls.py@@ -8,6 +8,7 @@ import pytestfrom src.agent.providers import Tool, ToolResultfrom src.agent.providers.claude import ClaudeProviderfrom src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider+from src.agent.providers.openai_responses import OpenAIResponsesProviderANSWER = '{"message":"Done."}'SCHEMA = {"type": "object", "properties": {"message": {"type": "string"}}, "required": ["message"], "additionalProperties": False}@@ -59,19 +60,56 @@ class FakeOpenAI:return self.calls[index]["tool_choice"]+class Item(NS):+ """An output item of the Responses API: the SDK's are models that can be dumped back into the next input."""++ def model_dump(self, **_):+ return {key: value.model_dump() if isinstance(value, Item) else [part.model_dump() for part in value] if isinstance(value, list) else value+ for key, value in vars(self).items()}+++class FakeResponses:+ """OpenAI's Responses API: a turn is the answer (a message with output_text) or tool calls, after a reasoning item."""++ def __init__(self, *turns, refusal=None, raw_arguments=None, status="completed", incomplete=None):+ self.turns, self.calls, self.refusal, self.raw_arguments = list(turns), [], refusal, raw_arguments+ self.status, self.incomplete = status, incomplete++ def create(self, **kwargs):+ self.calls.append(kwargs)+ turn = self.turns.pop(0)+ reasoning = Item(type="reasoning", id="rs_1", summary=[], encrypted_content="opaque")+ if isinstance(turn, str):+ part = Item(type="refusal", refusal=self.refusal) if self.refusal else Item(type="output_text", text=turn, annotations=[])+ output = [reasoning, Item(type="message", id="msg_1", role="assistant", status="completed", content=[part])]+ text = "" if self.refusal else turn+ else:+ output = [reasoning] + [Item(type="function_call", id=f"fc_{call_id}", call_id=call_id, name="get_host_summary", arguments=self.raw_arguments or json.dumps(args))+ for call_id, args in turn]+ text = ""+ return NS(output=output, output_text=text, status=self.status, incomplete_details=NS(reason=self.incomplete), model="gpt-test",+ usage=NS(input_tokens=100, output_tokens=10))++ def sent(self, index):+ return self.calls[index]["input"]++ def tool_choice(self, index):+ return self.calls[index]["tool_choice"]++def claude(fake):return ClaudeProvider("key", "claude-test", client=NS(messages=fake))def openai(fake):- return OpenAIProvider("key", "gpt-test", client=NS(chat=NS(completions=fake)))+ return OpenAIResponsesProvider("key", "gpt-test", client=NS(responses=fake))def gemini(fake):return OpenAIProvider("key", "gpt-test", name="Google Gemini", base_url=GEMINI_BASE_URL, client=NS(chat=NS(completions=fake)))-PROVIDERS = [pytest.param(claude, FakeClaude, id="claude"), pytest.param(openai, FakeOpenAI, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")]+PROVIDERS = [pytest.param(claude, FakeClaude, id="claude"), pytest.param(openai, FakeResponses, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")]@pytest.mark.parametrize("build, Fake", PROVIDERS)@@ -124,9 +162,11 @@ def test_several_tool_calls_in_one_turn_get_all_their_results(build, Fake):conversation.step()sent = fake.sent(1)- if Fake is FakeClaude: # one user message with both results+ if Fake is FakeClaude: # one user message with both resultsassert [block["tool_use_id"] for block in sent[-1]["content"]] == ["call_1", "call_2"]- else: # one tool message per result+ elif Fake is FakeResponses: # one function_call_output item per result+ assert [(item["type"], item["call_id"]) for item in sent[-2:]] == [("function_call_output", "call_1"), ("function_call_output", "call_2")]+ else: # one tool message per resultassert [message["tool_call_id"] for message in sent[-2:]] == ["call_1", "call_2"]@@ -141,6 +181,8 @@ def test_a_failed_tool_is_reported_as_an_error(build, Fake):result = fake.sent(1)[-1]if Fake is FakeClaude:assert result["content"][0]["is_error"] is True+ elif Fake is FakeResponses:+ assert result["output"] == "Error: No such server."else:assert result["content"] == "Error: No such server."@@ -176,18 +218,52 @@ def test_claude_refusing_midway_is_an_error():def test_openai_refusing_is_an_error():with pytest.raises(ValueError, match="refused"):- openai(FakeOpenAI(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+ openai(FakeResponses(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+++def test_gemini_refusing_is_an_error():+ with pytest.raises(ValueError, match="refused"):+ gemini(FakeOpenAI(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()[email protected]("build, Fake", [pytest.param(openai, FakeResponses, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")])@pytest.mark.parametrize("raw", ["{not json", "[1, 2]"])-def test_openai_tool_arguments_must_be_a_json_object(raw):+def test_openai_tool_arguments_must_be_a_json_object(build, Fake, raw):with pytest.raises(ValueError, match="tool arguments"):- openai(FakeOpenAI([("call_1", {})], raw_arguments=raw)).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+ build(Fake([("call_1", {})], raw_arguments=raw)).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+++def test_openai_uses_the_responses_api_with_strict_tools_reasoning_and_nothing_stored():+ fake = FakeResponses(ANSWER)+ openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply", effort="high").step()++ sent = fake.calls[0]+ assert sent["tools"][0] == {"type": "function", "name": "get_host_summary", "description": TOOLS[0].description, "parameters": TOOLS[0].schema, "strict": True}+ assert sent["text"]["format"] == {"type": "json_schema", "name": "reply", "schema": SCHEMA, "strict": True}+ assert sent["reasoning"] == {"effort": "high"} and sent["store"] is False and sent["include"] == ["reasoning.encrypted_content"]+ assert sent["instructions"] == "You are the agent." and sent["input"] == [{"role": "user", "content": "Why is it slow?"}]+++def test_openai_gets_its_reasoning_back_unchanged_with_the_tool_results():+ fake = FakeResponses([("call_1", {"server_id": 1})], ANSWER)+ conversation = openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")+ conversation.step()+ conversation.add_tool_results([ToolResult("call_1", "ok")])+ conversation.step()++ second = fake.sent(1)+ assert [item.get("type") for item in second[1:]] == ["reasoning", "function_call", "function_call_output"]+ assert second[1]["encrypted_content"] == "opaque" and second[2]["call_id"] == "call_1"+++def test_an_openai_answer_cut_off_is_an_error():+ with pytest.raises(ValueError, match="cut off"):+ openai(FakeResponses(ANSWER, status="incomplete", incomplete="max_output_tokens")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()-def test_openai_tools_are_strict_functions():+def test_gemini_tools_are_strict_functions():fake = FakeOpenAI(ANSWER)- openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+ gemini(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()function = fake.calls[0]["tools"][0]["function"]assert fake.calls[0]["tools"][0]["type"] == "function" and function["strict"] is True and function["parameters"] == TOOLS[0].schemadiff --git a/tests/assistant/test_agent_tools.py b/tests/assistant/test_agent_tools.pyindex c667d83..7fdb95e 100644--- a/tests/assistant/test_agent_tools.py+++ b/tests/assistant/test_agent_tools.py@@ -9,7 +9,7 @@ from sqlalchemy import create_enginefrom sqlalchemy.orm import sessionmakerfrom sqlalchemy.pool import StaticPool-from src.agent import tools+from src.agent import chat, diagnosis, toolsfrom src.agent.tools import common, serversfrom src.backend import history, rollupsfrom src.backend.models import (AppSetting, Base, Check, CheckResult, ContainerMetric, FirewallState, HostMetric, HostRollup, IncidentAlert, ListenerState,@@ -108,6 +108,32 @@ def test_every_tool_is_listed_once_with_a_strict_schema():assert set(servers.HOST_METRICS) <= set(history.HOST_METRICS)+def _unions(schema) -> int:+ """Parameters that may hold more than one type (anyOf, or a list of types), as Anthropic counts them."""+ if isinstance(schema, dict):+ return int("anyOf" in schema or isinstance(schema.get("type"), list)) + sum(_unions(value) for value in schema.values())+ if isinstance(schema, list):+ return sum(_unions(value) for value in schema)+ return 0++[email protected]("answer", [diagnosis.SCHEMA, chat.REPLY_SCHEMA], ids=["diagnosis", "chat"])+def test_no_request_has_more_union_typed_parameters_than_the_provider_accepts(answer):+ # Anthropic rejects strict schemas with more than 16 nullable or union-typed parameters over all tools and the+ # answer's schema of one request (#196), so the tools themselves have none.+ assert sum(_unions(tool.schema) for tool in tools.TOOLS) == 0+ assert _unions(answer) <= 16+++def test_an_optional_argument_left_empty_or_null_means_not_given(db):+ empty = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "", "to": " "}, now=NOW)+ null = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None}, now=NOW)++ assert empty.content == null.content and json.loads(empty.content)["to"] == "2026-11-07T12:00+00:00"+ with pytest.raises(tools.ToolError, match="non-empty"):+ tools.run(db, "server_info", {"server": ""}, now=NOW) # a required one may not be empty++def test_an_unknown_tool_is_refused(db):with pytest.raises(tools.ToolError, match="no tool named"):tools.run(db, "run_shell", {"command": "rm -rf /"}, now=NOW)