Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

fix(agent): make tool requests acceptable to Anthropic and OpenAI (#197)

Karlo Kajba Šimanić205c2aa6874f
diff --git a/SECURITY.md b/SECURITY.md
index 4ca3fc7..f68508f 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -78,7 +78,9 @@ the containers' names, images and figures, and similar incidents before it. Befo
masked for secrets (tokens, keys, passwords, credentials in URLs, private keys; `src/agent/masking.py`), and the
model is told the pack is untrusted data. Its answer is checked before anyone sees it: claims that cite no fact
from the pack are dropped, and fix commands that could destroy data or open the machine up are flagged and never
-marked recommended (`src/agent/safety.py`). Nothing the model writes is run. Turn off "Diagnose new incidents
+marked recommended (`src/agent/safety.py`). Nothing the model writes is run. Requests to OpenAI are sent with
+`store: false`, so OpenAI keeps no copy of the conversation for later retrieval (its own abuse-monitoring retention
+still applies). Turn off "Diagnose new incidents
automatically" in Admin › Incident agent to send nothing unless someone asks; without a key nothing is sent
and incidents keep the rule-based diagnosis, which is worked out locally.
diff --git a/TESTS.md b/TESTS.md
index ad4fe44..80d3cc8 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,
covers (and what it doesn't), and what every single test is for, so you can decide
whether a failure matters before you upgrade or change your setup.
-- **1563 test cases** from 849 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
+- **1574 test cases** from 857 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.
- **Safe to run next to a live install.** The tests use a throwaway in-memory database
and fakes; they never touch your running containers or your data. (Two collector
@@ -146,13 +146,13 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
|---|---|---|
| [`test_account_security.py`](#test_account_securitypy) | 146 | Two-factor sign-in, recovery codes, encrypted secrets, sessions you can review and end, password change, and the password check before turning two-factor on. |
| [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. |
-| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |
-| [`test_agent_service.py`](#test_agent_servicepy) | 9 | The chat's reply is one message; the OpenAI provider and conversation history. |
+| [`test_agent_errors.py`](#test_agent_errorspy) | 6 | Helpful, non-leaking error messages from the AI assistant: a plain sentence in the chat, the provider's own words only for admins. |
+| [`test_agent_service.py`](#test_agent_servicepy) | 10 | The chat's reply is one message; Test connection trying a real request; the Chat Completions provider and conversation history. |
| [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |
-| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 24 | Tool calling with Claude, OpenAI and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer. |
+| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. |
| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
| [`test_agent_investigation.py`](#test_agent_investigationpy) | 27 | The agent looking things up: the budget of tool calls per diagnosis and chat message, forced final answers, errors, the audit log, citations from tool results. |
-| [`test_agent_tools.py`](#test_agent_toolspy) | 49 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. |
+| [`test_agent_tools.py`](#test_agent_toolspy) | 52 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. |
| [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. |
| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |
| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |
@@ -312,7 +312,8 @@ shouldn't, or an admin might be locked out.
| Test | What it checks |
|---|---|
| `test_overloaded_provider_error_is_explained_to_the_user` | When the AI provider reports it is overloaded, the user gets a readable explanation. |
-| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the chat or logs; it is replaced with `[redacted]`. |
+| `test_the_chat_gets_a_plain_sentence_and_only_admins_the_providers_words` | A request the provider rejects is a plain sentence in the chat; only the admin's Test connection adds the provider's own error (#196). |
+| `test_provider_error_redacts_api_keys` | Error text from the provider never leaks an API key to the admin or the logs; it is replaced with `[redacted]`. |
| `test_openai_keys_are_redacted_too` | An OpenAI key (`sk-…`) in an error is redacted, and a rejected key is explained as such. |
| `test_a_missing_key_says_where_to_set_it` | Without a key the chat says to add one in Admin › Incident agent. |
| `test_an_unknown_model_points_at_the_model_setting` | A model the key can't use points the admin at the model in Admin › Incident agent. |
@@ -323,8 +324,9 @@ shouldn't, or an admin might be locked out.
|---|---|
| `test_without_a_tool_budget_the_chat_is_one_call_without_tools` | With the chat's look-up budget at 0, a message is one call to the model, with no tools and effort `low`. |
| `test_a_reply_without_a_message_is_an_error` | A reply whose message is missing or empty is an error, not a blank answer. |
+| `test_test_connection_tries_one_request_shaped_like_a_diagnosis_and_counts_it` | Test connection checks the key, then sends one request with every tool and the diagnosis schema (tools not allowed), counts it as a `test` call, and raises the provider's refusal (#196). |
| `test_the_reply_can_only_be_a_message` | The reply schema has one field, the message, and nothing else is allowed. |
-| `test_openai_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests use Structured Outputs in strict mode with the reply schema, and token usage is read from the response. |
+| `test_the_chat_completions_provider_asks_for_the_reply_schema_and_reports_tokens` | Requests to the Chat Completions provider (Gemini) use Structured Outputs in strict mode with the reply schema, and token usage is read from the response. |
| `test_a_refusal_is_an_error_not_an_empty_reply` | When the model refuses, the chat gets an error instead of an empty answer. |
| `test_test_connection_spends_no_tokens` | Admin's Test connection only looks the model up; it sends no prompt. |
| `test_a_conversation_sends_the_system_prompt_and_its_history` | Each request carries the system prompt and the conversation so far, in order. |
@@ -347,21 +349,27 @@ shouldn't, or an admin might be locked out.
### test_agent_tool_calls.py
-Most tests run once per provider (Claude, OpenAI, Gemini), with fake clients.
+Most tests run once per provider (Claude, OpenAI through the Responses API, Gemini through Chat Completions), with
+fake clients. The fakes can't know each provider's own limits (#196), which is why Test connection sends one real
+request.
| Test | What it checks |
|---|---|
| `test_with_tools_the_model_can_still_answer_at_once` *(3 inputs)* | With tools offered, a model that answers straight away gives the final answer and its token usage. |
| `test_a_tool_call_is_answered_and_the_history_goes_back_unchanged` *(3 inputs)* | A tool call comes back parsed; after its result, the next request repeats every earlier turn exactly as sent and adds the result. |
| `test_claude_gets_its_own_answer_back_untouched_thinking_included` | Claude's own turn, thinking included, goes back unchanged; its tools are strict and the system prompt is sent apart. |
-| `test_several_tool_calls_in_one_turn_get_all_their_results` *(3 inputs)* | Several calls in one turn are all answered: in one message for Claude, one tool message each for OpenAI and Gemini. |
+| `test_several_tool_calls_in_one_turn_get_all_their_results` *(3 inputs)* | Several calls in one turn are all answered: in one message for Claude, one `function_call_output` item each for OpenAI, one tool message each for Gemini. |
| `test_a_failed_tool_is_reported_as_an_error` *(3 inputs)* | A failed tool is marked as an error for the model. |
| `test_tool_results_are_masked_before_they_leave` *(3 inputs)* | A password in a tool result is masked before it is sent to the provider. |
| `test_without_tools_allowed_the_model_must_answer_but_the_tools_stay_listed` *(3 inputs)* | When the budget is used up the model is told to answer (`tool_choice` none), while the tool list stays the same. |
| `test_claude_refusing_midway_is_an_error` | A refusal from Claude in the middle of an investigation is an error. |
| `test_openai_refusing_is_an_error` | A refusal from OpenAI with tools offered is an error. |
-| `test_openai_tool_arguments_must_be_a_json_object` *(2 inputs)* | Tool arguments that aren't valid JSON, or aren't an object, are refused. |
-| `test_openai_tools_are_strict_functions` | OpenAI and Gemini get strict function tools and a strict answer schema. |
+| `test_gemini_refusing_is_an_error` | A refusal from Gemini with tools offered is an error. |
+| `test_openai_tool_arguments_must_be_a_json_object` *(4 inputs)* | For OpenAI and Gemini, tool arguments that aren't valid JSON, or aren't an object, are refused. |
+| `test_openai_uses_the_responses_api_with_strict_tools_reasoning_and_nothing_stored` | OpenAI gets strict function tools, a strict answer schema, the reasoning effort, the system prompt as instructions, `store: false` and the encrypted reasoning back. |
+| `test_openai_gets_its_reasoning_back_unchanged_with_the_tool_results` | The next request repeats OpenAI's reasoning item (encrypted) and its tool call unchanged, followed by the result. |
+| `test_an_openai_answer_cut_off_is_an_error` | A response stopped at the output limit is an error, not a half answer. |
+| `test_gemini_tools_are_strict_functions` | Gemini gets strict function tools and a strict answer schema. |
### test_agent_diagnosis.py
@@ -425,6 +433,8 @@ and a firewall.
| Test | What it checks |
|---|---|
+| `test_no_request_has_more_union_typed_parameters_than_the_provider_accepts` *(2 inputs)* | The tools have no nullable or union-typed parameter, and with the diagnosis or chat schema a request stays within Anthropic's limit of 16 (#196). |
+| `test_an_optional_argument_left_empty_or_null_means_not_given` | An optional argument sent as an empty string or null gives the same result (not given); a required one may not be empty. |
| `test_every_tool_is_listed_once_with_a_strict_schema` | The eleven tools are each listed once, every schema is strict (all fields required, nothing extra allowed), and every host metric offered can be read. |
| `test_an_unknown_tool_is_refused` | A call to a tool that isn't on the list (`run_shell`) is refused. |
| `test_bad_arguments_are_refused_with_a_reason_the_model_can_act_on` *(17 inputs)* | Missing or extra arguments, wrong types, values outside a list, oversized text, bad or reversed times, windows older than 90 days, container snapshots older than 7 days, and unknown checks or containers are refused before anything is read, with a reason (and the names that do exist). |
diff --git a/docs/api.md b/docs/api.md
index 0217320..69be2b6 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -38,7 +38,7 @@ All paths are relative to `/api` when going through the frontend/nginx.
| `GET /admin/agent/worker` | admin | Whether the agent worker is running, how many jobs wait, are running or wait to be retried, and the latest results. |
| `PUT /admin/agent/provider` | admin (rate limited) + `reauth_token` and `code` | Choose the agent's provider (`anthropic`, `openai`, `gemini`) and model, and optionally save a new API key, stored encrypted (409 without `SECRETS_KEY`). Returns the choice and whether a key is saved, never the key. |
| `DELETE /admin/agent/key` | admin | Remove the chosen provider's API key; the agent stops until a new one is saved. |
-| `POST /admin/agent/test` | admin | Checks the saved key and that the configured model exists, without spending tokens. |
+| `POST /admin/agent/test` | admin | Checks the saved key, that the configured model exists, and that the provider accepts a request shaped like a diagnosis (every tool and the answer schema, tools not allowed). That request costs a few thousand tokens and is counted in the usage as kind `test`; the reply adds `tokens`. At most 10 in 10 minutes. |
| `POST /monitoring/collectors/register` | join token (rate limited) | Collector asks to join; returns a poll secret and pairing code. |
| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. |
| `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. |
diff --git a/docs/assistant.md b/docs/assistant.md
index 6e2dc5a..2cf8684 100644
--- a/docs/assistant.md
+++ b/docs/assistant.md
@@ -23,8 +23,12 @@ change files on your servers. Everything about it is in `src/agent/` and in **Ad
| OpenAI | `gpt-6-luna` |
| Google Gemini | `gemini-3.1-flash-lite` |
- Another model works if it supports structured (JSON schema) output.
-4. **Test connection** checks the key and that the model exists, without spending tokens.
+ Another model works if it supports structured (JSON schema) output and tool calls. With OpenAI it must be a
+ reasoning model (the gpt-5 family and later): the agent sets how hard it thinks.
+4. **Test connection** checks the key, that the model exists, and that the provider accepts the agent's real
+ requests: one small request with every tool and the diagnosis schema. It costs a few thousand tokens (about
+ 2 000 to 4 500 in our tests), counted in the usage like any other call. Run it after changing the provider or
+ the model; a provider that refuses the request says why here, instead of every diagnosis failing later.
5. Enter the model's price per million tokens under **Prices** (from the provider's pricing page), so its calls
count towards the budget, and set a **Monthly budget** if you want one.
@@ -112,7 +116,10 @@ facts and the conversation so far (its latest 40 turns), and the model may look
[read-only tools](#what-the-agent-can-look-up) as a diagnosis, up to 4 calls per message by default. Only the
questions and replies stay in the conversation; the look-ups of an earlier message aren't sent again. Every reply
is held to a JSON schema with the provider's structured output: one message. All three providers sit behind one
-small interface (`src/agent/providers/`).
+small interface (`src/agent/providers/`): Claude through Anthropic's Messages API; OpenAI through its Responses API
+(the only one where its reasoning models can use tools), sent with `store: false` so nothing is kept at OpenAI, with
+the model's reasoning carried from one step to the next in encrypted form; Gemini through Google's OpenAI-compatible
+Chat Completions endpoint.
With Claude, an answer that may use tools thinks harder (effort `high`); a chat message without tools thinks
briefly (`low`). The other providers ignore the setting.
@@ -179,7 +186,10 @@ different windows, the label keeps both.
**Adding a tool.** Write the function in the module for its area (`servers.py`, `containers.py`, `activity.py`):
it gets the database session, the checked arguments, the time and a `Facts` collector, and returns a dict whose
facts each carry an `id` added with `facts.add`. Register it in `tools/__init__.py` with a description and a
-strict schema (every property required, `additionalProperties: false`, null for "not given"), and add tests to
+strict schema (every property required, `additionalProperties: false`, and an optional text argument made with
+`_optional`, a plain string where empty means "not given", never a nullable type: Anthropic refuses strict schemas
+with more than 16 nullable or union-typed parameters per request, counted over every tool and the answer's schema,
+and `test_no_request_has_more_union_typed_parameters_than_the_provider_accepts` checks it), and add tests to
`tests/assistant/test_agent_tools.py`: the shared tests then check that it stays bounded, masked, labelled and
read-only.
diff --git a/src/agent/chat.py b/src/agent/chat.py
index bc446ba..085c527 100644
--- a/src/agent/chat.py
+++ b/src/agent/chat.py
@@ -5,8 +5,10 @@ from __future__ import annotations
import json
import threading
+import time
from collections.abc import Callable
+from . import diagnosis, tools, usage
from .investigate import Auditor, Recorder, investigate
from .prompts import SYSTEM_PROMPT
from .providers import Completion, Provider, get_provider
@@ -108,6 +110,23 @@ def delete_agent_conversation(user_id: int, conversation_id: str) -> None:
_agent_service.delete_conversation(user_id, conversation_id)
-def check_provider_connection(db) -> dict:
- """For Admin's "Test connection": the key works and the configured model exists. Spends no tokens."""
- return get_provider(db).check()
+TRIAL_MESSAGES = [{"role": "system", "content": "This is a connection test. Do not call any tool."},
+ {"role": "user", "content": "Answer with the shortest values the schema allows."}]
+
+
+def check_provider_connection(db, user_id: int | None = None) -> dict:
+ """For Admin's "Test connection": the key works, the configured model exists, and the provider accepts a
+ request shaped like a diagnosis: every tool and the diagnosis schema, tools not allowed (#196). That one request
+ is the only thing here that costs tokens (a few thousand, mostly the tools' definitions), and it is
+ counted in the usage like any other (kind ``test``). Raises the provider's error when it is refused."""
+ provider = get_provider(db)
+ found = provider.check()
+ started = time.monotonic()
+ try:
+ completion = provider.start(TRIAL_MESSAGES, tools.TOOLS, diagnosis.SCHEMA, "incident_diagnosis", effort="low").step(allow_tools=False).completion
+ except Exception as exc:
+ usage.record(db, kind="test", status="error", model=provider.model, user_id=user_id, duration_ms=round((time.monotonic() - started) * 1000), error=f"{type(exc).__name__}: {exc}")
+ raise
+ usage.record(db, kind="test", model=completion.model or provider.model, input_tokens=completion.input_tokens, output_tokens=completion.output_tokens,
+ duration_ms=round((time.monotonic() - started) * 1000), user_id=user_id)
+ return {**found, "tokens": completion.input_tokens + completion.output_tokens}
diff --git a/src/agent/providers/__init__.py b/src/agent/providers/__init__.py
index 159c386..df14d6b 100644
--- a/src/agent/providers/__init__.py
+++ b/src/agent/providers/__init__.py
@@ -15,6 +15,7 @@ from src.backend.models import AppSetting
from .base import Completion, Conversation, Provider, Tool, ToolCall, ToolResult, Turn
from .claude import ClaudeProvider
from .openai_compatible import GEMINI_BASE_URL, OpenAIProvider
+from .openai_responses import OpenAIResponsesProvider
__all__ = ["Completion", "Conversation", "Provider", "Tool", "ToolCall", "ToolResult", "Turn", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]
@@ -85,7 +86,7 @@ def get_provider(db) -> Provider:
return ClaudeProvider(choice.key, choice.model)
if choice.provider == "gemini":
return OpenAIProvider(choice.key, choice.model, name="Google Gemini", base_url=GEMINI_BASE_URL)
- return OpenAIProvider(choice.key, choice.model)
+ return OpenAIResponsesProvider(choice.key, choice.model)
def save(db, provider: str, model: str | None, api_key: str | None) -> None:
diff --git a/src/agent/providers/openai_compatible.py b/src/agent/providers/openai_compatible.py
index 8ca87ad..f13b28a 100644
--- a/src/agent/providers/openai_compatible.py
+++ b/src/agent/providers/openai_compatible.py
@@ -1,4 +1,5 @@
-"""OpenAI, and Google Gemini through its OpenAI-compatible endpoint: both take the same Chat Completions request."""
+"""Google Gemini through its OpenAI-compatible endpoint, which takes a Chat Completions request. OpenAI itself uses
+the Responses API (openai_responses.py), since its reasoning models can't use tools through Chat Completions."""
from __future__ import annotations
import json
diff --git a/src/agent/providers/openai_responses.py b/src/agent/providers/openai_responses.py
new file mode 100644
index 0000000..b80901c
--- /dev/null
+++ b/src/agent/providers/openai_responses.py
@@ -0,0 +1,81 @@
+"""OpenAI through the Responses API: tools, reasoning and a strict JSON answer in one request (#196).
+
+OpenAI's reasoning models (the default, gpt-6-luna, among them) can't use tools through Chat Completions, so OpenAI
+has its own provider here; Gemini keeps the OpenAI-compatible Chat Completions endpoint (openai_compatible.py).
+
+Nothing is kept at OpenAI: every request is sent with ``store=False``. The model's reasoning still carries over
+from one step to the next, because each response's items, including the reasoning in encrypted form
+(``reasoning.encrypted_content``), go back unchanged with the next request.
+"""
+from __future__ import annotations
+
+import json
+
+from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion, Conversation, Tool, ToolCall, ToolResult, Turn
+
+
+class OpenAIResponsesProvider:
+ name = "OpenAI"
+
+ def __init__(self, api_key: str, model: str, client=None):
+ self.model = model
+ if client is None:
+ from openai import OpenAI # imported here so the rest of the app runs without the package
+ client = OpenAI(api_key=api_key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
+ self.client = client
+
+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> ResponsesConversation:
+ return ResponsesConversation(self, messages, tools, schema, schema_name, effort)
+
+ def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
+ return self.start(messages, [], schema, schema_name, effort).step().completion
+
+ def check(self) -> dict:
+ model = self.client.models.retrieve(self.model)
+ return {"provider": self.name, "model": getattr(model, "id", self.model)}
+
+
+def _item(item) -> dict:
+ """An output item as the next request's input, unchanged."""
+ return item.model_dump(mode="json", exclude_none=True) if hasattr(item, "model_dump") else dict(item)
+
+
+class ResponsesConversation(Conversation):
+ def __init__(self, provider: OpenAIResponsesProvider, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str):
+ self.provider, self.effort = provider, effort
+ self.instructions = "\n\n".join(message["content"] for message in messages if message["role"] == "system")
+ self.history = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]
+ self.text = {"format": {"type": "json_schema", "name": schema_name, "schema": schema, "strict": True}}
+ self.tools = [{"type": "function", "name": tool.name, "description": tool.description, "parameters": tool.schema, "strict": True} for tool in tools]
+
+ def step(self, allow_tools: bool = True) -> Turn:
+ extra = {"tools": self.tools, "tool_choice": "auto" if allow_tools else "none"} if self.tools else {}
+ response = self.provider.client.responses.create(
+ model=self.provider.model, instructions=self.instructions or None, input=list(self.history), text=self.text,
+ reasoning={"effort": self.effort}, store=False, include=["reasoning.encrypted_content"], **extra,
+ )
+ if getattr(response, "status", None) == "incomplete":
+ reason = getattr(getattr(response, "incomplete_details", None), "reason", None)
+ raise ValueError("The model's answer was cut off before it was complete." if reason == "max_output_tokens" else f"The model stopped before answering ({reason or 'incomplete'}).")
+ output = list(response.output or [])
+ for item in output:
+ if item.type == "message" and any(getattr(part, "type", None) == "refusal" for part in item.content or []):
+ raise ValueError("The model refused to answer.")
+ try:
+ calls = [ToolCall(item.call_id, item.name, json.loads(item.arguments)) for item in output if item.type == "function_call"]
+ except (json.JSONDecodeError, TypeError) as exc:
+ raise ValueError("The model sent tool arguments that aren't valid JSON.") from exc
+ if any(not isinstance(call.arguments, dict) for call in calls):
+ raise ValueError("The model sent tool arguments that aren't a JSON object.")
+ self.history.extend(_item(item) for item in output) # reasoning included, so the next step goes on from it
+ usage = getattr(response, "usage", None)
+ return Turn(Completion(
+ text="" if calls else response.output_text or "",
+ model=getattr(response, "model", self.provider.model),
+ input_tokens=getattr(usage, "input_tokens", 0) or 0,
+ output_tokens=getattr(usage, "output_tokens", 0) or 0, # reasoning tokens included
+ ), calls)
+
+ def _append_results(self, results: list[ToolResult]) -> None:
+ for result in results:
+ self.history.append({"type": "function_call_output", "call_id": result.call_id, "output": f"Error: {result.content}" if result.is_error else result.content})
diff --git a/src/agent/tools/__init__.py b/src/agent/tools/__init__.py
index 2c12eef..cb9a1a4 100644
--- a/src/agent/tools/__init__.py
+++ b/src/agent/tools/__init__.py
@@ -33,14 +33,20 @@ def _object(**properties) -> dict:
return {"type": "object", "properties": properties, "required": list(properties), "additionalProperties": False}
-def _nullable(kind: str, description: str) -> dict:
- return {"anyOf": [{"type": kind}, {"type": "null"}], "description": description}
+class _Optional(dict):
+ """A text argument that may be left empty ("" or null), which the tool reads as "not given". It is a plain
+ string in the schema because strict schemas allow at most 16 nullable or union-typed parameters per request,
+ counted over every tool and the answer's schema (Anthropic rejects more); see docs/assistant.md."""
+
+
+def _optional(description: str) -> dict:
+ return _Optional(type="string", description=description)
SERVER = {"type": "string", "description": "The server's hostname, as the evidence names it."}
-FROM = _nullable("string", "Start of the window, ISO 8601 in UTC (2026-11-07T10:00). null: 24 hours before `to`.")
-TO = _nullable("string", "End of the window, ISO 8601 in UTC. null: now.")
-AT = _nullable("string", "The moment to look at, ISO 8601 in UTC. null: now.")
+FROM = _optional("Start of the window, ISO 8601 in UTC (2026-11-07T10:00). Empty: 24 hours before `to`.")
+TO = _optional("End of the window, ISO 8601 in UTC. Empty: now.")
+AT = _optional("The moment to look at, ISO 8601 in UTC. Empty: now.")
WINDOW_NOTE = (f" Any window within the last {MAX_DAYS} days: every check-in for the last 7 days, hourly averages and peaks before that."
" Start wide, then narrow down to the interesting hours.")
@@ -66,7 +72,7 @@ _REGISTRY: dict[str, tuple[Tool, Handler]] = {tool.name: (tool, handler) for too
"and (for the last 7 days) each change of status, health, exit code, OOM kill or image." + WINDOW_NOTE,
_object(server=SERVER, container={"type": "string", "description": "The container's name."}, **{"from": FROM, "to": TO})), containers.container_history),
(Tool("disk_usage", "Every filesystem of a server (or one): used space and inodes now, usage over the last `days`, growth in GB per day and, when it grows, "
- "in about how many days it fills up.", _object(server=SERVER, mount=_nullable("string", "One mount point such as /var. null: all of them."),
+ "in about how many days it fills up.", _object(server=SERVER, mount=_optional("One mount point such as /var. Empty: all of them."),
days={"type": "integer", "description": f"How many days back to look at, 1 to {MAX_DAYS}."})), servers.disk_usage),
(Tool("events", "What the server recorded in a window: reboots, OOM kills, kernel faults and errors, image changes, containers added or removed, ports "
"opened or closed, firewall changes, package updates. The newest 30, oldest first." + WINDOW_NOTE,
@@ -77,35 +83,32 @@ _REGISTRY: dict[str, tuple[Tool, Handler]] = {tool.name: (tool, handler) for too
servers.compare_to_baseline),
(Tool("find_incidents", "Earlier incidents on one server or all of them: title, kind, severity, how long they lasted and how they were resolved. Newest "
"first. The window defaults to the last 30 days.",
- _object(server=_nullable("string", "A hostname. null: every server."),
- kind=_nullable("string", "Only this kind, as in the evidence's incident `kind` (host:disk, container:restart_loop, …). null: every kind."),
- **{"from": _nullable("string", "Start of the window, ISO 8601 in UTC. null: 30 days before `to`."), "to": TO})), activity.find_incidents),
+ _object(server=_optional("A hostname. Empty: every server."),
+ kind=_optional("Only this kind, as in the evidence's incident `kind` (host:disk, container:restart_loop, …). Empty: every kind."),
+ **{"from": _optional("Start of the window, ISO 8601 in UTC. Empty: 30 days before `to`."), "to": TO})), activity.find_incidents),
(Tool("fleet_snapshot", "Every server at one moment: online or not, open incidents and the worst of them, availability checks failing from it. "
"Shows whether a problem is on one machine or many.", _object(at=AT)), activity.fleet_snapshot),
(Tool("check_results", "Availability checks (HTTP, TCP, TLS, DNS, ping …) over a window: uptime, when they were down, the errors they got and response "
"times. Give one check by name, or a server for every check that runs from it." + WINDOW_NOTE,
- _object(check=_nullable("string", "The check's name."), server=_nullable("string", "A hostname: every check run from it."), **{"from": FROM, "to": TO})),
+ _object(check=_optional("The check's name."), server=_optional("A hostname: every check run from it."), **{"from": FROM, "to": TO})),
activity.check_results),
(Tool("network_state", "A server's listening ports (and whether other machines can reach them), its firewall configuration, and port and firewall changes "
"in the last 7 days.", _object(server=SERVER)), servers.network_state),
]}
-TOOLS: list[Tool] = [tool for tool, _ in _REGISTRY.values()]
+# What the provider gets: the same schemas as plain JSON (the _Optional marker is only for checking arguments here).
+TOOLS: list[Tool] = [Tool(tool.name, tool.description, json.loads(json.dumps(tool.schema))) for tool, _ in _REGISTRY.values()]
+
+
+def _blank(value) -> bool:
+ return value is None or (isinstance(value, str) and not value.strip())
def _check(schema: dict, value, path: str) -> None:
"""``value`` against the small part of JSON Schema the tools use. Raises ToolError naming the argument."""
- if "anyOf" in schema:
- for option in schema["anyOf"]:
- try:
- return _check(option, value, path)
- except ToolError:
- continue
- raise ToolError(f"{path} has the wrong type.")
+ if isinstance(schema, _Optional) and _blank(value):
+ return
kind = schema["type"]
- if kind == "null":
- if value is not None:
- raise ToolError(f"{path} must be null.")
- elif kind == "object":
+ if kind == "object":
if not isinstance(value, dict) or set(value) != set(schema["properties"]):
raise ToolError(f"Give exactly these arguments: {', '.join(schema['properties'])}.")
for key, sub in schema["properties"].items():
@@ -165,6 +168,8 @@ def run(db, name: str, arguments, now: datetime | None = None) -> Output:
raise ToolError(f"There is no tool named {str(name)[:60]!r}.")
tool, handler = _REGISTRY[name]
_check(tool.schema, arguments, "arguments")
+ # An optional argument left empty reaches the tool as None, whether the model sent "" or null.
+ arguments = {key: None if isinstance(tool.schema["properties"][key], _Optional) and _blank(value) else value for key, value in arguments.items()}
now = now or datetime.now(timezone.utc)
facts = Facts()
text = _fit(mask(handler(db, arguments, now, facts)))
diff --git a/src/backend/main.py b/src/backend/main.py
index fd9c9c5..b7dd55d 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -222,8 +222,10 @@ class MetricsIngestRequest(BaseModel):
containers: list[dict] = []
-def agent_error_message(exc: Exception) -> str:
- """Turn provider failures into useful, non-sensitive chat text."""
+def agent_error_message(exc: Exception, detail: bool = False) -> str:
+ """Turn provider failures into useful, non-sensitive text. Only ``detail=True`` (for admins, such as Test
+ connection) adds the provider's own words, redacted; the chat gets the plain sentence and the details go to the
+ server log and the agent's usage record (agent_error_detail)."""
if isinstance(exc, NoApiKey):
return "The agent has no API key. An admin adds one in Admin › Incident agent."
raw_message = str(exc).strip()
@@ -238,16 +240,17 @@ def agent_error_message(exc: Exception) -> str:
prefix = "The AI provider's rate limit or quota was reached. Please try again later."
elif any(marker in normalized for marker in ("timeout", "timed out", "deadline exceeded")):
prefix = "The AI provider did not respond before the request timed out. Please try again."
+ elif any(marker in normalized for marker in (" 400", "status 400", "invalid_request")):
+ prefix = "The AI provider rejected the request. An admin finds the reason in the backend's log, or with Test connection in Admin › Incident agent."
else:
prefix = "The AI provider could not complete this request."
+ return f"{prefix} Provider error: {agent_error_detail(exc)}" if detail and raw_message else prefix
- if not raw_message:
- return prefix
- # Errors can include request headers or provider diagnostics: keep a short,
- # redacted detail for the administrator without leaking credentials.
- detail = re.sub(r"(?i)(api[_ -]?key|authorization|token)\s*[:=]\s*\S+", r"\1=[redacted]", raw_message)
- detail = re.sub(r"(AIza|sk-)[\w-]+", "[redacted]", detail)[:300]
- return f"{prefix} Provider error: {detail}"
+
+def agent_error_detail(exc: Exception) -> str:
+ """The provider's own error, short and with keys redacted: errors can include request headers or diagnostics."""
+ text = re.sub(r"(?i)(api[_ -]?key|authorization|token)\s*[:=]\s*\S+", r"\1=[redacted]", str(exc).strip())
+ return re.sub(r"(AIza|sk-)[\w-]+", "[redacted]", text)[:300]
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
diff --git a/src/backend/models.py b/src/backend/models.py
index 561e5d8..034d2d8 100644
--- a/src/backend/models.py
+++ b/src/backend/models.py
@@ -509,7 +509,7 @@ class AgentRun(Base):
__table_args__ = (Index("ix_agent_runs_created", "created_at"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
- kind: Mapped[str] = mapped_column(String(16), nullable=False) # chat | diagnosis
+ kind: Mapped[str] = mapped_column(String(16), nullable=False) # chat | diagnosis | test (Admin's Test connection)
status: Mapped[str] = mapped_column(String(16), nullable=False) # ok | error
model: Mapped[str] = mapped_column(String(100), nullable=False)
input_tokens: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
diff --git a/src/backend/routes/admin.py b/src/backend/routes/admin.py
index 87663f1..166f989 100644
--- a/src/backend/routes/admin.py
+++ b/src/backend/routes/admin.py
@@ -161,13 +161,14 @@ def admin_agent_worker(session_id: Annotated[str | None, Cookie(alias=SESSION_CO
@router.post("/admin/agent/test")
def admin_agent_test(session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
- """Admin's "Test connection": the API key works and the configured model exists. Spends no tokens."""
+ """Admin's "Test connection": the API key works, the model exists, and the provider accepts the agent's tools
+ and answer schema (one small request of a few thousand tokens; see check_provider_connection)."""
admin = main._require_admin(session_id, db)
main.enforce_limit(main.redis_client, scope="agent-test", subject=admin["id"], limit=10, window_seconds=600)
try:
- result = main.check_provider_connection(db)
+ result = main.check_provider_connection(db, user_id=admin["id"])
except Exception as exc:
- raise HTTPException(status_code=502, detail=main.agent_error_message(exc)) from exc
+ raise HTTPException(status_code=502, detail=main.agent_error_message(exc, detail=True)) from exc
return {"ok": True, **result}
diff --git a/src/backend/routes/chat.py b/src/backend/routes/chat.py
index e00c534..3b05db5 100644
--- a/src/backend/routes/chat.py
+++ b/src/backend/routes/chat.py
@@ -4,6 +4,7 @@ tune them) and the shared helpers (`_incident_context`, `send_agent_message`, `_
this module is the HTTP layer."""
from __future__ import annotations
+import logging
from typing import Annotated
from fastapi import APIRouter, Cookie, Depends, HTTPException, Query
@@ -15,6 +16,7 @@ from ..config import SESSION_COOKIE_NAME
from ..db import get_db
from ..models import AuditLog, ChatMessage, Conversation, User
+logger = logging.getLogger(__name__)
router = APIRouter()
@@ -148,7 +150,11 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,
)
except Exception as exc:
user_message = main.agent_error_message(exc)
- agent_usage.record(db, kind="chat", status="error", model=agent_providers.current(db).model or "unknown", user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id, error=user_message)
+ # The person chatting gets a plain sentence; the provider's own words go to the log and the usage record, which
+ # only admins see.
+ logger.warning("The agent's chat request failed: %s", main.agent_error_detail(exc))
+ agent_usage.record(db, kind="chat", status="error", model=agent_providers.current(db).model or "unknown", user_id=user["id"], conversation_id=payload.conversation_id,
+ incident_id=payload.incident_id, error=f"{type(exc).__name__}: {main.agent_error_detail(exc)}")
main.record_audit_log(
db,
user_id=user["id"],
diff --git a/src/frontend/src/admin/AgentProvider.jsx b/src/frontend/src/admin/AgentProvider.jsx
index 32b9129..b6ced4c 100644
--- a/src/frontend/src/admin/AgentProvider.jsx
+++ b/src/frontend/src/admin/AgentProvider.jsx
@@ -50,9 +50,9 @@ export default function AgentProvider({ agent, user, secretsConfigured, onChange
return (
<>
<Row icon="AI" title="Model provider" detail={detail} tone={agent.key_configured ? undefined : 'warn'}>
- <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key and the model without spending tokens.' : 'Save an API key first.'} onClick={runTest}>{test.busy ? 'Testing…' : 'Test connection'}</button>
+ <button type="button" className="btn btn-sm" disabled={!agent.key_configured || test.busy} title={agent.key_configured ? 'Checks the key, the model, and one small request with the agent\'s tools (a few thousand tokens).' : 'Save an API key first.'} onClick={runTest}>{test.busy ? 'Testing…' : 'Test connection'}</button>
</Row>
- {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key and offers {tested.model}.</p>}</div>}
+ {(test.error || tested) && <div className="px-[18px] py-2.5">{test.error ? <p role="alert" className="m-0 text-xs text-warn">{test.error}</p> : <p role="status" className="m-0 text-xs text-ok">Connected. {tested.provider} accepts the key, offers {tested.model} and takes the agent's tools{tested.tokens ? ` (${tested.tokens.toLocaleString()} tokens)` : ''}.</p>}</div>}
<form className="grid items-start gap-3 px-[18px] py-3.5 md:grid-cols-[10rem_minmax(0,1fr)_minmax(0,1.5fr)_auto]" onSubmit={save}>
<Field label="Provider">
<select className="select h-9 w-full" value={provider} disabled={state.busy} onChange={(event) => choose(event.target.value)}>
diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.py
index 40877c8..dff2cf4 100644
--- a/tests/accounts/test_new_pages.py
+++ b/tests/accounts/test_new_pages.py
@@ -351,7 +351,7 @@ def test_a_server_is_renamed_and_keeps_the_name_and_the_install_describes_itself
def test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monkeypatch):
owner = signed_in(app, 1)
- monkeypatch.setattr(main, "check_provider_connection", lambda db: {"provider": "Anthropic", "model": "claude-test"})
+ monkeypatch.setattr(main, "check_provider_connection", lambda db, user_id=None: {"provider": "Anthropic", "model": "claude-test"})
saved = owner.put("/admin/agent/provider", json={"provider": "anthropic", "model": "claude-test", "api_key": "sk-ant-test-secret-value", **confirmed(app, owner)})
system = owner.get("/admin/system")
@@ -365,7 +365,7 @@ def test_admin_tests_the_agent_connection_and_the_key_never_comes_back(app, monk
def test_a_failed_agent_connection_test_explains_itself_without_the_key(app, monkeypatch):
owner = signed_in(app, 1)
- def rejected(db):
+ def rejected(db, user_id=None):
raise RuntimeError("Error code: 401 - Incorrect API key provided: sk-test-secret-value")
monkeypatch.setattr(main, "check_provider_connection", rejected)
diff --git a/tests/assistant/test_agent_errors.py b/tests/assistant/test_agent_errors.py
index 2cbad8e..2293274 100644
--- a/tests/assistant/test_agent_errors.py
+++ b/tests/assistant/test_agent_errors.py
@@ -1,23 +1,31 @@
from src.agent.providers import NoApiKey
-from src.backend.main import agent_error_message
+from src.backend.main import agent_error_detail, agent_error_message
def test_overloaded_provider_error_is_explained_to_the_user():
message = agent_error_message(Exception("503 RESOURCE_EXHAUSTED: model overloaded"))
assert "overloaded" in message.lower()
- assert "Provider error" in message
+
+
+def test_the_chat_gets_a_plain_sentence_and_only_admins_the_providers_words():
+ error = Exception("Error code: 400 - {'type': 'invalid_request_error', 'message': 'Schemas contains too many parameters with union types'}")
+
+ plain, detailed = agent_error_message(error), agent_error_message(error, detail=True)
+
+ assert plain.startswith("The AI provider rejected the request") and "Provider error" not in plain and "Schemas" not in plain
+ assert "Provider error" in detailed and "too many parameters" in detailed
def test_provider_error_redacts_api_keys():
- message = agent_error_message(Exception("status 429 api_key=AIzaSecretValue"))
+ message = agent_error_message(Exception("status 429 api_key=AIzaSecretValue"), detail=True)
assert "[redacted]" in message
- assert "AIzaSecretValue" not in message
+ assert "AIzaSecretValue" not in message and "AIzaSecretValue" not in agent_error_detail(Exception("api_key=AIzaSecretValue"))
def test_openai_keys_are_redacted_too():
- message = agent_error_message(Exception("Error code: 401 - Incorrect API key provided: sk-proj-abc123SECRET"))
+ message = agent_error_message(Exception("Error code: 401 - Incorrect API key provided: sk-proj-abc123SECRET"), detail=True)
assert "sk-proj-abc123SECRET" not in message
assert "rejected the API key" in message
diff --git a/tests/assistant/test_agent_providers.py b/tests/assistant/test_agent_providers.py
index 05f8032..c19bb88 100644
--- a/tests/assistant/test_agent_providers.py
+++ b/tests/assistant/test_agent_providers.py
@@ -11,6 +11,7 @@ from src.agent import chat as agent_service, providers
from src.agent.providers import Completion, NoApiKey
from src.agent.providers.claude import ClaudeProvider
from src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider
+from src.agent.providers.openai_responses import OpenAIResponsesProvider
from src.backend import secrets_box
from src.backend.models import AppSetting, Base
@@ -77,7 +78,7 @@ def test_without_a_choice_the_agent_has_no_provider(db):
providers.get_provider(db)
[email protected]("provider, cls, model", [("anthropic", ClaudeProvider, "claude-haiku-5-5"), ("openai", OpenAIProvider, "gpt-6-luna"), ("gemini", OpenAIProvider, "gemini-3.1-flash-lite")])
[email protected]("provider, cls, model", [("anthropic", ClaudeProvider, "claude-haiku-5-5"), ("openai", OpenAIResponsesProvider, "gpt-6-luna"), ("gemini", OpenAIProvider, "gemini-3.1-flash-lite")])
def test_each_provider_starts_with_its_own_default_model(db, provider, cls, model):
providers.save(db, provider, None, "test-key-123")
diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.py
index 7a81a23..d52a1f5 100644
--- a/tests/assistant/test_agent_service.py
+++ b/tests/assistant/test_agent_service.py
@@ -25,11 +25,42 @@ def test_a_reply_without_a_message_is_an_error():
make_service().send(1, "conversation", "Hello", ScriptedProvider('{"message":" "}'))
+def test_test_connection_tries_one_request_shaped_like_a_diagnosis_and_counts_it(monkeypatch):
+ from sqlalchemy import create_engine
+ from sqlalchemy.orm import sessionmaker
+ from sqlalchemy.pool import StaticPool
+
+ from src.agent import diagnosis, tools
+ from src.backend.models import AgentRun, Base
+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
+ Base.metadata.create_all(engine)
+ db = sessionmaker(bind=engine)()
+ provider = ScriptedProvider('{"summary":"x"}', tokens=(3100, 40))
+ provider.check = lambda: {"provider": "Anthropic", "model": "claude-test"}
+ monkeypatch.setattr(agent_service, "get_provider", lambda db: provider)
+
+ result = agent_service.check_provider_connection(db, user_id=1)
+
+ assert result == {"provider": "Anthropic", "model": "claude-test", "tokens": 3140}
+ assert provider.started[0]["tools"] == [tool.name for tool in tools.TOOLS] and provider.started[0]["schema_name"] == "incident_diagnosis"
+ assert provider.allowed == [False] # the tools are listed, as in a diagnosis, but not called
+ run = db.query(AgentRun).one()
+ assert (run.kind, run.status, run.input_tokens, run.user_id) == ("test", "ok", 3100, 1)
+
+ refused = ScriptedProvider(ValueError("Schemas contains too many parameters with union types"))
+ refused.check = provider.check
+ monkeypatch.setattr(agent_service, "get_provider", lambda db: refused)
+ with pytest.raises(ValueError, match="union types"):
+ agent_service.check_provider_connection(db)
+ assert [run.status for run in db.query(AgentRun).order_by(AgentRun.id)] == ["ok", "error"]
+ assert diagnosis.SCHEMA # the same schema a diagnosis uses
+
+
def test_the_reply_can_only_be_a_message():
assert agent_service.REPLY_SCHEMA["properties"].keys() == {"message"} and agent_service.REPLY_SCHEMA["additionalProperties"] is False
-# --- the OpenAI provider ----------------------------------------------------------------------------------------
+# --- the Chat Completions provider (Gemini's OpenAI-compatible endpoint) ----------------------------------------
from types import SimpleNamespace
@@ -53,7 +84,7 @@ def fake_client(completions):
return SimpleNamespace(chat=SimpleNamespace(completions=completions), models=SimpleNamespace(retrieve=lambda model: SimpleNamespace(id=model)))
-def test_openai_provider_asks_for_the_reply_schema_and_reports_tokens():
+def test_the_chat_completions_provider_asks_for_the_reply_schema_and_reports_tokens():
completions = FakeCompletions()
provider = OpenAIProvider("sk-test", "gpt-test", client=fake_client(completions))
diff --git a/tests/assistant/test_agent_tool_calls.py b/tests/assistant/test_agent_tool_calls.py
index fb18609..5f11bae 100644
--- a/tests/assistant/test_agent_tool_calls.py
+++ b/tests/assistant/test_agent_tool_calls.py
@@ -8,6 +8,7 @@ import pytest
from src.agent.providers import Tool, ToolResult
from src.agent.providers.claude import ClaudeProvider
from src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider
+from src.agent.providers.openai_responses import OpenAIResponsesProvider
ANSWER = '{"message":"Done."}'
SCHEMA = {"type": "object", "properties": {"message": {"type": "string"}}, "required": ["message"], "additionalProperties": False}
@@ -59,19 +60,56 @@ class FakeOpenAI:
return self.calls[index]["tool_choice"]
+class Item(NS):
+ """An output item of the Responses API: the SDK's are models that can be dumped back into the next input."""
+
+ def model_dump(self, **_):
+ return {key: value.model_dump() if isinstance(value, Item) else [part.model_dump() for part in value] if isinstance(value, list) else value
+ for key, value in vars(self).items()}
+
+
+class FakeResponses:
+ """OpenAI's Responses API: a turn is the answer (a message with output_text) or tool calls, after a reasoning item."""
+
+ def __init__(self, *turns, refusal=None, raw_arguments=None, status="completed", incomplete=None):
+ self.turns, self.calls, self.refusal, self.raw_arguments = list(turns), [], refusal, raw_arguments
+ self.status, self.incomplete = status, incomplete
+
+ def create(self, **kwargs):
+ self.calls.append(kwargs)
+ turn = self.turns.pop(0)
+ reasoning = Item(type="reasoning", id="rs_1", summary=[], encrypted_content="opaque")
+ if isinstance(turn, str):
+ part = Item(type="refusal", refusal=self.refusal) if self.refusal else Item(type="output_text", text=turn, annotations=[])
+ output = [reasoning, Item(type="message", id="msg_1", role="assistant", status="completed", content=[part])]
+ text = "" if self.refusal else turn
+ else:
+ output = [reasoning] + [Item(type="function_call", id=f"fc_{call_id}", call_id=call_id, name="get_host_summary", arguments=self.raw_arguments or json.dumps(args))
+ for call_id, args in turn]
+ text = ""
+ return NS(output=output, output_text=text, status=self.status, incomplete_details=NS(reason=self.incomplete), model="gpt-test",
+ usage=NS(input_tokens=100, output_tokens=10))
+
+ def sent(self, index):
+ return self.calls[index]["input"]
+
+ def tool_choice(self, index):
+ return self.calls[index]["tool_choice"]
+
+
def claude(fake):
return ClaudeProvider("key", "claude-test", client=NS(messages=fake))
def openai(fake):
- return OpenAIProvider("key", "gpt-test", client=NS(chat=NS(completions=fake)))
+ return OpenAIResponsesProvider("key", "gpt-test", client=NS(responses=fake))
def gemini(fake):
return OpenAIProvider("key", "gpt-test", name="Google Gemini", base_url=GEMINI_BASE_URL, client=NS(chat=NS(completions=fake)))
-PROVIDERS = [pytest.param(claude, FakeClaude, id="claude"), pytest.param(openai, FakeOpenAI, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")]
+PROVIDERS = [pytest.param(claude, FakeClaude, id="claude"), pytest.param(openai, FakeResponses, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")]
@pytest.mark.parametrize("build, Fake", PROVIDERS)
@@ -124,9 +162,11 @@ def test_several_tool_calls_in_one_turn_get_all_their_results(build, Fake):
conversation.step()
sent = fake.sent(1)
- if Fake is FakeClaude: # one user message with both results
+ if Fake is FakeClaude: # one user message with both results
assert [block["tool_use_id"] for block in sent[-1]["content"]] == ["call_1", "call_2"]
- else: # one tool message per result
+ elif Fake is FakeResponses: # one function_call_output item per result
+ assert [(item["type"], item["call_id"]) for item in sent[-2:]] == [("function_call_output", "call_1"), ("function_call_output", "call_2")]
+ else: # one tool message per result
assert [message["tool_call_id"] for message in sent[-2:]] == ["call_1", "call_2"]
@@ -141,6 +181,8 @@ def test_a_failed_tool_is_reported_as_an_error(build, Fake):
result = fake.sent(1)[-1]
if Fake is FakeClaude:
assert result["content"][0]["is_error"] is True
+ elif Fake is FakeResponses:
+ assert result["output"] == "Error: No such server."
else:
assert result["content"] == "Error: No such server."
@@ -176,18 +218,52 @@ def test_claude_refusing_midway_is_an_error():
def test_openai_refusing_is_an_error():
with pytest.raises(ValueError, match="refused"):
- openai(FakeOpenAI(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+ openai(FakeResponses(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+
+def test_gemini_refusing_is_an_error():
+ with pytest.raises(ValueError, match="refused"):
+ gemini(FakeOpenAI(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
[email protected]("build, Fake", [pytest.param(openai, FakeResponses, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")])
@pytest.mark.parametrize("raw", ["{not json", "[1, 2]"])
-def test_openai_tool_arguments_must_be_a_json_object(raw):
+def test_openai_tool_arguments_must_be_a_json_object(build, Fake, raw):
with pytest.raises(ValueError, match="tool arguments"):
- openai(FakeOpenAI([("call_1", {})], raw_arguments=raw)).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+ build(Fake([("call_1", {})], raw_arguments=raw)).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+
+def test_openai_uses_the_responses_api_with_strict_tools_reasoning_and_nothing_stored():
+ fake = FakeResponses(ANSWER)
+ openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply", effort="high").step()
+
+ sent = fake.calls[0]
+ assert sent["tools"][0] == {"type": "function", "name": "get_host_summary", "description": TOOLS[0].description, "parameters": TOOLS[0].schema, "strict": True}
+ assert sent["text"]["format"] == {"type": "json_schema", "name": "reply", "schema": SCHEMA, "strict": True}
+ assert sent["reasoning"] == {"effort": "high"} and sent["store"] is False and sent["include"] == ["reasoning.encrypted_content"]
+ assert sent["instructions"] == "You are the agent." and sent["input"] == [{"role": "user", "content": "Why is it slow?"}]
+
+
+def test_openai_gets_its_reasoning_back_unchanged_with_the_tool_results():
+ fake = FakeResponses([("call_1", {"server_id": 1})], ANSWER)
+ conversation = openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+ conversation.step()
+ conversation.add_tool_results([ToolResult("call_1", "ok")])
+ conversation.step()
+
+ second = fake.sent(1)
+ assert [item.get("type") for item in second[1:]] == ["reasoning", "function_call", "function_call_output"]
+ assert second[1]["encrypted_content"] == "opaque" and second[2]["call_id"] == "call_1"
+
+
+def test_an_openai_answer_cut_off_is_an_error():
+ with pytest.raises(ValueError, match="cut off"):
+ openai(FakeResponses(ANSWER, status="incomplete", incomplete="max_output_tokens")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
-def test_openai_tools_are_strict_functions():
+def test_gemini_tools_are_strict_functions():
fake = FakeOpenAI(ANSWER)
- openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+ gemini(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
function = fake.calls[0]["tools"][0]["function"]
assert fake.calls[0]["tools"][0]["type"] == "function" and function["strict"] is True and function["parameters"] == TOOLS[0].schema
diff --git a/tests/assistant/test_agent_tools.py b/tests/assistant/test_agent_tools.py
index c667d83..7fdb95e 100644
--- a/tests/assistant/test_agent_tools.py
+++ b/tests/assistant/test_agent_tools.py
@@ -9,7 +9,7 @@ from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import StaticPool
-from src.agent import tools
+from src.agent import chat, diagnosis, tools
from src.agent.tools import common, servers
from src.backend import history, rollups
from src.backend.models import (AppSetting, Base, Check, CheckResult, ContainerMetric, FirewallState, HostMetric, HostRollup, IncidentAlert, ListenerState,
@@ -108,6 +108,32 @@ def test_every_tool_is_listed_once_with_a_strict_schema():
assert set(servers.HOST_METRICS) <= set(history.HOST_METRICS)
+def _unions(schema) -> int:
+ """Parameters that may hold more than one type (anyOf, or a list of types), as Anthropic counts them."""
+ if isinstance(schema, dict):
+ return int("anyOf" in schema or isinstance(schema.get("type"), list)) + sum(_unions(value) for value in schema.values())
+ if isinstance(schema, list):
+ return sum(_unions(value) for value in schema)
+ return 0
+
+
[email protected]("answer", [diagnosis.SCHEMA, chat.REPLY_SCHEMA], ids=["diagnosis", "chat"])
+def test_no_request_has_more_union_typed_parameters_than_the_provider_accepts(answer):
+ # Anthropic rejects strict schemas with more than 16 nullable or union-typed parameters over all tools and the
+ # answer's schema of one request (#196), so the tools themselves have none.
+ assert sum(_unions(tool.schema) for tool in tools.TOOLS) == 0
+ assert _unions(answer) <= 16
+
+
+def test_an_optional_argument_left_empty_or_null_means_not_given(db):
+ empty = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "", "to": " "}, now=NOW)
+ null = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None}, now=NOW)
+
+ assert empty.content == null.content and json.loads(empty.content)["to"] == "2026-11-07T12:00+00:00"
+ with pytest.raises(tools.ToolError, match="non-empty"):
+ tools.run(db, "server_info", {"server": ""}, now=NOW) # a required one may not be empty
+
+
def test_an_unknown_tool_is_refused(db):
with pytest.raises(tools.ToolError, match="no tool named"):
tools.run(db, "run_shell", {"command": "rm -rf /"}, now=NOW)