Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat(agent): provider-neutral tool calling for Claude, OpenAI and Gemini (#186)
Karlo Kajba Šimanić9197e1567526
diff --git a/TESTS.md b/TESTS.mdindex 04f7276..8e7a8d0 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,covers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **1458 test cases** from 788 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).+- **1482 test cases** from 799 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collector@@ -149,6 +149,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. || [`test_agent_service.py`](#test_agent_servicepy) | 8 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. || [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |+| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 24 | Tool calling with Claude, OpenAI and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer. || [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. || [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. || [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |@@ -342,6 +343,24 @@ shouldn't, or an admin might be locked out.| `test_a_key_that_cant_be_decrypted_counts_as_no_key` | After `SECRETS_KEY` changes, an unreadable key turns the agent off instead of failing. || `test_a_removed_key_turns_the_agent_off` | Removing the key leaves the provider chosen but turns the agent off. |+### test_agent_tool_calls.py++Most tests run once per provider (Claude, OpenAI, Gemini), with fake clients.++| Test | What it checks |+|---|---|+| `test_with_tools_the_model_can_still_answer_at_once` *(3 inputs)* | With tools offered, a model that answers straight away gives the final answer and its token usage. |+| `test_a_tool_call_is_answered_and_the_history_goes_back_unchanged` *(3 inputs)* | A tool call comes back parsed; after its result, the next request repeats every earlier turn exactly as sent and adds the result. |+| `test_claude_gets_its_own_answer_back_untouched_thinking_included` | Claude's own turn, thinking included, goes back unchanged; its tools are strict and the system prompt is sent apart. |+| `test_several_tool_calls_in_one_turn_get_all_their_results` *(3 inputs)* | Several calls in one turn are all answered: in one message for Claude, one tool message each for OpenAI and Gemini. |+| `test_a_failed_tool_is_reported_as_an_error` *(3 inputs)* | A failed tool is marked as an error for the model. |+| `test_tool_results_are_masked_before_they_leave` *(3 inputs)* | A password in a tool result is masked before it is sent to the provider. |+| `test_without_tools_allowed_the_model_must_answer_but_the_tools_stay_listed` *(3 inputs)* | When the budget is used up the model is told to answer (`tool_choice` none), while the tool list stays the same. |+| `test_claude_refusing_midway_is_an_error` | A refusal from Claude in the middle of an investigation is an error. |+| `test_openai_refusing_is_an_error` | A refusal from OpenAI with tools offered is an error. |+| `test_openai_tool_arguments_must_be_a_json_object` *(2 inputs)* | Tool arguments that aren't valid JSON, or aren't an object, are refused. |+| `test_openai_tools_are_strict_functions` | OpenAI and Gemini get strict function tools and a strict answer schema. |+### test_agent_diagnosis.py| Test | What it checks |diff --git a/docs/development.md b/docs/development.mdindex 4740d29..38af3f8 100644--- a/docs/development.md+++ b/docs/development.md@@ -104,7 +104,7 @@ src/secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new keyreset_2fa.py command-line two-factor reset for a locked-out accountagent/ the incident agent; runs in the backend (chat) and the agent-worker container- providers/ the language model behind one interface: Claude, OpenAI, Gemini; the choice and its encrypted key+ providers/ the language model behind one interface (answers and tool calls): Claude, OpenAI, Gemini; the choice and its encrypted keychat.py, prompts.py, SYSTEM_PROMPT.md the incident chat and its instructionshealth_checks.py the fixed, read-only checks the agent may ask forjobs.py job queue in Redis: published after an incident opens or worsensdiff --git a/src/agent/providers/__init__.py b/src/agent/providers/__init__.pyindex 24fab9b..159c386 100644--- a/src/agent/providers/__init__.py+++ b/src/agent/providers/__init__.py@@ -12,11 +12,11 @@ from dataclasses import dataclassfrom src.backend import secrets_boxfrom src.backend.models import AppSetting-from .base import Completion, Provider+from .base import Completion, Conversation, Provider, Tool, ToolCall, ToolResult, Turnfrom .claude import ClaudeProviderfrom .openai_compatible import GEMINI_BASE_URL, OpenAIProvider-__all__ = ["Completion", "Provider", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]+__all__ = ["Completion", "Conversation", "Provider", "Tool", "ToolCall", "ToolResult", "Turn", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]logger = logging.getLogger(__name__)diff --git a/src/agent/providers/base.py b/src/agent/providers/base.pyindex d0fcd2c..da5b668 100644--- a/src/agent/providers/base.py+++ b/src/agent/providers/base.py@@ -1,10 +1,18 @@"""What every provider returns and offers. A provider never sees the database or the servers; the agent decides-what goes into the messages, and masks them first (masking.py)."""+what goes into the messages, and masks them first (masking.py).++A conversation with tools goes: ``provider.start(...)`` once, then ``step()`` until a turn has no tool calls. A turn+with tool calls is answered with ``add_tool_results()`` before the next step. The final answer is JSON matching the+schema, as with ``complete()``. Each provider keeps the history in its own format and only ever appends to it, so+what the model said (including Claude's thinking) goes back exactly as it came.+"""from __future__ import annotations-from dataclasses import dataclass+from dataclasses import dataclass, fieldfrom typing import Protocol+from ..masking import mask_text+TIMEOUT_SECONDS = 60MAX_RETRIES = 2@@ -17,13 +25,59 @@ class Completion:output_tokens: int = 0+@dataclass+class Tool:+ name: str+ description: str+ schema: dict # JSON schema of the arguments: an object with additionalProperties false and every property required+++@dataclass+class ToolCall:+ id: str+ name: str+ arguments: dict+++@dataclass+class ToolResult:+ call_id: str+ content: str+ is_error: bool = False+++@dataclass+class Turn:+ """One call to the model: tool calls to answer, or (with none) the final answer in ``completion.text``."""+ completion: Completion+ tool_calls: list[ToolCall] = field(default_factory=list)+++class Conversation:+ def step(self, allow_tools: bool = True) -> Turn:+ """One call to the model. ``allow_tools=False`` makes it answer now (the tools stay listed, so the cache+ and Claude's earlier thinking stay valid)."""+ raise NotImplementedError++ def add_tool_results(self, results: list[ToolResult]) -> None:+ """Answers every tool call of the last turn. Results come from the monitored machines, so they are masked+ here, where no tool can skip it."""+ self._append_results([ToolResult(result.call_id, mask_text(result.content), result.is_error) for result in results])++ def _append_results(self, results: list[ToolResult]) -> None:+ raise NotImplementedError++class Provider(Protocol):name: strmodel: str+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> Conversation:+ """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. ``effort`` ("low" | "medium" |+ "high") is how hard the model thinks, where the provider supports it."""+def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:- """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. Returns JSON text matching ``schema``.- ``effort`` ("low" | "medium" | "high") is how hard the model thinks, where the provider supports it."""+ """One answer without tools: JSON text matching ``schema``."""def check(self) -> dict:"""Confirms the key works and the model exists, without spending tokens."""diff --git a/src/agent/providers/claude.py b/src/agent/providers/claude.pyindex e6c09d8..ba43545 100644--- a/src/agent/providers/claude.py+++ b/src/agent/providers/claude.py@@ -1,7 +1,7 @@-"""Claude through the Anthropic API, with structured outputs so a reply always parses."""+"""Claude through the Anthropic API, with structured outputs so a final answer always parses."""from __future__ import annotations-from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion+from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion, Conversation, Tool, ToolCall, ToolResult, Turn# Enough for adaptive thinking plus the longest answer the schemas allow, and small enough for a non-streaming request.MAX_TOKENS = 16000@@ -17,31 +17,55 @@ class ClaudeProvider:client = anthropic.Anthropic(api_key=api_key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)self.client = client+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> ClaudeConversation:+ return ClaudeConversation(self, messages, tools, schema, effort)+def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:- system = "\n\n".join(message["content"] for message in messages if message["role"] == "system")- turns = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]- response = self.client.messages.create(- model=self.model,+ return self.start(messages, [], schema, schema_name, effort).step().completion++ def check(self) -> dict:+ model = self.client.models.retrieve(self.model)+ return {"provider": self.name, "model": getattr(model, "id", self.model)}+++class ClaudeConversation(Conversation):+ def __init__(self, provider: ClaudeProvider, messages: list[dict], tools: list[Tool], schema: dict, effort: str):+ self.provider, self.schema, self.effort = provider, schema, effort+ self.system = "\n\n".join(message["content"] for message in messages if message["role"] == "system")+ self.history = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]+ self.tools = [{"name": tool.name, "description": tool.description, "input_schema": tool.schema, "strict": True} for tool in tools]++ def step(self, allow_tools: bool = True) -> Turn:+ extra = {"tools": self.tools, "tool_choice": {"type": "auto" if allow_tools else "none"}} if self.tools else {}+ response = self.provider.client.messages.create(+ model=self.provider.model,max_tokens=MAX_TOKENS,- system=system,- messages=turns,+ system=self.system,+ messages=list(self.history),thinking={"type": "adaptive"},- output_config={"effort": effort, "format": {"type": "json_schema", "schema": schema}},- cache_control={"type": "ephemeral"}, # the system prompt and earlier turns repeat on every call+ output_config={"effort": self.effort, "format": {"type": "json_schema", "schema": self.schema}},+ cache_control={"type": "ephemeral"}, # the system prompt, tools and earlier turns repeat on every call+ **extra,)if response.stop_reason == "refusal":raise ValueError("The model refused to answer.")if response.stop_reason == "max_tokens":raise ValueError("The model's answer was cut off before it was complete.")+ self.history.append({"role": "assistant", "content": response.content}) # unchanged, thinking blocks included+ calls = [ToolCall(block.id, block.name, block.input) for block in response.content if block.type == "tool_use"]usage = response.usage- return Completion(- text="".join(block.text for block in response.content if block.type == "text"),- model=getattr(response, "model", self.model),+ text = "" if calls else "".join(block.text for block in response.content if block.type == "text")+ return Turn(Completion(+ text=text,+ model=getattr(response, "model", self.provider.model),# Cached input is billed differently, but it is still input: counting it keeps the budget on the safe side.input_tokens=(usage.input_tokens or 0) + (getattr(usage, "cache_creation_input_tokens", 0) or 0) + (getattr(usage, "cache_read_input_tokens", 0) or 0),output_tokens=usage.output_tokens or 0,- )+ ), calls)- def check(self) -> dict:- model = self.client.models.retrieve(self.model)- return {"provider": self.name, "model": getattr(model, "id", self.model)}+ def _append_results(self, results: list[ToolResult]) -> None:+ # All results of one turn go back in a single message.+ self.history.append({"role": "user", "content": [+ {"type": "tool_result", "tool_use_id": result.call_id, "content": result.content, **({"is_error": True} if result.is_error else {})}+ for result in results+ ]})diff --git a/src/agent/providers/openai_compatible.py b/src/agent/providers/openai_compatible.pyindex e46df4c..8ca87ad 100644--- a/src/agent/providers/openai_compatible.py+++ b/src/agent/providers/openai_compatible.py@@ -1,13 +1,15 @@"""OpenAI, and Google Gemini through its OpenAI-compatible endpoint: both take the same Chat Completions request."""from __future__ import annotations-from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion+import json++from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion, Conversation, Tool, ToolCall, ToolResult, TurnGEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/"class OpenAIProvider:- """Chat Completions with Structured Outputs, so a reply always parses."""+ """Chat Completions with Structured Outputs, so a final answer always parses."""def __init__(self, api_key: str, model: str, name: str = "OpenAI", base_url: str | None = None, client=None):self.name, self.model = name, model@@ -16,23 +18,49 @@ class OpenAIProvider:client = OpenAI(api_key=api_key, base_url=base_url, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)self.client = client+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> OpenAIConversation:+ return OpenAIConversation(self, messages, tools, schema, schema_name)+def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:- response = self.client.chat.completions.create(- model=self.model,- messages=messages,- response_format={"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}},- )- choice = response.choices[0]- if getattr(choice.message, "refusal", None):- raise ValueError(f"The model refused to answer: {choice.message.refusal}")- usage = getattr(response, "usage", None)- return Completion(- text=choice.message.content or "",- model=getattr(response, "model", self.model),- input_tokens=getattr(usage, "prompt_tokens", 0) or 0,- output_tokens=getattr(usage, "completion_tokens", 0) or 0,- )+ return self.start(messages, [], schema, schema_name, effort).step().completiondef check(self) -> dict:model = self.client.models.retrieve(self.model)return {"provider": self.name, "model": getattr(model, "id", self.model)}+++class OpenAIConversation(Conversation):+ def __init__(self, provider: OpenAIProvider, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str):+ self.provider = provider+ self.history = [dict(message) for message in messages]+ self.response_format = {"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}}+ self.tools = [{"type": "function", "function": {"name": tool.name, "description": tool.description, "parameters": tool.schema, "strict": True}} for tool in tools]++ def step(self, allow_tools: bool = True) -> Turn:+ extra = {"tools": self.tools, "tool_choice": "auto" if allow_tools else "none"} if self.tools else {}+ response = self.provider.client.chat.completions.create(model=self.provider.model, messages=list(self.history), response_format=self.response_format, **extra)+ message = response.choices[0].message+ if getattr(message, "refusal", None):+ raise ValueError(f"The model refused to answer: {message.refusal}")+ raw_calls = getattr(message, "tool_calls", None) or []+ try:+ calls = [ToolCall(call.id, call.function.name, json.loads(call.function.arguments)) for call in raw_calls]+ except (json.JSONDecodeError, TypeError) as exc:+ raise ValueError("The model sent tool arguments that aren't valid JSON.") from exc+ if any(not isinstance(call.arguments, dict) for call in calls):+ raise ValueError("The model sent tool arguments that aren't a JSON object.")+ turn = {"role": "assistant", "content": message.content}+ if raw_calls:+ turn["tool_calls"] = [{"id": call.id, "type": "function", "function": {"name": call.function.name, "arguments": call.function.arguments}} for call in raw_calls]+ self.history.append(turn)+ usage = getattr(response, "usage", None)+ return Turn(Completion(+ text="" if calls else message.content or "",+ model=getattr(response, "model", self.provider.model),+ input_tokens=getattr(usage, "prompt_tokens", 0) or 0,+ output_tokens=getattr(usage, "completion_tokens", 0) or 0,+ ), calls)++ def _append_results(self, results: list[ToolResult]) -> None:+ for result in results:+ self.history.append({"role": "tool", "tool_call_id": result.call_id, "content": f"Error: {result.content}" if result.is_error else result.content})diff --git a/tests/assistant/test_agent_tool_calls.py b/tests/assistant/test_agent_tool_calls.pynew file mode 100644index 0000000..fb18609--- /dev/null+++ b/tests/assistant/test_agent_tool_calls.py@@ -0,0 +1,194 @@+"""Tool calling through the provider-neutral interface, the same tests for Claude, OpenAI and Gemini with fake clients.+No test reaches a real API."""+import json+from types import SimpleNamespace as NS++import pytest++from src.agent.providers import Tool, ToolResult+from src.agent.providers.claude import ClaudeProvider+from src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider++ANSWER = '{"message":"Done."}'+SCHEMA = {"type": "object", "properties": {"message": {"type": "string"}}, "required": ["message"], "additionalProperties": False}+TOOLS = [Tool("get_host_summary", "The server's CPU, memory and disks right now.", {"type": "object", "properties": {"server_id": {"type": "integer"}}, "required": ["server_id"], "additionalProperties": False})]+MESSAGES = [{"role": "system", "content": "You are the agent."}, {"role": "user", "content": "Why is it slow?"}]+++# --- fake clients: each returns the scripted turns in order; a turn is a list of (call id, arguments) or the answer ---++class FakeClaude:+ def __init__(self, *turns, stop_reason=None):+ self.turns, self.calls, self.stop_reason = list(turns), [], stop_reason++ def create(self, **kwargs):+ self.calls.append(kwargs)+ turn = self.turns.pop(0)+ if isinstance(turn, str):+ content, stop = [NS(type="thinking", thinking=""), NS(type="text", text=turn)], "end_turn"+ else:+ content, stop = [NS(type="thinking", thinking="")] + [NS(type="tool_use", id=call_id, name="get_host_summary", input=args) for call_id, args in turn], "tool_use"+ usage = NS(input_tokens=100, output_tokens=10, cache_creation_input_tokens=0, cache_read_input_tokens=0)+ return NS(content=content, stop_reason=self.stop_reason or stop, model="claude-test", usage=usage)++ def sent(self, index):+ return self.calls[index]["messages"]++ def tool_choice(self, index):+ return self.calls[index]["tool_choice"]["type"]+++class FakeOpenAI:+ def __init__(self, *turns, refusal=None, raw_arguments=None):+ self.turns, self.calls, self.refusal, self.raw_arguments = list(turns), [], refusal, raw_arguments++ def create(self, **kwargs):+ self.calls.append(kwargs)+ turn = self.turns.pop(0)+ if isinstance(turn, str):+ message = NS(content=turn, refusal=self.refusal, tool_calls=None)+ else:+ calls = [NS(id=call_id, function=NS(name="get_host_summary", arguments=self.raw_arguments or json.dumps(args))) for call_id, args in turn]+ message = NS(content=None, refusal=self.refusal, tool_calls=calls)+ return NS(choices=[NS(message=message)], model="gpt-test", usage=NS(prompt_tokens=100, completion_tokens=10))++ def sent(self, index):+ return self.calls[index]["messages"]++ def tool_choice(self, index):+ return self.calls[index]["tool_choice"]+++def claude(fake):+ return ClaudeProvider("key", "claude-test", client=NS(messages=fake))+++def openai(fake):+ return OpenAIProvider("key", "gpt-test", client=NS(chat=NS(completions=fake)))+++def gemini(fake):+ return OpenAIProvider("key", "gpt-test", name="Google Gemini", base_url=GEMINI_BASE_URL, client=NS(chat=NS(completions=fake)))+++PROVIDERS = [pytest.param(claude, FakeClaude, id="claude"), pytest.param(openai, FakeOpenAI, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")]++[email protected]("build, Fake", PROVIDERS)+def test_with_tools_the_model_can_still_answer_at_once(build, Fake):+ fake = Fake(ANSWER)++ turn = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()++ assert turn.tool_calls == [] and turn.completion.text == ANSWER+ assert (turn.completion.input_tokens, turn.completion.output_tokens) == (100, 10)++[email protected]("build, Fake", PROVIDERS)+def test_a_tool_call_is_answered_and_the_history_goes_back_unchanged(build, Fake):+ fake = Fake([("call_1", {"server_id": 1})], ANSWER)+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")++ first = conversation.step()+ assert [(call.id, call.name, call.arguments) for call in first.tool_calls] == [("call_1", "get_host_summary", {"server_id": 1})]+ assert first.completion.text == ""+ conversation.add_tool_results([ToolResult("call_1", "CPU 95 %")])+ second = conversation.step()++ assert second.completion.text == ANSWER+ sent_first, sent_second = fake.sent(0), fake.sent(1)+ assert sent_second[:len(sent_first)] == sent_first # append-only: earlier turns are sent again as they were+ assert "CPU 95 %" in json.dumps(sent_second[-1], default=str)+++def test_claude_gets_its_own_answer_back_untouched_thinking_included():+ fake = FakeClaude([("call_1", {"server_id": 1})], ANSWER)+ conversation = claude(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")+ conversation.step()+ conversation.add_tool_results([ToolResult("call_1", "ok")])+ conversation.step()++ assistant = fake.sent(1)[1]+ assert assistant["role"] == "assistant" and assistant["content"][0].type == "thinking"+ tool = fake.calls[0]["tools"][0]+ assert tool["strict"] is True and tool["input_schema"] == TOOLS[0].schema and fake.calls[0]["system"] == "You are the agent."++[email protected]("build, Fake", PROVIDERS)+def test_several_tool_calls_in_one_turn_get_all_their_results(build, Fake):+ fake = Fake([("call_1", {"server_id": 1}), ("call_2", {"server_id": 2})], ANSWER)+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")++ assert [call.id for call in conversation.step().tool_calls] == ["call_1", "call_2"]+ conversation.add_tool_results([ToolResult("call_1", "one"), ToolResult("call_2", "two")])+ conversation.step()++ sent = fake.sent(1)+ if Fake is FakeClaude: # one user message with both results+ assert [block["tool_use_id"] for block in sent[-1]["content"]] == ["call_1", "call_2"]+ else: # one tool message per result+ assert [message["tool_call_id"] for message in sent[-2:]] == ["call_1", "call_2"]++[email protected]("build, Fake", PROVIDERS)+def test_a_failed_tool_is_reported_as_an_error(build, Fake):+ fake = Fake([("call_1", {"server_id": 9})], ANSWER)+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")+ conversation.step()+ conversation.add_tool_results([ToolResult("call_1", "No such server.", is_error=True)])+ conversation.step()++ result = fake.sent(1)[-1]+ if Fake is FakeClaude:+ assert result["content"][0]["is_error"] is True+ else:+ assert result["content"] == "Error: No such server."++[email protected]("build, Fake", PROVIDERS)+def test_tool_results_are_masked_before_they_leave(build, Fake):+ fake = Fake([("call_1", {"server_id": 1})], ANSWER)+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")+ conversation.step()+ conversation.add_tool_results([ToolResult("call_1", "DB_PASSWORD=hunter2 in the container's environment")])+ conversation.step()++ sent = json.dumps(fake.sent(1)[-1], default=str)+ assert "hunter2" not in sent and "[masked]" in sent++[email protected]("build, Fake", PROVIDERS)+def test_without_tools_allowed_the_model_must_answer_but_the_tools_stay_listed(build, Fake):+ fake = Fake([("call_1", {"server_id": 1})], ANSWER)+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")+ conversation.step()+ conversation.add_tool_results([ToolResult("call_1", "Tool budget used up; give your final answer now.", is_error=True)])+ conversation.step(allow_tools=False)++ assert (fake.tool_choice(0), fake.tool_choice(1)) == ("auto", "none")+ assert fake.calls[1]["tools"] == fake.calls[0]["tools"]+++def test_claude_refusing_midway_is_an_error():+ with pytest.raises(ValueError, match="refused"):+ claude(FakeClaude([("call_1", {})], stop_reason="refusal")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+++def test_openai_refusing_is_an_error():+ with pytest.raises(ValueError, match="refused"):+ openai(FakeOpenAI(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()++[email protected]("raw", ["{not json", "[1, 2]"])+def test_openai_tool_arguments_must_be_a_json_object(raw):+ with pytest.raises(ValueError, match="tool arguments"):+ openai(FakeOpenAI([("call_1", {})], raw_arguments=raw)).start(MESSAGES, TOOLS, SCHEMA, "reply").step()+++def test_openai_tools_are_strict_functions():+ fake = FakeOpenAI(ANSWER)+ openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()++ function = fake.calls[0]["tools"][0]["function"]+ assert fake.calls[0]["tools"][0]["type"] == "function" and function["strict"] is True and function["parameters"] == TOOLS[0].schema+ assert fake.calls[0]["response_format"]["json_schema"]["strict"] is True