Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

feat(agent): provider-neutral tool calling for Claude, OpenAI and Gemini (#186)

Karlo Kajba Šimanić9197e1567526
diff --git a/TESTS.md b/TESTS.md
index 04f7276..8e7a8d0 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,
covers (and what it doesn't), and what every single test is for, so you can decide
whether a failure matters before you upgrade or change your setup.
-- **1458 test cases** from 788 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
+- **1482 test cases** from 799 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.
- **Safe to run next to a live install.** The tests use a throwaway in-memory database
and fakes; they never touch your running containers or your data. (Two collector
@@ -149,6 +149,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.
| [`test_agent_errors.py`](#test_agent_errorspy) | 5 | Helpful, non-leaking error messages from the AI assistant. |
| [`test_agent_service.py`](#test_agent_servicepy) | 8 | The assistant can only run the fixed read-only checks; the OpenAI provider and conversation history. |
| [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. |
+| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 24 | Tool calling with Claude, OpenAI and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer. |
| [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |
| [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. |
| [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |
@@ -342,6 +343,24 @@ shouldn't, or an admin might be locked out.
| `test_a_key_that_cant_be_decrypted_counts_as_no_key` | After `SECRETS_KEY` changes, an unreadable key turns the agent off instead of failing. |
| `test_a_removed_key_turns_the_agent_off` | Removing the key leaves the provider chosen but turns the agent off. |
+### test_agent_tool_calls.py
+
+Most tests run once per provider (Claude, OpenAI, Gemini), with fake clients.
+
+| Test | What it checks |
+|---|---|
+| `test_with_tools_the_model_can_still_answer_at_once` *(3 inputs)* | With tools offered, a model that answers straight away gives the final answer and its token usage. |
+| `test_a_tool_call_is_answered_and_the_history_goes_back_unchanged` *(3 inputs)* | A tool call comes back parsed; after its result, the next request repeats every earlier turn exactly as sent and adds the result. |
+| `test_claude_gets_its_own_answer_back_untouched_thinking_included` | Claude's own turn, thinking included, goes back unchanged; its tools are strict and the system prompt is sent apart. |
+| `test_several_tool_calls_in_one_turn_get_all_their_results` *(3 inputs)* | Several calls in one turn are all answered: in one message for Claude, one tool message each for OpenAI and Gemini. |
+| `test_a_failed_tool_is_reported_as_an_error` *(3 inputs)* | A failed tool is marked as an error for the model. |
+| `test_tool_results_are_masked_before_they_leave` *(3 inputs)* | A password in a tool result is masked before it is sent to the provider. |
+| `test_without_tools_allowed_the_model_must_answer_but_the_tools_stay_listed` *(3 inputs)* | When the budget is used up the model is told to answer (`tool_choice` none), while the tool list stays the same. |
+| `test_claude_refusing_midway_is_an_error` | A refusal from Claude in the middle of an investigation is an error. |
+| `test_openai_refusing_is_an_error` | A refusal from OpenAI with tools offered is an error. |
+| `test_openai_tool_arguments_must_be_a_json_object` *(2 inputs)* | Tool arguments that aren't valid JSON, or aren't an object, are refused. |
+| `test_openai_tools_are_strict_functions` | OpenAI and Gemini get strict function tools and a strict answer schema. |
+
### test_agent_diagnosis.py
| Test | What it checks |
diff --git a/docs/development.md b/docs/development.md
index 4740d29..38af3f8 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -104,7 +104,7 @@ src/
secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key
reset_2fa.py command-line two-factor reset for a locked-out account
agent/ the incident agent; runs in the backend (chat) and the agent-worker container
- providers/ the language model behind one interface: Claude, OpenAI, Gemini; the choice and its encrypted key
+ providers/ the language model behind one interface (answers and tool calls): Claude, OpenAI, Gemini; the choice and its encrypted key
chat.py, prompts.py, SYSTEM_PROMPT.md the incident chat and its instructions
health_checks.py the fixed, read-only checks the agent may ask for
jobs.py job queue in Redis: published after an incident opens or worsens
diff --git a/src/agent/providers/__init__.py b/src/agent/providers/__init__.py
index 24fab9b..159c386 100644
--- a/src/agent/providers/__init__.py
+++ b/src/agent/providers/__init__.py
@@ -12,11 +12,11 @@ from dataclasses import dataclass
from src.backend import secrets_box
from src.backend.models import AppSetting
-from .base import Completion, Provider
+from .base import Completion, Conversation, Provider, Tool, ToolCall, ToolResult, Turn
from .claude import ClaudeProvider
from .openai_compatible import GEMINI_BASE_URL, OpenAIProvider
-__all__ = ["Completion", "Provider", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]
+__all__ = ["Completion", "Conversation", "Provider", "Tool", "ToolCall", "ToolResult", "Turn", "PROVIDERS", "NoApiKey", "current", "get_provider", "key_configured", "save", "remove_key"]
logger = logging.getLogger(__name__)
diff --git a/src/agent/providers/base.py b/src/agent/providers/base.py
index d0fcd2c..da5b668 100644
--- a/src/agent/providers/base.py
+++ b/src/agent/providers/base.py
@@ -1,10 +1,18 @@
"""What every provider returns and offers. A provider never sees the database or the servers; the agent decides
-what goes into the messages, and masks them first (masking.py)."""
+what goes into the messages, and masks them first (masking.py).
+
+A conversation with tools goes: ``provider.start(...)`` once, then ``step()`` until a turn has no tool calls. A turn
+with tool calls is answered with ``add_tool_results()`` before the next step. The final answer is JSON matching the
+schema, as with ``complete()``. Each provider keeps the history in its own format and only ever appends to it, so
+what the model said (including Claude's thinking) goes back exactly as it came.
+"""
from __future__ import annotations
-from dataclasses import dataclass
+from dataclasses import dataclass, field
from typing import Protocol
+from ..masking import mask_text
+
TIMEOUT_SECONDS = 60
MAX_RETRIES = 2
@@ -17,13 +25,59 @@ class Completion:
output_tokens: int = 0
+@dataclass
+class Tool:
+ name: str
+ description: str
+ schema: dict # JSON schema of the arguments: an object with additionalProperties false and every property required
+
+
+@dataclass
+class ToolCall:
+ id: str
+ name: str
+ arguments: dict
+
+
+@dataclass
+class ToolResult:
+ call_id: str
+ content: str
+ is_error: bool = False
+
+
+@dataclass
+class Turn:
+ """One call to the model: tool calls to answer, or (with none) the final answer in ``completion.text``."""
+ completion: Completion
+ tool_calls: list[ToolCall] = field(default_factory=list)
+
+
+class Conversation:
+ def step(self, allow_tools: bool = True) -> Turn:
+ """One call to the model. ``allow_tools=False`` makes it answer now (the tools stay listed, so the cache
+ and Claude's earlier thinking stay valid)."""
+ raise NotImplementedError
+
+ def add_tool_results(self, results: list[ToolResult]) -> None:
+ """Answers every tool call of the last turn. Results come from the monitored machines, so they are masked
+ here, where no tool can skip it."""
+ self._append_results([ToolResult(result.call_id, mask_text(result.content), result.is_error) for result in results])
+
+ def _append_results(self, results: list[ToolResult]) -> None:
+ raise NotImplementedError
+
+
class Provider(Protocol):
name: str
model: str
+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> Conversation:
+ """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. ``effort`` ("low" | "medium" |
+ "high") is how hard the model thinks, where the provider supports it."""
+
def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
- """``messages``: [{"role": "system" | "user" | "assistant", "content": str}]. Returns JSON text matching ``schema``.
- ``effort`` ("low" | "medium" | "high") is how hard the model thinks, where the provider supports it."""
+ """One answer without tools: JSON text matching ``schema``."""
def check(self) -> dict:
"""Confirms the key works and the model exists, without spending tokens."""
diff --git a/src/agent/providers/claude.py b/src/agent/providers/claude.py
index e6c09d8..ba43545 100644
--- a/src/agent/providers/claude.py
+++ b/src/agent/providers/claude.py
@@ -1,7 +1,7 @@
-"""Claude through the Anthropic API, with structured outputs so a reply always parses."""
+"""Claude through the Anthropic API, with structured outputs so a final answer always parses."""
from __future__ import annotations
-from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion
+from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion, Conversation, Tool, ToolCall, ToolResult, Turn
# Enough for adaptive thinking plus the longest answer the schemas allow, and small enough for a non-streaming request.
MAX_TOKENS = 16000
@@ -17,31 +17,55 @@ class ClaudeProvider:
client = anthropic.Anthropic(api_key=api_key, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
self.client = client
+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> ClaudeConversation:
+ return ClaudeConversation(self, messages, tools, schema, effort)
+
def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
- system = "\n\n".join(message["content"] for message in messages if message["role"] == "system")
- turns = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]
- response = self.client.messages.create(
- model=self.model,
+ return self.start(messages, [], schema, schema_name, effort).step().completion
+
+ def check(self) -> dict:
+ model = self.client.models.retrieve(self.model)
+ return {"provider": self.name, "model": getattr(model, "id", self.model)}
+
+
+class ClaudeConversation(Conversation):
+ def __init__(self, provider: ClaudeProvider, messages: list[dict], tools: list[Tool], schema: dict, effort: str):
+ self.provider, self.schema, self.effort = provider, schema, effort
+ self.system = "\n\n".join(message["content"] for message in messages if message["role"] == "system")
+ self.history = [{"role": message["role"], "content": message["content"]} for message in messages if message["role"] != "system"]
+ self.tools = [{"name": tool.name, "description": tool.description, "input_schema": tool.schema, "strict": True} for tool in tools]
+
+ def step(self, allow_tools: bool = True) -> Turn:
+ extra = {"tools": self.tools, "tool_choice": {"type": "auto" if allow_tools else "none"}} if self.tools else {}
+ response = self.provider.client.messages.create(
+ model=self.provider.model,
max_tokens=MAX_TOKENS,
- system=system,
- messages=turns,
+ system=self.system,
+ messages=list(self.history),
thinking={"type": "adaptive"},
- output_config={"effort": effort, "format": {"type": "json_schema", "schema": schema}},
- cache_control={"type": "ephemeral"}, # the system prompt and earlier turns repeat on every call
+ output_config={"effort": self.effort, "format": {"type": "json_schema", "schema": self.schema}},
+ cache_control={"type": "ephemeral"}, # the system prompt, tools and earlier turns repeat on every call
+ **extra,
)
if response.stop_reason == "refusal":
raise ValueError("The model refused to answer.")
if response.stop_reason == "max_tokens":
raise ValueError("The model's answer was cut off before it was complete.")
+ self.history.append({"role": "assistant", "content": response.content}) # unchanged, thinking blocks included
+ calls = [ToolCall(block.id, block.name, block.input) for block in response.content if block.type == "tool_use"]
usage = response.usage
- return Completion(
- text="".join(block.text for block in response.content if block.type == "text"),
- model=getattr(response, "model", self.model),
+ text = "" if calls else "".join(block.text for block in response.content if block.type == "text")
+ return Turn(Completion(
+ text=text,
+ model=getattr(response, "model", self.provider.model),
# Cached input is billed differently, but it is still input: counting it keeps the budget on the safe side.
input_tokens=(usage.input_tokens or 0) + (getattr(usage, "cache_creation_input_tokens", 0) or 0) + (getattr(usage, "cache_read_input_tokens", 0) or 0),
output_tokens=usage.output_tokens or 0,
- )
+ ), calls)
- def check(self) -> dict:
- model = self.client.models.retrieve(self.model)
- return {"provider": self.name, "model": getattr(model, "id", self.model)}
+ def _append_results(self, results: list[ToolResult]) -> None:
+ # All results of one turn go back in a single message.
+ self.history.append({"role": "user", "content": [
+ {"type": "tool_result", "tool_use_id": result.call_id, "content": result.content, **({"is_error": True} if result.is_error else {})}
+ for result in results
+ ]})
diff --git a/src/agent/providers/openai_compatible.py b/src/agent/providers/openai_compatible.py
index e46df4c..8ca87ad 100644
--- a/src/agent/providers/openai_compatible.py
+++ b/src/agent/providers/openai_compatible.py
@@ -1,13 +1,15 @@
"""OpenAI, and Google Gemini through its OpenAI-compatible endpoint: both take the same Chat Completions request."""
from __future__ import annotations
-from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion
+import json
+
+from .base import MAX_RETRIES, TIMEOUT_SECONDS, Completion, Conversation, Tool, ToolCall, ToolResult, Turn
GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/"
class OpenAIProvider:
- """Chat Completions with Structured Outputs, so a reply always parses."""
+ """Chat Completions with Structured Outputs, so a final answer always parses."""
def __init__(self, api_key: str, model: str, name: str = "OpenAI", base_url: str | None = None, client=None):
self.name, self.model = name, model
@@ -16,23 +18,49 @@ class OpenAIProvider:
client = OpenAI(api_key=api_key, base_url=base_url, timeout=TIMEOUT_SECONDS, max_retries=MAX_RETRIES)
self.client = client
+ def start(self, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str, effort: str = "medium") -> OpenAIConversation:
+ return OpenAIConversation(self, messages, tools, schema, schema_name)
+
def complete(self, messages: list[dict], schema: dict, schema_name: str, effort: str = "medium") -> Completion:
- response = self.client.chat.completions.create(
- model=self.model,
- messages=messages,
- response_format={"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}},
- )
- choice = response.choices[0]
- if getattr(choice.message, "refusal", None):
- raise ValueError(f"The model refused to answer: {choice.message.refusal}")
- usage = getattr(response, "usage", None)
- return Completion(
- text=choice.message.content or "",
- model=getattr(response, "model", self.model),
- input_tokens=getattr(usage, "prompt_tokens", 0) or 0,
- output_tokens=getattr(usage, "completion_tokens", 0) or 0,
- )
+ return self.start(messages, [], schema, schema_name, effort).step().completion
def check(self) -> dict:
model = self.client.models.retrieve(self.model)
return {"provider": self.name, "model": getattr(model, "id", self.model)}
+
+
+class OpenAIConversation(Conversation):
+ def __init__(self, provider: OpenAIProvider, messages: list[dict], tools: list[Tool], schema: dict, schema_name: str):
+ self.provider = provider
+ self.history = [dict(message) for message in messages]
+ self.response_format = {"type": "json_schema", "json_schema": {"name": schema_name, "strict": True, "schema": schema}}
+ self.tools = [{"type": "function", "function": {"name": tool.name, "description": tool.description, "parameters": tool.schema, "strict": True}} for tool in tools]
+
+ def step(self, allow_tools: bool = True) -> Turn:
+ extra = {"tools": self.tools, "tool_choice": "auto" if allow_tools else "none"} if self.tools else {}
+ response = self.provider.client.chat.completions.create(model=self.provider.model, messages=list(self.history), response_format=self.response_format, **extra)
+ message = response.choices[0].message
+ if getattr(message, "refusal", None):
+ raise ValueError(f"The model refused to answer: {message.refusal}")
+ raw_calls = getattr(message, "tool_calls", None) or []
+ try:
+ calls = [ToolCall(call.id, call.function.name, json.loads(call.function.arguments)) for call in raw_calls]
+ except (json.JSONDecodeError, TypeError) as exc:
+ raise ValueError("The model sent tool arguments that aren't valid JSON.") from exc
+ if any(not isinstance(call.arguments, dict) for call in calls):
+ raise ValueError("The model sent tool arguments that aren't a JSON object.")
+ turn = {"role": "assistant", "content": message.content}
+ if raw_calls:
+ turn["tool_calls"] = [{"id": call.id, "type": "function", "function": {"name": call.function.name, "arguments": call.function.arguments}} for call in raw_calls]
+ self.history.append(turn)
+ usage = getattr(response, "usage", None)
+ return Turn(Completion(
+ text="" if calls else message.content or "",
+ model=getattr(response, "model", self.provider.model),
+ input_tokens=getattr(usage, "prompt_tokens", 0) or 0,
+ output_tokens=getattr(usage, "completion_tokens", 0) or 0,
+ ), calls)
+
+ def _append_results(self, results: list[ToolResult]) -> None:
+ for result in results:
+ self.history.append({"role": "tool", "tool_call_id": result.call_id, "content": f"Error: {result.content}" if result.is_error else result.content})
diff --git a/tests/assistant/test_agent_tool_calls.py b/tests/assistant/test_agent_tool_calls.py
new file mode 100644
index 0000000..fb18609
--- /dev/null
+++ b/tests/assistant/test_agent_tool_calls.py
@@ -0,0 +1,194 @@
+"""Tool calling through the provider-neutral interface, the same tests for Claude, OpenAI and Gemini with fake clients.
+No test reaches a real API."""
+import json
+from types import SimpleNamespace as NS
+
+import pytest
+
+from src.agent.providers import Tool, ToolResult
+from src.agent.providers.claude import ClaudeProvider
+from src.agent.providers.openai_compatible import GEMINI_BASE_URL, OpenAIProvider
+
+ANSWER = '{"message":"Done."}'
+SCHEMA = {"type": "object", "properties": {"message": {"type": "string"}}, "required": ["message"], "additionalProperties": False}
+TOOLS = [Tool("get_host_summary", "The server's CPU, memory and disks right now.", {"type": "object", "properties": {"server_id": {"type": "integer"}}, "required": ["server_id"], "additionalProperties": False})]
+MESSAGES = [{"role": "system", "content": "You are the agent."}, {"role": "user", "content": "Why is it slow?"}]
+
+
+# --- fake clients: each returns the scripted turns in order; a turn is a list of (call id, arguments) or the answer ---
+
+class FakeClaude:
+ def __init__(self, *turns, stop_reason=None):
+ self.turns, self.calls, self.stop_reason = list(turns), [], stop_reason
+
+ def create(self, **kwargs):
+ self.calls.append(kwargs)
+ turn = self.turns.pop(0)
+ if isinstance(turn, str):
+ content, stop = [NS(type="thinking", thinking=""), NS(type="text", text=turn)], "end_turn"
+ else:
+ content, stop = [NS(type="thinking", thinking="")] + [NS(type="tool_use", id=call_id, name="get_host_summary", input=args) for call_id, args in turn], "tool_use"
+ usage = NS(input_tokens=100, output_tokens=10, cache_creation_input_tokens=0, cache_read_input_tokens=0)
+ return NS(content=content, stop_reason=self.stop_reason or stop, model="claude-test", usage=usage)
+
+ def sent(self, index):
+ return self.calls[index]["messages"]
+
+ def tool_choice(self, index):
+ return self.calls[index]["tool_choice"]["type"]
+
+
+class FakeOpenAI:
+ def __init__(self, *turns, refusal=None, raw_arguments=None):
+ self.turns, self.calls, self.refusal, self.raw_arguments = list(turns), [], refusal, raw_arguments
+
+ def create(self, **kwargs):
+ self.calls.append(kwargs)
+ turn = self.turns.pop(0)
+ if isinstance(turn, str):
+ message = NS(content=turn, refusal=self.refusal, tool_calls=None)
+ else:
+ calls = [NS(id=call_id, function=NS(name="get_host_summary", arguments=self.raw_arguments or json.dumps(args))) for call_id, args in turn]
+ message = NS(content=None, refusal=self.refusal, tool_calls=calls)
+ return NS(choices=[NS(message=message)], model="gpt-test", usage=NS(prompt_tokens=100, completion_tokens=10))
+
+ def sent(self, index):
+ return self.calls[index]["messages"]
+
+ def tool_choice(self, index):
+ return self.calls[index]["tool_choice"]
+
+
+def claude(fake):
+ return ClaudeProvider("key", "claude-test", client=NS(messages=fake))
+
+
+def openai(fake):
+ return OpenAIProvider("key", "gpt-test", client=NS(chat=NS(completions=fake)))
+
+
+def gemini(fake):
+ return OpenAIProvider("key", "gpt-test", name="Google Gemini", base_url=GEMINI_BASE_URL, client=NS(chat=NS(completions=fake)))
+
+
+PROVIDERS = [pytest.param(claude, FakeClaude, id="claude"), pytest.param(openai, FakeOpenAI, id="openai"), pytest.param(gemini, FakeOpenAI, id="gemini")]
+
+
[email protected]("build, Fake", PROVIDERS)
+def test_with_tools_the_model_can_still_answer_at_once(build, Fake):
+ fake = Fake(ANSWER)
+
+ turn = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+ assert turn.tool_calls == [] and turn.completion.text == ANSWER
+ assert (turn.completion.input_tokens, turn.completion.output_tokens) == (100, 10)
+
+
[email protected]("build, Fake", PROVIDERS)
+def test_a_tool_call_is_answered_and_the_history_goes_back_unchanged(build, Fake):
+ fake = Fake([("call_1", {"server_id": 1})], ANSWER)
+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+
+ first = conversation.step()
+ assert [(call.id, call.name, call.arguments) for call in first.tool_calls] == [("call_1", "get_host_summary", {"server_id": 1})]
+ assert first.completion.text == ""
+ conversation.add_tool_results([ToolResult("call_1", "CPU 95 %")])
+ second = conversation.step()
+
+ assert second.completion.text == ANSWER
+ sent_first, sent_second = fake.sent(0), fake.sent(1)
+ assert sent_second[:len(sent_first)] == sent_first # append-only: earlier turns are sent again as they were
+ assert "CPU 95 %" in json.dumps(sent_second[-1], default=str)
+
+
+def test_claude_gets_its_own_answer_back_untouched_thinking_included():
+ fake = FakeClaude([("call_1", {"server_id": 1})], ANSWER)
+ conversation = claude(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+ conversation.step()
+ conversation.add_tool_results([ToolResult("call_1", "ok")])
+ conversation.step()
+
+ assistant = fake.sent(1)[1]
+ assert assistant["role"] == "assistant" and assistant["content"][0].type == "thinking"
+ tool = fake.calls[0]["tools"][0]
+ assert tool["strict"] is True and tool["input_schema"] == TOOLS[0].schema and fake.calls[0]["system"] == "You are the agent."
+
+
[email protected]("build, Fake", PROVIDERS)
+def test_several_tool_calls_in_one_turn_get_all_their_results(build, Fake):
+ fake = Fake([("call_1", {"server_id": 1}), ("call_2", {"server_id": 2})], ANSWER)
+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+
+ assert [call.id for call in conversation.step().tool_calls] == ["call_1", "call_2"]
+ conversation.add_tool_results([ToolResult("call_1", "one"), ToolResult("call_2", "two")])
+ conversation.step()
+
+ sent = fake.sent(1)
+ if Fake is FakeClaude: # one user message with both results
+ assert [block["tool_use_id"] for block in sent[-1]["content"]] == ["call_1", "call_2"]
+ else: # one tool message per result
+ assert [message["tool_call_id"] for message in sent[-2:]] == ["call_1", "call_2"]
+
+
[email protected]("build, Fake", PROVIDERS)
+def test_a_failed_tool_is_reported_as_an_error(build, Fake):
+ fake = Fake([("call_1", {"server_id": 9})], ANSWER)
+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+ conversation.step()
+ conversation.add_tool_results([ToolResult("call_1", "No such server.", is_error=True)])
+ conversation.step()
+
+ result = fake.sent(1)[-1]
+ if Fake is FakeClaude:
+ assert result["content"][0]["is_error"] is True
+ else:
+ assert result["content"] == "Error: No such server."
+
+
[email protected]("build, Fake", PROVIDERS)
+def test_tool_results_are_masked_before_they_leave(build, Fake):
+ fake = Fake([("call_1", {"server_id": 1})], ANSWER)
+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+ conversation.step()
+ conversation.add_tool_results([ToolResult("call_1", "DB_PASSWORD=hunter2 in the container's environment")])
+ conversation.step()
+
+ sent = json.dumps(fake.sent(1)[-1], default=str)
+ assert "hunter2" not in sent and "[masked]" in sent
+
+
[email protected]("build, Fake", PROVIDERS)
+def test_without_tools_allowed_the_model_must_answer_but_the_tools_stay_listed(build, Fake):
+ fake = Fake([("call_1", {"server_id": 1})], ANSWER)
+ conversation = build(fake).start(MESSAGES, TOOLS, SCHEMA, "reply")
+ conversation.step()
+ conversation.add_tool_results([ToolResult("call_1", "Tool budget used up; give your final answer now.", is_error=True)])
+ conversation.step(allow_tools=False)
+
+ assert (fake.tool_choice(0), fake.tool_choice(1)) == ("auto", "none")
+ assert fake.calls[1]["tools"] == fake.calls[0]["tools"]
+
+
+def test_claude_refusing_midway_is_an_error():
+ with pytest.raises(ValueError, match="refused"):
+ claude(FakeClaude([("call_1", {})], stop_reason="refusal")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+
+def test_openai_refusing_is_an_error():
+ with pytest.raises(ValueError, match="refused"):
+ openai(FakeOpenAI(ANSWER, refusal="I can't help with that.")).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+
[email protected]("raw", ["{not json", "[1, 2]"])
+def test_openai_tool_arguments_must_be_a_json_object(raw):
+ with pytest.raises(ValueError, match="tool arguments"):
+ openai(FakeOpenAI([("call_1", {})], raw_arguments=raw)).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+
+def test_openai_tools_are_strict_functions():
+ fake = FakeOpenAI(ANSWER)
+ openai(fake).start(MESSAGES, TOOLS, SCHEMA, "reply").step()
+
+ function = fake.calls[0]["tools"][0]["function"]
+ assert fake.calls[0]["tools"][0]["type"] == "function" and function["strict"] is True and function["parameters"] == TOOLS[0].schema
+ assert fake.calls[0]["response_format"]["json_schema"]["strict"] is True