Skip to content

Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

Python

Preserve images across protocol translations (#1602)

Ali Khokhar4ef32fdb1e83
diff --git a/.env.example b/.env.example
index f7efe77b..ac7562bf 100644
--- a/.env.example
+++ b/.env.example
@@ -210,6 +210,7 @@ MODEL="nvidia_nim/nvidia/nemotron-3-super-120b-a12b"
# Optional live smoke model overrides. Provider smoke runs once per configured
# provider even when MODEL/MODEL_* route to a different provider.
# FCC_SMOKE_MODEL_NVIDIA_NIM=<provider/model-id>
+# FCC_SMOKE_MODEL_NVIDIA_NIM_VISION=<provider/model-id>
# FCC_SMOKE_MODEL_OPENAI=<provider/model-id>
# FCC_SMOKE_MODEL_AZURE_OPENAI=<provider/model-id>
# FCC_SMOKE_MODEL_OPEN_ROUTER=<provider/model-id>
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index bd595ae8..3b099b3a 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -1117,12 +1117,18 @@ is then coalesced into one turn so strict chat templates receive neither
and rejects unrepresentable blocks instead of dropping them. Provider policies
do not reinterpret this role mapping.
-User image conversion is a pure protocol operation. Core maps Anthropic base64
-and URL image sources to ordered OpenAI `image_url` content parts without
-fetching remote content. Provider adapters do not gate that conversion behind a
-provider-wide vision flag; the selected upstream model owns image capability,
-while any deliberate provider-specific attachment removal remains explicit
-compatibility policy.
+Image conversion is a pure protocol operation. Core validates and normalizes
+supported Anthropic base64/data-URL and remote URL image sources without
+fetching remote content, then maps them to ordered OpenAI image parts. The same
+rules apply to direct user images and images nested in `tool_result` content.
+Because a Chat Completions `tool` message cannot carry multimodal content, the
+Chat translation closes every parallel tool call first and moves each rich tool
+result, with an explicit call-ID label, into the immediately following ordered
+multimodal user content. Text-only tool results keep the ordinary tool-message
+shape. Provider adapters do not gate image conversion behind a provider-wide
+vision flag; the selected upstream model owns image capability, while any
+deliberate provider-specific attachment removal remains explicit compatibility
+policy.
Shared stream behavior lives under
[src/free_claude_code/core/anthropic/streaming/](src/free_claude_code/core/anthropic/streaming/). The shared layer owns the
@@ -1221,15 +1227,21 @@ Messages-to-Chat, Messages-to-Responses, and Responses-to-Chat. Native
Responses-to-Responses is an identity relay, not a fourth translation. FCC does
not maintain a universal canonical inference schema between these boundaries.
-The direct Responses-to-Chat cell preserves text, images, function/custom tool
-identity, call IDs, reasoning replay data, supported request options, and usage,
-then writes one coherent Responses lifecycle from Chat stream semantics. The
-Messages-to-Responses cell preserves the corresponding Anthropic semantics and
-writes Anthropic SSE from Responses events. Native Responses forwarding keeps
-nested request extensions and upstream response/item/call IDs, event order,
-reasoning payloads, and exact usage; it rewrites only FCC-owned routing fields
-such as the public model and enforces stateless streaming. Unsupported
-cross-protocol semantics fail in the owning translator before upstream I/O.
+The direct Responses-to-Chat cell preserves text, direct images,
+function/custom tool identity, call IDs, reasoning replay data, supported
+request options, and usage, then writes one coherent Responses lifecycle from
+Chat stream semantics. Image-bearing `function_call_output` content and
+`computer_call_output` screenshots use the same Chat tool-round rule: required
+tool messages are emitted first, followed by labeled multimodal user content in
+source order. The Messages-to-Responses cell preserves direct and nested
+tool-result images while writing Anthropic SSE from Responses events. Native
+Responses forwarding keeps nested request extensions—including file-ID image
+references—and upstream response/item/call IDs, event order, reasoning
+payloads, and exact usage; it rewrites only FCC-owned routing fields such as the
+public model and enforces stateless streaming. Cross-protocol image references
+that only contain an upstream file ID cannot be represented without fetching
+or storage, so the owning translator rejects them before upstream I/O instead
+of silently dropping the image.
Application-resolved reasoning controls remain on the immutable source request.
Provider adapters consume the resolved `ReasoningPolicy`; each cross-protocol
diff --git a/pyproject.toml b/pyproject.toml
index c7a1fcfd..675b0735 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "free-claude-code"
-version = "5.15.9"
+version = "5.15.10"
description = "Local proxy connecting coding agents to OpenAI-compatible AI providers"
readme = "README.md"
requires-python = ">=3.14.0"
diff --git a/smoke/README.md b/smoke/README.md
index 9639c914..c232878f 100644
--- a/smoke/README.md
+++ b/smoke/README.md
@@ -72,6 +72,7 @@ Heavy/side-effectful targets are opt-in:
| Target | Product scenarios | Required environment |
| --- | --- | --- |
| `nvidia_nim_cli` | Claude Code CLI feature matrix across NIM models | `NVIDIA_NIM_API_KEY`, Claude CLI |
+| `nvidia_nim_vision` | Claude-style image tool result reaches a NIM vision model as pixels | `NVIDIA_NIM_API_KEY`, `FCC_SMOKE_MODEL_NVIDIA_NIM_VISION` |
| `openrouter_free_cli` | Claude Code CLI feature matrix across OpenRouter free models | `OPENROUTER_API_KEY`, Claude CLI |
| `telegram` | getMe, send, edit, delete, optional manual inbound | token and chat/user ID |
| `discord` | channel access, send, edit, delete, optional manual inbound | token and channel ID |
@@ -127,6 +128,24 @@ $env:FCC_SMOKE_TARGETS = "messaging,config,extensibility"
uv run pytest smoke/product -n 0 -s --tb=short
```
+NVIDIA NIM vision regression (PowerShell):
+
+```powershell
+$env:FCC_LIVE_SMOKE = "1"
+$env:FCC_SMOKE_TARGETS = "nvidia_nim_vision"
+$env:FCC_SMOKE_MODEL_NVIDIA_NIM_VISION = "meta/llama-3.2-11b-vision-instruct"
+uv run pytest smoke/product/test_nvidia_nim_vision_product_live.py -n 0 -s --tb=short
+```
+
+NVIDIA NIM vision regression (POSIX):
+
+```bash
+FCC_LIVE_SMOKE=1 \
+FCC_SMOKE_TARGETS=nvidia_nim_vision \
+FCC_SMOKE_MODEL_NVIDIA_NIM_VISION=meta/llama-3.2-11b-vision-instruct \
+uv run pytest smoke/product/test_nvidia_nim_vision_product_live.py -n 0 -s --tb=short
+```
+
## Environment
- Runtime settings use the isolated managed `~/.fcc/.env`; `FCC_ENV_FILE` is
@@ -139,6 +158,8 @@ uv run pytest smoke/product -n 0 -s --tb=short
Use the uppercase provider ID, such as `FCC_SMOKE_MODEL_KILO`; the complete
variable inventory is in [.env.example](../.env.example). Values may include
the provider prefix or just the model name for that provider.
+- `FCC_SMOKE_MODEL_NVIDIA_NIM_VISION`: required explicit NIM vision model for
+ the opt-in `nvidia_nim_vision` target; it never falls back to the text model.
- `FCC_SMOKE_MODEL_MISTRAL_REASONING`: optional override for the dedicated
Mistral native reasoning smoke, default `mistral/mistral-medium-3-5`.
- `FCC_SMOKE_NIM_MODELS`: optional comma-separated NVIDIA NIM CLI matrix models
diff --git a/smoke/assets/vision-fcc.png b/smoke/assets/vision-fcc.png
new file mode 100644
index 00000000..9af2f052
Binary files /dev/null and b/smoke/assets/vision-fcc.png differ
diff --git a/smoke/capabilities.py b/smoke/capabilities.py
index 90983004..93c712d2 100644
--- a/smoke/capabilities.py
+++ b/smoke/capabilities.py
@@ -228,6 +228,24 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"test_provider_reasoning_tool_continuation_e2e",
),
),
+ CapabilityContract(
+ "streaming_conversion",
+ "vision_content",
+ "vision_protocol_matrix",
+ "free_claude_code.core.anthropic and free_claude_code.core.openai_responses",
+ "direct images and image-bearing tool outputs",
+ "native visual parts in each supported protocol cell",
+ "invalid or unresolvable translated images fail before upstream I/O",
+ (
+ "tests/contracts/test_protocol_matrix.py",
+ "tests/core/anthropic/test_image_sources.py",
+ "tests/core/openai_responses/test_chat_request.py",
+ ),
+ (
+ "test_nvidia_nim_vision_tool_result_e2e",
+ "test_nvidia_nim_vision_function_output_e2e",
+ ),
+ ),
CapabilityContract(
"streaming_conversion",
"heuristic_tools",
diff --git a/smoke/features.py b/smoke/features.py
index 7f423656..5b9a1e6c 100644
--- a/smoke/features.py
+++ b/smoke/features.py
@@ -221,6 +221,23 @@ FEATURE_INVENTORY: tuple[FeatureCoverage, ...] = (
("configured provider credentials/endpoints", "optional FCC_SMOKE_MODEL_*"),
"selected providers missing credentials are failing missing_env",
),
+ FeatureCoverage(
+ "vision_protocol_matrix",
+ "Direct and tool-result images remain visual across protocol translation",
+ (
+ "tests/contracts/test_protocol_matrix.py",
+ "tests/core/anthropic/test_image_sources.py",
+ "tests/core/openai_responses/test_chat_request.py",
+ ),
+ (),
+ (
+ "test_nvidia_nim_vision_tool_result_e2e",
+ "test_nvidia_nim_vision_function_output_e2e",
+ ),
+ ("nvidia_nim_vision",),
+ ("NVIDIA_NIM_API_KEY", "FCC_SMOKE_MODEL_NVIDIA_NIM_VISION"),
+ "skip unless the dedicated vision target and explicit model are configured",
+ ),
FeatureCoverage(
"per_model_mapping",
"Fable, Opus, Sonnet, Haiku, and fallback mappings route explicitly",
diff --git a/smoke/lib/config.py b/smoke/lib/config.py
index e0fd043f..36960fcc 100644
--- a/smoke/lib/config.py
+++ b/smoke/lib/config.py
@@ -33,7 +33,9 @@ DEFAULT_TARGETS = frozenset(
}
)
SIDE_EFFECT_TARGETS = frozenset({"discord", "telegram", "voice"})
-OPT_IN_TARGETS = frozenset({"nvidia_nim_cli", "openrouter_free_cli"})
+OPT_IN_TARGETS = frozenset(
+ {"nvidia_nim_cli", "nvidia_nim_vision", "openrouter_free_cli"}
+)
ALL_TARGETS = DEFAULT_TARGETS | SIDE_EFFECT_TARGETS | OPT_IN_TARGETS
TARGET_ALIASES = {
"contract": "api",
@@ -132,6 +134,10 @@ TARGET_REQUIRED_ENV: dict[str, tuple[str, ...]] = {
"NVIDIA_NIM_API_KEY",
"FCC_SMOKE_CLAUDE_BIN or claude on PATH",
),
+ "nvidia_nim_vision": (
+ "NVIDIA_NIM_API_KEY",
+ "FCC_SMOKE_MODEL_NVIDIA_NIM_VISION",
+ ),
"openrouter_free_cli": (
"OPENROUTER_API_KEY",
"FCC_SMOKE_CLAUDE_BIN or claude on PATH",
@@ -243,6 +249,18 @@ class SmokeConfig:
for full_model, source in nvidia_nim_cli_model_refs().items()
]
+ def nvidia_nim_vision_model(self) -> ProviderModel | None:
+ """Return only an explicitly selected NVIDIA NIM vision model."""
+ override_env = "FCC_SMOKE_MODEL_NVIDIA_NIM_VISION"
+ override = os.getenv(override_env)
+ if override is None:
+ return None
+ return ProviderModel(
+ provider="nvidia_nim",
+ full_model=_normalize_provider_model("nvidia_nim", override),
+ source=override_env,
+ )
+
def openrouter_free_cli_models(self) -> list[ProviderModel]:
"""Return OpenRouter free models for Claude Code CLI characterization."""
return [
diff --git a/smoke/product/test_nvidia_nim_vision_product_live.py b/smoke/product/test_nvidia_nim_vision_product_live.py
new file mode 100644
index 00000000..59b81f87
--- /dev/null
+++ b/smoke/product/test_nvidia_nim_vision_product_live.py
@@ -0,0 +1,154 @@
+import base64
+from pathlib import Path
+
+import httpx
+import pytest
+
+from free_claude_code.core.anthropic.stream_contracts import parse_sse_lines
+from smoke.lib.config import SmokeConfig, auth_headers
+from smoke.lib.e2e import ConversationDriver, SmokeServerDriver, assert_product_stream
+
+pytestmark = [pytest.mark.live, pytest.mark.smoke_target("nvidia_nim_vision")]
+
+_FIXTURE = Path(__file__).resolve().parents[1] / "assets" / "vision-fcc.png"
+_VISIBLE_TOKEN = "K7P4"
+
+
+def test_nvidia_nim_vision_tool_result_e2e(smoke_config: SmokeConfig) -> None:
+ """Keep a Claude-style image tool result visual through Chat egress."""
+ if not smoke_config.has_provider_configuration("nvidia_nim"):
+ pytest.skip("missing_env: NVIDIA_NIM_API_KEY is not configured")
+ provider_model = smoke_config.nvidia_nim_vision_model()
+ if provider_model is None:
+ pytest.skip("missing_env: FCC_SMOKE_MODEL_NVIDIA_NIM_VISION is required")
+
+ image_data = base64.b64encode(_FIXTURE.read_bytes()).decode("ascii")
+ payload = {
+ "model": provider_model.full_model,
+ "max_tokens": 128,
+ "messages": [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_use",
+ "id": "call_read_image",
+ "name": "Read",
+ "input": {"file_path": "vision-fcc.png"},
+ }
+ ],
+ },
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call_read_image",
+ "content": [
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": image_data,
+ },
+ },
+ {
+ "type": "text",
+ "text": (
+ "Return only the uppercase token visible in the "
+ "image."
+ ),
+ },
+ ],
+ }
+ ],
+ },
+ ],
+ }
+ with SmokeServerDriver(
+ smoke_config,
+ name="product-nvidia-nim-vision-tool-result",
+ env_overrides={
+ "MODEL": provider_model.full_model,
+ "MESSAGING_PLATFORM": "none",
+ "REASONING_POLICY": "off",
+ },
+ ).run() as server:
+ turn = ConversationDriver(server, smoke_config).stream(payload)
+
+ assert_product_stream(turn.events)
+ assert _VISIBLE_TOKEN in turn.text.upper(), (
+ f"{provider_model.full_model} did not read the expected fixture token: "
+ f"{turn.text!r}"
+ )
+
+
+def test_nvidia_nim_vision_function_output_e2e(smoke_config: SmokeConfig) -> None:
+ """Keep a Responses image function output visual through Chat egress."""
+ if not smoke_config.has_provider_configuration("nvidia_nim"):
+ pytest.skip("missing_env: NVIDIA_NIM_API_KEY is not configured")
+ provider_model = smoke_config.nvidia_nim_vision_model()
+ if provider_model is None:
+ pytest.skip("missing_env: FCC_SMOKE_MODEL_NVIDIA_NIM_VISION is required")
+
+ image_data = base64.b64encode(_FIXTURE.read_bytes()).decode("ascii")
+ payload = {
+ "model": provider_model.full_model,
+ "input": [
+ {
+ "type": "function_call",
+ "call_id": "call_read_image",
+ "name": "Read",
+ "arguments": '{"file_path":"vision-fcc.png"}',
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_read_image",
+ "output": [
+ {
+ "type": "input_image",
+ "image_url": f"data:image/png;base64,{image_data}",
+ },
+ {
+ "type": "input_text",
+ "text": "Return only the uppercase token visible in the image.",
+ },
+ ],
+ },
+ ],
+ "max_output_tokens": 128,
+ "stream": True,
+ }
+ with (
+ SmokeServerDriver(
+ smoke_config,
+ name="product-nvidia-nim-vision-function-output",
+ env_overrides={
+ "MODEL": provider_model.full_model,
+ "MESSAGING_PLATFORM": "none",
+ "REASONING_POLICY": "off",
+ },
+ ).run() as server,
+ httpx.stream(
+ "POST",
+ f"{server.base_url}/v1/responses",
+ headers=auth_headers(),
+ json=payload,
+ timeout=smoke_config.timeout_s,
+ ) as response,
+ ):
+ assert response.status_code == 200, response.read()
+ events = parse_sse_lines(response.iter_lines())
+
+ event_names = [event.event for event in events]
+ assert event_names[0] == "response.created", event_names
+ assert event_names[-1] == "response.completed", event_names
+ text = "".join(
+ str(event.data.get("delta", ""))
+ for event in events
+ if event.event == "response.output_text.delta"
+ )
+ assert _VISIBLE_TOKEN in text.upper(), (
+ f"{provider_model.full_model} did not read the expected fixture token: {text!r}"
+ )
diff --git a/src/free_claude_code/core/anthropic/__init__.py b/src/free_claude_code/core/anthropic/__init__.py
index c1a3160d..9b527926 100644
--- a/src/free_claude_code/core/anthropic/__init__.py
+++ b/src/free_claude_code/core/anthropic/__init__.py
@@ -6,7 +6,6 @@ from .conversion import (
OpenAIConversionError,
ReasoningReplayMode,
build_base_request_body,
- is_synthetic_openai_tool_turn_boundary,
)
from .errors import (
anthropic_error_payload,
@@ -94,7 +93,6 @@ __all__ = [
"get_block_attr",
"get_block_type",
"get_token_count",
- "is_synthetic_openai_tool_turn_boundary",
"map_stop_reason",
"serialize_tool_result_content",
"set_if_not_none",
diff --git a/src/free_claude_code/core/anthropic/conversion.py b/src/free_claude_code/core/anthropic/conversion.py
index 24cc0bb2..46ce097a 100644
--- a/src/free_claude_code/core/anthropic/conversion.py
+++ b/src/free_claude_code/core/anthropic/conversion.py
@@ -6,9 +6,22 @@ from dataclasses import dataclass, field
from enum import StrEnum
from typing import Any
+from free_claude_code.core.json_types import JsonObject
+from free_claude_code.core.openai_chat import (
+ IMAGE_TOOL_RESULT_MARKER,
+ close_chat_tool_result_turns,
+ image_tool_result_label,
+)
+
from .content import get_block_attr, get_block_type
+from .image_sources import AnthropicImageSourceError, portable_anthropic_image_url
from .models import MessagesRequest
from .request_serialization import serialize_tool_result_content
+from .tool_results import (
+ ToolResultImage,
+ ToolResultText,
+ decompose_tool_result_content,
+)
from .utils import set_if_not_none
@@ -204,31 +217,57 @@ def _openai_system_text(
return "\n\n".join(text_parts)
-def _openai_user_image_part(block: Any) -> dict[str, Any]:
+def _openai_user_image_part(block: Any) -> JsonObject:
"""Convert one Anthropic user image block without performing I/O."""
- source = get_block_attr(block, "source", {})
- source_type = get_block_attr(source, "type")
+ try:
+ url = portable_anthropic_image_url(get_block_attr(block, "source", {}))
+ except AnthropicImageSourceError as exc:
+ raise OpenAIConversionError(str(exc)) from exc
- if source_type == "base64":
- media_type = get_block_attr(source, "media_type")
- if not isinstance(media_type, str) or not media_type.strip():
- raise OpenAIConversionError(
- "Base64 image source requires a non-empty media_type."
- )
- data = get_block_attr(source, "data")
- if not isinstance(data, str) or not data.strip():
- raise OpenAIConversionError("Base64 image source requires non-empty data.")
- url = f"data:{media_type};base64,{data}"
- elif source_type == "url":
- url = get_block_attr(source, "url")
- if not isinstance(url, str) or not url.strip():
- raise OpenAIConversionError("URL image source requires a non-empty url.")
- else:
- raise OpenAIConversionError(
- f"Unsupported image source type {source_type!r}; expected 'base64' or 'url'."
+ return {"type": "image_url", "image_url": {"url": url}}
+
+
+@dataclass(frozen=True)
+class _OpenAIChatToolResult:
+ tool_message: JsonObject
+ rich_user_message: JsonObject | None = None
+
+
+def _openai_chat_tool_result(block: Any) -> _OpenAIChatToolResult:
+ tool_id = get_block_attr(block, "tool_use_id")
+ tool_content = get_block_attr(block, "content", "")
+ try:
+ decomposed = decompose_tool_result_content(tool_content)
+ except AnthropicImageSourceError as exc:
+ raise OpenAIConversionError(str(exc)) from exc
+
+ if not decomposed.has_images:
+ serialized = serialize_tool_result_content(tool_content)
+ return _OpenAIChatToolResult(
+ tool_message={
+ "role": "tool",
+ "tool_call_id": tool_id,
+ "content": serialized if serialized else "",
+ }
)
- return {"type": "image_url", "image_url": {"url": url}}
+ rich_parts: list[JsonObject] = [
+ {"type": "text", "text": image_tool_result_label(str(tool_id))}
+ ]
+ for part in decomposed.parts:
+ if isinstance(part, ToolResultText):
+ rich_parts.append({"type": "text", "text": part.text})
+ elif isinstance(part, ToolResultImage):
+ rich_parts.append({"type": "image_url", "image_url": {"url": part.url}})
+
+ return _OpenAIChatToolResult(
+ tool_message={
+ "role": "tool",
+ "tool_call_id": tool_id,
+ "content": IMAGE_TOOL_RESULT_MARKER,
+ },
+ rich_user_message={"role": "user", "content": rich_parts},
+ )
def _openai_user_content_parts(content: Any) -> list[dict[str, Any]]:
@@ -280,45 +319,13 @@ def _coalesce_openai_user_messages(
return result
-class _SyntheticOpenAIToolTurnBoundary(dict[str, Any]):
- """Identify an FCC-inserted assistant boundary until wire serialization."""
-
- __slots__ = ()
-
-
-def is_synthetic_openai_tool_turn_boundary(message: object) -> bool:
- """Return whether FCC inserted this message to close an OpenAI tool turn."""
- return isinstance(message, _SyntheticOpenAIToolTurnBoundary)
-
-
-def _close_openai_tool_result_turns(
- messages: list[dict[str, Any]],
-) -> list[dict[str, Any]]:
- """Close completed tool rounds before subsequent user input."""
- result: list[dict[str, Any]] = []
- for message in messages:
- if (
- message.get("role") == "user"
- and result
- and result[-1].get("role") == "tool"
- ):
- # Some OpenAI-compatible chat templates reject a user role directly
- # after tool output. Non-empty whitespace closes the assistant turn
- # without inventing model content.
- result.append(
- _SyntheticOpenAIToolTurnBoundary(role="assistant", content=" ")
- )
- result.append(message)
- return result
-
-
class _OpenAIChatHistoryLedger:
"""Assemble OpenAI chat history while respecting tool-result dependencies."""
def __init__(self) -> None:
self._output: list[dict[str, Any]] = []
self._segments: list[_TranscriptSegment] = []
- self._tool_results: dict[str, dict[str, Any]] = {}
+ self._tool_results: dict[str, _OpenAIChatToolResult] = {}
def add_plain(self, messages: list[dict[str, Any]]) -> None:
if messages:
@@ -369,17 +376,14 @@ class _OpenAIChatHistoryLedger:
if not tuid_s:
self.add_plain(AnthropicToOpenAIConverter._convert_user_message([block]))
return
- tool_content = get_block_attr(block, "content", "")
- serialized = serialize_tool_result_content(tool_content)
- tool_message = {
- "role": "tool",
- "tool_call_id": tuid,
- "content": serialized if serialized else "",
- }
+ converted = _openai_chat_tool_result(block)
if self._has_pending_tool_id(tuid_s):
- self._tool_results[tuid_s] = tool_message
+ self._tool_results[tuid_s] = converted
else:
- self.add_plain([tool_message])
+ messages = [converted.tool_message]
+ if converted.rich_user_message is not None:
+ messages.append(converted.rich_user_message)
+ self.add_plain(messages)
def _drain_ready_segments(self) -> None:
while self._segments:
@@ -402,8 +406,15 @@ class _OpenAIChatHistoryLedger:
break
self._segments.pop(0)
- for tool_id in segment.required_tool_ids:
- self._output.append(self._tool_results.pop(tool_id))
+ results = [
+ self._tool_results.pop(tool_id) for tool_id in segment.required_tool_ids
+ ]
+ self._output.extend(result.tool_message for result in results)
+ self._output.extend(
+ result.rich_user_message
+ for result in results
+ if result.rich_user_message is not None
+ )
deferred_messages = (
AnthropicToOpenAIConverter._deferred_post_tool_to_messages(segment)
)
@@ -413,10 +424,17 @@ class _OpenAIChatHistoryLedger:
if not segment.assistant_emitted:
self._output.append(segment.assistant_message)
segment.assistant_emitted = True
- for tool_id in segment.required_tool_ids:
- tool_result = self._tool_results.pop(tool_id, None)
- if tool_result is not None:
- self._output.append(tool_result)
+ results = [
+ tool_result
+ for tool_id in segment.required_tool_ids
+ if (tool_result := self._tool_results.pop(tool_id, None)) is not None
+ ]
+ self._output.extend(result.tool_message for result in results)
+ self._output.extend(
+ result.rich_user_message
+ for result in results
+ if result.rich_user_message is not None
+ )
self._output.extend(
AnthropicToOpenAIConverter._deferred_post_tool_to_messages(segment)
)
@@ -476,7 +494,7 @@ class AnthropicToOpenAIConverter:
ledger.add_tool_turn(segment)
ordered_messages = ledger.finish()
- closed_messages = _close_openai_tool_result_turns(ordered_messages)
+ closed_messages = close_chat_tool_result_turns(ordered_messages)
return _coalesce_openai_user_messages(closed_messages)
@staticmethod
@@ -667,6 +685,7 @@ class AnthropicToOpenAIConverter:
def _convert_user_message(content: list[Any]) -> list[dict[str, Any]]:
result: list[dict[str, Any]] = []
content_parts: list[dict[str, Any]] = []
+ tool_results: list[_OpenAIChatToolResult] = []
def flush_content() -> None:
if not content_parts:
@@ -680,27 +699,33 @@ class AnthropicToOpenAIConverter:
result.append({"role": "user", "content": message_content})
content_parts.clear()
+ def flush_tool_results() -> None:
+ if not tool_results:
+ return
+ result.extend(item.tool_message for item in tool_results)
+ result.extend(
+ item.rich_user_message
+ for item in tool_results
+ if item.rich_user_message is not None
+ )
+ tool_results.clear()
+
for block in content:
block_type = get_block_type(block)
if block_type == "text":
+ flush_tool_results()
content_parts.append(
{"type": "text", "text": get_block_attr(block, "text", "")}
)
elif block_type == "image":
+ flush_tool_results()
content_parts.append(_openai_user_image_part(block))
elif block_type == "tool_result":
flush_content()
- tool_content = get_block_attr(block, "content", "")
- serialized = serialize_tool_result_content(tool_content)
- result.append(
- {
- "role": "tool",
- "tool_call_id": get_block_attr(block, "tool_use_id"),
- "content": serialized if serialized else "",
- }
- )
+ tool_results.append(_openai_chat_tool_result(block))
+ flush_tool_results()
flush_content()
return result
diff --git a/src/free_claude_code/core/anthropic/image_sources.py b/src/free_claude_code/core/anthropic/image_sources.py
new file mode 100644
index 00000000..e1138179
--- /dev/null
+++ b/src/free_claude_code/core/anthropic/image_sources.py
@@ -0,0 +1,77 @@
+"""Portable Anthropic image-source normalization."""
+
+import base64
+import binascii
+import re
+from collections.abc import Mapping
+
+_SUPPORTED_MEDIA_TYPES = frozenset(
+ {"image/jpeg", "image/png", "image/gif", "image/webp"}
+)
+_DATA_URL_PATTERN = re.compile(
+ r"data:([^;,]+);base64,(.*)",
+ flags=re.IGNORECASE | re.DOTALL,
+)
+_ASCII_WHITESPACE = str.maketrans("", "", " \t\r\n\f\v")
+
+
+class AnthropicImageSourceError(ValueError):
+ """Raised when an Anthropic image source cannot cross protocol boundaries."""
+
+
+def _source_field(source: object, name: str) -> object:
+ if isinstance(source, Mapping):
+ return source.get(name)
+ return getattr(source, name, None)
+
+
+def portable_anthropic_image_url(source: object) -> str:
+ """Return a portable URL for one Anthropic image source."""
+ source_type = _source_field(source, "type")
+ if source_type == "url":
+ url = _source_field(source, "url")
+ if not isinstance(url, str) or not url.strip():
+ raise AnthropicImageSourceError("Image URL must be a non-empty string")
+ return url
+
+ if source_type != "base64":
+ label = source_type if isinstance(source_type, str) else "unknown"
+ raise AnthropicImageSourceError(
+ f"Image source type {label!r} cannot cross this protocol boundary"
+ )
+
+ media_type_value = _source_field(source, "media_type")
+ if not isinstance(media_type_value, str):
+ raise AnthropicImageSourceError("Image media type must be a string")
+ media_type = media_type_value.lower()
+ if media_type not in _SUPPORTED_MEDIA_TYPES:
+ raise AnthropicImageSourceError(
+ f"Unsupported image media type {media_type_value!r}"
+ )
+
+ data = _source_field(source, "data")
+ if not isinstance(data, str) or not data.strip():
+ raise AnthropicImageSourceError("Base64 image data must be non-empty")
+
+ raw_data = data.strip()
+ if raw_data[:5].lower() == "data:":
+ match = _DATA_URL_PATTERN.fullmatch(raw_data)
+ if match is None:
+ raise AnthropicImageSourceError("Malformed base64 image data URL")
+ embedded_media_type, payload = match.groups()
+ if embedded_media_type.lower() != media_type:
+ raise AnthropicImageSourceError(
+ "Image data URL media type does not match the declared media type"
+ )
+ else:
+ payload = raw_data
+
+ canonical_payload = payload.translate(_ASCII_WHITESPACE)
+ if not canonical_payload:
+ raise AnthropicImageSourceError("Base64 image data must be non-empty")
+ try:
+ base64.b64decode(canonical_payload, validate=True)
+ except (binascii.Error, ValueError) as exc:
+ raise AnthropicImageSourceError("Image data is not valid base64") from exc
+
+ return f"data:{media_type};base64,{canonical_payload}"
diff --git a/src/free_claude_code/core/anthropic/tool_results.py b/src/free_claude_code/core/anthropic/tool_results.py
new file mode 100644
index 00000000..af5a27cb
--- /dev/null
+++ b/src/free_claude_code/core/anthropic/tool_results.py
@@ -0,0 +1,63 @@
+"""Structured Anthropic tool-result content."""
+
+import json
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+
+from free_claude_code.core.anthropic.content import get_block_attr, get_block_type
+from free_claude_code.core.anthropic.image_sources import (
+ portable_anthropic_image_url,
+)
+
+
+@dataclass(frozen=True, slots=True)
+class ToolResultText:
+ text: str
+
+
+@dataclass(frozen=True, slots=True)
+class ToolResultImage:
+ url: str
+
+
+@dataclass(frozen=True, slots=True)
+class DecomposedToolResult:
+ parts: tuple[ToolResultText | ToolResultImage, ...]
+ has_images: bool
+
+
+def decompose_tool_result_content(content: object) -> DecomposedToolResult:
+ """Return ordered text/image values for one Anthropic tool result."""
+ if content is None:
+ blocks: Sequence[object] = ()
+ elif isinstance(content, (str, Mapping)) or hasattr(content, "type"):
+ blocks = (content,)
+ elif isinstance(content, Sequence):
+ blocks = content
+ else:
+ blocks = (content,)
+
+ parts: list[ToolResultText | ToolResultImage] = []
+ for block in blocks:
+ block_type = get_block_type(block)
+ if block_type == "text":
+ text = get_block_attr(block, "text", "")
+ if isinstance(text, str):
+ parts.append(ToolResultText(text))
+ else:
+ parts.append(ToolResultText(str(text)))
+ elif block_type == "image":
+ source = get_block_attr(block, "source")
+ parts.append(ToolResultImage(portable_anthropic_image_url(source)))
+ elif isinstance(block, str):
+ parts.append(ToolResultText(block))
+ elif isinstance(block, Mapping):
+ parts.append(ToolResultText(json.dumps(dict(block), ensure_ascii=False)))
+ else:
+ parts.append(ToolResultText(str(block)))
+
+ result = tuple(parts)
+ return DecomposedToolResult(
+ parts=result,
+ has_images=any(isinstance(part, ToolResultImage) for part in result),
+ )
diff --git a/src/free_claude_code/core/openai_chat.py b/src/free_claude_code/core/openai_chat.py
new file mode 100644
index 00000000..07ba63fc
--- /dev/null
+++ b/src/free_claude_code/core/openai_chat.py
@@ -0,0 +1,48 @@
+"""OpenAI Chat Completions wire-history helpers."""
+
+import json
+
+from free_claude_code.core.json_types import JsonObject, JsonValue
+
+IMAGE_TOOL_RESULT_MARKER = "[Image-bearing tool output follows in user content.]"
+
+
+def image_tool_result_label(tool_call_id: str) -> str:
+ """Label relocated visual output with its source function-call identity."""
+ return (
+ "Image-bearing output for tool call "
+ f"{json.dumps(tool_call_id, ensure_ascii=False)}:"
+ )
+
+
+def computer_screenshot_label(call_id: str) -> str:
+ """Label a Responses computer screenshot relocated into Chat user content."""
+ return f"Computer screenshot for call {json.dumps(call_id, ensure_ascii=False)}:"
+
+
+class _SyntheticChatToolTurnBoundary(dict[str, JsonValue]):
+ """Identify an FCC-inserted assistant boundary until wire serialization."""
+
+ __slots__ = ()
+
+
+def is_synthetic_chat_tool_turn_boundary(message: object) -> bool:
+ """Return whether FCC inserted this message to close a Chat tool turn."""
+ return isinstance(message, _SyntheticChatToolTurnBoundary)
+
+
+def close_chat_tool_result_turns(messages: list[JsonObject]) -> list[JsonObject]:
+ """Close completed Chat tool rounds before subsequent user input."""
+ result: list[JsonObject] = []
+ for message in messages:
+ if (
+ message.get("role") == "user"
+ and result
+ and result[-1].get("role") == "tool"
+ ):
+ # Some OpenAI-compatible chat templates reject a user role directly
+ # after tool output. Non-empty whitespace closes the assistant turn
+ # without inventing model content.
+ result.append(_SyntheticChatToolTurnBoundary(role="assistant", content=" "))
+ result.append(message)
+ return result
diff --git a/src/free_claude_code/core/openai_responses/chat_request.py b/src/free_claude_code/core/openai_responses/chat_request.py
index 6a920f12..ce6709d0 100644
--- a/src/free_claude_code/core/openai_responses/chat_request.py
+++ b/src/free_claude_code/core/openai_responses/chat_request.py
@@ -3,9 +3,16 @@
import json
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
+from typing import cast
from free_claude_code.core.anthropic import ReasoningReplayMode
from free_claude_code.core.json_types import JsonObject, JsonValue
+from free_claude_code.core.openai_chat import (
+ IMAGE_TOOL_RESULT_MARKER,
+ close_chat_tool_result_turns,
+ computer_screenshot_label,
+ image_tool_result_label,
+)
from free_claude_code.core.openai_tool_names import OpenAIToolNameCodec
from free_claude_code.core.trace import trace_event
@@ -94,17 +101,26 @@ class _ResponsesChatInputBuilder:
self._reasoning_replay = reasoning_replay
self._structured_reasoning_details = structured_reasoning_details
self._pending_reasoning = _PendingReasoning()
+ self._pending_rich_output_parts: list[dict[str, object]] = []
self._quarantined_call_ids: set[str] = set()
def add(self, item: JsonValue) -> None:
if isinstance(item, str):
+ self._flush_rich_outputs()
self._flush_reasoning()
self.messages.append({"role": "user", "content": item})
return
if not isinstance(item, Mapping):
+ self._flush_rich_outputs()
return
item_type = item.get("type")
+ if item_type not in {
+ "function_call_output",
+ "custom_tool_call_output",
+ "computer_call_output",
+ }:
+ self._flush_rich_outputs()
if item_type in (None, "message") or "role" in item:
self._add_message(item)
return
@@ -117,17 +133,20 @@ class _ResponsesChatInputBuilder:
if item_type in {"function_call_output", "custom_tool_call_output"}:
self._add_tool_output(item, function=item_type == "function_call_output")
return
+ if item_type == "computer_call_output":
+ self._add_computer_output(item)
+ return
if item_type in {"input_text", "output_text", "text"}:
self._flush_reasoning()
self.messages.append({"role": "user", "content": _text_from_part(item)})
return
if item_type == "input_image":
- image = _image_part(item)
- if image is not None:
- self._flush_reasoning()
- self.messages.append({"role": "user", "content": [image]})
+ image = _image_part(item, context="input_image")
+ self._flush_reasoning()
+ self.messages.append({"role": "user", "content": [image]})
def finish(self) -> tuple[list[str], list[dict[str, object]]]:
+ self._flush_rich_outputs()
self._flush_reasoning()
return self.system_parts, self.messages
@@ -208,13 +227,41 @@ class _ResponsesChatInputBuilder:
self._apply_pending_reasoning(previous)
else:
self._flush_reasoning()
+ output = item.get("output")
+ rich_parts = (
+ _rich_function_output_parts(output, call_id=call_id) if function else None
+ )
self.messages.append(
{
"role": "tool",
"tool_call_id": call_id,
- "content": _tool_output_text(item.get("output")),
+ "content": (
+ IMAGE_TOOL_RESULT_MARKER
+ if rich_parts is not None
+ else _tool_output_text(output)
+ ),
}
)
+ if rich_parts is not None:
+ self._pending_rich_output_parts.extend(rich_parts)
+
+ def _add_computer_output(self, item: Mapping[str, JsonValue]) -> None:
+ call_id = call_id_from_item(item)
+ output = item.get("output")
+ if not isinstance(output, Mapping) or output.get("type") != (
+ "computer_screenshot"
+ ):
+ raise ResponsesConversionError(
+ "computer_call_output.output must be a computer_screenshot object"
+ )
+ image = _image_part(output, context="computer_call_output.output")
+ self._flush_reasoning()
+ self._pending_rich_output_parts.extend(
+ [
+ {"type": "text", "text": computer_screenshot_label(call_id)},
+ image,
+ ]
+ )
def _last_tool_call_message(self) -> dict[str, object] | None:
if not self.messages:
@@ -234,6 +281,17 @@ class _ResponsesChatInputBuilder:
if len(message) > 2 or message.get("content"):
self.messages.append(message)
+ def _flush_rich_outputs(self) -> None:
+ if not self._pending_rich_output_parts:
+ return
+ self.messages.append(
+ {
+ "role": "user",
+ "content": list(self._pending_rich_output_parts),
+ }
+ )
+ self._pending_rich_output_parts.clear()
+
def _apply_pending_reasoning(self, message: dict[str, object]) -> None:
text, encrypted = self._pending_reasoning.take()
if text is not None:
@@ -259,7 +317,11 @@ def build_responses_chat_request(
builder.system_parts.append(request.instructions)
for item in _input_items(request.input):
builder.add(item)
- system_parts, messages = builder.finish()
+ system_parts, raw_messages = builder.finish()
+ messages = cast(
+ list[dict[str, object]],
+ close_chat_tool_result_turns(cast(list[JsonObject], raw_messages)),
+ )
if not messages:
raise ResponsesConversionError(
"Responses request must contain usable input for Chat Completions"
@@ -340,12 +402,8 @@ def _message_content(
):
parts.append({"type": "text", "text": _text_from_part(part)})
continue
- if (
- allow_images
- and part_type == "input_image"
- and (image := _image_part(part)) is not None
- ):
- parts.append(image)
+ if allow_images and part_type == "input_image":
+ parts.append(_image_part(part, context="input_image"))
if not parts:
return ""
@@ -371,20 +429,26 @@ def _text_from_part(part: Mapping[str, JsonValue]) -> str:
return ""
-def _image_part(part: Mapping[str, JsonValue]) -> dict[str, object] | None:
+def _image_part(part: Mapping[str, JsonValue], *, context: str) -> dict[str, object]:
source = part.get("image_url")
if isinstance(source, str) and source:
image_url: dict[str, object] = {"url": source}
elif isinstance(source, Mapping):
url = source.get("url")
if not isinstance(url, str) or not url:
- return None
+ raise ResponsesConversionError(
+ f"{context}.image_url requires a non-empty URL"
+ )
image_url = {"url": url}
source_detail = source.get("detail")
if isinstance(source_detail, str):
image_url["detail"] = source_detail
else:
- return None
+ if part.get("file_id") is not None:
+ raise ResponsesConversionError(
+ f"{context}.file_id cannot be represented in Chat Completions"
+ )
+ raise ResponsesConversionError(f"{context}.image_url requires a non-empty URL")
detail = part.get("detail")
if isinstance(detail, str):
image_url["detail"] = detail
@@ -411,6 +475,32 @@ def _tool_output_text(value: JsonValue) -> str:
return json.dumps(value, separators=(",", ":"))
+def _rich_function_output_parts(
+ value: JsonValue, *, call_id: str
+) -> list[dict[str, object]] | None:
+ if not isinstance(value, Sequence) or isinstance(value, str | bytes | bytearray):
+ return None
+ if not any(
+ isinstance(part, Mapping) and part.get("type") == "input_image"
+ for part in value
+ ):
+ return None
+
+ result: list[dict[str, object]] = [
+ {"type": "text", "text": image_tool_result_label(call_id)}
+ ]
+ for part in value:
+ if isinstance(part, Mapping) and part.get("type") == "input_text":
+ result.append({"type": "text", "text": _text_from_part(part)})
+ elif isinstance(part, Mapping) and part.get("type") == "input_image":
+ result.append(
+ _image_part(part, context="function_call_output.output.input_image")
+ )
+ else:
+ result.append({"type": "text", "text": _tool_output_text(part)})
+ return result
+
+
def _apply_reasoning_text(
message: dict[str, object], text: str, mode: ReasoningReplayMode
) -> None:
diff --git a/src/free_claude_code/core/openai_responses/provider_input.py b/src/free_claude_code/core/openai_responses/provider_input.py
index 13cd4034..da7caaac 100644
--- a/src/free_claude_code/core/openai_responses/provider_input.py
+++ b/src/free_claude_code/core/openai_responses/provider_input.py
@@ -5,10 +5,20 @@ from typing import Any
from free_claude_code.core.anthropic.content import get_block_attr, get_block_type
from free_claude_code.core.anthropic.conversion import resolve_anthropic_tool_choice
+from free_claude_code.core.anthropic.image_sources import (
+ AnthropicImageSourceError,
+ portable_anthropic_image_url,
+)
from free_claude_code.core.anthropic.models import MessagesRequest
from free_claude_code.core.anthropic.request_serialization import (
serialize_tool_result_content,
)
+from free_claude_code.core.anthropic.tool_results import (
+ ToolResultImage,
+ ToolResultText,
+ decompose_tool_result_content,
+)
+from free_claude_code.core.json_types import JsonObject
from free_claude_code.core.openai_tool_names import OpenAIToolNameCodec
from free_claude_code.core.reasoning import ReasoningPolicy
@@ -263,13 +273,25 @@ def _user_items(content: Any) -> list[dict[str, Any]]:
message_parts.append(_image_part(block))
elif block_type == "tool_result":
flush_message()
+ tool_content = get_block_attr(block, "content")
+ try:
+ decomposed = decompose_tool_result_content(tool_content)
+ except AnthropicImageSourceError as exc:
+ raise ResponsesConversionError(str(exc)) from exc
+ if decomposed.has_images:
+ output: str | list[JsonObject] = []
+ for part in decomposed.parts:
+ if isinstance(part, ToolResultText):
+ output.append({"type": "input_text", "text": part.text})
+ elif isinstance(part, ToolResultImage):
+ output.append({"type": "input_image", "image_url": part.url})
+ else:
+ output = serialize_tool_result_content(tool_content)
items.append(
{
"type": "function_call_output",
"call_id": str(get_block_attr(block, "tool_use_id", "")),
- "output": serialize_tool_result_content(
- get_block_attr(block, "content")
- ),
+ "output": output,
}
)
elif block_type == "document":
@@ -285,22 +307,10 @@ def _user_items(content: Any) -> list[dict[str, Any]]:
def _image_part(block: Any) -> dict[str, Any]:
- source = get_block_attr(block, "source", {})
- source_type = get_block_attr(source, "type")
- if source_type == "url":
- url = get_block_attr(source, "url")
- elif source_type == "base64":
- media_type = get_block_attr(source, "media_type")
- data = get_block_attr(source, "data")
- if not isinstance(media_type, str) or not isinstance(data, str):
- raise ResponsesConversionError("Base64 images require media_type and data.")
- url = f"data:{media_type};base64,{data}"
- else:
- raise ResponsesConversionError(
- f"Unsupported image source type {source_type!r}."
- )
- if not isinstance(url, str) or not url:
- raise ResponsesConversionError("Image source requires a non-empty URL.")
+ try:
+ url = portable_anthropic_image_url(get_block_attr(block, "source", {}))
+ except AnthropicImageSourceError as exc:
+ raise ResponsesConversionError(str(exc)) from exc
return {"type": "input_image", "image_url": url}
diff --git a/src/free_claude_code/providers/openai_chat/provider.py b/src/free_claude_code/providers/openai_chat/provider.py
index 889f1c5d..a113b64c 100644
--- a/src/free_claude_code/providers/openai_chat/provider.py
+++ b/src/free_claude_code/providers/openai_chat/provider.py
@@ -12,6 +12,7 @@ import httpx2
from loguru import logger
from openai import AsyncOpenAI, DefaultAsyncHttpx2Client
+from free_claude_code.application.errors import InvalidRequestError
from free_claude_code.application.model_metadata import ProviderModelInfo
from free_claude_code.core.anthropic import (
ContentBlockToolUse,
@@ -37,6 +38,7 @@ from free_claude_code.core.failures import ExecutionFailure
from free_claude_code.core.openai_responses import (
OpenAIResponsesRequest,
ResponsesChatRequest,
+ ResponsesConversionError,
build_responses_chat_request,
)
from free_claude_code.core.openai_tool_names import (
@@ -648,11 +650,16 @@ class OpenAIChatProvider(BaseProvider):
reasoning: ReasoningPolicy = DEFAULT_REASONING_POLICY,
) -> ResponsesChatRequest:
"""Build a Chat body directly from Responses ingress."""
- translated = build_responses_chat_request(
- request,
- reasoning_replay=self._profile.request_policy.reasoning_replay,
- structured_reasoning_details=self._profile.structured_reasoning_details,
- )
+ try:
+ translated = build_responses_chat_request(
+ request,
+ reasoning_replay=self._profile.request_policy.reasoning_replay,
+ structured_reasoning_details=(
+ self._profile.structured_reasoning_details
+ ),
+ )
+ except ResponsesConversionError as exc:
+ raise InvalidRequestError(str(exc)) from exc
body = translated.body
apply_openai_chat_body_policy(body, self._profile.request_policy)
self._profile.apply_reasoning_to_body(body, reasoning)
diff --git a/src/free_claude_code/providers/openai_chat/reasoning_details.py b/src/free_claude_code/providers/openai_chat/reasoning_details.py
index ff194289..93bbd337 100644
--- a/src/free_claude_code/providers/openai_chat/reasoning_details.py
+++ b/src/free_claude_code/providers/openai_chat/reasoning_details.py
@@ -4,10 +4,10 @@ import json
from collections.abc import Iterator, Mapping, Sequence
from typing import Any, Literal
-from free_claude_code.core.anthropic import (
- is_synthetic_openai_tool_turn_boundary,
-)
from free_claude_code.core.anthropic.models import MessagesRequest
+from free_claude_code.core.openai_chat import (
+ is_synthetic_chat_tool_turn_boundary,
+)
from free_claude_code.core.reasoning import ReasoningPolicy
from .stream_output import ChatStreamOutput
@@ -31,7 +31,7 @@ def apply_reasoning_details_replay(
if (
not isinstance(message, dict)
or message.get("role") != "assistant"
- or is_synthetic_openai_tool_turn_boundary(message)
+ or is_synthetic_chat_tool_turn_boundary(message)
):
continue
existing = message.get("reasoning_details")
diff --git a/tests/api/test_api.py b/tests/api/test_api.py
index 7b26fd25..eeb295aa 100644
--- a/tests/api/test_api.py
+++ b/tests/api/test_api.py
@@ -3,10 +3,18 @@ from unittest.mock import MagicMock, patch
import pytest
from fastapi.testclient import TestClient
+from free_claude_code.core.anthropic import ReasoningReplayMode
from free_claude_code.core.failures import ExecutionFailure, FailureKind
from free_claude_code.core.reasoning import ReasoningPolicy
from free_claude_code.providers.nvidia_nim import NvidiaNimProvider
+from free_claude_code.providers.openai_chat import (
+ NO_REASONING,
+ OpenAIChatProfile,
+ OpenAIChatProvider,
+ OpenAIChatRequestPolicy,
+)
from tests.api.support import create_test_app
+from tests.providers.support import immediate_admission, make_provider_config
app = create_test_app()
@@ -172,6 +180,54 @@ def test_create_message_ingress_error_has_request_id_without_terminal_header(
assert response.json()["request_id"] == response.headers["request-id"]
+def test_create_message_rejects_unportable_image_as_invalid_request() -> None:
+ with patch(
+ "free_claude_code.providers.openai_chat.provider.AsyncOpenAI",
+ return_value=MagicMock(),
+ ):
+ provider = OpenAIChatProvider(
+ make_provider_config(
+ api_key="test-key",
+ base_url="https://provider.invalid/v1",
+ ),
+ profile=OpenAIChatProfile(
+ OpenAIChatRequestPolicy(
+ provider_name="TEST_CHAT",
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ ),
+ NO_REASONING,
+ ),
+ admission=immediate_admission(provider_name="TEST_CHAT"),
+ )
+ test_app = create_test_app()
+ with (
+ patch("free_claude_code.api.routes.resolve_provider", return_value=provider),
+ TestClient(test_app) as test_client,
+ ):
+ response = test_client.post(
+ "/v1/messages",
+ json={
+ "model": "nvidia_nim/test-model",
+ "max_tokens": 32,
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "image",
+ "source": {"type": "file", "file_id": "file_1"},
+ }
+ ],
+ }
+ ],
+ },
+ )
+
+ assert response.status_code == 400
+ assert response.json()["error"]["type"] == "invalid_request_error"
+ assert "file" in response.json()["error"]["message"]
+
+
def test_create_message_schema_validation_has_request_id_without_terminal_header(
client: TestClient,
):
diff --git a/tests/api/test_openai_responses.py b/tests/api/test_openai_responses.py
index 4357008d..717b682d 100644
--- a/tests/api/test_openai_responses.py
+++ b/tests/api/test_openai_responses.py
@@ -6,6 +6,7 @@ import pytest
from fastapi.testclient import TestClient
from free_claude_code.application.errors import InvalidRequestError
+from free_claude_code.core.anthropic import ReasoningReplayMode
from free_claude_code.core.anthropic.stream_contracts import parse_sse_text
from free_claude_code.core.failures import ExecutionFailure, FailureKind
from free_claude_code.core.json_types import JsonObject
@@ -15,7 +16,14 @@ from free_claude_code.core.reasoning import (
ReasoningEffort,
ReasoningPolicy,
)
+from free_claude_code.providers.openai_chat import (
+ NO_REASONING,
+ OpenAIChatProfile,
+ OpenAIChatProvider,
+ OpenAIChatRequestPolicy,
+)
from tests.api.support import create_test_app
+from tests.providers.support import immediate_admission, make_provider_config
_PUBLIC_MODEL = "nvidia_nim/test-model"
_UPSTREAM_MODEL = "test-model"
@@ -165,6 +173,43 @@ def test_create_response_preflight_rejection_stays_an_ordinary_http_error() -> N
assert provider.requests == []
+def test_create_response_rejects_unportable_image_as_invalid_request() -> None:
+ with patch(
+ "free_claude_code.providers.openai_chat.provider.AsyncOpenAI",
+ return_value=MagicMock(),
+ ):
+ provider = OpenAIChatProvider(
+ make_provider_config(
+ api_key="test-key",
+ base_url="https://provider.invalid/v1",
+ ),
+ profile=OpenAIChatProfile(
+ OpenAIChatRequestPolicy(
+ provider_name="TEST_CHAT",
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ ),
+ NO_REASONING,
+ ),
+ admission=immediate_admission(provider_name="TEST_CHAT"),
+ )
+ app = create_test_app()
+ with (
+ patch("free_claude_code.api.routes.resolve_provider", return_value=provider),
+ TestClient(app) as client,
+ ):
+ response = client.post(
+ "/v1/responses",
+ json={
+ "model": _PUBLIC_MODEL,
+ "input": [{"type": "input_image", "file_id": "file_1"}],
+ },
+ )
+
+ assert response.status_code == 400
+ assert response.json()["error"]["type"] == "invalid_request_error"
+ assert "file_id" in response.json()["error"]["message"]
+
+
def test_create_response_preserves_unknown_top_level_extensions(
responses_client: tuple[TestClient, FakeProvider],
) -> None:
diff --git a/tests/contracts/test_protocol_matrix.py b/tests/contracts/test_protocol_matrix.py
index 5a81c31c..0d1e64ef 100644
--- a/tests/contracts/test_protocol_matrix.py
+++ b/tests/contracts/test_protocol_matrix.py
@@ -46,6 +46,75 @@ def _responses_request() -> OpenAIResponsesRequest:
)
+def _image_messages_request() -> MessagesRequest:
+ return MessagesRequest.model_validate(
+ {
+ "model": "upstream-model",
+ "messages": [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_use",
+ "id": "call_image",
+ "name": "read_image",
+ "input": {},
+ }
+ ],
+ },
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call_image",
+ "content": [
+ {"type": "text", "text": "result"},
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "url": "https://images.example.test/result.png",
+ },
+ },
+ ],
+ }
+ ],
+ },
+ ],
+ "max_tokens": 64,
+ }
+ )
+
+
+def _image_responses_request() -> OpenAIResponsesRequest:
+ return OpenAIResponsesRequest.model_validate(
+ {
+ "model": "upstream-model",
+ "input": [
+ {
+ "type": "function_call",
+ "call_id": "call_image",
+ "name": "read_image",
+ "arguments": "{}",
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_image",
+ "output": [
+ {"type": "input_text", "text": "result"},
+ {
+ "type": "input_image",
+ "image_url": "https://images.example.test/result.png",
+ },
+ ],
+ },
+ ],
+ "max_output_tokens": 64,
+ }
+ )
+
+
def _client(handler: Callable[[httpx2.Request], httpx2.Response]) -> AsyncOpenAI:
return AsyncOpenAI(
api_key="test-key",
@@ -313,3 +382,121 @@ async def test_responses_upstream_accepts_both_ingress_protocols_directly() -> N
responses_events[-1].data["response"]["output"][0]["content"][0]["text"]
== "responses-ok"
)
+
+
+async def test_image_tool_output_remains_visual_across_all_protocol_cells() -> None:
+ chat_requests: list[httpx2.Request] = []
+ responses_requests: list[httpx2.Request] = []
+
+ def chat_handler(request: httpx2.Request) -> httpx2.Response:
+ chat_requests.append(request)
+ return httpx2.Response(
+ 200,
+ headers={"content-type": "text/event-stream"},
+ text=_chat_sse("chat-ok"),
+ )
+
+ def responses_handler(request: httpx2.Request) -> httpx2.Response:
+ responses_requests.append(request)
+ return httpx2.Response(
+ 200,
+ headers={"content-type": "text/event-stream"},
+ text=_responses_sse("responses-ok"),
+ )
+
+ chat_client = _client(chat_handler)
+ config = make_provider_config(
+ api_key="test-key",
+ base_url="https://provider.invalid/v1",
+ )
+ with patch(
+ "free_claude_code.providers.openai_chat.provider.AsyncOpenAI",
+ return_value=chat_client,
+ ):
+ chat_provider = OpenAIChatProvider(
+ config,
+ profile=OpenAIChatProfile(
+ OpenAIChatRequestPolicy(
+ provider_name="MATRIX_CHAT",
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ ),
+ NO_REASONING,
+ ),
+ admission=immediate_admission(provider_name="MATRIX_CHAT"),
+ )
+
+ responses_client = _client(responses_handler)
+ responses_transport = OpenAIResponsesTransport(
+ client=responses_client,
+ admission=immediate_admission(provider_name="MATRIX_RESPONSES"),
+ provider_name="MATRIX_RESPONSES",
+ read_timeout_s=120.0,
+ log_raw_sse_events=False,
+ )
+ try:
+ await _collect(
+ chat_provider.stream_messages(
+ _image_messages_request(),
+ input_tokens=2,
+ request_id="req_matrix_image_messages_chat",
+ response_model="public-model",
+ )
+ )
+ await _collect(
+ chat_provider.stream_responses(
+ _image_responses_request(),
+ input_tokens=2,
+ request_id="req_matrix_image_responses_chat",
+ response_model="public-model",
+ )
+ )
+ await _collect(
+ responses_transport.stream_messages(
+ _image_messages_request(),
+ input_tokens=2,
+ request_id="req_matrix_image_messages_responses",
+ response_model="public-model",
+ reasoning=DEFAULT_REASONING_POLICY,
+ )
+ )
+ await _collect(
+ responses_transport.stream_responses(
+ _image_responses_request(),
+ input_tokens=2,
+ request_id="req_matrix_image_responses_responses",
+ response_model="public-model",
+ reasoning=DEFAULT_REASONING_POLICY,
+ )
+ )
+ finally:
+ await chat_provider.cleanup()
+ await responses_client.close()
+
+ image_url = "https://images.example.test/result.png"
+ messages_chat = json.loads(chat_requests[0].content)
+ responses_chat = json.loads(chat_requests[1].content)
+ messages_responses = json.loads(responses_requests[0].content)
+ responses_responses = json.loads(responses_requests[1].content)
+
+ for chat_body in (messages_chat, responses_chat):
+ tool_message = next(
+ message for message in chat_body["messages"] if message["role"] == "tool"
+ )
+ assert image_url not in tool_message["content"]
+ assert any(
+ part.get("image_url", {}).get("url") == image_url
+ for message in chat_body["messages"]
+ if message["role"] == "user" and isinstance(message["content"], list)
+ for part in message["content"]
+ )
+
+ assert messages_responses["input"][1] == {
+ "type": "function_call_output",
+ "call_id": "call_image",
+ "output": [
+ {"type": "input_text", "text": "result"},
+ {"type": "input_image", "image_url": image_url},
+ ],
+ }
+ assert responses_responses["input"] == _image_responses_request().input
diff --git a/tests/contracts/test_smoke_config.py b/tests/contracts/test_smoke_config.py
index 214c0f8f..2c7739aa 100644
--- a/tests/contracts/test_smoke_config.py
+++ b/tests/contracts/test_smoke_config.py
@@ -113,6 +113,10 @@ def test_nvidia_nim_cli_is_opt_in_smoke_target() -> None:
assert "openrouter_free_cli" in OPT_IN_TARGETS
assert "openrouter_free_cli" in ALL_TARGETS
assert "openrouter_free_cli" in TARGET_REQUIRED_ENV
+ assert "nvidia_nim_vision" not in DEFAULT_TARGETS
+ assert "nvidia_nim_vision" in OPT_IN_TARGETS
+ assert "nvidia_nim_vision" in ALL_TARGETS
+ assert "nvidia_nim_vision" in TARGET_REQUIRED_ENV
def test_ollama_provider_configuration_uses_base_url() -> None:
@@ -1050,6 +1054,25 @@ def test_smoke_config_returns_nvidia_nim_cli_provider_models(monkeypatch) -> Non
assert models[0].source == "nvidia_nim_cli_default"
+def test_smoke_config_requires_explicit_nvidia_nim_vision_model(monkeypatch) -> None:
+ config = _smoke_config()
+ monkeypatch.delenv("FCC_SMOKE_MODEL_NVIDIA_NIM_VISION", raising=False)
+
+ assert config.nvidia_nim_vision_model() is None
+
+ monkeypatch.setenv(
+ "FCC_SMOKE_MODEL_NVIDIA_NIM_VISION",
+ "meta/llama-3.2-11b-vision-instruct",
+ )
+ model = config.nvidia_nim_vision_model()
+
+ assert model == ProviderModel(
+ provider="nvidia_nim",
+ full_model="nvidia_nim/meta/llama-3.2-11b-vision-instruct",
+ source="FCC_SMOKE_MODEL_NVIDIA_NIM_VISION",
+ )
+
+
def test_openrouter_free_cli_default_models_are_normalized() -> None:
refs = openrouter_free_cli_model_refs({})
diff --git a/tests/core/anthropic/test_image_sources.py b/tests/core/anthropic/test_image_sources.py
new file mode 100644
index 00000000..fcebea0a
--- /dev/null
+++ b/tests/core/anthropic/test_image_sources.py
@@ -0,0 +1,113 @@
+import pytest
+
+from free_claude_code.core.anthropic.image_sources import (
+ AnthropicImageSourceError,
+ portable_anthropic_image_url,
+)
+
+
+ "media_type",
+ ("image/jpeg", "image/png", "image/gif", "image/webp"),
+)
+def test_portable_image_url_wraps_supported_base64(media_type: str) -> None:
+ assert (
+ portable_anthropic_image_url(
+ {"type": "base64", "media_type": media_type, "data": "aGVsbG8="}
+ )
+ == f"data:{media_type};base64,aGVsbG8="
+ )
+
+
+def test_portable_image_url_canonicalizes_equivalent_data_url() -> None:
+ assert (
+ portable_anthropic_image_url(
+ {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "data:IMAGE/PNG;BASE64,aGVs\r\nbG8=",
+ }
+ )
+ == "data:image/png;base64,aGVsbG8="
+ )
+
+
+def test_portable_image_url_removes_base64_line_wrapping() -> None:
+ assert (
+ portable_anthropic_image_url(
+ {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": " aG\tVs\r\nbG8= ",
+ }
+ )
+ == "data:image/png;base64,aGVsbG8="
+ )
+
+
+def test_portable_image_url_preserves_remote_url_without_fetching() -> None:
+ url = "https://images.example.test/vision.png"
+
+ assert portable_anthropic_image_url({"type": "url", "url": url}) == url
+
+
+ ("source", "message"),
+ (
+ ({"type": "base64", "media_type": "image/png", "data": ""}, "data"),
+ (
+ {
+ "type": "base64",
+ "media_type": "image/svg+xml",
+ "data": "aGVsbG8=",
+ },
+ "media type",
+ ),
+ (
+ {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "data:image/jpeg;base64,aGVsbG8=",
+ },
+ "does not match",
+ ),
+ (
+ {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "data:image/png,aGVsbG8=",
+ },
+ "data URL",
+ ),
+ (
+ {"type": "base64", "media_type": "image/png", "data": "@@@="},
+ "base64",
+ ),
+ (
+ {"type": "base64", "media_type": "image/png", "data": "abcde"},
+ "base64",
+ ),
+ ({"type": "file", "file_id": "file_123"}, "file"),
+ ({"type": "url", "url": ""}, "URL"),
+ ),
+)
+def test_portable_image_url_rejects_unrepresentable_source(
+ source: dict[str, str], message: str
+) -> None:
+ with pytest.raises(AnthropicImageSourceError, match=message):
+ portable_anthropic_image_url(source)
+
+
+def test_image_source_error_does_not_expose_payload() -> None:
+ secret_payload = "SECRET_IMAGE_PAYLOAD"
+
+ with pytest.raises(AnthropicImageSourceError) as exc_info:
+ portable_anthropic_image_url(
+ {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": secret_payload,
+ }
+ )
+
+ assert secret_payload not in str(exc_info.value)
diff --git a/tests/core/anthropic/test_tool_results.py b/tests/core/anthropic/test_tool_results.py
new file mode 100644
index 00000000..d1fb8d4a
--- /dev/null
+++ b/tests/core/anthropic/test_tool_results.py
@@ -0,0 +1,78 @@
+from free_claude_code.core.anthropic.tool_results import (
+ ToolResultImage,
+ ToolResultText,
+ decompose_tool_result_content,
+)
+
+
+def test_decompose_tool_result_preserves_mixed_part_order() -> None:
+ result = decompose_tool_result_content(
+ [
+ {"type": "text", "text": "before"},
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "aGVsbG8=",
+ },
+ },
+ {"type": "text", "text": "after"},
+ ]
+ )
+
+ assert result.has_images is True
+ assert result.parts == (
+ ToolResultText("before"),
+ ToolResultImage("data:image/png;base64,aGVsbG8="),
+ ToolResultText("after"),
+ )
+
+
+def test_decompose_tool_result_preserves_unknown_siblings_as_text() -> None:
+ result = decompose_tool_result_content(
+ [
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "url": "https://images.example.test/result.png",
+ },
+ },
+ {"type": "document", "title": "café"},
+ 7,
+ ]
+ )
+
+ assert result.parts == (
+ ToolResultImage("https://images.example.test/result.png"),
+ ToolResultText('{"type": "document", "title": "café"}'),
+ ToolResultText("7"),
+ )
+
+
+def test_decompose_text_only_result_does_not_claim_an_image() -> None:
+ result = decompose_tool_result_content(
+ [{"type": "text", "text": "first"}, {"answer": 42}]
+ )
+
+ assert result.has_images is False
+ assert result.parts == (
+ ToolResultText("first"),
+ ToolResultText('{"answer": 42}'),
+ )
+
+
+def test_decompose_single_image_dictionary() -> None:
+ result = decompose_tool_result_content(
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "url": "https://images.example.test/single.png",
+ },
+ }
+ )
+
+ assert result.has_images is True
+ assert result.parts == (ToolResultImage("https://images.example.test/single.png"),)
diff --git a/tests/core/openai_responses/test_chat_request.py b/tests/core/openai_responses/test_chat_request.py
index 73f95e5b..977a9745 100644
--- a/tests/core/openai_responses/test_chat_request.py
+++ b/tests/core/openai_responses/test_chat_request.py
@@ -155,6 +155,7 @@ def test_build_responses_chat_request_preserves_rich_supported_semantics() -> No
],
},
{"role": "tool", "tool_call_id": "call_1", "content": "FCC"},
+ {"role": "assistant", "content": " "},
{"role": "user", "content": "Continue"},
],
"tools": [
@@ -304,6 +305,310 @@ def test_build_responses_chat_request_quarantines_one_malformed_call_pair() -> N
]
+ "image",
+ (
+ {"type": "input_image", "file_id": "file_1"},
+ {"type": "input_image", "image_url": {"detail": "low"}},
+ {"type": "input_image", "image_url": ""},
+ ),
+)
+def test_build_responses_chat_request_rejects_unportable_direct_image(
+ image: dict[str, object],
+) -> None:
+ with pytest.raises(ResponsesConversionError, match="input_image"):
+ build_responses_chat_request(
+ _request(input=[image]),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+
+def test_build_responses_chat_request_preserves_chat_shaped_image_detail() -> None:
+ translated = build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "input_image",
+ "image_url": {
+ "url": "https://images.example.test/direct.png",
+ "detail": "low",
+ },
+ "detail": "high",
+ }
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+ assert translated.body["messages"] == [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "image_url",
+ "image_url": {
+ "url": "https://images.example.test/direct.png",
+ "detail": "high",
+ },
+ }
+ ],
+ }
+ ]
+
+
+def test_build_responses_chat_request_relocates_rich_function_output() -> None:
+ image_url = "data:image/png;base64,AA=="
+ translated = build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "function_call",
+ "call_id": "call_image",
+ "name": "read_image",
+ "arguments": "{}",
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_image",
+ "output": [
+ {"type": "input_text", "text": "before"},
+ {"type": "input_image", "image_url": image_url},
+ {"type": "input_file", "file_id": "file_1"},
+ {"type": "input_text", "text": "after"},
+ ],
+ },
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+ messages = translated.body["messages"]
+ assert isinstance(messages, list)
+ assert [message["role"] for message in messages] == [
+ "assistant",
+ "tool",
+ "assistant",
+ "user",
+ ]
+ assert messages[1] == {
+ "role": "tool",
+ "tool_call_id": "call_image",
+ "content": "[Image-bearing tool output follows in user content.]",
+ }
+ assert messages[3]["content"] == [
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_image":',
+ },
+ {"type": "text", "text": "before"},
+ {"type": "image_url", "image_url": {"url": image_url}},
+ {
+ "type": "text",
+ "text": '{"type":"input_file","file_id":"file_1"}',
+ },
+ {"type": "text", "text": "after"},
+ ]
+ assert image_url not in messages[1]["content"]
+
+
+def test_build_responses_chat_request_closes_parallel_tools_before_rich_outputs() -> (
+ None
+):
+ translated = build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "function_call",
+ "call_id": "call_a",
+ "name": "read_a",
+ "arguments": "{}",
+ },
+ {
+ "type": "function_call",
+ "call_id": "call_b",
+ "name": "read_b",
+ "arguments": "{}",
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_a",
+ "output": [
+ {
+ "type": "input_image",
+ "image_url": "https://images.example.test/a.png",
+ }
+ ],
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_b",
+ "output": [
+ {
+ "type": "input_image",
+ "image_url": "https://images.example.test/b.png",
+ }
+ ],
+ },
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+ messages = translated.body["messages"]
+ assert isinstance(messages, list)
+ assert [message["role"] for message in messages] == [
+ "assistant",
+ "tool",
+ "tool",
+ "assistant",
+ "user",
+ ]
+ assert [message["tool_call_id"] for message in messages[1:3]] == [
+ "call_a",
+ "call_b",
+ ]
+ assert messages[4]["content"] == [
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_a":',
+ },
+ {
+ "type": "image_url",
+ "image_url": {"url": "https://images.example.test/a.png"},
+ },
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_b":',
+ },
+ {
+ "type": "image_url",
+ "image_url": {"url": "https://images.example.test/b.png"},
+ },
+ ]
+
+
+def test_build_responses_chat_request_flushes_rich_output_before_later_input() -> None:
+ translated = build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "function_call",
+ "call_id": "call_image",
+ "name": "read_image",
+ "arguments": "{}",
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_image",
+ "output": [
+ {
+ "type": "input_image",
+ "image_url": "https://images.example.test/a.png",
+ }
+ ],
+ },
+ {"role": "user", "content": "Describe the image."},
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+ messages = translated.body["messages"]
+ assert isinstance(messages, list)
+ assert messages[-2]["content"][0]["text"].startswith(
+ "Image-bearing output for tool call"
+ )
+ assert messages[-1] == {"role": "user", "content": "Describe the image."}
+
+
+def test_build_responses_chat_request_relocates_computer_screenshot() -> None:
+ translated = build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "computer_call_output",
+ "call_id": "computer_1",
+ "output": {
+ "type": "computer_screenshot",
+ "image_url": "https://images.example.test/screen.png",
+ },
+ "acknowledged_safety_checks": [{"id": "check_1"}],
+ }
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+ assert translated.body["messages"] == [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": 'Computer screenshot for call "computer_1":',
+ },
+ {
+ "type": "image_url",
+ "image_url": {"url": "https://images.example.test/screen.png"},
+ },
+ ],
+ }
+ ]
+
+
+def test_build_responses_chat_request_keeps_reasoning_before_computer_output() -> None:
+ translated = build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "reasoning",
+ "content": [{"type": "reasoning_text", "text": "Think"}],
+ },
+ {
+ "type": "computer_call_output",
+ "call_id": "computer_1",
+ "output": {
+ "type": "computer_screenshot",
+ "image_url": "https://images.example.test/screen.png",
+ },
+ },
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.REASONING_CONTENT,
+ )
+
+ messages = translated.body["messages"]
+ assert isinstance(messages, list)
+ assert [message["role"] for message in messages] == ["assistant", "user"]
+ assert messages[0]["reasoning_content"] == "Think"
+
+
+ "output",
+ (
+ {"type": "computer_screenshot", "file_id": "file_1"},
+ {"type": "computer_screenshot", "image_url": ""},
+ {"type": "not_a_screenshot", "image_url": "https://x/image.png"},
+ ),
+)
+def test_build_responses_chat_request_rejects_unportable_computer_screenshot(
+ output: dict[str, object],
+) -> None:
+ with pytest.raises(ResponsesConversionError, match="computer_call_output"):
+ build_responses_chat_request(
+ _request(
+ input=[
+ {
+ "type": "computer_call_output",
+ "call_id": "computer_1",
+ "output": output,
+ }
+ ]
+ ),
+ reasoning_replay=ReasoningReplayMode.DISABLED,
+ )
+
+
def test_build_responses_chat_request_skips_unsupported_optional_items_and_choice() -> (
None
):
@@ -333,8 +638,8 @@ def test_build_responses_chat_request_rejects_no_usable_input() -> None:
)
-def test_build_responses_chat_request_rejects_message_with_only_skipped_parts() -> None:
- with pytest.raises(ResponsesConversionError, match="usable input"):
+def test_build_responses_chat_request_rejects_message_with_only_file_id_image() -> None:
+ with pytest.raises(ResponsesConversionError, match="input_image"):
build_responses_chat_request(
_request(
input=[
diff --git a/tests/core/openai_responses/test_native.py b/tests/core/openai_responses/test_native.py
index 9700e947..1b670d0f 100644
--- a/tests/core/openai_responses/test_native.py
+++ b/tests/core/openai_responses/test_native.py
@@ -49,6 +49,59 @@ def test_native_request_preserves_extensions_and_forces_stateless_streaming() ->
assert request.previous_response_id == "resp_previous"
+def test_native_request_preserves_complete_multimodal_input_tree() -> None:
+ input_tree = [
+ {
+ "type": "message",
+ "role": "user",
+ "content": [
+ {
+ "type": "input_image",
+ "image_url": "data:image/png;base64,AA==",
+ "detail": "high",
+ "future_image_option": True,
+ },
+ {"type": "input_image", "file_id": "file_image"},
+ {"type": "input_file", "file_id": "file_document"},
+ ],
+ },
+ {
+ "type": "function_call_output",
+ "call_id": "call_image",
+ "output": [
+ {"type": "input_text", "text": "result"},
+ {
+ "type": "input_image",
+ "image_url": "https://images.example.test/result.png",
+ },
+ {"type": "input_file", "file_id": "file_result"},
+ {"type": "future_output", "value": 7},
+ ],
+ },
+ {
+ "type": "computer_call_output",
+ "call_id": "computer_1",
+ "output": {
+ "type": "computer_screenshot",
+ "file_id": "file_screen",
+ "future_screenshot_option": "kept",
+ },
+ "acknowledged_safety_checks": [{"id": "check_1"}],
+ },
+ ]
+ request = OpenAIResponsesRequest.model_validate(
+ {"model": "gateway-model", "input": input_tree}
+ )
+
+ body = build_native_responses_request(
+ request,
+ model="upstream-model",
+ reasoning=ReasoningPolicy.provider_default(),
+ )
+
+ assert body["input"] == input_tree
+
+
def test_native_request_preserves_reasoning_when_fcc_did_not_override_it() -> None:
request = OpenAIResponsesRequest.model_validate(
{
diff --git a/tests/core/openai_responses/test_provider_input.py b/tests/core/openai_responses/test_provider_input.py
index 9ba7c778..afc9fac2 100644
--- a/tests/core/openai_responses/test_provider_input.py
+++ b/tests/core/openai_responses/test_provider_input.py
@@ -108,6 +108,137 @@ def test_build_responses_provider_request_preserves_multiturn_protocol() -> None
}
+def test_responses_provider_request_canonicalizes_direct_anthropic_image() -> None:
+ request = MessagesRequest.model_validate(
+ {
+ "model": "gpt-test",
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "data:IMAGE/PNG;BASE64,aGVs\r\nbG8=",
+ },
+ }
+ ],
+ }
+ ],
+ }
+ )
+
+ body = build_responses_provider_request(
+ request,
+ reasoning=ReasoningPolicy.provider_default(),
+ )
+
+ assert body["input"] == [
+ {
+ "type": "message",
+ "role": "user",
+ "content": [
+ {
+ "type": "input_image",
+ "image_url": "data:image/png;base64,aGVsbG8=",
+ }
+ ],
+ }
+ ]
+
+
+def test_responses_provider_request_preserves_rich_tool_output_order() -> None:
+ request = MessagesRequest.model_validate(
+ {
+ "model": "gpt-test",
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call_image",
+ "content": [
+ {"type": "text", "text": "before"},
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "url": "https://images.example.test/tool.png",
+ },
+ },
+ {"type": "document", "title": "reference"},
+ {"type": "text", "text": "after"},
+ ],
+ }
+ ],
+ }
+ ],
+ }
+ )
+
+ body = build_responses_provider_request(
+ request,
+ reasoning=ReasoningPolicy.provider_default(),
+ )
+
+ assert body["input"] == [
+ {
+ "type": "function_call_output",
+ "call_id": "call_image",
+ "output": [
+ {"type": "input_text", "text": "before"},
+ {
+ "type": "input_image",
+ "image_url": "https://images.example.test/tool.png",
+ },
+ {
+ "type": "input_text",
+ "text": '{"type": "document", "title": "reference"}',
+ },
+ {"type": "input_text", "text": "after"},
+ ],
+ }
+ ]
+
+
+ "source",
+ (
+ {"type": "base64", "media_type": "image/png", "data": "SECRET_BAD"},
+ {"type": "file", "file_id": "file_1"},
+ ),
+)
+def test_responses_provider_request_rejects_unportable_tool_image(source) -> None:
+ request = MessagesRequest.model_validate(
+ {
+ "model": "gpt-test",
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call_image",
+ "content": [{"type": "image", "source": source}],
+ }
+ ],
+ }
+ ],
+ }
+ )
+
+ with pytest.raises(ResponsesConversionError) as exc_info:
+ build_responses_provider_request(
+ request,
+ reasoning=ReasoningPolicy.provider_default(),
+ )
+
+ assert "SECRET_BAD" not in str(exc_info.value)
+
+
def test_build_responses_provider_request_accepts_claude_client_controls() -> None:
request = MessagesRequest.model_validate(
{
diff --git a/tests/core/test_openai_chat.py b/tests/core/test_openai_chat.py
new file mode 100644
index 00000000..7b8b4b90
--- /dev/null
+++ b/tests/core/test_openai_chat.py
@@ -0,0 +1,44 @@
+from free_claude_code.core.json_types import JsonObject
+from free_claude_code.core.openai_chat import (
+ close_chat_tool_result_turns,
+ is_synthetic_chat_tool_turn_boundary,
+)
+
+
+def test_close_chat_tool_result_turns_inserts_identifiable_assistant_boundary() -> None:
+ messages: list[JsonObject] = [
+ {
+ "role": "assistant",
+ "content": "",
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "read", "arguments": "{}"},
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "call_1", "content": "done"},
+ {"role": "user", "content": "continue"},
+ ]
+
+ closed = close_chat_tool_result_turns(messages)
+
+ assert [message["role"] for message in closed] == [
+ "assistant",
+ "tool",
+ "assistant",
+ "user",
+ ]
+ assert closed[2]["content"] == " "
+ assert is_synthetic_chat_tool_turn_boundary(closed[2]) is True
+ assert is_synthetic_chat_tool_turn_boundary(closed[0]) is False
+
+
+def test_close_chat_tool_result_turns_leaves_other_transitions_unchanged() -> None:
+ messages: list[JsonObject] = [
+ {"role": "user", "content": "start"},
+ {"role": "assistant", "content": "done"},
+ ]
+
+ assert close_chat_tool_result_turns(messages) == messages
diff --git a/tests/providers/test_converter.py b/tests/providers/test_converter.py
index b24f2c6f..47a98e37 100644
--- a/tests/providers/test_converter.py
+++ b/tests/providers/test_converter.py
@@ -7,9 +7,11 @@ from free_claude_code.core.anthropic import (
OpenAIConversionError,
ReasoningReplayMode,
build_base_request_body,
- is_synthetic_openai_tool_turn_boundary,
)
from free_claude_code.core.anthropic.models import MessagesRequest
+from free_claude_code.core.openai_chat import (
+ is_synthetic_chat_tool_turn_boundary,
+)
# --- Mock Classes ---
@@ -1132,6 +1134,263 @@ def test_convert_user_message_image_sources(source, expected_url):
]
+ "data",
+ (
+ "data:IMAGE/PNG;BASE64,aGVs\r\nbG8=",
+ " aG\tVs\r\nbG8= ",
+ ),
+)
+def test_convert_user_message_canonicalizes_equivalent_base64(data):
+ messages = [
+ MockMessage(
+ "user",
+ [
+ MockBlock(
+ type="image",
+ source={
+ "type": "base64",
+ "media_type": "image/png",
+ "data": data,
+ },
+ )
+ ],
+ )
+ ]
+
+ result = AnthropicToOpenAIConverter.convert_messages(messages)
+
+ assert result[0]["content"][0]["image_url"]["url"] == (
+ "data:image/png;base64,aGVsbG8="
+ )
+
+
+def test_convert_image_tool_result_to_schema_valid_chat_history():
+ image_url = "https://images.example.test/tool.png"
+ messages = [
+ MockMessage(
+ "assistant",
+ [MockBlock(type="tool_use", id="call_image", name="Read", input={})],
+ ),
+ MockMessage(
+ "user",
+ [
+ MockBlock(
+ type="tool_result",
+ tool_use_id="call_image",
+ content=[
+ {
+ "type": "image",
+ "source": {"type": "url", "url": image_url},
+ }
+ ],
+ )
+ ],
+ ),
+ ]
+
+ result = AnthropicToOpenAIConverter.convert_messages(messages)
+
+ assert [message["role"] for message in result] == [
+ "assistant",
+ "tool",
+ "assistant",
+ "user",
+ ]
+ assert result[1] == {
+ "role": "tool",
+ "tool_call_id": "call_image",
+ "content": "[Image-bearing tool output follows in user content.]",
+ }
+ assert is_synthetic_chat_tool_turn_boundary(result[2])
+ assert result[3]["content"] == [
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_image":',
+ },
+ {"type": "image_url", "image_url": {"url": image_url}},
+ ]
+ assert image_url not in result[1]["content"]
+
+
+def test_convert_mixed_image_tool_result_preserves_part_order_once():
+ image_url = "data:image/png;base64,aGVsbG8="
+ messages = [
+ MockMessage(
+ "assistant",
+ [MockBlock(type="tool_use", id="call_mixed", name="Read", input={})],
+ ),
+ MockMessage(
+ "user",
+ [
+ MockBlock(
+ type="tool_result",
+ tool_use_id="call_mixed",
+ content=[
+ {"type": "text", "text": "before"},
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": "aGVsbG8=",
+ },
+ },
+ {"type": "text", "text": "after"},
+ ],
+ )
+ ],
+ ),
+ ]
+
+ result = AnthropicToOpenAIConverter.convert_messages(messages)
+
+ assert result[3]["content"] == [
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_mixed":',
+ },
+ {"type": "text", "text": "before"},
+ {"type": "image_url", "image_url": {"url": image_url}},
+ {"type": "text", "text": "after"},
+ ]
+ assert (
+ sum(
+ part.get("text") == "before"
+ for message in result
+ if isinstance(message.get("content"), list)
+ for part in message["content"]
+ )
+ == 1
+ )
+
+
+def test_convert_parallel_image_tool_results_closes_all_tools_before_images():
+ messages = [
+ MockMessage(
+ "assistant",
+ [
+ MockBlock(type="tool_use", id="call_a", name="ReadA", input={}),
+ MockBlock(type="tool_use", id="call_b", name="ReadB", input={}),
+ ],
+ ),
+ MockMessage(
+ "user",
+ [
+ MockBlock(
+ type="tool_result",
+ tool_use_id="call_b",
+ content=[
+ {
+ "type": "image",
+ "source": {"type": "url", "url": "https://x/b.png"},
+ }
+ ],
+ ),
+ MockBlock(
+ type="tool_result",
+ tool_use_id="call_a",
+ content=[
+ {
+ "type": "image",
+ "source": {"type": "url", "url": "https://x/a.png"},
+ }
+ ],
+ ),
+ ],
+ ),
+ ]
+
+ result = AnthropicToOpenAIConverter.convert_messages(messages)
+
+ assert [message["role"] for message in result] == [
+ "assistant",
+ "tool",
+ "tool",
+ "assistant",
+ "user",
+ ]
+ assert [message["tool_call_id"] for message in result[1:3]] == [
+ "call_a",
+ "call_b",
+ ]
+ assert result[4]["content"] == [
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_a":',
+ },
+ {"type": "image_url", "image_url": {"url": "https://x/a.png"}},
+ {
+ "type": "text",
+ "text": 'Image-bearing output for tool call "call_b":',
+ },
+ {"type": "image_url", "image_url": {"url": "https://x/b.png"}},
+ ]
+
+
+def test_convert_image_tool_result_precedes_following_user_text():
+ messages = [
+ MockMessage(
+ "assistant",
+ [MockBlock(type="tool_use", id="call_image", name="Read", input={})],
+ ),
+ MockMessage(
+ "user",
+ [
+ MockBlock(
+ type="tool_result",
+ tool_use_id="call_image",
+ content=[
+ {
+ "type": "image",
+ "source": {"type": "url", "url": "https://x/a.png"},
+ }
+ ],
+ ),
+ MockBlock(type="text", text="Now describe it."),
+ ],
+ ),
+ ]
+
+ result = AnthropicToOpenAIConverter.convert_messages(messages)
+
+ assert result[3]["content"][-1] == {
+ "type": "text",
+ "text": "Now describe it.",
+ }
+
+
+ "source",
+ (
+ {"type": "base64", "media_type": "image/png", "data": "SECRET_BAD"},
+ {"type": "file", "file_id": "file_1"},
+ ),
+)
+def test_convert_image_tool_result_rejects_unportable_source_safely(source):
+ messages = [
+ MockMessage(
+ "assistant",
+ [MockBlock(type="tool_use", id="call_image", name="Read", input={})],
+ ),
+ MockMessage(
+ "user",
+ [
+ MockBlock(
+ type="tool_result",
+ tool_use_id="call_image",
+ content=[{"type": "image", "source": source}],
+ )
+ ],
+ ),
+ ]
+
+ with pytest.raises(OpenAIConversionError) as exc_info:
+ AnthropicToOpenAIConverter.convert_messages(messages)
+
+ assert "SECRET_BAD" not in str(exc_info.value)
+
+
def test_convert_user_message_preserves_interleaved_image_text_order():
messages = [
MockMessage(
@@ -1142,7 +1401,7 @@ def test_convert_user_message_preserves_interleaved_image_text_order():
source={
"type": "base64",
"media_type": "image/jpeg",
- "data": "FIRST",
+ "data": "RklSU1Q=",
},
),
MockBlock(type="text", text="Compare the first image with this one."),
@@ -1163,7 +1422,7 @@ def test_convert_user_message_preserves_interleaved_image_text_order():
"content": [
{
"type": "image_url",
- "image_url": {"url": "data:image/jpeg;base64,FIRST"},
+ "image_url": {"url": "data:image/jpeg;base64,RklSU1Q="},
},
{
"type": "text",
@@ -1216,15 +1475,15 @@ def test_convert_user_image_before_tool_result_preserves_message_order():
[
(
{"type": "base64", "media_type": "", "data": "AAAA"},
- "non-empty media_type",
+ "media type",
),
(
{"type": "base64", "media_type": "image/png", "data": ""},
- "non-empty data",
+ "data must be non-empty",
),
- ({"type": "url", "url": ""}, "non-empty url"),
- ({"type": "file", "file_id": "file_1"}, "Unsupported image source type"),
- ({}, "Unsupported image source type"),
+ ({"type": "url", "url": ""}, "URL must be a non-empty"),
+ ({"type": "file", "file_id": "file_1"}, "cannot cross"),
+ ({}, "cannot cross"),
],
)
def test_convert_user_message_rejects_unconvertible_image_sources(source, error):
@@ -1340,7 +1599,7 @@ def test_user_after_completed_tool_result_gets_neutral_assistant_boundary():
"user",
]
assert result[3] == {"role": "assistant", "content": " "}
- assert is_synthetic_openai_tool_turn_boundary(result[3])
+ assert is_synthetic_chat_tool_turn_boundary(result[3])
assert json.loads(json.dumps(result[3])) == {
"role": "assistant",
"content": " ",
diff --git a/uv.lock b/uv.lock
index 65e1cf89..2deae094 100644
--- a/uv.lock
+++ b/uv.lock
@@ -607,7 +607,7 @@ wheels = [
[[package]]
name = "free-claude-code"
-version = "5.15.9"
+version = "5.15.10"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },