Skip to content

Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

Python

Split API request handling by product surface (#923)

Ali Khokhardb60452c0c18
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 9fe57418..4fbfb43b 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -32,9 +32,10 @@ flowchart LR
Bots[Discord or Telegram Bots] --> Messaging[Messaging Bridge]
Messaging --> ClientCLI[Managed Client CLI Sessions]
ClientCLI --> ProxyAPI
- ProxyAPI --> Pipeline[ApiRequestPipeline]
- Pipeline --> Router[ModelRouter]
- Pipeline --> Providers[ProviderRegistry]
+ ProxyAPI --> Handlers[API Product Handlers]
+ Handlers --> Router[ModelRouter]
+ Handlers --> Executor[ProviderExecutionService]
+ Executor --> Providers[ProviderRegistry]
Providers --> OpenAIChat[OpenAI Chat Providers]
Providers --> NativeAnthropic[Anthropic Messages Providers]
```
@@ -43,8 +44,9 @@ flowchart LR
The installable wheel packages are declared in [pyproject.toml](pyproject.toml):
-- [api/](api/) owns the FastAPI app, route handlers, request pipeline, model
- catalog, admin APIs, local optimizations, and server-tool handling.
+- [api/](api/) owns the FastAPI app, route handlers, API product handlers, shared
+ provider execution, model catalog, admin APIs, local optimizations, and
+ server-tool handling.
- [cli/](cli/) owns console entrypoints, client CLI launchers, process/session
management, and client adapter contracts.
- [config/](config/) owns settings, provider metadata, filesystem paths,
@@ -105,10 +107,11 @@ The current package boundaries are intentional, but several modules still carry
large orchestration responsibilities. Treat these as refactor targets, not as
new places to add unrelated behavior:
-- [api/request_pipeline.py](api/request_pipeline.py) coordinates routing,
- message-only intercepts, provider execution, token counting, and Responses
- adaptation. Keep route handlers thin and keep new pre-provider behavior as
- explicit pipeline intercepts.
+- [api/handlers/](api/handlers/) owns customer-facing API product flows:
+ Claude Messages, OpenAI Responses, and token counting. Keep route handlers
+ thin, keep Claude-only behavior in the Messages handler, and use
+ [api/provider_execution.py](api/provider_execution.py) only for shared
+ provider resolution, preflight, tracing, token counting, and streaming.
- [providers/transports/](providers/transports/) owns provider transport
families. The OpenAI-chat and native Anthropic transport packages split thin
transport bases from per-request stream runners, recovery event construction,
@@ -221,36 +224,42 @@ proxy auth is disabled. Otherwise the token may be supplied through `x-api-key`,
`Authorization: Bearer ...`, or `anthropic-auth-token`. Comparisons use
constant-time matching.
-`ApiRequestPipeline` in
-[api/request_pipeline.py](api/request_pipeline.py) coordinates request handling.
-It validates non-empty messages, resolves models, runs explicit message-only
-intercepts for local server tools and optimizations, resolves a provider,
-preflights the upstream request, then streams Anthropic SSE back to the caller.
+[api/handlers/](api/handlers/) owns the public API product flows.
+`MessagesHandler` validates non-empty messages, resolves models, applies
+Claude-only safety-classifier and local optimization policy, handles local web
+server tools, then streams Anthropic SSE. `ResponsesHandler` owns streaming-only
+OpenAI Responses validation and conversion for Codex clients. `TokenCountHandler`
+owns Anthropic token counting. Shared provider execution lives in
+[api/provider_execution.py](api/provider_execution.py), which resolves a
+provider, preflights the upstream request, emits trace events, counts input
+tokens, and returns an Anthropic SSE iterator.
```mermaid
sequenceDiagram
participant Client
participant Route as FastAPIRoute
- participant Pipeline as ApiRequestPipeline
+ participant Handler as ProductHandler
participant Router as ModelRouter
+ participant Exec as ProviderExecution
participant Registry as ProviderRegistry
participant Provider
Client->>Route: POST /v1/messages
Route->>Route: require_api_key
- Route->>Pipeline: create_message
- Pipeline->>Router: resolve model and thinking
- Pipeline->>Pipeline: server tools or optimizations
- Pipeline->>Registry: resolve provider
+ Route->>Handler: create message
+ Handler->>Router: resolve model and thinking
+ Handler->>Handler: server tools or optimizations
+ Handler->>Exec: stream routed request
+ Exec->>Registry: resolve provider
Registry->>Provider: cached or new provider
- Pipeline->>Provider: preflight_stream
- Pipeline->>Provider: stream_response
+ Exec->>Provider: preflight_stream
+ Exec->>Provider: stream_response
Provider-->>Client: Anthropic SSE events
```
-OpenAI Responses uses the same provider execution path without message-only
-intercepts. `create_response()` delegates Responses protocol work to the
-`OpenAIResponsesAdapter` in
+OpenAI Responses uses the same provider execution primitive without importing
+Claude-only message intercepts. `ResponsesHandler` delegates protocol work to
+the `OpenAIResponsesAdapter` in
[core/openai_responses/adapter.py](core/openai_responses/adapter.py). The adapter
converts the Responses payload into an Anthropic Messages payload before
provider execution, then converts Anthropic SSE back to Responses SSE.
@@ -423,25 +432,25 @@ common low-value client requests before they reach a provider:
- suggestion mode;
- filepath extraction.
-The service runs these only after model routing and after local server-tool
+The Messages handler runs these only after model routing and after local server-tool
handling. Each optimization is controlled by settings flags.
Claude Code auto-mode safety-classifier requests are a message-only routing
-policy, not a short-circuit response. After routing, the pipeline detects the
+policy, not a short-circuit response. After routing, the Messages handler detects the
narrow classifier prompt shape and forces thinking off before provider execution
so Claude Code receives a parser-readable `<block>yes</block>` or
`<block>no</block>` verdict.
Local `web_search` and `web_fetch` handling lives under
[api/web_tools/](api/web_tools/). When `ENABLE_WEB_SERVER_TOOLS` is true, the
-service can stream local Anthropic server-tool responses without sending the
+Messages handler can stream local Anthropic server-tool responses without sending the
request upstream. [api/web_tools/egress.py](api/web_tools/egress.py) enforces URL
scheme and private-network restrictions for `web_fetch`.
OpenAI-chat upstream providers are identified by
`ProviderDescriptor.transport_type == "openai_chat"` in
[config/provider_catalog.py](config/provider_catalog.py). They cannot safely
-represent Anthropic server-tool blocks, so the service rejects unsupported
+represent Anthropic server-tool blocks, so the Messages handler rejects unsupported
server-tool requests before provider execution instead of performing a lossy
conversion. Forced `web_search` or `web_fetch` requests are handled locally when
`ENABLE_WEB_SERVER_TOOLS` is true; otherwise OpenAI-chat upstreams reject them
diff --git a/api/handlers/__init__.py b/api/handlers/__init__.py
new file mode 100644
index 00000000..e14c5d70
--- /dev/null
+++ b/api/handlers/__init__.py
@@ -0,0 +1,7 @@
+"""Product-flow handlers for public API routes."""
+
+from .messages import MessagesHandler
+from .responses import ResponsesHandler
+from .token_count import TokenCountHandler
+
+__all__ = ["MessagesHandler", "ResponsesHandler", "TokenCountHandler"]
diff --git a/api/handlers/messages.py b/api/handlers/messages.py
new file mode 100644
index 00000000..a20b47af
--- /dev/null
+++ b/api/handlers/messages.py
@@ -0,0 +1,172 @@
+"""Claude Messages API product flow."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from dataclasses import replace
+
+from loguru import logger
+
+from api.detection import is_safety_classifier_request
+from api.model_router import ModelRouter, RoutedMessagesRequest
+from api.models.anthropic import MessagesRequest
+from api.optimization_handlers import try_optimizations
+from api.provider_execution import ProviderExecutionService, TokenCounter
+from api.request_errors import require_non_empty_messages, unexpected_http_exception
+from api.response_streams import anthropic_sse_streaming_response
+from api.web_tools.egress import WebFetchEgressPolicy
+from api.web_tools.request import (
+ is_web_server_tool_request,
+ openai_chat_upstream_server_tool_error,
+)
+from api.web_tools.streaming import stream_web_server_tool_response
+from config.provider_catalog import PROVIDER_CATALOG
+from config.settings import Settings
+from core.anthropic import get_token_count
+from core.trace import trace_event
+from providers.base import BaseProvider
+from providers.exceptions import InvalidRequestError, ProviderError
+
+ProviderGetter = Callable[[str], BaseProvider]
+MessageIntercept = Callable[[RoutedMessagesRequest], object | None]
+
+_OPENAI_CHAT_UPSTREAM_IDS = frozenset(
+ provider_id
+ for provider_id, descriptor in PROVIDER_CATALOG.items()
+ if descriptor.transport_type == "openai_chat"
+)
+
+
+class MessagesHandler:
+ """Handle Anthropic-compatible Messages requests."""
+
+ def __init__(
+ self,
+ settings: Settings,
+ provider_getter: ProviderGetter,
+ *,
+ model_router: ModelRouter | None = None,
+ token_counter: TokenCounter = get_token_count,
+ provider_execution: ProviderExecutionService | None = None,
+ ) -> None:
+ self._settings = settings
+ self._model_router = model_router or ModelRouter(settings)
+ self._token_counter = token_counter
+ self._provider_execution = provider_execution or ProviderExecutionService(
+ settings,
+ provider_getter,
+ token_counter=token_counter,
+ )
+ self._message_intercepts: tuple[MessageIntercept, ...] = (
+ self._intercept_web_server_tool,
+ self._intercept_local_optimization,
+ )
+
+ def create(self, request_data: MessagesRequest) -> object:
+ """Create an Anthropic-compatible message response."""
+ try:
+ require_non_empty_messages(request_data.messages)
+ routed = self._model_router.resolve_messages_request(request_data)
+ routed = self._apply_message_routing_policies(routed)
+ self._reject_unsupported_server_tools(routed)
+
+ intercepted = self._run_message_intercepts(routed)
+ if intercepted is not None:
+ return intercepted
+
+ logger.debug("No optimization matched, routing to provider")
+ return anthropic_sse_streaming_response(
+ self._provider_execution.stream(
+ routed,
+ wire_api="messages",
+ raw_log_label="FULL_PAYLOAD",
+ raw_log_payload=routed.request.model_dump(),
+ )
+ )
+ except ProviderError:
+ raise
+ except Exception as exc:
+ raise unexpected_http_exception(
+ self._settings, exc, context="CREATE_MESSAGE_ERROR"
+ ) from exc
+
+ def _reject_unsupported_server_tools(self, routed: RoutedMessagesRequest) -> None:
+ if routed.resolved.provider_id not in _OPENAI_CHAT_UPSTREAM_IDS:
+ return
+ tool_err = openai_chat_upstream_server_tool_error(
+ routed.request,
+ web_tools_enabled=self._settings.enable_web_server_tools,
+ )
+ if tool_err is not None:
+ raise InvalidRequestError(tool_err)
+
+ def _apply_message_routing_policies(
+ self, routed: RoutedMessagesRequest
+ ) -> RoutedMessagesRequest:
+ if not is_safety_classifier_request(routed.request):
+ return routed
+ changed = routed.resolved.thinking_enabled
+ trace_event(
+ stage="routing",
+ event="api.optimization.safety_classifier_no_thinking",
+ source="api",
+ model=routed.request.model,
+ changed=changed,
+ )
+ if not changed:
+ return routed
+ return RoutedMessagesRequest(
+ request=routed.request,
+ resolved=replace(routed.resolved, thinking_enabled=False),
+ )
+
+ def _run_message_intercepts(self, routed: RoutedMessagesRequest) -> object | None:
+ for intercept in self._message_intercepts:
+ result = intercept(routed)
+ if result is not None:
+ return result
+ return None
+
+ def _intercept_web_server_tool(
+ self, routed: RoutedMessagesRequest
+ ) -> object | None:
+ if not self._settings.enable_web_server_tools:
+ return None
+ if not is_web_server_tool_request(routed.request):
+ return None
+
+ input_tokens = self._token_counter(
+ routed.request.messages, routed.request.system, routed.request.tools
+ )
+ trace_event(
+ stage="routing",
+ event="api.optimization.web_server_tool",
+ source="api",
+ model=routed.request.model,
+ )
+ egress = WebFetchEgressPolicy(
+ allow_private_network_targets=self._settings.web_fetch_allow_private_networks,
+ allowed_schemes=self._settings.web_fetch_allowed_scheme_set(),
+ )
+ return anthropic_sse_streaming_response(
+ stream_web_server_tool_response(
+ routed.request,
+ input_tokens=input_tokens,
+ web_fetch_egress=egress,
+ verbose_client_errors=self._settings.log_api_error_tracebacks,
+ ),
+ )
+
+ def _intercept_local_optimization(
+ self, routed: RoutedMessagesRequest
+ ) -> object | None:
+ optimized = try_optimizations(routed.request, self._settings)
+ if optimized is None:
+ return None
+ trace_event(
+ stage="routing",
+ event="api.optimization.short_circuit",
+ source="api",
+ model=routed.request.model,
+ )
+ return optimized
diff --git a/api/handlers/responses.py b/api/handlers/responses.py
new file mode 100644
index 00000000..09ad0e60
--- /dev/null
+++ b/api/handlers/responses.py
@@ -0,0 +1,113 @@
+"""OpenAI Responses API product flow for Codex clients."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+from fastapi.responses import JSONResponse
+
+from api.model_router import ModelRouter
+from api.models.anthropic import MessagesRequest
+from api.models.openai_responses import OpenAIResponsesRequest
+from api.provider_execution import ProviderExecutionService
+from api.request_errors import (
+ http_status_for_unexpected_api_exception,
+ log_unexpected_api_exception,
+ require_non_empty_messages,
+)
+from api.response_streams import openai_responses_sse_streaming_response
+from config.settings import Settings
+from core.anthropic import get_user_facing_error_message
+from core.openai_responses import OpenAIResponsesAdapter
+from providers.base import BaseProvider
+from providers.exceptions import InvalidRequestError, ProviderError
+
+ProviderGetter = Callable[[str], BaseProvider]
+
+
+class ResponsesHandler:
+ """Handle streaming OpenAI Responses-compatible requests."""
+
+ def __init__(
+ self,
+ settings: Settings,
+ provider_getter: ProviderGetter,
+ *,
+ model_router: ModelRouter | None = None,
+ responses_adapter: OpenAIResponsesAdapter | None = None,
+ provider_execution: ProviderExecutionService | None = None,
+ ) -> None:
+ self._settings = settings
+ self._model_router = model_router or ModelRouter(settings)
+ self._responses_adapter = responses_adapter or OpenAIResponsesAdapter()
+ self._provider_execution = provider_execution or ProviderExecutionService(
+ settings,
+ provider_getter,
+ )
+
+ async def create(self, request_data: OpenAIResponsesRequest) -> object:
+ """Create a streaming OpenAI Responses-compatible response."""
+ request_payload = request_data.model_dump(mode="json", exclude_none=True)
+ if request_data.stream is False:
+ invalid_request = InvalidRequestError(
+ "FCC /v1/responses supports streaming only; omit stream or set stream=true."
+ )
+ return JSONResponse(
+ status_code=invalid_request.status_code,
+ content=self._responses_adapter.error_payload(
+ message=invalid_request.message,
+ error_type=invalid_request.error_type,
+ ),
+ )
+
+ try:
+ anthropic_payload = self._responses_adapter.to_anthropic_payload(
+ request_payload
+ )
+ response_request = MessagesRequest(**anthropic_payload)
+ require_non_empty_messages(response_request.messages)
+ routed = self._model_router.resolve_messages_request(response_request)
+
+ streamed = self._provider_execution.stream(
+ routed,
+ wire_api="responses",
+ raw_log_label="FULL_RESPONSES_PAYLOAD",
+ raw_log_payload=request_payload,
+ )
+ return openai_responses_sse_streaming_response(
+ self._responses_adapter.iter_sse_from_anthropic(
+ streamed,
+ request_payload,
+ ),
+ headers=self._responses_adapter.sse_headers,
+ )
+ except OpenAIResponsesAdapter.ConversionError as exc:
+ invalid_request = InvalidRequestError(str(exc))
+ return JSONResponse(
+ status_code=invalid_request.status_code,
+ content=self._responses_adapter.error_payload(
+ message=invalid_request.message,
+ error_type=invalid_request.error_type,
+ ),
+ )
+ except ProviderError as exc:
+ return JSONResponse(
+ status_code=exc.status_code,
+ content=self._responses_adapter.error_payload(
+ message=exc.message,
+ error_type=exc.error_type,
+ ),
+ )
+ except Exception as exc:
+ log_unexpected_api_exception(
+ self._settings,
+ exc,
+ context="CREATE_RESPONSE_ERROR",
+ )
+ return JSONResponse(
+ status_code=http_status_for_unexpected_api_exception(exc),
+ content=self._responses_adapter.error_payload(
+ message=get_user_facing_error_message(exc),
+ error_type="api_error",
+ ),
+ )
diff --git a/api/handlers/token_count.py b/api/handlers/token_count.py
new file mode 100644
index 00000000..241a7e5a
--- /dev/null
+++ b/api/handlers/token_count.py
@@ -0,0 +1,80 @@
+"""Anthropic token-count API product flow."""
+
+from __future__ import annotations
+
+import uuid
+
+from fastapi import HTTPException
+from loguru import logger
+
+from api.model_router import ModelRouter
+from api.models.anthropic import TokenCountRequest
+from api.models.responses import TokenCountResponse
+from api.provider_execution import TokenCounter
+from api.request_errors import (
+ http_status_for_unexpected_api_exception,
+ log_unexpected_api_exception,
+ require_non_empty_messages,
+)
+from config.settings import Settings
+from core.anthropic import get_token_count, get_user_facing_error_message
+from core.trace import api_messages_request_snapshot, trace_event
+from providers.exceptions import ProviderError
+
+
+class TokenCountHandler:
+ """Handle Anthropic-compatible token count requests."""
+
+ def __init__(
+ self,
+ settings: Settings,
+ *,
+ model_router: ModelRouter | None = None,
+ token_counter: TokenCounter = get_token_count,
+ ) -> None:
+ self._settings = settings
+ self._model_router = model_router or ModelRouter(settings)
+ self._token_counter = token_counter
+
+ def count(self, request_data: TokenCountRequest) -> TokenCountResponse:
+ """Count tokens for a request after applying configured model routing."""
+ request_id = f"req_{uuid.uuid4().hex[:12]}"
+ with logger.contextualize(request_id=request_id):
+ try:
+ require_non_empty_messages(request_data.messages)
+ routed = self._model_router.resolve_token_count_request(request_data)
+ tokens = self._token_counter(
+ routed.request.messages, routed.request.system, routed.request.tools
+ )
+ trace_event(
+ stage="routing",
+ event="api.route.resolved",
+ source="api",
+ kind="count_tokens",
+ provider_id=routed.resolved.provider_id,
+ provider_model=routed.resolved.provider_model,
+ provider_model_ref=routed.resolved.provider_model_ref,
+ gateway_model=routed.request.model,
+ )
+ trace_event(
+ stage="ingress",
+ event="api.count_tokens.completed",
+ source="api",
+ message_count=len(routed.request.messages),
+ input_tokens=tokens,
+ snapshot=api_messages_request_snapshot(routed.request),
+ )
+ return TokenCountResponse(input_tokens=tokens)
+ except ProviderError:
+ raise
+ except Exception as exc:
+ log_unexpected_api_exception(
+ self._settings,
+ exc,
+ context="COUNT_TOKENS_ERROR",
+ request_id=request_id,
+ )
+ raise HTTPException(
+ status_code=http_status_for_unexpected_api_exception(exc),
+ detail=get_user_facing_error_message(exc),
+ ) from exc
diff --git a/api/provider_execution.py b/api/provider_execution.py
new file mode 100644
index 00000000..4d019eb3
--- /dev/null
+++ b/api/provider_execution.py
@@ -0,0 +1,111 @@
+"""Shared provider execution primitive for API product handlers."""
+
+from __future__ import annotations
+
+import uuid
+from collections.abc import AsyncIterator, Callable
+from typing import Any
+
+from loguru import logger
+
+from config.settings import Settings
+from core.anthropic import get_token_count
+from core.trace import api_messages_request_snapshot, trace_event, traced_async_stream
+from providers.base import BaseProvider
+
+from .model_router import RoutedMessagesRequest
+
+TokenCounter = Callable[[list[Any], str | list[Any] | None, list[Any] | None], int]
+ProviderGetter = Callable[[str], BaseProvider]
+
+
+class ProviderExecutionService:
+ """Resolve a provider and execute one routed Anthropic Messages stream."""
+
+ def __init__(
+ self,
+ settings: Settings,
+ provider_getter: ProviderGetter,
+ *,
+ token_counter: TokenCounter = get_token_count,
+ ) -> None:
+ self._settings = settings
+ self._provider_getter = provider_getter
+ self._token_counter = token_counter
+
+ def stream(
+ self,
+ routed: RoutedMessagesRequest,
+ *,
+ wire_api: str,
+ raw_log_label: str,
+ raw_log_payload: Any,
+ ) -> AsyncIterator[str]:
+ provider = self._provider_getter(routed.resolved.provider_id)
+ provider.preflight_stream(
+ routed.request,
+ thinking_enabled=routed.resolved.thinking_enabled,
+ )
+
+ route_trace: dict[str, Any] = {
+ "stage": "routing",
+ "event": "api.route.resolved",
+ "source": "api",
+ "provider_id": routed.resolved.provider_id,
+ "provider_model": routed.resolved.provider_model,
+ "provider_model_ref": routed.resolved.provider_model_ref,
+ "gateway_model": routed.request.model,
+ "thinking_enabled": routed.resolved.thinking_enabled,
+ }
+ if wire_api == "responses":
+ route_trace["wire_api"] = "responses"
+ trace_event(**route_trace)
+
+ request_id = f"req_{uuid.uuid4().hex[:12]}"
+ trace_event(
+ stage="ingress",
+ event=(
+ "api.responses.request.received"
+ if wire_api == "responses"
+ else "api.request.received"
+ ),
+ source="api",
+ message_count=len(routed.request.messages),
+ snapshot=api_messages_request_snapshot(routed.request),
+ request_id=request_id,
+ )
+
+ if self._settings.log_raw_api_payloads:
+ logger.debug(f"{raw_log_label} [{{}}]: {{}}", request_id, raw_log_payload)
+
+ input_tokens = self._token_counter(
+ routed.request.messages,
+ routed.request.system,
+ routed.request.tools,
+ )
+ return traced_async_stream(
+ provider.stream_response(
+ routed.request,
+ input_tokens=input_tokens,
+ request_id=request_id,
+ thinking_enabled=routed.resolved.thinking_enabled,
+ ),
+ stage="egress",
+ source="api",
+ complete_event=(
+ "api.responses.stream_completed"
+ if wire_api == "responses"
+ else "api.response.stream_completed"
+ ),
+ interrupted_event=(
+ "api.responses.stream_interrupted"
+ if wire_api == "responses"
+ else "api.response.stream_interrupted"
+ ),
+ chunk_event=None,
+ extra={
+ "request_id": request_id,
+ "provider_id": routed.resolved.provider_id,
+ "gateway_model": routed.request.model,
+ },
+ )
diff --git a/api/request_errors.py b/api/request_errors.py
new file mode 100644
index 00000000..cb193bab
--- /dev/null
+++ b/api/request_errors.py
@@ -0,0 +1,58 @@
+"""Shared API request validation and safe error logging."""
+
+from __future__ import annotations
+
+import traceback
+from typing import Any
+
+from fastapi import HTTPException
+from loguru import logger
+
+from config.settings import Settings
+from core.anthropic import get_user_facing_error_message
+from providers.exceptions import InvalidRequestError
+
+
+def require_non_empty_messages(messages: list[Any]) -> None:
+ if not messages:
+ raise InvalidRequestError("messages cannot be empty")
+
+
+def http_status_for_unexpected_api_exception(_exc: BaseException) -> int:
+ return 500
+
+
+def log_unexpected_api_exception(
+ settings: Settings,
+ exc: BaseException,
+ *,
+ context: str,
+ request_id: str | None = None,
+) -> None:
+ """Log API failures without echoing exception text unless opted in."""
+ if settings.log_api_error_tracebacks:
+ if request_id is not None:
+ logger.error("{} request_id={}: {}", context, request_id, exc)
+ else:
+ logger.error("{}: {}", context, exc)
+ logger.error(traceback.format_exc())
+ return
+ if request_id is not None:
+ logger.error(
+ "{} request_id={} exc_type={}",
+ context,
+ request_id,
+ type(exc).__name__,
+ )
+ else:
+ logger.error("{} exc_type={}", context, type(exc).__name__)
+
+
+def unexpected_http_exception(
+ settings: Settings, exc: Exception, *, context: str
+) -> HTTPException:
+ log_unexpected_api_exception(settings, exc, context=context)
+ return HTTPException(
+ status_code=http_status_for_unexpected_api_exception(exc),
+ detail=get_user_facing_error_message(exc),
+ )
diff --git a/api/request_pipeline.py b/api/request_pipeline.py
deleted file mode 100644
index eb0cacd4..00000000
--- a/api/request_pipeline.py
+++ /dev/null
@@ -1,424 +0,0 @@
-"""API request pipeline for routing, intercepts, and provider execution."""
-
-from __future__ import annotations
-
-import traceback
-import uuid
-from collections.abc import AsyncIterator, Callable
-from dataclasses import replace
-from typing import Any
-
-from fastapi import HTTPException
-from fastapi.responses import JSONResponse, StreamingResponse
-from loguru import logger
-
-from config.provider_catalog import PROVIDER_CATALOG
-from config.settings import Settings
-from core.anthropic import get_token_count, get_user_facing_error_message
-from core.anthropic.streaming import ANTHROPIC_SSE_RESPONSE_HEADERS
-from core.openai_responses import OpenAIResponsesAdapter
-from core.trace import api_messages_request_snapshot, trace_event, traced_async_stream
-from providers.base import BaseProvider
-from providers.exceptions import InvalidRequestError, ProviderError
-
-from .detection import is_safety_classifier_request
-from .model_router import ModelRouter, RoutedMessagesRequest
-from .models.anthropic import MessagesRequest, TokenCountRequest
-from .models.openai_responses import OpenAIResponsesRequest
-from .models.responses import TokenCountResponse
-from .optimization_handlers import try_optimizations
-from .web_tools.egress import WebFetchEgressPolicy
-from .web_tools.request import (
- is_web_server_tool_request,
- openai_chat_upstream_server_tool_error,
-)
-from .web_tools.streaming import stream_web_server_tool_response
-
-TokenCounter = Callable[[list[Any], str | list[Any] | None, list[Any] | None], int]
-ProviderGetter = Callable[[str], BaseProvider]
-MessageIntercept = Callable[[RoutedMessagesRequest], object | None]
-
-# Providers that use ``/chat/completions`` + Anthropic-to-OpenAI conversion.
-_OPENAI_CHAT_UPSTREAM_IDS = frozenset(
- provider_id
- for provider_id, descriptor in PROVIDER_CATALOG.items()
- if descriptor.transport_type == "openai_chat"
-)
-
-
-def anthropic_sse_streaming_response(body: AsyncIterator[str]) -> StreamingResponse:
- """Return a streaming response for Anthropic-style SSE streams."""
- return StreamingResponse(
- body,
- media_type="text/event-stream",
- headers=ANTHROPIC_SSE_RESPONSE_HEADERS,
- )
-
-
-def openai_responses_sse_streaming_response(
- body: AsyncIterator[str],
-) -> StreamingResponse:
- """Return a streaming response for OpenAI Responses-style SSE."""
- return StreamingResponse(
- body,
- media_type="text/event-stream",
- headers=OpenAIResponsesAdapter.sse_headers,
- )
-
-
-def _http_status_for_unexpected_pipeline_exception(_exc: BaseException) -> int:
- """HTTP status for uncaught non-provider failures."""
- return 500
-
-
-def _log_unexpected_pipeline_exception(
- settings: Settings,
- exc: BaseException,
- *,
- context: str,
- request_id: str | None = None,
-) -> None:
- """Log API failures without echoing exception text unless opted in."""
- if settings.log_api_error_tracebacks:
- if request_id is not None:
- logger.error("{} request_id={}: {}", context, request_id, exc)
- else:
- logger.error("{}: {}", context, exc)
- logger.error(traceback.format_exc())
- return
- if request_id is not None:
- logger.error(
- "{} request_id={} exc_type={}",
- context,
- request_id,
- type(exc).__name__,
- )
- else:
- logger.error("{} exc_type={}", context, type(exc).__name__)
-
-
-def _require_non_empty_messages(messages: list[Any]) -> None:
- if not messages:
- raise InvalidRequestError("messages cannot be empty")
-
-
-class ApiRequestPipeline:
- """Coordinate API request intercepts, routing, and provider stream execution."""
-
- def __init__(
- self,
- settings: Settings,
- provider_getter: ProviderGetter,
- model_router: ModelRouter | None = None,
- token_counter: TokenCounter = get_token_count,
- responses_adapter: OpenAIResponsesAdapter | None = None,
- ) -> None:
- self._settings = settings
- self._provider_getter = provider_getter
- self._model_router = model_router or ModelRouter(settings)
- self._token_counter = token_counter
- self._responses_adapter = responses_adapter or OpenAIResponsesAdapter()
- self._message_intercepts: tuple[MessageIntercept, ...] = (
- self._intercept_web_server_tool,
- self._intercept_local_optimization,
- )
-
- def create_message(self, request_data: MessagesRequest) -> object:
- """Create an Anthropic-compatible message response."""
- try:
- _require_non_empty_messages(request_data.messages)
- routed = self._model_router.resolve_messages_request(request_data)
- routed = self._apply_message_routing_policies(routed)
- self._reject_unsupported_server_tools(routed)
-
- intercepted = self._run_message_intercepts(routed)
- if intercepted is not None:
- return intercepted
-
- logger.debug("No optimization matched, routing to provider")
- return anthropic_sse_streaming_response(
- self._provider_stream(
- routed,
- wire_api="messages",
- raw_log_label="FULL_PAYLOAD",
- raw_log_payload=routed.request.model_dump(),
- )
- )
- except ProviderError:
- raise
- except Exception as e:
- _log_unexpected_pipeline_exception(
- self._settings, e, context="CREATE_MESSAGE_ERROR"
- )
- raise HTTPException(
- status_code=_http_status_for_unexpected_pipeline_exception(e),
- detail=get_user_facing_error_message(e),
- ) from e
-
- async def create_response(self, request_data: OpenAIResponsesRequest) -> object:
- """Create a streaming OpenAI Responses-compatible response."""
- request_payload = request_data.model_dump(mode="json", exclude_none=True)
- if request_data.stream is False:
- invalid_request = InvalidRequestError(
- "FCC /v1/responses supports streaming only; omit stream or set stream=true."
- )
- return JSONResponse(
- status_code=invalid_request.status_code,
- content=self._responses_adapter.error_payload(
- message=invalid_request.message,
- error_type=invalid_request.error_type,
- ),
- )
-
- try:
- anthropic_payload = self._responses_adapter.to_anthropic_payload(
- request_payload
- )
- response_request = MessagesRequest(**anthropic_payload)
- _require_non_empty_messages(response_request.messages)
- routed = self._model_router.resolve_messages_request(response_request)
- self._reject_unsupported_server_tools(routed)
-
- streamed = self._provider_stream(
- routed,
- wire_api="responses",
- raw_log_label="FULL_RESPONSES_PAYLOAD",
- raw_log_payload=request_payload,
- )
- return openai_responses_sse_streaming_response(
- self._responses_adapter.iter_sse_from_anthropic(
- streamed,
- request_payload,
- )
- )
- except OpenAIResponsesAdapter.ConversionError as exc:
- invalid_request = InvalidRequestError(str(exc))
- return JSONResponse(
- status_code=invalid_request.status_code,
- content=self._responses_adapter.error_payload(
- message=invalid_request.message,
- error_type=invalid_request.error_type,
- ),
- )
- except ProviderError as exc:
- return JSONResponse(
- status_code=exc.status_code,
- content=self._responses_adapter.error_payload(
- message=exc.message,
- error_type=exc.error_type,
- ),
- )
- except Exception as e:
- _log_unexpected_pipeline_exception(
- self._settings,
- e,
- context="CREATE_RESPONSE_ERROR",
- )
- return JSONResponse(
- status_code=_http_status_for_unexpected_pipeline_exception(e),
- content=self._responses_adapter.error_payload(
- message=get_user_facing_error_message(e),
- error_type="api_error",
- ),
- )
-
- def count_tokens(self, request_data: TokenCountRequest) -> TokenCountResponse:
- """Count tokens for a request after applying configured model routing."""
- request_id = f"req_{uuid.uuid4().hex[:12]}"
- with logger.contextualize(request_id=request_id):
- try:
- _require_non_empty_messages(request_data.messages)
- routed = self._model_router.resolve_token_count_request(request_data)
- tokens = self._token_counter(
- routed.request.messages, routed.request.system, routed.request.tools
- )
- trace_event(
- stage="routing",
- event="api.route.resolved",
- source="api",
- kind="count_tokens",
- provider_id=routed.resolved.provider_id,
- provider_model=routed.resolved.provider_model,
- provider_model_ref=routed.resolved.provider_model_ref,
- gateway_model=routed.request.model,
- )
- trace_event(
- stage="ingress",
- event="api.count_tokens.completed",
- source="api",
- message_count=len(routed.request.messages),
- input_tokens=tokens,
- snapshot=api_messages_request_snapshot(routed.request),
- )
- return TokenCountResponse(input_tokens=tokens)
- except ProviderError:
- raise
- except Exception as e:
- _log_unexpected_pipeline_exception(
- self._settings,
- e,
- context="COUNT_TOKENS_ERROR",
- request_id=request_id,
- )
- raise HTTPException(
- status_code=_http_status_for_unexpected_pipeline_exception(e),
- detail=get_user_facing_error_message(e),
- ) from e
-
- def _reject_unsupported_server_tools(self, routed: RoutedMessagesRequest) -> None:
- if routed.resolved.provider_id not in _OPENAI_CHAT_UPSTREAM_IDS:
- return
- tool_err = openai_chat_upstream_server_tool_error(
- routed.request,
- web_tools_enabled=self._settings.enable_web_server_tools,
- )
- if tool_err is not None:
- raise InvalidRequestError(tool_err)
-
- def _apply_message_routing_policies(
- self, routed: RoutedMessagesRequest
- ) -> RoutedMessagesRequest:
- if not is_safety_classifier_request(routed.request):
- return routed
- changed = routed.resolved.thinking_enabled
- trace_event(
- stage="routing",
- event="api.optimization.safety_classifier_no_thinking",
- source="api",
- model=routed.request.model,
- changed=changed,
- )
- if not changed:
- return routed
- return RoutedMessagesRequest(
- request=routed.request,
- resolved=replace(routed.resolved, thinking_enabled=False),
- )
-
- def _run_message_intercepts(self, routed: RoutedMessagesRequest) -> object | None:
- for intercept in self._message_intercepts:
- result = intercept(routed)
- if result is not None:
- return result
- return None
-
- def _intercept_web_server_tool(
- self, routed: RoutedMessagesRequest
- ) -> object | None:
- if not self._settings.enable_web_server_tools:
- return None
- if not is_web_server_tool_request(routed.request):
- return None
-
- input_tokens = self._token_counter(
- routed.request.messages, routed.request.system, routed.request.tools
- )
- trace_event(
- stage="routing",
- event="api.optimization.web_server_tool",
- source="api",
- model=routed.request.model,
- )
- egress = WebFetchEgressPolicy(
- allow_private_network_targets=self._settings.web_fetch_allow_private_networks,
- allowed_schemes=self._settings.web_fetch_allowed_scheme_set(),
- )
- return anthropic_sse_streaming_response(
- stream_web_server_tool_response(
- routed.request,
- input_tokens=input_tokens,
- web_fetch_egress=egress,
- verbose_client_errors=self._settings.log_api_error_tracebacks,
- ),
- )
-
- def _intercept_local_optimization(
- self, routed: RoutedMessagesRequest
- ) -> object | None:
- optimized = try_optimizations(routed.request, self._settings)
- if optimized is None:
- return None
- trace_event(
- stage="routing",
- event="api.optimization.short_circuit",
- source="api",
- model=routed.request.model,
- )
- return optimized
-
- def _provider_stream(
- self,
- routed: RoutedMessagesRequest,
- *,
- wire_api: str,
- raw_log_label: str,
- raw_log_payload: Any,
- ) -> AsyncIterator[str]:
- provider = self._provider_getter(routed.resolved.provider_id)
- provider.preflight_stream(
- routed.request,
- thinking_enabled=routed.resolved.thinking_enabled,
- )
-
- route_trace: dict[str, Any] = {
- "stage": "routing",
- "event": "api.route.resolved",
- "source": "api",
- "provider_id": routed.resolved.provider_id,
- "provider_model": routed.resolved.provider_model,
- "provider_model_ref": routed.resolved.provider_model_ref,
- "gateway_model": routed.request.model,
- "thinking_enabled": routed.resolved.thinking_enabled,
- }
- if wire_api == "responses":
- route_trace["wire_api"] = "responses"
- trace_event(**route_trace)
-
- request_id = f"req_{uuid.uuid4().hex[:12]}"
- trace_event(
- stage="ingress",
- event=(
- "api.responses.request.received"
- if wire_api == "responses"
- else "api.request.received"
- ),
- source="api",
- message_count=len(routed.request.messages),
- snapshot=api_messages_request_snapshot(routed.request),
- request_id=request_id,
- )
-
- if self._settings.log_raw_api_payloads:
- logger.debug(f"{raw_log_label} [{{}}]: {{}}", request_id, raw_log_payload)
-
- input_tokens = self._token_counter(
- routed.request.messages,
- routed.request.system,
- routed.request.tools,
- )
- return traced_async_stream(
- provider.stream_response(
- routed.request,
- input_tokens=input_tokens,
- request_id=request_id,
- thinking_enabled=routed.resolved.thinking_enabled,
- ),
- stage="egress",
- source="api",
- complete_event=(
- "api.responses.stream_completed"
- if wire_api == "responses"
- else "api.response.stream_completed"
- ),
- interrupted_event=(
- "api.responses.stream_interrupted"
- if wire_api == "responses"
- else "api.response.stream_interrupted"
- ),
- chunk_event=None,
- extra={
- "request_id": request_id,
- "provider_id": routed.resolved.provider_id,
- "gateway_model": routed.request.model,
- },
- )
diff --git a/api/response_streams.py b/api/response_streams.py
new file mode 100644
index 00000000..5651880e
--- /dev/null
+++ b/api/response_streams.py
@@ -0,0 +1,31 @@
+"""FastAPI streaming response wrappers for public API wire formats."""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator, Mapping
+
+from fastapi.responses import StreamingResponse
+
+from core.anthropic.streaming import ANTHROPIC_SSE_RESPONSE_HEADERS
+
+
+def anthropic_sse_streaming_response(body: AsyncIterator[str]) -> StreamingResponse:
+ """Return a streaming response for Anthropic-style SSE streams."""
+ return StreamingResponse(
+ body,
+ media_type="text/event-stream",
+ headers=ANTHROPIC_SSE_RESPONSE_HEADERS,
+ )
+
+
+def openai_responses_sse_streaming_response(
+ body: AsyncIterator[str],
+ *,
+ headers: Mapping[str, str],
+) -> StreamingResponse:
+ """Return a streaming response for OpenAI Responses-style SSE."""
+ return StreamingResponse(
+ body,
+ media_type="text/event-stream",
+ headers=dict(headers),
+ )
diff --git a/api/routes.py b/api/routes.py
index 08b7fef6..66b19d78 100644
--- a/api/routes.py
+++ b/api/routes.py
@@ -10,29 +10,51 @@ from providers.registry import ProviderRegistry
from . import dependencies
from .dependencies import get_settings, require_api_key
+from .handlers import MessagesHandler, ResponsesHandler, TokenCountHandler
from .model_catalog import build_models_list_response
from .models.anthropic import MessagesRequest, TokenCountRequest
from .models.openai_responses import OpenAIResponsesRequest
from .models.responses import ModelsListResponse
-from .request_pipeline import ApiRequestPipeline
router = APIRouter()
-def get_request_pipeline(
+def _provider_getter(request: Request, settings: Settings):
+ return lambda provider_type: dependencies.resolve_provider(
+ provider_type, app=request.app, settings=settings
+ )
+
+
+def get_messages_handler(
request: Request,
settings: Settings = Depends(get_settings),
-) -> ApiRequestPipeline:
- """Build the API request pipeline for route handlers."""
- return ApiRequestPipeline(
+) -> MessagesHandler:
+ """Build the Claude Messages product handler for route handlers."""
+ return MessagesHandler(
settings,
- provider_getter=lambda provider_type: dependencies.resolve_provider(
- provider_type, app=request.app, settings=settings
- ),
+ provider_getter=_provider_getter(request, settings),
token_counter=get_token_count,
)
+def get_responses_handler(
+ request: Request,
+ settings: Settings = Depends(get_settings),
+) -> ResponsesHandler:
+ """Build the OpenAI Responses product handler for route handlers."""
+ return ResponsesHandler(
+ settings,
+ provider_getter=_provider_getter(request, settings),
+ )
+
+
+def get_token_count_handler(
+ settings: Settings = Depends(get_settings),
+) -> TokenCountHandler:
+ """Build the token-count product handler for route handlers."""
+ return TokenCountHandler(settings, token_counter=get_token_count)
+
+
def _probe_response(allow: str) -> Response:
"""Return an empty success response for compatibility probes."""
return Response(status_code=204, headers={"Allow": allow})
@@ -44,11 +66,11 @@ def _probe_response(allow: str) -> Response:
@router.post("/v1/messages")
async def create_message(
request_data: MessagesRequest,
- pipeline: ApiRequestPipeline = Depends(get_request_pipeline),
+ handler: MessagesHandler = Depends(get_messages_handler),
_auth=Depends(require_api_key),
):
"""Create a message (always streaming)."""
- return pipeline.create_message(request_data)
+ return handler.create(request_data)
@router.api_route("/v1/messages", methods=["HEAD", "OPTIONS"])
@@ -60,11 +82,11 @@ async def probe_messages(_auth=Depends(require_api_key)):
@router.post("/v1/responses")
async def create_response(
request_data: OpenAIResponsesRequest,
- pipeline: ApiRequestPipeline = Depends(get_request_pipeline),
+ handler: ResponsesHandler = Depends(get_responses_handler),
_auth=Depends(require_api_key),
):
"""Create an OpenAI Responses-compatible response through this proxy."""
- return await pipeline.create_response(request_data)
+ return await handler.create(request_data)
@router.api_route("/v1/responses", methods=["HEAD", "OPTIONS"])
@@ -76,11 +98,11 @@ async def probe_responses(_auth=Depends(require_api_key)):
@router.post("/v1/messages/count_tokens")
async def count_tokens(
request_data: TokenCountRequest,
- pipeline: ApiRequestPipeline = Depends(get_request_pipeline),
+ handler: TokenCountHandler = Depends(get_token_count_handler),
_auth=Depends(require_api_key),
):
"""Count tokens for a request."""
- return pipeline.count_tokens(request_data)
+ return handler.count(request_data)
@router.api_route("/v1/messages/count_tokens", methods=["HEAD", "OPTIONS"])
diff --git a/pyproject.toml b/pyproject.toml
index cfc77f28..d0eb37dd 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "free-claude-code"
-version = "2.3.15"
+version = "2.3.16"
description = "Middleware between Claude Code CLI (Anthropic API) and NVIDIA NIM"
readme = "README.md"
requires-python = ">=3.14.0"
diff --git a/smoke/capabilities.py b/smoke/capabilities.py
index 2c5d080d..25d9805b 100644
--- a/smoke/capabilities.py
+++ b/smoke/capabilities.py
@@ -29,7 +29,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"routes_and_probes",
"anthropic_api_routes",
- "api.request_pipeline.ApiRequestPipeline",
+ "api.routes",
"Anthropic-compatible HTTP requests",
"JSON or Anthropic SSE responses",
"HTTPException or Anthropic error payload",
@@ -51,7 +51,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"client_request_shapes",
"drop_in_claude_code_replacement",
- "api.request_pipeline.ApiRequestPipeline",
+ "api.handlers.messages.MessagesHandler",
"Claude Code, VS Code, and JetBrains shaped requests",
"compatible streaming response",
"provider or validation error in Anthropic shape",
@@ -62,7 +62,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"responses_api",
"drop_in_codex_replacement",
- "api.request_pipeline.ApiRequestPipeline.create_response",
+ "api.handlers.responses.ResponsesHandler",
"OpenAI Responses requests from Codex",
"Responses SSE or JSON response",
"OpenAI-shaped error or conversion error",
@@ -81,7 +81,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"client_extensions",
"vscode_extension",
- "api.request_pipeline.ApiRequestPipeline",
+ "api.handlers.messages.MessagesHandler",
"VS Code beta query/header variants",
"accepted Anthropic-compatible response",
"HTTP error only for auth/provider failures",
@@ -92,7 +92,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"client_extensions",
"intellij_extension",
- "api.request_pipeline.ApiRequestPipeline",
+ "api.handlers.messages.MessagesHandler",
"JetBrains/ACP request variants",
"accepted Anthropic-compatible response",
"HTTP error only for auth/provider failures",
diff --git a/tests/api/test_request_pipeline.py b/tests/api/test_api_handlers.py
similarity index 73%
rename from tests/api/test_request_pipeline.py
rename to tests/api/test_api_handlers.py
index 95834786..beca4ee1 100644
--- a/tests/api/test_request_pipeline.py
+++ b/tests/api/test_api_handlers.py
@@ -7,9 +7,9 @@ from unittest.mock import MagicMock, patch
import pytest
from fastapi.responses import StreamingResponse
-from api.models.anthropic import Message, MessagesRequest
+from api.handlers import MessagesHandler, ResponsesHandler, TokenCountHandler
+from api.models.anthropic import Message, MessagesRequest, TokenCountRequest
from api.models.openai_responses import OpenAIResponsesRequest
-from api.request_pipeline import ApiRequestPipeline
from config.settings import Settings
from providers.base import BaseProvider, ProviderConfig
@@ -79,16 +79,16 @@ def _trace_events(trace_mock: MagicMock, event: str) -> list[dict[str, Any]]:
@pytest.mark.asyncio
-async def test_pipeline_provider_execution_passes_routed_request_and_stream_metadata():
+async def test_messages_handler_passes_routed_request_and_stream_metadata() -> None:
provider = FakeProvider()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ handler = MessagesHandler(Settings(), provider_getter=lambda _: provider)
request = MessagesRequest(
model="nvidia_nim/test-model",
max_tokens=100,
messages=[Message(role="user", content="hi")],
)
- response = pipeline.create_message(request)
+ response = handler.create(request)
assert isinstance(response, StreamingResponse)
body = await _streaming_body_text(response)
@@ -101,9 +101,9 @@ async def test_pipeline_provider_execution_passes_routed_request_and_stream_meta
@pytest.mark.asyncio
-async def test_pipeline_forces_no_thinking_for_safety_classifier_messages():
+async def test_messages_handler_forces_no_thinking_for_safety_classifier() -> None:
provider = FakeProvider()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ handler = MessagesHandler(Settings(), provider_getter=lambda _: provider)
request = MessagesRequest(
model="nvidia_nim/test-model",
max_tokens=100,
@@ -111,8 +111,8 @@ async def test_pipeline_forces_no_thinking_for_safety_classifier_messages():
messages=[Message(role="user", content=_CLASSIFIER_USER)],
)
- with patch("api.request_pipeline.trace_event") as trace_mock:
- response = pipeline.create_message(request)
+ with patch("api.handlers.messages.trace_event") as trace_mock:
+ response = handler.create(request)
assert isinstance(response, StreamingResponse)
await _streaming_body_text(response)
@@ -134,9 +134,9 @@ async def test_pipeline_forces_no_thinking_for_safety_classifier_messages():
@pytest.mark.asyncio
-async def test_pipeline_preserves_thinking_for_non_classifier_messages():
+async def test_messages_handler_preserves_thinking_for_non_classifier() -> None:
provider = FakeProvider()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ handler = MessagesHandler(Settings(), provider_getter=lambda _: provider)
request = MessagesRequest(
model="nvidia_nim/test-model",
max_tokens=100,
@@ -152,8 +152,8 @@ async def test_pipeline_preserves_thinking_for_non_classifier_messages():
],
)
- with patch("api.request_pipeline.trace_event") as trace_mock:
- response = pipeline.create_message(request)
+ with patch("api.handlers.messages.trace_event") as trace_mock:
+ response = handler.create(request)
assert isinstance(response, StreamingResponse)
await _streaming_body_text(response)
@@ -166,9 +166,9 @@ async def test_pipeline_preserves_thinking_for_non_classifier_messages():
@pytest.mark.asyncio
-async def test_pipeline_keeps_existing_no_thinking_for_classifier_messages():
+async def test_messages_handler_keeps_existing_no_thinking_for_classifier() -> None:
provider = FakeProvider()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ handler = MessagesHandler(Settings(), provider_getter=lambda _: provider)
request = MessagesRequest(
model="claude-3-freecc-no-thinking/nvidia_nim/test-model",
max_tokens=100,
@@ -176,8 +176,8 @@ async def test_pipeline_keeps_existing_no_thinking_for_classifier_messages():
messages=[Message(role="user", content=_CLASSIFIER_USER)],
)
- with patch("api.request_pipeline.trace_event") as trace_mock:
- response = pipeline.create_message(request)
+ with patch("api.handlers.messages.trace_event") as trace_mock:
+ response = handler.create(request)
assert isinstance(response, StreamingResponse)
await _streaming_body_text(response)
@@ -196,9 +196,9 @@ async def test_pipeline_keeps_existing_no_thinking_for_classifier_messages():
]
-def test_pipeline_message_optimization_intercepts_before_provider_execution():
+def test_messages_handler_optimization_intercepts_before_provider_execution() -> None:
provider_getter = MagicMock()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=provider_getter)
+ handler = MessagesHandler(Settings(), provider_getter=provider_getter)
request = MessagesRequest(
model="nvidia_nim/test-model",
max_tokens=100,
@@ -206,23 +206,23 @@ def test_pipeline_message_optimization_intercepts_before_provider_execution():
)
optimized = object()
- with patch("api.request_pipeline.try_optimizations", return_value=optimized):
- assert pipeline.create_message(request) is optimized
+ with patch("api.handlers.messages.try_optimizations", return_value=optimized):
+ assert handler.create(request) is optimized
provider_getter.assert_not_called()
@pytest.mark.asyncio
-async def test_pipeline_responses_bypass_message_only_optimizations():
+async def test_responses_handler_bypasses_message_only_optimizations() -> None:
provider = FakeProvider()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ handler = ResponsesHandler(Settings(), provider_getter=lambda _: provider)
with patch(
- "api.request_pipeline.try_optimizations",
+ "api.handlers.messages.try_optimizations",
side_effect=AssertionError("Responses must not use message optimizations"),
):
- response = await pipeline.create_response(
- request_data=OpenAIResponsesRequest(
+ response = await handler.create(
+ OpenAIResponsesRequest(
model="nvidia_nim/test-model",
input="quota check",
)
@@ -235,13 +235,13 @@ async def test_pipeline_responses_bypass_message_only_optimizations():
@pytest.mark.asyncio
-async def test_pipeline_responses_do_not_apply_safety_classifier_policy():
+async def test_responses_handler_does_not_apply_safety_classifier_policy() -> None:
provider = FakeProvider()
- pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ handler = ResponsesHandler(Settings(), provider_getter=lambda _: provider)
- with patch("api.request_pipeline.trace_event") as trace_mock:
- response = await pipeline.create_response(
- request_data=OpenAIResponsesRequest(
+ with patch("api.handlers.messages.trace_event") as trace_mock:
+ response = await handler.create(
+ OpenAIResponsesRequest(
model="nvidia_nim/test-model",
input=_CLASSIFIER_USER,
instructions=_CLASSIFIER_SYSTEM,
@@ -257,3 +257,19 @@ async def test_pipeline_responses_do_not_apply_safety_classifier_policy():
_trace_events(trace_mock, "api.optimization.safety_classifier_no_thinking")
== []
)
+
+
+def test_token_count_handler_routes_and_counts_tokens() -> None:
+ handler = TokenCountHandler(
+ Settings(),
+ token_counter=lambda messages, system, tools: len(messages) + 41,
+ )
+
+ response = handler.count(
+ TokenCountRequest(
+ model="nvidia_nim/test-model",
+ messages=[Message(role="user", content="hi")],
+ )
+ )
+
+ assert response.input_tokens == 42
diff --git a/tests/api/test_openai_responses.py b/tests/api/test_openai_responses.py
index c29c4aaf..909a11ca 100644
--- a/tests/api/test_openai_responses.py
+++ b/tests/api/test_openai_responses.py
@@ -81,7 +81,7 @@ def test_create_response_stream_bypasses_local_message_optimizations() -> None:
with (
patch("api.dependencies.resolve_provider", return_value=provider),
patch(
- "api.request_pipeline.try_optimizations",
+ "api.handlers.messages.try_optimizations",
side_effect=AssertionError("Responses must not use message optimizations"),
),
TestClient(app) as client,
diff --git a/tests/api/test_safe_logging.py b/tests/api/test_safe_logging.py
index f45bbcb6..aa685f07 100644
--- a/tests/api/test_safe_logging.py
+++ b/tests/api/test_safe_logging.py
@@ -7,9 +7,9 @@ from unittest.mock import MagicMock, patch
import pytest
from fastapi import HTTPException
-from api import request_pipeline as pipeline_mod
+from api import provider_execution, request_errors
+from api.handlers import MessagesHandler, TokenCountHandler
from api.models.anthropic import Message, MessagesRequest
-from api.request_pipeline import ApiRequestPipeline
from config.settings import Settings
from core.anthropic import AnthropicStreamLedger
@@ -23,7 +23,7 @@ def test_create_message_skips_full_payload_debug_log_by_default():
yield "event: ping\ndata: {}\n\n"
mock_provider.stream_response = fake_stream
- service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
+ service = MessagesHandler(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
@@ -31,8 +31,8 @@ def test_create_message_skips_full_payload_debug_log_by_default():
messages=[Message(role="user", content="secret-user-text")],
)
- with patch.object(pipeline_mod.logger, "debug") as mock_debug:
- service.create_message(request)
+ with patch.object(provider_execution.logger, "debug") as mock_debug:
+ service.create(request)
full_payload_calls = [
c
@@ -51,15 +51,15 @@ def test_create_message_logs_full_payload_when_opt_in():
yield "event: ping\ndata: {}\n\n"
mock_provider.stream_response = fake_stream
- service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
+ service = MessagesHandler(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
max_tokens=10,
messages=[Message(role="user", content="visible")],
)
- with patch.object(pipeline_mod.logger, "debug") as mock_debug:
- service.create_message(request)
+ with patch.object(provider_execution.logger, "debug") as mock_debug:
+ service.create(request)
keys = [c.args[0] for c in mock_debug.call_args_list if c.args]
assert any(k == "FULL_PAYLOAD [{}]: {}" for k in keys)
@@ -103,7 +103,7 @@ def test_create_message_unexpected_error_default_logs_exclude_exception_text():
raise RuntimeError(secret)
mock_provider.stream_response = stream_boom
- service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
+ service = MessagesHandler(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
max_tokens=10,
@@ -111,10 +111,10 @@ def test_create_message_unexpected_error_default_logs_exclude_exception_text():
)
with (
- patch.object(pipeline_mod.logger, "error") as log_err,
+ patch.object(request_errors.logger, "error") as log_err,
pytest.raises(HTTPException),
):
- service.create_message(request)
+ service.create(request)
blob = _flatten_log_calls(log_err)
assert secret not in blob
@@ -134,7 +134,7 @@ def test_create_message_unexpected_error_always_returns_500():
raise WeirdError("no")
mock_provider.stream_response = stream_boom
- service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
+ service = MessagesHandler(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
max_tokens=10,
@@ -142,7 +142,7 @@ def test_create_message_unexpected_error_always_returns_500():
)
with pytest.raises(HTTPException) as excinfo:
- service.create_message(request)
+ service.create(request)
assert excinfo.value.status_code == 500
@@ -181,9 +181,8 @@ def test_count_tokens_unexpected_error_default_logs_exclude_exception_text():
def boom(*_a, **_kw):
raise ValueError(secret)
- service = ApiRequestPipeline(
+ service = TokenCountHandler(
settings,
- provider_getter=lambda _: MagicMock(),
token_counter=boom,
)
from api.models.anthropic import TokenCountRequest
@@ -194,10 +193,10 @@ def test_count_tokens_unexpected_error_default_logs_exclude_exception_text():
)
with (
- patch.object(pipeline_mod.logger, "error") as log_err,
+ patch.object(request_errors.logger, "error") as log_err,
pytest.raises(HTTPException),
):
- service.count_tokens(req)
+ service.count(req)
blob = _flatten_log_calls(log_err)
assert secret not in blob
diff --git a/tests/api/test_web_server_tools.py b/tests/api/test_web_server_tools.py
index 140b8040..6ca12301 100644
--- a/tests/api/test_web_server_tools.py
+++ b/tests/api/test_web_server_tools.py
@@ -5,9 +5,9 @@ import httpx
import pytest
import api.web_tools.constants as web_tool_constants
+from api.handlers import MessagesHandler
from api.model_router import ModelRouter, ResolvedModel, RoutedMessagesRequest
from api.models.anthropic import Message, MessagesRequest, Tool
-from api.request_pipeline import ApiRequestPipeline
from api.web_tools import egress as web_egress
from api.web_tools.egress import (
WebFetchEgressPolicy,
@@ -112,7 +112,7 @@ def test_service_rejects_forced_server_tool_on_openai_when_disabled(
"""OpenAI Chat upstreams cannot run forced server tools without the local handler."""
settings = Settings()
assert settings.enable_web_server_tools is False
- service = ApiRequestPipeline(
+ service = MessagesHandler(
settings,
provider_getter=lambda _: MagicMock(),
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -130,7 +130,7 @@ def test_service_rejects_forced_server_tool_on_openai_when_disabled(
tool_choice={"type": "tool", "name": "web_search"},
)
with pytest.raises(InvalidRequestError, match="ENABLE_WEB_SERVER_TOOLS"):
- service.create_message(request)
+ service.create(request)
@pytest.mark.parametrize(
@@ -598,7 +598,7 @@ def test_service_rejects_listed_server_tools_on_openai_chat(
provider_id: str,
) -> None:
settings = Settings()
- service = ApiRequestPipeline(
+ service = MessagesHandler(
settings,
provider_getter=lambda _: MagicMock(),
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -610,7 +610,7 @@ def test_service_rejects_listed_server_tools_on_openai_chat(
tools=[Tool(name="web_search", type="web_search_20250305")],
)
with pytest.raises(InvalidRequestError, match="OpenAI Chat upstreams"):
- service.create_message(request)
+ service.create(request)
@pytest.mark.parametrize("provider_id", _ANTHROPIC_MESSAGES_PROVIDER_IDS)
@@ -626,7 +626,7 @@ def test_listed_server_tools_routed_on_anthropic_messages_providers(
mock_provider = MagicMock()
mock_provider.stream_response = fake_stream
- service = ApiRequestPipeline(
+ service = MessagesHandler(
settings,
provider_getter=lambda _: mock_provider,
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -637,7 +637,7 @@ def test_listed_server_tools_routed_on_anthropic_messages_providers(
messages=[Message(role="user", content="q")],
tools=[Tool(name="web_search", type="web_search_20250305")],
)
- service.create_message(request)
+ service.create(request)
mock_provider.preflight_stream.assert_called()
@@ -654,7 +654,7 @@ def test_forced_server_tools_routed_on_anthropic_messages_providers_when_local_d
mock_provider = MagicMock()
mock_provider.stream_response = fake_stream
- service = ApiRequestPipeline(
+ service = MessagesHandler(
settings,
provider_getter=lambda _: mock_provider,
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -666,5 +666,5 @@ def test_forced_server_tools_routed_on_anthropic_messages_providers_when_local_d
tools=[Tool(name="web_search", type="web_search_20250305")],
tool_choice={"type": "tool", "name": "web_search"},
)
- service.create_message(request)
+ service.create(request)
mock_provider.preflight_stream.assert_called()
diff --git a/tests/contracts/test_capability_map.py b/tests/contracts/test_capability_map.py
index 4b6e41a2..ccf3e8f2 100644
--- a/tests/contracts/test_capability_map.py
+++ b/tests/contracts/test_capability_map.py
@@ -46,3 +46,14 @@ def test_capability_contracts_are_decision_complete() -> None:
assert contract.failure.strip(), contract
assert contract.pytest_tests or contract.smoke_tests, contract
+
+
+def test_capability_contract_owners_do_not_reference_removed_request_pipeline() -> None:
+ stale = [
+ contract
+ for contract in CAPABILITY_CONTRACTS
+ if "api.request_pipeline" in contract.owner
+ or "ApiRequestPipeline" in contract.owner
+ ]
+
+ assert stale == []
diff --git a/tests/contracts/test_import_boundaries.py b/tests/contracts/test_import_boundaries.py
index 109a9555..2b224216 100644
--- a/tests/contracts/test_import_boundaries.py
+++ b/tests/contracts/test_import_boundaries.py
@@ -203,8 +203,11 @@ def test_anthropic_stream_engine_owns_provider_stream_state() -> None:
def test_openai_responses_uses_adapter_boundary() -> None:
repo_root = Path(__file__).resolve().parents[2]
responses_root = repo_root / "core" / "openai_responses"
+ api_root = repo_root / "api"
+ handlers_root = api_root / "handlers"
assert not (repo_root / "api" / "services.py").exists()
+ assert not (api_root / "request_pipeline.py").exists()
assert not (responses_root / "conversion.py").exists()
assert not (responses_root / "sse.py").exists()
assert not (responses_root / "output.py").exists()
@@ -223,12 +226,16 @@ def test_openai_responses_uses_adapter_boundary() -> None:
}:
assert (responses_root / filename).exists()
- pipeline_text = (repo_root / "api" / "request_pipeline.py").read_text(
- encoding="utf-8"
+ responses_handler = handlers_root / "responses.py"
+ responses_handler_text = responses_handler.read_text(encoding="utf-8")
+ assert (
+ "from core.openai_responses import OpenAIResponsesAdapter"
+ in responses_handler_text
)
- assert "from core.openai_responses import OpenAIResponsesAdapter" in pipeline_text
routes_text = (repo_root / "api" / "routes.py").read_text(encoding="utf-8")
- assert "ApiRequestPipeline" in routes_text
+ assert "ApiRequestPipeline" not in routes_text
+ assert "request_pipeline" not in routes_text
+ assert "from .handlers import" in routes_text
assert "api.services" not in routes_text
for old_helper in {
"responses_request_to_anthropic_payload",
@@ -237,7 +244,7 @@ def test_openai_responses_uses_adapter_boundary() -> None:
"collect_openai_response_from_anthropic_sse",
"iter_message_response_as_openai_responses",
}:
- assert old_helper not in pipeline_text
+ assert old_helper not in responses_handler_text
offenders: list[str] = []
for path in (repo_root / "api").rglob("*.py"):
@@ -247,6 +254,27 @@ def test_openai_responses_uses_adapter_boundary() -> None:
offenders.append(f"{rel}: {imported}")
assert sorted(offenders) == []
+ adapter_importers: list[str] = []
+ for path in (repo_root / "api").rglob("*.py"):
+ imports = set(_imports_from(path, repo_root))
+ if "core.openai_responses" in imports:
+ adapter_importers.append(path.relative_to(repo_root).as_posix())
+ assert sorted(adapter_importers) == ["api/handlers/responses.py"]
+
+ response_handler_imports = set(_imports_from(responses_handler, repo_root))
+ for forbidden in {
+ "api.optimization_handlers",
+ "api.detection",
+ "api.web_tools",
+ }:
+ assert forbidden not in response_handler_imports
+
+ provider_execution_text = (api_root / "provider_execution.py").read_text(
+ encoding="utf-8"
+ )
+ assert "StreamingResponse" not in provider_execution_text
+ assert "OpenAIResponsesAdapter" not in provider_execution_text
+
adapter_text = (responses_root / "adapter.py").read_text(encoding="utf-8")
for deleted_api in {
"from_anthropic_message",
diff --git a/uv.lock b/uv.lock
index e73cf773..a0b33736 100644
--- a/uv.lock
+++ b/uv.lock
@@ -561,7 +561,7 @@ wheels = [
[[package]]
name = "free-claude-code"
-version = "2.3.15"
+version = "2.3.16"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },