Skip to content

Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

Python

Refactor API request pipeline (#851)

Ali Khokharb2af3f63d20a
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 9432ba3f..d377b158 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -32,9 +32,9 @@ flowchart LR
Bots[Discord or Telegram Bots] --> Messaging[Messaging Bridge]
Messaging --> ClientCLI[Managed Client CLI Sessions]
ClientCLI --> ProxyAPI
- ProxyAPI --> Service[ClaudeProxyService]
- Service --> Router[ModelRouter]
- Service --> Providers[ProviderRegistry]
+ ProxyAPI --> Pipeline[ApiRequestPipeline]
+ Pipeline --> Router[ModelRouter]
+ Pipeline --> Providers[ProviderRegistry]
Providers --> OpenAIChat[OpenAI Chat Providers]
Providers --> NativeAnthropic[Anthropic Messages Providers]
```
@@ -43,8 +43,8 @@ flowchart LR
The installable wheel packages are declared in [pyproject.toml](pyproject.toml):
-- [api/](api/) owns the FastAPI app, route handlers, request orchestration,
- admin APIs, local optimizations, and server-tool handling.
+- [api/](api/) owns the FastAPI app, route handlers, request pipeline, model
+ catalog, admin APIs, local optimizations, and server-tool handling.
- [cli/](cli/) owns console entrypoints, client CLI launchers, process/session
management, and client adapter contracts.
- [config/](config/) owns settings, provider metadata, filesystem paths,
@@ -105,10 +105,10 @@ The current package boundaries are intentional, but several modules still carry
large orchestration responsibilities. Treat these as refactor targets, not as
new places to add unrelated behavior:
-- [api/services.py](api/services.py) coordinates routing, optimizations, local
- server tools, provider execution, and Responses adaptation. Future changes
- should keep route handlers thin and move separable use-case logic behind small
- helpers.
+- [api/request_pipeline.py](api/request_pipeline.py) coordinates routing,
+ message-only intercepts, provider execution, token counting, and Responses
+ adaptation. Keep route handlers thin and keep new pre-provider behavior as
+ explicit pipeline intercepts.
- [providers/transports/](providers/transports/) owns provider transport
families. The OpenAI-chat and native Anthropic transport packages split thin
transport bases from per-request stream runners, recovery event construction,
@@ -221,38 +221,39 @@ proxy auth is disabled. Otherwise the token may be supplied through `x-api-key`,
`Authorization: Bearer ...`, or `anthropic-auth-token`. Comparisons use
constant-time matching.
-`ClaudeProxyService` in [api/services.py](api/services.py) coordinates request
-handling. It validates non-empty messages, resolves models, handles local server
-tools and optimizations, resolves a provider, preflights the upstream request,
-then streams Anthropic SSE back to the caller.
+`ApiRequestPipeline` in
+[api/request_pipeline.py](api/request_pipeline.py) coordinates request handling.
+It validates non-empty messages, resolves models, runs explicit message-only
+intercepts for local server tools and optimizations, resolves a provider,
+preflights the upstream request, then streams Anthropic SSE back to the caller.
```mermaid
sequenceDiagram
participant Client
participant Route as FastAPIRoute
- participant Service as ClaudeProxyService
+ participant Pipeline as ApiRequestPipeline
participant Router as ModelRouter
participant Registry as ProviderRegistry
participant Provider
Client->>Route: POST /v1/messages
Route->>Route: require_api_key
- Route->>Service: create_message
- Service->>Router: resolve model and thinking
- Service->>Service: server tools or optimizations
- Service->>Registry: resolve provider
+ Route->>Pipeline: create_message
+ Pipeline->>Router: resolve model and thinking
+ Pipeline->>Pipeline: server tools or optimizations
+ Pipeline->>Registry: resolve provider
Registry->>Provider: cached or new provider
- Service->>Provider: preflight_stream
- Service->>Provider: stream_response
+ Pipeline->>Provider: preflight_stream
+ Pipeline->>Provider: stream_response
Provider-->>Client: Anthropic SSE events
```
-OpenAI Responses uses the same core route. `create_response()` delegates
-Responses protocol work to the `OpenAIResponsesAdapter` in
+OpenAI Responses uses the same provider execution path without message-only
+intercepts. `create_response()` delegates Responses protocol work to the
+`OpenAIResponsesAdapter` in
[core/openai_responses/adapter.py](core/openai_responses/adapter.py). The adapter
-converts the Responses payload into an Anthropic Messages payload, lets the
-service reuse `create_message()`, then converts the final response or SSE stream
-back to Responses format.
+converts the Responses payload into an Anthropic Messages payload before
+provider execution, then converts Anthropic SSE back to Responses SSE.
## Model Routing
diff --git a/api/model_catalog.py b/api/model_catalog.py
new file mode 100644
index 00000000..68b07153
--- /dev/null
+++ b/api/model_catalog.py
@@ -0,0 +1,133 @@
+"""Model-list response construction for Claude-compatible clients."""
+
+from __future__ import annotations
+
+from config.settings import Settings
+from providers.registry import ProviderRegistry
+
+from .gateway_model_ids import gateway_model_id, no_thinking_gateway_model_id
+from .models.responses import ModelResponse, ModelsListResponse
+
+DISCOVERED_MODEL_CREATED_AT = "1970-01-01T00:00:00Z"
+
+
+SUPPORTED_CLAUDE_MODELS = [
+ ModelResponse(
+ id="claude-opus-4-20250514",
+ display_name="Claude Opus 4",
+ created_at="2025-05-14T00:00:00Z",
+ ),
+ ModelResponse(
+ id="claude-sonnet-4-20250514",
+ display_name="Claude Sonnet 4",
+ created_at="2025-05-14T00:00:00Z",
+ ),
+ ModelResponse(
+ id="claude-haiku-4-20250514",
+ display_name="Claude Haiku 4",
+ created_at="2025-05-14T00:00:00Z",
+ ),
+ ModelResponse(
+ id="claude-3-opus-20240229",
+ display_name="Claude 3 Opus",
+ created_at="2024-02-29T00:00:00Z",
+ ),
+ ModelResponse(
+ id="claude-3-5-sonnet-20241022",
+ display_name="Claude 3.5 Sonnet",
+ created_at="2024-10-22T00:00:00Z",
+ ),
+ ModelResponse(
+ id="claude-3-haiku-20240307",
+ display_name="Claude 3 Haiku",
+ created_at="2024-03-07T00:00:00Z",
+ ),
+ ModelResponse(
+ id="claude-3-5-haiku-20241022",
+ display_name="Claude 3.5 Haiku",
+ created_at="2024-10-22T00:00:00Z",
+ ),
+]
+
+
+def build_models_list_response(
+ settings: Settings, provider_registry: ProviderRegistry | None
+) -> ModelsListResponse:
+ """Return configured, cached, and compatibility model ids."""
+ models: list[ModelResponse] = []
+ seen: set[str] = set()
+
+ for ref in settings.configured_chat_model_refs():
+ supports_thinking = None
+ if provider_registry is not None:
+ supports_thinking = provider_registry.cached_model_supports_thinking(
+ ref.provider_id, ref.model_id
+ )
+ _append_provider_model_variants(
+ models,
+ seen,
+ ref.model_ref,
+ supports_thinking=supports_thinking,
+ )
+
+ if provider_registry is not None:
+ for model_info in provider_registry.cached_prefixed_model_infos():
+ _append_provider_model_variants(
+ models,
+ seen,
+ model_info.model_id,
+ supports_thinking=model_info.supports_thinking,
+ )
+
+ for model in SUPPORTED_CLAUDE_MODELS:
+ _append_unique_model(models, seen, model)
+
+ return ModelsListResponse(
+ data=models,
+ first_id=models[0].id if models else None,
+ has_more=False,
+ last_id=models[-1].id if models else None,
+ )
+
+
+def _discovered_model_response(model_id: str, *, display_name: str) -> ModelResponse:
+ return ModelResponse(
+ id=model_id,
+ display_name=display_name,
+ created_at=DISCOVERED_MODEL_CREATED_AT,
+ )
+
+
+def _append_unique_model(
+ models: list[ModelResponse], seen: set[str], model: ModelResponse
+) -> None:
+ if model.id in seen:
+ return
+ seen.add(model.id)
+ models.append(model)
+
+
+def _append_provider_model_variants(
+ models: list[ModelResponse],
+ seen: set[str],
+ provider_model_ref: str,
+ *,
+ supports_thinking: bool | None = None,
+) -> None:
+ if supports_thinking is not False:
+ _append_unique_model(
+ models,
+ seen,
+ _discovered_model_response(
+ gateway_model_id(provider_model_ref),
+ display_name=provider_model_ref,
+ ),
+ )
+ _append_unique_model(
+ models,
+ seen,
+ _discovered_model_response(
+ no_thinking_gateway_model_id(provider_model_ref),
+ display_name=f"{provider_model_ref} (no thinking)",
+ ),
+ )
diff --git a/api/services.py b/api/request_pipeline.py
similarity index 52%
rename from api/services.py
rename to api/request_pipeline.py
index fd0bc824..600449db 100644
--- a/api/services.py
+++ b/api/request_pipeline.py
@@ -1,4 +1,4 @@
-"""Application services for the Claude-compatible API."""
+"""API request pipeline for routing, intercepts, and provider execution."""
from __future__ import annotations
@@ -20,7 +20,7 @@ from core.trace import api_messages_request_snapshot, trace_event, traced_async_
from providers.base import BaseProvider
from providers.exceptions import InvalidRequestError, ProviderError
-from .model_router import ModelRouter
+from .model_router import ModelRouter, RoutedMessagesRequest
from .models.anthropic import MessagesRequest, TokenCountRequest
from .models.openai_responses import OpenAIResponsesRequest
from .models.responses import TokenCountResponse
@@ -33,10 +33,10 @@ from .web_tools.request import (
from .web_tools.streaming import stream_web_server_tool_response
TokenCounter = Callable[[list[Any], str | list[Any] | None, list[Any] | None], int]
-
ProviderGetter = Callable[[str], BaseProvider]
+MessageIntercept = Callable[[RoutedMessagesRequest], object | None]
-# Providers that use ``/chat/completions`` + Anthropic-to-OpenAI conversion (not native Messages).
+# Providers that use ``/chat/completions`` + Anthropic-to-OpenAI conversion.
_OPENAI_CHAT_UPSTREAM_IDS = frozenset(
provider_id
for provider_id, descriptor in PROVIDER_CATALOG.items()
@@ -44,10 +44,8 @@ _OPENAI_CHAT_UPSTREAM_IDS = frozenset(
)
-def anthropic_sse_streaming_response(
- body: AsyncIterator[str],
-) -> StreamingResponse:
- """Return a :class:`StreamingResponse` for Anthropic-style SSE streams."""
+def anthropic_sse_streaming_response(body: AsyncIterator[str]) -> StreamingResponse:
+ """Return a streaming response for Anthropic-style SSE streams."""
return StreamingResponse(
body,
media_type="text/event-stream",
@@ -66,19 +64,19 @@ def openai_responses_sse_streaming_response(
)
-def _http_status_for_unexpected_service_exception(_exc: BaseException) -> int:
- """HTTP status for uncaught non-provider failures (stable client contract)."""
+def _http_status_for_unexpected_pipeline_exception(_exc: BaseException) -> int:
+ """HTTP status for uncaught non-provider failures."""
return 500
-def _log_unexpected_service_exception(
+def _log_unexpected_pipeline_exception(
settings: Settings,
exc: BaseException,
*,
context: str,
request_id: str | None = None,
) -> None:
- """Log service-layer failures without echoing exception text unless opted in."""
+ """Log API failures without echoing exception text unless opted in."""
if settings.log_api_error_tracebacks:
if request_id is not None:
logger.error("{} request_id={}: {}", context, request_id, exc)
@@ -102,8 +100,8 @@ def _require_non_empty_messages(messages: list[Any]) -> None:
raise InvalidRequestError("messages cannot be empty")
-class ClaudeProxyService:
- """Coordinate request optimization, model routing, token count, and providers."""
+class ApiRequestPipeline:
+ """Coordinate API request intercepts, routing, and provider stream execution."""
def __init__(
self,
@@ -112,135 +110,50 @@ class ClaudeProxyService:
model_router: ModelRouter | None = None,
token_counter: TokenCounter = get_token_count,
responses_adapter: OpenAIResponsesAdapter | None = None,
- ):
+ ) -> None:
self._settings = settings
self._provider_getter = provider_getter
self._model_router = model_router or ModelRouter(settings)
self._token_counter = token_counter
self._responses_adapter = responses_adapter or OpenAIResponsesAdapter()
+ self._message_intercepts: tuple[MessageIntercept, ...] = (
+ self._intercept_web_server_tool,
+ self._intercept_local_optimization,
+ )
def create_message(self, request_data: MessagesRequest) -> object:
- """Create a message response or streaming response."""
+ """Create an Anthropic-compatible message response."""
try:
_require_non_empty_messages(request_data.messages)
-
routed = self._model_router.resolve_messages_request(request_data)
- if routed.resolved.provider_id in _OPENAI_CHAT_UPSTREAM_IDS:
- tool_err = openai_chat_upstream_server_tool_error(
- routed.request,
- web_tools_enabled=self._settings.enable_web_server_tools,
- )
- if tool_err is not None:
- raise InvalidRequestError(tool_err)
+ self._reject_unsupported_server_tools(routed)
- if self._settings.enable_web_server_tools and is_web_server_tool_request(
- routed.request
- ):
- input_tokens = self._token_counter(
- routed.request.messages, routed.request.system, routed.request.tools
- )
- trace_event(
- stage="routing",
- event="api.optimization.web_server_tool",
- source="api",
- model=routed.request.model,
- )
- egress = WebFetchEgressPolicy(
- allow_private_network_targets=self._settings.web_fetch_allow_private_networks,
- allowed_schemes=self._settings.web_fetch_allowed_scheme_set(),
- )
- return anthropic_sse_streaming_response(
- stream_web_server_tool_response(
- routed.request,
- input_tokens=input_tokens,
- web_fetch_egress=egress,
- verbose_client_errors=self._settings.log_api_error_tracebacks,
- ),
- )
+ intercepted = self._run_message_intercepts(routed)
+ if intercepted is not None:
+ return intercepted
- optimized = try_optimizations(routed.request, self._settings)
- if optimized is not None:
- trace_event(
- stage="routing",
- event="api.optimization.short_circuit",
- source="api",
- model=routed.request.model,
- )
- return optimized
logger.debug("No optimization matched, routing to provider")
-
- provider = self._provider_getter(routed.resolved.provider_id)
- provider.preflight_stream(
- routed.request,
- thinking_enabled=routed.resolved.thinking_enabled,
- )
-
- trace_event(
- stage="routing",
- event="api.route.resolved",
- source="api",
- provider_id=routed.resolved.provider_id,
- provider_model=routed.resolved.provider_model,
- provider_model_ref=routed.resolved.provider_model_ref,
- gateway_model=routed.request.model,
- thinking_enabled=routed.resolved.thinking_enabled,
- )
-
- request_id = f"req_{uuid.uuid4().hex[:12]}"
- trace_event(
- stage="ingress",
- event="api.request.received",
- source="api",
- message_count=len(routed.request.messages),
- snapshot=api_messages_request_snapshot(routed.request),
- request_id=request_id,
- )
-
- if self._settings.log_raw_api_payloads:
- logger.debug(
- "FULL_PAYLOAD [{}]: {}", request_id, routed.request.model_dump()
+ return anthropic_sse_streaming_response(
+ self._provider_stream(
+ routed,
+ wire_api="messages",
+ raw_log_label="FULL_PAYLOAD",
+ raw_log_payload=routed.request.model_dump(),
)
-
- input_tokens = self._token_counter(
- routed.request.messages,
- routed.request.system,
- routed.request.tools,
)
-
- streamed = traced_async_stream(
- provider.stream_response(
- routed.request,
- input_tokens=input_tokens,
- request_id=request_id,
- thinking_enabled=routed.resolved.thinking_enabled,
- ),
- stage="egress",
- source="api",
- complete_event="api.response.stream_completed",
- interrupted_event="api.response.stream_interrupted",
- chunk_event=None,
- extra={
- "request_id": request_id,
- "provider_id": routed.resolved.provider_id,
- "gateway_model": routed.request.model,
- },
- )
- return anthropic_sse_streaming_response(streamed)
-
except ProviderError:
raise
except Exception as e:
- _log_unexpected_service_exception(
+ _log_unexpected_pipeline_exception(
self._settings, e, context="CREATE_MESSAGE_ERROR"
)
raise HTTPException(
- status_code=_http_status_for_unexpected_service_exception(e),
+ status_code=_http_status_for_unexpected_pipeline_exception(e),
detail=get_user_facing_error_message(e),
) from e
async def create_response(self, request_data: OpenAIResponsesRequest) -> object:
- """Create an OpenAI Responses-compatible response through the provider router."""
-
+ """Create a streaming OpenAI Responses-compatible response."""
request_payload = request_data.model_dump(mode="json", exclude_none=True)
if request_data.stream is False:
invalid_request = InvalidRequestError(
@@ -261,73 +174,13 @@ class ClaudeProxyService:
response_request = MessagesRequest(**anthropic_payload)
_require_non_empty_messages(response_request.messages)
routed = self._model_router.resolve_messages_request(response_request)
+ self._reject_unsupported_server_tools(routed)
- if routed.resolved.provider_id in _OPENAI_CHAT_UPSTREAM_IDS:
- tool_err = openai_chat_upstream_server_tool_error(
- routed.request,
- web_tools_enabled=self._settings.enable_web_server_tools,
- )
- if tool_err is not None:
- raise InvalidRequestError(tool_err)
-
- provider = self._provider_getter(routed.resolved.provider_id)
- provider.preflight_stream(
- routed.request,
- thinking_enabled=routed.resolved.thinking_enabled,
- )
-
- trace_event(
- stage="routing",
- event="api.route.resolved",
- source="api",
- provider_id=routed.resolved.provider_id,
- provider_model=routed.resolved.provider_model,
- provider_model_ref=routed.resolved.provider_model_ref,
- gateway_model=routed.request.model,
- thinking_enabled=routed.resolved.thinking_enabled,
+ streamed = self._provider_stream(
+ routed,
wire_api="responses",
- )
-
- request_id = f"req_{uuid.uuid4().hex[:12]}"
- trace_event(
- stage="ingress",
- event="api.responses.request.received",
- source="api",
- message_count=len(routed.request.messages),
- snapshot=api_messages_request_snapshot(routed.request),
- request_id=request_id,
- )
-
- if self._settings.log_raw_api_payloads:
- logger.debug(
- "FULL_RESPONSES_PAYLOAD [{}]: {}",
- request_id,
- request_payload,
- )
-
- input_tokens = self._token_counter(
- routed.request.messages,
- routed.request.system,
- routed.request.tools,
- )
-
- streamed = traced_async_stream(
- provider.stream_response(
- routed.request,
- input_tokens=input_tokens,
- request_id=request_id,
- thinking_enabled=routed.resolved.thinking_enabled,
- ),
- stage="egress",
- source="api",
- complete_event="api.responses.stream_completed",
- interrupted_event="api.responses.stream_interrupted",
- chunk_event=None,
- extra={
- "request_id": request_id,
- "provider_id": routed.resolved.provider_id,
- "gateway_model": routed.request.model,
- },
+ raw_log_label="FULL_RESPONSES_PAYLOAD",
+ raw_log_payload=request_payload,
)
return openai_responses_sse_streaming_response(
self._responses_adapter.iter_sse_from_anthropic(
@@ -353,13 +206,13 @@ class ClaudeProxyService:
),
)
except Exception as e:
- _log_unexpected_service_exception(
+ _log_unexpected_pipeline_exception(
self._settings,
e,
context="CREATE_RESPONSE_ERROR",
)
return JSONResponse(
- status_code=_http_status_for_unexpected_service_exception(e),
+ status_code=_http_status_for_unexpected_pipeline_exception(e),
content=self._responses_adapter.error_payload(
message=get_user_facing_error_message(e),
error_type="api_error",
@@ -398,13 +251,151 @@ class ClaudeProxyService:
except ProviderError:
raise
except Exception as e:
- _log_unexpected_service_exception(
+ _log_unexpected_pipeline_exception(
self._settings,
e,
context="COUNT_TOKENS_ERROR",
request_id=request_id,
)
raise HTTPException(
- status_code=_http_status_for_unexpected_service_exception(e),
+ status_code=_http_status_for_unexpected_pipeline_exception(e),
detail=get_user_facing_error_message(e),
) from e
+
+ def _reject_unsupported_server_tools(self, routed: RoutedMessagesRequest) -> None:
+ if routed.resolved.provider_id not in _OPENAI_CHAT_UPSTREAM_IDS:
+ return
+ tool_err = openai_chat_upstream_server_tool_error(
+ routed.request,
+ web_tools_enabled=self._settings.enable_web_server_tools,
+ )
+ if tool_err is not None:
+ raise InvalidRequestError(tool_err)
+
+ def _run_message_intercepts(self, routed: RoutedMessagesRequest) -> object | None:
+ for intercept in self._message_intercepts:
+ result = intercept(routed)
+ if result is not None:
+ return result
+ return None
+
+ def _intercept_web_server_tool(
+ self, routed: RoutedMessagesRequest
+ ) -> object | None:
+ if not self._settings.enable_web_server_tools:
+ return None
+ if not is_web_server_tool_request(routed.request):
+ return None
+
+ input_tokens = self._token_counter(
+ routed.request.messages, routed.request.system, routed.request.tools
+ )
+ trace_event(
+ stage="routing",
+ event="api.optimization.web_server_tool",
+ source="api",
+ model=routed.request.model,
+ )
+ egress = WebFetchEgressPolicy(
+ allow_private_network_targets=self._settings.web_fetch_allow_private_networks,
+ allowed_schemes=self._settings.web_fetch_allowed_scheme_set(),
+ )
+ return anthropic_sse_streaming_response(
+ stream_web_server_tool_response(
+ routed.request,
+ input_tokens=input_tokens,
+ web_fetch_egress=egress,
+ verbose_client_errors=self._settings.log_api_error_tracebacks,
+ ),
+ )
+
+ def _intercept_local_optimization(
+ self, routed: RoutedMessagesRequest
+ ) -> object | None:
+ optimized = try_optimizations(routed.request, self._settings)
+ if optimized is None:
+ return None
+ trace_event(
+ stage="routing",
+ event="api.optimization.short_circuit",
+ source="api",
+ model=routed.request.model,
+ )
+ return optimized
+
+ def _provider_stream(
+ self,
+ routed: RoutedMessagesRequest,
+ *,
+ wire_api: str,
+ raw_log_label: str,
+ raw_log_payload: Any,
+ ) -> AsyncIterator[str]:
+ provider = self._provider_getter(routed.resolved.provider_id)
+ provider.preflight_stream(
+ routed.request,
+ thinking_enabled=routed.resolved.thinking_enabled,
+ )
+
+ route_trace: dict[str, Any] = {
+ "stage": "routing",
+ "event": "api.route.resolved",
+ "source": "api",
+ "provider_id": routed.resolved.provider_id,
+ "provider_model": routed.resolved.provider_model,
+ "provider_model_ref": routed.resolved.provider_model_ref,
+ "gateway_model": routed.request.model,
+ "thinking_enabled": routed.resolved.thinking_enabled,
+ }
+ if wire_api == "responses":
+ route_trace["wire_api"] = "responses"
+ trace_event(**route_trace)
+
+ request_id = f"req_{uuid.uuid4().hex[:12]}"
+ trace_event(
+ stage="ingress",
+ event=(
+ "api.responses.request.received"
+ if wire_api == "responses"
+ else "api.request.received"
+ ),
+ source="api",
+ message_count=len(routed.request.messages),
+ snapshot=api_messages_request_snapshot(routed.request),
+ request_id=request_id,
+ )
+
+ if self._settings.log_raw_api_payloads:
+ logger.debug(f"{raw_log_label} [{{}}]: {{}}", request_id, raw_log_payload)
+
+ input_tokens = self._token_counter(
+ routed.request.messages,
+ routed.request.system,
+ routed.request.tools,
+ )
+ return traced_async_stream(
+ provider.stream_response(
+ routed.request,
+ input_tokens=input_tokens,
+ request_id=request_id,
+ thinking_enabled=routed.resolved.thinking_enabled,
+ ),
+ stage="egress",
+ source="api",
+ complete_event=(
+ "api.responses.stream_completed"
+ if wire_api == "responses"
+ else "api.response.stream_completed"
+ ),
+ interrupted_event=(
+ "api.responses.stream_interrupted"
+ if wire_api == "responses"
+ else "api.response.stream_interrupted"
+ ),
+ chunk_event=None,
+ extra={
+ "request_id": request_id,
+ "provider_id": routed.resolved.provider_id,
+ "gateway_model": routed.request.model,
+ },
+ )
diff --git a/api/routes.py b/api/routes.py
index e7ec9487..73581b0a 100644
--- a/api/routes.py
+++ b/api/routes.py
@@ -10,62 +10,21 @@ from providers.registry import ProviderRegistry
from . import dependencies
from .dependencies import get_settings, require_api_key
-from .gateway_model_ids import gateway_model_id, no_thinking_gateway_model_id
+from .model_catalog import build_models_list_response
from .models.anthropic import MessagesRequest, TokenCountRequest
from .models.openai_responses import OpenAIResponsesRequest
-from .models.responses import ModelResponse, ModelsListResponse
-from .services import ClaudeProxyService
+from .models.responses import ModelsListResponse
+from .request_pipeline import ApiRequestPipeline
router = APIRouter()
-DISCOVERED_MODEL_CREATED_AT = "1970-01-01T00:00:00Z"
-
-
-SUPPORTED_CLAUDE_MODELS = [
- ModelResponse(
- id="claude-opus-4-20250514",
- display_name="Claude Opus 4",
- created_at="2025-05-14T00:00:00Z",
- ),
- ModelResponse(
- id="claude-sonnet-4-20250514",
- display_name="Claude Sonnet 4",
- created_at="2025-05-14T00:00:00Z",
- ),
- ModelResponse(
- id="claude-haiku-4-20250514",
- display_name="Claude Haiku 4",
- created_at="2025-05-14T00:00:00Z",
- ),
- ModelResponse(
- id="claude-3-opus-20240229",
- display_name="Claude 3 Opus",
- created_at="2024-02-29T00:00:00Z",
- ),
- ModelResponse(
- id="claude-3-5-sonnet-20241022",
- display_name="Claude 3.5 Sonnet",
- created_at="2024-10-22T00:00:00Z",
- ),
- ModelResponse(
- id="claude-3-haiku-20240307",
- display_name="Claude 3 Haiku",
- created_at="2024-03-07T00:00:00Z",
- ),
- ModelResponse(
- id="claude-3-5-haiku-20241022",
- display_name="Claude 3.5 Haiku",
- created_at="2024-10-22T00:00:00Z",
- ),
-]
-
-
-def get_proxy_service(
+
+def get_request_pipeline(
request: Request,
settings: Settings = Depends(get_settings),
-) -> ClaudeProxyService:
- """Build the request service for route handlers."""
- return ClaudeProxyService(
+) -> ApiRequestPipeline:
+ """Build the API request pipeline for route handlers."""
+ return ApiRequestPipeline(
settings,
provider_getter=lambda provider_type: dependencies.resolve_provider(
provider_type, app=request.app, settings=settings
@@ -79,99 +38,17 @@ def _probe_response(allow: str) -> Response:
return Response(status_code=204, headers={"Allow": allow})
-def _discovered_model_response(model_id: str, *, display_name: str) -> ModelResponse:
- return ModelResponse(
- id=model_id,
- display_name=display_name,
- created_at=DISCOVERED_MODEL_CREATED_AT,
- )
-
-
-def _append_unique_model(
- models: list[ModelResponse], seen: set[str], model: ModelResponse
-) -> None:
- if model.id in seen:
- return
- seen.add(model.id)
- models.append(model)
-
-
-def _append_provider_model_variants(
- models: list[ModelResponse],
- seen: set[str],
- provider_model_ref: str,
- *,
- supports_thinking: bool | None = None,
-) -> None:
- if supports_thinking is not False:
- _append_unique_model(
- models,
- seen,
- _discovered_model_response(
- gateway_model_id(provider_model_ref),
- display_name=provider_model_ref,
- ),
- )
- _append_unique_model(
- models,
- seen,
- _discovered_model_response(
- no_thinking_gateway_model_id(provider_model_ref),
- display_name=f"{provider_model_ref} (no thinking)",
- ),
- )
-
-
-def _build_models_list_response(
- settings: Settings, provider_registry: ProviderRegistry | None
-) -> ModelsListResponse:
- models: list[ModelResponse] = []
- seen: set[str] = set()
-
- for ref in settings.configured_chat_model_refs():
- supports_thinking = None
- if provider_registry is not None:
- supports_thinking = provider_registry.cached_model_supports_thinking(
- ref.provider_id, ref.model_id
- )
- _append_provider_model_variants(
- models,
- seen,
- ref.model_ref,
- supports_thinking=supports_thinking,
- )
-
- if provider_registry is not None:
- for model_info in provider_registry.cached_prefixed_model_infos():
- _append_provider_model_variants(
- models,
- seen,
- model_info.model_id,
- supports_thinking=model_info.supports_thinking,
- )
-
- for model in SUPPORTED_CLAUDE_MODELS:
- _append_unique_model(models, seen, model)
-
- return ModelsListResponse(
- data=models,
- first_id=models[0].id if models else None,
- has_more=False,
- last_id=models[-1].id if models else None,
- )
-
-
# =============================================================================
# Routes
# =============================================================================
@router.post("/v1/messages")
async def create_message(
request_data: MessagesRequest,
- service: ClaudeProxyService = Depends(get_proxy_service),
+ pipeline: ApiRequestPipeline = Depends(get_request_pipeline),
_auth=Depends(require_api_key),
):
"""Create a message (always streaming)."""
- return service.create_message(request_data)
+ return pipeline.create_message(request_data)
@router.api_route("/v1/messages", methods=["HEAD", "OPTIONS"])
@@ -183,11 +60,11 @@ async def probe_messages(_auth=Depends(require_api_key)):
@router.post("/v1/responses")
async def create_response(
request_data: OpenAIResponsesRequest,
- service: ClaudeProxyService = Depends(get_proxy_service),
+ pipeline: ApiRequestPipeline = Depends(get_request_pipeline),
_auth=Depends(require_api_key),
):
"""Create an OpenAI Responses-compatible response through this proxy."""
- return await service.create_response(request_data)
+ return await pipeline.create_response(request_data)
@router.api_route("/v1/responses", methods=["HEAD", "OPTIONS"])
@@ -199,11 +76,11 @@ async def probe_responses(_auth=Depends(require_api_key)):
@router.post("/v1/messages/count_tokens")
async def count_tokens(
request_data: TokenCountRequest,
- service: ClaudeProxyService = Depends(get_proxy_service),
+ pipeline: ApiRequestPipeline = Depends(get_request_pipeline),
_auth=Depends(require_api_key),
):
"""Count tokens for a request."""
- return service.count_tokens(request_data)
+ return pipeline.count_tokens(request_data)
@router.api_route("/v1/messages/count_tokens", methods=["HEAD", "OPTIONS"])
@@ -252,7 +129,7 @@ async def list_models(
trace_event(stage="ingress", event="api.models.list", source="api")
registry = getattr(request.app.state, "provider_registry", None)
provider_registry = registry if isinstance(registry, ProviderRegistry) else None
- return _build_models_list_response(settings, provider_registry)
+ return build_models_list_response(settings, provider_registry)
@router.post("/stop")
diff --git a/pyproject.toml b/pyproject.toml
index 574d334b..cf1c4258 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "free-claude-code"
-version = "2.3.5"
+version = "2.3.6"
description = "Middleware between Claude Code CLI (Anthropic API) and NVIDIA NIM"
readme = "README.md"
requires-python = ">=3.14.0"
diff --git a/smoke/capabilities.py b/smoke/capabilities.py
index ef6f1d06..2557eda6 100644
--- a/smoke/capabilities.py
+++ b/smoke/capabilities.py
@@ -29,7 +29,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"routes_and_probes",
"anthropic_api_routes",
- "api.services.ClaudeProxyService",
+ "api.request_pipeline.ApiRequestPipeline",
"Anthropic-compatible HTTP requests",
"JSON or Anthropic SSE responses",
"HTTPException or Anthropic error payload",
@@ -51,7 +51,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"client_request_shapes",
"drop_in_claude_code_replacement",
- "api.services.ClaudeProxyService",
+ "api.request_pipeline.ApiRequestPipeline",
"Claude Code, VS Code, and JetBrains shaped requests",
"compatible streaming response",
"provider or validation error in Anthropic shape",
@@ -62,7 +62,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"responses_api",
"drop_in_codex_replacement",
- "api.services.ClaudeProxyService.create_response",
+ "api.request_pipeline.ApiRequestPipeline.create_response",
"OpenAI Responses requests from Codex",
"Responses SSE or JSON response",
"OpenAI-shaped error or conversion error",
@@ -80,7 +80,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"client_extensions",
"vscode_extension",
- "api.services.ClaudeProxyService",
+ "api.request_pipeline.ApiRequestPipeline",
"VS Code beta query/header variants",
"accepted Anthropic-compatible response",
"HTTP error only for auth/provider failures",
@@ -91,7 +91,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = (
"api_compatibility",
"client_extensions",
"intellij_extension",
- "api.services.ClaudeProxyService",
+ "api.request_pipeline.ApiRequestPipeline",
"JetBrains/ACP request variants",
"accepted Anthropic-compatible response",
"HTTP error only for auth/provider failures",
diff --git a/tests/api/test_openai_responses.py b/tests/api/test_openai_responses.py
index fb8bed47..741cb224 100644
--- a/tests/api/test_openai_responses.py
+++ b/tests/api/test_openai_responses.py
@@ -81,7 +81,7 @@ def test_create_response_stream_bypasses_local_message_optimizations() -> None:
with (
patch("api.dependencies.resolve_provider", return_value=provider),
patch(
- "api.services.try_optimizations",
+ "api.request_pipeline.try_optimizations",
side_effect=AssertionError("Responses must not use message optimizations"),
),
TestClient(app) as client,
diff --git a/tests/api/test_request_pipeline.py b/tests/api/test_request_pipeline.py
new file mode 100644
index 00000000..119263c6
--- /dev/null
+++ b/tests/api/test_request_pipeline.py
@@ -0,0 +1,122 @@
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+from unittest.mock import MagicMock, patch
+
+import pytest
+from fastapi.responses import StreamingResponse
+
+from api.models.anthropic import Message, MessagesRequest
+from api.models.openai_responses import OpenAIResponsesRequest
+from api.request_pipeline import ApiRequestPipeline
+from config.settings import Settings
+from providers.base import BaseProvider, ProviderConfig
+
+
+class FakeProvider(BaseProvider):
+ def __init__(self) -> None:
+ super().__init__(ProviderConfig(api_key="test"))
+ self.preflight_calls: list[tuple[Any, bool | None]] = []
+ self.requests: list[Any] = []
+ self.stream_kwargs: list[dict[str, Any]] = []
+
+ def preflight_stream(
+ self, request: Any, *, thinking_enabled: bool | None = None
+ ) -> None:
+ self.preflight_calls.append((request, thinking_enabled))
+
+ async def cleanup(self) -> None:
+ return None
+
+ async def list_model_ids(self) -> frozenset[str]:
+ return frozenset({"test-model"})
+
+ async def stream_response(
+ self,
+ request: Any,
+ input_tokens: int = 0,
+ *,
+ request_id: str | None = None,
+ thinking_enabled: bool | None = None,
+ ) -> AsyncIterator[str]:
+ self.requests.append(request)
+ self.stream_kwargs.append(
+ {
+ "input_tokens": input_tokens,
+ "request_id": request_id,
+ "thinking_enabled": thinking_enabled,
+ }
+ )
+ yield 'event: message_start\ndata: {"type":"message_start"}\n\n'
+ yield 'event: message_stop\ndata: {"type":"message_stop"}\n\n'
+
+
+async def _streaming_body_text(response: StreamingResponse) -> str:
+ parts: list[str] = []
+ async for chunk in response.body_iterator:
+ if isinstance(chunk, bytes):
+ parts.append(chunk.decode("utf-8"))
+ else:
+ parts.append(str(chunk))
+ return "".join(parts)
+
+
+async def test_pipeline_provider_execution_passes_routed_request_and_stream_metadata():
+ provider = FakeProvider()
+ pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+ request = MessagesRequest(
+ model="nvidia_nim/test-model",
+ max_tokens=100,
+ messages=[Message(role="user", content="hi")],
+ )
+
+ response = pipeline.create_message(request)
+ assert isinstance(response, StreamingResponse)
+
+ body = await _streaming_body_text(response)
+ assert "message_start" in body
+ assert provider.requests[0].model == "test-model"
+ assert provider.stream_kwargs[0]["input_tokens"] > 0
+ assert provider.stream_kwargs[0]["request_id"].startswith("req_")
+ assert provider.stream_kwargs[0]["thinking_enabled"] is True
+ assert len(provider.preflight_calls) == 1
+
+
+def test_pipeline_message_optimization_intercepts_before_provider_execution():
+ provider_getter = MagicMock()
+ pipeline = ApiRequestPipeline(Settings(), provider_getter=provider_getter)
+ request = MessagesRequest(
+ model="nvidia_nim/test-model",
+ max_tokens=100,
+ messages=[Message(role="user", content="quota check")],
+ )
+ optimized = object()
+
+ with patch("api.request_pipeline.try_optimizations", return_value=optimized):
+ assert pipeline.create_message(request) is optimized
+
+ provider_getter.assert_not_called()
+
+
+async def test_pipeline_responses_bypass_message_only_optimizations():
+ provider = FakeProvider()
+ pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)
+
+ with patch(
+ "api.request_pipeline.try_optimizations",
+ side_effect=AssertionError("Responses must not use message optimizations"),
+ ):
+ response = await pipeline.create_response(
+ request_data=OpenAIResponsesRequest(
+ model="nvidia_nim/test-model",
+ input="quota check",
+ )
+ )
+
+ assert isinstance(response, StreamingResponse)
+ body = await _streaming_body_text(response)
+ assert "response.completed" in body
+ assert provider.requests[0].messages[0].content == "quota check"
diff --git a/tests/api/test_safe_logging.py b/tests/api/test_safe_logging.py
index 7c3f9bc7..1b3edde8 100644
--- a/tests/api/test_safe_logging.py
+++ b/tests/api/test_safe_logging.py
@@ -7,9 +7,9 @@ from unittest.mock import MagicMock, patch
import pytest
from fastapi import HTTPException
-from api import services as services_mod
+from api import request_pipeline as pipeline_mod
from api.models.anthropic import Message, MessagesRequest
-from api.services import ClaudeProxyService
+from api.request_pipeline import ApiRequestPipeline
from config.settings import Settings
from core.anthropic.sse import SSEBuilder
@@ -23,7 +23,7 @@ def test_create_message_skips_full_payload_debug_log_by_default():
yield "event: ping\ndata: {}\n\n"
mock_provider.stream_response = fake_stream
- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)
+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
@@ -31,7 +31,7 @@ def test_create_message_skips_full_payload_debug_log_by_default():
messages=[Message(role="user", content="secret-user-text")],
)
- with patch.object(services_mod.logger, "debug") as mock_debug:
+ with patch.object(pipeline_mod.logger, "debug") as mock_debug:
service.create_message(request)
full_payload_calls = [
@@ -51,14 +51,14 @@ def test_create_message_logs_full_payload_when_opt_in():
yield "event: ping\ndata: {}\n\n"
mock_provider.stream_response = fake_stream
- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)
+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
max_tokens=10,
messages=[Message(role="user", content="visible")],
)
- with patch.object(services_mod.logger, "debug") as mock_debug:
+ with patch.object(pipeline_mod.logger, "debug") as mock_debug:
service.create_message(request)
keys = [c.args[0] for c in mock_debug.call_args_list if c.args]
@@ -103,7 +103,7 @@ def test_create_message_unexpected_error_default_logs_exclude_exception_text():
raise RuntimeError(secret)
mock_provider.stream_response = stream_boom
- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)
+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
max_tokens=10,
@@ -111,7 +111,7 @@ def test_create_message_unexpected_error_default_logs_exclude_exception_text():
)
with (
- patch.object(services_mod.logger, "error") as log_err,
+ patch.object(pipeline_mod.logger, "error") as log_err,
pytest.raises(HTTPException),
):
service.create_message(request)
@@ -134,7 +134,7 @@ def test_create_message_unexpected_error_always_returns_500():
raise WeirdError("no")
mock_provider.stream_response = stream_boom
- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)
+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)
request = MessagesRequest(
model="claude-3-haiku-20240307",
max_tokens=10,
@@ -181,7 +181,7 @@ def test_count_tokens_unexpected_error_default_logs_exclude_exception_text():
def boom(*_a, **_kw):
raise ValueError(secret)
- service = ClaudeProxyService(
+ service = ApiRequestPipeline(
settings,
provider_getter=lambda _: MagicMock(),
token_counter=boom,
@@ -194,7 +194,7 @@ def test_count_tokens_unexpected_error_default_logs_exclude_exception_text():
)
with (
- patch.object(services_mod.logger, "error") as log_err,
+ patch.object(pipeline_mod.logger, "error") as log_err,
pytest.raises(HTTPException),
):
service.count_tokens(req)
diff --git a/tests/api/test_web_server_tools.py b/tests/api/test_web_server_tools.py
index 6d6df63a..140b8040 100644
--- a/tests/api/test_web_server_tools.py
+++ b/tests/api/test_web_server_tools.py
@@ -7,7 +7,7 @@ import pytest
import api.web_tools.constants as web_tool_constants
from api.model_router import ModelRouter, ResolvedModel, RoutedMessagesRequest
from api.models.anthropic import Message, MessagesRequest, Tool
-from api.services import ClaudeProxyService
+from api.request_pipeline import ApiRequestPipeline
from api.web_tools import egress as web_egress
from api.web_tools.egress import (
WebFetchEgressPolicy,
@@ -112,7 +112,7 @@ def test_service_rejects_forced_server_tool_on_openai_when_disabled(
"""OpenAI Chat upstreams cannot run forced server tools without the local handler."""
settings = Settings()
assert settings.enable_web_server_tools is False
- service = ClaudeProxyService(
+ service = ApiRequestPipeline(
settings,
provider_getter=lambda _: MagicMock(),
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -598,7 +598,7 @@ def test_service_rejects_listed_server_tools_on_openai_chat(
provider_id: str,
) -> None:
settings = Settings()
- service = ClaudeProxyService(
+ service = ApiRequestPipeline(
settings,
provider_getter=lambda _: MagicMock(),
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -626,7 +626,7 @@ def test_listed_server_tools_routed_on_anthropic_messages_providers(
mock_provider = MagicMock()
mock_provider.stream_response = fake_stream
- service = ClaudeProxyService(
+ service = ApiRequestPipeline(
settings,
provider_getter=lambda _: mock_provider,
model_router=FixedProviderModelRouter(settings, provider_id),
@@ -654,7 +654,7 @@ def test_forced_server_tools_routed_on_anthropic_messages_providers_when_local_d
mock_provider = MagicMock()
mock_provider.stream_response = fake_stream
- service = ClaudeProxyService(
+ service = ApiRequestPipeline(
settings,
provider_getter=lambda _: mock_provider,
model_router=FixedProviderModelRouter(settings, provider_id),
diff --git a/tests/contracts/test_import_boundaries.py b/tests/contracts/test_import_boundaries.py
index 9ee9c073..d21dfebe 100644
--- a/tests/contracts/test_import_boundaries.py
+++ b/tests/contracts/test_import_boundaries.py
@@ -159,6 +159,7 @@ def test_openai_responses_uses_adapter_boundary() -> None:
repo_root = Path(__file__).resolve().parents[2]
responses_root = repo_root / "core" / "openai_responses"
+ assert not (repo_root / "api" / "services.py").exists()
assert not (responses_root / "conversion.py").exists()
assert not (responses_root / "sse.py").exists()
assert not (responses_root / "output.py").exists()
@@ -177,8 +178,13 @@ def test_openai_responses_uses_adapter_boundary() -> None:
}:
assert (responses_root / filename).exists()
- services_text = (repo_root / "api" / "services.py").read_text(encoding="utf-8")
- assert "from core.openai_responses import OpenAIResponsesAdapter" in services_text
+ pipeline_text = (repo_root / "api" / "request_pipeline.py").read_text(
+ encoding="utf-8"
+ )
+ assert "from core.openai_responses import OpenAIResponsesAdapter" in pipeline_text
+ routes_text = (repo_root / "api" / "routes.py").read_text(encoding="utf-8")
+ assert "ApiRequestPipeline" in routes_text
+ assert "api.services" not in routes_text
for old_helper in {
"responses_request_to_anthropic_payload",
"anthropic_message_response_to_openai_response",
@@ -186,7 +192,7 @@ def test_openai_responses_uses_adapter_boundary() -> None:
"collect_openai_response_from_anthropic_sse",
"iter_message_response_as_openai_responses",
}:
- assert old_helper not in services_text
+ assert old_helper not in pipeline_text
offenders: list[str] = []
for path in (repo_root / "api").rglob("*.py"):
diff --git a/uv.lock b/uv.lock
index 8284ce2a..843eebb1 100644
--- a/uv.lock
+++ b/uv.lock
@@ -561,7 +561,7 @@ wheels = [
[[package]]
name = "free-claude-code"
-version = "2.3.5"
+version = "2.3.6"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },