Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)
Python
Refactor API request pipeline (#851)
Ali Khokharb2af3f63d20a
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.mdindex 9432ba3f..d377b158 100644--- a/ARCHITECTURE.md+++ b/ARCHITECTURE.md@@ -32,9 +32,9 @@ flowchart LRBots[Discord or Telegram Bots] --> Messaging[Messaging Bridge]Messaging --> ClientCLI[Managed Client CLI Sessions]ClientCLI --> ProxyAPI- ProxyAPI --> Service[ClaudeProxyService]- Service --> Router[ModelRouter]- Service --> Providers[ProviderRegistry]+ ProxyAPI --> Pipeline[ApiRequestPipeline]+ Pipeline --> Router[ModelRouter]+ Pipeline --> Providers[ProviderRegistry]Providers --> OpenAIChat[OpenAI Chat Providers]Providers --> NativeAnthropic[Anthropic Messages Providers]```@@ -43,8 +43,8 @@ flowchart LRThe installable wheel packages are declared in [pyproject.toml](pyproject.toml):-- [api/](api/) owns the FastAPI app, route handlers, request orchestration,- admin APIs, local optimizations, and server-tool handling.+- [api/](api/) owns the FastAPI app, route handlers, request pipeline, model+ catalog, admin APIs, local optimizations, and server-tool handling.- [cli/](cli/) owns console entrypoints, client CLI launchers, process/sessionmanagement, and client adapter contracts.- [config/](config/) owns settings, provider metadata, filesystem paths,@@ -105,10 +105,10 @@ The current package boundaries are intentional, but several modules still carrylarge orchestration responsibilities. Treat these as refactor targets, not asnew places to add unrelated behavior:-- [api/services.py](api/services.py) coordinates routing, optimizations, local- server tools, provider execution, and Responses adaptation. Future changes- should keep route handlers thin and move separable use-case logic behind small- helpers.+- [api/request_pipeline.py](api/request_pipeline.py) coordinates routing,+ message-only intercepts, provider execution, token counting, and Responses+ adaptation. Keep route handlers thin and keep new pre-provider behavior as+ explicit pipeline intercepts.- [providers/transports/](providers/transports/) owns provider transportfamilies. The OpenAI-chat and native Anthropic transport packages split thintransport bases from per-request stream runners, recovery event construction,@@ -221,38 +221,39 @@ proxy auth is disabled. Otherwise the token may be supplied through `x-api-key`,`Authorization: Bearer ...`, or `anthropic-auth-token`. Comparisons useconstant-time matching.-`ClaudeProxyService` in [api/services.py](api/services.py) coordinates request-handling. It validates non-empty messages, resolves models, handles local server-tools and optimizations, resolves a provider, preflights the upstream request,-then streams Anthropic SSE back to the caller.+`ApiRequestPipeline` in+[api/request_pipeline.py](api/request_pipeline.py) coordinates request handling.+It validates non-empty messages, resolves models, runs explicit message-only+intercepts for local server tools and optimizations, resolves a provider,+preflights the upstream request, then streams Anthropic SSE back to the caller.```mermaidsequenceDiagramparticipant Clientparticipant Route as FastAPIRoute- participant Service as ClaudeProxyService+ participant Pipeline as ApiRequestPipelineparticipant Router as ModelRouterparticipant Registry as ProviderRegistryparticipant ProviderClient->>Route: POST /v1/messagesRoute->>Route: require_api_key- Route->>Service: create_message- Service->>Router: resolve model and thinking- Service->>Service: server tools or optimizations- Service->>Registry: resolve provider+ Route->>Pipeline: create_message+ Pipeline->>Router: resolve model and thinking+ Pipeline->>Pipeline: server tools or optimizations+ Pipeline->>Registry: resolve providerRegistry->>Provider: cached or new provider- Service->>Provider: preflight_stream- Service->>Provider: stream_response+ Pipeline->>Provider: preflight_stream+ Pipeline->>Provider: stream_responseProvider-->>Client: Anthropic SSE events```-OpenAI Responses uses the same core route. `create_response()` delegates-Responses protocol work to the `OpenAIResponsesAdapter` in+OpenAI Responses uses the same provider execution path without message-only+intercepts. `create_response()` delegates Responses protocol work to the+`OpenAIResponsesAdapter` in[core/openai_responses/adapter.py](core/openai_responses/adapter.py). The adapter-converts the Responses payload into an Anthropic Messages payload, lets the-service reuse `create_message()`, then converts the final response or SSE stream-back to Responses format.+converts the Responses payload into an Anthropic Messages payload before+provider execution, then converts Anthropic SSE back to Responses SSE.## Model Routingdiff --git a/api/model_catalog.py b/api/model_catalog.pynew file mode 100644index 00000000..68b07153--- /dev/null+++ b/api/model_catalog.py@@ -0,0 +1,133 @@+"""Model-list response construction for Claude-compatible clients."""++from __future__ import annotations++from config.settings import Settings+from providers.registry import ProviderRegistry++from .gateway_model_ids import gateway_model_id, no_thinking_gateway_model_id+from .models.responses import ModelResponse, ModelsListResponse++DISCOVERED_MODEL_CREATED_AT = "1970-01-01T00:00:00Z"+++SUPPORTED_CLAUDE_MODELS = [+ ModelResponse(+ id="claude-opus-4-20250514",+ display_name="Claude Opus 4",+ created_at="2025-05-14T00:00:00Z",+ ),+ ModelResponse(+ id="claude-sonnet-4-20250514",+ display_name="Claude Sonnet 4",+ created_at="2025-05-14T00:00:00Z",+ ),+ ModelResponse(+ id="claude-haiku-4-20250514",+ display_name="Claude Haiku 4",+ created_at="2025-05-14T00:00:00Z",+ ),+ ModelResponse(+ id="claude-3-opus-20240229",+ display_name="Claude 3 Opus",+ created_at="2024-02-29T00:00:00Z",+ ),+ ModelResponse(+ id="claude-3-5-sonnet-20241022",+ display_name="Claude 3.5 Sonnet",+ created_at="2024-10-22T00:00:00Z",+ ),+ ModelResponse(+ id="claude-3-haiku-20240307",+ display_name="Claude 3 Haiku",+ created_at="2024-03-07T00:00:00Z",+ ),+ ModelResponse(+ id="claude-3-5-haiku-20241022",+ display_name="Claude 3.5 Haiku",+ created_at="2024-10-22T00:00:00Z",+ ),+]+++def build_models_list_response(+ settings: Settings, provider_registry: ProviderRegistry | None+) -> ModelsListResponse:+ """Return configured, cached, and compatibility model ids."""+ models: list[ModelResponse] = []+ seen: set[str] = set()++ for ref in settings.configured_chat_model_refs():+ supports_thinking = None+ if provider_registry is not None:+ supports_thinking = provider_registry.cached_model_supports_thinking(+ ref.provider_id, ref.model_id+ )+ _append_provider_model_variants(+ models,+ seen,+ ref.model_ref,+ supports_thinking=supports_thinking,+ )++ if provider_registry is not None:+ for model_info in provider_registry.cached_prefixed_model_infos():+ _append_provider_model_variants(+ models,+ seen,+ model_info.model_id,+ supports_thinking=model_info.supports_thinking,+ )++ for model in SUPPORTED_CLAUDE_MODELS:+ _append_unique_model(models, seen, model)++ return ModelsListResponse(+ data=models,+ first_id=models[0].id if models else None,+ has_more=False,+ last_id=models[-1].id if models else None,+ )+++def _discovered_model_response(model_id: str, *, display_name: str) -> ModelResponse:+ return ModelResponse(+ id=model_id,+ display_name=display_name,+ created_at=DISCOVERED_MODEL_CREATED_AT,+ )+++def _append_unique_model(+ models: list[ModelResponse], seen: set[str], model: ModelResponse+) -> None:+ if model.id in seen:+ return+ seen.add(model.id)+ models.append(model)+++def _append_provider_model_variants(+ models: list[ModelResponse],+ seen: set[str],+ provider_model_ref: str,+ *,+ supports_thinking: bool | None = None,+) -> None:+ if supports_thinking is not False:+ _append_unique_model(+ models,+ seen,+ _discovered_model_response(+ gateway_model_id(provider_model_ref),+ display_name=provider_model_ref,+ ),+ )+ _append_unique_model(+ models,+ seen,+ _discovered_model_response(+ no_thinking_gateway_model_id(provider_model_ref),+ display_name=f"{provider_model_ref} (no thinking)",+ ),+ )diff --git a/api/services.py b/api/request_pipeline.pysimilarity index 52%rename from api/services.pyrename to api/request_pipeline.pyindex fd0bc824..600449db 100644--- a/api/services.py+++ b/api/request_pipeline.py@@ -1,4 +1,4 @@-"""Application services for the Claude-compatible API."""+"""API request pipeline for routing, intercepts, and provider execution."""from __future__ import annotations@@ -20,7 +20,7 @@ from core.trace import api_messages_request_snapshot, trace_event, traced_async_from providers.base import BaseProviderfrom providers.exceptions import InvalidRequestError, ProviderError-from .model_router import ModelRouter+from .model_router import ModelRouter, RoutedMessagesRequestfrom .models.anthropic import MessagesRequest, TokenCountRequestfrom .models.openai_responses import OpenAIResponsesRequestfrom .models.responses import TokenCountResponse@@ -33,10 +33,10 @@ from .web_tools.request import (from .web_tools.streaming import stream_web_server_tool_responseTokenCounter = Callable[[list[Any], str | list[Any] | None, list[Any] | None], int]-ProviderGetter = Callable[[str], BaseProvider]+MessageIntercept = Callable[[RoutedMessagesRequest], object | None]-# Providers that use ``/chat/completions`` + Anthropic-to-OpenAI conversion (not native Messages).+# Providers that use ``/chat/completions`` + Anthropic-to-OpenAI conversion._OPENAI_CHAT_UPSTREAM_IDS = frozenset(provider_idfor provider_id, descriptor in PROVIDER_CATALOG.items()@@ -44,10 +44,8 @@ _OPENAI_CHAT_UPSTREAM_IDS = frozenset()-def anthropic_sse_streaming_response(- body: AsyncIterator[str],-) -> StreamingResponse:- """Return a :class:`StreamingResponse` for Anthropic-style SSE streams."""+def anthropic_sse_streaming_response(body: AsyncIterator[str]) -> StreamingResponse:+ """Return a streaming response for Anthropic-style SSE streams."""return StreamingResponse(body,media_type="text/event-stream",@@ -66,19 +64,19 @@ def openai_responses_sse_streaming_response()-def _http_status_for_unexpected_service_exception(_exc: BaseException) -> int:- """HTTP status for uncaught non-provider failures (stable client contract)."""+def _http_status_for_unexpected_pipeline_exception(_exc: BaseException) -> int:+ """HTTP status for uncaught non-provider failures."""return 500-def _log_unexpected_service_exception(+def _log_unexpected_pipeline_exception(settings: Settings,exc: BaseException,*,context: str,request_id: str | None = None,) -> None:- """Log service-layer failures without echoing exception text unless opted in."""+ """Log API failures without echoing exception text unless opted in."""if settings.log_api_error_tracebacks:if request_id is not None:logger.error("{} request_id={}: {}", context, request_id, exc)@@ -102,8 +100,8 @@ def _require_non_empty_messages(messages: list[Any]) -> None:raise InvalidRequestError("messages cannot be empty")-class ClaudeProxyService:- """Coordinate request optimization, model routing, token count, and providers."""+class ApiRequestPipeline:+ """Coordinate API request intercepts, routing, and provider stream execution."""def __init__(self,@@ -112,135 +110,50 @@ class ClaudeProxyService:model_router: ModelRouter | None = None,token_counter: TokenCounter = get_token_count,responses_adapter: OpenAIResponsesAdapter | None = None,- ):+ ) -> None:self._settings = settingsself._provider_getter = provider_getterself._model_router = model_router or ModelRouter(settings)self._token_counter = token_counterself._responses_adapter = responses_adapter or OpenAIResponsesAdapter()+ self._message_intercepts: tuple[MessageIntercept, ...] = (+ self._intercept_web_server_tool,+ self._intercept_local_optimization,+ )def create_message(self, request_data: MessagesRequest) -> object:- """Create a message response or streaming response."""+ """Create an Anthropic-compatible message response."""try:_require_non_empty_messages(request_data.messages)-routed = self._model_router.resolve_messages_request(request_data)- if routed.resolved.provider_id in _OPENAI_CHAT_UPSTREAM_IDS:- tool_err = openai_chat_upstream_server_tool_error(- routed.request,- web_tools_enabled=self._settings.enable_web_server_tools,- )- if tool_err is not None:- raise InvalidRequestError(tool_err)+ self._reject_unsupported_server_tools(routed)- if self._settings.enable_web_server_tools and is_web_server_tool_request(- routed.request- ):- input_tokens = self._token_counter(- routed.request.messages, routed.request.system, routed.request.tools- )- trace_event(- stage="routing",- event="api.optimization.web_server_tool",- source="api",- model=routed.request.model,- )- egress = WebFetchEgressPolicy(- allow_private_network_targets=self._settings.web_fetch_allow_private_networks,- allowed_schemes=self._settings.web_fetch_allowed_scheme_set(),- )- return anthropic_sse_streaming_response(- stream_web_server_tool_response(- routed.request,- input_tokens=input_tokens,- web_fetch_egress=egress,- verbose_client_errors=self._settings.log_api_error_tracebacks,- ),- )+ intercepted = self._run_message_intercepts(routed)+ if intercepted is not None:+ return intercepted- optimized = try_optimizations(routed.request, self._settings)- if optimized is not None:- trace_event(- stage="routing",- event="api.optimization.short_circuit",- source="api",- model=routed.request.model,- )- return optimizedlogger.debug("No optimization matched, routing to provider")-- provider = self._provider_getter(routed.resolved.provider_id)- provider.preflight_stream(- routed.request,- thinking_enabled=routed.resolved.thinking_enabled,- )-- trace_event(- stage="routing",- event="api.route.resolved",- source="api",- provider_id=routed.resolved.provider_id,- provider_model=routed.resolved.provider_model,- provider_model_ref=routed.resolved.provider_model_ref,- gateway_model=routed.request.model,- thinking_enabled=routed.resolved.thinking_enabled,- )-- request_id = f"req_{uuid.uuid4().hex[:12]}"- trace_event(- stage="ingress",- event="api.request.received",- source="api",- message_count=len(routed.request.messages),- snapshot=api_messages_request_snapshot(routed.request),- request_id=request_id,- )-- if self._settings.log_raw_api_payloads:- logger.debug(- "FULL_PAYLOAD [{}]: {}", request_id, routed.request.model_dump()+ return anthropic_sse_streaming_response(+ self._provider_stream(+ routed,+ wire_api="messages",+ raw_log_label="FULL_PAYLOAD",+ raw_log_payload=routed.request.model_dump(),)-- input_tokens = self._token_counter(- routed.request.messages,- routed.request.system,- routed.request.tools,)-- streamed = traced_async_stream(- provider.stream_response(- routed.request,- input_tokens=input_tokens,- request_id=request_id,- thinking_enabled=routed.resolved.thinking_enabled,- ),- stage="egress",- source="api",- complete_event="api.response.stream_completed",- interrupted_event="api.response.stream_interrupted",- chunk_event=None,- extra={- "request_id": request_id,- "provider_id": routed.resolved.provider_id,- "gateway_model": routed.request.model,- },- )- return anthropic_sse_streaming_response(streamed)-except ProviderError:raiseexcept Exception as e:- _log_unexpected_service_exception(+ _log_unexpected_pipeline_exception(self._settings, e, context="CREATE_MESSAGE_ERROR")raise HTTPException(- status_code=_http_status_for_unexpected_service_exception(e),+ status_code=_http_status_for_unexpected_pipeline_exception(e),detail=get_user_facing_error_message(e),) from easync def create_response(self, request_data: OpenAIResponsesRequest) -> object:- """Create an OpenAI Responses-compatible response through the provider router."""-+ """Create a streaming OpenAI Responses-compatible response."""request_payload = request_data.model_dump(mode="json", exclude_none=True)if request_data.stream is False:invalid_request = InvalidRequestError(@@ -261,73 +174,13 @@ class ClaudeProxyService:response_request = MessagesRequest(**anthropic_payload)_require_non_empty_messages(response_request.messages)routed = self._model_router.resolve_messages_request(response_request)+ self._reject_unsupported_server_tools(routed)- if routed.resolved.provider_id in _OPENAI_CHAT_UPSTREAM_IDS:- tool_err = openai_chat_upstream_server_tool_error(- routed.request,- web_tools_enabled=self._settings.enable_web_server_tools,- )- if tool_err is not None:- raise InvalidRequestError(tool_err)-- provider = self._provider_getter(routed.resolved.provider_id)- provider.preflight_stream(- routed.request,- thinking_enabled=routed.resolved.thinking_enabled,- )-- trace_event(- stage="routing",- event="api.route.resolved",- source="api",- provider_id=routed.resolved.provider_id,- provider_model=routed.resolved.provider_model,- provider_model_ref=routed.resolved.provider_model_ref,- gateway_model=routed.request.model,- thinking_enabled=routed.resolved.thinking_enabled,+ streamed = self._provider_stream(+ routed,wire_api="responses",- )-- request_id = f"req_{uuid.uuid4().hex[:12]}"- trace_event(- stage="ingress",- event="api.responses.request.received",- source="api",- message_count=len(routed.request.messages),- snapshot=api_messages_request_snapshot(routed.request),- request_id=request_id,- )-- if self._settings.log_raw_api_payloads:- logger.debug(- "FULL_RESPONSES_PAYLOAD [{}]: {}",- request_id,- request_payload,- )-- input_tokens = self._token_counter(- routed.request.messages,- routed.request.system,- routed.request.tools,- )-- streamed = traced_async_stream(- provider.stream_response(- routed.request,- input_tokens=input_tokens,- request_id=request_id,- thinking_enabled=routed.resolved.thinking_enabled,- ),- stage="egress",- source="api",- complete_event="api.responses.stream_completed",- interrupted_event="api.responses.stream_interrupted",- chunk_event=None,- extra={- "request_id": request_id,- "provider_id": routed.resolved.provider_id,- "gateway_model": routed.request.model,- },+ raw_log_label="FULL_RESPONSES_PAYLOAD",+ raw_log_payload=request_payload,)return openai_responses_sse_streaming_response(self._responses_adapter.iter_sse_from_anthropic(@@ -353,13 +206,13 @@ class ClaudeProxyService:),)except Exception as e:- _log_unexpected_service_exception(+ _log_unexpected_pipeline_exception(self._settings,e,context="CREATE_RESPONSE_ERROR",)return JSONResponse(- status_code=_http_status_for_unexpected_service_exception(e),+ status_code=_http_status_for_unexpected_pipeline_exception(e),content=self._responses_adapter.error_payload(message=get_user_facing_error_message(e),error_type="api_error",@@ -398,13 +251,151 @@ class ClaudeProxyService:except ProviderError:raiseexcept Exception as e:- _log_unexpected_service_exception(+ _log_unexpected_pipeline_exception(self._settings,e,context="COUNT_TOKENS_ERROR",request_id=request_id,)raise HTTPException(- status_code=_http_status_for_unexpected_service_exception(e),+ status_code=_http_status_for_unexpected_pipeline_exception(e),detail=get_user_facing_error_message(e),) from e++ def _reject_unsupported_server_tools(self, routed: RoutedMessagesRequest) -> None:+ if routed.resolved.provider_id not in _OPENAI_CHAT_UPSTREAM_IDS:+ return+ tool_err = openai_chat_upstream_server_tool_error(+ routed.request,+ web_tools_enabled=self._settings.enable_web_server_tools,+ )+ if tool_err is not None:+ raise InvalidRequestError(tool_err)++ def _run_message_intercepts(self, routed: RoutedMessagesRequest) -> object | None:+ for intercept in self._message_intercepts:+ result = intercept(routed)+ if result is not None:+ return result+ return None++ def _intercept_web_server_tool(+ self, routed: RoutedMessagesRequest+ ) -> object | None:+ if not self._settings.enable_web_server_tools:+ return None+ if not is_web_server_tool_request(routed.request):+ return None++ input_tokens = self._token_counter(+ routed.request.messages, routed.request.system, routed.request.tools+ )+ trace_event(+ stage="routing",+ event="api.optimization.web_server_tool",+ source="api",+ model=routed.request.model,+ )+ egress = WebFetchEgressPolicy(+ allow_private_network_targets=self._settings.web_fetch_allow_private_networks,+ allowed_schemes=self._settings.web_fetch_allowed_scheme_set(),+ )+ return anthropic_sse_streaming_response(+ stream_web_server_tool_response(+ routed.request,+ input_tokens=input_tokens,+ web_fetch_egress=egress,+ verbose_client_errors=self._settings.log_api_error_tracebacks,+ ),+ )++ def _intercept_local_optimization(+ self, routed: RoutedMessagesRequest+ ) -> object | None:+ optimized = try_optimizations(routed.request, self._settings)+ if optimized is None:+ return None+ trace_event(+ stage="routing",+ event="api.optimization.short_circuit",+ source="api",+ model=routed.request.model,+ )+ return optimized++ def _provider_stream(+ self,+ routed: RoutedMessagesRequest,+ *,+ wire_api: str,+ raw_log_label: str,+ raw_log_payload: Any,+ ) -> AsyncIterator[str]:+ provider = self._provider_getter(routed.resolved.provider_id)+ provider.preflight_stream(+ routed.request,+ thinking_enabled=routed.resolved.thinking_enabled,+ )++ route_trace: dict[str, Any] = {+ "stage": "routing",+ "event": "api.route.resolved",+ "source": "api",+ "provider_id": routed.resolved.provider_id,+ "provider_model": routed.resolved.provider_model,+ "provider_model_ref": routed.resolved.provider_model_ref,+ "gateway_model": routed.request.model,+ "thinking_enabled": routed.resolved.thinking_enabled,+ }+ if wire_api == "responses":+ route_trace["wire_api"] = "responses"+ trace_event(**route_trace)++ request_id = f"req_{uuid.uuid4().hex[:12]}"+ trace_event(+ stage="ingress",+ event=(+ "api.responses.request.received"+ if wire_api == "responses"+ else "api.request.received"+ ),+ source="api",+ message_count=len(routed.request.messages),+ snapshot=api_messages_request_snapshot(routed.request),+ request_id=request_id,+ )++ if self._settings.log_raw_api_payloads:+ logger.debug(f"{raw_log_label} [{{}}]: {{}}", request_id, raw_log_payload)++ input_tokens = self._token_counter(+ routed.request.messages,+ routed.request.system,+ routed.request.tools,+ )+ return traced_async_stream(+ provider.stream_response(+ routed.request,+ input_tokens=input_tokens,+ request_id=request_id,+ thinking_enabled=routed.resolved.thinking_enabled,+ ),+ stage="egress",+ source="api",+ complete_event=(+ "api.responses.stream_completed"+ if wire_api == "responses"+ else "api.response.stream_completed"+ ),+ interrupted_event=(+ "api.responses.stream_interrupted"+ if wire_api == "responses"+ else "api.response.stream_interrupted"+ ),+ chunk_event=None,+ extra={+ "request_id": request_id,+ "provider_id": routed.resolved.provider_id,+ "gateway_model": routed.request.model,+ },+ )diff --git a/api/routes.py b/api/routes.pyindex e7ec9487..73581b0a 100644--- a/api/routes.py+++ b/api/routes.py@@ -10,62 +10,21 @@ from providers.registry import ProviderRegistryfrom . import dependenciesfrom .dependencies import get_settings, require_api_key-from .gateway_model_ids import gateway_model_id, no_thinking_gateway_model_id+from .model_catalog import build_models_list_responsefrom .models.anthropic import MessagesRequest, TokenCountRequestfrom .models.openai_responses import OpenAIResponsesRequest-from .models.responses import ModelResponse, ModelsListResponse-from .services import ClaudeProxyService+from .models.responses import ModelsListResponse+from .request_pipeline import ApiRequestPipelinerouter = APIRouter()-DISCOVERED_MODEL_CREATED_AT = "1970-01-01T00:00:00Z"---SUPPORTED_CLAUDE_MODELS = [- ModelResponse(- id="claude-opus-4-20250514",- display_name="Claude Opus 4",- created_at="2025-05-14T00:00:00Z",- ),- ModelResponse(- id="claude-sonnet-4-20250514",- display_name="Claude Sonnet 4",- created_at="2025-05-14T00:00:00Z",- ),- ModelResponse(- id="claude-haiku-4-20250514",- display_name="Claude Haiku 4",- created_at="2025-05-14T00:00:00Z",- ),- ModelResponse(- id="claude-3-opus-20240229",- display_name="Claude 3 Opus",- created_at="2024-02-29T00:00:00Z",- ),- ModelResponse(- id="claude-3-5-sonnet-20241022",- display_name="Claude 3.5 Sonnet",- created_at="2024-10-22T00:00:00Z",- ),- ModelResponse(- id="claude-3-haiku-20240307",- display_name="Claude 3 Haiku",- created_at="2024-03-07T00:00:00Z",- ),- ModelResponse(- id="claude-3-5-haiku-20241022",- display_name="Claude 3.5 Haiku",- created_at="2024-10-22T00:00:00Z",- ),-]---def get_proxy_service(++def get_request_pipeline(request: Request,settings: Settings = Depends(get_settings),-) -> ClaudeProxyService:- """Build the request service for route handlers."""- return ClaudeProxyService(+) -> ApiRequestPipeline:+ """Build the API request pipeline for route handlers."""+ return ApiRequestPipeline(settings,provider_getter=lambda provider_type: dependencies.resolve_provider(provider_type, app=request.app, settings=settings@@ -79,99 +38,17 @@ def _probe_response(allow: str) -> Response:return Response(status_code=204, headers={"Allow": allow})-def _discovered_model_response(model_id: str, *, display_name: str) -> ModelResponse:- return ModelResponse(- id=model_id,- display_name=display_name,- created_at=DISCOVERED_MODEL_CREATED_AT,- )---def _append_unique_model(- models: list[ModelResponse], seen: set[str], model: ModelResponse-) -> None:- if model.id in seen:- return- seen.add(model.id)- models.append(model)---def _append_provider_model_variants(- models: list[ModelResponse],- seen: set[str],- provider_model_ref: str,- *,- supports_thinking: bool | None = None,-) -> None:- if supports_thinking is not False:- _append_unique_model(- models,- seen,- _discovered_model_response(- gateway_model_id(provider_model_ref),- display_name=provider_model_ref,- ),- )- _append_unique_model(- models,- seen,- _discovered_model_response(- no_thinking_gateway_model_id(provider_model_ref),- display_name=f"{provider_model_ref} (no thinking)",- ),- )---def _build_models_list_response(- settings: Settings, provider_registry: ProviderRegistry | None-) -> ModelsListResponse:- models: list[ModelResponse] = []- seen: set[str] = set()-- for ref in settings.configured_chat_model_refs():- supports_thinking = None- if provider_registry is not None:- supports_thinking = provider_registry.cached_model_supports_thinking(- ref.provider_id, ref.model_id- )- _append_provider_model_variants(- models,- seen,- ref.model_ref,- supports_thinking=supports_thinking,- )-- if provider_registry is not None:- for model_info in provider_registry.cached_prefixed_model_infos():- _append_provider_model_variants(- models,- seen,- model_info.model_id,- supports_thinking=model_info.supports_thinking,- )-- for model in SUPPORTED_CLAUDE_MODELS:- _append_unique_model(models, seen, model)-- return ModelsListResponse(- data=models,- first_id=models[0].id if models else None,- has_more=False,- last_id=models[-1].id if models else None,- )--# =============================================================================# Routes# =============================================================================@router.post("/v1/messages")async def create_message(request_data: MessagesRequest,- service: ClaudeProxyService = Depends(get_proxy_service),+ pipeline: ApiRequestPipeline = Depends(get_request_pipeline),_auth=Depends(require_api_key),):"""Create a message (always streaming)."""- return service.create_message(request_data)+ return pipeline.create_message(request_data)@router.api_route("/v1/messages", methods=["HEAD", "OPTIONS"])@@ -183,11 +60,11 @@ async def probe_messages(_auth=Depends(require_api_key)):@router.post("/v1/responses")async def create_response(request_data: OpenAIResponsesRequest,- service: ClaudeProxyService = Depends(get_proxy_service),+ pipeline: ApiRequestPipeline = Depends(get_request_pipeline),_auth=Depends(require_api_key),):"""Create an OpenAI Responses-compatible response through this proxy."""- return await service.create_response(request_data)+ return await pipeline.create_response(request_data)@router.api_route("/v1/responses", methods=["HEAD", "OPTIONS"])@@ -199,11 +76,11 @@ async def probe_responses(_auth=Depends(require_api_key)):@router.post("/v1/messages/count_tokens")async def count_tokens(request_data: TokenCountRequest,- service: ClaudeProxyService = Depends(get_proxy_service),+ pipeline: ApiRequestPipeline = Depends(get_request_pipeline),_auth=Depends(require_api_key),):"""Count tokens for a request."""- return service.count_tokens(request_data)+ return pipeline.count_tokens(request_data)@router.api_route("/v1/messages/count_tokens", methods=["HEAD", "OPTIONS"])@@ -252,7 +129,7 @@ async def list_models(trace_event(stage="ingress", event="api.models.list", source="api")registry = getattr(request.app.state, "provider_registry", None)provider_registry = registry if isinstance(registry, ProviderRegistry) else None- return _build_models_list_response(settings, provider_registry)+ return build_models_list_response(settings, provider_registry)@router.post("/stop")diff --git a/pyproject.toml b/pyproject.tomlindex 574d334b..cf1c4258 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "hatchling.build"[project]name = "free-claude-code"-version = "2.3.5"+version = "2.3.6"description = "Middleware between Claude Code CLI (Anthropic API) and NVIDIA NIM"readme = "README.md"requires-python = ">=3.14.0"diff --git a/smoke/capabilities.py b/smoke/capabilities.pyindex ef6f1d06..2557eda6 100644--- a/smoke/capabilities.py+++ b/smoke/capabilities.py@@ -29,7 +29,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = ("api_compatibility","routes_and_probes","anthropic_api_routes",- "api.services.ClaudeProxyService",+ "api.request_pipeline.ApiRequestPipeline","Anthropic-compatible HTTP requests","JSON or Anthropic SSE responses","HTTPException or Anthropic error payload",@@ -51,7 +51,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = ("api_compatibility","client_request_shapes","drop_in_claude_code_replacement",- "api.services.ClaudeProxyService",+ "api.request_pipeline.ApiRequestPipeline","Claude Code, VS Code, and JetBrains shaped requests","compatible streaming response","provider or validation error in Anthropic shape",@@ -62,7 +62,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = ("api_compatibility","responses_api","drop_in_codex_replacement",- "api.services.ClaudeProxyService.create_response",+ "api.request_pipeline.ApiRequestPipeline.create_response","OpenAI Responses requests from Codex","Responses SSE or JSON response","OpenAI-shaped error or conversion error",@@ -80,7 +80,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = ("api_compatibility","client_extensions","vscode_extension",- "api.services.ClaudeProxyService",+ "api.request_pipeline.ApiRequestPipeline","VS Code beta query/header variants","accepted Anthropic-compatible response","HTTP error only for auth/provider failures",@@ -91,7 +91,7 @@ CAPABILITY_CONTRACTS: tuple[CapabilityContract, ...] = ("api_compatibility","client_extensions","intellij_extension",- "api.services.ClaudeProxyService",+ "api.request_pipeline.ApiRequestPipeline","JetBrains/ACP request variants","accepted Anthropic-compatible response","HTTP error only for auth/provider failures",diff --git a/tests/api/test_openai_responses.py b/tests/api/test_openai_responses.pyindex fb8bed47..741cb224 100644--- a/tests/api/test_openai_responses.py+++ b/tests/api/test_openai_responses.py@@ -81,7 +81,7 @@ def test_create_response_stream_bypasses_local_message_optimizations() -> None:with (patch("api.dependencies.resolve_provider", return_value=provider),patch(- "api.services.try_optimizations",+ "api.request_pipeline.try_optimizations",side_effect=AssertionError("Responses must not use message optimizations"),),TestClient(app) as client,diff --git a/tests/api/test_request_pipeline.py b/tests/api/test_request_pipeline.pynew file mode 100644index 00000000..119263c6--- /dev/null+++ b/tests/api/test_request_pipeline.py@@ -0,0 +1,122 @@+from __future__ import annotations++from collections.abc import AsyncIterator+from typing import Any+from unittest.mock import MagicMock, patch++import pytest+from fastapi.responses import StreamingResponse++from api.models.anthropic import Message, MessagesRequest+from api.models.openai_responses import OpenAIResponsesRequest+from api.request_pipeline import ApiRequestPipeline+from config.settings import Settings+from providers.base import BaseProvider, ProviderConfig+++class FakeProvider(BaseProvider):+ def __init__(self) -> None:+ super().__init__(ProviderConfig(api_key="test"))+ self.preflight_calls: list[tuple[Any, bool | None]] = []+ self.requests: list[Any] = []+ self.stream_kwargs: list[dict[str, Any]] = []++ def preflight_stream(+ self, request: Any, *, thinking_enabled: bool | None = None+ ) -> None:+ self.preflight_calls.append((request, thinking_enabled))++ async def cleanup(self) -> None:+ return None++ async def list_model_ids(self) -> frozenset[str]:+ return frozenset({"test-model"})++ async def stream_response(+ self,+ request: Any,+ input_tokens: int = 0,+ *,+ request_id: str | None = None,+ thinking_enabled: bool | None = None,+ ) -> AsyncIterator[str]:+ self.requests.append(request)+ self.stream_kwargs.append(+ {+ "input_tokens": input_tokens,+ "request_id": request_id,+ "thinking_enabled": thinking_enabled,+ }+ )+ yield 'event: message_start\ndata: {"type":"message_start"}\n\n'+ yield 'event: message_stop\ndata: {"type":"message_stop"}\n\n'+++async def _streaming_body_text(response: StreamingResponse) -> str:+ parts: list[str] = []+ async for chunk in response.body_iterator:+ if isinstance(chunk, bytes):+ parts.append(chunk.decode("utf-8"))+ else:+ parts.append(str(chunk))+ return "".join(parts)+++async def test_pipeline_provider_execution_passes_routed_request_and_stream_metadata():+ provider = FakeProvider()+ pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)+ request = MessagesRequest(+ model="nvidia_nim/test-model",+ max_tokens=100,+ messages=[Message(role="user", content="hi")],+ )++ response = pipeline.create_message(request)+ assert isinstance(response, StreamingResponse)++ body = await _streaming_body_text(response)+ assert "message_start" in body+ assert provider.requests[0].model == "test-model"+ assert provider.stream_kwargs[0]["input_tokens"] > 0+ assert provider.stream_kwargs[0]["request_id"].startswith("req_")+ assert provider.stream_kwargs[0]["thinking_enabled"] is True+ assert len(provider.preflight_calls) == 1+++def test_pipeline_message_optimization_intercepts_before_provider_execution():+ provider_getter = MagicMock()+ pipeline = ApiRequestPipeline(Settings(), provider_getter=provider_getter)+ request = MessagesRequest(+ model="nvidia_nim/test-model",+ max_tokens=100,+ messages=[Message(role="user", content="quota check")],+ )+ optimized = object()++ with patch("api.request_pipeline.try_optimizations", return_value=optimized):+ assert pipeline.create_message(request) is optimized++ provider_getter.assert_not_called()+++async def test_pipeline_responses_bypass_message_only_optimizations():+ provider = FakeProvider()+ pipeline = ApiRequestPipeline(Settings(), provider_getter=lambda _: provider)++ with patch(+ "api.request_pipeline.try_optimizations",+ side_effect=AssertionError("Responses must not use message optimizations"),+ ):+ response = await pipeline.create_response(+ request_data=OpenAIResponsesRequest(+ model="nvidia_nim/test-model",+ input="quota check",+ )+ )++ assert isinstance(response, StreamingResponse)+ body = await _streaming_body_text(response)+ assert "response.completed" in body+ assert provider.requests[0].messages[0].content == "quota check"diff --git a/tests/api/test_safe_logging.py b/tests/api/test_safe_logging.pyindex 7c3f9bc7..1b3edde8 100644--- a/tests/api/test_safe_logging.py+++ b/tests/api/test_safe_logging.py@@ -7,9 +7,9 @@ from unittest.mock import MagicMock, patchimport pytestfrom fastapi import HTTPException-from api import services as services_mod+from api import request_pipeline as pipeline_modfrom api.models.anthropic import Message, MessagesRequest-from api.services import ClaudeProxyService+from api.request_pipeline import ApiRequestPipelinefrom config.settings import Settingsfrom core.anthropic.sse import SSEBuilder@@ -23,7 +23,7 @@ def test_create_message_skips_full_payload_debug_log_by_default():yield "event: ping\ndata: {}\n\n"mock_provider.stream_response = fake_stream- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)request = MessagesRequest(model="claude-3-haiku-20240307",@@ -31,7 +31,7 @@ def test_create_message_skips_full_payload_debug_log_by_default():messages=[Message(role="user", content="secret-user-text")],)- with patch.object(services_mod.logger, "debug") as mock_debug:+ with patch.object(pipeline_mod.logger, "debug") as mock_debug:service.create_message(request)full_payload_calls = [@@ -51,14 +51,14 @@ def test_create_message_logs_full_payload_when_opt_in():yield "event: ping\ndata: {}\n\n"mock_provider.stream_response = fake_stream- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)request = MessagesRequest(model="claude-3-haiku-20240307",max_tokens=10,messages=[Message(role="user", content="visible")],)- with patch.object(services_mod.logger, "debug") as mock_debug:+ with patch.object(pipeline_mod.logger, "debug") as mock_debug:service.create_message(request)keys = [c.args[0] for c in mock_debug.call_args_list if c.args]@@ -103,7 +103,7 @@ def test_create_message_unexpected_error_default_logs_exclude_exception_text():raise RuntimeError(secret)mock_provider.stream_response = stream_boom- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)request = MessagesRequest(model="claude-3-haiku-20240307",max_tokens=10,@@ -111,7 +111,7 @@ def test_create_message_unexpected_error_default_logs_exclude_exception_text():)with (- patch.object(services_mod.logger, "error") as log_err,+ patch.object(pipeline_mod.logger, "error") as log_err,pytest.raises(HTTPException),):service.create_message(request)@@ -134,7 +134,7 @@ def test_create_message_unexpected_error_always_returns_500():raise WeirdError("no")mock_provider.stream_response = stream_boom- service = ClaudeProxyService(settings, provider_getter=lambda _: mock_provider)+ service = ApiRequestPipeline(settings, provider_getter=lambda _: mock_provider)request = MessagesRequest(model="claude-3-haiku-20240307",max_tokens=10,@@ -181,7 +181,7 @@ def test_count_tokens_unexpected_error_default_logs_exclude_exception_text():def boom(*_a, **_kw):raise ValueError(secret)- service = ClaudeProxyService(+ service = ApiRequestPipeline(settings,provider_getter=lambda _: MagicMock(),token_counter=boom,@@ -194,7 +194,7 @@ def test_count_tokens_unexpected_error_default_logs_exclude_exception_text():)with (- patch.object(services_mod.logger, "error") as log_err,+ patch.object(pipeline_mod.logger, "error") as log_err,pytest.raises(HTTPException),):service.count_tokens(req)diff --git a/tests/api/test_web_server_tools.py b/tests/api/test_web_server_tools.pyindex 6d6df63a..140b8040 100644--- a/tests/api/test_web_server_tools.py+++ b/tests/api/test_web_server_tools.py@@ -7,7 +7,7 @@ import pytestimport api.web_tools.constants as web_tool_constantsfrom api.model_router import ModelRouter, ResolvedModel, RoutedMessagesRequestfrom api.models.anthropic import Message, MessagesRequest, Tool-from api.services import ClaudeProxyService+from api.request_pipeline import ApiRequestPipelinefrom api.web_tools import egress as web_egressfrom api.web_tools.egress import (WebFetchEgressPolicy,@@ -112,7 +112,7 @@ def test_service_rejects_forced_server_tool_on_openai_when_disabled("""OpenAI Chat upstreams cannot run forced server tools without the local handler."""settings = Settings()assert settings.enable_web_server_tools is False- service = ClaudeProxyService(+ service = ApiRequestPipeline(settings,provider_getter=lambda _: MagicMock(),model_router=FixedProviderModelRouter(settings, provider_id),@@ -598,7 +598,7 @@ def test_service_rejects_listed_server_tools_on_openai_chat(provider_id: str,) -> None:settings = Settings()- service = ClaudeProxyService(+ service = ApiRequestPipeline(settings,provider_getter=lambda _: MagicMock(),model_router=FixedProviderModelRouter(settings, provider_id),@@ -626,7 +626,7 @@ def test_listed_server_tools_routed_on_anthropic_messages_providers(mock_provider = MagicMock()mock_provider.stream_response = fake_stream- service = ClaudeProxyService(+ service = ApiRequestPipeline(settings,provider_getter=lambda _: mock_provider,model_router=FixedProviderModelRouter(settings, provider_id),@@ -654,7 +654,7 @@ def test_forced_server_tools_routed_on_anthropic_messages_providers_when_local_dmock_provider = MagicMock()mock_provider.stream_response = fake_stream- service = ClaudeProxyService(+ service = ApiRequestPipeline(settings,provider_getter=lambda _: mock_provider,model_router=FixedProviderModelRouter(settings, provider_id),diff --git a/tests/contracts/test_import_boundaries.py b/tests/contracts/test_import_boundaries.pyindex 9ee9c073..d21dfebe 100644--- a/tests/contracts/test_import_boundaries.py+++ b/tests/contracts/test_import_boundaries.py@@ -159,6 +159,7 @@ def test_openai_responses_uses_adapter_boundary() -> None:repo_root = Path(__file__).resolve().parents[2]responses_root = repo_root / "core" / "openai_responses"+ assert not (repo_root / "api" / "services.py").exists()assert not (responses_root / "conversion.py").exists()assert not (responses_root / "sse.py").exists()assert not (responses_root / "output.py").exists()@@ -177,8 +178,13 @@ def test_openai_responses_uses_adapter_boundary() -> None:}:assert (responses_root / filename).exists()- services_text = (repo_root / "api" / "services.py").read_text(encoding="utf-8")- assert "from core.openai_responses import OpenAIResponsesAdapter" in services_text+ pipeline_text = (repo_root / "api" / "request_pipeline.py").read_text(+ encoding="utf-8"+ )+ assert "from core.openai_responses import OpenAIResponsesAdapter" in pipeline_text+ routes_text = (repo_root / "api" / "routes.py").read_text(encoding="utf-8")+ assert "ApiRequestPipeline" in routes_text+ assert "api.services" not in routes_textfor old_helper in {"responses_request_to_anthropic_payload","anthropic_message_response_to_openai_response",@@ -186,7 +192,7 @@ def test_openai_responses_uses_adapter_boundary() -> None:"collect_openai_response_from_anthropic_sse","iter_message_response_as_openai_responses",}:- assert old_helper not in services_text+ assert old_helper not in pipeline_textoffenders: list[str] = []for path in (repo_root / "api").rglob("*.py"):diff --git a/uv.lock b/uv.lockindex 8284ce2a..843eebb1 100644--- a/uv.lock+++ b/uv.lock@@ -561,7 +561,7 @@ wheels = [[[package]]name = "free-claude-code"-version = "2.3.5"+version = "2.3.6"source = { editable = "." }dependencies = [{ name = "aiohttp" },