Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(latency): pass route-selection spend metadata to LiteLLM (#54)
eric-liu-nvidiae62fe46750a8
diff --git a/.agents/skills/switchyard-lib-core/SKILL.md b/.agents/skills/switchyard-lib-core/SKILL.mdindex de8b3b054..17f7409d4 100644--- a/.agents/skills/switchyard-lib-core/SKILL.md+++ b/.agents/skills/switchyard-lib-core/SKILL.md@@ -43,6 +43,7 @@ right validation set. If the change is driven by a launcher need, also read| Stats / telemetry | Reuse `StatsRequestProcessor`, `StatsResponseProcessor`, `StatsLlmBackend`, and `StatsAccumulator`. A profile config should thread one accumulator through all three when stats are enabled. Do not write a parallel collector. || A fixed-path endpoint contributed by per-route components | Set `Endpoint.register_once = True`; `build_switchyard_app(...)` mounts the first instance while still running every component's lifecycle. Leave the default `False` for configurable endpoint classes that may mount distinct instances. || Per-endpoint attribution on `/metrics` for a Python backend that can't be wrapped by `StatsLlmBackend` | Set `ctx.selected_model = endpoint_id` before returning the response. Also set `ctx.backend_call_latency_ms = upstream_call_ms` so the response processor can compute routing overhead. `LatencyServiceLLMBackend.call` is the reference. |+| Spend/tokenomics attribution when a front proxy (LiteLLM) sits above Switchyard | Stamp each upstream attempt with the `x-litellm-spend-logs-metadata` request header (selection re-stamped per failover attempt, one `router_correlation_id` uuid per client request) and record the successful attempt's selection in `ctx.metadata[CTX_ROUTE_SELECTION]`; the endpoint layer maps it to the `x-switchyard-*` response headers via `route_selection_headers` in `lib/endpoints/route_selection.py` — stamped by `dispatch.serialize_chain_result` on success and by `upstream_error.handle_chain_exception` when a failure follows a billed upstream success. Header values are sanitized there (client-controlled `router_model` must stay legal header material). `LatencyServiceLLMBackend.call` is the reference; the payload contract lives on `CTX_ROUTE_SELECTION` in `switchyard/lib/proxy_context.py`. || State metrics on `/metrics` | Register a `PrometheusEmitter` via `switchyard.lib.endpoints.prometheus_emitter.register(...)` and unregister on `shutdown()`. This is for backend-owned state, not request-flow counters. || Error-rate / retry-recovery counters | Use `switchyard.lib.endpoints.outcome_metrics`. FastAPI middleware records client outcomes. A retrying Python backend records each upstream attempt itself (and `record_retry_recovered()`) and sets `CTX_UPSTREAM_ATTEMPTS_RECORDED` so the endpoint skips its fallback — `LatencyServiceLLMBackend.call` is the reference. Single-attempt backends (Rust native / passthrough / multi) record nothing themselves; the endpoint fallback (`record_upstream_attempt_success` / `record_upstream_attempt_failure` in `upstream_error.py`, called from `dispatch_chat_request` and `handle_chain_exception`) counts their one attempt. Don't add a `model` label — these counters are layer-aggregate. Keep labels bounded. For a per-model error breakdown, emit a backend-owned counter via the `PrometheusEmitter` instead (labels bounded to config-derived ids: the route id `config.route_model` + endpoint ids, else the `other` sentinel) — `LatencyServiceLLMBackend._render_prometheus_lines` exposes `switchyard_latency_upstream_attempts_total{requested_model,upstream_model,outcome,code}` next to the aggregate, leaving the shared counter model-free. || Per-event error log | Use `switchyard.lib.endpoints.upstream_error_log.log_upstream_attempt_failure(...)` on the failure path. Events belong in logs/traces, not Prometheus sample timestamps. |diff --git a/docs/internal/latency_service_routing.md b/docs/internal/latency_service_routing.mdindex 9df19dfb8..9df6ffac1 100644--- a/docs/internal/latency_service_routing.md+++ b/docs/internal/latency_service_routing.md@@ -431,6 +431,52 @@ front of Switchyard (e.g. LiteLLM) should tag its own failures the same way andpropagate these headers from below. Mid-stream failures after HTTP 200 has beencommitted cannot carry headers and are not annotated today.+## Route-selection spend attribution++For tokenomics reporting, a LiteLLM front proxy needs to tie its provider spend-log+rows back to the Switchyard logical route that selected them. The backend stamps+**every upstream attempt** with a spend-logs metadata request header (LiteLLM copies+its JSON value into the provider spend-log DB row):++```text+x-litellm-spend-logs-metadata: {"router_model": "<client-facing route id>",+ "router_strategy": "latency",+ "router_selected_endpoint": "<endpoint id>",+ "router_selected_model": "<upstream model>",+ "router_selected_provider": "<upstream model's leading path segment>",+ "router_correlation_id": "<uuid4>"}+```++One correlation id is generated per client request and shared by every failover+attempt; the selection fields are re-stamped per attempt, so each provider row+records the endpoint it actually hit. `router_model` is the model the client asked+for, falling back to the configured `route_model` when the body carried no model.++**Successful** responses then return the selection that served the request, so the+front proxy can enrich its own (parent) spend-log row with the same correlation id+the provider row received:++| Header | Value |+|---|---|+| `x-switchyard-router-model` | `router_model` from the selection |+| `x-switchyard-selected-model` | `router_selected_model` |+| `x-switchyard-selected-provider` | `router_selected_provider` |+| `x-switchyard-router-correlation-id` | `router_correlation_id` |++Streaming responses carry the same headers (the upstream call completes before the+SSE response is committed). Error responses carry the failure-source headers above;+the selection headers join them only when an upstream call already **succeeded**+before the failure (e.g. a response-translation error after a billed 200), so the+provider row's correlation id stays joinable — a request that never reached a+successful upstream claims no selection. Because the client controls `router_model`,+header values are re-validated as header material: a value that is not+latin-1-encodable or contains control characters is omitted rather than breaking+(or splitting) the response; the outbound JSON header is immune (`json.dumps`+escapes it) and still records the raw value. The cross-component contract is+`CTX_ROUTE_SELECTION` in `switchyard/lib/proxy_context.py` (written by the backend,+read by the endpoint layer via `switchyard/lib/endpoints/route_selection.py`);+wire-level proofs live in `tests/test_route_selection_headers.py`.+## ObservabilityWhen `enable_stats` is `true` (the default), `LatencyServiceProfileConfig` wires adiff --git a/switchyard/lib/backends/latency_service_llm_backend.py b/switchyard/lib/backends/latency_service_llm_backend.pyindex fa8f573cd..1942df790 100644--- a/switchyard/lib/backends/latency_service_llm_backend.py+++ b/switchyard/lib/backends/latency_service_llm_backend.py@@ -20,10 +20,13 @@ selected, so Chat endpoints keep receiving Chat Completions whileResponses-mode endpoints receive the OpenAI Responses API natively."""+import jsonimport loggingimport randomimport threadingimport time+import uuid+from collections.abc import Mappingfrom dataclasses import dataclassfrom openai import APIStatusError, AsyncStream@@ -46,6 +49,7 @@ from switchyard.lib.prometheus_exposition import format_number, render_labelsfrom switchyard.lib.proxy_context import (CTX_CALLER_API_KEY,CTX_ERROR_SOURCE,+ CTX_ROUTE_SELECTION,CTX_UPSTREAM_ATTEMPTS_RECORDED,CTX_UPSTREAM_HTTP_BODY,CTX_UPSTREAM_HTTP_STATUS,@@ -63,6 +67,14 @@ from switchyard_rust.translation import TranslationEnginelog = logging.getLogger(__name__)+#: Request header read by a LiteLLM front proxy; its JSON value is copied into+#: the provider spend-log DB row, tying that row back to the Switchyard route+#: that selected it (see :data:`CTX_ROUTE_SELECTION` for the payload contract).+SPEND_LOGS_METADATA_HEADER = "x-litellm-spend-logs-metadata"++#: ``router_strategy`` value identifying this backend's selection algorithm.+ROUTER_STRATEGY_LATENCY = "latency"+@dataclass(frozen=True)class _RouteDecision:@@ -356,6 +368,15 @@ class LatencyServiceLLMBackend(LLMBackend):# -- Request processing (hot path — no Latency Service call) ------------async def call(self, ctx: ProxyContext, request: ChatRequest) -> ChatResponse:+ """Serve *request* on the best healthy endpoint, failing over on error.++ Candidates are ordered by health and observed latency (with optional+ session affinity); every upstream attempt is stamped with the+ spend-logs metadata header, and the attempt that succeeds records its+ route selection on *ctx* (see :data:`CTX_ROUTE_SELECTION`). Exhausting+ all candidates re-raises the last upstream error, annotated for the+ client-facing error envelope.+ """# This backend records its own per-attempt ``outcome_metrics`` counters# below (one per failover attempt). Claim attempt accounting for this# request so the endpoint-layer fallback in ``dispatch`` /@@ -380,6 +401,10 @@ class LatencyServiceLLMBackend(LLMBackend):# Captured before the per-attempt ``body["model"]`` override so the span# records the model the client asked for, not the selected endpoint.incoming_model = request.model+ # One correlation id per client request, shared by every failover+ # attempt's spend-logs header and by the response headers — the join+ # key between a front proxy's spend-log row and the provider rows.+ correlation_id = str(uuid.uuid4())# Resolve the session-affinity pin once (keyed on the stable conversation# prefix). ``None`` when affinity is disabled or the conversation isn't# pinned yet, so every attempt routes purely by health + latency.@@ -426,6 +451,15 @@ class LatencyServiceLLMBackend(LLMBackend):target_request_type = self._request_types[model_id]body = self._body_for_endpoint_request_type(ctx, request, target_request_type)body["model"] = upstream_model+ # Per-attempt route-selection record: each upstream call is stamped+ # with the endpoint it actually hits, so a failover retry's+ # spend-log row doesn't inherit the failed attempt's selection.+ route_selection = self._route_selection(+ incoming_model=incoming_model,+ model_id=model_id,+ upstream_model=upstream_model,+ correlation_id=correlation_id,+ )log.debug("LatencyServiceLLMBackend: attempt=%d model=%s upstream=%s ""request_type=%s stream=%s",@@ -450,6 +484,9 @@ class LatencyServiceLLMBackend(LLMBackend):target_request_type,api_key=api_key_override,body=body,+ extra_headers={+ SPEND_LOGS_METADATA_HEADER: json.dumps(route_selection),+ },)except APIStatusError as exc:set_tags(attempt_span, {@@ -547,6 +584,12 @@ class LatencyServiceLLMBackend(LLMBackend):# Rust ``StatsLlmBackend`` that normally publishes this signal.ctx.backend_call_latency_ms = backend_latency_ms+ # The selection that actually served the request, surfaced to+ # the endpoint layer as ``x-switchyard-*`` response headers so+ # a front proxy can enrich its own spend-log row with the same+ # correlation id the provider row received.+ ctx.metadata[CTX_ROUTE_SELECTION] = route_selection+# Pin this conversation to the endpoint that served it so later# turns reuse it (warm cache). Re-pinning on every success also# follows a recovery: if the previous pin degraded and we@@ -605,6 +648,33 @@ class LatencyServiceLLMBackend(LLMBackend):)return dict(normalized.body)+ def _route_selection(+ self,+ *,+ incoming_model: str | None,+ model_id: str,+ upstream_model: str,+ correlation_id: str,+ ) -> dict[str, str | None]:+ """Build the route-selection payload for one upstream attempt.++ This is the JSON value of the outbound ``x-litellm-spend-logs-metadata``+ header and, for the attempt that succeeds, the+ :data:`CTX_ROUTE_SELECTION` record behind the ``x-switchyard-*``+ response headers. ``router_model`` falls back to the configured route id+ when the client body carried no model; ``router_selected_provider`` is+ the upstream model's leading path segment (IH/LiteLLM naming, e.g.+ ``"openai/openai/gpt-5.4"`` → ``"openai"``).+ """+ return {+ "router_model": incoming_model or self._config.route_model,+ "router_strategy": ROUTER_STRATEGY_LATENCY,+ "router_selected_endpoint": model_id,+ "router_selected_model": upstream_model,+ "router_selected_provider": upstream_model.split("/", 1)[0],+ "router_correlation_id": correlation_id,+ }+async def _call_endpoint(self,model_id: str,@@ -612,14 +682,44 @@ class LatencyServiceLLMBackend(LLMBackend):*,api_key: str | None,body: dict[str, object],+ extra_headers: dict[str, str],) -> object:+ """Make one upstream SDK call with the protected *extra_headers* stamped on.++ ``extra_headers`` rides the client wrapper's ``**kwargs`` into the+ OpenAI SDK's per-request header merge (over ``default_headers``). A+ passthrough body may itself carry an SDK-style ``extra_headers`` field+ (shape-preserving translation keeps unknown keys); it is still+ forwarded — it used to ride ``**body`` into the same SDK parameter —+ but the protected headers must win the merge so callers can't spoof+ them, and header names are case-insensitive on the wire while dict+ keys are not, so every case variant of a protected name is stripped+ from the client mapping (a client-cased duplicate would otherwise+ reach the wire alongside ours and win first-match parsing on the+ receiving proxy). A non-mapping value is dropped: it could only ever+ have broken the SDK call.+ """+ client_extra = body.pop("extra_headers", None)+ protected = {name.lower() for name in extra_headers}+ headers: dict[str, object] = (+ {+ name: value+ for name, value in client_extra.items()+ if not (isinstance(name, str) and name.lower() in protected)+ }+ if isinstance(client_extra, Mapping)+ else {}+ )+ headers.update(extra_headers)if target_request_type == ChatRequestType.OPENAI_RESPONSES:return await self._clients[model_id].aresponses(api_key=api_key,+ extra_headers=headers,**body,)return await self._clients[model_id].acompletion(api_key=api_key,+ extra_headers=headers,**body,)diff --git a/switchyard/lib/endpoints/anthropic_messages_endpoint.py b/switchyard/lib/endpoints/anthropic_messages_endpoint.pyindex beb3a8581..a497a4075 100644--- a/switchyard/lib/endpoints/anthropic_messages_endpoint.py+++ b/switchyard/lib/endpoints/anthropic_messages_endpoint.py@@ -74,6 +74,7 @@ class AnthropicMessagesEndpoint(NemoSwitchyardEndpoint):request: Request,body: Annotated[dict[str, Any], Body(...)],) -> Response:+ """Anthropic-compatible Messages endpoint."""obj = request.app.state.switchyard_strip_unsupported_output_config(body)model = str(body.get("model", "<none>"))@@ -106,7 +107,9 @@ class AnthropicMessagesEndpoint(NemoSwitchyardEndpoint):stream,type(result).__name__,)- return serialize_chain_result(result, stream=stream, sse_iter=iter_anthropic_sse)+ return serialize_chain_result(+ result, stream=stream, sse_iter=iter_anthropic_sse, ctx=ctx+ )except (SwitchyardContextPoolExhaustedError, SwitchyardContextWindowExceededError) as exc:return context_exhausted_response(exc, inbound="anthropic")except Exception as exc:diff --git a/switchyard/lib/endpoints/dispatch.py b/switchyard/lib/endpoints/dispatch.pyindex bdac8cd56..12f55e1a9 100644--- a/switchyard/lib/endpoints/dispatch.py+++ b/switchyard/lib/endpoints/dispatch.py@@ -9,6 +9,7 @@ from typing import Any, castfrom fastapi.responses import JSONResponse, Response, StreamingResponsefrom switchyard.lib.endpoints.error_envelope import error_response+from switchyard.lib.endpoints.route_selection import route_selection_headersfrom switchyard.lib.endpoints.upstream_error import record_upstream_attempt_successfrom switchyard.lib.proxy_context import ProxyContextfrom switchyard.lib.roles import TranslatedResponse@@ -96,16 +97,30 @@ def serialize_chain_result(*,stream: bool,sse_iter: Callable[[Any], AsyncIterator[str]],+ ctx: ProxyContext,) -> Response:"""Serialize a chain result to the appropriate HTTP response.- Returns the result as-is if it is already a ``Response``, wraps it in a+ Returns the result itself if it is already a ``Response``, wraps it in a``StreamingResponse`` when streaming is requested, or JSON-serializes it.+ Any route selection recorded on *ctx* is stamped as ``x-switchyard-*``+ response headers on every branch, pre-built responses included (streaming+ too — the backend call completed before the response object is built, so+ the selection is final). ``ctx`` is required so a new endpoint cannot+ silently opt out of spend attribution."""+ headers = route_selection_headers(ctx)if isinstance(result, Response):+ # Merge rather than pass through untouched: no current chain path+ # yields a pre-built Response after a billed upstream success, but if+ # one ever does, dropping the recorded selection here would silently+ # break spend attribution.+ result.headers.update(headers)return resultif stream and hasattr(result, "__aiter__"):- return StreamingResponse(sse_iter(result), media_type="text/event-stream")+ return StreamingResponse(+ sse_iter(result), media_type="text/event-stream", headers=headers+ )if hasattr(result, "model_dump"):- return JSONResponse(content=result.model_dump())- return JSONResponse(content=result)+ return JSONResponse(content=result.model_dump(), headers=headers)+ return JSONResponse(content=result, headers=headers)diff --git a/switchyard/lib/endpoints/openai_chat_endpoint.py b/switchyard/lib/endpoints/openai_chat_endpoint.pyindex 18cef6318..fb3880df5 100644--- a/switchyard/lib/endpoints/openai_chat_endpoint.py+++ b/switchyard/lib/endpoints/openai_chat_endpoint.py@@ -73,6 +73,7 @@ class OpenAIChatEndpoint(NemoSwitchyardEndpoint):request: Request,body: Annotated[dict[str, Any], Body(...)],) -> Response:+ """OpenAI-compatible Chat Completions endpoint."""obj = request.app.state.switchyardchat_request = ChatRequest.openai_chat(body)# Reject semantically invalid input (e.g. empty messages) at the@@ -89,7 +90,7 @@ class OpenAIChatEndpoint(NemoSwitchyardEndpoint):try:result: Any = await dispatch_chat_request(obj, chat_request, ctx)return serialize_chain_result(- result, stream=stream, sse_iter=iter_chat_completion_sse+ result, stream=stream, sse_iter=iter_chat_completion_sse, ctx=ctx)except (SwitchyardContextPoolExhaustedError, SwitchyardContextWindowExceededError) as exc:return context_exhausted_response(exc, inbound="openai")diff --git a/switchyard/lib/endpoints/responses_endpoint.py b/switchyard/lib/endpoints/responses_endpoint.pyindex b9a9a3be7..d5ae97e2e 100644--- a/switchyard/lib/endpoints/responses_endpoint.py+++ b/switchyard/lib/endpoints/responses_endpoint.py@@ -57,6 +57,7 @@ class ResponsesEndpoint(NemoSwitchyardEndpoint):request: Request,body: Annotated[dict[str, Any], Body(...)],) -> Response:+ """OpenAI-compatible Responses endpoint."""obj = request.app.state.switchyardmodel = str(body.get("model", "<none>"))stream = bool(body.get("stream"))@@ -84,7 +85,9 @@ class ResponsesEndpoint(NemoSwitchyardEndpoint):stream,type(result).__name__,)- return serialize_chain_result(result, stream=stream, sse_iter=iter_preframed_sse)+ return serialize_chain_result(+ result, stream=stream, sse_iter=iter_preframed_sse, ctx=ctx+ )except (SwitchyardContextPoolExhaustedError, SwitchyardContextWindowExceededError) as exc:return context_exhausted_response(exc, inbound="openai-responses")except Exception as exc:diff --git a/switchyard/lib/endpoints/route_selection.py b/switchyard/lib/endpoints/route_selection.pynew file mode 100644index 000000000..209110535--- /dev/null+++ b/switchyard/lib/endpoints/route_selection.py@@ -0,0 +1,68 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Route-selection response headers for spend/tokenomics attribution.++Maps the :data:`CTX_ROUTE_SELECTION` record a routing backend stored on+``ctx`` (see :mod:`switchyard.lib.proxy_context`) to the ``x-switchyard-*``+response headers a front proxy such as LiteLLM copies into its parent+spend-log row. Shared by the success serializer (``dispatch``) and the+error path (``upstream_error``) — a failure that happens *after* a billed+upstream success must still expose the selection, or the provider spend-log+row's correlation id becomes unjoinable.+"""++from collections.abc import Mapping++from switchyard.lib.proxy_context import CTX_ROUTE_SELECTION, ProxyContext++#: Response headers exposing the route selection behind an upstream call,+#: carrying the same correlation id the provider row received via the+#: outbound ``x-litellm-spend-logs-metadata`` header.+ROUTER_MODEL_HEADER = "x-switchyard-router-model"+SELECTED_MODEL_HEADER = "x-switchyard-selected-model"+SELECTED_PROVIDER_HEADER = "x-switchyard-selected-provider"+ROUTER_CORRELATION_ID_HEADER = "x-switchyard-router-correlation-id"++_ROUTE_SELECTION_RESPONSE_HEADERS = (+ (ROUTER_MODEL_HEADER, "router_model"),+ (SELECTED_MODEL_HEADER, "router_selected_model"),+ (SELECTED_PROVIDER_HEADER, "router_selected_provider"),+ (ROUTER_CORRELATION_ID_HEADER, "router_correlation_id"),+)+++def _is_header_value_safe(value: str) -> bool:+ """Whether *value* can be emitted as an HTTP/1.1 response-header value.++ ``router_model`` echoes the client-supplied model string, so it must be+ re-validated as header material: Starlette encodes response-header values+ as latin-1 (a non-encodable value would fail response construction after+ the upstream call already succeeded and was billed), and CTL characters —+ CR/LF above all — would be a response-splitting vector on permissive+ ASGI stacks.+ """+ try:+ value.encode("latin-1")+ except UnicodeEncodeError:+ return False+ return not any(ord(ch) < 0x20 or ord(ch) == 0x7F for ch in value)+++def route_selection_headers(ctx: ProxyContext) -> dict[str, str]:+ """Response headers for the route selection recorded on *ctx*, if any.++ Empty when no routing backend recorded a selection (passthrough chains,+ failures before any upstream success). A recorded field that is absent or+ not emittable as a header value is skipped — headers never carry+ placeholder or unsafe values.+ """+ selection = ctx.metadata.get(CTX_ROUTE_SELECTION)+ if not isinstance(selection, Mapping):+ return {}+ headers: dict[str, str] = {}+ for header_name, selection_key in _ROUTE_SELECTION_RESPONSE_HEADERS:+ value = selection.get(selection_key)+ if isinstance(value, str) and value and _is_header_value_safe(value):+ headers[header_name] = value+ return headersdiff --git a/switchyard/lib/endpoints/upstream_error.py b/switchyard/lib/endpoints/upstream_error.pyindex d51b39d46..285eda79b 100644--- a/switchyard/lib/endpoints/upstream_error.py+++ b/switchyard/lib/endpoints/upstream_error.py@@ -23,6 +23,7 @@ from switchyard.lib.endpoints.error_envelope import (error_response,upstream_error_response,)+from switchyard.lib.endpoints.route_selection import route_selection_headersfrom switchyard.lib.proxy_context import (CTX_ERROR_SOURCE,CTX_UPSTREAM_ATTEMPTS_RECORDED,@@ -197,14 +198,23 @@ def handle_chain_exception(record_upstream_attempt_failure(ctx, exc)upstream = upstream_response_from_ctx(ctx, inbound=inbound, exc=exc)if upstream is not None:- return upstream- _log.error(log_msg, exc_info=exc)- return internal_chain_error_response(- exc,- inbound=inbound,- error_source=_ctx_error_source(ctx, default=ERROR_SOURCE_SWITCHYARD),- upstream_model=_ctx_upstream_model(ctx),- )+ response = upstream+ else:+ _log.error(log_msg, exc_info=exc)+ response = internal_chain_error_response(+ exc,+ inbound=inbound,+ error_source=_ctx_error_source(ctx, default=ERROR_SOURCE_SWITCHYARD),+ upstream_model=_ctx_upstream_model(ctx),+ )+ # A selection on ctx means an upstream call already SUCCEEDED (and was+ # billed, with the spend-logs header stamped) before this failure — e.g.+ # response-side translation rejected the 200 payload. Surface the+ # selection headers on the error response too, so the front proxy can+ # still join its (failed) parent spend-log row to the billed provider row.+ for header_name, value in route_selection_headers(ctx).items():+ response.headers[header_name] = value+ return responsedef context_exhausted_response(exc: BaseException, inbound: Inbound) -> JSONResponse:diff --git a/switchyard/lib/proxy_context.py b/switchyard/lib/proxy_context.pyindex 34bd5c29b..19ae2779e 100644--- a/switchyard/lib/proxy_context.py+++ b/switchyard/lib/proxy_context.py@@ -89,6 +89,19 @@ ERROR_SOURCE_PROVIDER = "provider"#: ``x-switchyard-upstream-model`` response header.CTX_UPSTREAM_MODEL = "_upstream_model"+#: Route-selection record for spend/tokenomics attribution, written by a+#: routing backend after a successful upstream call. A dict with the keys+#: ``router_model`` (client-facing route id), ``router_strategy``,+#: ``router_selected_endpoint``, ``router_selected_model``,+#: ``router_selected_provider``, and ``router_correlation_id``. The same+#: payload is stamped on the outbound upstream call as the+#: ``x-litellm-spend-logs-metadata`` header (one per attempt, so provider+#: spend-log rows record the endpoint they actually hit); the endpoint layer+#: reads this key to return the ``x-switchyard-*`` route-selection response+#: headers, letting a front proxy enrich its own spend-log row with the same+#: correlation id.+CTX_ROUTE_SELECTION = "_route_selection"+__all__ = ["CTX_CALLER_API_KEY",@@ -97,6 +110,7 @@ __all__ = ["CTX_ORIGINAL_MODEL","CTX_ORIGINAL_REQUEST","CTX_PROXY_ACTUAL_MODEL",+ "CTX_ROUTE_SELECTION","CTX_ROUTING","CTX_TARGET_FORMAT","CTX_UPSTREAM_ATTEMPTS_RECORDED",diff --git a/tests/_chain_test_helpers.py b/tests/_chain_test_helpers.pyindex 3e91687be..a3dfc3ec4 100644--- a/tests/_chain_test_helpers.py+++ b/tests/_chain_test_helpers.py@@ -151,11 +151,18 @@ class _OpenAICompatStub:protocol_version = "HTTP/1.1"def do_POST(self) -> None:+ """Record the request (path, body, headers) and pop the queued reply."""length = int(self.headers.get("content-length", "0"))raw = self.rfile.read(length)body = json.loads(raw.decode("utf-8"))with owner._lock:- owner._requests.append({"path": self.path, "body": body})+ # Header names lower-cased so tests can look them up+ # without caring how the client cased them on the wire.+ owner._requests.append({+ "path": self.path,+ "body": body,+ "headers": {k.lower(): v for k, v in self.headers.items()},+ })if owner._responses:status, content, content_type = owner._responses.pop(0)else:@@ -174,7 +181,13 @@ class _OpenAICompatStub:return Noneself._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)- self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)+ # ``shutdown()`` blocks up to serve_forever's poll interval; the 0.5s+ # default adds half a second of idle teardown to every test using the+ # stub, so poll frequently.+ server = self._server+ self._thread = threading.Thread(+ target=lambda: server.serve_forever(poll_interval=0.05), daemon=True+ )self._thread.start()return selfdiff --git a/tests/test_latency_service_llm_backend.py b/tests/test_latency_service_llm_backend.pyindex 6b6d0c40f..00e9addaf 100644--- a/tests/test_latency_service_llm_backend.py+++ b/tests/test_latency_service_llm_backend.py@@ -3,9 +3,11 @@"""Unit tests for :class:`LatencyServiceLLMBackend` (usage case)."""+import jsonimport randomimport threadingimport time+import uuidfrom unittest.mock import AsyncMock, MagicMock, patchimport httpx@@ -22,6 +24,7 @@ from switchyard.lib.backends.health_poller import (HealthPoller,)from switchyard.lib.backends.latency_service_llm_backend import (+ SPEND_LOGS_METADATA_HEADER,LatencyServiceLLMBackend,)from switchyard.lib.config.latency_service_backend_config import (@@ -30,6 +33,7 @@ from switchyard.lib.config.latency_service_backend_config import ()from switchyard.lib.proxy_context import (CTX_ERROR_SOURCE,+ CTX_ROUTE_SELECTION,CTX_UPSTREAM_HTTP_STATUS,CTX_UPSTREAM_MODEL,ProxyContext,@@ -983,6 +987,189 @@ class TestCall:assert call_kwargs["model"] == "openai/gpt-5.5"+# ---------------------------------------------------------------------------+# Route-selection spend-logs metadata (tokenomics attribution)+# ---------------------------------------------------------------------------+++def _spend_logs_payload(mock: AsyncMock, call_index: int = -1) -> dict[str, object]:+ """Parse the spend-logs metadata header stamped on one recorded SDK call."""+ call_kwargs = mock.call_args_list[call_index].kwargs+ header_value = call_kwargs["extra_headers"][SPEND_LOGS_METADATA_HEADER]+ payload = json.loads(header_value)+ assert isinstance(payload, dict)+ return payload+++class TestRouteSelectionSpendLogs:+ """Every upstream attempt is stamped with route-selection metadata.++ The ``x-litellm-spend-logs-metadata`` request header ties a LiteLLM+ provider spend-log row back to the Switchyard route that selected it, and+ the successful attempt's selection is recorded on+ ``ctx.metadata[CTX_ROUTE_SELECTION]`` so the endpoint layer can return the+ matching ``x-switchyard-*`` response headers.+ """++ async def test_outbound_header_carries_route_selection(self):+ """The outbound header records the full selection for the chosen endpoint."""+ config = LatencyServiceBackendConfig(+ latency_service_url=LATENCY_SERVICE_URL,+ endpoints=[+ LatencyServiceEndpoint(+ model="openai/gpt-5.5",+ upstream_model="openai/openai/gpt-5.5",+ base_url="https://inference-api.test/v1",+ api_key="k",+ ),+ ],+ )+ backend = _make_backend(config)+ backend._clients["openai/gpt-5.5"].acompletion = AsyncMock(+ return_value=_make_completion()+ )++ await backend.call(ProxyContext(), _openai_request(model="nvidia/switchyard/gpt-5.5"))++ payload = _spend_logs_payload(backend._clients["openai/gpt-5.5"].acompletion)+ assert payload["router_model"] == "nvidia/switchyard/gpt-5.5"+ assert payload["router_strategy"] == "latency"+ assert payload["router_selected_endpoint"] == "openai/gpt-5.5"+ assert payload["router_selected_model"] == "openai/openai/gpt-5.5"+ assert payload["router_selected_provider"] == "openai"+ # Must parse as a UUID — the join key between spend-log rows.+ uuid.UUID(str(payload["router_correlation_id"]))++ async def test_ctx_records_selection_matching_outbound_header(self):+ """ctx records exactly the payload the wire header carried."""+ backend = _make_backend(_config("model-A"))+ backend._clients["model-A"].acompletion = AsyncMock(+ return_value=_make_completion()+ )++ ctx = ProxyContext()+ await backend.call(ctx, _openai_request())++ payload = _spend_logs_payload(backend._clients["model-A"].acompletion)+ assert ctx.metadata[CTX_ROUTE_SELECTION] == payload+ # Endpoint id without a provider prefix: the provider segment+ # degenerates to the whole id (split on the first "/").+ assert payload["router_selected_provider"] == "model-A"++ async def test_failover_restamps_selection_and_keeps_correlation_id(self):+ """Each attempt is stamped with its own endpoint under one correlation id."""+ backend = _make_backend(_config("model-A", "model-B"))+ _set_health(+ backend,+ {+ "model-A": EndpointHealthStatus.HEALTHY,+ "model-B": EndpointHealthStatus.DEGRADED,+ },+ )+ failing = AsyncMock(side_effect=_api_status_error(500))+ succeeding = AsyncMock(return_value=_make_completion())+ backend._clients["model-A"].acompletion = failing+ backend._clients["model-B"].acompletion = succeeding++ ctx = ProxyContext()+ await backend.call(ctx, _openai_request())++ first = _spend_logs_payload(failing)+ second = _spend_logs_payload(succeeding)+ # Each attempt records the endpoint it actually hit…+ assert first["router_selected_endpoint"] == "model-A"+ assert second["router_selected_endpoint"] == "model-B"+ # …while sharing one correlation id per client request.+ assert first["router_correlation_id"] == second["router_correlation_id"]+ # ctx carries the selection that served the request, not the failure.+ assert ctx.metadata[CTX_ROUTE_SELECTION] == second++ async def test_correlation_id_is_fresh_per_request(self):+ """Two client requests never share a correlation id."""+ backend = _make_backend(_config("model-A"))+ mock = AsyncMock(return_value=_make_completion())+ backend._clients["model-A"].acompletion = mock++ await backend.call(ProxyContext(), _openai_request())+ await backend.call(ProxyContext(), _openai_request())++ first = _spend_logs_payload(mock, call_index=0)+ second = _spend_logs_payload(mock, call_index=1)+ assert first["router_correlation_id"] != second["router_correlation_id"]++ async def test_router_model_falls_back_to_configured_route_model(self):+ """A model-less client body attributes to the configured route id."""+ backend = _make_backend(+ _config("model-A", route_model="nvidia/switchyard/gpt-5.5")+ )+ backend._clients["model-A"].acompletion = AsyncMock(+ return_value=_make_completion()+ )++ body: dict = {"messages": [{"role": "user", "content": "hi"}]}+ await backend.call(ProxyContext(), ChatRequest.openai_chat(body))++ payload = _spend_logs_payload(backend._clients["model-A"].acompletion)+ assert payload["router_model"] == "nvidia/switchyard/gpt-5.5"++ async def test_no_selection_recorded_when_all_attempts_fail(self):+ """No billed success → no selection recorded on ctx."""+ backend = _make_backend(_config("model-A"))+ backend._clients["model-A"].acompletion = AsyncMock(+ side_effect=_api_status_error(401),+ )++ ctx = ProxyContext()+ with pytest.raises(openai.APIStatusError):+ await backend.call(ctx, _openai_request())++ assert CTX_ROUTE_SELECTION not in ctx.metadata++ async def test_responses_surface_is_stamped_too(self):+ """The Responses-API surface is stamped like the Chat surface."""+ backend = _make_backend(_config("model-A", request_type="openai_responses"))+ backend._clients["model-A"].aresponses = AsyncMock(+ return_value={"id": "resp-test", "object": "response", "output": []}+ )++ await backend.call(+ ProxyContext(),+ ChatRequest.openai_responses({"model": "incoming-model", "input": "hi"}),+ )++ payload = _spend_logs_payload(backend._clients["model-A"].aresponses)+ assert payload["router_selected_endpoint"] == "model-A"+ assert payload["router_strategy"] == "latency"++ async def test_cased_spoof_of_spend_logs_header_is_stripped(self):+ """A differently-cased spoof key cannot ride client extra_headers to the wire.++ Header names are case-insensitive on the wire while dict merges are+ not: exactly one instance of the protected header — ours — may reach+ the SDK, while benign client headers still pass through.+ """+ backend = _make_backend(_config("model-A"))+ mock = AsyncMock(return_value=_make_completion())+ backend._clients["model-A"].acompletion = mock++ await backend.call(+ ProxyContext(),+ _openai_request(+ extra_headers={+ "X-LiteLLM-Spend-Logs-Metadata": "spoofed",+ "x-client-tag": "42",+ },+ ),+ )++ headers = mock.call_args.kwargs["extra_headers"]+ spoof_keys = [k for k in headers if k.lower() == SPEND_LOGS_METADATA_HEADER]+ assert spoof_keys == [SPEND_LOGS_METADATA_HEADER]+ payload = _spend_logs_payload(mock)+ assert payload["router_selected_endpoint"] == "model-A"+ assert headers["x-client-tag"] == "42"++# ---------------------------------------------------------------------------# Credential policy# ---------------------------------------------------------------------------diff --git a/tests/test_route_selection_headers.py b/tests/test_route_selection_headers.pynew file mode 100644index 000000000..350498f46--- /dev/null+++ b/tests/test_route_selection_headers.py@@ -0,0 +1,495 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""End-to-end gate for route-selection spend-attribution headers.++For tokenomics reporting, a front proxy (e.g. LiteLLM) must be able to tie+its provider spend-log rows back to the Switchyard logical route that+selected them:++* outbound — every upstream attempt carries an ``x-litellm-spend-logs-metadata``+ request header whose JSON payload records the route selection plus a+ per-request correlation id;+* inbound — the Switchyard response returns ``x-switchyard-*`` headers with+ the successful attempt's selection and the same correlation id, so the+ parent spend-log row can be enriched to match the provider row.++The app-level tests run the real ``OpenAILLMClient`` against a loopback HTTP+stub, so the outbound header is asserted on the wire — not on a mock.+"""++from __future__ import annotations++import json+from collections.abc import Iterator+from typing import Any++from fastapi import FastAPI+from fastapi.responses import JSONResponse+from fastapi.testclient import TestClient+from pytest import fixture+from pytest_mock import MockerFixture++from switchyard.lib.backends.health_poller import HealthPoller+from switchyard.lib.backends.latency_service_llm_backend import (+ SPEND_LOGS_METADATA_HEADER,+)+from switchyard.lib.config.latency_service_backend_config import (+ LatencyServiceBackendConfig,+ LatencyServiceEndpoint,+)+from switchyard.lib.endpoints.dispatch import serialize_chain_result+from switchyard.lib.endpoints.route_selection import (+ ROUTER_CORRELATION_ID_HEADER,+ ROUTER_MODEL_HEADER,+ SELECTED_MODEL_HEADER,+ SELECTED_PROVIDER_HEADER,+ route_selection_headers,+)+from switchyard.lib.endpoints.upstream_error import handle_chain_exception+from switchyard.lib.profiles import LatencyServiceProfileConfig, ProfileSwitchyard+from switchyard.lib.proxy_context import CTX_ROUTE_SELECTION, ProxyContext+from switchyard.server.switchyard_app import build_switchyard_app+from tests._chain_test_helpers import _backend_payload, _OpenAICompatStub, _sse_body, _stream_chunk++ROUTE_MODEL = "nvidia/switchyard/test-route"+ENDPOINT_ID = "openai/test-model"+UPSTREAM_MODEL = "openai/openai/test-model"+++def _selection(**overrides: object) -> dict[str, object]:+ """Selection record as the latency backend writes it, with *overrides* applied."""+ payload: dict[str, object] = {+ "router_model": ROUTE_MODEL,+ "router_strategy": "latency",+ "router_selected_endpoint": ENDPOINT_ID,+ "router_selected_model": UPSTREAM_MODEL,+ "router_selected_provider": "openai",+ "router_correlation_id": "11111111-2222-3333-4444-555555555555",+ }+ payload.update(overrides)+ return payload+++# ---------------------------------------------------------------------------+# Unit: ctx → response-header mapping+# ---------------------------------------------------------------------------+++class TestRouteSelectionHeaders:+ """Mapping of the ctx selection record to ``x-switchyard-*`` headers."""++ def test_empty_without_selection(self) -> None:+ """A ctx without a recorded selection yields no headers."""+ assert route_selection_headers(ProxyContext()) == {}++ def test_maps_selection_to_response_headers(self) -> None:+ """Every exposed selection field maps to its response header."""+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection()++ assert route_selection_headers(ctx) == {+ ROUTER_MODEL_HEADER: ROUTE_MODEL,+ SELECTED_MODEL_HEADER: UPSTREAM_MODEL,+ SELECTED_PROVIDER_HEADER: "openai",+ ROUTER_CORRELATION_ID_HEADER: "11111111-2222-3333-4444-555555555555",+ }++ def test_skips_absent_fields_instead_of_stamping_placeholders(self) -> None:+ """An absent field is skipped — never stamped as a placeholder value."""+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection(router_model=None)++ headers = route_selection_headers(ctx)++ assert ROUTER_MODEL_HEADER not in headers+ assert headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL++ def test_ignores_non_mapping_value(self) -> None:+ """A malformed (non-mapping) ctx record yields no headers."""+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = "bogus"++ assert route_selection_headers(ctx) == {}++ def test_skips_values_unsafe_as_header_material(self) -> None:+ """The client-controlled router_model must be re-validated as a header.++ A CRLF-bearing value would be a response-splitting vector, and a+ non-latin-1 value would crash Starlette response construction after+ the upstream call already succeeded and was billed.+ """+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection(router_model="gpt\r\nx-evil: 1")+ headers = route_selection_headers(ctx)+ assert ROUTER_MODEL_HEADER not in headers+ assert headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL++ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection(router_model="gpt-4中文")+ headers = route_selection_headers(ctx)+ assert ROUTER_MODEL_HEADER not in headers+ assert headers[ROUTER_CORRELATION_ID_HEADER] == (+ "11111111-2222-3333-4444-555555555555"+ )+++class TestSerializeChainResultHeaders:+ """``serialize_chain_result`` stamps the selection on every branch."""++ @staticmethod+ async def _sse_iter(_result: Any) -> Any:+ """Minimal SSE iterator satisfying the serializer signature."""+ yield "data: {}\n\n"++ def test_json_response_carries_selection_headers(self) -> None:+ """A JSON-serialized result carries the selection headers."""+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection()++ response = serialize_chain_result(+ {"ok": True}, stream=False, sse_iter=self._sse_iter, ctx=ctx+ )++ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL+ assert (+ response.headers[ROUTER_CORRELATION_ID_HEADER]+ == "11111111-2222-3333-4444-555555555555"+ )++ def test_streaming_response_carries_selection_headers(self) -> None:+ """The SSE branch stamps the headers too.++ The backend call completes before the StreamingResponse is built, so+ the selection is final by the time SSE headers are committed.+ """+ class _EmptyStream:+ """Async iterator yielding nothing, standing in for a backend stream."""++ def __aiter__(self) -> _EmptyStream:+ return self++ async def __anext__(self) -> str:+ raise StopAsyncIteration++ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection()++ response = serialize_chain_result(+ _EmptyStream(), stream=True, sse_iter=self._sse_iter, ctx=ctx+ )++ assert response.media_type == "text/event-stream"+ assert response.headers[ROUTER_MODEL_HEADER] == ROUTE_MODEL++ def test_selection_free_ctx_no_selection_headers(self) -> None:+ """A selection-free ctx adds no ``x-switchyard-*`` headers."""+ response = serialize_chain_result(+ {"ok": True}, stream=False, sse_iter=self._sse_iter, ctx=ProxyContext()+ )++ assert ROUTER_CORRELATION_ID_HEADER not in response.headers++ def test_prebuilt_response_passes_through_with_selection_merged(self) -> None:+ """A result that is already a ``Response`` gains the selection headers.++ No current chain path yields a pre-built Response after a billed+ upstream success, but the serializer contract — a recorded selection+ is never dropped — must hold on that branch too, with the response's+ own status, body, and headers preserved.+ """+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection()+ prebuilt = JSONResponse(+ content={"ok": True}, status_code=404, headers={"x-existing": "1"}+ )++ response = serialize_chain_result(+ prebuilt, stream=False, sse_iter=self._sse_iter, ctx=ctx+ )++ assert response is prebuilt+ assert response.status_code == 404+ assert response.headers["x-existing"] == "1"+ assert json.loads(response.body) == {"ok": True}+ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == (+ "11111111-2222-3333-4444-555555555555"+ )+++class TestErrorPathSelectionHeaders:+ """``handle_chain_exception`` keeps a billed selection on error responses."""++ def test_failure_after_billed_success_keeps_selection_headers(self) -> None:+ """A post-backend failure must still expose the billed selection.++ The upstream call succeeded (provider spend-log row written with the+ stamped correlation id) before e.g. response translation raised; the+ error response must carry the selection headers or that provider row+ becomes unjoinable.+ """+ ctx = ProxyContext()+ ctx.metadata[CTX_ROUTE_SELECTION] = _selection()++ response = handle_chain_exception(+ RuntimeError("response translation failed"),+ ctx,+ inbound="openai",+ log_msg="test",+ )++ assert response.status_code == 500+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == (+ "11111111-2222-3333-4444-555555555555"+ )+ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL++ def test_failure_without_selection_has_no_selection_headers(self) -> None:+ """No billed success → the error response claims no selection."""+ response = handle_chain_exception(+ RuntimeError("backend never succeeded"),+ ProxyContext(),+ inbound="openai",+ log_msg="test",+ )++ assert response.status_code == 500+ assert ROUTER_CORRELATION_ID_HEADER not in response.headers+++# ---------------------------------------------------------------------------+# End-to-end: HTTP in → wire header out → response headers back+# ---------------------------------------------------------------------------+++@fixture+def latency_app(mocker: MockerFixture) -> Iterator[tuple[FastAPI, _OpenAICompatStub]]:+ """Latency-service app whose single endpoint targets a loopback stub.++ The backend uses the real ``OpenAILLMClient``/OpenAI SDK, so the stub+ records the actual HTTP headers the upstream would receive. Only the+ health poller is stubbed out (no Latency Service in tests).+ """+ with _OpenAICompatStub() as stub:+ config = LatencyServiceBackendConfig(+ latency_service_url="http://latency-service.test:8080",+ route_model=ROUTE_MODEL,+ endpoints=[+ LatencyServiceEndpoint(+ model=ENDPOINT_ID,+ upstream_model=UPSTREAM_MODEL,+ base_url=stub.base_url,+ api_key="test-key",+ ),+ ],+ )+ mocker.patch.object(HealthPoller, "start")+ mocker.patch.object(HealthPoller, "stop")+ switchyard = ProfileSwitchyard(+ LatencyServiceProfileConfig.from_config(config)+ .build()+ .with_runtime_components(enable_stats=config.enable_stats)+ )+ yield build_switchyard_app(switchyard), stub+++def _wire_spend_logs_payload(stub: _OpenAICompatStub) -> dict[str, object]:+ """Parse the spend-logs metadata header the stub received on the wire."""+ headers = stub.requests[-1]["headers"]+ assert isinstance(headers, dict)+ payload = json.loads(str(headers[SPEND_LOGS_METADATA_HEADER]))+ assert isinstance(payload, dict)+ return payload+++def test_chat_completion_roundtrips_selection_and_correlation_id(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """The wire spend-logs header and the response headers share one selection."""+ app, stub = latency_app+ stub.respond_json(_backend_payload(content="hello", model=UPSTREAM_MODEL))++ with TestClient(app) as client:+ response = client.post(+ "/v1/chat/completions",+ json={+ "model": ROUTE_MODEL,+ "messages": [{"role": "user", "content": "hi"}],+ },+ )++ assert response.status_code == 200+ payload = _wire_spend_logs_payload(stub)+ assert payload["router_model"] == ROUTE_MODEL+ assert payload["router_strategy"] == "latency"+ assert payload["router_selected_endpoint"] == ENDPOINT_ID+ assert payload["router_selected_model"] == UPSTREAM_MODEL+ assert payload["router_selected_provider"] == "openai"+ assert response.headers[ROUTER_MODEL_HEADER] == ROUTE_MODEL+ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL+ assert response.headers[SELECTED_PROVIDER_HEADER] == "openai"+ # The join key: the response header must equal the id the provider row got.+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == payload["router_correlation_id"]+++def test_streaming_chat_completion_carries_selection_headers(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """A streaming completion returns the same selection headers before the body."""+ app, stub = latency_app+ stub.respond_sse(_sse_body([_stream_chunk(content="hel"), _stream_chunk(finish="stop")]))++ with TestClient(app) as client:+ response = client.post(+ "/v1/chat/completions",+ json={+ "model": ROUTE_MODEL,+ "messages": [{"role": "user", "content": "hi"}],+ "stream": True,+ },+ )++ assert response.status_code == 200+ assert response.headers["content-type"].startswith("text/event-stream")+ assert "data:" in response.text+ payload = _wire_spend_logs_payload(stub)+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == payload["router_correlation_id"]+ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL+++def test_anthropic_messages_endpoint_carries_selection_headers(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """Anthropic inbound rides the same dispatch path and gets the same headers."""+ app, stub = latency_app+ stub.respond_json(_backend_payload(content="hello", model=UPSTREAM_MODEL))++ with TestClient(app) as client:+ response = client.post(+ "/v1/messages",+ json={+ "model": ROUTE_MODEL,+ "max_tokens": 64,+ "messages": [{"role": "user", "content": "hi"}],+ },+ )++ assert response.status_code == 200+ payload = _wire_spend_logs_payload(stub)+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == payload["router_correlation_id"]+ assert response.headers[SELECTED_PROVIDER_HEADER] == "openai"+++def test_responses_endpoint_carries_selection_headers(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """OpenAI Responses inbound rides the same dispatch path and gets the same headers."""+ app, stub = latency_app+ stub.respond_json(_backend_payload(content="hello", model=UPSTREAM_MODEL))++ with TestClient(app) as client:+ response = client.post(+ "/v1/responses",+ json={"model": ROUTE_MODEL, "input": "hi"},+ )++ assert response.status_code == 200+ payload = _wire_spend_logs_payload(stub)+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == payload["router_correlation_id"]+ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL+++def test_upstream_error_response_has_no_selection_headers(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """No upstream call succeeded — the error response must not claim a selection.++ The failed attempt itself is still stamped on the wire (the provider row+ for the failure keeps its attribution); the client-facing headers are+ reserved for a recorded selection, i.e. a billed upstream success — which+ never happened here. (A failure *after* a billed success does carry them;+ see TestErrorPathSelectionHeaders.)+ """+ app, stub = latency_app+ stub.respond_json({"error": {"message": "bad key"}}, status=401)++ with TestClient(app, raise_server_exceptions=False) as client:+ response = client.post(+ "/v1/chat/completions",+ json={+ "model": ROUTE_MODEL,+ "messages": [{"role": "user", "content": "hi"}],+ },+ )++ assert response.status_code == 401+ assert ROUTER_CORRELATION_ID_HEADER not in response.headers+ assert SPEND_LOGS_METADATA_HEADER in stub.requests[-1]["headers"] # type: ignore[operator]+++def test_hostile_model_string_never_breaks_the_response(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """A model string that is not legal header material must not 500 or inject.++ Single-chain profile serving forwards any client model string, so+ router_model is client-controlled; the billed 200 must survive, with the+ unsafe header skipped and the config-derived headers intact.+ """+ app, stub = latency_app++ with TestClient(app) as client:+ for hostile_model in ("gpt-4中文", "gpt\r\nx-evil: 1"):+ stub.respond_json(_backend_payload(content="hello", model=UPSTREAM_MODEL))+ response = client.post(+ "/v1/chat/completions",+ json={+ "model": hostile_model,+ "messages": [{"role": "user", "content": "hi"}],+ },+ )++ assert response.status_code == 200+ assert ROUTER_MODEL_HEADER not in response.headers+ assert "x-evil" not in response.headers+ assert response.headers[SELECTED_MODEL_HEADER] == UPSTREAM_MODEL+ # The outbound header is JSON (ensure_ascii escapes control and+ # non-ASCII chars), so the wire payload still records the model.+ assert _wire_spend_logs_payload(stub)["router_model"] == hostile_model+++def test_client_body_extra_headers_forwarded_not_collided(+ latency_app: tuple[FastAPI, _OpenAICompatStub],+) -> None:+ """A passthrough body carrying an SDK-style extra_headers field keeps working.++ Pre-spend-logs it rode ``**body`` into the SDK's header kwarg; it must+ still reach the wire — and must not be able to spoof the spend-logs+ header under any key casing, since header names are case-insensitive on+ the wire.+ """+ app, stub = latency_app+ stub.respond_json(_backend_payload(content="hello", model=UPSTREAM_MODEL))++ with TestClient(app) as client:+ response = client.post(+ "/v1/chat/completions",+ json={+ "model": ROUTE_MODEL,+ "messages": [{"role": "user", "content": "hi"}],+ "extra_headers": {+ "x-client-tag": "42",+ SPEND_LOGS_METADATA_HEADER: "spoofed",+ "X-LiteLLM-Spend-Logs-Metadata": "spoofed-cased",+ },+ },+ )++ assert response.status_code == 200+ headers = stub.requests[-1]["headers"]+ assert isinstance(headers, dict)+ assert headers["x-client-tag"] == "42"+ payload = _wire_spend_logs_payload(stub)+ assert payload["router_selected_endpoint"] == ENDPOINT_ID # not "spoofed"+ assert response.headers[ROUTER_CORRELATION_ID_HEADER] == payload["router_correlation_id"]