Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(litellm): use stage router integration (#226)
Dhruv Nandakumar3acf3d86f10f
diff --git a/examples/experimental/litellm/README.md b/examples/experimental/litellm/README.mdindex 23b4196e0..546a0adca 100644--- a/examples/experimental/litellm/README.md+++ b/examples/experimental/litellm/README.md@@ -9,7 +9,8 @@ Compose configuration, and example code are intended as a copyable startingpoint for your own application.LiteLLM provides the OpenAI-compatible gateway, model aliases, and OpenRouter-provider integration. Switchyard makes the routing decision.+provider integration. Switchyard's Stage router makes the routing decision from+the coding agent's recent tool history.`LiteLLMSyClient` connects Switchyard's normalized libsy requests to theselected alias through LiteLLM's asynchronous Completion API and theDockerized gateway. Together, they let an application keep routing policy in@@ -19,15 +20,18 @@ through OpenRouter.## Request flow```text-your application → libsy normalized request → Switchyard router+your application → libsy normalized request → Switchyard Stage router→ LiteLLMSyClient → LiteLLM async Completion API→ Dockerized gateway alias → OpenRouter Chat Completions→ selected model```-The bundled router selects either `strong` or `fast`; the client asks LiteLLM's-Completion API to call that alias through the gateway rather than embedding a-provider model ID in application code.+The bundled router treats `strong` as the capable tier and `fast` as the+efficient tier. It uses the `efficient_first` picker, so an initial request or+an ambiguous turn falls open to `fast`. Decisive error-recovery signals, such as+a critical failed tool result, route the turn to `strong`. The client asks+LiteLLM's Completion API to call the selected alias through the gateway rather+than embedding a provider model ID in application code.## Quick start@@ -89,8 +93,9 @@ curl -fsS http://127.0.0.1:4000/health/livelinessuv run python example.py```-`weights=[1, 3]` means 25% `strong` and 75% `fast` over many calls. `seed=42`-makes the sequence repeatable.+The bundled request includes an out-of-memory tool failure, so the Stage router+selects the capable `strong` alias. Remove the assistant tool call and tool+result to see the same router fall open to `fast`.## Use in your application@@ -110,18 +115,47 @@ async def main() -> None:"messages": [{"role": "user",- "content": [{"type": "text", "text": "Say hello briefly."}],- }+ "content": [{"type": "text", "text": "Fix the failing tests."}],+ },+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": {"command": "pytest"},+ }+ ],+ },+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [+ {+ "type": "text",+ "text": "fatal runtime error: out of memory",+ }+ ],+ "is_error": True,+ }+ ],+ },],"reasoning": {"effort": "low"},"output": {"max_output_tokens": 64},}strong = LiteLLMSyClient("strong")fast = LiteLLMSyClient("fast")- router = algorithms.random(- [LlmTarget("strong", strong), LlmTarget("fast", fast)],- weights=[1, 3],- seed=42,+ router = algorithms.stage_router(+ LlmTarget("strong", strong),+ LlmTarget("fast", fast),+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,)try:decisions, response = await router.run(request)@@ -137,14 +171,23 @@ asyncio.run(main())## Known limitations-This example intentionally supports buffered, text-only Chat Completions.-Tools, media, structured output, and streaming fail explicitly.+This example intentionally supports buffered text and OpenAI-compatible+function-tool traffic. It translates normalized function definitions, tool+choices, assistant tool calls, and text-only tool results into gateway requests,+and normalizes returned function calls for `libsy`.+Media, instructions, non-text tool results, structured output, provider-specific+extensions, preserved raw payloads, and streaming fail explicitly.The pinned LiteLLM release does not yet recognize Kimi K3's current`reasoning_effort` support. `LiteLLMSyClient` forwards LiteLLM's`allowed_openai_params` hint to the gateway so OpenRouter receives thatsupported parameter. Recheck this compatibility hint when upgrading LiteLLM.+LiteLLM 1.92 enters its optional proxy MCP path before distinguishing ordinary+OpenAI function tools. `LiteLLMSyClient` disables that bridge for these calls so+the integration does not need LiteLLM's proxy dependencies. Recheck this pinned+compatibility behavior when upgrading LiteLLM.+## TestFrom the package directory, run the offline tests without provider calls:@@ -154,7 +197,8 @@ PYTHONPATH=. uv run --project . pytest tests -m "not e2e" -v```The E2E test starts its own LiteLLM gateway and makes two paid OpenRouter Chat-Completions calls, one to each configured model.+Completions calls through one Stage router. A no-signal request falls open to+`fast`; a critical failed-tool request routes to `strong`.`SWITCHYARD_LITELLM_E2E=1` is the explicit spend opt-in:```bash@@ -184,7 +228,7 @@ bash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \--server-config examples/experimental/litellm/benchmark-route.toml \--task-list-file examples/experimental/litellm/benchmark-tasks.txt \- --model litellm-random \+ --model litellm-stage \--agent codex \--reasoning-effort low \--n-tasks 3 \@@ -194,9 +238,14 @@ bash benchmark/run-baseline.sh \--foreground```-The sample runs `broken-python`, `cosign-keyless-signing`, and-`jq-data-processing`. Three autonomous coding-agent tasks can make many paid-model calls. Inspect the resulting run under `benchmark/tb_runs/`, including+The native benchmark route uses the same efficient-first Stage policy but does+not use `LiteLLMSyClient`; Switchyard server translates requests directly to+the shared LiteLLM gateway. The gateway drops parameters unsupported by the+selected provider model, so the benchmark's reasoning effort reaches `strong`+without causing `fast` requests to fail. The sample runs `broken-python`,+`cosign-keyless-signing`, and `jq-data-processing`. Three autonomous coding-agent+tasks can make many paid model calls. Inspect the resulting run under+`benchmark/tb_runs/`, including`run_manifest.json`, `harbor.log`, `routing_stats_final.json`, and`jobs/<job>/<task>/agent/trajectory.json`.@@ -226,7 +275,7 @@ bash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \--server-config examples/experimental/litellm/benchmark-route.toml \--task-list-file examples/experimental/litellm/benchmark-tasks.txt \- --model litellm-random \+ --model litellm-stage \--agent codex \--reasoning-effort low \--n-tasks 3 \@@ -287,4 +336,4 @@ deployment guidance before exposing it remotely.- [GPT-5.6 Sol on OpenRouter](https://openrouter.ai/openai/gpt-5.6-sol)- [Kimi K3 on OpenRouter](https://openrouter.ai/moonshotai/kimi-k3-20260715)- [LiteLLM gateway quick start](https://docs.litellm.ai/docs/proxy/quick_start)-- [Switchyard random-routing docs](../../../docs/routing_algorithms/random_routing.md)+- [Switchyard Stage-router docs](../../../docs/routing_algorithms/stage_router_routing.md)diff --git a/examples/experimental/litellm/benchmark-route.toml b/examples/experimental/litellm/benchmark-route.tomlindex 1ef063bd6..10c7413a4 100644--- a/examples/experimental/litellm/benchmark-route.toml+++ b/examples/experimental/litellm/benchmark-route.toml@@ -15,9 +15,11 @@ llm_client = "litellm"id = "fast"llm_client = "litellm"-[routes.litellm_random]-id = "litellm-random"-type = "random"-targets = ["strong", "fast"]-weights = [0.5, 0.5]-seed = 42+[routes.litellm_stage]+id = "litellm-stage"+type = "stage_router"+capable_target = "strong"+efficient_target = "fast"+picker = "efficient_first"+confidence_threshold = 0.5+recent_turn_window = 3diff --git a/examples/experimental/litellm/example.py b/examples/experimental/litellm/example.pyindex 362f21a6c..bb67c27ea 100644--- a/examples/experimental/litellm/example.py+++ b/examples/experimental/litellm/example.py@@ -2,7 +2,7 @@# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.# SPDX-License-Identifier: Apache-2.0-"""Route one text request through LiteLLM with libsy's random algorithm."""+"""Route one coding-agent turn through LiteLLM with libsy's Stage router."""import asyncio@@ -12,7 +12,7 @@ from switchyard.libsy import LlmTarget, algorithmsdef sy_request() -> dict[str, object]:- """Build a normalized libsy text request."""+ """Build a normalized turn whose critical tool failure needs the capable tier."""return {"model": "auto","messages": [@@ -21,10 +21,37 @@ def sy_request() -> dict[str, object]:"content": [{"type": "text",- "text": "Explain weighted random LLM routing in one sentence.",+ "text": "Fix the failing tests.",}],- }+ },+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": {"command": "pytest"},+ }+ ],+ },+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [+ {+ "type": "text",+ "text": "fatal runtime error: out of memory",+ }+ ],+ "is_error": True,+ }+ ],+ },],"reasoning": {"effort": "low"},"output": {"max_output_tokens": 96},@@ -32,20 +59,19 @@ def sy_request() -> dict[str, object]:async def main() -> None:- """Run the weighted router and print its normalized result."""+ """Run the Stage router and print its normalized result."""strong_client = LiteLLMSyClient("strong")fast_client = LiteLLMSyClient("fast")- router = algorithms.random(- [- LlmTarget("strong", strong_client),- LlmTarget("fast", fast_client),- ],- weights=[1, 3],- seed=42,+ router = algorithms.stage_router(+ LlmTarget("strong", strong_client),+ LlmTarget("fast", fast_client),+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,)try:decisions, response = await router.run(sy_request())- print("Random:", decisions, response)+ print("Stage router:", decisions, response)finally:await strong_client.aclose()await fast_client.aclose()diff --git a/examples/experimental/litellm/litellm-config.yaml b/examples/experimental/litellm/litellm-config.yamlindex 83f32d0f9..8ab38a113 100644--- a/examples/experimental/litellm/litellm-config.yaml+++ b/examples/experimental/litellm/litellm-config.yaml@@ -7,3 +7,6 @@ model_list:litellm_params:model: openrouter/moonshotai/kimi-k3api_key: os.environ/OPENROUTER_API_KEY++litellm_settings:+ drop_params: truediff --git a/examples/experimental/litellm/src/switchyard_litellm/client.py b/examples/experimental/litellm/src/switchyard_litellm/client.pyindex d61e97f1c..dc0286c1f 100644--- a/examples/experimental/litellm/src/switchyard_litellm/client.py+++ b/examples/experimental/litellm/src/switchyard_litellm/client.py@@ -5,12 +5,13 @@from __future__ import annotations+import jsonfrom collections.abc import Mapping, Sequencefrom typing import Any, castfrom litellm import ModelResponse, acompletion-_ROLES = {"system", "developer", "user", "assistant"}+_TEXT_ROLES = {"system", "developer", "user"}_STOP_REASONS = {"stop": "end_turn","length": "max_tokens",@@ -67,6 +68,47 @@ def _reject_preservation_payload(request: Mapping[str, object]) -> None:raise ValueError("preservation is not supported")+def _text_block(raw_block: object, path: str) -> dict[str, str]:+ block = _mapping(raw_block, path)+ if block.get("type") != "text" or not isinstance(block.get("text"), str):+ raise ValueError(f"{path} must be a text block")+ return {"type": "text", "text": cast(str, block["text"])}+++def _tool_call(raw_block: object, path: str) -> dict[str, object]:+ block = _mapping(raw_block, path)+ call_id = block.get("id")+ name = block.get("name")+ arguments = block.get("arguments")+ if block.get("type") != "tool_call" or not isinstance(call_id, str):+ raise ValueError(f"{path} must be a tool-call block")+ if not isinstance(name, str):+ raise ValueError(f"{path}.name must be a string")+ if not isinstance(arguments, Mapping):+ raise ValueError(f"{path}.arguments must be a mapping")+ return {+ "id": call_id,+ "type": "function",+ "function": {+ "name": name,+ "arguments": json.dumps(arguments),+ },+ }+++def _tool_result(raw_block: object, path: str) -> dict[str, object]:+ block = _mapping(raw_block, path)+ call_id = block.get("tool_call_id")+ if block.get("type") != "tool_result" or not isinstance(call_id, str):+ raise ValueError(f"{path} must be a tool-result block")+ content = _sequence(block.get("content"), f"{path}.content")+ text = [+ _text_block(raw_content, f"{path}.content[{index}]")["text"]+ for index, raw_content in enumerate(content)+ ]+ return {"role": "tool", "tool_call_id": call_id, "content": "\n".join(text)}++def _messages(request: Mapping[str, object]) -> list[dict[str, object]]:messages = _sequence(request.get("messages"), "messages")converted: list[dict[str, object]] = []@@ -74,22 +116,88 @@ def _messages(request: Mapping[str, object]) -> list[dict[str, object]]:message_path = f"messages[{message_index}]"message = _mapping(raw_message, message_path)role = message.get("role")- if not isinstance(role, str) or role not in _ROLES:- raise ValueError(f"{message_path}.role is not supported")blocks = _sequence(message.get("content"), f"{message_path}.content")- content: list[dict[str, str]] = []- for block_index, raw_block in enumerate(blocks):- block_path = f"{message_path}.content[{block_index}]"- block = _mapping(raw_block, block_path)- if block.get("type") != "text" or not isinstance(block.get("text"), str):- raise ValueError(f"{block_path} must be a text block")- content.append({"type": "text", "text": cast(str, block["text"])})- converted.append({"role": role, "content": content})+ if role in _TEXT_ROLES:+ content = [+ _text_block(raw_block, f"{message_path}.content[{block_index}]")+ for block_index, raw_block in enumerate(blocks)+ ]+ converted.append({"role": role, "content": content})+ elif role == "assistant":+ content: list[dict[str, str]] = []+ tool_calls: list[dict[str, object]] = []+ for block_index, raw_block in enumerate(blocks):+ block_path = f"{message_path}.content[{block_index}]"+ block = _mapping(raw_block, block_path)+ if block.get("type") == "text":+ content.append(_text_block(block, block_path))+ elif block.get("type") == "tool_call":+ tool_calls.append(_tool_call(block, block_path))+ else:+ raise ValueError(f"{block_path} is not supported")+ converted_message: dict[str, object] = {+ "role": "assistant",+ "content": content or None,+ }+ if tool_calls:+ converted_message["tool_calls"] = tool_calls+ converted.append(converted_message)+ elif role == "tool":+ converted.extend(+ _tool_result(raw_block, f"{message_path}.content[{block_index}]")+ for block_index, raw_block in enumerate(blocks)+ )+ else:+ raise ValueError(f"{message_path}.role is not supported")if not converted:raise ValueError("messages must not be empty")return converted+def _tools(request: Mapping[str, object]) -> list[dict[str, object]]:+ tools = _sequence(request.get("tools", []), "tools")+ converted: list[dict[str, object]] = []+ for index, raw_tool in enumerate(tools):+ path = f"tools[{index}]"+ tool = _mapping(raw_tool, path)+ name = tool.get("name")+ description = tool.get("description")+ parameters = tool.get("parameters")+ strict = tool.get("strict")+ if not isinstance(name, str):+ raise ValueError(f"{path}.name must be a string")+ if description is not None and not isinstance(description, str):+ raise ValueError(f"{path}.description must be a string or null")+ if not isinstance(parameters, Mapping):+ raise ValueError(f"{path}.parameters must be a mapping")+ if strict is not None and not isinstance(strict, bool):+ raise ValueError(f"{path}.strict must be a boolean or null")+ function: dict[str, object] = {"name": name, "parameters": dict(parameters)}+ if description is not None:+ function["description"] = description+ if strict is not None:+ function["strict"] = strict+ converted.append({"type": "function", "function": function})+ return converted+++def _tool_choice(request: Mapping[str, object]) -> object | None:+ value = request.get("tool_choice")+ if value is None:+ return None+ choice = _mapping(value, "tool_choice")+ choice_type = choice.get("type")+ if choice_type in {"auto", "required", "none"}:+ return choice_type+ if choice_type == "tool":+ data = _mapping(choice.get("data"), "tool_choice.data")+ name = data.get("name")+ if not isinstance(name, str):+ raise ValueError("tool_choice.data.name must be a string")+ return {"type": "function", "function": {"name": name}}+ raise ValueError("tool_choice is not supported")++def _optional_mapping(request: Mapping[str, object],field: str,@@ -103,10 +211,7 @@ def _optional_mapping(def _payload(request: Mapping[str, object], model: str) -> dict[str, Any]:if request.get("stream") is True:raise ValueError("stream=True is not supported")- for field in ("instructions", "tools"):- _reject_sequence_payload(request, field)- if request.get("tool_choice") is not None:- raise ValueError("tool_choice is not supported")+ _reject_sequence_payload(request, "instructions")_reject_extensions_payload(request)_reject_preservation_payload(request)@@ -125,6 +230,12 @@ def _payload(request: Mapping[str, object], model: str) -> dict[str, Any]:"messages": _messages(request),"stream": False,}+ tools = _tools(request)+ if tools:+ payload["tools"] = tools+ tool_choice = _tool_choice(request)+ if tool_choice is not None:+ payload["tool_choice"] = tool_choicefor source, target in (("temperature", "temperature"), ("top_p", "top_p")):value = sampling.get(source)if value is not None:@@ -163,7 +274,23 @@ def _response(response: ModelResponse) -> dict[str, object]:raise ValueError("LiteLLM returned no choices")choice = response.choices[0]text = choice.message.content- if not isinstance(text, str) or not text:+ content: list[dict[str, object]] = []+ if isinstance(text, str) and text:+ content.append({"type": "text", "text": text})+ for tool_call in choice.message.tool_calls or []:+ try:+ arguments = json.loads(tool_call.function.arguments)+ except (TypeError, json.JSONDecodeError) as error:+ raise ValueError("LiteLLM returned invalid tool-call arguments") from error+ if not isinstance(arguments, Mapping):+ raise ValueError("LiteLLM returned invalid tool-call arguments")+ content.append({+ "type": "tool_call",+ "id": tool_call.id,+ "name": tool_call.function.name,+ "arguments": arguments,+ })+ if not content:raise ValueError("LiteLLM returned no text content")return {"id": response.id,@@ -171,7 +298,7 @@ def _response(response: ModelResponse) -> dict[str, object]:"outputs": [{"role": "assistant",- "content": [{"type": "text", "text": text}],+ "content": content,"stop_reason": _STOP_REASONS.get(choice.finish_reason, "unknown"),}],@@ -205,6 +332,9 @@ class LiteLLMSyClient:api_base=self._base_url,api_key=self._api_key,num_retries=0,+ # LiteLLM otherwise imports its optional proxy MCP stack for ordinary+ # OpenAI function tools before it determines that they are not MCP tools.+ _skip_mcp_handler=True,allowed_openai_params=["reasoning_effort"],extra_body={"allowed_openai_params": ["reasoning_effort"]},)diff --git a/examples/experimental/litellm/tests/test_client.py b/examples/experimental/litellm/tests/test_client.pyindex b879592cf..5e3654816 100644--- a/examples/experimental/litellm/tests/test_client.py+++ b/examples/experimental/litellm/tests/test_client.py@@ -57,6 +57,26 @@ def gateway_response(model: str = "moonshotai/kimi-k3") -> dict[str, object]:}+def gateway_tool_response() -> dict[str, object]:+ response = gateway_response()+ response["choices"][0]["message"] = {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_2",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest -q"}',+ },+ }+ ],+ }+ response["choices"][0]["finish_reason"] = "tool_calls"+ return response++async def test_call_uses_litellm_async_completion(monkeypatch: pytest.MonkeyPatch,) -> None:@@ -132,6 +152,203 @@ async def test_call_translates_request_and_normalizes_response() -> None:}+async def test_call_translates_function_tool_history() -> None:+ route = respx.post(f"{BASE_URL}/chat/completions").mock(+ return_value=httpx.Response(200, json=gateway_response())+ )+ request = request_body()+ request["messages"] = [+ {+ "role": "user",+ "content": [{"type": "text", "text": "Run the tests."}],+ },+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": {"command": "pytest"},+ }+ ],+ },+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [{"type": "text", "text": "1 failed"}],+ "is_error": True,+ }+ ],+ },+ ]+ request["tools"] = [+ {+ "name": "Bash",+ "description": "Run a shell command.",+ "parameters": {+ "type": "object",+ "properties": {"command": {"type": "string"}},+ "required": ["command"],+ },+ "strict": True,+ }+ ]+ request["tool_choice"] = {"type": "tool", "data": {"name": "Bash"}}++ client = LiteLLMSyClient("fast", base_url=BASE_URL)+ try:+ await client.call(request)+ finally:+ await client.aclose()++ sent = json.loads(route.calls[0].request.content)+ assert sent["messages"] == [+ {+ "role": "user",+ "content": [{"type": "text", "text": "Run the tests."}],+ },+ {+ "role": "assistant",+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command": "pytest"}',+ },+ }+ ],+ },+ {"role": "tool", "tool_call_id": "call_1", "content": "1 failed"},+ ]+ assert sent["tools"] == [+ {+ "type": "function",+ "function": {+ "name": "Bash",+ "description": "Run a shell command.",+ "parameters": {+ "type": "object",+ "properties": {"command": {"type": "string"}},+ "required": ["command"],+ },+ "strict": True,+ },+ }+ ]+ assert sent["tool_choice"] == {+ "type": "function",+ "function": {"name": "Bash"},+ }+++async def test_call_normalizes_a_function_tool_response(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def fake_acompletion(**_: object) -> ModelResponse:+ return ModelResponse(**gateway_tool_response())++ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)+ client = LiteLLMSyClient("fast", base_url=BASE_URL)+ try:+ response = await client.call(request_body())+ finally:+ await client.aclose()++ assert response["outputs"] == [+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_2",+ "name": "Bash",+ "arguments": {"command": "pytest -q"},+ }+ ],+ "stop_reason": "tool_use",+ }+ ]+++async def test_call_rejects_non_mapping_tool_call_arguments(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def fake_acompletion(**_: object) -> ModelResponse:+ return ModelResponse(**gateway_response())++ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)+ request = request_body()+ request["messages"] = [+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": ["pytest"],+ }+ ],+ }+ ]+ client = LiteLLMSyClient("fast", base_url=BASE_URL)+ try:+ with pytest.raises(ValueError, match=r"messages\[0\]\.content\[0\]\.arguments"):+ await client.call(request)+ finally:+ await client.aclose()+++async def test_call_rejects_non_mapping_response_tool_call_arguments(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ payload = gateway_tool_response()+ tool_call = payload["choices"][0]["message"]["tool_calls"][0]+ tool_call["function"]["arguments"] = '["pytest"]'++ async def fake_acompletion(**_: object) -> ModelResponse:+ return ModelResponse(**payload)++ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)+ client = LiteLLMSyClient("fast", base_url=BASE_URL)+ try:+ with pytest.raises(ValueError, match="invalid tool-call arguments"):+ await client.call(request_body())+ finally:+ await client.aclose()++[email protected]("choice", ["auto", "required", "none"])+async def test_call_translates_standard_tool_choice(+ monkeypatch: pytest.MonkeyPatch,+ choice: str,+) -> None:+ captured: dict[str, object] = {}++ async def fake_acompletion(**kwargs: object) -> ModelResponse:+ captured.update(kwargs)+ return ModelResponse(**gateway_response())++ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)+ request = request_body()+ request["tools"] = [{"name": "Bash", "description": None, "parameters": {}}]+ request["tool_choice"] = {"type": choice}+ client = LiteLLMSyClient("fast", base_url=BASE_URL)+ try:+ await client.call(request)+ finally:+ await client.aclose()++ assert captured["tool_choice"] == choice++@pytest.mark.parametrize(("mutate", "match"),[@@ -148,18 +365,42 @@ async def test_call_translates_request_and_normalizes_response() -> None:),"instructions",),- (- lambda body: body["tools"].append(- {"name": "lookup", "description": None, "parameters": {}}- ),- "tools",- ),(lambda body: body["messages"][0]["content"].append({"type": "image", "source": {"type": "url", "data": {"url": "x"}}}),r"messages\[0\]\.content\[1\]",),+ (+ lambda body: body.update(+ messages=[+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [+ {+ "type": "image",+ "source": {+ "type": "url",+ "data": {"url": "x"},+ },+ }+ ],+ "is_error": False,+ }+ ],+ }+ ]+ ),+ r"messages\[0\]\.content\[0\]\.content\[0\]",+ ),+ (+ lambda body: body.update(tool_choice={"type": "raw", "data": {}}),+ "tool_choice",+ ),(lambda body: body["sampling"].update(top_k=4),"sampling.top_k",diff --git a/examples/experimental/litellm/tests/test_e2e.py b/examples/experimental/litellm/tests/test_e2e.pyindex f1dbfb743..d4a733357 100644--- a/examples/experimental/litellm/tests/test_e2e.py+++ b/examples/experimental/litellm/tests/test_e2e.py@@ -93,42 +93,72 @@ def test_paid_e2e_requires_openrouter_key(next(fixture)-def _request() -> dict[str, object]:- return {- "model": "auto",- "messages": [+def _request(*, critical_error: bool = False) -> dict[str, object]:+ messages: list[dict[str, object]] = [+ {+ "role": "user",+ "content": [+ {+ "type": "text",+ "text": "Reply with a short greeting for a Switchyard E2E test.",+ }+ ],+ }+ ]+ if critical_error:+ messages.extend([+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": {"command": "pytest"},+ }+ ],+ },{- "role": "user",+ "role": "tool","content": [{- "type": "text",- "text": "Reply with a short greeting for a Switchyard E2E test.",+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [+ {+ "type": "text",+ "text": "fatal runtime error: out of memory",+ }+ ],+ "is_error": True,}],- }- ],+ },+ ])+ return {+ "model": "auto",+ "messages": messages,"reasoning": {"effort": "low"},"output": {"max_output_tokens": 128},}@pytest.mark.e2e-async def test_random_router_calls_both_real_openrouter_models(+async def test_stage_router_calls_both_real_openrouter_models(litellm_base_url: str,) -> None:strong_client = LiteLLMSyClient("strong", base_url=litellm_base_url)fast_client = LiteLLMSyClient("fast", base_url=litellm_base_url)- targets = [+ router = algorithms.stage_router(LlmTarget("strong", strong_client),LlmTarget("fast", fast_client),- ]+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ )try:- strong_trace, strong_response = await algorithms.random(- targets, weights=[1, 0], seed=42- ).run(_request())- fast_trace, fast_response = await algorithms.random(- targets, weights=[0, 1], seed=42- ).run(_request())+ fast_trace, fast_response = await router.run(_request())+ strong_trace, strong_response = await router.run(_request(critical_error=True))finally:await strong_client.aclose()await fast_client.aclose()diff --git a/examples/experimental/litellm/tests/test_gateway_config.py b/examples/experimental/litellm/tests/test_gateway_config.pyindex b0df7f27c..f70171ca6 100644--- a/examples/experimental/litellm/tests/test_gateway_config.py+++ b/examples/experimental/litellm/tests/test_gateway_config.py@@ -24,6 +24,7 @@ def test_gateway_uses_openrouter_models_and_credential() -> None:"api_key": "os.environ/OPENROUTER_API_KEY",},}+ assert config["litellm_settings"] == {"drop_params": True}compose = yaml.safe_load((PACKAGE_ROOT / "compose.yaml").read_text())assert compose["services"]["litellm"]["environment"] == {diff --git a/examples/experimental/litellm/tests/test_random_routing.py b/examples/experimental/litellm/tests/test_stage_routing.pysimilarity index 51%rename from examples/experimental/litellm/tests/test_random_routing.pyrename to examples/experimental/litellm/tests/test_stage_routing.pyindex 46a4b3ab3..1e8bf4202 100644--- a/examples/experimental/litellm/tests/test_random_routing.py+++ b/examples/experimental/litellm/tests/test_stage_routing.py@@ -12,15 +12,44 @@ from switchyard.libsy import LlmTarget, algorithmsBASE_URL = "http://gateway.test/v1"-def request_body() -> dict[str, object]:- return {- "messages": [+def request_body(*, critical_error: bool = False) -> dict[str, object]:+ messages: list[dict[str, object]] = [+ {+ "role": "user",+ "content": [{"type": "text", "text": "Fix the failing tests."}],+ }+ ]+ if critical_error:+ messages.extend([{- "role": "user",- "content": [{"type": "text", "text": "Route this request."}],- }- ]- }+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": {"command": "pytest"},+ }+ ],+ },+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [+ {+ "type": "text",+ "text": "fatal runtime error: out of memory",+ }+ ],+ "is_error": True,+ }+ ],+ },+ ])+ return {"model": "auto", "messages": messages}def gateway_response(model: str) -> dict[str, object]:@@ -45,7 +74,7 @@ def gateway_response(model: str) -> dict[str, object]:@respx.mock-async def test_random_router_can_drive_both_litellm_targets() -> None:+async def test_stage_router_drives_both_litellm_targets() -> None:seen: list[str] = []def respond(request: httpx.Request) -> httpx.Response:@@ -56,21 +85,24 @@ async def test_random_router_can_drive_both_litellm_targets() -> None:respx.post(f"{BASE_URL}/chat/completions").mock(side_effect=respond)strong_client = LiteLLMSyClient("strong", base_url=BASE_URL)fast_client = LiteLLMSyClient("fast", base_url=BASE_URL)- targets = [+ router = algorithms.stage_router(LlmTarget("strong", strong_client),LlmTarget("fast", fast_client),- ]+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ )try:- strong_router = algorithms.random(targets, weights=[1, 0], seed=42)- fast_router = algorithms.random(targets, weights=[0, 1], seed=42)- strong_decisions, strong_response = await strong_router.run(request_body())- fast_decisions, fast_response = await fast_router.run(request_body())+ fast_decisions, fast_response = await router.run(request_body())+ strong_decisions, strong_response = await router.run(+ request_body(critical_error=True)+ )finally:await strong_client.aclose()await fast_client.aclose()- assert [item["selected_model"] for item in strong_decisions] == ["strong"]assert [item["selected_model"] for item in fast_decisions] == ["fast"]- assert strong_response["outputs"][0]["content"][0]["text"] == "strong"+ assert [item["selected_model"] for item in strong_decisions] == ["strong"]assert fast_response["outputs"][0]["content"][0]["text"] == "fast"- assert seen == ["strong", "fast"]+ assert strong_response["outputs"][0]["content"][0]["text"] == "strong"+ assert seen == ["fast", "strong"]