Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(litellm): use stage router integration (#226)

Dhruv Nandakumar3acf3d86f10f
diff --git a/examples/experimental/litellm/README.md b/examples/experimental/litellm/README.md
index 23b4196e0..546a0adca 100644
--- a/examples/experimental/litellm/README.md
+++ b/examples/experimental/litellm/README.md
@@ -9,7 +9,8 @@ Compose configuration, and example code are intended as a copyable starting
point for your own application.
LiteLLM provides the OpenAI-compatible gateway, model aliases, and OpenRouter
-provider integration. Switchyard makes the routing decision.
+provider integration. Switchyard's Stage router makes the routing decision from
+the coding agent's recent tool history.
`LiteLLMSyClient` connects Switchyard's normalized libsy requests to the
selected alias through LiteLLM's asynchronous Completion API and the
Dockerized gateway. Together, they let an application keep routing policy in
@@ -19,15 +20,18 @@ through OpenRouter.
## Request flow
```text
-your application → libsy normalized request → Switchyard router
+your application → libsy normalized request → Switchyard Stage router
→ LiteLLMSyClient → LiteLLM async Completion API
→ Dockerized gateway alias → OpenRouter Chat Completions
→ selected model
```
-The bundled router selects either `strong` or `fast`; the client asks LiteLLM's
-Completion API to call that alias through the gateway rather than embedding a
-provider model ID in application code.
+The bundled router treats `strong` as the capable tier and `fast` as the
+efficient tier. It uses the `efficient_first` picker, so an initial request or
+an ambiguous turn falls open to `fast`. Decisive error-recovery signals, such as
+a critical failed tool result, route the turn to `strong`. The client asks
+LiteLLM's Completion API to call the selected alias through the gateway rather
+than embedding a provider model ID in application code.
## Quick start
@@ -89,8 +93,9 @@ curl -fsS http://127.0.0.1:4000/health/liveliness
uv run python example.py
```
-`weights=[1, 3]` means 25% `strong` and 75% `fast` over many calls. `seed=42`
-makes the sequence repeatable.
+The bundled request includes an out-of-memory tool failure, so the Stage router
+selects the capable `strong` alias. Remove the assistant tool call and tool
+result to see the same router fall open to `fast`.
## Use in your application
@@ -110,18 +115,47 @@ async def main() -> None:
"messages": [
{
"role": "user",
- "content": [{"type": "text", "text": "Say hello briefly."}],
- }
+ "content": [{"type": "text", "text": "Fix the failing tests."}],
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": {"command": "pytest"},
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [
+ {
+ "type": "text",
+ "text": "fatal runtime error: out of memory",
+ }
+ ],
+ "is_error": True,
+ }
+ ],
+ },
],
"reasoning": {"effort": "low"},
"output": {"max_output_tokens": 64},
}
strong = LiteLLMSyClient("strong")
fast = LiteLLMSyClient("fast")
- router = algorithms.random(
- [LlmTarget("strong", strong), LlmTarget("fast", fast)],
- weights=[1, 3],
- seed=42,
+ router = algorithms.stage_router(
+ LlmTarget("strong", strong),
+ LlmTarget("fast", fast),
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
)
try:
decisions, response = await router.run(request)
@@ -137,14 +171,23 @@ asyncio.run(main())
## Known limitations
-This example intentionally supports buffered, text-only Chat Completions.
-Tools, media, structured output, and streaming fail explicitly.
+This example intentionally supports buffered text and OpenAI-compatible
+function-tool traffic. It translates normalized function definitions, tool
+choices, assistant tool calls, and text-only tool results into gateway requests,
+and normalizes returned function calls for `libsy`.
+Media, instructions, non-text tool results, structured output, provider-specific
+extensions, preserved raw payloads, and streaming fail explicitly.
The pinned LiteLLM release does not yet recognize Kimi K3's current
`reasoning_effort` support. `LiteLLMSyClient` forwards LiteLLM's
`allowed_openai_params` hint to the gateway so OpenRouter receives that
supported parameter. Recheck this compatibility hint when upgrading LiteLLM.
+LiteLLM 1.92 enters its optional proxy MCP path before distinguishing ordinary
+OpenAI function tools. `LiteLLMSyClient` disables that bridge for these calls so
+the integration does not need LiteLLM's proxy dependencies. Recheck this pinned
+compatibility behavior when upgrading LiteLLM.
+
## Test
From the package directory, run the offline tests without provider calls:
@@ -154,7 +197,8 @@ PYTHONPATH=. uv run --project . pytest tests -m "not e2e" -v
```
The E2E test starts its own LiteLLM gateway and makes two paid OpenRouter Chat
-Completions calls, one to each configured model.
+Completions calls through one Stage router. A no-signal request falls open to
+`fast`; a critical failed-tool request routes to `strong`.
`SWITCHYARD_LITELLM_E2E=1` is the explicit spend opt-in:
```bash
@@ -184,7 +228,7 @@ bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
--server-config examples/experimental/litellm/benchmark-route.toml \
--task-list-file examples/experimental/litellm/benchmark-tasks.txt \
- --model litellm-random \
+ --model litellm-stage \
--agent codex \
--reasoning-effort low \
--n-tasks 3 \
@@ -194,9 +238,14 @@ bash benchmark/run-baseline.sh \
--foreground
```
-The sample runs `broken-python`, `cosign-keyless-signing`, and
-`jq-data-processing`. Three autonomous coding-agent tasks can make many paid
-model calls. Inspect the resulting run under `benchmark/tb_runs/`, including
+The native benchmark route uses the same efficient-first Stage policy but does
+not use `LiteLLMSyClient`; Switchyard server translates requests directly to
+the shared LiteLLM gateway. The gateway drops parameters unsupported by the
+selected provider model, so the benchmark's reasoning effort reaches `strong`
+without causing `fast` requests to fail. The sample runs `broken-python`,
+`cosign-keyless-signing`, and `jq-data-processing`. Three autonomous coding-agent
+tasks can make many paid model calls. Inspect the resulting run under
+`benchmark/tb_runs/`, including
`run_manifest.json`, `harbor.log`, `routing_stats_final.json`, and
`jobs/<job>/<task>/agent/trajectory.json`.
@@ -226,7 +275,7 @@ bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
--server-config examples/experimental/litellm/benchmark-route.toml \
--task-list-file examples/experimental/litellm/benchmark-tasks.txt \
- --model litellm-random \
+ --model litellm-stage \
--agent codex \
--reasoning-effort low \
--n-tasks 3 \
@@ -287,4 +336,4 @@ deployment guidance before exposing it remotely.
- [GPT-5.6 Sol on OpenRouter](https://openrouter.ai/openai/gpt-5.6-sol)
- [Kimi K3 on OpenRouter](https://openrouter.ai/moonshotai/kimi-k3-20260715)
- [LiteLLM gateway quick start](https://docs.litellm.ai/docs/proxy/quick_start)
-- [Switchyard random-routing docs](../../../docs/routing_algorithms/random_routing.md)
+- [Switchyard Stage-router docs](../../../docs/routing_algorithms/stage_router_routing.md)
diff --git a/examples/experimental/litellm/benchmark-route.toml b/examples/experimental/litellm/benchmark-route.toml
index 1ef063bd6..10c7413a4 100644
--- a/examples/experimental/litellm/benchmark-route.toml
+++ b/examples/experimental/litellm/benchmark-route.toml
@@ -15,9 +15,11 @@ llm_client = "litellm"
id = "fast"
llm_client = "litellm"
-[routes.litellm_random]
-id = "litellm-random"
-type = "random"
-targets = ["strong", "fast"]
-weights = [0.5, 0.5]
-seed = 42
+[routes.litellm_stage]
+id = "litellm-stage"
+type = "stage_router"
+capable_target = "strong"
+efficient_target = "fast"
+picker = "efficient_first"
+confidence_threshold = 0.5
+recent_turn_window = 3
diff --git a/examples/experimental/litellm/example.py b/examples/experimental/litellm/example.py
index 362f21a6c..bb67c27ea 100644
--- a/examples/experimental/litellm/example.py
+++ b/examples/experimental/litellm/example.py
@@ -2,7 +2,7 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
-"""Route one text request through LiteLLM with libsy's random algorithm."""
+"""Route one coding-agent turn through LiteLLM with libsy's Stage router."""
import asyncio
@@ -12,7 +12,7 @@ from switchyard.libsy import LlmTarget, algorithms
def sy_request() -> dict[str, object]:
- """Build a normalized libsy text request."""
+ """Build a normalized turn whose critical tool failure needs the capable tier."""
return {
"model": "auto",
"messages": [
@@ -21,10 +21,37 @@ def sy_request() -> dict[str, object]:
"content": [
{
"type": "text",
- "text": "Explain weighted random LLM routing in one sentence.",
+ "text": "Fix the failing tests.",
}
],
- }
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": {"command": "pytest"},
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [
+ {
+ "type": "text",
+ "text": "fatal runtime error: out of memory",
+ }
+ ],
+ "is_error": True,
+ }
+ ],
+ },
],
"reasoning": {"effort": "low"},
"output": {"max_output_tokens": 96},
@@ -32,20 +59,19 @@ def sy_request() -> dict[str, object]:
async def main() -> None:
- """Run the weighted router and print its normalized result."""
+ """Run the Stage router and print its normalized result."""
strong_client = LiteLLMSyClient("strong")
fast_client = LiteLLMSyClient("fast")
- router = algorithms.random(
- [
- LlmTarget("strong", strong_client),
- LlmTarget("fast", fast_client),
- ],
- weights=[1, 3],
- seed=42,
+ router = algorithms.stage_router(
+ LlmTarget("strong", strong_client),
+ LlmTarget("fast", fast_client),
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
)
try:
decisions, response = await router.run(sy_request())
- print("Random:", decisions, response)
+ print("Stage router:", decisions, response)
finally:
await strong_client.aclose()
await fast_client.aclose()
diff --git a/examples/experimental/litellm/litellm-config.yaml b/examples/experimental/litellm/litellm-config.yaml
index 83f32d0f9..8ab38a113 100644
--- a/examples/experimental/litellm/litellm-config.yaml
+++ b/examples/experimental/litellm/litellm-config.yaml
@@ -7,3 +7,6 @@ model_list:
litellm_params:
model: openrouter/moonshotai/kimi-k3
api_key: os.environ/OPENROUTER_API_KEY
+
+litellm_settings:
+ drop_params: true
diff --git a/examples/experimental/litellm/src/switchyard_litellm/client.py b/examples/experimental/litellm/src/switchyard_litellm/client.py
index d61e97f1c..dc0286c1f 100644
--- a/examples/experimental/litellm/src/switchyard_litellm/client.py
+++ b/examples/experimental/litellm/src/switchyard_litellm/client.py
@@ -5,12 +5,13 @@
from __future__ import annotations
+import json
from collections.abc import Mapping, Sequence
from typing import Any, cast
from litellm import ModelResponse, acompletion
-_ROLES = {"system", "developer", "user", "assistant"}
+_TEXT_ROLES = {"system", "developer", "user"}
_STOP_REASONS = {
"stop": "end_turn",
"length": "max_tokens",
@@ -67,6 +68,47 @@ def _reject_preservation_payload(request: Mapping[str, object]) -> None:
raise ValueError("preservation is not supported")
+def _text_block(raw_block: object, path: str) -> dict[str, str]:
+ block = _mapping(raw_block, path)
+ if block.get("type") != "text" or not isinstance(block.get("text"), str):
+ raise ValueError(f"{path} must be a text block")
+ return {"type": "text", "text": cast(str, block["text"])}
+
+
+def _tool_call(raw_block: object, path: str) -> dict[str, object]:
+ block = _mapping(raw_block, path)
+ call_id = block.get("id")
+ name = block.get("name")
+ arguments = block.get("arguments")
+ if block.get("type") != "tool_call" or not isinstance(call_id, str):
+ raise ValueError(f"{path} must be a tool-call block")
+ if not isinstance(name, str):
+ raise ValueError(f"{path}.name must be a string")
+ if not isinstance(arguments, Mapping):
+ raise ValueError(f"{path}.arguments must be a mapping")
+ return {
+ "id": call_id,
+ "type": "function",
+ "function": {
+ "name": name,
+ "arguments": json.dumps(arguments),
+ },
+ }
+
+
+def _tool_result(raw_block: object, path: str) -> dict[str, object]:
+ block = _mapping(raw_block, path)
+ call_id = block.get("tool_call_id")
+ if block.get("type") != "tool_result" or not isinstance(call_id, str):
+ raise ValueError(f"{path} must be a tool-result block")
+ content = _sequence(block.get("content"), f"{path}.content")
+ text = [
+ _text_block(raw_content, f"{path}.content[{index}]")["text"]
+ for index, raw_content in enumerate(content)
+ ]
+ return {"role": "tool", "tool_call_id": call_id, "content": "\n".join(text)}
+
+
def _messages(request: Mapping[str, object]) -> list[dict[str, object]]:
messages = _sequence(request.get("messages"), "messages")
converted: list[dict[str, object]] = []
@@ -74,22 +116,88 @@ def _messages(request: Mapping[str, object]) -> list[dict[str, object]]:
message_path = f"messages[{message_index}]"
message = _mapping(raw_message, message_path)
role = message.get("role")
- if not isinstance(role, str) or role not in _ROLES:
- raise ValueError(f"{message_path}.role is not supported")
blocks = _sequence(message.get("content"), f"{message_path}.content")
- content: list[dict[str, str]] = []
- for block_index, raw_block in enumerate(blocks):
- block_path = f"{message_path}.content[{block_index}]"
- block = _mapping(raw_block, block_path)
- if block.get("type") != "text" or not isinstance(block.get("text"), str):
- raise ValueError(f"{block_path} must be a text block")
- content.append({"type": "text", "text": cast(str, block["text"])})
- converted.append({"role": role, "content": content})
+ if role in _TEXT_ROLES:
+ content = [
+ _text_block(raw_block, f"{message_path}.content[{block_index}]")
+ for block_index, raw_block in enumerate(blocks)
+ ]
+ converted.append({"role": role, "content": content})
+ elif role == "assistant":
+ content: list[dict[str, str]] = []
+ tool_calls: list[dict[str, object]] = []
+ for block_index, raw_block in enumerate(blocks):
+ block_path = f"{message_path}.content[{block_index}]"
+ block = _mapping(raw_block, block_path)
+ if block.get("type") == "text":
+ content.append(_text_block(block, block_path))
+ elif block.get("type") == "tool_call":
+ tool_calls.append(_tool_call(block, block_path))
+ else:
+ raise ValueError(f"{block_path} is not supported")
+ converted_message: dict[str, object] = {
+ "role": "assistant",
+ "content": content or None,
+ }
+ if tool_calls:
+ converted_message["tool_calls"] = tool_calls
+ converted.append(converted_message)
+ elif role == "tool":
+ converted.extend(
+ _tool_result(raw_block, f"{message_path}.content[{block_index}]")
+ for block_index, raw_block in enumerate(blocks)
+ )
+ else:
+ raise ValueError(f"{message_path}.role is not supported")
if not converted:
raise ValueError("messages must not be empty")
return converted
+def _tools(request: Mapping[str, object]) -> list[dict[str, object]]:
+ tools = _sequence(request.get("tools", []), "tools")
+ converted: list[dict[str, object]] = []
+ for index, raw_tool in enumerate(tools):
+ path = f"tools[{index}]"
+ tool = _mapping(raw_tool, path)
+ name = tool.get("name")
+ description = tool.get("description")
+ parameters = tool.get("parameters")
+ strict = tool.get("strict")
+ if not isinstance(name, str):
+ raise ValueError(f"{path}.name must be a string")
+ if description is not None and not isinstance(description, str):
+ raise ValueError(f"{path}.description must be a string or null")
+ if not isinstance(parameters, Mapping):
+ raise ValueError(f"{path}.parameters must be a mapping")
+ if strict is not None and not isinstance(strict, bool):
+ raise ValueError(f"{path}.strict must be a boolean or null")
+ function: dict[str, object] = {"name": name, "parameters": dict(parameters)}
+ if description is not None:
+ function["description"] = description
+ if strict is not None:
+ function["strict"] = strict
+ converted.append({"type": "function", "function": function})
+ return converted
+
+
+def _tool_choice(request: Mapping[str, object]) -> object | None:
+ value = request.get("tool_choice")
+ if value is None:
+ return None
+ choice = _mapping(value, "tool_choice")
+ choice_type = choice.get("type")
+ if choice_type in {"auto", "required", "none"}:
+ return choice_type
+ if choice_type == "tool":
+ data = _mapping(choice.get("data"), "tool_choice.data")
+ name = data.get("name")
+ if not isinstance(name, str):
+ raise ValueError("tool_choice.data.name must be a string")
+ return {"type": "function", "function": {"name": name}}
+ raise ValueError("tool_choice is not supported")
+
+
def _optional_mapping(
request: Mapping[str, object],
field: str,
@@ -103,10 +211,7 @@ def _optional_mapping(
def _payload(request: Mapping[str, object], model: str) -> dict[str, Any]:
if request.get("stream") is True:
raise ValueError("stream=True is not supported")
- for field in ("instructions", "tools"):
- _reject_sequence_payload(request, field)
- if request.get("tool_choice") is not None:
- raise ValueError("tool_choice is not supported")
+ _reject_sequence_payload(request, "instructions")
_reject_extensions_payload(request)
_reject_preservation_payload(request)
@@ -125,6 +230,12 @@ def _payload(request: Mapping[str, object], model: str) -> dict[str, Any]:
"messages": _messages(request),
"stream": False,
}
+ tools = _tools(request)
+ if tools:
+ payload["tools"] = tools
+ tool_choice = _tool_choice(request)
+ if tool_choice is not None:
+ payload["tool_choice"] = tool_choice
for source, target in (("temperature", "temperature"), ("top_p", "top_p")):
value = sampling.get(source)
if value is not None:
@@ -163,7 +274,23 @@ def _response(response: ModelResponse) -> dict[str, object]:
raise ValueError("LiteLLM returned no choices")
choice = response.choices[0]
text = choice.message.content
- if not isinstance(text, str) or not text:
+ content: list[dict[str, object]] = []
+ if isinstance(text, str) and text:
+ content.append({"type": "text", "text": text})
+ for tool_call in choice.message.tool_calls or []:
+ try:
+ arguments = json.loads(tool_call.function.arguments)
+ except (TypeError, json.JSONDecodeError) as error:
+ raise ValueError("LiteLLM returned invalid tool-call arguments") from error
+ if not isinstance(arguments, Mapping):
+ raise ValueError("LiteLLM returned invalid tool-call arguments")
+ content.append({
+ "type": "tool_call",
+ "id": tool_call.id,
+ "name": tool_call.function.name,
+ "arguments": arguments,
+ })
+ if not content:
raise ValueError("LiteLLM returned no text content")
return {
"id": response.id,
@@ -171,7 +298,7 @@ def _response(response: ModelResponse) -> dict[str, object]:
"outputs": [
{
"role": "assistant",
- "content": [{"type": "text", "text": text}],
+ "content": content,
"stop_reason": _STOP_REASONS.get(choice.finish_reason, "unknown"),
}
],
@@ -205,6 +332,9 @@ class LiteLLMSyClient:
api_base=self._base_url,
api_key=self._api_key,
num_retries=0,
+ # LiteLLM otherwise imports its optional proxy MCP stack for ordinary
+ # OpenAI function tools before it determines that they are not MCP tools.
+ _skip_mcp_handler=True,
allowed_openai_params=["reasoning_effort"],
extra_body={"allowed_openai_params": ["reasoning_effort"]},
)
diff --git a/examples/experimental/litellm/tests/test_client.py b/examples/experimental/litellm/tests/test_client.py
index b879592cf..5e3654816 100644
--- a/examples/experimental/litellm/tests/test_client.py
+++ b/examples/experimental/litellm/tests/test_client.py
@@ -57,6 +57,26 @@ def gateway_response(model: str = "moonshotai/kimi-k3") -> dict[str, object]:
}
+def gateway_tool_response() -> dict[str, object]:
+ response = gateway_response()
+ response["choices"][0]["message"] = {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_2",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest -q"}',
+ },
+ }
+ ],
+ }
+ response["choices"][0]["finish_reason"] = "tool_calls"
+ return response
+
+
async def test_call_uses_litellm_async_completion(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -132,6 +152,203 @@ async def test_call_translates_request_and_normalizes_response() -> None:
}
+async def test_call_translates_function_tool_history() -> None:
+ route = respx.post(f"{BASE_URL}/chat/completions").mock(
+ return_value=httpx.Response(200, json=gateway_response())
+ )
+ request = request_body()
+ request["messages"] = [
+ {
+ "role": "user",
+ "content": [{"type": "text", "text": "Run the tests."}],
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": {"command": "pytest"},
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [{"type": "text", "text": "1 failed"}],
+ "is_error": True,
+ }
+ ],
+ },
+ ]
+ request["tools"] = [
+ {
+ "name": "Bash",
+ "description": "Run a shell command.",
+ "parameters": {
+ "type": "object",
+ "properties": {"command": {"type": "string"}},
+ "required": ["command"],
+ },
+ "strict": True,
+ }
+ ]
+ request["tool_choice"] = {"type": "tool", "data": {"name": "Bash"}}
+
+ client = LiteLLMSyClient("fast", base_url=BASE_URL)
+ try:
+ await client.call(request)
+ finally:
+ await client.aclose()
+
+ sent = json.loads(route.calls[0].request.content)
+ assert sent["messages"] == [
+ {
+ "role": "user",
+ "content": [{"type": "text", "text": "Run the tests."}],
+ },
+ {
+ "role": "assistant",
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command": "pytest"}',
+ },
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "call_1", "content": "1 failed"},
+ ]
+ assert sent["tools"] == [
+ {
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "description": "Run a shell command.",
+ "parameters": {
+ "type": "object",
+ "properties": {"command": {"type": "string"}},
+ "required": ["command"],
+ },
+ "strict": True,
+ },
+ }
+ ]
+ assert sent["tool_choice"] == {
+ "type": "function",
+ "function": {"name": "Bash"},
+ }
+
+
+async def test_call_normalizes_a_function_tool_response(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def fake_acompletion(**_: object) -> ModelResponse:
+ return ModelResponse(**gateway_tool_response())
+
+ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
+ client = LiteLLMSyClient("fast", base_url=BASE_URL)
+ try:
+ response = await client.call(request_body())
+ finally:
+ await client.aclose()
+
+ assert response["outputs"] == [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_2",
+ "name": "Bash",
+ "arguments": {"command": "pytest -q"},
+ }
+ ],
+ "stop_reason": "tool_use",
+ }
+ ]
+
+
+async def test_call_rejects_non_mapping_tool_call_arguments(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def fake_acompletion(**_: object) -> ModelResponse:
+ return ModelResponse(**gateway_response())
+
+ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
+ request = request_body()
+ request["messages"] = [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": ["pytest"],
+ }
+ ],
+ }
+ ]
+ client = LiteLLMSyClient("fast", base_url=BASE_URL)
+ try:
+ with pytest.raises(ValueError, match=r"messages\[0\]\.content\[0\]\.arguments"):
+ await client.call(request)
+ finally:
+ await client.aclose()
+
+
+async def test_call_rejects_non_mapping_response_tool_call_arguments(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ payload = gateway_tool_response()
+ tool_call = payload["choices"][0]["message"]["tool_calls"][0]
+ tool_call["function"]["arguments"] = '["pytest"]'
+
+ async def fake_acompletion(**_: object) -> ModelResponse:
+ return ModelResponse(**payload)
+
+ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
+ client = LiteLLMSyClient("fast", base_url=BASE_URL)
+ try:
+ with pytest.raises(ValueError, match="invalid tool-call arguments"):
+ await client.call(request_body())
+ finally:
+ await client.aclose()
+
+
[email protected]("choice", ["auto", "required", "none"])
+async def test_call_translates_standard_tool_choice(
+ monkeypatch: pytest.MonkeyPatch,
+ choice: str,
+) -> None:
+ captured: dict[str, object] = {}
+
+ async def fake_acompletion(**kwargs: object) -> ModelResponse:
+ captured.update(kwargs)
+ return ModelResponse(**gateway_response())
+
+ monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
+ request = request_body()
+ request["tools"] = [{"name": "Bash", "description": None, "parameters": {}}]
+ request["tool_choice"] = {"type": choice}
+ client = LiteLLMSyClient("fast", base_url=BASE_URL)
+ try:
+ await client.call(request)
+ finally:
+ await client.aclose()
+
+ assert captured["tool_choice"] == choice
+
+
@pytest.mark.parametrize(
("mutate", "match"),
[
@@ -148,18 +365,42 @@ async def test_call_translates_request_and_normalizes_response() -> None:
),
"instructions",
),
- (
- lambda body: body["tools"].append(
- {"name": "lookup", "description": None, "parameters": {}}
- ),
- "tools",
- ),
(
lambda body: body["messages"][0]["content"].append(
{"type": "image", "source": {"type": "url", "data": {"url": "x"}}}
),
r"messages\[0\]\.content\[1\]",
),
+ (
+ lambda body: body.update(
+ messages=[
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "data": {"url": "x"},
+ },
+ }
+ ],
+ "is_error": False,
+ }
+ ],
+ }
+ ]
+ ),
+ r"messages\[0\]\.content\[0\]\.content\[0\]",
+ ),
+ (
+ lambda body: body.update(tool_choice={"type": "raw", "data": {}}),
+ "tool_choice",
+ ),
(
lambda body: body["sampling"].update(top_k=4),
"sampling.top_k",
diff --git a/examples/experimental/litellm/tests/test_e2e.py b/examples/experimental/litellm/tests/test_e2e.py
index f1dbfb743..d4a733357 100644
--- a/examples/experimental/litellm/tests/test_e2e.py
+++ b/examples/experimental/litellm/tests/test_e2e.py
@@ -93,42 +93,72 @@ def test_paid_e2e_requires_openrouter_key(
next(fixture)
-def _request() -> dict[str, object]:
- return {
- "model": "auto",
- "messages": [
+def _request(*, critical_error: bool = False) -> dict[str, object]:
+ messages: list[dict[str, object]] = [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": "Reply with a short greeting for a Switchyard E2E test.",
+ }
+ ],
+ }
+ ]
+ if critical_error:
+ messages.extend([
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": {"command": "pytest"},
+ }
+ ],
+ },
{
- "role": "user",
+ "role": "tool",
"content": [
{
- "type": "text",
- "text": "Reply with a short greeting for a Switchyard E2E test.",
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [
+ {
+ "type": "text",
+ "text": "fatal runtime error: out of memory",
+ }
+ ],
+ "is_error": True,
}
],
- }
- ],
+ },
+ ])
+ return {
+ "model": "auto",
+ "messages": messages,
"reasoning": {"effort": "low"},
"output": {"max_output_tokens": 128},
}
@pytest.mark.e2e
-async def test_random_router_calls_both_real_openrouter_models(
+async def test_stage_router_calls_both_real_openrouter_models(
litellm_base_url: str,
) -> None:
strong_client = LiteLLMSyClient("strong", base_url=litellm_base_url)
fast_client = LiteLLMSyClient("fast", base_url=litellm_base_url)
- targets = [
+ router = algorithms.stage_router(
LlmTarget("strong", strong_client),
LlmTarget("fast", fast_client),
- ]
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ )
try:
- strong_trace, strong_response = await algorithms.random(
- targets, weights=[1, 0], seed=42
- ).run(_request())
- fast_trace, fast_response = await algorithms.random(
- targets, weights=[0, 1], seed=42
- ).run(_request())
+ fast_trace, fast_response = await router.run(_request())
+ strong_trace, strong_response = await router.run(_request(critical_error=True))
finally:
await strong_client.aclose()
await fast_client.aclose()
diff --git a/examples/experimental/litellm/tests/test_gateway_config.py b/examples/experimental/litellm/tests/test_gateway_config.py
index b0df7f27c..f70171ca6 100644
--- a/examples/experimental/litellm/tests/test_gateway_config.py
+++ b/examples/experimental/litellm/tests/test_gateway_config.py
@@ -24,6 +24,7 @@ def test_gateway_uses_openrouter_models_and_credential() -> None:
"api_key": "os.environ/OPENROUTER_API_KEY",
},
}
+ assert config["litellm_settings"] == {"drop_params": True}
compose = yaml.safe_load((PACKAGE_ROOT / "compose.yaml").read_text())
assert compose["services"]["litellm"]["environment"] == {
diff --git a/examples/experimental/litellm/tests/test_random_routing.py b/examples/experimental/litellm/tests/test_stage_routing.py
similarity index 51%
rename from examples/experimental/litellm/tests/test_random_routing.py
rename to examples/experimental/litellm/tests/test_stage_routing.py
index 46a4b3ab3..1e8bf4202 100644
--- a/examples/experimental/litellm/tests/test_random_routing.py
+++ b/examples/experimental/litellm/tests/test_stage_routing.py
@@ -12,15 +12,44 @@ from switchyard.libsy import LlmTarget, algorithms
BASE_URL = "http://gateway.test/v1"
-def request_body() -> dict[str, object]:
- return {
- "messages": [
+def request_body(*, critical_error: bool = False) -> dict[str, object]:
+ messages: list[dict[str, object]] = [
+ {
+ "role": "user",
+ "content": [{"type": "text", "text": "Fix the failing tests."}],
+ }
+ ]
+ if critical_error:
+ messages.extend([
{
- "role": "user",
- "content": [{"type": "text", "text": "Route this request."}],
- }
- ]
- }
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": {"command": "pytest"},
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [
+ {
+ "type": "text",
+ "text": "fatal runtime error: out of memory",
+ }
+ ],
+ "is_error": True,
+ }
+ ],
+ },
+ ])
+ return {"model": "auto", "messages": messages}
def gateway_response(model: str) -> dict[str, object]:
@@ -45,7 +74,7 @@ def gateway_response(model: str) -> dict[str, object]:
@respx.mock
-async def test_random_router_can_drive_both_litellm_targets() -> None:
+async def test_stage_router_drives_both_litellm_targets() -> None:
seen: list[str] = []
def respond(request: httpx.Request) -> httpx.Response:
@@ -56,21 +85,24 @@ async def test_random_router_can_drive_both_litellm_targets() -> None:
respx.post(f"{BASE_URL}/chat/completions").mock(side_effect=respond)
strong_client = LiteLLMSyClient("strong", base_url=BASE_URL)
fast_client = LiteLLMSyClient("fast", base_url=BASE_URL)
- targets = [
+ router = algorithms.stage_router(
LlmTarget("strong", strong_client),
LlmTarget("fast", fast_client),
- ]
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ )
try:
- strong_router = algorithms.random(targets, weights=[1, 0], seed=42)
- fast_router = algorithms.random(targets, weights=[0, 1], seed=42)
- strong_decisions, strong_response = await strong_router.run(request_body())
- fast_decisions, fast_response = await fast_router.run(request_body())
+ fast_decisions, fast_response = await router.run(request_body())
+ strong_decisions, strong_response = await router.run(
+ request_body(critical_error=True)
+ )
finally:
await strong_client.aclose()
await fast_client.aclose()
- assert [item["selected_model"] for item in strong_decisions] == ["strong"]
assert [item["selected_model"] for item in fast_decisions] == ["fast"]
- assert strong_response["outputs"][0]["content"][0]["text"] == "strong"
+ assert [item["selected_model"] for item in strong_decisions] == ["strong"]
assert fast_response["outputs"][0]["content"][0]["text"] == "fast"
- assert seen == ["strong", "fast"]
+ assert strong_response["outputs"][0]["content"][0]["text"] == "strong"
+ assert seen == ["fast", "strong"]