Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

refactor(litellm): replace client integration with routing plugin (#532)

Dhruv Nandakumard5aa63383a7f
diff --git a/.dockerignore b/.dockerignore
index 21ce6789e..cd04b9ce0 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -1,4 +1,10 @@
.git
+.env
+**/.env
+.env.*
+**/.env.*
+!.env.example
+!**/.env.example
.mypy_cache
.pytest_cache
.ruff_cache
diff --git a/docs/routing_algorithms/random_routing.md b/docs/routing_algorithms/random_routing.md
index e889afce2..a90422fc5 100644
--- a/docs/routing_algorithms/random_routing.md
+++ b/docs/routing_algorithms/random_routing.md
@@ -99,7 +99,7 @@ curl http://localhost:4000/v1/chat/completions \
## Experimental LiteLLM integration
The package's
-[developer guide](../../examples/experimental/litellm/README.md) is the
+[developer guide](../../examples/litellm/README.md) is the
canonical documentation for integrating Switchyard with a Dockerized LiteLLM
gateway. Its source-installable client adapts libsy's dictionary contract and
demonstrates weighted random routing between model aliases backed by
diff --git a/examples/experimental/litellm/.env.example b/examples/experimental/litellm/.env.example
deleted file mode 100644
index 9b229f342..000000000
--- a/examples/experimental/litellm/.env.example
+++ /dev/null
@@ -1 +0,0 @@
-OPENROUTER_API_KEY=
diff --git a/examples/experimental/litellm/README.md b/examples/experimental/litellm/README.md
deleted file mode 100644
index 084ce55ec..000000000
--- a/examples/experimental/litellm/README.md
+++ /dev/null
@@ -1,357 +0,0 @@
-# Switchyard + LiteLLM
-
-> **Experimental integration:** This example and its Python APIs are
-> experimental and may change without notice.
-
-Build on this package when you want to put Switchyard routing in front of a
-LiteLLM gateway. It demonstrates the integration end to end, and its client,
-Compose configuration, and example code are intended as a copyable starting
-point for your own application.
-
-LiteLLM provides the OpenAI-compatible gateway, model aliases, and OpenRouter
-provider integration. Switchyard's Stage router makes the routing decision from
-the coding agent's recent tool history.
-`LiteLLMSyClient` connects Switchyard's normalized libsy requests to their
-selected aliases through LiteLLM's asynchronous Completion API and the
-Dockerized gateway. Together, they let an application keep routing policy in
-Switchyard while LiteLLM owns model access and sends Chat Completions inference
-through OpenRouter.
-
-## Request flow
-
-```text
-your application → libsy normalized request → Switchyard Stage router
- → LiteLLMSyClient → LiteLLM async Completion API
- → Dockerized gateway alias → OpenRouter Chat Completions
- → selected model
-```
-
-The bundled router treats `strong` as the capable tier and `fast` as the
-efficient tier. It uses the `efficient_first` picker, so an initial request or
-an ambiguous turn falls open to `fast`. Decisive error-recovery signals, such as
-a critical failed tool result, route the turn to `strong`. The client asks
-LiteLLM's Completion API to call the selected alias through the gateway rather
-than embedding a provider model ID in application code.
-
-## Quick start
-
-This example uses these pinned gateway aliases and image:
-
-- `strong` maps to OpenRouter's `openai/gpt-5.6-sol` model.
-- `fast` maps to OpenRouter's `moonshotai/kimi-k3` model.
-- The Python client pins `litellm==1.92.0`.
-- The gateway runs `ghcr.io/berriai/litellm:v1.92.0`.
-
-### Prerequisites
-
-You need Python 3.12, [uv](https://docs.astral.sh/uv/), Docker Compose, an
-`OPENROUTER_API_KEY`, and OpenRouter account access to both model IDs. The
-example intentionally standardizes on Python 3.12; the pinned LiteLLM release
-cannot build on Python 3.14.
-
-### Configure
-
-From the repository root:
-
-```bash
-cd examples/experimental/litellm
-cp .env.example .env
-# Edit .env and set OPENROUTER_API_KEY.
-```
-
-### Install from the checkout
-
-From the package directory:
-
-```bash
-uv sync --locked --python 3.12
-```
-
-### Install the package from source in another uv environment
-
-From the repository root:
-
-```bash
-LITELLM_ENV_DIR="$(mktemp -d "${TMPDIR:-/tmp}/switchyard-litellm-env.XXXXXX")"
-uv venv --python 3.12 "$LITELLM_ENV_DIR"
-uv pip install --python "$LITELLM_ENV_DIR/bin/python" ./examples/experimental/litellm
-"$LITELLM_ENV_DIR/bin/python" -c "from switchyard_litellm import LiteLLMSyClient"
-rm -rf -- "$LITELLM_ENV_DIR"
-```
-
-### Start and inspect LiteLLM
-
-From the package directory:
-
-```bash
-docker compose up -d --wait
-docker compose ps
-curl -fsS http://127.0.0.1:4000/health/liveliness
-```
-
-### Run the example
-
-```bash
-uv run --locked --python 3.12 python example.py
-```
-
-The bundled request includes an out-of-memory tool failure, so the Stage router
-selects the capable `strong` alias. Remove the assistant tool call and tool
-result to see the same router fall open to `fast`.
-
-## Use in your application
-
-After installing the package, adapt this program to your application's
-normalized request shape and routing policy:
-
-```python
-import asyncio
-
-from switchyard.libsy import LlmResponse, Step, algorithms
-from switchyard_litellm import LiteLLMSyClient
-
-
-async def main() -> None:
- request = {
- "model": "auto",
- "messages": [
- {
- "role": "user",
- "content": [{"type": "text", "text": "Fix the failing tests."}],
- },
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_1",
- "name": "Bash",
- "arguments": {"command": "pytest"},
- }
- ],
- },
- {
- "role": "tool",
- "content": [
- {
- "type": "tool_result",
- "tool_call_id": "call_1",
- "content": [
- {
- "type": "text",
- "text": "fatal runtime error: out of memory",
- }
- ],
- "is_error": True,
- }
- ],
- },
- ],
- "reasoning": {"effort": "low"},
- "output": {"max_output_tokens": 64},
- }
- client = LiteLLMSyClient()
- router = algorithms.stage_router(
- "strong",
- "fast",
- picker="efficient_first",
- confidence_threshold=0.5,
- recent_window=3,
- )
- try:
- async for step in router.run_stream(request):
- match step:
- case Step.CallModel(call):
- try:
- response = await client.call(call.request)
- except Exception as error:
- call.fail(error)
- else:
- call.respond(LlmResponse.Agg(response))
- case Step.Done(outcome):
- print("Decision:", outcome.selected_model_id)
- match outcome.response:
- case LlmResponse.Agg(response):
- print("Response:", response)
- case LlmResponse.Stream(stream):
- async for event in stream:
- print("Response event:", event)
- case None:
- print("Response:", await client.call(outcome.request))
- finally:
- await client.aclose()
-
-
-asyncio.run(main())
-```
-
-## Known limitations
-
-This example intentionally supports buffered text and OpenAI-compatible
-function-tool traffic. It translates normalized function definitions, tool
-choices, assistant tool calls, and text-only tool results into gateway requests,
-and normalizes returned function calls for `libsy`.
-Media, instructions, non-text tool results, structured output, provider-specific
-extensions, preserved raw payloads, and streaming fail explicitly.
-
-The pinned LiteLLM release does not yet recognize Kimi K3's current
-`reasoning_effort` support. `LiteLLMSyClient` forwards LiteLLM's
-`allowed_openai_params` hint to the gateway so OpenRouter receives that
-supported parameter. Recheck this compatibility hint when upgrading LiteLLM.
-
-LiteLLM 1.92 enters its optional proxy MCP path before distinguishing ordinary
-OpenAI function tools. `LiteLLMSyClient` disables that bridge for these calls so
-the integration does not need LiteLLM's proxy dependencies. Recheck this pinned
-compatibility behavior when upgrading LiteLLM.
-
-## Test
-
-From the package directory, run the offline tests without provider calls:
-
-```bash
-PYTHONPATH=. uv run --project . --locked --python 3.12 \
- pytest tests -m "not e2e" -v
-```
-
-The E2E test starts its own LiteLLM gateway and makes two paid OpenRouter Chat
-Completions calls through one Stage router. A no-signal request falls open to
-`fast`; a critical failed-tool request routes to `strong`.
-`SWITCHYARD_LITELLM_E2E=1` is the explicit spend opt-in:
-
-```bash
-SWITCHYARD_LITELLM_E2E=1 \
-uv run --project . --locked --python 3.12 --env-file .env \
- pytest tests/test_e2e.py -m e2e -v
-```
-
-## Run the optional three-task Harbor smoke benchmark
-
-See the repository's [Harbor benchmark guide](../../../benchmark/README.md) for
-the one-time Harbor patch and dataset preparation. This is a separate test
-boundary: the package E2E tests `LiteLLMSyClient`, while Harbor tests
-`Harbor → Switchyard server → LiteLLM → OpenRouter`.
-
-From the repository root, start the example gateway:
-
-```bash
-docker compose --env-file examples/experimental/litellm/.env \
- -f examples/experimental/litellm/compose.yaml up -d --wait
-```
-
-Then run the three-task smoke benchmark in the foreground:
-
-```bash
-SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \
-bash benchmark/run-baseline.sh \
- --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --server-config examples/experimental/litellm/benchmark-route.toml \
- --task-list-file examples/experimental/litellm/benchmark-tasks.txt \
- --model litellm-stage \
- --agent codex \
- --reasoning-effort low \
- --n-tasks 3 \
- --n-concurrent 1 \
- --max-retries 0 \
- --port 4100 \
- --foreground
-```
-
-The native benchmark route uses the same efficient-first Stage policy but does
-not use `LiteLLMSyClient`; Switchyard server translates requests directly to
-the shared LiteLLM gateway. The gateway drops parameters unsupported by the
-selected provider model, so the benchmark's reasoning effort reaches `strong`
-without causing `fast` requests to fail. The sample runs `broken-python`,
-`cosign-keyless-signing`, and `jq-data-processing`. Three autonomous coding-agent
-tasks can make many paid model calls. Inspect the resulting run under
-`benchmark/tb_runs/`, including
-`run_manifest.json`, `harbor.log`, `routing_stats_final.json`, and
-`jobs/<job>/<task>/agent/trajectory.json`.
-
-The verified run used the benchmark runner's fallback Codex model template:
-the task image's pinned Codex 0.144.5 requires `supports_reasoning_summaries`,
-which a newer host-generated catalog may omit. If Harbor fails with that missing
-field, use the following recovery procedure. It changes no repository source
-and is only for that exact error.
-
-Create a temporary `codex` shim that makes the host-only
-`codex debug models --bundled` probe fail. The runner then uses its existing
-`_fallback_codex_model_template()`:
-
-```bash
-CODEX_SHIM_DIR="$(mktemp -d "${TMPDIR:-/tmp}/switchyard-codex.XXXXXX")"
-printf '#!/usr/bin/env bash\nexit 1\n' > "$CODEX_SHIM_DIR/codex"
-chmod +x "$CODEX_SHIM_DIR/codex"
-```
-
-From the repository root, rerun the benchmark with the temporary directory at
-the front of `PATH`:
-
-```bash
-PATH="$CODEX_SHIM_DIR:$PATH" \
-SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \
-bash benchmark/run-baseline.sh \
- --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --server-config examples/experimental/litellm/benchmark-route.toml \
- --task-list-file examples/experimental/litellm/benchmark-tasks.txt \
- --model litellm-stage \
- --agent codex \
- --reasoning-effort low \
- --n-tasks 3 \
- --n-concurrent 1 \
- --max-retries 0 \
- --port 4100 \
- --foreground
-```
-
-After the run, remove only the shim and its empty temporary directory:
-
-```bash
-rm -f "$CODEX_SHIM_DIR/codex"
-rmdir "$CODEX_SHIM_DIR"
-```
-
-## Stop
-
-For the package-directory quick start:
-
-```bash
-docker compose down
-```
-
-After the repository-root Harbor workflow:
-
-```bash
-docker compose --env-file examples/experimental/litellm/.env \
- -f examples/experimental/litellm/compose.yaml down
-```
-
-## Troubleshooting
-
-- Ensure `OPENROUTER_API_KEY` is present and not blank in `.env`.
-- Confirm your OpenRouter account can access `openai/gpt-5.6-sol` and
- `moonshotai/kimi-k3`.
-- If host port 4000 is occupied, set `LITELLM_PORT` to an unused port and use
- it in the health check, gateway URL, and `LiteLLMSyClient` `base_url`. The
- bundled `example.py` assumes port 4000 unless you edit it.
-- If the container is unhealthy, from the repository root inspect:
-
- ```bash
- docker compose --env-file examples/experimental/litellm/.env \
- -f examples/experimental/litellm/compose.yaml logs litellm
- ```
-- Harbor must use the Docker network named exactly `switchyard-litellm`.
-- The benchmark uses Switchyard port 4100 to avoid LiteLLM's host port 4000.
-
-## Security
-
-The gateway is unauthenticated and intended only for local development. Its
-port is bound to loopback only. Use LiteLLM authentication and its production
-deployment guidance before exposing it remotely.
-
-## References
-
-- [OpenRouter quickstart](https://openrouter.ai/docs/quickstart)
-- [GPT-5.6 Sol on OpenRouter](https://openrouter.ai/openai/gpt-5.6-sol)
-- [Kimi K3 on OpenRouter](https://openrouter.ai/moonshotai/kimi-k3)
-- [LiteLLM gateway quick start](https://docs.litellm.ai/docs/proxy/quick_start)
-- [Switchyard Stage-router docs](../../../docs/routing_algorithms/stage_router_routing.md)
diff --git a/examples/experimental/litellm/benchmark-route.toml b/examples/experimental/litellm/benchmark-route.toml
deleted file mode 100644
index 10c7413a4..000000000
--- a/examples/experimental/litellm/benchmark-route.toml
+++ /dev/null
@@ -1,25 +0,0 @@
-schema_version = 1
-
-[llm_clients.litellm]
-format = "openai_chat"
-base_url = "http://litellm:4000/v1"
-
-[llm_clients.litellm.extra_headers]
-authorization = "Bearer not-needed"
-
-[targets.strong]
-id = "strong"
-llm_client = "litellm"
-
-[targets.fast]
-id = "fast"
-llm_client = "litellm"
-
-[routes.litellm_stage]
-id = "litellm-stage"
-type = "stage_router"
-capable_target = "strong"
-efficient_target = "fast"
-picker = "efficient_first"
-confidence_threshold = 0.5
-recent_turn_window = 3
diff --git a/examples/experimental/litellm/benchmark-tasks.txt b/examples/experimental/litellm/benchmark-tasks.txt
deleted file mode 100644
index 222e6ed4f..000000000
--- a/examples/experimental/litellm/benchmark-tasks.txt
+++ /dev/null
@@ -1,4 +0,0 @@
-# Small deterministic TB Lite smoke sample.
-broken-python
-cosign-keyless-signing
-jq-data-processing
diff --git a/examples/experimental/litellm/example.py b/examples/experimental/litellm/example.py
deleted file mode 100644
index 96567d7e2..000000000
--- a/examples/experimental/litellm/example.py
+++ /dev/null
@@ -1,96 +0,0 @@
-#!/usr/bin/env python3
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-"""Route one coding-agent turn through LiteLLM with libsy's Stage router."""
-
-import asyncio
-
-from switchyard_litellm import LiteLLMSyClient
-
-from switchyard.libsy import LlmResponse, Step, algorithms
-
-
-def sy_request() -> dict[str, object]:
- """Build a normalized turn whose critical tool failure needs the capable tier."""
- return {
- "model": "auto",
- "messages": [
- {
- "role": "user",
- "content": [
- {
- "type": "text",
- "text": "Fix the failing tests.",
- }
- ],
- },
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_1",
- "name": "Bash",
- "arguments": {"command": "pytest"},
- }
- ],
- },
- {
- "role": "tool",
- "content": [
- {
- "type": "tool_result",
- "tool_call_id": "call_1",
- "content": [
- {
- "type": "text",
- "text": "fatal runtime error: out of memory",
- }
- ],
- "is_error": True,
- }
- ],
- },
- ],
- "reasoning": {"effort": "low"},
- "output": {"max_output_tokens": 96},
- }
-
-
-async def main() -> None:
- """Run the Stage router and print its normalized result."""
- client = LiteLLMSyClient()
- router = algorithms.stage_router(
- "strong",
- "fast",
- picker="efficient_first",
- confidence_threshold=0.5,
- recent_window=3,
- )
- try:
- async for step in router.run_stream(sy_request()):
- match step:
- case Step.CallModel(call):
- try:
- response = await client.call(call.request)
- except Exception as error:
- call.fail(error)
- else:
- call.respond(LlmResponse.Agg(response))
- case Step.Done(outcome):
- print("Decision:", outcome.selected_model_id)
- match outcome.response:
- case LlmResponse.Agg(response):
- print("Response:", response)
- case LlmResponse.Stream(stream):
- async for event in stream:
- print("Response event:", event)
- case None:
- print("Response:", await client.call(outcome.request))
- finally:
- await client.aclose()
-
-
-if __name__ == "__main__":
- asyncio.run(main())
diff --git a/examples/experimental/litellm/litellm-config.yaml b/examples/experimental/litellm/litellm-config.yaml
deleted file mode 100644
index 8ab38a113..000000000
--- a/examples/experimental/litellm/litellm-config.yaml
+++ /dev/null
@@ -1,12 +0,0 @@
-model_list:
- - model_name: strong
- litellm_params:
- model: openrouter/openai/gpt-5.6-sol
- api_key: os.environ/OPENROUTER_API_KEY
- - model_name: fast
- litellm_params:
- model: openrouter/moonshotai/kimi-k3
- api_key: os.environ/OPENROUTER_API_KEY
-
-litellm_settings:
- drop_params: true
diff --git a/examples/experimental/litellm/src/switchyard_litellm/__init__.py b/examples/experimental/litellm/src/switchyard_litellm/__init__.py
deleted file mode 100644
index 67131cfd7..000000000
--- a/examples/experimental/litellm/src/switchyard_litellm/__init__.py
+++ /dev/null
@@ -1,8 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-"""Experimental LiteLLM client for Switchyard libsy."""
-
-from .client import LiteLLMSyClient
-
-__all__ = ["LiteLLMSyClient"]
diff --git a/examples/experimental/litellm/src/switchyard_litellm/client.py b/examples/experimental/litellm/src/switchyard_litellm/client.py
deleted file mode 100644
index 277fbe6fa..000000000
--- a/examples/experimental/litellm/src/switchyard_litellm/client.py
+++ /dev/null
@@ -1,354 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-"""LiteLLM gateway adapter for normalized libsy requests."""
-
-from __future__ import annotations
-
-import json
-from collections.abc import Mapping, Sequence
-from typing import Any, cast
-
-from litellm import ModelResponse, acompletion
-from litellm.exceptions import (
- ContextWindowExceededError as LiteLLMContextWindowExceededError,
-)
-
-from switchyard.libsy import ContextWindowExceededError
-
-_TEXT_ROLES = {"system", "developer", "user"}
-_STOP_REASONS = {
- "stop": "end_turn",
- "length": "max_tokens",
- "tool_calls": "tool_use",
- "content_filter": "content_filter",
-}
-
-
-def _mapping(value: object, path: str) -> Mapping[str, object]:
- if not isinstance(value, Mapping):
- raise ValueError(f"{path} must be a mapping")
- return cast(Mapping[str, object], value)
-
-
-def _sequence(value: object, path: str) -> Sequence[object]:
- if isinstance(value, (str, bytes, bytearray)) or not isinstance(value, Sequence):
- raise ValueError(f"{path} must be a sequence")
- return cast(Sequence[object], value)
-
-
-def _reject_sequence_payload(
- request: Mapping[str, object],
- field: str,
-) -> None:
- value = request.get(field)
- if value is None:
- return
- if _sequence(value, field):
- raise ValueError(f"{field} is not supported")
-
-
-def _reject_extensions_payload(request: Mapping[str, object]) -> None:
- value = request.get("extensions")
- if value is None:
- return
- extensions = _mapping(value, "extensions")
- if set(extensions) - {"fields"}:
- raise ValueError("extensions is not supported")
- if "fields" in extensions and _mapping(extensions["fields"], "extensions.fields"):
- raise ValueError("extensions is not supported")
-
-
-def _reject_preservation_payload(request: Mapping[str, object]) -> None:
- value = request.get("preservation")
- if value is None:
- return
- preservation = _mapping(value, "preservation")
- if set(preservation) - {"requests", "responses"}:
- raise ValueError("preservation is not supported")
- for field in ("requests", "responses"):
- if field in preservation and _mapping(
- preservation[field], f"preservation.{field}"
- ):
- raise ValueError("preservation is not supported")
-
-
-def _text_block(raw_block: object, path: str) -> dict[str, str]:
- block = _mapping(raw_block, path)
- if block.get("type") != "text" or not isinstance(block.get("text"), str):
- raise ValueError(f"{path} must be a text block")
- return {"type": "text", "text": cast(str, block["text"])}
-
-
-def _tool_call(raw_block: object, path: str) -> dict[str, object]:
- block = _mapping(raw_block, path)
- call_id = block.get("id")
- name = block.get("name")
- arguments = block.get("arguments")
- if block.get("type") != "tool_call" or not isinstance(call_id, str):
- raise ValueError(f"{path} must be a tool-call block")
- if not isinstance(name, str):
- raise ValueError(f"{path}.name must be a string")
- if not isinstance(arguments, Mapping):
- raise ValueError(f"{path}.arguments must be a mapping")
- return {
- "id": call_id,
- "type": "function",
- "function": {
- "name": name,
- "arguments": json.dumps(arguments),
- },
- }
-
-
-def _tool_result(raw_block: object, path: str) -> dict[str, object]:
- block = _mapping(raw_block, path)
- call_id = block.get("tool_call_id")
- if block.get("type") != "tool_result" or not isinstance(call_id, str):
- raise ValueError(f"{path} must be a tool-result block")
- content = _sequence(block.get("content"), f"{path}.content")
- text = [
- _text_block(raw_content, f"{path}.content[{index}]")["text"]
- for index, raw_content in enumerate(content)
- ]
- return {"role": "tool", "tool_call_id": call_id, "content": "\n".join(text)}
-
-
-def _messages(request: Mapping[str, object]) -> list[dict[str, object]]:
- messages = _sequence(request.get("messages"), "messages")
- converted: list[dict[str, object]] = []
- for message_index, raw_message in enumerate(messages):
- message_path = f"messages[{message_index}]"
- message = _mapping(raw_message, message_path)
- role = message.get("role")
- blocks = _sequence(message.get("content"), f"{message_path}.content")
- if role in _TEXT_ROLES:
- content = [
- _text_block(raw_block, f"{message_path}.content[{block_index}]")
- for block_index, raw_block in enumerate(blocks)
- ]
- converted.append({"role": role, "content": content})
- elif role == "assistant":
- content: list[dict[str, str]] = []
- tool_calls: list[dict[str, object]] = []
- for block_index, raw_block in enumerate(blocks):
- block_path = f"{message_path}.content[{block_index}]"
- block = _mapping(raw_block, block_path)
- if block.get("type") == "text":
- content.append(_text_block(block, block_path))
- elif block.get("type") == "tool_call":
- tool_calls.append(_tool_call(block, block_path))
- else:
- raise ValueError(f"{block_path} is not supported")
- converted_message: dict[str, object] = {
- "role": "assistant",
- "content": content or None,
- }
- if tool_calls:
- converted_message["tool_calls"] = tool_calls
- converted.append(converted_message)
- elif role == "tool":
- converted.extend(
- _tool_result(raw_block, f"{message_path}.content[{block_index}]")
- for block_index, raw_block in enumerate(blocks)
- )
- else:
- raise ValueError(f"{message_path}.role is not supported")
- if not converted:
- raise ValueError("messages must not be empty")
- return converted
-
-
-def _tools(request: Mapping[str, object]) -> list[dict[str, object]]:
- tools = _sequence(request.get("tools", []), "tools")
- converted: list[dict[str, object]] = []
- for index, raw_tool in enumerate(tools):
- path = f"tools[{index}]"
- tool = _mapping(raw_tool, path)
- name = tool.get("name")
- description = tool.get("description")
- parameters = tool.get("parameters")
- strict = tool.get("strict")
- if not isinstance(name, str):
- raise ValueError(f"{path}.name must be a string")
- if description is not None and not isinstance(description, str):
- raise ValueError(f"{path}.description must be a string or null")
- if not isinstance(parameters, Mapping):
- raise ValueError(f"{path}.parameters must be a mapping")
- if strict is not None and not isinstance(strict, bool):
- raise ValueError(f"{path}.strict must be a boolean or null")
- function: dict[str, object] = {"name": name, "parameters": dict(parameters)}
- if description is not None:
- function["description"] = description
- if strict is not None:
- function["strict"] = strict
- converted.append({"type": "function", "function": function})
- return converted
-
-
-def _tool_choice(request: Mapping[str, object]) -> object | None:
- value = request.get("tool_choice")
- if value is None:
- return None
- choice = _mapping(value, "tool_choice")
- choice_type = choice.get("type")
- if choice_type in {"auto", "required", "none"}:
- return choice_type
- if choice_type == "tool":
- data = _mapping(choice.get("data"), "tool_choice.data")
- name = data.get("name")
- if not isinstance(name, str):
- raise ValueError("tool_choice.data.name must be a string")
- return {"type": "function", "function": {"name": name}}
- raise ValueError("tool_choice is not supported")
-
-
-def _optional_mapping(
- request: Mapping[str, object],
- field: str,
-) -> Mapping[str, object]:
- value = request.get(field)
- if value is None:
- return {}
- return _mapping(value, field)
-
-
-async def _payload(request: Mapping[str, object]) -> dict[str, Any]:
- model = request.get("model")
- if not isinstance(model, str) or not model:
- raise ValueError("model must be a non-empty string")
- if request.get("stream") is True:
- raise ValueError("stream=True is not supported")
- _reject_sequence_payload(request, "instructions")
- _reject_extensions_payload(request)
- _reject_preservation_payload(request)
-
- sampling = _optional_mapping(request, "sampling")
- output = _optional_mapping(request, "output")
- reasoning = _optional_mapping(request, "reasoning")
- if sampling.get("top_k") is not None:
- raise ValueError("sampling.top_k is not supported")
- if output.get("response_format") is not None:
- raise ValueError("output.response_format is not supported")
- if reasoning.get("raw") is not None:
- raise ValueError("reasoning.raw is not supported")
-
- payload: dict[str, Any] = {
- # LiteLLM needs this prefix to select its OpenAI-compatible transport,
- # then strips it before sending the selected alias to the gateway.
- "model": f"openai/{model}",
- "messages": _messages(request),
- "stream": False,
- }
- tools = _tools(request)
- if tools:
- payload["tools"] = tools
- tool_choice = _tool_choice(request)
- if tool_choice is not None:
- payload["tool_choice"] = tool_choice
- for source, target in (("temperature", "temperature"), ("top_p", "top_p")):
- value = sampling.get(source)
- if value is not None:
- payload[target] = value
- max_tokens = output.get("max_output_tokens")
- if max_tokens is not None:
- payload["max_completion_tokens"] = max_tokens
- effort = reasoning.get("effort")
- if effort is not None:
- payload["reasoning_effort"] = effort
- return payload
-
-
-def _usage(response: ModelResponse) -> dict[str, int]:
- usage = response.usage
- if usage is None:
- return {}
- prompt_details = usage.prompt_tokens_details
- completion_details = usage.completion_tokens_details
- cached_value = prompt_details.cached_tokens if prompt_details is not None else None
- cached = cached_value if cached_value is not None else 0
- normalized = {
- "input_tokens": max(usage.prompt_tokens - cached, 0),
- "output_tokens": usage.completion_tokens,
- "total_tokens": usage.total_tokens,
- }
- if cached_value is not None:
- normalized["cached_input_tokens"] = cached
- if completion_details is not None and completion_details.reasoning_tokens is not None:
- normalized["reasoning_tokens"] = completion_details.reasoning_tokens
- return normalized
-
-
-def _response(response: ModelResponse) -> dict[str, object]:
- if not response.choices:
- raise ValueError("LiteLLM returned no choices")
- choice = response.choices[0]
- text = choice.message.content
- content: list[dict[str, object]] = []
- if isinstance(text, str) and text:
- content.append({"type": "text", "text": text})
- for tool_call in choice.message.tool_calls or []:
- try:
- arguments = json.loads(tool_call.function.arguments)
- except (TypeError, json.JSONDecodeError) as error:
- raise ValueError("LiteLLM returned invalid tool-call arguments") from error
- if not isinstance(arguments, Mapping):
- raise ValueError("LiteLLM returned invalid tool-call arguments")
- content.append({
- "type": "tool_call",
- "id": tool_call.id,
- "name": tool_call.function.name,
- "arguments": arguments,
- })
- if not content and choice.finish_reason != "content_filter":
- raise ValueError("LiteLLM returned no text content")
- return {
- "id": response.id,
- "model": response.model,
- "outputs": [
- {
- "role": "assistant",
- "content": content,
- "stop_reason": _STOP_REASONS.get(choice.finish_reason, "unknown"),
- }
- ],
- "usage": _usage(response),
- }
-
-
-class LiteLLMSyClient:
- """Call the LiteLLM gateway alias selected in a libsy request."""
-
- def __init__(
- self,
- *,
- base_url: str = "http://127.0.0.1:4000/v1",
- api_key: str = "not-needed",
- ) -> None:
- self._base_url = base_url
- self._api_key = api_key
-
- async def call(
- self,
- sy_request: Mapping[str, object],
- ) -> Mapping[str, object]:
- """Send one normalized, buffered text request through LiteLLM."""
- payload = await _payload(sy_request)
- try:
- response = await acompletion(
- **payload,
- api_base=self._base_url,
- api_key=self._api_key,
- num_retries=0,
- # LiteLLM otherwise imports its optional proxy MCP stack for ordinary
- # OpenAI function tools before it determines that they are not MCP tools.
- _skip_mcp_handler=True,
- allowed_openai_params=["reasoning_effort"],
- extra_body={"allowed_openai_params": ["reasoning_effort"]},
- )
- except LiteLLMContextWindowExceededError as error:
- raise ContextWindowExceededError(str(error)) from error
- return _response(cast(ModelResponse, response))
-
- async def aclose(self) -> None:
- """Retain asynchronous lifecycle compatibility without owning a transport."""
diff --git a/examples/experimental/litellm/tests/test_client.py b/examples/experimental/litellm/tests/test_client.py
deleted file mode 100644
index 0f7e6555a..000000000
--- a/examples/experimental/litellm/tests/test_client.py
+++ /dev/null
@@ -1,628 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-import copy
-import json
-
-import httpx
-import pytest
-import respx
-from litellm import (
- AuthenticationError,
- ModelResponse,
- RateLimitError,
-)
-from litellm.exceptions import (
- ContextWindowExceededError as LiteLLMContextWindowExceededError,
-)
-from litellm.types.utils import Choices, Message, Usage
-from switchyard_litellm import LiteLLMSyClient
-
-from switchyard.libsy import ContextWindowExceededError
-
-BASE_URL = "http://gateway.test/v1"
-
-
-def request_body() -> dict[str, object]:
- return {
- "model": "fast",
- "instructions": [],
- "messages": [
- {
- "role": "user",
- "content": [{"type": "text", "text": "Say hello."}],
- }
- ],
- "tools": [],
- "tool_choice": None,
- "sampling": {"temperature": 0.2, "top_p": 0.9, "top_k": None},
- "output": {"max_output_tokens": 64, "response_format": None},
- "reasoning": {"effort": "low", "raw": None},
- "stream": False,
- "extensions": {"fields": {}},
- "preservation": {"requests": {}, "responses": {}},
- }
-
-
-def gateway_response(model: str = "moonshotai/kimi-k3") -> dict[str, object]:
- return {
- "id": "chatcmpl-test",
- "object": "chat.completion",
- "created": 1,
- "model": model,
- "choices": [
- {
- "index": 0,
- "message": {"role": "assistant", "content": "Hello."},
- "finish_reason": "stop",
- }
- ],
- "usage": {
- "prompt_tokens": 12,
- "completion_tokens": 7,
- "total_tokens": 19,
- "prompt_tokens_details": {"cached_tokens": 2},
- "completion_tokens_details": {"reasoning_tokens": 3},
- },
- }
-
-
-def gateway_tool_response() -> dict[str, object]:
- response = gateway_response()
- response["choices"][0]["message"] = {
- "role": "assistant",
- "content": None,
- "tool_calls": [
- {
- "id": "call_2",
- "type": "function",
- "function": {
- "name": "Bash",
- "arguments": '{"command":"pytest -q"}',
- },
- }
- ],
- }
- response["choices"][0]["finish_reason"] = "tool_calls"
- return response
-
-
-async def test_call_uses_litellm_async_completion(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- captured: dict[str, object] = {}
-
- async def fake_acompletion(**kwargs: object) -> ModelResponse:
- captured.update(kwargs)
- return ModelResponse(**gateway_response())
-
- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- await client.call(request_body())
- finally:
- await client.aclose()
-
- assert captured["model"] == "openai/fast"
- assert captured["api_base"] == BASE_URL
- assert captured["api_key"] == "not-needed"
- assert captured["num_retries"] == 0
- assert captured["allowed_openai_params"] == ["reasoning_effort"]
- assert captured["stream"] is False
-
-
-async def test_call_translates_request_and_normalizes_response() -> None:
- route = respx.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(200, json=gateway_response())
- )
- request = request_body()
- original = copy.deepcopy(request)
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- response = await client.call(request)
- finally:
- await client.aclose()
-
- sent = json.loads(route.calls[0].request.content)
- assert sent == {
- "model": "fast",
- "messages": [
- {
- "role": "user",
- "content": [{"type": "text", "text": "Say hello."}],
- }
- ],
- "temperature": 0.2,
- "top_p": 0.9,
- "max_completion_tokens": 64,
- "reasoning_effort": "low",
- "allowed_openai_params": ["reasoning_effort"],
- }
- assert sent["model"] == "fast"
- assert sent["model"] != "openai/fast"
- assert request == original
- assert response == {
- "id": "chatcmpl-test",
- "model": "moonshotai/kimi-k3",
- "outputs": [
- {
- "role": "assistant",
- "content": [{"type": "text", "text": "Hello."}],
- "stop_reason": "end_turn",
- }
- ],
- "usage": {
- "input_tokens": 10,
- "cached_input_tokens": 2,
- "output_tokens": 7,
- "total_tokens": 19,
- "reasoning_tokens": 3,
- },
- }
-
-
-async def test_call_translates_function_tool_history() -> None:
- route = respx.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(200, json=gateway_response())
- )
- request = request_body()
- request["messages"] = [
- {
- "role": "user",
- "content": [{"type": "text", "text": "Run the tests."}],
- },
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_1",
- "name": "Bash",
- "arguments": {"command": "pytest"},
- }
- ],
- },
- {
- "role": "tool",
- "content": [
- {
- "type": "tool_result",
- "tool_call_id": "call_1",
- "content": [{"type": "text", "text": "1 failed"}],
- "is_error": True,
- }
- ],
- },
- ]
- request["tools"] = [
- {
- "name": "Bash",
- "description": "Run a shell command.",
- "parameters": {
- "type": "object",
- "properties": {"command": {"type": "string"}},
- "required": ["command"],
- },
- "strict": True,
- }
- ]
- request["tool_choice"] = {"type": "tool", "data": {"name": "Bash"}}
-
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- await client.call(request)
- finally:
- await client.aclose()
-
- sent = json.loads(route.calls[0].request.content)
- assert sent["messages"] == [
- {
- "role": "user",
- "content": [{"type": "text", "text": "Run the tests."}],
- },
- {
- "role": "assistant",
- "tool_calls": [
- {
- "id": "call_1",
- "type": "function",
- "function": {
- "name": "Bash",
- "arguments": '{"command": "pytest"}',
- },
- }
- ],
- },
- {"role": "tool", "tool_call_id": "call_1", "content": "1 failed"},
- ]
- assert sent["tools"] == [
- {
- "type": "function",
- "function": {
- "name": "Bash",
- "description": "Run a shell command.",
- "parameters": {
- "type": "object",
- "properties": {"command": {"type": "string"}},
- "required": ["command"],
- },
- "strict": True,
- },
- }
- ]
- assert sent["tool_choice"] == {
- "type": "function",
- "function": {"name": "Bash"},
- }
-
-
-async def test_call_normalizes_a_function_tool_response(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- async def fake_acompletion(**_: object) -> ModelResponse:
- return ModelResponse(**gateway_tool_response())
-
- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- response = await client.call(request_body())
- finally:
- await client.aclose()
-
- assert response["outputs"] == [
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_2",
- "name": "Bash",
- "arguments": {"command": "pytest -q"},
- }
- ],
- "stop_reason": "tool_use",
- }
- ]
-
-
-async def test_call_rejects_non_mapping_tool_call_arguments(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- async def fake_acompletion(**_: object) -> ModelResponse:
- return ModelResponse(**gateway_response())
-
- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
- request = request_body()
- request["messages"] = [
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_1",
- "name": "Bash",
- "arguments": ["pytest"],
- }
- ],
- }
- ]
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ValueError, match=r"messages\[0\]\.content\[0\]\.arguments"):
- await client.call(request)
- finally:
- await client.aclose()
-
-
-async def test_call_rejects_non_mapping_response_tool_call_arguments(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- payload = gateway_tool_response()
- tool_call = payload["choices"][0]["message"]["tool_calls"][0]
- tool_call["function"]["arguments"] = '["pytest"]'
-
- async def fake_acompletion(**_: object) -> ModelResponse:
- return ModelResponse(**payload)
-
- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ValueError, match="invalid tool-call arguments"):
- await client.call(request_body())
- finally:
- await client.aclose()
-
-
[email protected]("choice", ["auto", "required", "none"])
-async def test_call_translates_standard_tool_choice(
- monkeypatch: pytest.MonkeyPatch,
- choice: str,
-) -> None:
- captured: dict[str, object] = {}
-
- async def fake_acompletion(**kwargs: object) -> ModelResponse:
- captured.update(kwargs)
- return ModelResponse(**gateway_response())
-
- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
- request = request_body()
- request["tools"] = [{"name": "Bash", "description": None, "parameters": {}}]
- request["tool_choice"] = {"type": choice}
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- await client.call(request)
- finally:
- await client.aclose()
-
- assert captured["tool_choice"] == choice
-
-
- ("mutate", "match"),
- [
- (lambda body: body.update(messages="not-a-sequence"), "messages"),
- (lambda body: body.update(sampling=[]), "sampling"),
- (lambda body: body.update(output=[]), "output"),
- (lambda body: body.update(reasoning=[]), "reasoning"),
- (lambda body: body.update(extensions=[]), "extensions"),
- (lambda body: body.update(preservation=[]), "preservation"),
- (lambda body: body.update(stream=True), "stream"),
- (
- lambda body: body["instructions"].append(
- {"role": "system", "content": [{"type": "text", "text": "x"}]}
- ),
- "instructions",
- ),
- (
- lambda body: body["messages"][0]["content"].append(
- {"type": "image", "source": {"type": "url", "data": {"url": "x"}}}
- ),
- r"messages\[0\]\.content\[1\]",
- ),
- (
- lambda body: body.update(
- messages=[
- {
- "role": "tool",
- "content": [
- {
- "type": "tool_result",
- "tool_call_id": "call_1",
- "content": [
- {
- "type": "image",
- "source": {
- "type": "url",
- "data": {"url": "x"},
- },
- }
- ],
- "is_error": False,
- }
- ],
- }
- ]
- ),
- r"messages\[0\]\.content\[0\]\.content\[0\]",
- ),
- (
- lambda body: body.update(tool_choice={"type": "raw", "data": {}}),
- "tool_choice",
- ),
- (
- lambda body: body["sampling"].update(top_k=4),
- "sampling.top_k",
- ),
- (
- lambda body: body["output"].update(response_format={"type": "json_object"}),
- "output.response_format",
- ),
- (
- lambda body: body["reasoning"].update(raw={"provider": "value"}),
- "reasoning.raw",
- ),
- (
- lambda body: body["extensions"]["fields"].update(provider="value"),
- "extensions",
- ),
- (
- lambda body: body["extensions"]["fields"].update(provider=None),
- "extensions",
- ),
- (
- lambda body: body["preservation"]["requests"].update(provider=None),
- "preservation",
- ),
- (
- lambda body: body["preservation"]["responses"].update(provider={}),
- "preservation",
- ),
- ],
-)
-async def test_call_rejects_unsupported_normalized_fields(
- mutate: object,
- match: str,
-) -> None:
- request = request_body()
- assert callable(mutate)
- mutate(request)
- with respx.mock(assert_all_called=False) as router:
- router.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(200, json=gateway_response())
- )
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ValueError, match=match):
- await client.call(request)
- finally:
- await client.aclose()
-
-
-async def test_call_rejects_missing_messages() -> None:
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ValueError, match="messages"):
- await client.call({"model": "fast"})
- finally:
- await client.aclose()
-
-
-async def test_call_rejects_a_response_without_text() -> None:
- payload = gateway_response()
- payload["choices"][0]["message"]["content"] = None
- respx.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(200, json=payload)
- )
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ValueError, match="no text content"):
- await client.call(request_body())
- finally:
- await client.aclose()
-
-
-async def test_call_rejects_a_response_without_choices(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- payload = gateway_response()
- payload["choices"] = []
-
- async def fake_acompletion(**_: object) -> ModelResponse:
- return ModelResponse(**payload)
-
- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ValueError, match="no choices"):
- await client.call(request_body())
- finally:
- await client.aclose()
-
-
-async def test_litellm_errors_propagate() -> None:
- respx.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(
- 401,
- json={
- "error": {
- "message": "Incorrect API key provided",
- "type": "authentication_error",
- "code": "invalid_api_key",
- }
- },
- )
- )
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(AuthenticationError):
- await client.call(request_body())
- finally:
- await client.aclose()
-
-
-async def test_retryable_litellm_error_is_not_retried() -> None:
- route = respx.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(
- 429,
- json={
- "error": {
- "message": "rate limited",
- "type": "rate_limit_error",
- "code": "rate_limit_exceeded",
- }
- },
- )
- )
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(RateLimitError):
- await client.call(request_body())
- finally:
- await client.aclose()
-
- assert route.call_count == 1
-
-
-async def test_context_window_error_is_preserved(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- async def fail_with_context_window(**_: object) -> ModelResponse:
- raise LiteLLMContextWindowExceededError(
- message="request is too large",
- model="fast",
- llm_provider="openai",
- )
-
- monkeypatch.setattr(
- "switchyard_litellm.client.acompletion", fail_with_context_window
- )
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- with pytest.raises(ContextWindowExceededError, match="request is too large"):
- await client.call(request_body())
- finally:
- await client.aclose()
-
-
-async def test_cached_token_count_preserves_explicit_zero() -> None:
- payload = gateway_response()
- payload["usage"]["prompt_tokens_details"]["cached_tokens"] = 0
- respx.post(f"{BASE_URL}/chat/completions").mock(
- return_value=httpx.Response(200, json=payload)
- )
- client = LiteLLMSyClient(base_url=BASE_URL)
- try:
- response = await client.call(request_body())
- finally:
- await client.aclose()
-
- assert response["usage"]["cached_input_tokens"] == 0
-
-
-def test_response_content_filter_empty_content() -> None:
- """Empty content with a content_filter finish reason must be normalized."""
- from switchyard_litellm.client import _response
-
- response = ModelResponse(
- id="chatcmpl-test",
- model="openai/strong",
- choices=[
- Choices(
- message=Message(content=None, tool_calls=None),
- finish_reason="content_filter",
- )
- ],
- usage=Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0),
- )
- result = _response(response)
- outputs = result["outputs"]
- assert isinstance(outputs, list)
- output = outputs[0]
- assert isinstance(output, dict)
- assert output["content"] == []
- assert output["stop_reason"] == "content_filter"
-
-
-def test_response_empty_content_without_content_filter_raises() -> None:
- """Empty content with any other finish reason must still raise."""
- from switchyard_litellm.client import _response
-
- response = ModelResponse(
- id="chatcmpl-test",
- model="openai/strong",
- choices=[
- Choices(
- message=Message(content=None, tool_calls=None),
- finish_reason="stop",
- )
- ],
- usage=Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0),
- )
- with pytest.raises(ValueError, match="LiteLLM returned no text content"):
- _response(response)
diff --git a/examples/experimental/litellm/tests/test_e2e.py b/examples/experimental/litellm/tests/test_e2e.py
deleted file mode 100644
index 9586eea9b..000000000
--- a/examples/experimental/litellm/tests/test_e2e.py
+++ /dev/null
@@ -1,191 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-import os
-import shutil
-import socket
-import subprocess
-from collections.abc import Iterator
-from pathlib import Path
-
-import pytest
-from switchyard_litellm import LiteLLMSyClient
-
-from switchyard.libsy import Algorithm, LlmResponse, Step, algorithms
-
-PACKAGE_ROOT = Path(__file__).resolve().parents[1]
-
-
-def _free_port() -> int:
- with socket.socket() as sock:
- sock.bind(("127.0.0.1", 0))
- return int(sock.getsockname()[1])
-
-
[email protected](scope="session")
-def litellm_base_url() -> Iterator[str]:
- if os.environ.get("SWITCHYARD_LITELLM_E2E") != "1":
- pytest.skip("SWITCHYARD_LITELLM_E2E=1 is required for paid E2E tests")
- if not os.environ.get("OPENROUTER_API_KEY"):
- pytest.skip("OPENROUTER_API_KEY is required for paid E2E tests")
- if shutil.which("docker") is None:
- pytest.skip("Docker is required for paid E2E tests")
- subprocess.run(
- ["docker", "compose", "version"],
- check=True,
- stdout=subprocess.DEVNULL,
- )
-
- port = _free_port()
- project = f"switchyard-litellm-e2e-{os.getpid()}"
- network = f"{project}-network"
- env = {
- **os.environ,
- "LITELLM_PORT": str(port),
- "LITELLM_NETWORK": network,
- }
- compose = ["docker", "compose", "--project-name", project]
- try:
- subprocess.run(
- [*compose, "up", "-d", "--wait"],
- cwd=PACKAGE_ROOT,
- env=env,
- check=True,
- )
- yield f"http://127.0.0.1:{port}/v1"
- finally:
- subprocess.run(
- [*compose, "down", "--volumes", "--remove-orphans"],
- cwd=PACKAGE_ROOT,
- env=env,
- check=True,
- )
-
-
-def test_paid_e2e_requires_explicit_spend_opt_in(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- monkeypatch.setenv("OPENROUTER_API_KEY", "test-key")
- monkeypatch.delenv("SWITCHYARD_LITELLM_E2E", raising=False)
-
- def fail_if_docker_is_checked(_: str) -> str | None:
- pytest.fail("Docker must not be checked without the paid E2E opt-in")
-
- monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)
- fixture = litellm_base_url.__wrapped__()
- with pytest.raises(pytest.skip.Exception, match="SWITCHYARD_LITELLM_E2E"):
- next(fixture)
-
-
-def test_paid_e2e_requires_openrouter_key(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- monkeypatch.setenv("SWITCHYARD_LITELLM_E2E", "1")
- monkeypatch.setenv("OPENAI_API_KEY", "legacy-key")
- monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
-
- def fail_if_docker_is_checked(_: str) -> str | None:
- pytest.fail("Docker must not be checked without OPENROUTER_API_KEY")
-
- monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)
- fixture = litellm_base_url.__wrapped__()
- with pytest.raises(pytest.skip.Exception, match="OPENROUTER_API_KEY"):
- next(fixture)
-
-
-def _request(*, critical_error: bool = False) -> dict[str, object]:
- messages: list[dict[str, object]] = [
- {
- "role": "user",
- "content": [
- {
- "type": "text",
- "text": "Reply with a short greeting for a Switchyard E2E test.",
- }
- ],
- }
- ]
- if critical_error:
- messages.extend([
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_1",
- "name": "Bash",
- "arguments": {"command": "pytest"},
- }
- ],
- },
- {
- "role": "tool",
- "content": [
- {
- "type": "tool_result",
- "tool_call_id": "call_1",
- "content": [
- {
- "type": "text",
- "text": "fatal runtime error: out of memory",
- }
- ],
- "is_error": True,
- }
- ],
- },
- ])
- return {
- "model": "auto",
- "messages": messages,
- "reasoning": {"effort": "low"},
- "output": {"max_output_tokens": 128},
- }
-
-
-async def _run_router(
- router: Algorithm,
- request: dict[str, object],
- client: LiteLLMSyClient,
-) -> tuple[str, dict[str, object]]:
- async for step in router.run_stream(request):
- match step:
- case Step.CallModel(call):
- call.respond(LlmResponse.Agg(await client.call(call.request)))
- case Step.Done(outcome):
- match outcome.response:
- case LlmResponse.Agg(response):
- return outcome.selected_model_id, response
- case LlmResponse.Stream(_):
- raise AssertionError("LiteLLM example expects buffered responses")
- case None:
- return outcome.selected_model_id, await client.call(outcome.request)
- raise AssertionError("algorithm stream ended without an outcome")
-
-
-async def test_stage_router_calls_both_real_openrouter_models(
- litellm_base_url: str,
-) -> None:
- client = LiteLLMSyClient(base_url=litellm_base_url)
- router = algorithms.stage_router(
- "strong",
- "fast",
- picker="efficient_first",
- confidence_threshold=0.5,
- recent_window=3,
- )
- try:
- fast_trace, fast_response = await _run_router(router, _request(), client)
- strong_trace, strong_response = await _run_router(
- router, _request(critical_error=True), client
- )
- finally:
- await client.aclose()
-
- assert strong_trace == "strong"
- assert fast_trace == "fast"
- for response in (strong_response, fast_response):
- text = response["outputs"][0]["content"][0]["text"]
- assert isinstance(text, str)
- assert text.strip()
diff --git a/examples/experimental/litellm/tests/test_gateway_config.py b/examples/experimental/litellm/tests/test_gateway_config.py
deleted file mode 100644
index 8042cdfc1..000000000
--- a/examples/experimental/litellm/tests/test_gateway_config.py
+++ /dev/null
@@ -1,51 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-import tomllib
-from pathlib import Path
-
-import yaml
-
-PACKAGE_ROOT = Path(__file__).resolve().parents[1]
-REPOSITORY_ROOT = PACKAGE_ROOT.parents[2]
-
-
-def test_example_targets_python_312() -> None:
- project = tomllib.loads((PACKAGE_ROOT / "pyproject.toml").read_text())
-
- assert project["project"]["requires-python"] == ">=3.12,<3.13"
-
-
-def test_lockfile_matches_root_package_version() -> None:
- root_project = tomllib.loads((REPOSITORY_ROOT / "pyproject.toml").read_text())
- lock = tomllib.loads((PACKAGE_ROOT / "uv.lock").read_text())
- locked_root = next(
- package for package in lock["package"] if package["name"] == "nemo-switchyard"
- )
-
- assert locked_root["version"] == root_project["project"]["version"]
-
-
-def test_gateway_uses_openrouter_models_and_credential() -> None:
- config = yaml.safe_load((PACKAGE_ROOT / "litellm-config.yaml").read_text())
- models = {
- item["model_name"]: item["litellm_params"] for item in config["model_list"]
- }
-
- assert models == {
- "strong": {
- "model": "openrouter/openai/gpt-5.6-sol",
- "api_key": "os.environ/OPENROUTER_API_KEY",
- },
- "fast": {
- "model": "openrouter/moonshotai/kimi-k3",
- "api_key": "os.environ/OPENROUTER_API_KEY",
- },
- }
- assert config["litellm_settings"] == {"drop_params": True}
-
- compose = yaml.safe_load((PACKAGE_ROOT / "compose.yaml").read_text())
- assert compose["services"]["litellm"]["environment"] == {
- "OPENROUTER_API_KEY": "${OPENROUTER_API_KEY:?set OPENROUTER_API_KEY}"
- }
- assert (PACKAGE_ROOT / ".env.example").read_text() == "OPENROUTER_API_KEY=\n"
diff --git a/examples/experimental/litellm/tests/test_stage_routing.py b/examples/experimental/litellm/tests/test_stage_routing.py
deleted file mode 100644
index e6fb84877..000000000
--- a/examples/experimental/litellm/tests/test_stage_routing.py
+++ /dev/null
@@ -1,128 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-import json
-
-import httpx
-import respx
-from switchyard_litellm import LiteLLMSyClient
-
-from switchyard.libsy import Algorithm, LlmResponse, Step, algorithms
-
-BASE_URL = "http://gateway.test/v1"
-
-
-def request_body(*, critical_error: bool = False) -> dict[str, object]:
- messages: list[dict[str, object]] = [
- {
- "role": "user",
- "content": [{"type": "text", "text": "Fix the failing tests."}],
- }
- ]
- if critical_error:
- messages.extend([
- {
- "role": "assistant",
- "content": [
- {
- "type": "tool_call",
- "id": "call_1",
- "name": "Bash",
- "arguments": {"command": "pytest"},
- }
- ],
- },
- {
- "role": "tool",
- "content": [
- {
- "type": "tool_result",
- "tool_call_id": "call_1",
- "content": [
- {
- "type": "text",
- "text": "fatal runtime error: out of memory",
- }
- ],
- "is_error": True,
- }
- ],
- },
- ])
- return {"model": "auto", "messages": messages}
-
-
-def gateway_response(model: str) -> dict[str, object]:
- return {
- "id": f"chatcmpl-{model}",
- "object": "chat.completion",
- "created": 1,
- "model": model,
- "choices": [
- {
- "index": 0,
- "message": {"role": "assistant", "content": model},
- "finish_reason": "stop",
- }
- ],
- "usage": {
- "prompt_tokens": 4,
- "completion_tokens": 1,
- "total_tokens": 5,
- },
- }
-
-
-async def _run_router(
- router: Algorithm,
- request: dict[str, object],
- client: LiteLLMSyClient,
-) -> tuple[str, dict[str, object]]:
- async for step in router.run_stream(request):
- match step:
- case Step.CallModel(call):
- call.respond(LlmResponse.Agg(await client.call(call.request)))
- case Step.Done(outcome):
- match outcome.response:
- case LlmResponse.Agg(response):
- return outcome.selected_model_id, response
- case LlmResponse.Stream(_):
- raise AssertionError("LiteLLM example expects buffered responses")
- case None:
- return outcome.selected_model_id, await client.call(outcome.request)
- raise AssertionError("algorithm stream ended without an outcome")
-
-
-async def test_stage_router_drives_both_litellm_models() -> None:
- seen: list[str] = []
-
- def respond(request: httpx.Request) -> httpx.Response:
- model = json.loads(request.content)["model"]
- seen.append(model)
- return httpx.Response(200, json=gateway_response(model))
-
- respx.post(f"{BASE_URL}/chat/completions").mock(side_effect=respond)
- client = LiteLLMSyClient(base_url=BASE_URL)
- router = algorithms.stage_router(
- "strong",
- "fast",
- picker="efficient_first",
- confidence_threshold=0.5,
- recent_window=3,
- )
- try:
- fast_target, fast_response = await _run_router(
- router, request_body(), client
- )
- strong_target, strong_response = await _run_router(
- router, request_body(critical_error=True), client
- )
- finally:
- await client.aclose()
-
- assert fast_target == "fast"
- assert strong_target == "strong"
- assert fast_response["outputs"][0]["content"][0]["text"] == "fast"
- assert strong_response["outputs"][0]["content"][0]["text"] == "strong"
- assert seen == ["fast", "strong"]
diff --git a/examples/litellm/README.md b/examples/litellm/README.md
new file mode 100644
index 000000000..20ee65315
--- /dev/null
+++ b/examples/litellm/README.md
@@ -0,0 +1,364 @@
+# Switchyard routing plugin for LiteLLM
+
+> **Experimental integration:** This example and its Python API may change without notice.
+
+> **Checkout-only dependency:** The current published `nemo-switchyard` release does not contain
+> the decision-only Python API used here. Run this example from this repository checkout, where
+> `[tool.uv.sources]` binds the example to the adjacent Switchyard source. Do not install or publish
+> `switchyard-litellm` as a standalone package until a compatible Switchyard release is available
+> and its dependency floor can be updated.
+
+This package integrates Switchyard at LiteLLM's native routing-plugin boundary. Applications keep
+using LiteLLM's `Router` or OpenAI-compatible proxy. The same configured object has two LiteLLM
+roles: before deployment selection it narrows the candidate set to the model selected by
+Switchyard; after selection it applies any supported Switchyard request rewrite before LiteLLM
+translates and sends the provider request.
+
+The integration pins LiteLLM 1.97.0. Model inventory and routing policy are both owned by the
+deployer:
+
+- LiteLLM YAML defines the public model group, candidate models, credentials, and provider options.
+- Switchyard TOML selects the algorithm and all of its supported routing parameters.
+
+The checked-in profiles use two OpenRouter models as demonstration values. Those model IDs are not
+built into `switchyard_litellm` and can be replaced without changing package source.
+
+## Layout
+
+```text
+litellm/
+├── deployment/
+│ ├── .env.example
+│ ├── Dockerfile
+│ ├── compose.yaml
+│ └── profiles/
+│ ├── stage/{litellm.yaml,switchyard.toml}
+│ └── random/{litellm.yaml,switchyard.toml}
+├── examples/python_router.py
+├── src/switchyard_litellm/
+│ ├── configuration/
+│ └── plugins/
+└── tests/{unit,integration}/
+```
+
+## Request flow
+
+```text
+application → LiteLLM Router → Switchyard routing plugin
+ → Algorithm.run_stream()
+ → one selected candidate + optional request delta
+ → LiteLLM deployment selection
+ → same object as deployment callback → provider inference
+```
+
+The candidate-bound Stage and Random plugins construct Switchyard algorithms from the live
+candidate list supplied by LiteLLM. The low-level `SwitchyardRoutingPlugin` then:
+
+1. converts LiteLLM's `structured_messages` to a normalized Switchyard request;
+2. runs the supplied algorithm until `Step.Done`;
+3. validates and converts any supported request delta;
+4. replaces `context.candidate_models` with the selected exact deployment; and
+5. records the decision and private delta under `context.signals["switchyard"]`.
+
+LiteLLM copies those signals into request metadata. After LiteLLM selects a deployment, the same
+object's `async_pre_call_deployment_hook` applies the delta and removes it from the metadata passed
+downstream. The proxy profile therefore registers the dotted object as both a routing plugin and a
+callback.
+
+LiteLLM remains responsible for credentials, retries, provider translation, inference, and the
+OpenAI-compatible API.
+
+## Supported algorithm semantics
+
+Compatibility is determined by behavior rather than an algorithm-name allowlist.
+`StageRoutingPlugin` and `RandomRoutingPlugin` provide the deployer-oriented bindings supported by
+the checked-in configuration loader.
+
+| Behavior | Result |
+|---|---|
+| Random selects a current candidate | Supported |
+| Stage uses only request-history signals | Supported |
+| Algorithm requests `Step.CallModel` | Rejected |
+| Algorithm returns an existing response | Rejected |
+| Algorithm rewrites text instructions/messages | Applied after deployment selection |
+| Algorithm rewrites tools/tool choice | Applied after deployment selection |
+| Algorithm rewrites sampling, output, reasoning effort, or stream | Applied after deployment selection |
+| Algorithm rewrites supported provider-extension fields | Applied after deployment selection |
+| Algorithm rewrites raw reasoning, preservation, or unsupported content/extensions | Rejected |
+| Algorithm selects outside the current candidate pool | Rejected |
+| Algorithm stream ends without `Step.Done` | Rejected |
+
+Escalation and classifier-backed routers are not compatible because a LiteLLM routing plugin cannot
+service intermediate model calls. Stage's system prompts and handoff notes do not require an
+intermediate call, so the dual-role Stage plugin supports them. Unsupported behavior fails closed
+before LiteLLM sends inference.
+
+LiteLLM 1.97.0's routing context exposes structured messages but does not expose the caller's tools,
+sampling controls, output controls, or provider-specific arguments. The adapter therefore uses
+delta semantics: a field explicitly changed by Switchyard overrides the corresponding LiteLLM
+argument, while a field Switchyard leaves at the adapter default does not clear or replace the
+caller's original value. A Switchyard algorithm can produce an outbound override for tools,
+`tool_choice`, `temperature`, `top_p`, `top_k`, `max_completion_tokens`, `response_format`,
+`reasoning_effort`, `stream`, and the safe OpenAI Chat extension fields supported by the adapter.
+It cannot inspect the original value of those fields through the current routing-plugin API, or
+intentionally clear a caller value back to an adapter default such as `None` or an empty tool list.
+
+The message adapter supports `system`, `developer`, and `user` text; assistant text and OpenAI
+function tool calls; and text tool results. It rejects malformed function arguments, media or audio
+content, legacy `function` messages, and unknown roles or content blocks.
+
+## Quick start with the local proxy
+
+Prerequisites are Docker Compose and an OpenRouter key with access to the example models. From this
+directory:
+
+```bash
+cp deployment/.env.example deployment/.env
+# Set OPENROUTER_API_KEY in deployment/.env.
+docker compose -f deployment/compose.yaml up -d --build --wait
+curl -fsS http://127.0.0.1:4000/health/liveliness
+```
+
+The default profile is `stage`. Send a request to its public `switchyard` model group:
+
+```bash
+curl -i http://127.0.0.1:4000/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "switchyard",
+ "messages": [{"role": "user", "content": "Reply with the word hello."}],
+ "max_tokens": 64
+ }'
+```
+
+LiteLLM keeps the response body's `model` field equal to the public group. The
+`x-litellm-model-name` response header identifies the concrete model selected by Switchyard.
+
+Start the Random profile instead with:
+
+```bash
+SWITCHYARD_LITELLM_PROFILE=random \
+ docker compose -f deployment/compose.yaml up -d --build --wait
+```
+
+Stop the proxy without deleting repository data:
+
+```bash
+docker compose -f deployment/compose.yaml down
+```
+
+## Configure a deployment profile
+
+Each directory under `deployment/profiles/` is a complete selectable profile:
+
+```text
+profiles/my-profile/
+├── litellm.yaml # model inventory and LiteLLM settings
+└── switchyard.toml # Switchyard algorithm and parameters
+```
+
+The LiteLLM YAML registers the environment-configured plugin object:
+
+```yaml
+model_list:
+ - model_name: switchyard
+ litellm_params:
+ model: provider/capable-model
+ api_key: os.environ/PROVIDER_API_KEY
+ - model_name: switchyard
+ litellm_params:
+ model: provider/efficient-model
+ api_key: os.environ/PROVIDER_API_KEY
+
+router_settings:
+ plugins:
+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN
+
+litellm_settings:
+ callbacks:
+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN
+```
+
+Compose mounts the selected profile at `/app/deployment` and sets
+`SWITCHYARD_LITELLM_CONFIG=/app/deployment/switchyard.toml`. LiteLLM imports the pre-created object
+from the dotted path above when the proxy starts.
+
+### Stage fields
+
+```toml
+algorithm = "stage"
+picker = "efficient_first"
+confidence_threshold = 0.5
+recent_window = 3
+only_on_wrong_signal_escalation = true
+escalation_note = "The efficient tier failed; continue from its work."
+deescalation_note = "The capable tier completed the recovery."
+capable_system_prompt = "Handle this request as the capable tier."
+efficient_system_prompt = "Handle this request as the efficient tier."
+```
+
+| Field | Required | Accepted value |
+|---|---:|---|
+| `algorithm` | yes | `"stage"` |
+| `picker` | yes | `"capable_first"` or `"efficient_first"` |
+| `confidence_threshold` | yes | finite number from `0` through `1` |
+| `recent_window` | no | nonnegative integer; omitted means no fixed recent window |
+| `only_on_wrong_signal_escalation` | no | Boolean; defaults to `true` |
+| `escalation_note` | no | nonempty text appended when Stage hands work to the capable tier |
+| `deescalation_note` | no | nonempty text appended when Stage hands work back; requires `escalation_note` |
+| `capable_system_prompt` | no | nonempty system instruction used for the capable tier |
+| `efficient_system_prompt` | no | nonempty system instruction used for the efficient tier |
+
+Stage requires exactly two unique candidate model IDs. In pinned LiteLLM 1.97.0, the routing context
+preserves the matching `model_list` declaration order. Declare the capable model first and the
+efficient model second, as in the checked-in profile; an integration test locks this ordering
+contract. `picker` controls which tier is chosen without an escalation signal; it does not change
+those roles.
+
+### Random fields
+
+```toml
+algorithm = "random"
+seed = 6
+weights = [0.25, 0.75]
+```
+
+| Field | Required | Accepted value |
+|---|---:|---|
+| `algorithm` | yes | `"random"` |
+| `seed` | no | integer from `0` through `2^64 - 1` |
+| `weights` | no | nonempty array of finite, nonnegative numbers with at least one positive value |
+
+Random selects across every unique candidate ID in first-seen LiteLLM order. `weights` maps to that
+same order, so its length must equal the number of unique candidates in the request. Omitting
+`weights` gives each candidate equal weight; omitting `seed` uses Switchyard's unseeded behavior.
+
+Static errors—missing files, malformed TOML, unknown keys, wrong types, invalid ranges, and invalid
+weight values—fail while LiteLLM imports the plugin at proxy startup. Constraints that depend on
+LiteLLM's live candidates fail at request time: Stage's two-candidate requirement and Random's
+candidate-to-weight count.
+
+To add a profile, copy an existing directory, edit both files, and select its directory name:
+
+```bash
+SWITCHYARD_LITELLM_PROFILE=my-profile \
+ docker compose -f deployment/compose.yaml up -d --build --wait
+```
+
+On a successful decision without a rewrite, the plugin preserves signals from earlier plugins and
+adds:
+
+```python
+context.signals["switchyard"] = {
+ "selected_model_id": "provider/efficient-model",
+ "fallback_models": ["provider/capable-model"],
+}
+```
+
+`fallback_models` is diagnostic metadata only. The plugin narrows the candidate set; it does not add
+a LiteLLM fallback policy. When Switchyard rewrites a supported request field, the signal also
+temporarily contains a private `request_patch`. The callback consumes that patch after deployment
+selection and excludes it from the metadata it returns downstream.
+
+## Use the plugin with LiteLLM's Python Router
+
+For programmatic use, construct the plugin directly; TOML and the environment-backed import module
+are proxy deployment conveniences.
+
+```python
+import litellm
+from litellm import Router
+from switchyard_litellm import StageRoutingPlugin
+
+model_list = [
+ {
+ "model_name": "switchyard",
+ "litellm_params": {"model": "provider/capable-model"},
+ },
+ {
+ "model_name": "switchyard",
+ "litellm_params": {"model": "provider/efficient-model"},
+ },
+]
+plugin = StageRoutingPlugin(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+)
+router = Router(model_list=model_list, plugins=[plugin])
+litellm.callbacks.append(plugin)
+```
+
+The explicit callback registration is required for programmatic `Router` use because LiteLLM's
+constructor accepts routing plugins but not deployment callbacks. In a long-running application,
+register the object once during startup. The proxy YAML shown above performs both registrations
+declaratively.
+
+Install the locked example environment and run the complete example from this directory:
+
+```bash
+uv sync --locked --python 3.12
+uv run --locked --env-file deployment/.env python examples/python_router.py
+```
+
+## Develop against a LiteLLM source checkout
+
+The pinned container is the reproducible default. To work on LiteLLM itself, follow LiteLLM's
+[local development setup](https://docs.litellm.ai/docs/extras/contributing_code#1-setting-up-your-local-dev-environment),
+including its proxy dependencies, then install this Switchyard checkout into that environment.
+Point the plugin loader at a routing TOML and LiteLLM at the matching model YAML:
+
+```bash
+export SWITCHYARD_LITELLM_CONFIG=/absolute/path/to/switchyard-new/examples/litellm/deployment/profiles/stage/switchyard.toml
+PYTHONPATH=/absolute/path/to/switchyard-new/examples/litellm/src \
+ uv run litellm --config \
+ /absolute/path/to/switchyard-new/examples/litellm/deployment/profiles/stage/litellm.yaml
+```
+
+Keep LiteLLM at v1.97.0 when reproducing this example's verified behavior.
+
+## Tests
+
+From the repository root, run the offline suite without an API key or provider calls:
+
+```bash
+PYTHONPATH=examples/litellm/src \
+ uv run --project examples/litellm --locked \
+ pytest examples/litellm/tests -m "not e2e" -v
+```
+
+The paid suite has an explicit opt-in. It builds and starts the local proxy once for each profile
+and verifies the concrete target through LiteLLM's `x-litellm-model-name` response header:
+
+```bash
+SWITCHYARD_LITELLM_E2E=1 \
+ uv run --env-file examples/litellm/deployment/.env \
+ --project examples/litellm --locked \
+ pytest examples/litellm/tests/integration/test_e2e.py -m e2e -v
+```
+
+## Troubleshooting
+
+- If Compose says the key is missing, place `OPENROUTER_API_KEY` in `deployment/.env`, or pass
+ another file with `docker compose --env-file /path/to/.env -f deployment/compose.yaml ...`.
+- If startup reports `SWITCHYARD_LITELLM_CONFIG`, verify that the selected profile contains a
+ readable `switchyard.toml` and that its `algorithm` and fields match the tables above.
+- If port 4000 is occupied, set `LITELLM_PORT` before starting Compose and use that port in requests.
+- If the service is unhealthy, run `docker compose -f deployment/compose.yaml logs litellm`.
+- If routing fails only when a request arrives, verify candidate ordering/count and Random weight
+ count before checking the request's supported message shapes.
+
+## Security
+
+The bundled proxy is unauthenticated and binds only to loopback. It is intended for local
+development. Local `.env` files are excluded from the Docker build context; keep credentials out of
+YAML, TOML, source control, and command output. Configure LiteLLM authentication and follow its
+production deployment guidance before exposing the service.
+
+## References
+
+- [LiteLLM routing plugins](https://docs.litellm.ai/docs/routing_plugins)
+- [Switchyard decision-only API PR #459](https://github.com/NVIDIA-NeMo/Switchyard/pull/459)
+- [Switchyard Python binding updates PR #479](https://github.com/NVIDIA-NeMo/Switchyard/pull/479)
+- [Switchyard Stage routing](../../../docs/routing_algorithms/stage_router_routing.md)
+- [Switchyard Random routing](../../../docs/routing_algorithms/random_routing.md)
diff --git a/examples/litellm/deployment/.env.example b/examples/litellm/deployment/.env.example
new file mode 100644
index 000000000..fbe1e2245
--- /dev/null
+++ b/examples/litellm/deployment/.env.example
@@ -0,0 +1,2 @@
+OPENROUTER_API_KEY=
+SWITCHYARD_LITELLM_PROFILE=stage
diff --git a/examples/litellm/deployment/Dockerfile b/examples/litellm/deployment/Dockerfile
new file mode 100644
index 000000000..482a77b17
--- /dev/null
+++ b/examples/litellm/deployment/Dockerfile
@@ -0,0 +1,41 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+FROM rust:1.96.1-slim-bookworm AS switchyard-builder
+
+RUN apt-get update \
+ && apt-get install --yes --no-install-recommends patchelf python3 python3-pip python3-venv \
+ && rm -rf /var/lib/apt/lists/*
+RUN python3 -m venv /opt/maturin \
+ && /opt/maturin/bin/pip install --no-cache-dir "maturin==1.14.1"
+
+WORKDIR /src
+COPY . .
+RUN /opt/maturin/bin/maturin build \
+ --release \
+ --locked \
+ --compatibility manylinux_2_34 \
+ --out /wheels \
+ --interpreter /usr/bin/python3
+
+FROM ghcr.io/astral-sh/uv:0.8.15 AS uv
+
+FROM ghcr.io/berriai/litellm:v1.97.0
+
+COPY --from=uv /uv /usr/local/bin/uv
+COPY --from=switchyard-builder /wheels /wheels
+RUN uv pip install --python /app/.venv/bin/python --no-deps /wheels/*.whl \
+ && rm -rf /wheels
+
+COPY examples/litellm/src /app/switchyard-plugin/src
+ENV PYTHONPATH=/app/switchyard-plugin/src
+
+RUN python -c "from importlib.metadata import version; \
+from switchyard.libsy import algorithms; \
+from switchyard_litellm import SwitchyardRoutingPlugin; \
+assert version('litellm') == '1.97.0'; \
+SwitchyardRoutingPlugin(algorithms.random(['smoke']))"
+
+RUN addgroup -S -g 10001 litellm \
+ && adduser -S -D -u 10001 -G litellm -h /home/litellm litellm
+USER litellm
diff --git a/examples/experimental/litellm/compose.yaml b/examples/litellm/deployment/compose.yaml
similarity index 64%
rename from examples/experimental/litellm/compose.yaml
rename to examples/litellm/deployment/compose.yaml
index 54ef48e5b..0ac934644 100644
--- a/examples/experimental/litellm/compose.yaml
+++ b/examples/litellm/deployment/compose.yaml
@@ -1,15 +1,19 @@
services:
litellm:
- image: ghcr.io/berriai/litellm:v1.92.0
+ build:
+ context: ../../..
+ dockerfile: examples/litellm/deployment/Dockerfile
+ image: switchyard-litellm:1.97.0
command:
- --config
- - /app/config.yaml
+ - /app/deployment/litellm.yaml
environment:
OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:?set OPENROUTER_API_KEY}
+ SWITCHYARD_LITELLM_CONFIG: /app/deployment/switchyard.toml
ports:
- "127.0.0.1:${LITELLM_PORT:-4000}:4000"
volumes:
- - ./litellm-config.yaml:/app/config.yaml:ro
+ - ./profiles/${SWITCHYARD_LITELLM_PROFILE:-stage}:/app/deployment:ro
healthcheck:
test:
- CMD-SHELL
diff --git a/examples/litellm/deployment/profiles/random/litellm.yaml b/examples/litellm/deployment/profiles/random/litellm.yaml
new file mode 100644
index 000000000..890683c6c
--- /dev/null
+++ b/examples/litellm/deployment/profiles/random/litellm.yaml
@@ -0,0 +1,17 @@
+model_list:
+ - model_name: switchyard
+ litellm_params:
+ model: openrouter/openai/gpt-5.6-sol
+ api_key: os.environ/OPENROUTER_API_KEY
+ - model_name: switchyard
+ litellm_params:
+ model: openrouter/openai/gpt-5.6-terra
+ api_key: os.environ/OPENROUTER_API_KEY
+
+router_settings:
+ plugins:
+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN
+
+litellm_settings:
+ callbacks:
+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN
diff --git a/examples/litellm/deployment/profiles/random/switchyard.toml b/examples/litellm/deployment/profiles/random/switchyard.toml
new file mode 100644
index 000000000..a28c8724b
--- /dev/null
+++ b/examples/litellm/deployment/profiles/random/switchyard.toml
@@ -0,0 +1,2 @@
+algorithm = "random"
+seed = 6
diff --git a/examples/litellm/deployment/profiles/stage/litellm.yaml b/examples/litellm/deployment/profiles/stage/litellm.yaml
new file mode 100644
index 000000000..6748d0599
--- /dev/null
+++ b/examples/litellm/deployment/profiles/stage/litellm.yaml
@@ -0,0 +1,18 @@
+model_list:
+ # Stage roles follow declaration order: capable first, efficient second.
+ - model_name: switchyard
+ litellm_params:
+ model: openrouter/openai/gpt-5.6-sol
+ api_key: os.environ/OPENROUTER_API_KEY
+ - model_name: switchyard
+ litellm_params:
+ model: openrouter/openai/gpt-5.6-terra
+ api_key: os.environ/OPENROUTER_API_KEY
+
+router_settings:
+ plugins:
+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN
+
+litellm_settings:
+ callbacks:
+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN
diff --git a/examples/litellm/deployment/profiles/stage/switchyard.toml b/examples/litellm/deployment/profiles/stage/switchyard.toml
new file mode 100644
index 000000000..d51af3754
--- /dev/null
+++ b/examples/litellm/deployment/profiles/stage/switchyard.toml
@@ -0,0 +1,9 @@
+algorithm = "stage"
+picker = "efficient_first"
+confidence_threshold = 0.5
+recent_window = 3
+only_on_wrong_signal_escalation = true
+escalation_note = "The efficient tier failed; continue from its work."
+deescalation_note = "The capable tier completed the recovery."
+capable_system_prompt = "Handle this request as the capable tier."
+efficient_system_prompt = "Handle this request as the efficient tier."
diff --git a/examples/litellm/examples/python_router.py b/examples/litellm/examples/python_router.py
new file mode 100644
index 000000000..8003e03cf
--- /dev/null
+++ b/examples/litellm/examples/python_router.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Send one request through LiteLLM's Router and the Switchyard Stage plugin."""
+
+import asyncio
+
+import litellm
+from litellm.router import Router
+from litellm.types.llms.openai import AllMessageValues
+from switchyard_litellm import StageRoutingPlugin
+
+MODEL_GROUP = "switchyard"
+MODEL_LIST: list[dict[str, object]] = [
+ {
+ "model_name": MODEL_GROUP,
+ "litellm_params": {
+ "model": "openrouter/openai/gpt-5.6-sol",
+ "api_key": "os.environ/OPENROUTER_API_KEY",
+ },
+ },
+ {
+ "model_name": MODEL_GROUP,
+ "litellm_params": {
+ "model": "openrouter/openai/gpt-5.6-terra",
+ "api_key": "os.environ/OPENROUTER_API_KEY",
+ },
+ },
+]
+STAGE_ROUTING_PLUGIN = StageRoutingPlugin(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+)
+
+
+def critical_tool_history() -> list[AllMessageValues]:
+ """Build a turn whose critical tool result should select GPT-5.6 Sol."""
+ return [
+ {"role": "user", "content": "Fix the failing tests."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "tool_call_id": "call_1",
+ "content": "fatal runtime error: out of memory",
+ },
+ ]
+
+
+async def main() -> None:
+ """Route the example through the plugin-controlled LiteLLM pipeline."""
+ router = Router(model_list=MODEL_LIST, plugins=[STAGE_ROUTING_PLUGIN])
+ litellm.callbacks.append(STAGE_ROUTING_PLUGIN)
+ try:
+ response = await router.acompletion(
+ model=MODEL_GROUP,
+ messages=critical_tool_history(),
+ max_tokens=64,
+ )
+ print("Selected model:", response.model)
+ print("Response:", response.choices[0].message.content)
+ finally:
+ litellm.callbacks.remove(STAGE_ROUTING_PLUGIN)
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/examples/experimental/litellm/pyproject.toml b/examples/litellm/pyproject.toml
similarity index 76%
rename from examples/experimental/litellm/pyproject.toml
rename to examples/litellm/pyproject.toml
index 6dfbfe5de..b7368fb60 100644
--- a/examples/experimental/litellm/pyproject.toml
+++ b/examples/litellm/pyproject.toml
@@ -5,24 +5,23 @@ build-backend = "setuptools.build_meta"
[project]
name = "switchyard-litellm"
version = "0.1.0"
-description = "Experimental LiteLLM gateway client for Switchyard libsy"
+description = "Experimental Switchyard routing plugin for LiteLLM"
readme = "README.md"
requires-python = ">=3.12,<3.13"
license = "Apache-2.0"
dependencies = [
"nemo-switchyard>=0.1.0",
- "litellm==1.92.0",
+ "litellm==1.97.0",
]
[dependency-groups]
dev = [
"pytest>=9,<10",
"pytest-asyncio>=1.3,<2",
- "respx>=0.23,<1",
]
[tool.uv.sources]
-nemo-switchyard = { path = "../../..", editable = true }
+nemo-switchyard = { path = "../..", editable = true }
[tool.setuptools.packages.find]
where = ["src"]
diff --git a/examples/litellm/src/switchyard_litellm/__init__.py b/examples/litellm/src/switchyard_litellm/__init__.py
new file mode 100644
index 000000000..f3ce414d5
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/__init__.py
@@ -0,0 +1,8 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Experimental Switchyard routing plugin for LiteLLM."""
+
+from .plugins import RandomRoutingPlugin, StageRoutingPlugin, SwitchyardRoutingPlugin
+
+__all__ = ["RandomRoutingPlugin", "StageRoutingPlugin", "SwitchyardRoutingPlugin"]
diff --git a/examples/litellm/src/switchyard_litellm/configuration/__init__.py b/examples/litellm/src/switchyard_litellm/configuration/__init__.py
new file mode 100644
index 000000000..03a6102b8
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/configuration/__init__.py
@@ -0,0 +1,18 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Load deployer-owned Switchyard routing plugin configuration."""
+
+from .loader import (
+ CONFIG_ENV,
+ RoutingPlugin,
+ load_routing_plugin,
+ load_routing_plugin_from_environment,
+)
+
+__all__ = [
+ "CONFIG_ENV",
+ "RoutingPlugin",
+ "load_routing_plugin",
+ "load_routing_plugin_from_environment",
+]
diff --git a/examples/litellm/src/switchyard_litellm/configuration/configured_plugin.py b/examples/litellm/src/switchyard_litellm/configuration/configured_plugin.py
new file mode 100644
index 000000000..afa5a7f94
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/configuration/configured_plugin.py
@@ -0,0 +1,10 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Expose the environment-configured plugin object imported by LiteLLM."""
+
+from .loader import load_routing_plugin_from_environment
+
+ROUTING_PLUGIN = load_routing_plugin_from_environment()
+
+__all__ = ["ROUTING_PLUGIN"]
diff --git a/examples/litellm/src/switchyard_litellm/configuration/loader.py b/examples/litellm/src/switchyard_litellm/configuration/loader.py
new file mode 100644
index 000000000..e6bac488d
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/configuration/loader.py
@@ -0,0 +1,271 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Create candidate-bound LiteLLM plugins from Switchyard TOML configuration.
+
+``tomllib`` decodes TOML values into Python types. This module's helpers enforce
+the plugin schema and value constraints after that decoding step.
+"""
+
+from __future__ import annotations
+
+import math
+import os
+import tomllib
+from collections.abc import Collection, Mapping
+from pathlib import Path
+
+from switchyard_litellm.plugins import RandomRoutingPlugin, StageRoutingPlugin
+
+RoutingPlugin = RandomRoutingPlugin | StageRoutingPlugin
+
+CONFIG_ENV = "SWITCHYARD_LITELLM_CONFIG"
+_STAGE_KEYS = frozenset(
+ {
+ "algorithm",
+ "picker",
+ "confidence_threshold",
+ "recent_window",
+ "escalation_note",
+ "deescalation_note",
+ "only_on_wrong_signal_escalation",
+ "capable_system_prompt",
+ "efficient_system_prompt",
+ }
+)
+_RANDOM_KEYS = frozenset({"algorithm", "seed", "weights"})
+_MAX_RANDOM_SEED = (1 << 64) - 1
+
+
+def load_routing_plugin(path: str | Path) -> RoutingPlugin:
+ """Load a Stage or Random routing plugin from TOML."""
+ config_path = Path(path)
+ try:
+ with config_path.open("rb") as config_file:
+ values: Mapping[str, object] = tomllib.load(config_file)
+ except (OSError, tomllib.TOMLDecodeError) as error:
+ raise ValueError(
+ f"Could not load Switchyard routing config {config_path}: {error}"
+ ) from error
+
+ algorithm = _required_string(values, "algorithm", config_path)
+ if algorithm == "stage":
+ return _stage_plugin(values, config_path)
+ if algorithm == "random":
+ return _random_plugin(values, config_path)
+ raise ValueError(
+ f"Switchyard routing config {config_path} has unsupported algorithm {algorithm!r}"
+ )
+
+
+def load_routing_plugin_from_environment(
+ environment: Mapping[str, str] = os.environ,
+) -> RoutingPlugin:
+ """Load the routing plugin named by ``SWITCHYARD_LITELLM_CONFIG``."""
+ path = environment.get(CONFIG_ENV)
+ if not path:
+ raise ValueError(f"{CONFIG_ENV} must name a Switchyard routing TOML file")
+ return load_routing_plugin(path)
+
+
+def _stage_plugin(values: Mapping[str, object], config_path: Path) -> StageRoutingPlugin:
+ """Validate parsed Stage fields and construct its candidate-bound plugin."""
+ _reject_unknown_keys(values, _STAGE_KEYS, config_path)
+ picker = _required_string(values, "picker", config_path)
+ if picker not in {"capable_first", "efficient_first"}:
+ raise ValueError(
+ f"Switchyard routing config {config_path} picker must be "
+ "'capable_first' or 'efficient_first'"
+ )
+ escalation_note = _optional_string(values, "escalation_note", config_path)
+ deescalation_note = _optional_string(values, "deescalation_note", config_path)
+ if deescalation_note is not None and escalation_note is None:
+ raise ValueError(
+ f"Switchyard routing config {config_path} deescalation_note requires escalation_note"
+ )
+ return StageRoutingPlugin(
+ picker=picker,
+ confidence_threshold=_required_confidence_threshold(values, config_path),
+ recent_window=_optional_integer(
+ values,
+ "recent_window",
+ config_path,
+ ),
+ escalation_note=escalation_note,
+ deescalation_note=deescalation_note,
+ only_on_wrong_signal_escalation=_optional_boolean(
+ values,
+ "only_on_wrong_signal_escalation",
+ config_path,
+ default=True,
+ ),
+ capable_system_prompt=_optional_string(
+ values,
+ "capable_system_prompt",
+ config_path,
+ ),
+ efficient_system_prompt=_optional_string(
+ values,
+ "efficient_system_prompt",
+ config_path,
+ ),
+ )
+
+
+def _random_plugin(values: Mapping[str, object], config_path: Path) -> RandomRoutingPlugin:
+ """Validate parsed Random fields and construct its candidate-bound plugin."""
+ _reject_unknown_keys(values, _RANDOM_KEYS, config_path)
+ return RandomRoutingPlugin(
+ seed=_optional_integer(
+ values,
+ "seed",
+ config_path,
+ maximum=_MAX_RANDOM_SEED,
+ ),
+ weights=_optional_weights(values, config_path),
+ )
+
+
+def _reject_unknown_keys(
+ values: Mapping[str, object],
+ allowed: Collection[str],
+ config_path: Path,
+) -> None:
+ """Reject parsed TOML keys outside an algorithm's supported schema."""
+ unknown = sorted(set(values) - set(allowed))
+ if unknown:
+ raise ValueError(
+ f"Switchyard routing config {config_path} has unknown keys: {', '.join(unknown)}"
+ )
+
+
+def _required_string(
+ values: Mapping[str, object],
+ key: str,
+ config_path: Path,
+) -> str:
+ """Return a required nonempty string already decoded by ``tomllib``."""
+ value = values.get(key)
+ if not isinstance(value, str) or not value:
+ raise ValueError(f"Switchyard routing config {config_path} {key} must be a nonempty string")
+ return value
+
+
+def _optional_string(
+ values: Mapping[str, object],
+ key: str,
+ config_path: Path,
+) -> str | None:
+ """Return an optional nonempty string already decoded by ``tomllib``."""
+ if key not in values:
+ return None
+ value = values[key]
+ if not isinstance(value, str) or not value:
+ raise ValueError(f"Switchyard routing config {config_path} {key} must be a nonempty string")
+ return value
+
+
+def _required_confidence_threshold(
+ values: Mapping[str, object],
+ config_path: Path,
+) -> float:
+ """Validate and return the parsed Stage confidence threshold."""
+ value = values.get("confidence_threshold")
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise ValueError(
+ f"Switchyard routing config {config_path} confidence_threshold must be a finite number"
+ )
+ threshold = float(value)
+ if not math.isfinite(threshold):
+ raise ValueError(
+ f"Switchyard routing config {config_path} confidence_threshold must be a finite number"
+ )
+ if not 0.0 <= threshold <= 1.0:
+ raise ValueError(
+ f"Switchyard routing config {config_path} confidence_threshold "
+ "must be between 0 and 1 inclusive"
+ )
+ return threshold
+
+
+def _optional_integer(
+ values: Mapping[str, object],
+ key: str,
+ config_path: Path,
+ *,
+ maximum: int | None = None,
+) -> int | None:
+ """Validate and return an optional nonnegative TOML integer."""
+ if key not in values:
+ return None
+ value = values[key]
+ if isinstance(value, bool) or not isinstance(value, int):
+ raise ValueError(
+ f"Switchyard routing config {config_path} {key} must be a nonnegative integer"
+ )
+ if value < 0 or maximum is not None and value > maximum:
+ upper_bound = f" no greater than {maximum}" if maximum is not None else ""
+ raise ValueError(
+ f"Switchyard routing config {config_path} {key} must be a nonnegative integer"
+ f"{upper_bound}"
+ )
+ return value
+
+
+def _optional_boolean(
+ values: Mapping[str, object],
+ key: str,
+ config_path: Path,
+ *,
+ default: bool,
+) -> bool:
+ """Validate and return an optional TOML Boolean with its default."""
+ value = values.get(key, default)
+ if not isinstance(value, bool):
+ raise ValueError(f"Switchyard routing config {config_path} {key} must be a boolean")
+ return value
+
+
+def _optional_weights(
+ values: Mapping[str, object],
+ config_path: Path,
+) -> list[float] | None:
+ """Validate and normalize optional Random weights decoded from TOML."""
+ if "weights" not in values:
+ return None
+ value = values["weights"]
+ if not isinstance(value, list):
+ raise ValueError(
+ f"Switchyard routing config {config_path} weights must be a nonempty array"
+ )
+ if not value:
+ raise ValueError(
+ f"Switchyard routing config {config_path} weights must be a nonempty array"
+ )
+
+ weights: list[float] = []
+ for item in value:
+ if isinstance(item, bool) or not isinstance(item, (int, float)):
+ raise ValueError(
+ f"Switchyard routing config {config_path} weights must contain only numbers"
+ )
+ weight = float(item)
+ if not math.isfinite(weight):
+ raise ValueError(f"Switchyard routing config {config_path} weights must be finite")
+ if weight < 0.0:
+ raise ValueError(f"Switchyard routing config {config_path} weights must be nonnegative")
+ weights.append(weight)
+
+ if not any(weight > 0.0 for weight in weights):
+ raise ValueError(
+ f"Switchyard routing config {config_path} weights must contain a positive value"
+ )
+ return weights
+
+
+__all__ = [
+ "CONFIG_ENV",
+ "RoutingPlugin",
+ "load_routing_plugin",
+ "load_routing_plugin_from_environment",
+]
diff --git a/examples/litellm/src/switchyard_litellm/plugins/__init__.py b/examples/litellm/src/switchyard_litellm/plugins/__init__.py
new file mode 100644
index 000000000..17944e1b0
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/plugins/__init__.py
@@ -0,0 +1,10 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Candidate-bound Switchyard routing plugins for LiteLLM."""
+
+from .random_routing_plugin import RandomRoutingPlugin
+from .stage_routing_plugin import StageRoutingPlugin
+from .switchyard_routing_plugin import SwitchyardRoutingPlugin
+
+__all__ = ["RandomRoutingPlugin", "StageRoutingPlugin", "SwitchyardRoutingPlugin"]
diff --git a/examples/litellm/src/switchyard_litellm/plugins/lite_llm_request_rewriter.py b/examples/litellm/src/switchyard_litellm/plugins/lite_llm_request_rewriter.py
new file mode 100644
index 000000000..e792ca4f7
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/plugins/lite_llm_request_rewriter.py
@@ -0,0 +1,133 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Apply Switchyard request deltas at LiteLLM's selected-deployment boundary."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from typing import Any, cast
+
+from litellm.integrations.custom_logger import CustomLogger
+from litellm.types.utils import CallTypes
+
+_PATCH_FIELDS = frozenset(
+ {
+ "messages",
+ "tools",
+ "tool_choice",
+ "temperature",
+ "top_p",
+ "top_k",
+ "max_completion_tokens",
+ "response_format",
+ "reasoning_effort",
+ "stream",
+ "parallel_tool_calls",
+ "prompt_cache_key",
+ "prompt_cache_retention",
+ "safety_identifier",
+ "service_tier",
+ "store",
+ "stream_options",
+ "top_logprobs",
+ "user",
+ "stop",
+ }
+)
+
+
+class LiteLLMRequestRewriter(CustomLogger):
+ """Act as the callback half of a dual-role LiteLLM routing plugin."""
+
+ def __init__(self) -> None:
+ super().__init__()
+
+ async def async_pre_call_deployment_hook(
+ self,
+ kwargs: dict[str, Any],
+ call_type: CallTypes | None,
+ ) -> dict[str, Any] | None:
+ """Apply this plugin's private request patch after deployment selection."""
+ del call_type
+ located = self._locate_patch(kwargs)
+ if located is None:
+ return None
+ metadata_key, metadata, signals, switchyard_signal, patch = located
+
+ raw_set_values = patch.get("set", {})
+ if not isinstance(raw_set_values, Mapping):
+ raise ValueError("Switchyard LiteLLM request patch set must be a mapping")
+ set_values = cast(Mapping[str, object], raw_set_values)
+ unknown_set_fields = sorted(set(set_values) - _PATCH_FIELDS)
+ if unknown_set_fields:
+ raise ValueError(
+ "Switchyard LiteLLM request patch cannot set fields: "
+ + ", ".join(unknown_set_fields)
+ )
+ raw_remove_values = patch.get("remove", [])
+ if (
+ isinstance(raw_remove_values, (str, bytes, bytearray))
+ or not isinstance(raw_remove_values, Sequence)
+ or not all(isinstance(item, str) for item in raw_remove_values)
+ ):
+ raise ValueError("Switchyard LiteLLM request patch remove must be a string sequence")
+ unknown_remove_fields = sorted(set(raw_remove_values) - _PATCH_FIELDS)
+ if unknown_remove_fields:
+ raise ValueError(
+ "Switchyard LiteLLM request patch cannot remove fields: "
+ + ", ".join(unknown_remove_fields)
+ )
+
+ rewritten = dict(kwargs)
+ for field in cast(Sequence[str], raw_remove_values):
+ rewritten.pop(field, None)
+ rewritten.update(set_values)
+
+ sanitized_signal = dict(switchyard_signal)
+ sanitized_signal.pop("request_patch", None)
+ sanitized_signals = dict(signals)
+ sanitized_signals["switchyard"] = sanitized_signal
+ sanitized_metadata = dict(metadata)
+ sanitized_metadata["routing_plugin_signals"] = sanitized_signals
+ rewritten[metadata_key] = sanitized_metadata
+ return rewritten
+
+ @staticmethod
+ def _locate_patch(
+ kwargs: Mapping[str, object],
+ ) -> tuple[
+ str,
+ Mapping[str, object],
+ Mapping[str, object],
+ Mapping[str, object],
+ Mapping[str, object],
+ ] | None:
+ """Find a Switchyard request patch in either LiteLLM metadata field."""
+ for metadata_key in ("litellm_metadata", "metadata"):
+ raw_metadata = kwargs.get(metadata_key)
+ if not isinstance(raw_metadata, Mapping):
+ continue
+ metadata = cast(Mapping[str, object], raw_metadata)
+ raw_signals = metadata.get("routing_plugin_signals")
+ if not isinstance(raw_signals, Mapping):
+ continue
+ signals = cast(Mapping[str, object], raw_signals)
+ raw_switchyard_signal = signals.get("switchyard")
+ if not isinstance(raw_switchyard_signal, Mapping):
+ continue
+ switchyard_signal = cast(Mapping[str, object], raw_switchyard_signal)
+ raw_patch = switchyard_signal.get("request_patch")
+ if not isinstance(raw_patch, Mapping):
+ continue
+ return (
+ metadata_key,
+ metadata,
+ signals,
+ switchyard_signal,
+ cast(Mapping[str, object], raw_patch),
+ )
+ return None
+
+
+__all__ = ["LiteLLMRequestRewriter"]
diff --git a/examples/litellm/src/switchyard_litellm/plugins/random_routing_plugin.py b/examples/litellm/src/switchyard_litellm/plugins/random_routing_plugin.py
new file mode 100644
index 000000000..6e6435271
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/plugins/random_routing_plugin.py
@@ -0,0 +1,46 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Bind Random routing to the candidates supplied by LiteLLM."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from litellm.types.router import RoutingContext
+
+from switchyard.libsy import algorithms
+
+from .lite_llm_request_rewriter import LiteLLMRequestRewriter
+from .switchyard_routing_plugin import SwitchyardRoutingPlugin
+
+
+class RandomRoutingPlugin(LiteLLMRequestRewriter):
+ """Randomly select from the current unique LiteLLM candidates."""
+
+ def __init__(
+ self,
+ *,
+ weights: Sequence[float] | None = None,
+ seed: int | None = None,
+ ) -> None:
+ super().__init__()
+ self._weights = tuple(weights) if weights is not None else None
+ self._seed = seed
+ self._plugins: dict[tuple[str, ...], SwitchyardRoutingPlugin] = {}
+
+ async def run(self, context: RoutingContext) -> RoutingContext:
+ """Reuse one Random algorithm for each live LiteLLM candidate pool."""
+ candidates = tuple(dict.fromkeys(context.candidate_models))
+ if not candidates:
+ raise ValueError("Random routing requires at least one LiteLLM candidate")
+ plugin = self._plugins.get(candidates)
+ if plugin is None:
+ plugin = SwitchyardRoutingPlugin(
+ algorithms.random(candidates, weights=self._weights, seed=self._seed)
+ )
+ self._plugins[candidates] = plugin
+ return await plugin.run(context)
+
+
+__all__ = ["RandomRoutingPlugin"]
diff --git a/examples/litellm/src/switchyard_litellm/plugins/request_rewrite.py b/examples/litellm/src/switchyard_litellm/plugins/request_rewrite.py
new file mode 100644
index 000000000..fea62d496
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/plugins/request_rewrite.py
@@ -0,0 +1,286 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Translate Switchyard request deltas into LiteLLM completion arguments."""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping, Sequence
+from typing import cast
+
+_REQUEST_FIELDS = frozenset(
+ {
+ "model",
+ "instructions",
+ "messages",
+ "tools",
+ "tool_choice",
+ "sampling",
+ "output",
+ "reasoning",
+ "stream",
+ "extensions",
+ "preservation",
+ }
+)
+_EXTENSION_FIELDS = {
+ "parallel_tool_calls": "parallel_tool_calls",
+ "prompt_cache_key": "prompt_cache_key",
+ "prompt_cache_retention": "prompt_cache_retention",
+ "safety_identifier": "safety_identifier",
+ "service_tier": "service_tier",
+ "store": "store",
+ "stream_options": "stream_options",
+ "top_logprobs": "top_logprobs",
+ "user": "user",
+ "stop_sequences": "stop",
+}
+
+
+def _mapping(value: object, path: str) -> Mapping[str, object]:
+ """Require a mapping and include its request path in validation errors."""
+ if not isinstance(value, Mapping):
+ raise ValueError(f"{path} must be a mapping")
+ return cast(Mapping[str, object], value)
+
+
+def _sequence(value: object, path: str) -> Sequence[object]:
+ """Require a non-string sequence at the given request path."""
+ if isinstance(value, (str, bytes, bytearray)) or not isinstance(value, Sequence):
+ raise ValueError(f"{path} must be a sequence")
+ return cast(Sequence[object], value)
+
+
+def _text_content(value: object, path: str, separator: str) -> str:
+ """Join normalized text blocks for a LiteLLM message field."""
+ parts: list[str] = []
+ for index, raw_block in enumerate(_sequence(value, path)):
+ block_path = f"{path}[{index}]"
+ block = _mapping(raw_block, block_path)
+ text = block.get("text")
+ if block.get("type") != "text" or not isinstance(text, str):
+ raise ValueError(f"{block_path} cannot safely apply non-text content to LiteLLM")
+ parts.append(text)
+ return separator.join(parts)
+
+
+def _instructions(value: object) -> list[dict[str, object]]:
+ """Convert normalized system and developer instructions to chat messages."""
+ instructions: list[dict[str, object]] = []
+ for index, raw_instruction in enumerate(_sequence(value, "request.instructions")):
+ path = f"request.instructions[{index}]"
+ instruction = _mapping(raw_instruction, path)
+ role = instruction.get("role")
+ if role not in {"system", "developer"}:
+ raise ValueError(f"{path} cannot safely apply role {role!r} to LiteLLM")
+ instructions.append(
+ {
+ "role": role,
+ "content": _text_content(instruction.get("content"), f"{path}.content", "\n\n"),
+ }
+ )
+ return instructions
+
+
+def _tool_call(block: Mapping[str, object], path: str) -> dict[str, object]:
+ """Convert one normalized tool call to OpenAI function-call shape."""
+ call_id = block.get("id")
+ name = block.get("name")
+ if not isinstance(call_id, str) or not call_id:
+ raise ValueError(f"{path}.id cannot safely apply an empty tool-call ID to LiteLLM")
+ if not isinstance(name, str) or not name:
+ raise ValueError(f"{path}.name cannot safely apply an empty tool name to LiteLLM")
+ return {
+ "id": call_id,
+ "type": "function",
+ "function": {
+ "name": name,
+ "arguments": json.dumps(block.get("arguments"), separators=(",", ":")),
+ },
+ }
+
+
+def _tool_result(block: Mapping[str, object], path: str) -> dict[str, object]:
+ """Convert one normalized tool result to an OpenAI tool message."""
+ tool_call_id = block.get("tool_call_id")
+ if not isinstance(tool_call_id, str) or not tool_call_id:
+ raise ValueError(f"{path}.tool_call_id cannot safely apply an empty ID to LiteLLM")
+ return {
+ "role": "tool",
+ "tool_call_id": tool_call_id,
+ "content": _text_content(block.get("content"), f"{path}.content", " "),
+ }
+
+
+def _messages(value: object) -> list[dict[str, object]]:
+ """Convert normalized Switchyard messages to LiteLLM chat messages."""
+ messages: list[dict[str, object]] = []
+ for index, raw_message in enumerate(_sequence(value, "request.messages")):
+ path = f"request.messages[{index}]"
+ message = _mapping(raw_message, path)
+ role = message.get("role")
+ if role not in {"system", "developer", "user", "assistant", "tool"}:
+ raise ValueError(f"{path} cannot safely apply role {role!r} to LiteLLM")
+
+ text_blocks: list[dict[str, object]] = []
+ tool_calls: list[dict[str, object]] = []
+ tool_results: list[dict[str, object]] = []
+ for block_index, raw_block in enumerate(_sequence(message.get("content"), f"{path}.content")):
+ block_path = f"{path}.content[{block_index}]"
+ block = _mapping(raw_block, block_path)
+ block_type = block.get("type")
+ if block_type == "text" and isinstance(block.get("text"), str):
+ text_blocks.append(dict(block))
+ elif block_type == "tool_call" and role == "assistant":
+ tool_calls.append(_tool_call(block, block_path))
+ elif block_type == "tool_result" and role in {"tool", "user"}:
+ tool_results.append(_tool_result(block, block_path))
+ else:
+ raise ValueError(
+ f"{block_path} cannot safely apply content type {block_type!r} to LiteLLM"
+ )
+
+ messages.extend(tool_results)
+ if text_blocks or tool_calls or not tool_results:
+ converted: dict[str, object] = {
+ "role": role,
+ "content": "\n".join(cast(str, block["text"]) for block in text_blocks),
+ }
+ if tool_calls:
+ converted["tool_calls"] = tool_calls
+ if not text_blocks:
+ converted["content"] = None
+ messages.append(converted)
+ return messages
+
+
+def _tools(value: object) -> list[dict[str, object]]:
+ """Convert normalized tool definitions to OpenAI function tools."""
+ tools: list[dict[str, object]] = []
+ for index, raw_tool in enumerate(_sequence(value, "request.tools")):
+ path = f"request.tools[{index}]"
+ tool = _mapping(raw_tool, path)
+ name = tool.get("name")
+ if not isinstance(name, str) or not name:
+ raise ValueError(f"{path}.name cannot safely apply an empty tool name to LiteLLM")
+ function: dict[str, object] = {
+ "name": name,
+ "description": tool.get("description") or "",
+ "parameters": tool.get("parameters", {}),
+ }
+ strict = tool.get("strict")
+ if strict is not None:
+ if not isinstance(strict, bool):
+ raise ValueError(f"{path}.strict cannot safely apply a non-Boolean value")
+ function["strict"] = strict
+ tools.append({"type": "function", "function": function})
+ return tools
+
+
+def _tool_choice(value: object) -> object:
+ """Convert normalized tool-selection policy to LiteLLM form."""
+ if value is None:
+ return None
+ choice = _mapping(value, "request.tool_choice")
+ choice_type = choice.get("type")
+ if choice_type in {"auto", "required", "none"}:
+ return choice_type
+ if choice_type == "tool":
+ data = _mapping(choice.get("data"), "request.tool_choice.data")
+ name = data.get("name")
+ if not isinstance(name, str) or not name:
+ raise ValueError("request.tool_choice cannot safely apply an empty tool name")
+ return {"type": "function", "function": {"name": name}}
+ if choice_type == "raw":
+ return choice.get("data")
+ raise ValueError(f"request.tool_choice cannot safely apply type {choice_type!r}")
+
+
+def _changed_fields(
+ original: Mapping[str, object],
+ rewritten: Mapping[str, object],
+ fields: Sequence[str],
+) -> list[str]:
+ """Return the requested fields whose normalized values changed."""
+ return [field for field in fields if original.get(field) != rewritten.get(field)]
+
+
+def build_request_patch(
+ original_request: Mapping[str, object],
+ rewritten_request: Mapping[str, object],
+ *,
+ selected_model_id: str,
+) -> dict[str, object]:
+ """Build a LiteLLM argument patch for fields explicitly changed by Switchyard."""
+ if set(original_request) != _REQUEST_FIELDS or set(rewritten_request) != _REQUEST_FIELDS:
+ raise ValueError("Switchyard request shape changed and cannot safely apply to LiteLLM")
+ if rewritten_request.get("model") != selected_model_id:
+ raise ValueError("Switchyard request model cannot safely apply to the selected deployment")
+ if original_request.get("preservation") != rewritten_request.get("preservation"):
+ raise ValueError("Switchyard preservation rewrite cannot safely apply to LiteLLM")
+
+ set_values: dict[str, object] = {}
+ remove_values: list[str] = []
+ if _changed_fields(original_request, rewritten_request, ["instructions", "messages"]):
+ set_values["messages"] = [
+ *_instructions(rewritten_request.get("instructions")),
+ *_messages(rewritten_request.get("messages")),
+ ]
+ if original_request.get("tools") != rewritten_request.get("tools"):
+ set_values["tools"] = _tools(rewritten_request.get("tools"))
+ if original_request.get("tool_choice") != rewritten_request.get("tool_choice"):
+ set_values["tool_choice"] = _tool_choice(rewritten_request.get("tool_choice"))
+
+ original_sampling = _mapping(original_request.get("sampling"), "original.sampling")
+ rewritten_sampling = _mapping(rewritten_request.get("sampling"), "request.sampling")
+ for field in ("temperature", "top_p", "top_k"):
+ if original_sampling.get(field) != rewritten_sampling.get(field):
+ set_values[field] = rewritten_sampling.get(field)
+
+ original_output = _mapping(original_request.get("output"), "original.output")
+ rewritten_output = _mapping(rewritten_request.get("output"), "request.output")
+ if original_output.get("max_output_tokens") != rewritten_output.get("max_output_tokens"):
+ set_values["max_completion_tokens"] = rewritten_output.get("max_output_tokens")
+ remove_values.append("max_tokens")
+ if original_output.get("response_format") != rewritten_output.get("response_format"):
+ set_values["response_format"] = rewritten_output.get("response_format")
+
+ original_reasoning = _mapping(original_request.get("reasoning"), "original.reasoning")
+ rewritten_reasoning = _mapping(rewritten_request.get("reasoning"), "request.reasoning")
+ if original_reasoning.get("raw") != rewritten_reasoning.get("raw"):
+ raise ValueError("Switchyard raw reasoning rewrite cannot safely apply to LiteLLM")
+ if original_reasoning.get("effort") != rewritten_reasoning.get("effort"):
+ set_values["reasoning_effort"] = rewritten_reasoning.get("effort")
+
+ if original_request.get("stream") != rewritten_request.get("stream"):
+ stream = rewritten_request.get("stream")
+ if not isinstance(stream, bool):
+ raise ValueError("Switchyard stream rewrite cannot safely apply to LiteLLM")
+ set_values["stream"] = stream
+
+ original_extensions = _mapping(original_request.get("extensions"), "original.extensions")
+ rewritten_extensions = _mapping(rewritten_request.get("extensions"), "request.extensions")
+ original_fields = _mapping(original_extensions.get("fields"), "original.extensions.fields")
+ rewritten_fields = _mapping(rewritten_extensions.get("fields"), "request.extensions.fields")
+ changed_extension_fields = set(original_fields) | set(rewritten_fields)
+ for field in sorted(changed_extension_fields):
+ if original_fields.get(field) == rewritten_fields.get(field):
+ continue
+ litellm_field = _EXTENSION_FIELDS.get(field)
+ if litellm_field is None:
+ raise ValueError(
+ f"Switchyard extension {field!r} cannot safely apply to LiteLLM"
+ )
+ if field in rewritten_fields:
+ set_values[litellm_field] = rewritten_fields[field]
+ else:
+ remove_values.append(litellm_field)
+
+ patch: dict[str, object] = {"set": set_values}
+ if remove_values:
+ patch["remove"] = remove_values
+ return patch
+
+
+__all__ = ["build_request_patch"]
diff --git a/examples/litellm/src/switchyard_litellm/plugins/stage_routing_plugin.py b/examples/litellm/src/switchyard_litellm/plugins/stage_routing_plugin.py
new file mode 100644
index 000000000..a17e23f1b
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/plugins/stage_routing_plugin.py
@@ -0,0 +1,71 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Bind Stage routing to the candidates supplied by LiteLLM."""
+
+from __future__ import annotations
+
+from litellm.types.router import RoutingContext
+
+from switchyard.libsy import algorithms
+
+from .lite_llm_request_rewriter import LiteLLMRequestRewriter
+from .switchyard_routing_plugin import SwitchyardRoutingPlugin
+
+
+class StageRoutingPlugin(LiteLLMRequestRewriter):
+ """Route between LiteLLM's first capable and second efficient candidate.
+
+ LiteLLM 1.97 preserves ``model_list`` declaration order when it constructs
+ the routing context. The checked-in profile and integration tests make that
+ order an explicit deployment contract.
+ """
+
+ def __init__(
+ self,
+ *,
+ picker: str,
+ confidence_threshold: float,
+ recent_window: int | None = None,
+ escalation_note: str | None = None,
+ deescalation_note: str | None = None,
+ only_on_wrong_signal_escalation: bool = True,
+ capable_system_prompt: str | None = None,
+ efficient_system_prompt: str | None = None,
+ ) -> None:
+ super().__init__()
+ self._picker = picker
+ self._confidence_threshold = confidence_threshold
+ self._recent_window = recent_window
+ self._escalation_note = escalation_note
+ self._deescalation_note = deescalation_note
+ self._only_on_wrong_signal_escalation = only_on_wrong_signal_escalation
+ self._capable_system_prompt = capable_system_prompt
+ self._efficient_system_prompt = efficient_system_prompt
+
+ async def run(self, context: RoutingContext) -> RoutingContext:
+ """Build Stage routing from the current LiteLLM candidate order."""
+ candidates = list(dict.fromkeys(context.candidate_models))
+ if len(candidates) != 2:
+ raise ValueError(
+ "Stage routing requires exactly two unique LiteLLM candidates "
+ "ordered as capable then efficient"
+ )
+ plugin = SwitchyardRoutingPlugin(
+ algorithms.stage_router(
+ candidates[0],
+ candidates[1],
+ picker=self._picker,
+ confidence_threshold=self._confidence_threshold,
+ recent_window=self._recent_window,
+ escalation_note=self._escalation_note,
+ deescalation_note=self._deescalation_note,
+ only_on_wrong_signal_escalation=self._only_on_wrong_signal_escalation,
+ capable_system_prompt=self._capable_system_prompt,
+ efficient_system_prompt=self._efficient_system_prompt,
+ )
+ )
+ return await plugin.run(context)
+
+
+__all__ = ["StageRoutingPlugin"]
diff --git a/examples/litellm/src/switchyard_litellm/plugins/switchyard_routing_plugin.py b/examples/litellm/src/switchyard_litellm/plugins/switchyard_routing_plugin.py
new file mode 100644
index 000000000..1532204a1
--- /dev/null
+++ b/examples/litellm/src/switchyard_litellm/plugins/switchyard_routing_plugin.py
@@ -0,0 +1,209 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Use a compatible Switchyard algorithm to narrow LiteLLM routing candidates."""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping, Sequence
+from typing import cast
+
+from litellm.types.router import RoutingContext
+
+from switchyard.libsy import Algorithm, Step
+
+from .lite_llm_request_rewriter import LiteLLMRequestRewriter
+from .request_rewrite import build_request_patch
+
+
+def _mapping(value: object, path: str) -> Mapping[str, object]:
+ """Require a mapping and preserve its message path in validation errors."""
+ if not isinstance(value, Mapping):
+ raise ValueError(f"{path} must be a mapping")
+ return cast(Mapping[str, object], value)
+
+
+def _sequence(value: object, path: str) -> Sequence[object]:
+ """Require a non-string sequence at the given message path."""
+ if isinstance(value, (str, bytes, bytearray)) or not isinstance(value, Sequence):
+ raise ValueError(f"{path} must be a sequence")
+ return cast(Sequence[object], value)
+
+
+def _text_blocks(value: object, path: str, *, allow_none: bool = False) -> list[dict[str, object]]:
+ """Normalize LiteLLM string or text-block content for Switchyard."""
+ if value is None and allow_none:
+ return []
+ if isinstance(value, str):
+ return [{"type": "text", "text": value}]
+
+ blocks: list[dict[str, object]] = []
+ for index, raw_block in enumerate(_sequence(value, path)):
+ block_path = f"{path}[{index}]"
+ block = _mapping(raw_block, block_path)
+ if block.get("type") != "text" or not isinstance(block.get("text"), str):
+ raise ValueError(f"{block_path} content type is not supported")
+ blocks.append({"type": "text", "text": cast(str, block["text"])})
+ return blocks
+
+
+def _tool_calls(value: object, path: str) -> list[dict[str, object]]:
+ """Normalize OpenAI function calls and parse their JSON arguments."""
+ calls: list[dict[str, object]] = []
+ for index, raw_call in enumerate(_sequence(value, path)):
+ call_path = f"{path}[{index}]"
+ call = _mapping(raw_call, call_path)
+ call_id = call.get("id")
+ if call.get("type") != "function":
+ raise ValueError(f"{call_path}.type is not supported")
+ if not isinstance(call_id, str) or not call_id:
+ raise ValueError(f"{call_path}.id must be a non-empty string")
+ function = _mapping(call.get("function"), f"{call_path}.function")
+ name = function.get("name")
+ arguments = function.get("arguments")
+ if not isinstance(name, str) or not name:
+ raise ValueError(f"{call_path}.function.name must be a non-empty string")
+ if not isinstance(arguments, str):
+ raise ValueError(f"{call_path}.function.arguments must be a JSON string")
+ try:
+ parsed_arguments = json.loads(arguments)
+ except json.JSONDecodeError as error:
+ raise ValueError(
+ f"{call_path}.function.arguments must be a valid JSON object"
+ ) from error
+ if not isinstance(parsed_arguments, Mapping):
+ raise ValueError(f"{call_path}.function.arguments must be a valid JSON object")
+ calls.append(
+ {
+ "type": "tool_call",
+ "id": call_id,
+ "name": name,
+ "arguments": dict(parsed_arguments),
+ }
+ )
+ return calls
+
+
+def _messages(structured_messages: Sequence[object]) -> list[dict[str, object]]:
+ """Normalize LiteLLM structured chat history for Switchyard algorithms."""
+ if not structured_messages:
+ raise ValueError("structured_messages must not be empty")
+
+ converted: list[dict[str, object]] = []
+ for index, raw_message in enumerate(structured_messages):
+ path = f"structured_messages[{index}]"
+ message = _mapping(raw_message, path)
+ role = message.get("role")
+ if role in {"system", "developer", "user"}:
+ converted.append(
+ {
+ "role": role,
+ "content": _text_blocks(message.get("content"), f"{path}.content"),
+ }
+ )
+ elif role == "assistant":
+ content: list[dict[str, object]] = list(
+ _text_blocks(message.get("content"), f"{path}.content", allow_none=True)
+ )
+ raw_calls = message.get("tool_calls")
+ if raw_calls is not None:
+ content.extend(_tool_calls(raw_calls, f"{path}.tool_calls"))
+ if not content:
+ raise ValueError(f"{path} must contain text or tool calls")
+ converted.append({"role": "assistant", "content": content})
+ elif role == "tool":
+ call_id = message.get("tool_call_id")
+ if not isinstance(call_id, str) or not call_id:
+ raise ValueError(f"{path}.tool_call_id must be a non-empty string")
+ converted.append(
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": call_id,
+ "content": _text_blocks(message.get("content"), f"{path}.content"),
+ "is_error": None,
+ }
+ ],
+ }
+ )
+ else:
+ raise ValueError(f"{path}.role is not supported")
+ return converted
+
+
+def _request(structured_messages: Sequence[object]) -> dict[str, object]:
+ """Build the canonical normalized request used to detect unsupported rewrites."""
+ return {
+ "model": "auto",
+ "instructions": [],
+ "messages": _messages(structured_messages),
+ "tools": [],
+ "tool_choice": None,
+ "sampling": {"temperature": None, "top_p": None, "top_k": None},
+ "output": {"max_output_tokens": None, "response_format": None},
+ "reasoning": {"effort": None, "raw": None},
+ "stream": False,
+ "extensions": {"fields": {}},
+ "preservation": {"requests": {}, "responses": {}},
+ }
+
+
+class SwitchyardRoutingPlugin(LiteLLMRequestRewriter):
+ """Narrow LiteLLM candidates with a decision-only Switchyard algorithm.
+
+ Compatible algorithms must finish without intermediate model calls or an
+ already-produced response. Supported request rewrites are carried to the
+ selected deployment by the object's LiteLLM callback role.
+ """
+
+ def __init__(self, algorithm: Algorithm) -> None:
+ super().__init__()
+ self._algorithm = algorithm
+
+ async def run(self, context: RoutingContext) -> RoutingContext:
+ """Run Switchyard and retain only its selected LiteLLM candidate."""
+ candidates = list(context.candidate_models)
+ request = _request(context.structured_messages)
+
+ async for step in self._algorithm.run_stream(request):
+ match step:
+ case Step.CallModel(_):
+ raise ValueError(
+ "Switchyard algorithm requested an intermediate model call, "
+ "which a LiteLLM routing plugin cannot serve"
+ )
+ case Step.Done(outcome):
+ if outcome.response is not None:
+ raise ValueError(
+ "Switchyard algorithm produced a response while routing, "
+ "which a LiteLLM routing plugin cannot return"
+ )
+ selected = outcome.selected_model_id
+ if selected not in candidates:
+ raise ValueError(
+ f"Switchyard selected {selected!r}, which is not in LiteLLM's "
+ "candidate pool"
+ )
+ request_patch = build_request_patch(
+ request,
+ outcome.request,
+ selected_model_id=selected,
+ )
+ context.candidate_models = [
+ candidate for candidate in candidates if candidate == selected
+ ]
+ context.signals["switchyard"] = {
+ "selected_model_id": selected,
+ "fallback_models": outcome.fallback_models,
+ }
+ if request_patch["set"] or request_patch.get("remove"):
+ context.signals["switchyard"]["request_patch"] = request_patch
+ return context
+
+ raise ValueError("Switchyard algorithm stream ended without a routing outcome")
+
+
+__all__ = ["SwitchyardRoutingPlugin"]
diff --git a/examples/experimental/litellm/tests/conftest.py b/examples/litellm/tests/conftest.py
similarity index 100%
rename from examples/experimental/litellm/tests/conftest.py
rename to examples/litellm/tests/conftest.py
diff --git a/examples/litellm/tests/integration/test_e2e.py b/examples/litellm/tests/integration/test_e2e.py
new file mode 100644
index 000000000..f271312ad
--- /dev/null
+++ b/examples/litellm/tests/integration/test_e2e.py
@@ -0,0 +1,187 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+import shutil
+import socket
+import subprocess
+from collections.abc import Iterator
+from pathlib import Path
+
+import httpx
+import pytest
+
+PACKAGE_ROOT = Path(__file__).resolve().parents[2]
+DEPLOYMENT_ROOT = PACKAGE_ROOT / "deployment"
+MODEL_GROUP = "switchyard"
+SOL_MODEL = "openrouter/openai/gpt-5.6-sol"
+TERRA_MODEL = "openrouter/openai/gpt-5.6-terra"
+
+
+def _free_port() -> int:
+ """Reserve and return an available loopback port for one Compose project."""
+ with socket.socket() as sock:
+ sock.bind(("127.0.0.1", 0))
+ return int(sock.getsockname()[1])
+
+
+def _require_live_environment() -> None:
+ """Skip paid tests unless the explicit opt-in, key, and Docker are available."""
+ if os.environ.get("SWITCHYARD_LITELLM_E2E") != "1":
+ pytest.skip("SWITCHYARD_LITELLM_E2E=1 is required for paid E2E tests")
+ if not os.environ.get("OPENROUTER_API_KEY"):
+ pytest.skip("OPENROUTER_API_KEY is required for paid E2E tests")
+ if shutil.which("docker") is None:
+ pytest.skip("Docker is required for paid E2E tests")
+ subprocess.run(
+ ["docker", "compose", "version"],
+ check=True,
+ stdout=subprocess.DEVNULL,
+ )
+
+
+def litellm_base_url(request: pytest.FixtureRequest) -> Iterator[str]:
+ """Build one profile, yield its local proxy URL, and tear it down."""
+ _require_live_environment()
+
+ profile = str(request.param)
+ port = _free_port()
+ project = f"switchyard-litellm-{profile}-{os.getpid()}"
+ env = {
+ **os.environ,
+ "LITELLM_NETWORK": f"{project}-network",
+ "LITELLM_PORT": str(port),
+ "SWITCHYARD_LITELLM_PROFILE": profile,
+ }
+ compose = ["docker", "compose", "--project-name", project]
+ try:
+ subprocess.run(
+ [*compose, "up", "-d", "--build", "--wait"],
+ cwd=DEPLOYMENT_ROOT,
+ env=env,
+ check=True,
+ )
+ yield f"http://127.0.0.1:{port}/v1"
+ finally:
+ subprocess.run(
+ [*compose, "down", "--volumes", "--remove-orphans"],
+ cwd=DEPLOYMENT_ROOT,
+ env=env,
+ check=True,
+ )
+
+
+def test_paid_e2e_requires_explicit_spend_opt_in(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Avoid checking Docker or spending unless live E2E is explicitly enabled."""
+ monkeypatch.setenv("OPENROUTER_API_KEY", "test-key")
+ monkeypatch.delenv("SWITCHYARD_LITELLM_E2E", raising=False)
+
+ def fail_if_docker_is_checked(_: str) -> str | None:
+ pytest.fail("Docker must not be checked without the paid E2E opt-in")
+
+ monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)
+ with pytest.raises(pytest.skip.Exception, match="SWITCHYARD_LITELLM_E2E"):
+ _require_live_environment()
+
+
+def test_paid_e2e_requires_openrouter_key(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Require the profile's OpenRouter credential before checking Docker."""
+ monkeypatch.setenv("SWITCHYARD_LITELLM_E2E", "1")
+ monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
+
+ def fail_if_docker_is_checked(_: str) -> str | None:
+ pytest.fail("Docker must not be checked without OPENROUTER_API_KEY")
+
+ monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)
+ with pytest.raises(pytest.skip.Exception, match="OPENROUTER_API_KEY"):
+ _require_live_environment()
+
+
+def _critical_tool_history() -> list[dict[str, object]]:
+ """Build a critical failed-tool turn that Stage must send to capable."""
+ return [
+ {"role": "user", "content": "Fix the failing tests, then report the result."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "tool_call_id": "call_1",
+ "content": "fatal runtime error: out of memory",
+ },
+ ]
+
+
+async def _complete(
+ base_url: str,
+ messages: list[dict[str, object]],
+) -> tuple[dict[str, object], str]:
+ """Send one chat request and return its body and concrete selected model."""
+ async with httpx.AsyncClient(timeout=180) as client:
+ response = await client.post(
+ f"{base_url}/chat/completions",
+ json={"model": MODEL_GROUP, "messages": messages, "max_tokens": 64},
+ )
+ response.raise_for_status()
+ return response.json(), response.headers["x-litellm-model-name"]
+
+
+def _assert_selected(
+ response: dict[str, object],
+ selected_model: str,
+ expected: str,
+) -> None:
+ """Check the public group response and concrete deployment selection."""
+ assert response.get("model") == MODEL_GROUP
+ assert selected_model == expected
+ choices = response.get("choices")
+ assert isinstance(choices, list) and choices
+
+
[email protected]("litellm_base_url", ["stage"], indirect=True)
+async def test_stage_plugin_routes_live_requests_to_both_gpt_5_6_models(
+ litellm_base_url: str,
+) -> None:
+ """Exercise efficient and capable Stage decisions against live deployments."""
+ efficient, efficient_model = await _complete(
+ litellm_base_url,
+ [{"role": "user", "content": "Reply with the word hello."}],
+ )
+ capable, capable_model = await _complete(litellm_base_url, _critical_tool_history())
+
+ _assert_selected(efficient, efficient_model, TERRA_MODEL)
+ _assert_selected(capable, capable_model, SOL_MODEL)
+
+
[email protected]("litellm_base_url", ["random"], indirect=True)
+async def test_random_plugin_routes_live_requests_to_both_gpt_5_6_models(
+ litellm_base_url: str,
+) -> None:
+ """Exercise the seeded Random profile against both live deployments."""
+ selected = []
+ for index in range(2):
+ _, selected_model = await _complete(
+ litellm_base_url,
+ [{"role": "user", "content": f"Reply with the number {index}."}],
+ )
+ selected.append(selected_model)
+
+ assert selected == [SOL_MODEL, TERRA_MODEL]
diff --git a/examples/litellm/tests/integration/test_gateway_config.py b/examples/litellm/tests/integration/test_gateway_config.py
new file mode 100644
index 000000000..b2cb1050e
--- /dev/null
+++ b/examples/litellm/tests/integration/test_gateway_config.py
@@ -0,0 +1,101 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import runpy
+import tomllib
+from pathlib import Path
+
+import yaml
+from switchyard_litellm import StageRoutingPlugin
+
+PACKAGE_ROOT = Path(__file__).resolve().parents[2]
+REPOSITORY_ROOT = PACKAGE_ROOT.parents[1]
+DEPLOYMENT_ROOT = PACKAGE_ROOT / "deployment"
+
+
+def test_example_targets_python_312() -> None:
+ project = tomllib.loads((PACKAGE_ROOT / "pyproject.toml").read_text())
+
+ assert project["project"]["requires-python"] == ">=3.12,<3.13"
+ assert project["project"]["dependencies"] == [
+ "nemo-switchyard>=0.1.0",
+ "litellm==1.97.0",
+ ]
+
+
+def test_lockfile_matches_root_package_version() -> None:
+ root_project = tomllib.loads((REPOSITORY_ROOT / "pyproject.toml").read_text())
+ lock = tomllib.loads((PACKAGE_ROOT / "uv.lock").read_text())
+ locked_root = next(
+ package for package in lock["package"] if package["name"] == "nemo-switchyard"
+ )
+
+ assert locked_root["version"] == root_project["project"]["version"]
+
+
+def test_compose_builds_the_pinned_plugin_image_and_selects_a_profile() -> None:
+ compose = yaml.safe_load((DEPLOYMENT_ROOT / "compose.yaml").read_text())
+ service = compose["services"]["litellm"]
+
+ assert service["build"] == {
+ "context": "../../..",
+ "dockerfile": "examples/litellm/deployment/Dockerfile",
+ }
+ assert service["image"] == "switchyard-litellm:1.97.0"
+ assert service["command"] == ["--config", "/app/deployment/litellm.yaml"]
+ assert service["environment"] == {
+ "OPENROUTER_API_KEY": "${OPENROUTER_API_KEY:?set OPENROUTER_API_KEY}",
+ "SWITCHYARD_LITELLM_CONFIG": "/app/deployment/switchyard.toml",
+ }
+ assert service["volumes"] == [
+ "./profiles/${SWITCHYARD_LITELLM_PROFILE:-stage}:/app/deployment:ro"
+ ]
+ assert (DEPLOYMENT_ROOT / ".env.example").read_text() == (
+ "OPENROUTER_API_KEY=\nSWITCHYARD_LITELLM_PROFILE=stage\n"
+ )
+
+
+def test_dockerfile_runs_the_final_image_as_litellm_user() -> None:
+ """Require the final runtime stage to select the dedicated non-root account."""
+ instructions = [
+ line.strip()
+ for line in (DEPLOYMENT_ROOT / "Dockerfile").read_text().splitlines()
+ if line.strip() and not line.lstrip().startswith("#")
+ ]
+ final_stage = max(index for index, line in enumerate(instructions) if line.startswith("FROM "))
+ runtime_users = [
+ line.split(maxsplit=1)[1]
+ for line in instructions[final_stage + 1 :]
+ if line.startswith("USER ")
+ ]
+
+ assert runtime_users == ["litellm"]
+
+
+def test_deployment_profiles_contain_only_model_and_policy_configuration() -> None:
+ for profile in ("stage", "random"):
+ assert {path.name for path in (DEPLOYMENT_ROOT / "profiles" / profile).iterdir()} == {
+ "litellm.yaml",
+ "switchyard.toml",
+ }
+
+
+def test_python_router_example_owns_its_programmatic_stage_plugin() -> None:
+ namespace = runpy.run_path(str(PACKAGE_ROOT / "examples" / "python_router.py"))
+
+ assert isinstance(namespace["STAGE_ROUTING_PLUGIN"], StageRoutingPlugin)
+ assert [entry["model_name"] for entry in namespace["MODEL_LIST"]] == [
+ "switchyard",
+ "switchyard",
+ ]
+
+
+def test_docker_context_excludes_local_environment_files() -> None:
+ patterns = {
+ line.strip()
+ for line in (REPOSITORY_ROOT / ".dockerignore").read_text().splitlines()
+ if line.strip()
+ }
+
+ assert {".env", "**/.env", ".env.*", "**/.env.*"} <= patterns
+ assert {"!.env.example", "!**/.env.example"} <= patterns
diff --git a/examples/litellm/tests/integration/test_profile_configuration.py b/examples/litellm/tests/integration/test_profile_configuration.py
new file mode 100644
index 000000000..7743bf7bf
--- /dev/null
+++ b/examples/litellm/tests/integration/test_profile_configuration.py
@@ -0,0 +1,99 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import sys
+import tomllib
+from pathlib import Path
+
+import pytest
+import yaml
+from litellm.proxy.types_utils.utils import get_instance_fn
+from switchyard_litellm import RandomRoutingPlugin, StageRoutingPlugin
+from switchyard_litellm.configuration import (
+ CONFIG_ENV,
+ load_routing_plugin_from_environment,
+)
+
+PACKAGE_ROOT = Path(__file__).resolve().parents[2]
+PROFILE_ROOT = PACKAGE_ROOT / "deployment" / "profiles"
+PLUGIN_PATH = "switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN"
+
+
+def _expected_models() -> list[dict[str, object]]:
+ return [
+ {
+ "model_name": "switchyard",
+ "litellm_params": {
+ "model": "openrouter/openai/gpt-5.6-sol",
+ "api_key": "os.environ/OPENROUTER_API_KEY",
+ },
+ },
+ {
+ "model_name": "switchyard",
+ "litellm_params": {
+ "model": "openrouter/openai/gpt-5.6-terra",
+ "api_key": "os.environ/OPENROUTER_API_KEY",
+ },
+ },
+ ]
+
+
+def test_environment_loader_requires_a_path() -> None:
+ with pytest.raises(ValueError, match=CONFIG_ENV):
+ load_routing_plugin_from_environment({})
+
+
+def test_environment_loader_uses_the_configured_path(tmp_path: Path) -> None:
+ config_path = tmp_path / "switchyard.toml"
+ config_path.write_text('algorithm = "random"\nseed = 9\n')
+
+ plugin = load_routing_plugin_from_environment({CONFIG_ENV: str(config_path)})
+
+ assert isinstance(plugin, RandomRoutingPlugin)
+
+
+def test_checked_in_profiles_separate_models_from_routing_policy() -> None:
+ expected_policies = {
+ "stage": {
+ "algorithm": "stage",
+ "picker": "efficient_first",
+ "confidence_threshold": 0.5,
+ "recent_window": 3,
+ "only_on_wrong_signal_escalation": True,
+ "escalation_note": "The efficient tier failed; continue from its work.",
+ "deescalation_note": "The capable tier completed the recovery.",
+ "capable_system_prompt": "Handle this request as the capable tier.",
+ "efficient_system_prompt": "Handle this request as the efficient tier.",
+ },
+ "random": {"algorithm": "random", "seed": 6},
+ }
+
+ for profile, policy in expected_policies.items():
+ profile_root = PROFILE_ROOT / profile
+ assert tomllib.loads((profile_root / "switchyard.toml").read_text()) == policy
+ assert yaml.safe_load((profile_root / "litellm.yaml").read_text()) == {
+ "model_list": _expected_models(),
+ "router_settings": {"plugins": [PLUGIN_PATH]},
+ "litellm_settings": {"callbacks": [PLUGIN_PATH]},
+ }
+
+
+ ("profile", "plugin_type"),
+ [("stage", StageRoutingPlugin), ("random", RandomRoutingPlugin)],
+)
+def test_litellm_resolves_the_profile_configured_plugin(
+ monkeypatch: pytest.MonkeyPatch,
+ profile: str,
+ plugin_type: type[StageRoutingPlugin] | type[RandomRoutingPlugin],
+) -> None:
+ profile_root = PROFILE_ROOT / profile
+ monkeypatch.setenv(CONFIG_ENV, str(profile_root / "switchyard.toml"))
+ sys.modules.pop("switchyard_litellm.configuration.configured_plugin", None)
+
+ plugin = get_instance_fn(
+ value=PLUGIN_PATH,
+ config_file_path=str(profile_root / "litellm.yaml"),
+ )
+
+ assert isinstance(plugin, plugin_type)
diff --git a/examples/litellm/tests/integration/test_stage_routing.py b/examples/litellm/tests/integration/test_stage_routing.py
new file mode 100644
index 000000000..b93d34b4f
--- /dev/null
+++ b/examples/litellm/tests/integration/test_stage_routing.py
@@ -0,0 +1,166 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from typing import Any
+
+import litellm
+import pytest
+from litellm import Router
+from litellm.integrations.custom_logger import CustomLogger
+from switchyard_litellm import RandomRoutingPlugin, StageRoutingPlugin
+
+MODEL_GROUP = "deployer-route"
+CAPABLE_MODEL = "openrouter/example/deployer-capable"
+EFFICIENT_MODEL = "openrouter/example/deployer-efficient"
+
+
+def stage_plugin() -> StageRoutingPlugin:
+ """Build the deployer-configurable Stage policy used by the integration test."""
+ return StageRoutingPlugin(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ )
+
+
+def model_list() -> list[dict[str, Any]]:
+ """Provide real LiteLLM deployments whose provider calls are replaced by responses."""
+ return [
+ {
+ "model_name": MODEL_GROUP,
+ "litellm_params": {
+ "model": CAPABLE_MODEL,
+ "mock_response": "served by capable",
+ },
+ },
+ {
+ "model_name": MODEL_GROUP,
+ "litellm_params": {
+ "model": EFFICIENT_MODEL,
+ "mock_response": "served by efficient",
+ },
+ },
+ ]
+
+
+def critical_tool_history() -> list[dict[str, Any]]:
+ """Build OpenAI chat history containing a decisive Stage escalation signal."""
+ return [
+ {"role": "user", "content": "Fix the failing tests."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "tool_call_id": "call_1",
+ "content": "fatal runtime error: out of memory",
+ },
+ ]
+
+
+def test_litellm_preserves_declaration_order_for_stage_candidates() -> None:
+ """Lock the pinned LiteLLM contract that gives Stage capable then efficient."""
+ router = Router(model_list=model_list(), plugins=[stage_plugin()])
+ deployments = router.get_model_list(model_name=MODEL_GROUP)
+
+ assert deployments is not None
+ assert [deployment["litellm_params"]["model"] for deployment in deployments] == [
+ CAPABLE_MODEL,
+ EFFICIENT_MODEL,
+ ]
+
+
+async def test_stage_plugin_controls_real_litellm_router_pipeline() -> None:
+ router = Router(model_list=model_list(), plugins=[stage_plugin()])
+
+ efficient = await router.acompletion(
+ model=MODEL_GROUP,
+ messages=[{"role": "user", "content": "Say hello."}],
+ )
+ capable = await router.acompletion(
+ model=MODEL_GROUP,
+ messages=critical_tool_history(),
+ )
+
+ assert efficient.choices[0].message.content == "served by efficient"
+ assert capable.choices[0].message.content == "served by capable"
+
+
+async def test_seeded_random_plugin_reaches_both_litellm_candidates() -> None:
+ router = Router(model_list=model_list(), plugins=[RandomRoutingPlugin(seed=6)])
+
+ served = {
+ (
+ await router.acompletion(
+ model=MODEL_GROUP,
+ messages=[{"role": "user", "content": f"Request {index}"}],
+ )
+ )
+ .choices[0]
+ .message.content
+ for index in range(2)
+ }
+
+ assert served == {"served by capable", "served by efficient"}
+
+
+class DeploymentRequestRecorder(CustomLogger):
+ """Record the request after all deployment callbacks have transformed it."""
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.requests: list[dict[str, Any]] = []
+
+ async def async_pre_call_deployment_hook(
+ self,
+ kwargs: dict[str, Any],
+ call_type: object,
+ ) -> None:
+ self.requests.append(kwargs)
+
+
+async def test_stage_rewrite_reaches_the_selected_litellm_deployment(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ plugin = StageRoutingPlugin(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ escalation_note="The efficient tier failed.",
+ capable_system_prompt="Use the capable tier.",
+ )
+ recorder = DeploymentRequestRecorder()
+ monkeypatch.setattr(litellm, "callbacks", [plugin, recorder])
+ router = Router(model_list=model_list(), plugins=[plugin])
+
+ response = await router.acompletion(
+ model=MODEL_GROUP,
+ messages=critical_tool_history(),
+ temperature=0.2,
+ max_tokens=64,
+ )
+
+ assert response.choices[0].message.content == "served by capable"
+ assert recorder.requests
+ deployment_request = recorder.requests[-1]
+ assert deployment_request["messages"][0] == {
+ "role": "system",
+ "content": "Use the capable tier.",
+ }
+ assert deployment_request["messages"][-1] == {
+ "role": "user",
+ "content": "The efficient tier failed.",
+ }
+ assert deployment_request["temperature"] == 0.2
+ assert deployment_request["max_tokens"] == 64
diff --git a/examples/litellm/tests/unit/test_candidate_routing_plugins.py b/examples/litellm/tests/unit/test_candidate_routing_plugins.py
new file mode 100644
index 000000000..0f1cfa1e5
--- /dev/null
+++ b/examples/litellm/tests/unit/test_candidate_routing_plugins.py
@@ -0,0 +1,126 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from typing import Any
+
+import pytest
+import switchyard_litellm
+from litellm.integrations.custom_logger import CustomLogger
+from litellm.types.router import RoutingContext
+
+
+def routing_context(
+ messages: list[dict[str, Any]],
+ candidates: list[str],
+) -> RoutingContext:
+ """Build the LiteLLM boundary object passed to candidate-bound plugins."""
+ return RoutingContext(
+ raw_messages=messages,
+ structured_messages=messages,
+ candidate_models=candidates,
+ metadata={},
+ signals={},
+ )
+
+
+async def test_stage_uses_deployer_candidates_in_capable_then_efficient_order() -> None:
+ plugin_type = getattr(switchyard_litellm, "StageRoutingPlugin", None)
+ assert plugin_type is not None, "StageRoutingPlugin must be part of the public integration API"
+ plugin = plugin_type(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ )
+ assert isinstance(plugin, CustomLogger)
+ context = routing_context(
+ [{"role": "user", "content": "Say hello."}],
+ ["provider/deployer-capable", "provider/deployer-efficient"],
+ )
+
+ await plugin.run(context)
+
+ assert context.candidate_models == ["provider/deployer-efficient"]
+ assert context.signals["switchyard"] == {
+ "selected_model_id": "provider/deployer-efficient",
+ "fallback_models": ["provider/deployer-capable"],
+ }
+
+
+ "candidates",
+ [
+ ["provider/only-one"],
+ ["provider/one", "provider/two", "provider/three"],
+ ],
+)
+async def test_stage_rejects_candidate_pools_without_two_unique_models(
+ candidates: list[str],
+) -> None:
+ plugin = switchyard_litellm.StageRoutingPlugin(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ )
+ context = routing_context([{"role": "user", "content": "Hello."}], candidates)
+
+ with pytest.raises(ValueError, match="exactly two unique LiteLLM candidates"):
+ await plugin.run(context)
+
+
+async def test_stage_treats_duplicate_deployments_as_one_candidate_model() -> None:
+ plugin = switchyard_litellm.StageRoutingPlugin(
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ )
+ context = routing_context(
+ [{"role": "user", "content": "Hello."}],
+ ["provider/capable", "provider/capable", "provider/efficient"],
+ )
+
+ await plugin.run(context)
+
+ assert context.candidate_models == ["provider/efficient"]
+
+
+async def test_seeded_random_uses_and_retains_the_full_deployer_candidate_pool() -> None:
+ plugin_type = getattr(switchyard_litellm, "RandomRoutingPlugin", None)
+ assert plugin_type is not None, "RandomRoutingPlugin must be part of the public integration API"
+ plugin = plugin_type(seed=6)
+ assert isinstance(plugin, CustomLogger)
+ candidates = ["provider/alpha", "provider/beta", "provider/gamma"]
+ selected: list[str] = []
+
+ for index in range(5):
+ context = routing_context(
+ [{"role": "user", "content": f"Request {index}."}],
+ candidates.copy(),
+ )
+ await plugin.run(context)
+ selected.extend(context.candidate_models)
+
+ assert selected == [
+ "provider/beta",
+ "provider/gamma",
+ "provider/gamma",
+ "provider/gamma",
+ "provider/alpha",
+ ]
+
+
+async def test_random_rejects_an_empty_candidate_pool() -> None:
+ plugin = switchyard_litellm.RandomRoutingPlugin(seed=6)
+ context = routing_context([{"role": "user", "content": "Hello."}], [])
+
+ with pytest.raises(ValueError, match="requires at least one LiteLLM candidate"):
+ await plugin.run(context)
+
+
+async def test_random_weights_apply_to_unique_candidate_models() -> None:
+ plugin = switchyard_litellm.RandomRoutingPlugin(weights=[0.0, 1.0], seed=6)
+ context = routing_context(
+ [{"role": "user", "content": "Hello."}],
+ ["provider/alpha", "provider/alpha", "provider/beta"],
+ )
+
+ await plugin.run(context)
+
+ assert context.candidate_models == ["provider/beta"]
diff --git a/examples/litellm/tests/unit/test_routing_plugin_configuration.py b/examples/litellm/tests/unit/test_routing_plugin_configuration.py
new file mode 100644
index 000000000..123fb7681
--- /dev/null
+++ b/examples/litellm/tests/unit/test_routing_plugin_configuration.py
@@ -0,0 +1,248 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from pathlib import Path
+from typing import Any
+
+import pytest
+from litellm.types.router import RoutingContext
+from switchyard_litellm.configuration import load_routing_plugin
+
+
+def routing_context(
+ messages: list[dict[str, Any]],
+ candidates: list[str],
+) -> RoutingContext:
+ """Build the LiteLLM boundary object passed to configured plugins."""
+ return RoutingContext(
+ raw_messages=messages,
+ structured_messages=messages,
+ candidate_models=candidates,
+ metadata={},
+ signals={},
+ )
+
+
+async def test_loads_stage_picker_and_threshold(tmp_path: Path) -> None:
+ config_path = tmp_path / "switchyard.toml"
+ config_path.write_text(
+ 'algorithm = "stage"\n'
+ 'picker = "capable_first"\n'
+ "confidence_threshold = 0.5\n"
+ "recent_window = 3\n"
+ "only_on_wrong_signal_escalation = true\n"
+ )
+ plugin = load_routing_plugin(config_path)
+ context = routing_context(
+ [{"role": "user", "content": "Say hello."}],
+ ["provider/capable", "provider/efficient"],
+ )
+
+ await plugin.run(context)
+
+ assert context.candidate_models == ["provider/capable"]
+
+
+async def test_loads_stage_prompt_and_handoff_rewrites(tmp_path: Path) -> None:
+ config_path = tmp_path / "switchyard.toml"
+ config_path.write_text(
+ 'algorithm = "stage"\n'
+ 'picker = "capable_first"\n'
+ "confidence_threshold = 0.5\n"
+ 'escalation_note = "The efficient tier failed."\n'
+ 'deescalation_note = "The capable tier recovered."\n'
+ 'capable_system_prompt = "Use the capable tier."\n'
+ 'efficient_system_prompt = "Use the efficient tier."\n'
+ )
+ plugin = load_routing_plugin(config_path)
+ context = routing_context(
+ [{"role": "user", "content": "Say hello."}],
+ ["provider/capable", "provider/efficient"],
+ )
+
+ await plugin.run(context)
+ rewritten = await plugin.async_pre_call_deployment_hook(
+ {
+ "messages": [{"role": "user", "content": "Say hello."}],
+ "metadata": {"routing_plugin_signals": context.signals},
+ },
+ None,
+ )
+
+ assert rewritten is not None
+ assert rewritten["messages"][0] == {
+ "role": "system",
+ "content": "Use the capable tier.",
+ }
+
+ settled_messages = [
+ {"role": "user", "content": "Implement the change."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "write_1",
+ "type": "function",
+ "function": {
+ "name": "Write",
+ "arguments": '{"file_path":"x.py","content":"pass"}',
+ },
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "write_1", "content": "Wrote x.py"},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "test_1",
+ "type": "function",
+ "function": {"name": "Bash", "arguments": '{"command":"pytest"}'},
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "test_1", "content": "5 passed in 0.12s"},
+ ]
+ settled_context = routing_context(
+ settled_messages,
+ ["provider/capable", "provider/efficient"],
+ )
+
+ await plugin.run(settled_context)
+ settled_rewrite = await plugin.async_pre_call_deployment_hook(
+ {
+ "messages": settled_messages,
+ "metadata": {"routing_plugin_signals": settled_context.signals},
+ },
+ None,
+ )
+
+ assert settled_context.candidate_models == ["provider/efficient"]
+ assert settled_rewrite is not None
+ assert settled_rewrite["messages"][0] == {
+ "role": "system",
+ "content": "Use the efficient tier.",
+ }
+ assert settled_rewrite["messages"][-1] == {
+ "role": "user",
+ "content": "The capable tier recovered.",
+ }
+
+
+async def test_loads_random_weights(tmp_path: Path) -> None:
+ config_path = tmp_path / "switchyard.toml"
+ config_path.write_text('algorithm = "random"\nseed = 6\nweights = [0.0, 1.0]\n')
+ plugin = load_routing_plugin(config_path)
+ context = routing_context(
+ [{"role": "user", "content": "Say hello."}],
+ ["provider/alpha", "provider/beta"],
+ )
+
+ await plugin.run(context)
+
+ assert context.candidate_models == ["provider/beta"]
+
+
+ ("contents", "match"),
+ [
+ ("", "algorithm"),
+ ("algorithm = 7\n", "algorithm"),
+ ('algorithm = "unknown"\n', "unsupported algorithm"),
+ ('algorithm = "stage"\nconfidence_threshold = 0.5\n', "picker"),
+ (
+ 'algorithm = "stage"\npicker = "fast"\nconfidence_threshold = 0.5\n',
+ "picker",
+ ),
+ ('algorithm = "stage"\npicker = "efficient_first"\n', "confidence_threshold"),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = true\n',
+ "confidence_threshold",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = nan\n',
+ "finite",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = -0.1\n',
+ "between 0 and 1",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = 1.1\n',
+ "between 0 and 1",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\n'
+ "confidence_threshold = 0.5\nrecent_window = true\n",
+ "recent_window",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\n'
+ "confidence_threshold = 0.5\nrecent_window = -1\n",
+ "recent_window",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\n'
+ "confidence_threshold = 0.5\nonly_on_wrong_signal_escalation = 1\n",
+ "only_on_wrong_signal_escalation",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\n'
+ 'confidence_threshold = 0.5\ncapable_system_prompt = ""\n',
+ "capable_system_prompt",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\n'
+ 'confidence_threshold = 0.5\ndeescalation_note = "Recovered."\n',
+ "requires escalation_note",
+ ),
+ (
+ 'algorithm = "stage"\npicker = "efficient_first"\n'
+ "confidence_threshold = 0.5\nextra = true\n",
+ "unknown keys",
+ ),
+ ('algorithm = "random"\nextra = true\n', "unknown keys"),
+ ('algorithm = "random"\nseed = true\n', "seed"),
+ ('algorithm = "random"\nseed = -1\n', "seed"),
+ ('algorithm = "random"\nseed = 18446744073709551616\n', "seed"),
+ ('algorithm = "random"\nweights = 1.0\n', "weights"),
+ ('algorithm = "random"\nweights = []\n', "nonempty"),
+ ('algorithm = "random"\nweights = [true]\n', "weights"),
+ ('algorithm = "random"\nweights = [-1.0, 2.0]\n', "nonnegative"),
+ ('algorithm = "random"\nweights = [nan, 1.0]\n', "finite"),
+ ('algorithm = "random"\nweights = [0.0, 0.0]\n', "positive"),
+ ],
+)
+def test_rejects_invalid_static_configuration(
+ tmp_path: Path,
+ contents: str,
+ match: str,
+) -> None:
+ config_path = tmp_path / "switchyard.toml"
+ config_path.write_text(contents)
+
+ with pytest.raises(ValueError, match=match) as error:
+ load_routing_plugin(config_path)
+
+ assert str(config_path) in str(error.value)
+
+
+def test_wraps_missing_file_with_its_path(tmp_path: Path) -> None:
+ config_path = tmp_path / "missing.toml"
+
+ with pytest.raises(ValueError, match="Could not load") as error:
+ load_routing_plugin(config_path)
+
+ assert str(config_path) in str(error.value)
+
+
+def test_wraps_malformed_toml_with_its_path(tmp_path: Path) -> None:
+ config_path = tmp_path / "switchyard.toml"
+ config_path.write_text('algorithm = "random"\nweights = [\n')
+
+ with pytest.raises(ValueError, match="Could not load") as error:
+ load_routing_plugin(config_path)
+
+ assert str(config_path) in str(error.value)
diff --git a/examples/litellm/tests/unit/test_switchyard_routing_plugin.py b/examples/litellm/tests/unit/test_switchyard_routing_plugin.py
new file mode 100644
index 000000000..3038f0bb5
--- /dev/null
+++ b/examples/litellm/tests/unit/test_switchyard_routing_plugin.py
@@ -0,0 +1,448 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+import pytest
+from litellm.integrations.custom_logger import CustomLogger
+from litellm.types.router import RoutingContext
+from switchyard_litellm import SwitchyardRoutingPlugin
+from switchyard_litellm.plugins.request_rewrite import build_request_patch
+from switchyard_litellm.plugins.switchyard_routing_plugin import _request
+
+from switchyard.libsy import Step, TaskClassifierConfig, algorithms
+
+SOL = "openrouter/openai/gpt-5.6-sol"
+TERRA = "openrouter/openai/gpt-5.6-terra"
+
+
+def routing_context(
+ messages: list[dict[str, Any]],
+ *,
+ candidates: list[str] | None = None,
+) -> RoutingContext:
+ """Build the exact LiteLLM boundary object passed to a routing plugin."""
+ return RoutingContext(
+ raw_messages=messages,
+ structured_messages=messages,
+ candidate_models=candidates or [SOL, TERRA],
+ metadata={"tenant": "test"},
+ signals={"earlier-plugin": {"allowed": True}},
+ )
+
+
+def stage_plugin(**kwargs: object) -> SwitchyardRoutingPlugin:
+ """Build the supported signal-only Stage configuration."""
+ return SwitchyardRoutingPlugin(
+ algorithms.stage_router(
+ SOL,
+ TERRA,
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ **kwargs,
+ )
+ )
+
+
+async def test_stage_converts_tool_history_and_narrows_to_capable_candidate() -> None:
+ context = routing_context(
+ [
+ {"role": "system", "content": "You are a coding agent."},
+ {"role": "developer", "content": [{"type": "text", "text": "Be concise."}]},
+ {"role": "user", "content": "Fix the failing tests."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "tool_call_id": "call_1",
+ "content": "fatal runtime error: out of memory",
+ },
+ ],
+ candidates=[SOL, TERRA, SOL],
+ )
+
+ result = await stage_plugin().run(context)
+
+ assert result is context
+ assert result.candidate_models == [SOL, SOL]
+ assert result.signals == {
+ "earlier-plugin": {"allowed": True},
+ "switchyard": {
+ "selected_model_id": SOL,
+ "fallback_models": [TERRA],
+ },
+ }
+
+
+async def test_litellm_conversion_preserves_stage_tool_signal_input() -> None:
+ """Match direct libsy and LiteLLM-plugin routing over the same tool history."""
+ litellm_messages = [
+ {"role": "user", "content": "Fix the failing tests."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "tool_call_id": "call_1",
+ "content": "fatal runtime error: out of memory",
+ },
+ ]
+ original_messages: list[dict[str, object]] = [
+ {
+ "role": "user",
+ "content": [{"type": "text", "text": "Fix the failing tests."}],
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "Bash",
+ "arguments": {"command": "pytest"},
+ }
+ ],
+ },
+ {
+ "role": "tool",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_call_id": "call_1",
+ "content": [
+ {
+ "type": "text",
+ "text": "fatal runtime error: out of memory",
+ }
+ ],
+ "is_error": None,
+ }
+ ],
+ },
+ ]
+ original_request: dict[str, object] = {
+ "model": "auto",
+ "messages": original_messages,
+ }
+
+ direct_outcome = None
+ direct_algorithm = algorithms.stage_router(
+ SOL,
+ TERRA,
+ picker="efficient_first",
+ confidence_threshold=0.5,
+ recent_window=3,
+ )
+ async for step in direct_algorithm.run_stream(original_request):
+ match step:
+ case Step.Done(outcome):
+ direct_outcome = outcome
+
+ # ToolSignals reads only normalized messages, so exact equality protects every signal input.
+ assert _request(litellm_messages)["messages"] == original_messages
+ assert direct_outcome is not None
+ assert direct_outcome.selected_model_id == SOL
+ assert direct_outcome.fallback_models == [TERRA]
+
+ context = routing_context(litellm_messages)
+ await stage_plugin().run(context)
+
+ assert context.signals["switchyard"] == {
+ "selected_model_id": direct_outcome.selected_model_id,
+ "fallback_models": direct_outcome.fallback_models,
+ }
+
+
+async def test_stage_selects_efficient_candidate_without_tool_signals() -> None:
+ context = routing_context([{"role": "user", "content": "Say hello."}])
+
+ await stage_plugin().run(context)
+
+ assert context.candidate_models == [TERRA]
+ assert context.signals["switchyard"] == {
+ "selected_model_id": TERRA,
+ "fallback_models": [SOL],
+ }
+
+
+ ("messages", "match"),
+ [
+ (
+ [
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "Bash", "arguments": "not-json"},
+ }
+ ],
+ }
+ ],
+ "valid JSON object",
+ ),
+ (
+ [
+ {
+ "role": "user",
+ "content": [
+ {"type": "image_url", "image_url": {"url": "https://example.test/x"}}
+ ],
+ }
+ ],
+ "content.*not supported",
+ ),
+ ([{"role": "function", "content": "legacy"}], "role.*not supported"),
+ ],
+)
+async def test_unsupported_structured_messages_fail_closed(
+ messages: list[dict[str, Any]],
+ match: str,
+) -> None:
+ with pytest.raises(ValueError, match=match):
+ await stage_plugin().run(routing_context(messages))
+
+
+async def test_selection_outside_current_litellm_pool_fails_closed() -> None:
+ plugin = SwitchyardRoutingPlugin(algorithms.random(["openrouter/openai/not-allowed"]))
+
+ with pytest.raises(ValueError, match="not in LiteLLM's candidate pool"):
+ await plugin.run(routing_context([{"role": "user", "content": "hello"}]))
+
+
+async def test_classifier_backed_algorithm_fails_on_intermediate_model_call() -> None:
+ plugin = SwitchyardRoutingPlugin(
+ algorithms.llm_task_classifier(
+ "openrouter/openai/gpt-5.6-judge",
+ TERRA,
+ SOL,
+ config=TaskClassifierConfig(0.5),
+ )
+ )
+
+ with pytest.raises(ValueError, match="intermediate model call"):
+ await plugin.run(routing_context([{"role": "user", "content": "hello"}]))
+
+
+async def test_algorithm_that_produces_a_response_fails_closed() -> None:
+ with pytest.raises(ValueError, match="produced a response while routing"):
+ await SwitchyardRoutingPlugin(algorithms.noop()).run(
+ routing_context([{"role": "user", "content": "hello"}])
+ )
+
+
+async def test_stage_request_rewrite_is_applied_after_litellm_selects_a_deployment() -> None:
+ messages = [
+ {"role": "user", "content": "Fix the failing tests."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "call_1", "content": "out of memory"},
+ ]
+ context = routing_context(messages)
+ plugin = stage_plugin(
+ capable_system_prompt="Use the capable tier.",
+ escalation_note="The efficient tier failed.",
+ )
+
+ await plugin.run(context)
+ rewritten = await plugin.async_pre_call_deployment_hook(
+ {
+ "messages": messages,
+ "temperature": 0.2,
+ "tools": [{"type": "function", "function": {"name": "Bash"}}],
+ "metadata": {"routing_plugin_signals": context.signals},
+ },
+ None,
+ )
+
+ assert isinstance(plugin, CustomLogger)
+ assert rewritten is not None
+ assert rewritten["messages"] == [
+ {"role": "system", "content": "Use the capable tier."},
+ {"role": "user", "content": "Fix the failing tests."},
+ {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "call_1",
+ "type": "function",
+ "function": {
+ "name": "Bash",
+ "arguments": '{"command":"pytest"}',
+ },
+ }
+ ],
+ },
+ {"role": "tool", "tool_call_id": "call_1", "content": "out of memory"},
+ {"role": "user", "content": "The efficient tier failed."},
+ ]
+ assert rewritten["temperature"] == 0.2
+ assert rewritten["tools"] == [{"type": "function", "function": {"name": "Bash"}}]
+ assert "request_patch" not in rewritten["metadata"]["routing_plugin_signals"]["switchyard"]
+ assert "request_patch" in context.signals["switchyard"]
+
+
+def test_request_patch_maps_supported_switchyard_request_overrides() -> None:
+ original = {
+ "model": "auto",
+ "instructions": [],
+ "messages": [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}],
+ "tools": [],
+ "tool_choice": None,
+ "sampling": {"temperature": None, "top_p": None, "top_k": None},
+ "output": {"max_output_tokens": None, "response_format": None},
+ "reasoning": {"effort": None, "raw": None},
+ "stream": False,
+ "extensions": {"fields": {}},
+ "preservation": {"requests": {}, "responses": {}},
+ }
+ rewritten = {
+ **original,
+ "model": SOL,
+ "instructions": [
+ {
+ "role": "developer",
+ "content": [{"type": "text", "text": "Use the selected tier."}],
+ }
+ ],
+ "tools": [
+ {
+ "name": "search",
+ "description": "Search documents",
+ "parameters": {"type": "object", "properties": {}},
+ "strict": True,
+ }
+ ],
+ "tool_choice": {"type": "tool", "data": {"name": "search"}},
+ "sampling": {"temperature": 0.3, "top_p": 0.8, "top_k": 20},
+ "output": {
+ "max_output_tokens": 512,
+ "response_format": {"type": "json_object"},
+ },
+ "reasoning": {"effort": "high", "raw": None},
+ "stream": True,
+ "extensions": {
+ "fields": {
+ "parallel_tool_calls": False,
+ "stop_sequences": ["END"],
+ }
+ },
+ }
+
+ assert build_request_patch(original, rewritten, selected_model_id=SOL) == {
+ "set": {
+ "messages": [
+ {"role": "developer", "content": "Use the selected tier."},
+ {"role": "user", "content": "Hello"},
+ ],
+ "tools": [
+ {
+ "type": "function",
+ "function": {
+ "name": "search",
+ "description": "Search documents",
+ "parameters": {"type": "object", "properties": {}},
+ "strict": True,
+ },
+ }
+ ],
+ "tool_choice": {"type": "function", "function": {"name": "search"}},
+ "temperature": 0.3,
+ "top_p": 0.8,
+ "top_k": 20,
+ "max_completion_tokens": 512,
+ "response_format": {"type": "json_object"},
+ "reasoning_effort": "high",
+ "stream": True,
+ "parallel_tool_calls": False,
+ "stop": ["END"],
+ },
+ "remove": ["max_tokens"],
+ }
+
+
+ "update",
+ [
+ {"reasoning": {"effort": None, "raw": {"budget_tokens": 512}}},
+ {"extensions": {"fields": {"api_key": "must-not-be-forwarded"}}},
+ {"preservation": {"requests": {"openai_chat": {}}, "responses": {}}},
+ ],
+)
+def test_request_patch_rejects_unsafe_or_unrepresentable_overrides(
+ update: dict[str, object],
+) -> None:
+ original = {
+ "model": "auto",
+ "instructions": [],
+ "messages": [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}],
+ "tools": [],
+ "tool_choice": None,
+ "sampling": {"temperature": None, "top_p": None, "top_k": None},
+ "output": {"max_output_tokens": None, "response_format": None},
+ "reasoning": {"effort": None, "raw": None},
+ "stream": False,
+ "extensions": {"fields": {}},
+ "preservation": {"requests": {}, "responses": {}},
+ }
+ rewritten = {**original, "model": SOL, **update}
+
+ with pytest.raises(ValueError, match="cannot safely apply"):
+ build_request_patch(original, rewritten, selected_model_id=SOL)
+
+
+class EmptyAlgorithm:
+ """Algorithm-shaped test double whose stream violates the terminal-step contract."""
+
+ async def run_stream(self, request: dict[str, object]) -> AsyncIterator[object]:
+ if request:
+ return
+ yield object()
+
+
+async def test_algorithm_stream_without_terminal_outcome_fails_closed() -> None:
+ plugin = SwitchyardRoutingPlugin(EmptyAlgorithm()) # type: ignore[arg-type]
+
+ with pytest.raises(ValueError, match="without a routing outcome"):
+ await plugin.run(routing_context([{"role": "user", "content": "hello"}]))
diff --git a/examples/experimental/litellm/uv.lock b/examples/litellm/uv.lock
similarity index 97%
rename from examples/experimental/litellm/uv.lock
rename to examples/litellm/uv.lock
index 950a05de7..326c7e84d 100644
--- a/examples/experimental/litellm/uv.lock
+++ b/examples/litellm/uv.lock
@@ -384,7 +384,7 @@ wheels = [
[[package]]
name = "litellm"
-version = "1.92.0"
+version = "1.97.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aiohttp" },
@@ -396,14 +396,20 @@ dependencies = [
{ name = "jsonschema" },
{ name = "openai" },
{ name = "pydantic" },
+ { name = "pydantic-settings" },
{ name = "python-dotenv" },
{ name = "tiktoken" },
{ name = "tokenizers" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/46/ea/5522be7e0dbcaa7c3fddfd2834f387c6e8eab47c0cc65f2a74657c815af7/litellm-1.92.0.tar.gz", hash = "sha256:773adf5503ee1793289689c899394a83df8122993760d9acd782e32aa798db9d", size = 15098143, upload-time = "2026-07-12T01:15:59.828Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/1a/08/db78a9f53e5688ad0f9e50d5c3bfe616e445aac120eafcca907f04634e48/litellm-1.97.0.tar.gz", hash = "sha256:6f7ce326a2e5385ef850e0b0768d41f502ec79278860090a838511cea067b067", size = 17762282, upload-time = "2026-08-16T00:05:44.724Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/cf/26/8061907b67aa04b7cdf2a41cbc4f8aebfbc722c909a7e4b2aea25def7053/litellm-1.92.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:9c7b2849591a35f7039dc7386a81a8e68fccb5ac2fecaa5122192c1172556b29", size = 19291180, upload-time = "2026-07-12T01:15:48.728Z" },
- { url = "https://files.pythonhosted.org/packages/ba/6c/dc9b0b580ee8af9117abd729d1de25d74fae487a0029ca77049a09f3ad33/litellm-1.92.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:f17499155366afd1eaaeb6fd0c7b55cbd2239dcd72f2fe1f8071a969cc402fae", size = 19289559, upload-time = "2026-07-12T01:15:51.376Z" },
+ { url = "https://files.pythonhosted.org/packages/35/d4/a04f8bda468fb25a0a8a392a6922c9ecf6c122ad9bf9e2f69ebd173e1cac/litellm-1.97.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ff401dd5d66f54b9b474f0652c419fb7bf883fbf5ca64c0bc363acdc98b758b5", size = 24235645, upload-time = "2026-08-16T00:05:20.055Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/51/a055bf5df38112f07970937d95416b22379ee84664b739bfe87a8292e098/litellm-1.97.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:2983b40ed5d8b1bcbbfbc0d66fefa21b04db91b87c05dd680ae77cba74e561ef", size = 23893493, upload-time = "2026-08-16T00:05:23.79Z" },
+ { url = "https://files.pythonhosted.org/packages/14/28/b1cf429493aec5260ac2737dd82f6200c729fd26b336dc21cf001cfbcd8a/litellm-1.97.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e3a1f70d693716b4e8a8108f0a464b0e2c555e274ddbb8ef89c4c59c80be14ed", size = 24031743, upload-time = "2026-08-16T00:05:27.138Z" },
+ { url = "https://files.pythonhosted.org/packages/08/4a/22c49e8bd068bfdab0daba235cc2d2752d76d855ccb8f6dafe7dabf01ca4/litellm-1.97.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:5b56dce7df44a6a9e6caf5379de2578a8cb82831ceabd3d71cc99b370a1015e7", size = 24397362, upload-time = "2026-08-16T00:05:30.89Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/22/f60558230969ac7d860bd93aff6bf6a69bcd8a3ef4f35eea211174ef4e81/litellm-1.97.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:6b360ddc3162c2ed39b64d3f9957a7af70cd9c60cf71f7a4dfa355a0bf05bebc", size = 24107078, upload-time = "2026-08-16T00:05:34.463Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/e4/6c74ff4b188d9399a036d41e86c4c616d95ce5fd9d9b3c42646bd02f270e/litellm-1.97.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:3c4f1dd45e14127f2303769a7ae79482697823e329ae503513d014ceea4dd704", size = 24493640, upload-time = "2026-08-16T00:05:37.85Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/d1/1475c4ecdf43221ab8ee8d0fcc6b469c26d0d80913d0773e18de81cf18f1/litellm-1.97.0-cp310-abi3-win_amd64.whl", hash = "sha256:dce3377207234fc5c5b275a5e234ba056a5051fd178ae8e9a6aeb5d056f12095", size = 24289278, upload-time = "2026-08-16T00:05:41.441Z" },
]
[[package]]
@@ -455,7 +461,7 @@ wheels = [
[[package]]
name = "nemo-switchyard"
version = "0.2.0"
-source = { editable = "../../../" }
+source = { editable = "../../" }
[package.metadata]
@@ -586,6 +592,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" },
]
+[[package]]
+name = "pydantic-settings"
+version = "2.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pydantic" },
+ { name = "python-dotenv" },
+ { name = "typing-inspection" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/68/ca/31c57507b13119d7d3cfa1576dad2911a4861e3be07b579395f4e9d393f9/pydantic_settings-2.15.0.tar.gz", hash = "sha256:694b793e84f766ba76a90ebdefc01d0a9a045dab0382bee70393da93712ad117", size = 261253, upload-time = "2026-08-07T09:24:57.419Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/30/a4/2bffa9f8e804325a09867f0e9d30795c80ea9f8d62560bd1b6ad6220eb2f/pydantic_settings-2.15.0-py3-none-any.whl", hash = "sha256:0ba092c291c94baceb5eff768aa0d56400a457585bc0175925a5a5510303da42", size = 69413, upload-time = "2026-08-07T09:24:55.839Z" },
+]
+
[[package]]
name = "pygments"
version = "2.20.0"
@@ -704,18 +724,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" },
]
-[[package]]
-name = "respx"
-version = "0.23.1"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "httpx" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/43/98/4e55c9c486404ec12373708d015ebce157966965a5ebe7f28ff2c784d41b/respx-0.23.1.tar.gz", hash = "sha256:242dcc6ce6b5b9bf621f5870c82a63997e8e82bc7c947f9ffe272b8f3dd5a780", size = 29243, upload-time = "2026-04-08T14:37:16.008Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/1d/4a/221da6ca167db45693d8d26c7dc79ccfc978a440251bf6721c9aaf251ac0/respx-0.23.1-py2.py3-none-any.whl", hash = "sha256:b18004b029935384bccfa6d7d9d74b4ec9af73a081cc28600fffc0447f4b8c1a", size = 25557, upload-time = "2026-04-08T14:37:14.613Z" },
-]
-
[[package]]
name = "rpds-py"
version = "2026.6.3"
@@ -761,20 +769,18 @@ dependencies = [
dev = [
{ name = "pytest" },
{ name = "pytest-asyncio" },
- { name = "respx" },
]
[package.metadata]
requires-dist = [
- { name = "litellm", specifier = "==1.92.0" },
- { name = "nemo-switchyard", editable = "../../../" },
+ { name = "litellm", specifier = "==1.97.0" },
+ { name = "nemo-switchyard", editable = "../../" },
]
[package.metadata.requires-dev]
dev = [
{ name = "pytest", specifier = ">=9,<10" },
{ name = "pytest-asyncio", specifier = ">=1.3,<2" },
- { name = "respx", specifier = ">=0.23,<1" },
]
[[package]]