Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
refactor(litellm): replace client integration with routing plugin (#532)
Dhruv Nandakumard5aa63383a7f
diff --git a/.dockerignore b/.dockerignoreindex 21ce6789e..cd04b9ce0 100644--- a/.dockerignore+++ b/.dockerignore@@ -1,4 +1,10 @@.git+.env+**/.env+.env.*+**/.env.*+!.env.example+!**/.env.example.mypy_cache.pytest_cache.ruff_cachediff --git a/docs/routing_algorithms/random_routing.md b/docs/routing_algorithms/random_routing.mdindex e889afce2..a90422fc5 100644--- a/docs/routing_algorithms/random_routing.md+++ b/docs/routing_algorithms/random_routing.md@@ -99,7 +99,7 @@ curl http://localhost:4000/v1/chat/completions \## Experimental LiteLLM integrationThe package's-[developer guide](../../examples/experimental/litellm/README.md) is the+[developer guide](../../examples/litellm/README.md) is thecanonical documentation for integrating Switchyard with a Dockerized LiteLLMgateway. Its source-installable client adapts libsy's dictionary contract anddemonstrates weighted random routing between model aliases backed bydiff --git a/examples/experimental/litellm/.env.example b/examples/experimental/litellm/.env.exampledeleted file mode 100644index 9b229f342..000000000--- a/examples/experimental/litellm/.env.example+++ /dev/null@@ -1 +0,0 @@-OPENROUTER_API_KEY=diff --git a/examples/experimental/litellm/README.md b/examples/experimental/litellm/README.mddeleted file mode 100644index 084ce55ec..000000000--- a/examples/experimental/litellm/README.md+++ /dev/null@@ -1,357 +0,0 @@-# Switchyard + LiteLLM--> **Experimental integration:** This example and its Python APIs are-> experimental and may change without notice.--Build on this package when you want to put Switchyard routing in front of a-LiteLLM gateway. It demonstrates the integration end to end, and its client,-Compose configuration, and example code are intended as a copyable starting-point for your own application.--LiteLLM provides the OpenAI-compatible gateway, model aliases, and OpenRouter-provider integration. Switchyard's Stage router makes the routing decision from-the coding agent's recent tool history.-`LiteLLMSyClient` connects Switchyard's normalized libsy requests to their-selected aliases through LiteLLM's asynchronous Completion API and the-Dockerized gateway. Together, they let an application keep routing policy in-Switchyard while LiteLLM owns model access and sends Chat Completions inference-through OpenRouter.--## Request flow--```text-your application → libsy normalized request → Switchyard Stage router- → LiteLLMSyClient → LiteLLM async Completion API- → Dockerized gateway alias → OpenRouter Chat Completions- → selected model-```--The bundled router treats `strong` as the capable tier and `fast` as the-efficient tier. It uses the `efficient_first` picker, so an initial request or-an ambiguous turn falls open to `fast`. Decisive error-recovery signals, such as-a critical failed tool result, route the turn to `strong`. The client asks-LiteLLM's Completion API to call the selected alias through the gateway rather-than embedding a provider model ID in application code.--## Quick start--This example uses these pinned gateway aliases and image:--- `strong` maps to OpenRouter's `openai/gpt-5.6-sol` model.-- `fast` maps to OpenRouter's `moonshotai/kimi-k3` model.-- The Python client pins `litellm==1.92.0`.-- The gateway runs `ghcr.io/berriai/litellm:v1.92.0`.--### Prerequisites--You need Python 3.12, [uv](https://docs.astral.sh/uv/), Docker Compose, an-`OPENROUTER_API_KEY`, and OpenRouter account access to both model IDs. The-example intentionally standardizes on Python 3.12; the pinned LiteLLM release-cannot build on Python 3.14.--### Configure--From the repository root:--```bash-cd examples/experimental/litellm-cp .env.example .env-# Edit .env and set OPENROUTER_API_KEY.-```--### Install from the checkout--From the package directory:--```bash-uv sync --locked --python 3.12-```--### Install the package from source in another uv environment--From the repository root:--```bash-LITELLM_ENV_DIR="$(mktemp -d "${TMPDIR:-/tmp}/switchyard-litellm-env.XXXXXX")"-uv venv --python 3.12 "$LITELLM_ENV_DIR"-uv pip install --python "$LITELLM_ENV_DIR/bin/python" ./examples/experimental/litellm-"$LITELLM_ENV_DIR/bin/python" -c "from switchyard_litellm import LiteLLMSyClient"-rm -rf -- "$LITELLM_ENV_DIR"-```--### Start and inspect LiteLLM--From the package directory:--```bash-docker compose up -d --wait-docker compose ps-curl -fsS http://127.0.0.1:4000/health/liveliness-```--### Run the example--```bash-uv run --locked --python 3.12 python example.py-```--The bundled request includes an out-of-memory tool failure, so the Stage router-selects the capable `strong` alias. Remove the assistant tool call and tool-result to see the same router fall open to `fast`.--## Use in your application--After installing the package, adapt this program to your application's-normalized request shape and routing policy:--```python-import asyncio--from switchyard.libsy import LlmResponse, Step, algorithms-from switchyard_litellm import LiteLLMSyClient---async def main() -> None:- request = {- "model": "auto",- "messages": [- {- "role": "user",- "content": [{"type": "text", "text": "Fix the failing tests."}],- },- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_1",- "name": "Bash",- "arguments": {"command": "pytest"},- }- ],- },- {- "role": "tool",- "content": [- {- "type": "tool_result",- "tool_call_id": "call_1",- "content": [- {- "type": "text",- "text": "fatal runtime error: out of memory",- }- ],- "is_error": True,- }- ],- },- ],- "reasoning": {"effort": "low"},- "output": {"max_output_tokens": 64},- }- client = LiteLLMSyClient()- router = algorithms.stage_router(- "strong",- "fast",- picker="efficient_first",- confidence_threshold=0.5,- recent_window=3,- )- try:- async for step in router.run_stream(request):- match step:- case Step.CallModel(call):- try:- response = await client.call(call.request)- except Exception as error:- call.fail(error)- else:- call.respond(LlmResponse.Agg(response))- case Step.Done(outcome):- print("Decision:", outcome.selected_model_id)- match outcome.response:- case LlmResponse.Agg(response):- print("Response:", response)- case LlmResponse.Stream(stream):- async for event in stream:- print("Response event:", event)- case None:- print("Response:", await client.call(outcome.request))- finally:- await client.aclose()---asyncio.run(main())-```--## Known limitations--This example intentionally supports buffered text and OpenAI-compatible-function-tool traffic. It translates normalized function definitions, tool-choices, assistant tool calls, and text-only tool results into gateway requests,-and normalizes returned function calls for `libsy`.-Media, instructions, non-text tool results, structured output, provider-specific-extensions, preserved raw payloads, and streaming fail explicitly.--The pinned LiteLLM release does not yet recognize Kimi K3's current-`reasoning_effort` support. `LiteLLMSyClient` forwards LiteLLM's-`allowed_openai_params` hint to the gateway so OpenRouter receives that-supported parameter. Recheck this compatibility hint when upgrading LiteLLM.--LiteLLM 1.92 enters its optional proxy MCP path before distinguishing ordinary-OpenAI function tools. `LiteLLMSyClient` disables that bridge for these calls so-the integration does not need LiteLLM's proxy dependencies. Recheck this pinned-compatibility behavior when upgrading LiteLLM.--## Test--From the package directory, run the offline tests without provider calls:--```bash-PYTHONPATH=. uv run --project . --locked --python 3.12 \- pytest tests -m "not e2e" -v-```--The E2E test starts its own LiteLLM gateway and makes two paid OpenRouter Chat-Completions calls through one Stage router. A no-signal request falls open to-`fast`; a critical failed-tool request routes to `strong`.-`SWITCHYARD_LITELLM_E2E=1` is the explicit spend opt-in:--```bash-SWITCHYARD_LITELLM_E2E=1 \-uv run --project . --locked --python 3.12 --env-file .env \- pytest tests/test_e2e.py -m e2e -v-```--## Run the optional three-task Harbor smoke benchmark--See the repository's [Harbor benchmark guide](../../../benchmark/README.md) for-the one-time Harbor patch and dataset preparation. This is a separate test-boundary: the package E2E tests `LiteLLMSyClient`, while Harbor tests-`Harbor → Switchyard server → LiteLLM → OpenRouter`.--From the repository root, start the example gateway:--```bash-docker compose --env-file examples/experimental/litellm/.env \- -f examples/experimental/litellm/compose.yaml up -d --wait-```--Then run the three-task smoke benchmark in the foreground:--```bash-SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \-bash benchmark/run-baseline.sh \- --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --server-config examples/experimental/litellm/benchmark-route.toml \- --task-list-file examples/experimental/litellm/benchmark-tasks.txt \- --model litellm-stage \- --agent codex \- --reasoning-effort low \- --n-tasks 3 \- --n-concurrent 1 \- --max-retries 0 \- --port 4100 \- --foreground-```--The native benchmark route uses the same efficient-first Stage policy but does-not use `LiteLLMSyClient`; Switchyard server translates requests directly to-the shared LiteLLM gateway. The gateway drops parameters unsupported by the-selected provider model, so the benchmark's reasoning effort reaches `strong`-without causing `fast` requests to fail. The sample runs `broken-python`,-`cosign-keyless-signing`, and `jq-data-processing`. Three autonomous coding-agent-tasks can make many paid model calls. Inspect the resulting run under-`benchmark/tb_runs/`, including-`run_manifest.json`, `harbor.log`, `routing_stats_final.json`, and-`jobs/<job>/<task>/agent/trajectory.json`.--The verified run used the benchmark runner's fallback Codex model template:-the task image's pinned Codex 0.144.5 requires `supports_reasoning_summaries`,-which a newer host-generated catalog may omit. If Harbor fails with that missing-field, use the following recovery procedure. It changes no repository source-and is only for that exact error.--Create a temporary `codex` shim that makes the host-only-`codex debug models --bundled` probe fail. The runner then uses its existing-`_fallback_codex_model_template()`:--```bash-CODEX_SHIM_DIR="$(mktemp -d "${TMPDIR:-/tmp}/switchyard-codex.XXXXXX")"-printf '#!/usr/bin/env bash\nexit 1\n' > "$CODEX_SHIM_DIR/codex"-chmod +x "$CODEX_SHIM_DIR/codex"-```--From the repository root, rerun the benchmark with the temporary directory at-the front of `PATH`:--```bash-PATH="$CODEX_SHIM_DIR:$PATH" \-SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \-bash benchmark/run-baseline.sh \- --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --server-config examples/experimental/litellm/benchmark-route.toml \- --task-list-file examples/experimental/litellm/benchmark-tasks.txt \- --model litellm-stage \- --agent codex \- --reasoning-effort low \- --n-tasks 3 \- --n-concurrent 1 \- --max-retries 0 \- --port 4100 \- --foreground-```--After the run, remove only the shim and its empty temporary directory:--```bash-rm -f "$CODEX_SHIM_DIR/codex"-rmdir "$CODEX_SHIM_DIR"-```--## Stop--For the package-directory quick start:--```bash-docker compose down-```--After the repository-root Harbor workflow:--```bash-docker compose --env-file examples/experimental/litellm/.env \- -f examples/experimental/litellm/compose.yaml down-```--## Troubleshooting--- Ensure `OPENROUTER_API_KEY` is present and not blank in `.env`.-- Confirm your OpenRouter account can access `openai/gpt-5.6-sol` and- `moonshotai/kimi-k3`.-- If host port 4000 is occupied, set `LITELLM_PORT` to an unused port and use- it in the health check, gateway URL, and `LiteLLMSyClient` `base_url`. The- bundled `example.py` assumes port 4000 unless you edit it.-- If the container is unhealthy, from the repository root inspect:-- ```bash- docker compose --env-file examples/experimental/litellm/.env \- -f examples/experimental/litellm/compose.yaml logs litellm- ```-- Harbor must use the Docker network named exactly `switchyard-litellm`.-- The benchmark uses Switchyard port 4100 to avoid LiteLLM's host port 4000.--## Security--The gateway is unauthenticated and intended only for local development. Its-port is bound to loopback only. Use LiteLLM authentication and its production-deployment guidance before exposing it remotely.--## References--- [OpenRouter quickstart](https://openrouter.ai/docs/quickstart)-- [GPT-5.6 Sol on OpenRouter](https://openrouter.ai/openai/gpt-5.6-sol)-- [Kimi K3 on OpenRouter](https://openrouter.ai/moonshotai/kimi-k3)-- [LiteLLM gateway quick start](https://docs.litellm.ai/docs/proxy/quick_start)-- [Switchyard Stage-router docs](../../../docs/routing_algorithms/stage_router_routing.md)diff --git a/examples/experimental/litellm/benchmark-route.toml b/examples/experimental/litellm/benchmark-route.tomldeleted file mode 100644index 10c7413a4..000000000--- a/examples/experimental/litellm/benchmark-route.toml+++ /dev/null@@ -1,25 +0,0 @@-schema_version = 1--[llm_clients.litellm]-format = "openai_chat"-base_url = "http://litellm:4000/v1"--[llm_clients.litellm.extra_headers]-authorization = "Bearer not-needed"--[targets.strong]-id = "strong"-llm_client = "litellm"--[targets.fast]-id = "fast"-llm_client = "litellm"--[routes.litellm_stage]-id = "litellm-stage"-type = "stage_router"-capable_target = "strong"-efficient_target = "fast"-picker = "efficient_first"-confidence_threshold = 0.5-recent_turn_window = 3diff --git a/examples/experimental/litellm/benchmark-tasks.txt b/examples/experimental/litellm/benchmark-tasks.txtdeleted file mode 100644index 222e6ed4f..000000000--- a/examples/experimental/litellm/benchmark-tasks.txt+++ /dev/null@@ -1,4 +0,0 @@-# Small deterministic TB Lite smoke sample.-broken-python-cosign-keyless-signing-jq-data-processingdiff --git a/examples/experimental/litellm/example.py b/examples/experimental/litellm/example.pydeleted file mode 100644index 96567d7e2..000000000--- a/examples/experimental/litellm/example.py+++ /dev/null@@ -1,96 +0,0 @@-#!/usr/bin/env python3-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--"""Route one coding-agent turn through LiteLLM with libsy's Stage router."""--import asyncio--from switchyard_litellm import LiteLLMSyClient--from switchyard.libsy import LlmResponse, Step, algorithms---def sy_request() -> dict[str, object]:- """Build a normalized turn whose critical tool failure needs the capable tier."""- return {- "model": "auto",- "messages": [- {- "role": "user",- "content": [- {- "type": "text",- "text": "Fix the failing tests.",- }- ],- },- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_1",- "name": "Bash",- "arguments": {"command": "pytest"},- }- ],- },- {- "role": "tool",- "content": [- {- "type": "tool_result",- "tool_call_id": "call_1",- "content": [- {- "type": "text",- "text": "fatal runtime error: out of memory",- }- ],- "is_error": True,- }- ],- },- ],- "reasoning": {"effort": "low"},- "output": {"max_output_tokens": 96},- }---async def main() -> None:- """Run the Stage router and print its normalized result."""- client = LiteLLMSyClient()- router = algorithms.stage_router(- "strong",- "fast",- picker="efficient_first",- confidence_threshold=0.5,- recent_window=3,- )- try:- async for step in router.run_stream(sy_request()):- match step:- case Step.CallModel(call):- try:- response = await client.call(call.request)- except Exception as error:- call.fail(error)- else:- call.respond(LlmResponse.Agg(response))- case Step.Done(outcome):- print("Decision:", outcome.selected_model_id)- match outcome.response:- case LlmResponse.Agg(response):- print("Response:", response)- case LlmResponse.Stream(stream):- async for event in stream:- print("Response event:", event)- case None:- print("Response:", await client.call(outcome.request))- finally:- await client.aclose()---if __name__ == "__main__":- asyncio.run(main())diff --git a/examples/experimental/litellm/litellm-config.yaml b/examples/experimental/litellm/litellm-config.yamldeleted file mode 100644index 8ab38a113..000000000--- a/examples/experimental/litellm/litellm-config.yaml+++ /dev/null@@ -1,12 +0,0 @@-model_list:- - model_name: strong- litellm_params:- model: openrouter/openai/gpt-5.6-sol- api_key: os.environ/OPENROUTER_API_KEY- - model_name: fast- litellm_params:- model: openrouter/moonshotai/kimi-k3- api_key: os.environ/OPENROUTER_API_KEY--litellm_settings:- drop_params: truediff --git a/examples/experimental/litellm/src/switchyard_litellm/__init__.py b/examples/experimental/litellm/src/switchyard_litellm/__init__.pydeleted file mode 100644index 67131cfd7..000000000--- a/examples/experimental/litellm/src/switchyard_litellm/__init__.py+++ /dev/null@@ -1,8 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--"""Experimental LiteLLM client for Switchyard libsy."""--from .client import LiteLLMSyClient--__all__ = ["LiteLLMSyClient"]diff --git a/examples/experimental/litellm/src/switchyard_litellm/client.py b/examples/experimental/litellm/src/switchyard_litellm/client.pydeleted file mode 100644index 277fbe6fa..000000000--- a/examples/experimental/litellm/src/switchyard_litellm/client.py+++ /dev/null@@ -1,354 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--"""LiteLLM gateway adapter for normalized libsy requests."""--from __future__ import annotations--import json-from collections.abc import Mapping, Sequence-from typing import Any, cast--from litellm import ModelResponse, acompletion-from litellm.exceptions import (- ContextWindowExceededError as LiteLLMContextWindowExceededError,-)--from switchyard.libsy import ContextWindowExceededError--_TEXT_ROLES = {"system", "developer", "user"}-_STOP_REASONS = {- "stop": "end_turn",- "length": "max_tokens",- "tool_calls": "tool_use",- "content_filter": "content_filter",-}---def _mapping(value: object, path: str) -> Mapping[str, object]:- if not isinstance(value, Mapping):- raise ValueError(f"{path} must be a mapping")- return cast(Mapping[str, object], value)---def _sequence(value: object, path: str) -> Sequence[object]:- if isinstance(value, (str, bytes, bytearray)) or not isinstance(value, Sequence):- raise ValueError(f"{path} must be a sequence")- return cast(Sequence[object], value)---def _reject_sequence_payload(- request: Mapping[str, object],- field: str,-) -> None:- value = request.get(field)- if value is None:- return- if _sequence(value, field):- raise ValueError(f"{field} is not supported")---def _reject_extensions_payload(request: Mapping[str, object]) -> None:- value = request.get("extensions")- if value is None:- return- extensions = _mapping(value, "extensions")- if set(extensions) - {"fields"}:- raise ValueError("extensions is not supported")- if "fields" in extensions and _mapping(extensions["fields"], "extensions.fields"):- raise ValueError("extensions is not supported")---def _reject_preservation_payload(request: Mapping[str, object]) -> None:- value = request.get("preservation")- if value is None:- return- preservation = _mapping(value, "preservation")- if set(preservation) - {"requests", "responses"}:- raise ValueError("preservation is not supported")- for field in ("requests", "responses"):- if field in preservation and _mapping(- preservation[field], f"preservation.{field}"- ):- raise ValueError("preservation is not supported")---def _text_block(raw_block: object, path: str) -> dict[str, str]:- block = _mapping(raw_block, path)- if block.get("type") != "text" or not isinstance(block.get("text"), str):- raise ValueError(f"{path} must be a text block")- return {"type": "text", "text": cast(str, block["text"])}---def _tool_call(raw_block: object, path: str) -> dict[str, object]:- block = _mapping(raw_block, path)- call_id = block.get("id")- name = block.get("name")- arguments = block.get("arguments")- if block.get("type") != "tool_call" or not isinstance(call_id, str):- raise ValueError(f"{path} must be a tool-call block")- if not isinstance(name, str):- raise ValueError(f"{path}.name must be a string")- if not isinstance(arguments, Mapping):- raise ValueError(f"{path}.arguments must be a mapping")- return {- "id": call_id,- "type": "function",- "function": {- "name": name,- "arguments": json.dumps(arguments),- },- }---def _tool_result(raw_block: object, path: str) -> dict[str, object]:- block = _mapping(raw_block, path)- call_id = block.get("tool_call_id")- if block.get("type") != "tool_result" or not isinstance(call_id, str):- raise ValueError(f"{path} must be a tool-result block")- content = _sequence(block.get("content"), f"{path}.content")- text = [- _text_block(raw_content, f"{path}.content[{index}]")["text"]- for index, raw_content in enumerate(content)- ]- return {"role": "tool", "tool_call_id": call_id, "content": "\n".join(text)}---def _messages(request: Mapping[str, object]) -> list[dict[str, object]]:- messages = _sequence(request.get("messages"), "messages")- converted: list[dict[str, object]] = []- for message_index, raw_message in enumerate(messages):- message_path = f"messages[{message_index}]"- message = _mapping(raw_message, message_path)- role = message.get("role")- blocks = _sequence(message.get("content"), f"{message_path}.content")- if role in _TEXT_ROLES:- content = [- _text_block(raw_block, f"{message_path}.content[{block_index}]")- for block_index, raw_block in enumerate(blocks)- ]- converted.append({"role": role, "content": content})- elif role == "assistant":- content: list[dict[str, str]] = []- tool_calls: list[dict[str, object]] = []- for block_index, raw_block in enumerate(blocks):- block_path = f"{message_path}.content[{block_index}]"- block = _mapping(raw_block, block_path)- if block.get("type") == "text":- content.append(_text_block(block, block_path))- elif block.get("type") == "tool_call":- tool_calls.append(_tool_call(block, block_path))- else:- raise ValueError(f"{block_path} is not supported")- converted_message: dict[str, object] = {- "role": "assistant",- "content": content or None,- }- if tool_calls:- converted_message["tool_calls"] = tool_calls- converted.append(converted_message)- elif role == "tool":- converted.extend(- _tool_result(raw_block, f"{message_path}.content[{block_index}]")- for block_index, raw_block in enumerate(blocks)- )- else:- raise ValueError(f"{message_path}.role is not supported")- if not converted:- raise ValueError("messages must not be empty")- return converted---def _tools(request: Mapping[str, object]) -> list[dict[str, object]]:- tools = _sequence(request.get("tools", []), "tools")- converted: list[dict[str, object]] = []- for index, raw_tool in enumerate(tools):- path = f"tools[{index}]"- tool = _mapping(raw_tool, path)- name = tool.get("name")- description = tool.get("description")- parameters = tool.get("parameters")- strict = tool.get("strict")- if not isinstance(name, str):- raise ValueError(f"{path}.name must be a string")- if description is not None and not isinstance(description, str):- raise ValueError(f"{path}.description must be a string or null")- if not isinstance(parameters, Mapping):- raise ValueError(f"{path}.parameters must be a mapping")- if strict is not None and not isinstance(strict, bool):- raise ValueError(f"{path}.strict must be a boolean or null")- function: dict[str, object] = {"name": name, "parameters": dict(parameters)}- if description is not None:- function["description"] = description- if strict is not None:- function["strict"] = strict- converted.append({"type": "function", "function": function})- return converted---def _tool_choice(request: Mapping[str, object]) -> object | None:- value = request.get("tool_choice")- if value is None:- return None- choice = _mapping(value, "tool_choice")- choice_type = choice.get("type")- if choice_type in {"auto", "required", "none"}:- return choice_type- if choice_type == "tool":- data = _mapping(choice.get("data"), "tool_choice.data")- name = data.get("name")- if not isinstance(name, str):- raise ValueError("tool_choice.data.name must be a string")- return {"type": "function", "function": {"name": name}}- raise ValueError("tool_choice is not supported")---def _optional_mapping(- request: Mapping[str, object],- field: str,-) -> Mapping[str, object]:- value = request.get(field)- if value is None:- return {}- return _mapping(value, field)---async def _payload(request: Mapping[str, object]) -> dict[str, Any]:- model = request.get("model")- if not isinstance(model, str) or not model:- raise ValueError("model must be a non-empty string")- if request.get("stream") is True:- raise ValueError("stream=True is not supported")- _reject_sequence_payload(request, "instructions")- _reject_extensions_payload(request)- _reject_preservation_payload(request)-- sampling = _optional_mapping(request, "sampling")- output = _optional_mapping(request, "output")- reasoning = _optional_mapping(request, "reasoning")- if sampling.get("top_k") is not None:- raise ValueError("sampling.top_k is not supported")- if output.get("response_format") is not None:- raise ValueError("output.response_format is not supported")- if reasoning.get("raw") is not None:- raise ValueError("reasoning.raw is not supported")-- payload: dict[str, Any] = {- # LiteLLM needs this prefix to select its OpenAI-compatible transport,- # then strips it before sending the selected alias to the gateway.- "model": f"openai/{model}",- "messages": _messages(request),- "stream": False,- }- tools = _tools(request)- if tools:- payload["tools"] = tools- tool_choice = _tool_choice(request)- if tool_choice is not None:- payload["tool_choice"] = tool_choice- for source, target in (("temperature", "temperature"), ("top_p", "top_p")):- value = sampling.get(source)- if value is not None:- payload[target] = value- max_tokens = output.get("max_output_tokens")- if max_tokens is not None:- payload["max_completion_tokens"] = max_tokens- effort = reasoning.get("effort")- if effort is not None:- payload["reasoning_effort"] = effort- return payload---def _usage(response: ModelResponse) -> dict[str, int]:- usage = response.usage- if usage is None:- return {}- prompt_details = usage.prompt_tokens_details- completion_details = usage.completion_tokens_details- cached_value = prompt_details.cached_tokens if prompt_details is not None else None- cached = cached_value if cached_value is not None else 0- normalized = {- "input_tokens": max(usage.prompt_tokens - cached, 0),- "output_tokens": usage.completion_tokens,- "total_tokens": usage.total_tokens,- }- if cached_value is not None:- normalized["cached_input_tokens"] = cached- if completion_details is not None and completion_details.reasoning_tokens is not None:- normalized["reasoning_tokens"] = completion_details.reasoning_tokens- return normalized---def _response(response: ModelResponse) -> dict[str, object]:- if not response.choices:- raise ValueError("LiteLLM returned no choices")- choice = response.choices[0]- text = choice.message.content- content: list[dict[str, object]] = []- if isinstance(text, str) and text:- content.append({"type": "text", "text": text})- for tool_call in choice.message.tool_calls or []:- try:- arguments = json.loads(tool_call.function.arguments)- except (TypeError, json.JSONDecodeError) as error:- raise ValueError("LiteLLM returned invalid tool-call arguments") from error- if not isinstance(arguments, Mapping):- raise ValueError("LiteLLM returned invalid tool-call arguments")- content.append({- "type": "tool_call",- "id": tool_call.id,- "name": tool_call.function.name,- "arguments": arguments,- })- if not content and choice.finish_reason != "content_filter":- raise ValueError("LiteLLM returned no text content")- return {- "id": response.id,- "model": response.model,- "outputs": [- {- "role": "assistant",- "content": content,- "stop_reason": _STOP_REASONS.get(choice.finish_reason, "unknown"),- }- ],- "usage": _usage(response),- }---class LiteLLMSyClient:- """Call the LiteLLM gateway alias selected in a libsy request."""-- def __init__(- self,- *,- base_url: str = "http://127.0.0.1:4000/v1",- api_key: str = "not-needed",- ) -> None:- self._base_url = base_url- self._api_key = api_key-- async def call(- self,- sy_request: Mapping[str, object],- ) -> Mapping[str, object]:- """Send one normalized, buffered text request through LiteLLM."""- payload = await _payload(sy_request)- try:- response = await acompletion(- **payload,- api_base=self._base_url,- api_key=self._api_key,- num_retries=0,- # LiteLLM otherwise imports its optional proxy MCP stack for ordinary- # OpenAI function tools before it determines that they are not MCP tools.- _skip_mcp_handler=True,- allowed_openai_params=["reasoning_effort"],- extra_body={"allowed_openai_params": ["reasoning_effort"]},- )- except LiteLLMContextWindowExceededError as error:- raise ContextWindowExceededError(str(error)) from error- return _response(cast(ModelResponse, response))-- async def aclose(self) -> None:- """Retain asynchronous lifecycle compatibility without owning a transport."""diff --git a/examples/experimental/litellm/tests/test_client.py b/examples/experimental/litellm/tests/test_client.pydeleted file mode 100644index 0f7e6555a..000000000--- a/examples/experimental/litellm/tests/test_client.py+++ /dev/null@@ -1,628 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--import copy-import json--import httpx-import pytest-import respx-from litellm import (- AuthenticationError,- ModelResponse,- RateLimitError,-)-from litellm.exceptions import (- ContextWindowExceededError as LiteLLMContextWindowExceededError,-)-from litellm.types.utils import Choices, Message, Usage-from switchyard_litellm import LiteLLMSyClient--from switchyard.libsy import ContextWindowExceededError--BASE_URL = "http://gateway.test/v1"---def request_body() -> dict[str, object]:- return {- "model": "fast",- "instructions": [],- "messages": [- {- "role": "user",- "content": [{"type": "text", "text": "Say hello."}],- }- ],- "tools": [],- "tool_choice": None,- "sampling": {"temperature": 0.2, "top_p": 0.9, "top_k": None},- "output": {"max_output_tokens": 64, "response_format": None},- "reasoning": {"effort": "low", "raw": None},- "stream": False,- "extensions": {"fields": {}},- "preservation": {"requests": {}, "responses": {}},- }---def gateway_response(model: str = "moonshotai/kimi-k3") -> dict[str, object]:- return {- "id": "chatcmpl-test",- "object": "chat.completion",- "created": 1,- "model": model,- "choices": [- {- "index": 0,- "message": {"role": "assistant", "content": "Hello."},- "finish_reason": "stop",- }- ],- "usage": {- "prompt_tokens": 12,- "completion_tokens": 7,- "total_tokens": 19,- "prompt_tokens_details": {"cached_tokens": 2},- "completion_tokens_details": {"reasoning_tokens": 3},- },- }---def gateway_tool_response() -> dict[str, object]:- response = gateway_response()- response["choices"][0]["message"] = {- "role": "assistant",- "content": None,- "tool_calls": [- {- "id": "call_2",- "type": "function",- "function": {- "name": "Bash",- "arguments": '{"command":"pytest -q"}',- },- }- ],- }- response["choices"][0]["finish_reason"] = "tool_calls"- return response---async def test_call_uses_litellm_async_completion(- monkeypatch: pytest.MonkeyPatch,-) -> None:- captured: dict[str, object] = {}-- async def fake_acompletion(**kwargs: object) -> ModelResponse:- captured.update(kwargs)- return ModelResponse(**gateway_response())-- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)- client = LiteLLMSyClient(base_url=BASE_URL)- try:- await client.call(request_body())- finally:- await client.aclose()-- assert captured["model"] == "openai/fast"- assert captured["api_base"] == BASE_URL- assert captured["api_key"] == "not-needed"- assert captured["num_retries"] == 0- assert captured["allowed_openai_params"] == ["reasoning_effort"]- assert captured["stream"] is False---async def test_call_translates_request_and_normalizes_response() -> None:- route = respx.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(200, json=gateway_response())- )- request = request_body()- original = copy.deepcopy(request)- client = LiteLLMSyClient(base_url=BASE_URL)- try:- response = await client.call(request)- finally:- await client.aclose()-- sent = json.loads(route.calls[0].request.content)- assert sent == {- "model": "fast",- "messages": [- {- "role": "user",- "content": [{"type": "text", "text": "Say hello."}],- }- ],- "temperature": 0.2,- "top_p": 0.9,- "max_completion_tokens": 64,- "reasoning_effort": "low",- "allowed_openai_params": ["reasoning_effort"],- }- assert sent["model"] == "fast"- assert sent["model"] != "openai/fast"- assert request == original- assert response == {- "id": "chatcmpl-test",- "model": "moonshotai/kimi-k3",- "outputs": [- {- "role": "assistant",- "content": [{"type": "text", "text": "Hello."}],- "stop_reason": "end_turn",- }- ],- "usage": {- "input_tokens": 10,- "cached_input_tokens": 2,- "output_tokens": 7,- "total_tokens": 19,- "reasoning_tokens": 3,- },- }---async def test_call_translates_function_tool_history() -> None:- route = respx.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(200, json=gateway_response())- )- request = request_body()- request["messages"] = [- {- "role": "user",- "content": [{"type": "text", "text": "Run the tests."}],- },- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_1",- "name": "Bash",- "arguments": {"command": "pytest"},- }- ],- },- {- "role": "tool",- "content": [- {- "type": "tool_result",- "tool_call_id": "call_1",- "content": [{"type": "text", "text": "1 failed"}],- "is_error": True,- }- ],- },- ]- request["tools"] = [- {- "name": "Bash",- "description": "Run a shell command.",- "parameters": {- "type": "object",- "properties": {"command": {"type": "string"}},- "required": ["command"],- },- "strict": True,- }- ]- request["tool_choice"] = {"type": "tool", "data": {"name": "Bash"}}-- client = LiteLLMSyClient(base_url=BASE_URL)- try:- await client.call(request)- finally:- await client.aclose()-- sent = json.loads(route.calls[0].request.content)- assert sent["messages"] == [- {- "role": "user",- "content": [{"type": "text", "text": "Run the tests."}],- },- {- "role": "assistant",- "tool_calls": [- {- "id": "call_1",- "type": "function",- "function": {- "name": "Bash",- "arguments": '{"command": "pytest"}',- },- }- ],- },- {"role": "tool", "tool_call_id": "call_1", "content": "1 failed"},- ]- assert sent["tools"] == [- {- "type": "function",- "function": {- "name": "Bash",- "description": "Run a shell command.",- "parameters": {- "type": "object",- "properties": {"command": {"type": "string"}},- "required": ["command"],- },- "strict": True,- },- }- ]- assert sent["tool_choice"] == {- "type": "function",- "function": {"name": "Bash"},- }---async def test_call_normalizes_a_function_tool_response(- monkeypatch: pytest.MonkeyPatch,-) -> None:- async def fake_acompletion(**_: object) -> ModelResponse:- return ModelResponse(**gateway_tool_response())-- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)- client = LiteLLMSyClient(base_url=BASE_URL)- try:- response = await client.call(request_body())- finally:- await client.aclose()-- assert response["outputs"] == [- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_2",- "name": "Bash",- "arguments": {"command": "pytest -q"},- }- ],- "stop_reason": "tool_use",- }- ]---async def test_call_rejects_non_mapping_tool_call_arguments(- monkeypatch: pytest.MonkeyPatch,-) -> None:- async def fake_acompletion(**_: object) -> ModelResponse:- return ModelResponse(**gateway_response())-- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)- request = request_body()- request["messages"] = [- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_1",- "name": "Bash",- "arguments": ["pytest"],- }- ],- }- ]- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ValueError, match=r"messages\[0\]\.content\[0\]\.arguments"):- await client.call(request)- finally:- await client.aclose()---async def test_call_rejects_non_mapping_response_tool_call_arguments(- monkeypatch: pytest.MonkeyPatch,-) -> None:- payload = gateway_tool_response()- tool_call = payload["choices"][0]["message"]["tool_calls"][0]- tool_call["function"]["arguments"] = '["pytest"]'-- async def fake_acompletion(**_: object) -> ModelResponse:- return ModelResponse(**payload)-- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ValueError, match="invalid tool-call arguments"):- await client.call(request_body())- finally:- await client.aclose()--[email protected]("choice", ["auto", "required", "none"])-async def test_call_translates_standard_tool_choice(- monkeypatch: pytest.MonkeyPatch,- choice: str,-) -> None:- captured: dict[str, object] = {}-- async def fake_acompletion(**kwargs: object) -> ModelResponse:- captured.update(kwargs)- return ModelResponse(**gateway_response())-- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)- request = request_body()- request["tools"] = [{"name": "Bash", "description": None, "parameters": {}}]- request["tool_choice"] = {"type": choice}- client = LiteLLMSyClient(base_url=BASE_URL)- try:- await client.call(request)- finally:- await client.aclose()-- assert captured["tool_choice"] == choice--- ("mutate", "match"),- [- (lambda body: body.update(messages="not-a-sequence"), "messages"),- (lambda body: body.update(sampling=[]), "sampling"),- (lambda body: body.update(output=[]), "output"),- (lambda body: body.update(reasoning=[]), "reasoning"),- (lambda body: body.update(extensions=[]), "extensions"),- (lambda body: body.update(preservation=[]), "preservation"),- (lambda body: body.update(stream=True), "stream"),- (- lambda body: body["instructions"].append(- {"role": "system", "content": [{"type": "text", "text": "x"}]}- ),- "instructions",- ),- (- lambda body: body["messages"][0]["content"].append(- {"type": "image", "source": {"type": "url", "data": {"url": "x"}}}- ),- r"messages\[0\]\.content\[1\]",- ),- (- lambda body: body.update(- messages=[- {- "role": "tool",- "content": [- {- "type": "tool_result",- "tool_call_id": "call_1",- "content": [- {- "type": "image",- "source": {- "type": "url",- "data": {"url": "x"},- },- }- ],- "is_error": False,- }- ],- }- ]- ),- r"messages\[0\]\.content\[0\]\.content\[0\]",- ),- (- lambda body: body.update(tool_choice={"type": "raw", "data": {}}),- "tool_choice",- ),- (- lambda body: body["sampling"].update(top_k=4),- "sampling.top_k",- ),- (- lambda body: body["output"].update(response_format={"type": "json_object"}),- "output.response_format",- ),- (- lambda body: body["reasoning"].update(raw={"provider": "value"}),- "reasoning.raw",- ),- (- lambda body: body["extensions"]["fields"].update(provider="value"),- "extensions",- ),- (- lambda body: body["extensions"]["fields"].update(provider=None),- "extensions",- ),- (- lambda body: body["preservation"]["requests"].update(provider=None),- "preservation",- ),- (- lambda body: body["preservation"]["responses"].update(provider={}),- "preservation",- ),- ],-)-async def test_call_rejects_unsupported_normalized_fields(- mutate: object,- match: str,-) -> None:- request = request_body()- assert callable(mutate)- mutate(request)- with respx.mock(assert_all_called=False) as router:- router.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(200, json=gateway_response())- )- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ValueError, match=match):- await client.call(request)- finally:- await client.aclose()---async def test_call_rejects_missing_messages() -> None:- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ValueError, match="messages"):- await client.call({"model": "fast"})- finally:- await client.aclose()---async def test_call_rejects_a_response_without_text() -> None:- payload = gateway_response()- payload["choices"][0]["message"]["content"] = None- respx.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(200, json=payload)- )- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ValueError, match="no text content"):- await client.call(request_body())- finally:- await client.aclose()---async def test_call_rejects_a_response_without_choices(- monkeypatch: pytest.MonkeyPatch,-) -> None:- payload = gateway_response()- payload["choices"] = []-- async def fake_acompletion(**_: object) -> ModelResponse:- return ModelResponse(**payload)-- monkeypatch.setattr("switchyard_litellm.client.acompletion", fake_acompletion)- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ValueError, match="no choices"):- await client.call(request_body())- finally:- await client.aclose()---async def test_litellm_errors_propagate() -> None:- respx.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(- 401,- json={- "error": {- "message": "Incorrect API key provided",- "type": "authentication_error",- "code": "invalid_api_key",- }- },- )- )- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(AuthenticationError):- await client.call(request_body())- finally:- await client.aclose()---async def test_retryable_litellm_error_is_not_retried() -> None:- route = respx.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(- 429,- json={- "error": {- "message": "rate limited",- "type": "rate_limit_error",- "code": "rate_limit_exceeded",- }- },- )- )- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(RateLimitError):- await client.call(request_body())- finally:- await client.aclose()-- assert route.call_count == 1---async def test_context_window_error_is_preserved(- monkeypatch: pytest.MonkeyPatch,-) -> None:- async def fail_with_context_window(**_: object) -> ModelResponse:- raise LiteLLMContextWindowExceededError(- message="request is too large",- model="fast",- llm_provider="openai",- )-- monkeypatch.setattr(- "switchyard_litellm.client.acompletion", fail_with_context_window- )- client = LiteLLMSyClient(base_url=BASE_URL)- try:- with pytest.raises(ContextWindowExceededError, match="request is too large"):- await client.call(request_body())- finally:- await client.aclose()---async def test_cached_token_count_preserves_explicit_zero() -> None:- payload = gateway_response()- payload["usage"]["prompt_tokens_details"]["cached_tokens"] = 0- respx.post(f"{BASE_URL}/chat/completions").mock(- return_value=httpx.Response(200, json=payload)- )- client = LiteLLMSyClient(base_url=BASE_URL)- try:- response = await client.call(request_body())- finally:- await client.aclose()-- assert response["usage"]["cached_input_tokens"] == 0---def test_response_content_filter_empty_content() -> None:- """Empty content with a content_filter finish reason must be normalized."""- from switchyard_litellm.client import _response-- response = ModelResponse(- id="chatcmpl-test",- model="openai/strong",- choices=[- Choices(- message=Message(content=None, tool_calls=None),- finish_reason="content_filter",- )- ],- usage=Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0),- )- result = _response(response)- outputs = result["outputs"]- assert isinstance(outputs, list)- output = outputs[0]- assert isinstance(output, dict)- assert output["content"] == []- assert output["stop_reason"] == "content_filter"---def test_response_empty_content_without_content_filter_raises() -> None:- """Empty content with any other finish reason must still raise."""- from switchyard_litellm.client import _response-- response = ModelResponse(- id="chatcmpl-test",- model="openai/strong",- choices=[- Choices(- message=Message(content=None, tool_calls=None),- finish_reason="stop",- )- ],- usage=Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0),- )- with pytest.raises(ValueError, match="LiteLLM returned no text content"):- _response(response)diff --git a/examples/experimental/litellm/tests/test_e2e.py b/examples/experimental/litellm/tests/test_e2e.pydeleted file mode 100644index 9586eea9b..000000000--- a/examples/experimental/litellm/tests/test_e2e.py+++ /dev/null@@ -1,191 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--import os-import shutil-import socket-import subprocess-from collections.abc import Iterator-from pathlib import Path--import pytest-from switchyard_litellm import LiteLLMSyClient--from switchyard.libsy import Algorithm, LlmResponse, Step, algorithms--PACKAGE_ROOT = Path(__file__).resolve().parents[1]---def _free_port() -> int:- with socket.socket() as sock:- sock.bind(("127.0.0.1", 0))- return int(sock.getsockname()[1])--[email protected](scope="session")-def litellm_base_url() -> Iterator[str]:- if os.environ.get("SWITCHYARD_LITELLM_E2E") != "1":- pytest.skip("SWITCHYARD_LITELLM_E2E=1 is required for paid E2E tests")- if not os.environ.get("OPENROUTER_API_KEY"):- pytest.skip("OPENROUTER_API_KEY is required for paid E2E tests")- if shutil.which("docker") is None:- pytest.skip("Docker is required for paid E2E tests")- subprocess.run(- ["docker", "compose", "version"],- check=True,- stdout=subprocess.DEVNULL,- )-- port = _free_port()- project = f"switchyard-litellm-e2e-{os.getpid()}"- network = f"{project}-network"- env = {- **os.environ,- "LITELLM_PORT": str(port),- "LITELLM_NETWORK": network,- }- compose = ["docker", "compose", "--project-name", project]- try:- subprocess.run(- [*compose, "up", "-d", "--wait"],- cwd=PACKAGE_ROOT,- env=env,- check=True,- )- yield f"http://127.0.0.1:{port}/v1"- finally:- subprocess.run(- [*compose, "down", "--volumes", "--remove-orphans"],- cwd=PACKAGE_ROOT,- env=env,- check=True,- )---def test_paid_e2e_requires_explicit_spend_opt_in(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("OPENROUTER_API_KEY", "test-key")- monkeypatch.delenv("SWITCHYARD_LITELLM_E2E", raising=False)-- def fail_if_docker_is_checked(_: str) -> str | None:- pytest.fail("Docker must not be checked without the paid E2E opt-in")-- monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)- fixture = litellm_base_url.__wrapped__()- with pytest.raises(pytest.skip.Exception, match="SWITCHYARD_LITELLM_E2E"):- next(fixture)---def test_paid_e2e_requires_openrouter_key(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("SWITCHYARD_LITELLM_E2E", "1")- monkeypatch.setenv("OPENAI_API_KEY", "legacy-key")- monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)-- def fail_if_docker_is_checked(_: str) -> str | None:- pytest.fail("Docker must not be checked without OPENROUTER_API_KEY")-- monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)- fixture = litellm_base_url.__wrapped__()- with pytest.raises(pytest.skip.Exception, match="OPENROUTER_API_KEY"):- next(fixture)---def _request(*, critical_error: bool = False) -> dict[str, object]:- messages: list[dict[str, object]] = [- {- "role": "user",- "content": [- {- "type": "text",- "text": "Reply with a short greeting for a Switchyard E2E test.",- }- ],- }- ]- if critical_error:- messages.extend([- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_1",- "name": "Bash",- "arguments": {"command": "pytest"},- }- ],- },- {- "role": "tool",- "content": [- {- "type": "tool_result",- "tool_call_id": "call_1",- "content": [- {- "type": "text",- "text": "fatal runtime error: out of memory",- }- ],- "is_error": True,- }- ],- },- ])- return {- "model": "auto",- "messages": messages,- "reasoning": {"effort": "low"},- "output": {"max_output_tokens": 128},- }---async def _run_router(- router: Algorithm,- request: dict[str, object],- client: LiteLLMSyClient,-) -> tuple[str, dict[str, object]]:- async for step in router.run_stream(request):- match step:- case Step.CallModel(call):- call.respond(LlmResponse.Agg(await client.call(call.request)))- case Step.Done(outcome):- match outcome.response:- case LlmResponse.Agg(response):- return outcome.selected_model_id, response- case LlmResponse.Stream(_):- raise AssertionError("LiteLLM example expects buffered responses")- case None:- return outcome.selected_model_id, await client.call(outcome.request)- raise AssertionError("algorithm stream ended without an outcome")---async def test_stage_router_calls_both_real_openrouter_models(- litellm_base_url: str,-) -> None:- client = LiteLLMSyClient(base_url=litellm_base_url)- router = algorithms.stage_router(- "strong",- "fast",- picker="efficient_first",- confidence_threshold=0.5,- recent_window=3,- )- try:- fast_trace, fast_response = await _run_router(router, _request(), client)- strong_trace, strong_response = await _run_router(- router, _request(critical_error=True), client- )- finally:- await client.aclose()-- assert strong_trace == "strong"- assert fast_trace == "fast"- for response in (strong_response, fast_response):- text = response["outputs"][0]["content"][0]["text"]- assert isinstance(text, str)- assert text.strip()diff --git a/examples/experimental/litellm/tests/test_gateway_config.py b/examples/experimental/litellm/tests/test_gateway_config.pydeleted file mode 100644index 8042cdfc1..000000000--- a/examples/experimental/litellm/tests/test_gateway_config.py+++ /dev/null@@ -1,51 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--import tomllib-from pathlib import Path--import yaml--PACKAGE_ROOT = Path(__file__).resolve().parents[1]-REPOSITORY_ROOT = PACKAGE_ROOT.parents[2]---def test_example_targets_python_312() -> None:- project = tomllib.loads((PACKAGE_ROOT / "pyproject.toml").read_text())-- assert project["project"]["requires-python"] == ">=3.12,<3.13"---def test_lockfile_matches_root_package_version() -> None:- root_project = tomllib.loads((REPOSITORY_ROOT / "pyproject.toml").read_text())- lock = tomllib.loads((PACKAGE_ROOT / "uv.lock").read_text())- locked_root = next(- package for package in lock["package"] if package["name"] == "nemo-switchyard"- )-- assert locked_root["version"] == root_project["project"]["version"]---def test_gateway_uses_openrouter_models_and_credential() -> None:- config = yaml.safe_load((PACKAGE_ROOT / "litellm-config.yaml").read_text())- models = {- item["model_name"]: item["litellm_params"] for item in config["model_list"]- }-- assert models == {- "strong": {- "model": "openrouter/openai/gpt-5.6-sol",- "api_key": "os.environ/OPENROUTER_API_KEY",- },- "fast": {- "model": "openrouter/moonshotai/kimi-k3",- "api_key": "os.environ/OPENROUTER_API_KEY",- },- }- assert config["litellm_settings"] == {"drop_params": True}-- compose = yaml.safe_load((PACKAGE_ROOT / "compose.yaml").read_text())- assert compose["services"]["litellm"]["environment"] == {- "OPENROUTER_API_KEY": "${OPENROUTER_API_KEY:?set OPENROUTER_API_KEY}"- }- assert (PACKAGE_ROOT / ".env.example").read_text() == "OPENROUTER_API_KEY=\n"diff --git a/examples/experimental/litellm/tests/test_stage_routing.py b/examples/experimental/litellm/tests/test_stage_routing.pydeleted file mode 100644index e6fb84877..000000000--- a/examples/experimental/litellm/tests/test_stage_routing.py+++ /dev/null@@ -1,128 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--import json--import httpx-import respx-from switchyard_litellm import LiteLLMSyClient--from switchyard.libsy import Algorithm, LlmResponse, Step, algorithms--BASE_URL = "http://gateway.test/v1"---def request_body(*, critical_error: bool = False) -> dict[str, object]:- messages: list[dict[str, object]] = [- {- "role": "user",- "content": [{"type": "text", "text": "Fix the failing tests."}],- }- ]- if critical_error:- messages.extend([- {- "role": "assistant",- "content": [- {- "type": "tool_call",- "id": "call_1",- "name": "Bash",- "arguments": {"command": "pytest"},- }- ],- },- {- "role": "tool",- "content": [- {- "type": "tool_result",- "tool_call_id": "call_1",- "content": [- {- "type": "text",- "text": "fatal runtime error: out of memory",- }- ],- "is_error": True,- }- ],- },- ])- return {"model": "auto", "messages": messages}---def gateway_response(model: str) -> dict[str, object]:- return {- "id": f"chatcmpl-{model}",- "object": "chat.completion",- "created": 1,- "model": model,- "choices": [- {- "index": 0,- "message": {"role": "assistant", "content": model},- "finish_reason": "stop",- }- ],- "usage": {- "prompt_tokens": 4,- "completion_tokens": 1,- "total_tokens": 5,- },- }---async def _run_router(- router: Algorithm,- request: dict[str, object],- client: LiteLLMSyClient,-) -> tuple[str, dict[str, object]]:- async for step in router.run_stream(request):- match step:- case Step.CallModel(call):- call.respond(LlmResponse.Agg(await client.call(call.request)))- case Step.Done(outcome):- match outcome.response:- case LlmResponse.Agg(response):- return outcome.selected_model_id, response- case LlmResponse.Stream(_):- raise AssertionError("LiteLLM example expects buffered responses")- case None:- return outcome.selected_model_id, await client.call(outcome.request)- raise AssertionError("algorithm stream ended without an outcome")---async def test_stage_router_drives_both_litellm_models() -> None:- seen: list[str] = []-- def respond(request: httpx.Request) -> httpx.Response:- model = json.loads(request.content)["model"]- seen.append(model)- return httpx.Response(200, json=gateway_response(model))-- respx.post(f"{BASE_URL}/chat/completions").mock(side_effect=respond)- client = LiteLLMSyClient(base_url=BASE_URL)- router = algorithms.stage_router(- "strong",- "fast",- picker="efficient_first",- confidence_threshold=0.5,- recent_window=3,- )- try:- fast_target, fast_response = await _run_router(- router, request_body(), client- )- strong_target, strong_response = await _run_router(- router, request_body(critical_error=True), client- )- finally:- await client.aclose()-- assert fast_target == "fast"- assert strong_target == "strong"- assert fast_response["outputs"][0]["content"][0]["text"] == "fast"- assert strong_response["outputs"][0]["content"][0]["text"] == "strong"- assert seen == ["fast", "strong"]diff --git a/examples/litellm/README.md b/examples/litellm/README.mdnew file mode 100644index 000000000..20ee65315--- /dev/null+++ b/examples/litellm/README.md@@ -0,0 +1,364 @@+# Switchyard routing plugin for LiteLLM++> **Experimental integration:** This example and its Python API may change without notice.++> **Checkout-only dependency:** The current published `nemo-switchyard` release does not contain+> the decision-only Python API used here. Run this example from this repository checkout, where+> `[tool.uv.sources]` binds the example to the adjacent Switchyard source. Do not install or publish+> `switchyard-litellm` as a standalone package until a compatible Switchyard release is available+> and its dependency floor can be updated.++This package integrates Switchyard at LiteLLM's native routing-plugin boundary. Applications keep+using LiteLLM's `Router` or OpenAI-compatible proxy. The same configured object has two LiteLLM+roles: before deployment selection it narrows the candidate set to the model selected by+Switchyard; after selection it applies any supported Switchyard request rewrite before LiteLLM+translates and sends the provider request.++The integration pins LiteLLM 1.97.0. Model inventory and routing policy are both owned by the+deployer:++- LiteLLM YAML defines the public model group, candidate models, credentials, and provider options.+- Switchyard TOML selects the algorithm and all of its supported routing parameters.++The checked-in profiles use two OpenRouter models as demonstration values. Those model IDs are not+built into `switchyard_litellm` and can be replaced without changing package source.++## Layout++```text+litellm/+├── deployment/+│ ├── .env.example+│ ├── Dockerfile+│ ├── compose.yaml+│ └── profiles/+│ ├── stage/{litellm.yaml,switchyard.toml}+│ └── random/{litellm.yaml,switchyard.toml}+├── examples/python_router.py+├── src/switchyard_litellm/+│ ├── configuration/+│ └── plugins/+└── tests/{unit,integration}/+```++## Request flow++```text+application → LiteLLM Router → Switchyard routing plugin+ → Algorithm.run_stream()+ → one selected candidate + optional request delta+ → LiteLLM deployment selection+ → same object as deployment callback → provider inference+```++The candidate-bound Stage and Random plugins construct Switchyard algorithms from the live+candidate list supplied by LiteLLM. The low-level `SwitchyardRoutingPlugin` then:++1. converts LiteLLM's `structured_messages` to a normalized Switchyard request;+2. runs the supplied algorithm until `Step.Done`;+3. validates and converts any supported request delta;+4. replaces `context.candidate_models` with the selected exact deployment; and+5. records the decision and private delta under `context.signals["switchyard"]`.++LiteLLM copies those signals into request metadata. After LiteLLM selects a deployment, the same+object's `async_pre_call_deployment_hook` applies the delta and removes it from the metadata passed+downstream. The proxy profile therefore registers the dotted object as both a routing plugin and a+callback.++LiteLLM remains responsible for credentials, retries, provider translation, inference, and the+OpenAI-compatible API.++## Supported algorithm semantics++Compatibility is determined by behavior rather than an algorithm-name allowlist.+`StageRoutingPlugin` and `RandomRoutingPlugin` provide the deployer-oriented bindings supported by+the checked-in configuration loader.++| Behavior | Result |+|---|---|+| Random selects a current candidate | Supported |+| Stage uses only request-history signals | Supported |+| Algorithm requests `Step.CallModel` | Rejected |+| Algorithm returns an existing response | Rejected |+| Algorithm rewrites text instructions/messages | Applied after deployment selection |+| Algorithm rewrites tools/tool choice | Applied after deployment selection |+| Algorithm rewrites sampling, output, reasoning effort, or stream | Applied after deployment selection |+| Algorithm rewrites supported provider-extension fields | Applied after deployment selection |+| Algorithm rewrites raw reasoning, preservation, or unsupported content/extensions | Rejected |+| Algorithm selects outside the current candidate pool | Rejected |+| Algorithm stream ends without `Step.Done` | Rejected |++Escalation and classifier-backed routers are not compatible because a LiteLLM routing plugin cannot+service intermediate model calls. Stage's system prompts and handoff notes do not require an+intermediate call, so the dual-role Stage plugin supports them. Unsupported behavior fails closed+before LiteLLM sends inference.++LiteLLM 1.97.0's routing context exposes structured messages but does not expose the caller's tools,+sampling controls, output controls, or provider-specific arguments. The adapter therefore uses+delta semantics: a field explicitly changed by Switchyard overrides the corresponding LiteLLM+argument, while a field Switchyard leaves at the adapter default does not clear or replace the+caller's original value. A Switchyard algorithm can produce an outbound override for tools,+`tool_choice`, `temperature`, `top_p`, `top_k`, `max_completion_tokens`, `response_format`,+`reasoning_effort`, `stream`, and the safe OpenAI Chat extension fields supported by the adapter.+It cannot inspect the original value of those fields through the current routing-plugin API, or+intentionally clear a caller value back to an adapter default such as `None` or an empty tool list.++The message adapter supports `system`, `developer`, and `user` text; assistant text and OpenAI+function tool calls; and text tool results. It rejects malformed function arguments, media or audio+content, legacy `function` messages, and unknown roles or content blocks.++## Quick start with the local proxy++Prerequisites are Docker Compose and an OpenRouter key with access to the example models. From this+directory:++```bash+cp deployment/.env.example deployment/.env+# Set OPENROUTER_API_KEY in deployment/.env.+docker compose -f deployment/compose.yaml up -d --build --wait+curl -fsS http://127.0.0.1:4000/health/liveliness+```++The default profile is `stage`. Send a request to its public `switchyard` model group:++```bash+curl -i http://127.0.0.1:4000/v1/chat/completions \+ -H 'Content-Type: application/json' \+ -d '{+ "model": "switchyard",+ "messages": [{"role": "user", "content": "Reply with the word hello."}],+ "max_tokens": 64+ }'+```++LiteLLM keeps the response body's `model` field equal to the public group. The+`x-litellm-model-name` response header identifies the concrete model selected by Switchyard.++Start the Random profile instead with:++```bash+SWITCHYARD_LITELLM_PROFILE=random \+ docker compose -f deployment/compose.yaml up -d --build --wait+```++Stop the proxy without deleting repository data:++```bash+docker compose -f deployment/compose.yaml down+```++## Configure a deployment profile++Each directory under `deployment/profiles/` is a complete selectable profile:++```text+profiles/my-profile/+├── litellm.yaml # model inventory and LiteLLM settings+└── switchyard.toml # Switchyard algorithm and parameters+```++The LiteLLM YAML registers the environment-configured plugin object:++```yaml+model_list:+ - model_name: switchyard+ litellm_params:+ model: provider/capable-model+ api_key: os.environ/PROVIDER_API_KEY+ - model_name: switchyard+ litellm_params:+ model: provider/efficient-model+ api_key: os.environ/PROVIDER_API_KEY++router_settings:+ plugins:+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN++litellm_settings:+ callbacks:+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN+```++Compose mounts the selected profile at `/app/deployment` and sets+`SWITCHYARD_LITELLM_CONFIG=/app/deployment/switchyard.toml`. LiteLLM imports the pre-created object+from the dotted path above when the proxy starts.++### Stage fields++```toml+algorithm = "stage"+picker = "efficient_first"+confidence_threshold = 0.5+recent_window = 3+only_on_wrong_signal_escalation = true+escalation_note = "The efficient tier failed; continue from its work."+deescalation_note = "The capable tier completed the recovery."+capable_system_prompt = "Handle this request as the capable tier."+efficient_system_prompt = "Handle this request as the efficient tier."+```++| Field | Required | Accepted value |+|---|---:|---|+| `algorithm` | yes | `"stage"` |+| `picker` | yes | `"capable_first"` or `"efficient_first"` |+| `confidence_threshold` | yes | finite number from `0` through `1` |+| `recent_window` | no | nonnegative integer; omitted means no fixed recent window |+| `only_on_wrong_signal_escalation` | no | Boolean; defaults to `true` |+| `escalation_note` | no | nonempty text appended when Stage hands work to the capable tier |+| `deescalation_note` | no | nonempty text appended when Stage hands work back; requires `escalation_note` |+| `capable_system_prompt` | no | nonempty system instruction used for the capable tier |+| `efficient_system_prompt` | no | nonempty system instruction used for the efficient tier |++Stage requires exactly two unique candidate model IDs. In pinned LiteLLM 1.97.0, the routing context+preserves the matching `model_list` declaration order. Declare the capable model first and the+efficient model second, as in the checked-in profile; an integration test locks this ordering+contract. `picker` controls which tier is chosen without an escalation signal; it does not change+those roles.++### Random fields++```toml+algorithm = "random"+seed = 6+weights = [0.25, 0.75]+```++| Field | Required | Accepted value |+|---|---:|---|+| `algorithm` | yes | `"random"` |+| `seed` | no | integer from `0` through `2^64 - 1` |+| `weights` | no | nonempty array of finite, nonnegative numbers with at least one positive value |++Random selects across every unique candidate ID in first-seen LiteLLM order. `weights` maps to that+same order, so its length must equal the number of unique candidates in the request. Omitting+`weights` gives each candidate equal weight; omitting `seed` uses Switchyard's unseeded behavior.++Static errors—missing files, malformed TOML, unknown keys, wrong types, invalid ranges, and invalid+weight values—fail while LiteLLM imports the plugin at proxy startup. Constraints that depend on+LiteLLM's live candidates fail at request time: Stage's two-candidate requirement and Random's+candidate-to-weight count.++To add a profile, copy an existing directory, edit both files, and select its directory name:++```bash+SWITCHYARD_LITELLM_PROFILE=my-profile \+ docker compose -f deployment/compose.yaml up -d --build --wait+```++On a successful decision without a rewrite, the plugin preserves signals from earlier plugins and+adds:++```python+context.signals["switchyard"] = {+ "selected_model_id": "provider/efficient-model",+ "fallback_models": ["provider/capable-model"],+}+```++`fallback_models` is diagnostic metadata only. The plugin narrows the candidate set; it does not add+a LiteLLM fallback policy. When Switchyard rewrites a supported request field, the signal also+temporarily contains a private `request_patch`. The callback consumes that patch after deployment+selection and excludes it from the metadata it returns downstream.++## Use the plugin with LiteLLM's Python Router++For programmatic use, construct the plugin directly; TOML and the environment-backed import module+are proxy deployment conveniences.++```python+import litellm+from litellm import Router+from switchyard_litellm import StageRoutingPlugin++model_list = [+ {+ "model_name": "switchyard",+ "litellm_params": {"model": "provider/capable-model"},+ },+ {+ "model_name": "switchyard",+ "litellm_params": {"model": "provider/efficient-model"},+ },+]+plugin = StageRoutingPlugin(+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+)+router = Router(model_list=model_list, plugins=[plugin])+litellm.callbacks.append(plugin)+```++The explicit callback registration is required for programmatic `Router` use because LiteLLM's+constructor accepts routing plugins but not deployment callbacks. In a long-running application,+register the object once during startup. The proxy YAML shown above performs both registrations+declaratively.++Install the locked example environment and run the complete example from this directory:++```bash+uv sync --locked --python 3.12+uv run --locked --env-file deployment/.env python examples/python_router.py+```++## Develop against a LiteLLM source checkout++The pinned container is the reproducible default. To work on LiteLLM itself, follow LiteLLM's+[local development setup](https://docs.litellm.ai/docs/extras/contributing_code#1-setting-up-your-local-dev-environment),+including its proxy dependencies, then install this Switchyard checkout into that environment.+Point the plugin loader at a routing TOML and LiteLLM at the matching model YAML:++```bash+export SWITCHYARD_LITELLM_CONFIG=/absolute/path/to/switchyard-new/examples/litellm/deployment/profiles/stage/switchyard.toml+PYTHONPATH=/absolute/path/to/switchyard-new/examples/litellm/src \+ uv run litellm --config \+ /absolute/path/to/switchyard-new/examples/litellm/deployment/profiles/stage/litellm.yaml+```++Keep LiteLLM at v1.97.0 when reproducing this example's verified behavior.++## Tests++From the repository root, run the offline suite without an API key or provider calls:++```bash+PYTHONPATH=examples/litellm/src \+ uv run --project examples/litellm --locked \+ pytest examples/litellm/tests -m "not e2e" -v+```++The paid suite has an explicit opt-in. It builds and starts the local proxy once for each profile+and verifies the concrete target through LiteLLM's `x-litellm-model-name` response header:++```bash+SWITCHYARD_LITELLM_E2E=1 \+ uv run --env-file examples/litellm/deployment/.env \+ --project examples/litellm --locked \+ pytest examples/litellm/tests/integration/test_e2e.py -m e2e -v+```++## Troubleshooting++- If Compose says the key is missing, place `OPENROUTER_API_KEY` in `deployment/.env`, or pass+ another file with `docker compose --env-file /path/to/.env -f deployment/compose.yaml ...`.+- If startup reports `SWITCHYARD_LITELLM_CONFIG`, verify that the selected profile contains a+ readable `switchyard.toml` and that its `algorithm` and fields match the tables above.+- If port 4000 is occupied, set `LITELLM_PORT` before starting Compose and use that port in requests.+- If the service is unhealthy, run `docker compose -f deployment/compose.yaml logs litellm`.+- If routing fails only when a request arrives, verify candidate ordering/count and Random weight+ count before checking the request's supported message shapes.++## Security++The bundled proxy is unauthenticated and binds only to loopback. It is intended for local+development. Local `.env` files are excluded from the Docker build context; keep credentials out of+YAML, TOML, source control, and command output. Configure LiteLLM authentication and follow its+production deployment guidance before exposing the service.++## References++- [LiteLLM routing plugins](https://docs.litellm.ai/docs/routing_plugins)+- [Switchyard decision-only API PR #459](https://github.com/NVIDIA-NeMo/Switchyard/pull/459)+- [Switchyard Python binding updates PR #479](https://github.com/NVIDIA-NeMo/Switchyard/pull/479)+- [Switchyard Stage routing](../../../docs/routing_algorithms/stage_router_routing.md)+- [Switchyard Random routing](../../../docs/routing_algorithms/random_routing.md)diff --git a/examples/litellm/deployment/.env.example b/examples/litellm/deployment/.env.examplenew file mode 100644index 000000000..fbe1e2245--- /dev/null+++ b/examples/litellm/deployment/.env.example@@ -0,0 +1,2 @@+OPENROUTER_API_KEY=+SWITCHYARD_LITELLM_PROFILE=stagediff --git a/examples/litellm/deployment/Dockerfile b/examples/litellm/deployment/Dockerfilenew file mode 100644index 000000000..482a77b17--- /dev/null+++ b/examples/litellm/deployment/Dockerfile@@ -0,0 +1,41 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++FROM rust:1.96.1-slim-bookworm AS switchyard-builder++RUN apt-get update \+ && apt-get install --yes --no-install-recommends patchelf python3 python3-pip python3-venv \+ && rm -rf /var/lib/apt/lists/*+RUN python3 -m venv /opt/maturin \+ && /opt/maturin/bin/pip install --no-cache-dir "maturin==1.14.1"++WORKDIR /src+COPY . .+RUN /opt/maturin/bin/maturin build \+ --release \+ --locked \+ --compatibility manylinux_2_34 \+ --out /wheels \+ --interpreter /usr/bin/python3++FROM ghcr.io/astral-sh/uv:0.8.15 AS uv++FROM ghcr.io/berriai/litellm:v1.97.0++COPY --from=uv /uv /usr/local/bin/uv+COPY --from=switchyard-builder /wheels /wheels+RUN uv pip install --python /app/.venv/bin/python --no-deps /wheels/*.whl \+ && rm -rf /wheels++COPY examples/litellm/src /app/switchyard-plugin/src+ENV PYTHONPATH=/app/switchyard-plugin/src++RUN python -c "from importlib.metadata import version; \+from switchyard.libsy import algorithms; \+from switchyard_litellm import SwitchyardRoutingPlugin; \+assert version('litellm') == '1.97.0'; \+SwitchyardRoutingPlugin(algorithms.random(['smoke']))"++RUN addgroup -S -g 10001 litellm \+ && adduser -S -D -u 10001 -G litellm -h /home/litellm litellm+USER litellmdiff --git a/examples/experimental/litellm/compose.yaml b/examples/litellm/deployment/compose.yamlsimilarity index 64%rename from examples/experimental/litellm/compose.yamlrename to examples/litellm/deployment/compose.yamlindex 54ef48e5b..0ac934644 100644--- a/examples/experimental/litellm/compose.yaml+++ b/examples/litellm/deployment/compose.yaml@@ -1,15 +1,19 @@services:litellm:- image: ghcr.io/berriai/litellm:v1.92.0+ build:+ context: ../../..+ dockerfile: examples/litellm/deployment/Dockerfile+ image: switchyard-litellm:1.97.0command:- --config- - /app/config.yaml+ - /app/deployment/litellm.yamlenvironment:OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:?set OPENROUTER_API_KEY}+ SWITCHYARD_LITELLM_CONFIG: /app/deployment/switchyard.tomlports:- "127.0.0.1:${LITELLM_PORT:-4000}:4000"volumes:- - ./litellm-config.yaml:/app/config.yaml:ro+ - ./profiles/${SWITCHYARD_LITELLM_PROFILE:-stage}:/app/deployment:rohealthcheck:test:- CMD-SHELLdiff --git a/examples/litellm/deployment/profiles/random/litellm.yaml b/examples/litellm/deployment/profiles/random/litellm.yamlnew file mode 100644index 000000000..890683c6c--- /dev/null+++ b/examples/litellm/deployment/profiles/random/litellm.yaml@@ -0,0 +1,17 @@+model_list:+ - model_name: switchyard+ litellm_params:+ model: openrouter/openai/gpt-5.6-sol+ api_key: os.environ/OPENROUTER_API_KEY+ - model_name: switchyard+ litellm_params:+ model: openrouter/openai/gpt-5.6-terra+ api_key: os.environ/OPENROUTER_API_KEY++router_settings:+ plugins:+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN++litellm_settings:+ callbacks:+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGINdiff --git a/examples/litellm/deployment/profiles/random/switchyard.toml b/examples/litellm/deployment/profiles/random/switchyard.tomlnew file mode 100644index 000000000..a28c8724b--- /dev/null+++ b/examples/litellm/deployment/profiles/random/switchyard.toml@@ -0,0 +1,2 @@+algorithm = "random"+seed = 6diff --git a/examples/litellm/deployment/profiles/stage/litellm.yaml b/examples/litellm/deployment/profiles/stage/litellm.yamlnew file mode 100644index 000000000..6748d0599--- /dev/null+++ b/examples/litellm/deployment/profiles/stage/litellm.yaml@@ -0,0 +1,18 @@+model_list:+ # Stage roles follow declaration order: capable first, efficient second.+ - model_name: switchyard+ litellm_params:+ model: openrouter/openai/gpt-5.6-sol+ api_key: os.environ/OPENROUTER_API_KEY+ - model_name: switchyard+ litellm_params:+ model: openrouter/openai/gpt-5.6-terra+ api_key: os.environ/OPENROUTER_API_KEY++router_settings:+ plugins:+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN++litellm_settings:+ callbacks:+ - switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGINdiff --git a/examples/litellm/deployment/profiles/stage/switchyard.toml b/examples/litellm/deployment/profiles/stage/switchyard.tomlnew file mode 100644index 000000000..d51af3754--- /dev/null+++ b/examples/litellm/deployment/profiles/stage/switchyard.toml@@ -0,0 +1,9 @@+algorithm = "stage"+picker = "efficient_first"+confidence_threshold = 0.5+recent_window = 3+only_on_wrong_signal_escalation = true+escalation_note = "The efficient tier failed; continue from its work."+deescalation_note = "The capable tier completed the recovery."+capable_system_prompt = "Handle this request as the capable tier."+efficient_system_prompt = "Handle this request as the efficient tier."diff --git a/examples/litellm/examples/python_router.py b/examples/litellm/examples/python_router.pynew file mode 100644index 000000000..8003e03cf--- /dev/null+++ b/examples/litellm/examples/python_router.py@@ -0,0 +1,81 @@+#!/usr/bin/env python3+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Send one request through LiteLLM's Router and the Switchyard Stage plugin."""++import asyncio++import litellm+from litellm.router import Router+from litellm.types.llms.openai import AllMessageValues+from switchyard_litellm import StageRoutingPlugin++MODEL_GROUP = "switchyard"+MODEL_LIST: list[dict[str, object]] = [+ {+ "model_name": MODEL_GROUP,+ "litellm_params": {+ "model": "openrouter/openai/gpt-5.6-sol",+ "api_key": "os.environ/OPENROUTER_API_KEY",+ },+ },+ {+ "model_name": MODEL_GROUP,+ "litellm_params": {+ "model": "openrouter/openai/gpt-5.6-terra",+ "api_key": "os.environ/OPENROUTER_API_KEY",+ },+ },+]+STAGE_ROUTING_PLUGIN = StageRoutingPlugin(+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+)+++def critical_tool_history() -> list[AllMessageValues]:+ """Build a turn whose critical tool result should select GPT-5.6 Sol."""+ return [+ {"role": "user", "content": "Fix the failing tests."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {+ "role": "tool",+ "tool_call_id": "call_1",+ "content": "fatal runtime error: out of memory",+ },+ ]+++async def main() -> None:+ """Route the example through the plugin-controlled LiteLLM pipeline."""+ router = Router(model_list=MODEL_LIST, plugins=[STAGE_ROUTING_PLUGIN])+ litellm.callbacks.append(STAGE_ROUTING_PLUGIN)+ try:+ response = await router.acompletion(+ model=MODEL_GROUP,+ messages=critical_tool_history(),+ max_tokens=64,+ )+ print("Selected model:", response.model)+ print("Response:", response.choices[0].message.content)+ finally:+ litellm.callbacks.remove(STAGE_ROUTING_PLUGIN)+++if __name__ == "__main__":+ asyncio.run(main())diff --git a/examples/experimental/litellm/pyproject.toml b/examples/litellm/pyproject.tomlsimilarity index 76%rename from examples/experimental/litellm/pyproject.tomlrename to examples/litellm/pyproject.tomlindex 6dfbfe5de..b7368fb60 100644--- a/examples/experimental/litellm/pyproject.toml+++ b/examples/litellm/pyproject.toml@@ -5,24 +5,23 @@ build-backend = "setuptools.build_meta"[project]name = "switchyard-litellm"version = "0.1.0"-description = "Experimental LiteLLM gateway client for Switchyard libsy"+description = "Experimental Switchyard routing plugin for LiteLLM"readme = "README.md"requires-python = ">=3.12,<3.13"license = "Apache-2.0"dependencies = ["nemo-switchyard>=0.1.0",- "litellm==1.92.0",+ "litellm==1.97.0",][dependency-groups]dev = ["pytest>=9,<10","pytest-asyncio>=1.3,<2",- "respx>=0.23,<1",][tool.uv.sources]-nemo-switchyard = { path = "../../..", editable = true }+nemo-switchyard = { path = "../..", editable = true }[tool.setuptools.packages.find]where = ["src"]diff --git a/examples/litellm/src/switchyard_litellm/__init__.py b/examples/litellm/src/switchyard_litellm/__init__.pynew file mode 100644index 000000000..f3ce414d5--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/__init__.py@@ -0,0 +1,8 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Experimental Switchyard routing plugin for LiteLLM."""++from .plugins import RandomRoutingPlugin, StageRoutingPlugin, SwitchyardRoutingPlugin++__all__ = ["RandomRoutingPlugin", "StageRoutingPlugin", "SwitchyardRoutingPlugin"]diff --git a/examples/litellm/src/switchyard_litellm/configuration/__init__.py b/examples/litellm/src/switchyard_litellm/configuration/__init__.pynew file mode 100644index 000000000..03a6102b8--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/configuration/__init__.py@@ -0,0 +1,18 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Load deployer-owned Switchyard routing plugin configuration."""++from .loader import (+ CONFIG_ENV,+ RoutingPlugin,+ load_routing_plugin,+ load_routing_plugin_from_environment,+)++__all__ = [+ "CONFIG_ENV",+ "RoutingPlugin",+ "load_routing_plugin",+ "load_routing_plugin_from_environment",+]diff --git a/examples/litellm/src/switchyard_litellm/configuration/configured_plugin.py b/examples/litellm/src/switchyard_litellm/configuration/configured_plugin.pynew file mode 100644index 000000000..afa5a7f94--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/configuration/configured_plugin.py@@ -0,0 +1,10 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Expose the environment-configured plugin object imported by LiteLLM."""++from .loader import load_routing_plugin_from_environment++ROUTING_PLUGIN = load_routing_plugin_from_environment()++__all__ = ["ROUTING_PLUGIN"]diff --git a/examples/litellm/src/switchyard_litellm/configuration/loader.py b/examples/litellm/src/switchyard_litellm/configuration/loader.pynew file mode 100644index 000000000..e6bac488d--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/configuration/loader.py@@ -0,0 +1,271 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Create candidate-bound LiteLLM plugins from Switchyard TOML configuration.++``tomllib`` decodes TOML values into Python types. This module's helpers enforce+the plugin schema and value constraints after that decoding step.+"""++from __future__ import annotations++import math+import os+import tomllib+from collections.abc import Collection, Mapping+from pathlib import Path++from switchyard_litellm.plugins import RandomRoutingPlugin, StageRoutingPlugin++RoutingPlugin = RandomRoutingPlugin | StageRoutingPlugin++CONFIG_ENV = "SWITCHYARD_LITELLM_CONFIG"+_STAGE_KEYS = frozenset(+ {+ "algorithm",+ "picker",+ "confidence_threshold",+ "recent_window",+ "escalation_note",+ "deescalation_note",+ "only_on_wrong_signal_escalation",+ "capable_system_prompt",+ "efficient_system_prompt",+ }+)+_RANDOM_KEYS = frozenset({"algorithm", "seed", "weights"})+_MAX_RANDOM_SEED = (1 << 64) - 1+++def load_routing_plugin(path: str | Path) -> RoutingPlugin:+ """Load a Stage or Random routing plugin from TOML."""+ config_path = Path(path)+ try:+ with config_path.open("rb") as config_file:+ values: Mapping[str, object] = tomllib.load(config_file)+ except (OSError, tomllib.TOMLDecodeError) as error:+ raise ValueError(+ f"Could not load Switchyard routing config {config_path}: {error}"+ ) from error++ algorithm = _required_string(values, "algorithm", config_path)+ if algorithm == "stage":+ return _stage_plugin(values, config_path)+ if algorithm == "random":+ return _random_plugin(values, config_path)+ raise ValueError(+ f"Switchyard routing config {config_path} has unsupported algorithm {algorithm!r}"+ )+++def load_routing_plugin_from_environment(+ environment: Mapping[str, str] = os.environ,+) -> RoutingPlugin:+ """Load the routing plugin named by ``SWITCHYARD_LITELLM_CONFIG``."""+ path = environment.get(CONFIG_ENV)+ if not path:+ raise ValueError(f"{CONFIG_ENV} must name a Switchyard routing TOML file")+ return load_routing_plugin(path)+++def _stage_plugin(values: Mapping[str, object], config_path: Path) -> StageRoutingPlugin:+ """Validate parsed Stage fields and construct its candidate-bound plugin."""+ _reject_unknown_keys(values, _STAGE_KEYS, config_path)+ picker = _required_string(values, "picker", config_path)+ if picker not in {"capable_first", "efficient_first"}:+ raise ValueError(+ f"Switchyard routing config {config_path} picker must be "+ "'capable_first' or 'efficient_first'"+ )+ escalation_note = _optional_string(values, "escalation_note", config_path)+ deescalation_note = _optional_string(values, "deescalation_note", config_path)+ if deescalation_note is not None and escalation_note is None:+ raise ValueError(+ f"Switchyard routing config {config_path} deescalation_note requires escalation_note"+ )+ return StageRoutingPlugin(+ picker=picker,+ confidence_threshold=_required_confidence_threshold(values, config_path),+ recent_window=_optional_integer(+ values,+ "recent_window",+ config_path,+ ),+ escalation_note=escalation_note,+ deescalation_note=deescalation_note,+ only_on_wrong_signal_escalation=_optional_boolean(+ values,+ "only_on_wrong_signal_escalation",+ config_path,+ default=True,+ ),+ capable_system_prompt=_optional_string(+ values,+ "capable_system_prompt",+ config_path,+ ),+ efficient_system_prompt=_optional_string(+ values,+ "efficient_system_prompt",+ config_path,+ ),+ )+++def _random_plugin(values: Mapping[str, object], config_path: Path) -> RandomRoutingPlugin:+ """Validate parsed Random fields and construct its candidate-bound plugin."""+ _reject_unknown_keys(values, _RANDOM_KEYS, config_path)+ return RandomRoutingPlugin(+ seed=_optional_integer(+ values,+ "seed",+ config_path,+ maximum=_MAX_RANDOM_SEED,+ ),+ weights=_optional_weights(values, config_path),+ )+++def _reject_unknown_keys(+ values: Mapping[str, object],+ allowed: Collection[str],+ config_path: Path,+) -> None:+ """Reject parsed TOML keys outside an algorithm's supported schema."""+ unknown = sorted(set(values) - set(allowed))+ if unknown:+ raise ValueError(+ f"Switchyard routing config {config_path} has unknown keys: {', '.join(unknown)}"+ )+++def _required_string(+ values: Mapping[str, object],+ key: str,+ config_path: Path,+) -> str:+ """Return a required nonempty string already decoded by ``tomllib``."""+ value = values.get(key)+ if not isinstance(value, str) or not value:+ raise ValueError(f"Switchyard routing config {config_path} {key} must be a nonempty string")+ return value+++def _optional_string(+ values: Mapping[str, object],+ key: str,+ config_path: Path,+) -> str | None:+ """Return an optional nonempty string already decoded by ``tomllib``."""+ if key not in values:+ return None+ value = values[key]+ if not isinstance(value, str) or not value:+ raise ValueError(f"Switchyard routing config {config_path} {key} must be a nonempty string")+ return value+++def _required_confidence_threshold(+ values: Mapping[str, object],+ config_path: Path,+) -> float:+ """Validate and return the parsed Stage confidence threshold."""+ value = values.get("confidence_threshold")+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ raise ValueError(+ f"Switchyard routing config {config_path} confidence_threshold must be a finite number"+ )+ threshold = float(value)+ if not math.isfinite(threshold):+ raise ValueError(+ f"Switchyard routing config {config_path} confidence_threshold must be a finite number"+ )+ if not 0.0 <= threshold <= 1.0:+ raise ValueError(+ f"Switchyard routing config {config_path} confidence_threshold "+ "must be between 0 and 1 inclusive"+ )+ return threshold+++def _optional_integer(+ values: Mapping[str, object],+ key: str,+ config_path: Path,+ *,+ maximum: int | None = None,+) -> int | None:+ """Validate and return an optional nonnegative TOML integer."""+ if key not in values:+ return None+ value = values[key]+ if isinstance(value, bool) or not isinstance(value, int):+ raise ValueError(+ f"Switchyard routing config {config_path} {key} must be a nonnegative integer"+ )+ if value < 0 or maximum is not None and value > maximum:+ upper_bound = f" no greater than {maximum}" if maximum is not None else ""+ raise ValueError(+ f"Switchyard routing config {config_path} {key} must be a nonnegative integer"+ f"{upper_bound}"+ )+ return value+++def _optional_boolean(+ values: Mapping[str, object],+ key: str,+ config_path: Path,+ *,+ default: bool,+) -> bool:+ """Validate and return an optional TOML Boolean with its default."""+ value = values.get(key, default)+ if not isinstance(value, bool):+ raise ValueError(f"Switchyard routing config {config_path} {key} must be a boolean")+ return value+++def _optional_weights(+ values: Mapping[str, object],+ config_path: Path,+) -> list[float] | None:+ """Validate and normalize optional Random weights decoded from TOML."""+ if "weights" not in values:+ return None+ value = values["weights"]+ if not isinstance(value, list):+ raise ValueError(+ f"Switchyard routing config {config_path} weights must be a nonempty array"+ )+ if not value:+ raise ValueError(+ f"Switchyard routing config {config_path} weights must be a nonempty array"+ )++ weights: list[float] = []+ for item in value:+ if isinstance(item, bool) or not isinstance(item, (int, float)):+ raise ValueError(+ f"Switchyard routing config {config_path} weights must contain only numbers"+ )+ weight = float(item)+ if not math.isfinite(weight):+ raise ValueError(f"Switchyard routing config {config_path} weights must be finite")+ if weight < 0.0:+ raise ValueError(f"Switchyard routing config {config_path} weights must be nonnegative")+ weights.append(weight)++ if not any(weight > 0.0 for weight in weights):+ raise ValueError(+ f"Switchyard routing config {config_path} weights must contain a positive value"+ )+ return weights+++__all__ = [+ "CONFIG_ENV",+ "RoutingPlugin",+ "load_routing_plugin",+ "load_routing_plugin_from_environment",+]diff --git a/examples/litellm/src/switchyard_litellm/plugins/__init__.py b/examples/litellm/src/switchyard_litellm/plugins/__init__.pynew file mode 100644index 000000000..17944e1b0--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/plugins/__init__.py@@ -0,0 +1,10 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Candidate-bound Switchyard routing plugins for LiteLLM."""++from .random_routing_plugin import RandomRoutingPlugin+from .stage_routing_plugin import StageRoutingPlugin+from .switchyard_routing_plugin import SwitchyardRoutingPlugin++__all__ = ["RandomRoutingPlugin", "StageRoutingPlugin", "SwitchyardRoutingPlugin"]diff --git a/examples/litellm/src/switchyard_litellm/plugins/lite_llm_request_rewriter.py b/examples/litellm/src/switchyard_litellm/plugins/lite_llm_request_rewriter.pynew file mode 100644index 000000000..e792ca4f7--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/plugins/lite_llm_request_rewriter.py@@ -0,0 +1,133 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Apply Switchyard request deltas at LiteLLM's selected-deployment boundary."""++from __future__ import annotations++from collections.abc import Mapping, Sequence+from typing import Any, cast++from litellm.integrations.custom_logger import CustomLogger+from litellm.types.utils import CallTypes++_PATCH_FIELDS = frozenset(+ {+ "messages",+ "tools",+ "tool_choice",+ "temperature",+ "top_p",+ "top_k",+ "max_completion_tokens",+ "response_format",+ "reasoning_effort",+ "stream",+ "parallel_tool_calls",+ "prompt_cache_key",+ "prompt_cache_retention",+ "safety_identifier",+ "service_tier",+ "store",+ "stream_options",+ "top_logprobs",+ "user",+ "stop",+ }+)+++class LiteLLMRequestRewriter(CustomLogger):+ """Act as the callback half of a dual-role LiteLLM routing plugin."""++ def __init__(self) -> None:+ super().__init__()++ async def async_pre_call_deployment_hook(+ self,+ kwargs: dict[str, Any],+ call_type: CallTypes | None,+ ) -> dict[str, Any] | None:+ """Apply this plugin's private request patch after deployment selection."""+ del call_type+ located = self._locate_patch(kwargs)+ if located is None:+ return None+ metadata_key, metadata, signals, switchyard_signal, patch = located++ raw_set_values = patch.get("set", {})+ if not isinstance(raw_set_values, Mapping):+ raise ValueError("Switchyard LiteLLM request patch set must be a mapping")+ set_values = cast(Mapping[str, object], raw_set_values)+ unknown_set_fields = sorted(set(set_values) - _PATCH_FIELDS)+ if unknown_set_fields:+ raise ValueError(+ "Switchyard LiteLLM request patch cannot set fields: "+ + ", ".join(unknown_set_fields)+ )+ raw_remove_values = patch.get("remove", [])+ if (+ isinstance(raw_remove_values, (str, bytes, bytearray))+ or not isinstance(raw_remove_values, Sequence)+ or not all(isinstance(item, str) for item in raw_remove_values)+ ):+ raise ValueError("Switchyard LiteLLM request patch remove must be a string sequence")+ unknown_remove_fields = sorted(set(raw_remove_values) - _PATCH_FIELDS)+ if unknown_remove_fields:+ raise ValueError(+ "Switchyard LiteLLM request patch cannot remove fields: "+ + ", ".join(unknown_remove_fields)+ )++ rewritten = dict(kwargs)+ for field in cast(Sequence[str], raw_remove_values):+ rewritten.pop(field, None)+ rewritten.update(set_values)++ sanitized_signal = dict(switchyard_signal)+ sanitized_signal.pop("request_patch", None)+ sanitized_signals = dict(signals)+ sanitized_signals["switchyard"] = sanitized_signal+ sanitized_metadata = dict(metadata)+ sanitized_metadata["routing_plugin_signals"] = sanitized_signals+ rewritten[metadata_key] = sanitized_metadata+ return rewritten++ @staticmethod+ def _locate_patch(+ kwargs: Mapping[str, object],+ ) -> tuple[+ str,+ Mapping[str, object],+ Mapping[str, object],+ Mapping[str, object],+ Mapping[str, object],+ ] | None:+ """Find a Switchyard request patch in either LiteLLM metadata field."""+ for metadata_key in ("litellm_metadata", "metadata"):+ raw_metadata = kwargs.get(metadata_key)+ if not isinstance(raw_metadata, Mapping):+ continue+ metadata = cast(Mapping[str, object], raw_metadata)+ raw_signals = metadata.get("routing_plugin_signals")+ if not isinstance(raw_signals, Mapping):+ continue+ signals = cast(Mapping[str, object], raw_signals)+ raw_switchyard_signal = signals.get("switchyard")+ if not isinstance(raw_switchyard_signal, Mapping):+ continue+ switchyard_signal = cast(Mapping[str, object], raw_switchyard_signal)+ raw_patch = switchyard_signal.get("request_patch")+ if not isinstance(raw_patch, Mapping):+ continue+ return (+ metadata_key,+ metadata,+ signals,+ switchyard_signal,+ cast(Mapping[str, object], raw_patch),+ )+ return None+++__all__ = ["LiteLLMRequestRewriter"]diff --git a/examples/litellm/src/switchyard_litellm/plugins/random_routing_plugin.py b/examples/litellm/src/switchyard_litellm/plugins/random_routing_plugin.pynew file mode 100644index 000000000..6e6435271--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/plugins/random_routing_plugin.py@@ -0,0 +1,46 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Bind Random routing to the candidates supplied by LiteLLM."""++from __future__ import annotations++from collections.abc import Sequence++from litellm.types.router import RoutingContext++from switchyard.libsy import algorithms++from .lite_llm_request_rewriter import LiteLLMRequestRewriter+from .switchyard_routing_plugin import SwitchyardRoutingPlugin+++class RandomRoutingPlugin(LiteLLMRequestRewriter):+ """Randomly select from the current unique LiteLLM candidates."""++ def __init__(+ self,+ *,+ weights: Sequence[float] | None = None,+ seed: int | None = None,+ ) -> None:+ super().__init__()+ self._weights = tuple(weights) if weights is not None else None+ self._seed = seed+ self._plugins: dict[tuple[str, ...], SwitchyardRoutingPlugin] = {}++ async def run(self, context: RoutingContext) -> RoutingContext:+ """Reuse one Random algorithm for each live LiteLLM candidate pool."""+ candidates = tuple(dict.fromkeys(context.candidate_models))+ if not candidates:+ raise ValueError("Random routing requires at least one LiteLLM candidate")+ plugin = self._plugins.get(candidates)+ if plugin is None:+ plugin = SwitchyardRoutingPlugin(+ algorithms.random(candidates, weights=self._weights, seed=self._seed)+ )+ self._plugins[candidates] = plugin+ return await plugin.run(context)+++__all__ = ["RandomRoutingPlugin"]diff --git a/examples/litellm/src/switchyard_litellm/plugins/request_rewrite.py b/examples/litellm/src/switchyard_litellm/plugins/request_rewrite.pynew file mode 100644index 000000000..fea62d496--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/plugins/request_rewrite.py@@ -0,0 +1,286 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Translate Switchyard request deltas into LiteLLM completion arguments."""++from __future__ import annotations++import json+from collections.abc import Mapping, Sequence+from typing import cast++_REQUEST_FIELDS = frozenset(+ {+ "model",+ "instructions",+ "messages",+ "tools",+ "tool_choice",+ "sampling",+ "output",+ "reasoning",+ "stream",+ "extensions",+ "preservation",+ }+)+_EXTENSION_FIELDS = {+ "parallel_tool_calls": "parallel_tool_calls",+ "prompt_cache_key": "prompt_cache_key",+ "prompt_cache_retention": "prompt_cache_retention",+ "safety_identifier": "safety_identifier",+ "service_tier": "service_tier",+ "store": "store",+ "stream_options": "stream_options",+ "top_logprobs": "top_logprobs",+ "user": "user",+ "stop_sequences": "stop",+}+++def _mapping(value: object, path: str) -> Mapping[str, object]:+ """Require a mapping and include its request path in validation errors."""+ if not isinstance(value, Mapping):+ raise ValueError(f"{path} must be a mapping")+ return cast(Mapping[str, object], value)+++def _sequence(value: object, path: str) -> Sequence[object]:+ """Require a non-string sequence at the given request path."""+ if isinstance(value, (str, bytes, bytearray)) or not isinstance(value, Sequence):+ raise ValueError(f"{path} must be a sequence")+ return cast(Sequence[object], value)+++def _text_content(value: object, path: str, separator: str) -> str:+ """Join normalized text blocks for a LiteLLM message field."""+ parts: list[str] = []+ for index, raw_block in enumerate(_sequence(value, path)):+ block_path = f"{path}[{index}]"+ block = _mapping(raw_block, block_path)+ text = block.get("text")+ if block.get("type") != "text" or not isinstance(text, str):+ raise ValueError(f"{block_path} cannot safely apply non-text content to LiteLLM")+ parts.append(text)+ return separator.join(parts)+++def _instructions(value: object) -> list[dict[str, object]]:+ """Convert normalized system and developer instructions to chat messages."""+ instructions: list[dict[str, object]] = []+ for index, raw_instruction in enumerate(_sequence(value, "request.instructions")):+ path = f"request.instructions[{index}]"+ instruction = _mapping(raw_instruction, path)+ role = instruction.get("role")+ if role not in {"system", "developer"}:+ raise ValueError(f"{path} cannot safely apply role {role!r} to LiteLLM")+ instructions.append(+ {+ "role": role,+ "content": _text_content(instruction.get("content"), f"{path}.content", "\n\n"),+ }+ )+ return instructions+++def _tool_call(block: Mapping[str, object], path: str) -> dict[str, object]:+ """Convert one normalized tool call to OpenAI function-call shape."""+ call_id = block.get("id")+ name = block.get("name")+ if not isinstance(call_id, str) or not call_id:+ raise ValueError(f"{path}.id cannot safely apply an empty tool-call ID to LiteLLM")+ if not isinstance(name, str) or not name:+ raise ValueError(f"{path}.name cannot safely apply an empty tool name to LiteLLM")+ return {+ "id": call_id,+ "type": "function",+ "function": {+ "name": name,+ "arguments": json.dumps(block.get("arguments"), separators=(",", ":")),+ },+ }+++def _tool_result(block: Mapping[str, object], path: str) -> dict[str, object]:+ """Convert one normalized tool result to an OpenAI tool message."""+ tool_call_id = block.get("tool_call_id")+ if not isinstance(tool_call_id, str) or not tool_call_id:+ raise ValueError(f"{path}.tool_call_id cannot safely apply an empty ID to LiteLLM")+ return {+ "role": "tool",+ "tool_call_id": tool_call_id,+ "content": _text_content(block.get("content"), f"{path}.content", " "),+ }+++def _messages(value: object) -> list[dict[str, object]]:+ """Convert normalized Switchyard messages to LiteLLM chat messages."""+ messages: list[dict[str, object]] = []+ for index, raw_message in enumerate(_sequence(value, "request.messages")):+ path = f"request.messages[{index}]"+ message = _mapping(raw_message, path)+ role = message.get("role")+ if role not in {"system", "developer", "user", "assistant", "tool"}:+ raise ValueError(f"{path} cannot safely apply role {role!r} to LiteLLM")++ text_blocks: list[dict[str, object]] = []+ tool_calls: list[dict[str, object]] = []+ tool_results: list[dict[str, object]] = []+ for block_index, raw_block in enumerate(_sequence(message.get("content"), f"{path}.content")):+ block_path = f"{path}.content[{block_index}]"+ block = _mapping(raw_block, block_path)+ block_type = block.get("type")+ if block_type == "text" and isinstance(block.get("text"), str):+ text_blocks.append(dict(block))+ elif block_type == "tool_call" and role == "assistant":+ tool_calls.append(_tool_call(block, block_path))+ elif block_type == "tool_result" and role in {"tool", "user"}:+ tool_results.append(_tool_result(block, block_path))+ else:+ raise ValueError(+ f"{block_path} cannot safely apply content type {block_type!r} to LiteLLM"+ )++ messages.extend(tool_results)+ if text_blocks or tool_calls or not tool_results:+ converted: dict[str, object] = {+ "role": role,+ "content": "\n".join(cast(str, block["text"]) for block in text_blocks),+ }+ if tool_calls:+ converted["tool_calls"] = tool_calls+ if not text_blocks:+ converted["content"] = None+ messages.append(converted)+ return messages+++def _tools(value: object) -> list[dict[str, object]]:+ """Convert normalized tool definitions to OpenAI function tools."""+ tools: list[dict[str, object]] = []+ for index, raw_tool in enumerate(_sequence(value, "request.tools")):+ path = f"request.tools[{index}]"+ tool = _mapping(raw_tool, path)+ name = tool.get("name")+ if not isinstance(name, str) or not name:+ raise ValueError(f"{path}.name cannot safely apply an empty tool name to LiteLLM")+ function: dict[str, object] = {+ "name": name,+ "description": tool.get("description") or "",+ "parameters": tool.get("parameters", {}),+ }+ strict = tool.get("strict")+ if strict is not None:+ if not isinstance(strict, bool):+ raise ValueError(f"{path}.strict cannot safely apply a non-Boolean value")+ function["strict"] = strict+ tools.append({"type": "function", "function": function})+ return tools+++def _tool_choice(value: object) -> object:+ """Convert normalized tool-selection policy to LiteLLM form."""+ if value is None:+ return None+ choice = _mapping(value, "request.tool_choice")+ choice_type = choice.get("type")+ if choice_type in {"auto", "required", "none"}:+ return choice_type+ if choice_type == "tool":+ data = _mapping(choice.get("data"), "request.tool_choice.data")+ name = data.get("name")+ if not isinstance(name, str) or not name:+ raise ValueError("request.tool_choice cannot safely apply an empty tool name")+ return {"type": "function", "function": {"name": name}}+ if choice_type == "raw":+ return choice.get("data")+ raise ValueError(f"request.tool_choice cannot safely apply type {choice_type!r}")+++def _changed_fields(+ original: Mapping[str, object],+ rewritten: Mapping[str, object],+ fields: Sequence[str],+) -> list[str]:+ """Return the requested fields whose normalized values changed."""+ return [field for field in fields if original.get(field) != rewritten.get(field)]+++def build_request_patch(+ original_request: Mapping[str, object],+ rewritten_request: Mapping[str, object],+ *,+ selected_model_id: str,+) -> dict[str, object]:+ """Build a LiteLLM argument patch for fields explicitly changed by Switchyard."""+ if set(original_request) != _REQUEST_FIELDS or set(rewritten_request) != _REQUEST_FIELDS:+ raise ValueError("Switchyard request shape changed and cannot safely apply to LiteLLM")+ if rewritten_request.get("model") != selected_model_id:+ raise ValueError("Switchyard request model cannot safely apply to the selected deployment")+ if original_request.get("preservation") != rewritten_request.get("preservation"):+ raise ValueError("Switchyard preservation rewrite cannot safely apply to LiteLLM")++ set_values: dict[str, object] = {}+ remove_values: list[str] = []+ if _changed_fields(original_request, rewritten_request, ["instructions", "messages"]):+ set_values["messages"] = [+ *_instructions(rewritten_request.get("instructions")),+ *_messages(rewritten_request.get("messages")),+ ]+ if original_request.get("tools") != rewritten_request.get("tools"):+ set_values["tools"] = _tools(rewritten_request.get("tools"))+ if original_request.get("tool_choice") != rewritten_request.get("tool_choice"):+ set_values["tool_choice"] = _tool_choice(rewritten_request.get("tool_choice"))++ original_sampling = _mapping(original_request.get("sampling"), "original.sampling")+ rewritten_sampling = _mapping(rewritten_request.get("sampling"), "request.sampling")+ for field in ("temperature", "top_p", "top_k"):+ if original_sampling.get(field) != rewritten_sampling.get(field):+ set_values[field] = rewritten_sampling.get(field)++ original_output = _mapping(original_request.get("output"), "original.output")+ rewritten_output = _mapping(rewritten_request.get("output"), "request.output")+ if original_output.get("max_output_tokens") != rewritten_output.get("max_output_tokens"):+ set_values["max_completion_tokens"] = rewritten_output.get("max_output_tokens")+ remove_values.append("max_tokens")+ if original_output.get("response_format") != rewritten_output.get("response_format"):+ set_values["response_format"] = rewritten_output.get("response_format")++ original_reasoning = _mapping(original_request.get("reasoning"), "original.reasoning")+ rewritten_reasoning = _mapping(rewritten_request.get("reasoning"), "request.reasoning")+ if original_reasoning.get("raw") != rewritten_reasoning.get("raw"):+ raise ValueError("Switchyard raw reasoning rewrite cannot safely apply to LiteLLM")+ if original_reasoning.get("effort") != rewritten_reasoning.get("effort"):+ set_values["reasoning_effort"] = rewritten_reasoning.get("effort")++ if original_request.get("stream") != rewritten_request.get("stream"):+ stream = rewritten_request.get("stream")+ if not isinstance(stream, bool):+ raise ValueError("Switchyard stream rewrite cannot safely apply to LiteLLM")+ set_values["stream"] = stream++ original_extensions = _mapping(original_request.get("extensions"), "original.extensions")+ rewritten_extensions = _mapping(rewritten_request.get("extensions"), "request.extensions")+ original_fields = _mapping(original_extensions.get("fields"), "original.extensions.fields")+ rewritten_fields = _mapping(rewritten_extensions.get("fields"), "request.extensions.fields")+ changed_extension_fields = set(original_fields) | set(rewritten_fields)+ for field in sorted(changed_extension_fields):+ if original_fields.get(field) == rewritten_fields.get(field):+ continue+ litellm_field = _EXTENSION_FIELDS.get(field)+ if litellm_field is None:+ raise ValueError(+ f"Switchyard extension {field!r} cannot safely apply to LiteLLM"+ )+ if field in rewritten_fields:+ set_values[litellm_field] = rewritten_fields[field]+ else:+ remove_values.append(litellm_field)++ patch: dict[str, object] = {"set": set_values}+ if remove_values:+ patch["remove"] = remove_values+ return patch+++__all__ = ["build_request_patch"]diff --git a/examples/litellm/src/switchyard_litellm/plugins/stage_routing_plugin.py b/examples/litellm/src/switchyard_litellm/plugins/stage_routing_plugin.pynew file mode 100644index 000000000..a17e23f1b--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/plugins/stage_routing_plugin.py@@ -0,0 +1,71 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Bind Stage routing to the candidates supplied by LiteLLM."""++from __future__ import annotations++from litellm.types.router import RoutingContext++from switchyard.libsy import algorithms++from .lite_llm_request_rewriter import LiteLLMRequestRewriter+from .switchyard_routing_plugin import SwitchyardRoutingPlugin+++class StageRoutingPlugin(LiteLLMRequestRewriter):+ """Route between LiteLLM's first capable and second efficient candidate.++ LiteLLM 1.97 preserves ``model_list`` declaration order when it constructs+ the routing context. The checked-in profile and integration tests make that+ order an explicit deployment contract.+ """++ def __init__(+ self,+ *,+ picker: str,+ confidence_threshold: float,+ recent_window: int | None = None,+ escalation_note: str | None = None,+ deescalation_note: str | None = None,+ only_on_wrong_signal_escalation: bool = True,+ capable_system_prompt: str | None = None,+ efficient_system_prompt: str | None = None,+ ) -> None:+ super().__init__()+ self._picker = picker+ self._confidence_threshold = confidence_threshold+ self._recent_window = recent_window+ self._escalation_note = escalation_note+ self._deescalation_note = deescalation_note+ self._only_on_wrong_signal_escalation = only_on_wrong_signal_escalation+ self._capable_system_prompt = capable_system_prompt+ self._efficient_system_prompt = efficient_system_prompt++ async def run(self, context: RoutingContext) -> RoutingContext:+ """Build Stage routing from the current LiteLLM candidate order."""+ candidates = list(dict.fromkeys(context.candidate_models))+ if len(candidates) != 2:+ raise ValueError(+ "Stage routing requires exactly two unique LiteLLM candidates "+ "ordered as capable then efficient"+ )+ plugin = SwitchyardRoutingPlugin(+ algorithms.stage_router(+ candidates[0],+ candidates[1],+ picker=self._picker,+ confidence_threshold=self._confidence_threshold,+ recent_window=self._recent_window,+ escalation_note=self._escalation_note,+ deescalation_note=self._deescalation_note,+ only_on_wrong_signal_escalation=self._only_on_wrong_signal_escalation,+ capable_system_prompt=self._capable_system_prompt,+ efficient_system_prompt=self._efficient_system_prompt,+ )+ )+ return await plugin.run(context)+++__all__ = ["StageRoutingPlugin"]diff --git a/examples/litellm/src/switchyard_litellm/plugins/switchyard_routing_plugin.py b/examples/litellm/src/switchyard_litellm/plugins/switchyard_routing_plugin.pynew file mode 100644index 000000000..1532204a1--- /dev/null+++ b/examples/litellm/src/switchyard_litellm/plugins/switchyard_routing_plugin.py@@ -0,0 +1,209 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Use a compatible Switchyard algorithm to narrow LiteLLM routing candidates."""++from __future__ import annotations++import json+from collections.abc import Mapping, Sequence+from typing import cast++from litellm.types.router import RoutingContext++from switchyard.libsy import Algorithm, Step++from .lite_llm_request_rewriter import LiteLLMRequestRewriter+from .request_rewrite import build_request_patch+++def _mapping(value: object, path: str) -> Mapping[str, object]:+ """Require a mapping and preserve its message path in validation errors."""+ if not isinstance(value, Mapping):+ raise ValueError(f"{path} must be a mapping")+ return cast(Mapping[str, object], value)+++def _sequence(value: object, path: str) -> Sequence[object]:+ """Require a non-string sequence at the given message path."""+ if isinstance(value, (str, bytes, bytearray)) or not isinstance(value, Sequence):+ raise ValueError(f"{path} must be a sequence")+ return cast(Sequence[object], value)+++def _text_blocks(value: object, path: str, *, allow_none: bool = False) -> list[dict[str, object]]:+ """Normalize LiteLLM string or text-block content for Switchyard."""+ if value is None and allow_none:+ return []+ if isinstance(value, str):+ return [{"type": "text", "text": value}]++ blocks: list[dict[str, object]] = []+ for index, raw_block in enumerate(_sequence(value, path)):+ block_path = f"{path}[{index}]"+ block = _mapping(raw_block, block_path)+ if block.get("type") != "text" or not isinstance(block.get("text"), str):+ raise ValueError(f"{block_path} content type is not supported")+ blocks.append({"type": "text", "text": cast(str, block["text"])})+ return blocks+++def _tool_calls(value: object, path: str) -> list[dict[str, object]]:+ """Normalize OpenAI function calls and parse their JSON arguments."""+ calls: list[dict[str, object]] = []+ for index, raw_call in enumerate(_sequence(value, path)):+ call_path = f"{path}[{index}]"+ call = _mapping(raw_call, call_path)+ call_id = call.get("id")+ if call.get("type") != "function":+ raise ValueError(f"{call_path}.type is not supported")+ if not isinstance(call_id, str) or not call_id:+ raise ValueError(f"{call_path}.id must be a non-empty string")+ function = _mapping(call.get("function"), f"{call_path}.function")+ name = function.get("name")+ arguments = function.get("arguments")+ if not isinstance(name, str) or not name:+ raise ValueError(f"{call_path}.function.name must be a non-empty string")+ if not isinstance(arguments, str):+ raise ValueError(f"{call_path}.function.arguments must be a JSON string")+ try:+ parsed_arguments = json.loads(arguments)+ except json.JSONDecodeError as error:+ raise ValueError(+ f"{call_path}.function.arguments must be a valid JSON object"+ ) from error+ if not isinstance(parsed_arguments, Mapping):+ raise ValueError(f"{call_path}.function.arguments must be a valid JSON object")+ calls.append(+ {+ "type": "tool_call",+ "id": call_id,+ "name": name,+ "arguments": dict(parsed_arguments),+ }+ )+ return calls+++def _messages(structured_messages: Sequence[object]) -> list[dict[str, object]]:+ """Normalize LiteLLM structured chat history for Switchyard algorithms."""+ if not structured_messages:+ raise ValueError("structured_messages must not be empty")++ converted: list[dict[str, object]] = []+ for index, raw_message in enumerate(structured_messages):+ path = f"structured_messages[{index}]"+ message = _mapping(raw_message, path)+ role = message.get("role")+ if role in {"system", "developer", "user"}:+ converted.append(+ {+ "role": role,+ "content": _text_blocks(message.get("content"), f"{path}.content"),+ }+ )+ elif role == "assistant":+ content: list[dict[str, object]] = list(+ _text_blocks(message.get("content"), f"{path}.content", allow_none=True)+ )+ raw_calls = message.get("tool_calls")+ if raw_calls is not None:+ content.extend(_tool_calls(raw_calls, f"{path}.tool_calls"))+ if not content:+ raise ValueError(f"{path} must contain text or tool calls")+ converted.append({"role": "assistant", "content": content})+ elif role == "tool":+ call_id = message.get("tool_call_id")+ if not isinstance(call_id, str) or not call_id:+ raise ValueError(f"{path}.tool_call_id must be a non-empty string")+ converted.append(+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": call_id,+ "content": _text_blocks(message.get("content"), f"{path}.content"),+ "is_error": None,+ }+ ],+ }+ )+ else:+ raise ValueError(f"{path}.role is not supported")+ return converted+++def _request(structured_messages: Sequence[object]) -> dict[str, object]:+ """Build the canonical normalized request used to detect unsupported rewrites."""+ return {+ "model": "auto",+ "instructions": [],+ "messages": _messages(structured_messages),+ "tools": [],+ "tool_choice": None,+ "sampling": {"temperature": None, "top_p": None, "top_k": None},+ "output": {"max_output_tokens": None, "response_format": None},+ "reasoning": {"effort": None, "raw": None},+ "stream": False,+ "extensions": {"fields": {}},+ "preservation": {"requests": {}, "responses": {}},+ }+++class SwitchyardRoutingPlugin(LiteLLMRequestRewriter):+ """Narrow LiteLLM candidates with a decision-only Switchyard algorithm.++ Compatible algorithms must finish without intermediate model calls or an+ already-produced response. Supported request rewrites are carried to the+ selected deployment by the object's LiteLLM callback role.+ """++ def __init__(self, algorithm: Algorithm) -> None:+ super().__init__()+ self._algorithm = algorithm++ async def run(self, context: RoutingContext) -> RoutingContext:+ """Run Switchyard and retain only its selected LiteLLM candidate."""+ candidates = list(context.candidate_models)+ request = _request(context.structured_messages)++ async for step in self._algorithm.run_stream(request):+ match step:+ case Step.CallModel(_):+ raise ValueError(+ "Switchyard algorithm requested an intermediate model call, "+ "which a LiteLLM routing plugin cannot serve"+ )+ case Step.Done(outcome):+ if outcome.response is not None:+ raise ValueError(+ "Switchyard algorithm produced a response while routing, "+ "which a LiteLLM routing plugin cannot return"+ )+ selected = outcome.selected_model_id+ if selected not in candidates:+ raise ValueError(+ f"Switchyard selected {selected!r}, which is not in LiteLLM's "+ "candidate pool"+ )+ request_patch = build_request_patch(+ request,+ outcome.request,+ selected_model_id=selected,+ )+ context.candidate_models = [+ candidate for candidate in candidates if candidate == selected+ ]+ context.signals["switchyard"] = {+ "selected_model_id": selected,+ "fallback_models": outcome.fallback_models,+ }+ if request_patch["set"] or request_patch.get("remove"):+ context.signals["switchyard"]["request_patch"] = request_patch+ return context++ raise ValueError("Switchyard algorithm stream ended without a routing outcome")+++__all__ = ["SwitchyardRoutingPlugin"]diff --git a/examples/experimental/litellm/tests/conftest.py b/examples/litellm/tests/conftest.pysimilarity index 100%rename from examples/experimental/litellm/tests/conftest.pyrename to examples/litellm/tests/conftest.pydiff --git a/examples/litellm/tests/integration/test_e2e.py b/examples/litellm/tests/integration/test_e2e.pynew file mode 100644index 000000000..f271312ad--- /dev/null+++ b/examples/litellm/tests/integration/test_e2e.py@@ -0,0 +1,187 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++import os+import shutil+import socket+import subprocess+from collections.abc import Iterator+from pathlib import Path++import httpx+import pytest++PACKAGE_ROOT = Path(__file__).resolve().parents[2]+DEPLOYMENT_ROOT = PACKAGE_ROOT / "deployment"+MODEL_GROUP = "switchyard"+SOL_MODEL = "openrouter/openai/gpt-5.6-sol"+TERRA_MODEL = "openrouter/openai/gpt-5.6-terra"+++def _free_port() -> int:+ """Reserve and return an available loopback port for one Compose project."""+ with socket.socket() as sock:+ sock.bind(("127.0.0.1", 0))+ return int(sock.getsockname()[1])+++def _require_live_environment() -> None:+ """Skip paid tests unless the explicit opt-in, key, and Docker are available."""+ if os.environ.get("SWITCHYARD_LITELLM_E2E") != "1":+ pytest.skip("SWITCHYARD_LITELLM_E2E=1 is required for paid E2E tests")+ if not os.environ.get("OPENROUTER_API_KEY"):+ pytest.skip("OPENROUTER_API_KEY is required for paid E2E tests")+ if shutil.which("docker") is None:+ pytest.skip("Docker is required for paid E2E tests")+ subprocess.run(+ ["docker", "compose", "version"],+ check=True,+ stdout=subprocess.DEVNULL,+ )+++def litellm_base_url(request: pytest.FixtureRequest) -> Iterator[str]:+ """Build one profile, yield its local proxy URL, and tear it down."""+ _require_live_environment()++ profile = str(request.param)+ port = _free_port()+ project = f"switchyard-litellm-{profile}-{os.getpid()}"+ env = {+ **os.environ,+ "LITELLM_NETWORK": f"{project}-network",+ "LITELLM_PORT": str(port),+ "SWITCHYARD_LITELLM_PROFILE": profile,+ }+ compose = ["docker", "compose", "--project-name", project]+ try:+ subprocess.run(+ [*compose, "up", "-d", "--build", "--wait"],+ cwd=DEPLOYMENT_ROOT,+ env=env,+ check=True,+ )+ yield f"http://127.0.0.1:{port}/v1"+ finally:+ subprocess.run(+ [*compose, "down", "--volumes", "--remove-orphans"],+ cwd=DEPLOYMENT_ROOT,+ env=env,+ check=True,+ )+++def test_paid_e2e_requires_explicit_spend_opt_in(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Avoid checking Docker or spending unless live E2E is explicitly enabled."""+ monkeypatch.setenv("OPENROUTER_API_KEY", "test-key")+ monkeypatch.delenv("SWITCHYARD_LITELLM_E2E", raising=False)++ def fail_if_docker_is_checked(_: str) -> str | None:+ pytest.fail("Docker must not be checked without the paid E2E opt-in")++ monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)+ with pytest.raises(pytest.skip.Exception, match="SWITCHYARD_LITELLM_E2E"):+ _require_live_environment()+++def test_paid_e2e_requires_openrouter_key(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Require the profile's OpenRouter credential before checking Docker."""+ monkeypatch.setenv("SWITCHYARD_LITELLM_E2E", "1")+ monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)++ def fail_if_docker_is_checked(_: str) -> str | None:+ pytest.fail("Docker must not be checked without OPENROUTER_API_KEY")++ monkeypatch.setattr(shutil, "which", fail_if_docker_is_checked)+ with pytest.raises(pytest.skip.Exception, match="OPENROUTER_API_KEY"):+ _require_live_environment()+++def _critical_tool_history() -> list[dict[str, object]]:+ """Build a critical failed-tool turn that Stage must send to capable."""+ return [+ {"role": "user", "content": "Fix the failing tests, then report the result."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {+ "role": "tool",+ "tool_call_id": "call_1",+ "content": "fatal runtime error: out of memory",+ },+ ]+++async def _complete(+ base_url: str,+ messages: list[dict[str, object]],+) -> tuple[dict[str, object], str]:+ """Send one chat request and return its body and concrete selected model."""+ async with httpx.AsyncClient(timeout=180) as client:+ response = await client.post(+ f"{base_url}/chat/completions",+ json={"model": MODEL_GROUP, "messages": messages, "max_tokens": 64},+ )+ response.raise_for_status()+ return response.json(), response.headers["x-litellm-model-name"]+++def _assert_selected(+ response: dict[str, object],+ selected_model: str,+ expected: str,+) -> None:+ """Check the public group response and concrete deployment selection."""+ assert response.get("model") == MODEL_GROUP+ assert selected_model == expected+ choices = response.get("choices")+ assert isinstance(choices, list) and choices++[email protected]("litellm_base_url", ["stage"], indirect=True)+async def test_stage_plugin_routes_live_requests_to_both_gpt_5_6_models(+ litellm_base_url: str,+) -> None:+ """Exercise efficient and capable Stage decisions against live deployments."""+ efficient, efficient_model = await _complete(+ litellm_base_url,+ [{"role": "user", "content": "Reply with the word hello."}],+ )+ capable, capable_model = await _complete(litellm_base_url, _critical_tool_history())++ _assert_selected(efficient, efficient_model, TERRA_MODEL)+ _assert_selected(capable, capable_model, SOL_MODEL)++[email protected]("litellm_base_url", ["random"], indirect=True)+async def test_random_plugin_routes_live_requests_to_both_gpt_5_6_models(+ litellm_base_url: str,+) -> None:+ """Exercise the seeded Random profile against both live deployments."""+ selected = []+ for index in range(2):+ _, selected_model = await _complete(+ litellm_base_url,+ [{"role": "user", "content": f"Reply with the number {index}."}],+ )+ selected.append(selected_model)++ assert selected == [SOL_MODEL, TERRA_MODEL]diff --git a/examples/litellm/tests/integration/test_gateway_config.py b/examples/litellm/tests/integration/test_gateway_config.pynew file mode 100644index 000000000..b2cb1050e--- /dev/null+++ b/examples/litellm/tests/integration/test_gateway_config.py@@ -0,0 +1,101 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++import runpy+import tomllib+from pathlib import Path++import yaml+from switchyard_litellm import StageRoutingPlugin++PACKAGE_ROOT = Path(__file__).resolve().parents[2]+REPOSITORY_ROOT = PACKAGE_ROOT.parents[1]+DEPLOYMENT_ROOT = PACKAGE_ROOT / "deployment"+++def test_example_targets_python_312() -> None:+ project = tomllib.loads((PACKAGE_ROOT / "pyproject.toml").read_text())++ assert project["project"]["requires-python"] == ">=3.12,<3.13"+ assert project["project"]["dependencies"] == [+ "nemo-switchyard>=0.1.0",+ "litellm==1.97.0",+ ]+++def test_lockfile_matches_root_package_version() -> None:+ root_project = tomllib.loads((REPOSITORY_ROOT / "pyproject.toml").read_text())+ lock = tomllib.loads((PACKAGE_ROOT / "uv.lock").read_text())+ locked_root = next(+ package for package in lock["package"] if package["name"] == "nemo-switchyard"+ )++ assert locked_root["version"] == root_project["project"]["version"]+++def test_compose_builds_the_pinned_plugin_image_and_selects_a_profile() -> None:+ compose = yaml.safe_load((DEPLOYMENT_ROOT / "compose.yaml").read_text())+ service = compose["services"]["litellm"]++ assert service["build"] == {+ "context": "../../..",+ "dockerfile": "examples/litellm/deployment/Dockerfile",+ }+ assert service["image"] == "switchyard-litellm:1.97.0"+ assert service["command"] == ["--config", "/app/deployment/litellm.yaml"]+ assert service["environment"] == {+ "OPENROUTER_API_KEY": "${OPENROUTER_API_KEY:?set OPENROUTER_API_KEY}",+ "SWITCHYARD_LITELLM_CONFIG": "/app/deployment/switchyard.toml",+ }+ assert service["volumes"] == [+ "./profiles/${SWITCHYARD_LITELLM_PROFILE:-stage}:/app/deployment:ro"+ ]+ assert (DEPLOYMENT_ROOT / ".env.example").read_text() == (+ "OPENROUTER_API_KEY=\nSWITCHYARD_LITELLM_PROFILE=stage\n"+ )+++def test_dockerfile_runs_the_final_image_as_litellm_user() -> None:+ """Require the final runtime stage to select the dedicated non-root account."""+ instructions = [+ line.strip()+ for line in (DEPLOYMENT_ROOT / "Dockerfile").read_text().splitlines()+ if line.strip() and not line.lstrip().startswith("#")+ ]+ final_stage = max(index for index, line in enumerate(instructions) if line.startswith("FROM "))+ runtime_users = [+ line.split(maxsplit=1)[1]+ for line in instructions[final_stage + 1 :]+ if line.startswith("USER ")+ ]++ assert runtime_users == ["litellm"]+++def test_deployment_profiles_contain_only_model_and_policy_configuration() -> None:+ for profile in ("stage", "random"):+ assert {path.name for path in (DEPLOYMENT_ROOT / "profiles" / profile).iterdir()} == {+ "litellm.yaml",+ "switchyard.toml",+ }+++def test_python_router_example_owns_its_programmatic_stage_plugin() -> None:+ namespace = runpy.run_path(str(PACKAGE_ROOT / "examples" / "python_router.py"))++ assert isinstance(namespace["STAGE_ROUTING_PLUGIN"], StageRoutingPlugin)+ assert [entry["model_name"] for entry in namespace["MODEL_LIST"]] == [+ "switchyard",+ "switchyard",+ ]+++def test_docker_context_excludes_local_environment_files() -> None:+ patterns = {+ line.strip()+ for line in (REPOSITORY_ROOT / ".dockerignore").read_text().splitlines()+ if line.strip()+ }++ assert {".env", "**/.env", ".env.*", "**/.env.*"} <= patterns+ assert {"!.env.example", "!**/.env.example"} <= patternsdiff --git a/examples/litellm/tests/integration/test_profile_configuration.py b/examples/litellm/tests/integration/test_profile_configuration.pynew file mode 100644index 000000000..7743bf7bf--- /dev/null+++ b/examples/litellm/tests/integration/test_profile_configuration.py@@ -0,0 +1,99 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++import sys+import tomllib+from pathlib import Path++import pytest+import yaml+from litellm.proxy.types_utils.utils import get_instance_fn+from switchyard_litellm import RandomRoutingPlugin, StageRoutingPlugin+from switchyard_litellm.configuration import (+ CONFIG_ENV,+ load_routing_plugin_from_environment,+)++PACKAGE_ROOT = Path(__file__).resolve().parents[2]+PROFILE_ROOT = PACKAGE_ROOT / "deployment" / "profiles"+PLUGIN_PATH = "switchyard_litellm.configuration.configured_plugin.ROUTING_PLUGIN"+++def _expected_models() -> list[dict[str, object]]:+ return [+ {+ "model_name": "switchyard",+ "litellm_params": {+ "model": "openrouter/openai/gpt-5.6-sol",+ "api_key": "os.environ/OPENROUTER_API_KEY",+ },+ },+ {+ "model_name": "switchyard",+ "litellm_params": {+ "model": "openrouter/openai/gpt-5.6-terra",+ "api_key": "os.environ/OPENROUTER_API_KEY",+ },+ },+ ]+++def test_environment_loader_requires_a_path() -> None:+ with pytest.raises(ValueError, match=CONFIG_ENV):+ load_routing_plugin_from_environment({})+++def test_environment_loader_uses_the_configured_path(tmp_path: Path) -> None:+ config_path = tmp_path / "switchyard.toml"+ config_path.write_text('algorithm = "random"\nseed = 9\n')++ plugin = load_routing_plugin_from_environment({CONFIG_ENV: str(config_path)})++ assert isinstance(plugin, RandomRoutingPlugin)+++def test_checked_in_profiles_separate_models_from_routing_policy() -> None:+ expected_policies = {+ "stage": {+ "algorithm": "stage",+ "picker": "efficient_first",+ "confidence_threshold": 0.5,+ "recent_window": 3,+ "only_on_wrong_signal_escalation": True,+ "escalation_note": "The efficient tier failed; continue from its work.",+ "deescalation_note": "The capable tier completed the recovery.",+ "capable_system_prompt": "Handle this request as the capable tier.",+ "efficient_system_prompt": "Handle this request as the efficient tier.",+ },+ "random": {"algorithm": "random", "seed": 6},+ }++ for profile, policy in expected_policies.items():+ profile_root = PROFILE_ROOT / profile+ assert tomllib.loads((profile_root / "switchyard.toml").read_text()) == policy+ assert yaml.safe_load((profile_root / "litellm.yaml").read_text()) == {+ "model_list": _expected_models(),+ "router_settings": {"plugins": [PLUGIN_PATH]},+ "litellm_settings": {"callbacks": [PLUGIN_PATH]},+ }+++ ("profile", "plugin_type"),+ [("stage", StageRoutingPlugin), ("random", RandomRoutingPlugin)],+)+def test_litellm_resolves_the_profile_configured_plugin(+ monkeypatch: pytest.MonkeyPatch,+ profile: str,+ plugin_type: type[StageRoutingPlugin] | type[RandomRoutingPlugin],+) -> None:+ profile_root = PROFILE_ROOT / profile+ monkeypatch.setenv(CONFIG_ENV, str(profile_root / "switchyard.toml"))+ sys.modules.pop("switchyard_litellm.configuration.configured_plugin", None)++ plugin = get_instance_fn(+ value=PLUGIN_PATH,+ config_file_path=str(profile_root / "litellm.yaml"),+ )++ assert isinstance(plugin, plugin_type)diff --git a/examples/litellm/tests/integration/test_stage_routing.py b/examples/litellm/tests/integration/test_stage_routing.pynew file mode 100644index 000000000..b93d34b4f--- /dev/null+++ b/examples/litellm/tests/integration/test_stage_routing.py@@ -0,0 +1,166 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++from typing import Any++import litellm+import pytest+from litellm import Router+from litellm.integrations.custom_logger import CustomLogger+from switchyard_litellm import RandomRoutingPlugin, StageRoutingPlugin++MODEL_GROUP = "deployer-route"+CAPABLE_MODEL = "openrouter/example/deployer-capable"+EFFICIENT_MODEL = "openrouter/example/deployer-efficient"+++def stage_plugin() -> StageRoutingPlugin:+ """Build the deployer-configurable Stage policy used by the integration test."""+ return StageRoutingPlugin(+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ )+++def model_list() -> list[dict[str, Any]]:+ """Provide real LiteLLM deployments whose provider calls are replaced by responses."""+ return [+ {+ "model_name": MODEL_GROUP,+ "litellm_params": {+ "model": CAPABLE_MODEL,+ "mock_response": "served by capable",+ },+ },+ {+ "model_name": MODEL_GROUP,+ "litellm_params": {+ "model": EFFICIENT_MODEL,+ "mock_response": "served by efficient",+ },+ },+ ]+++def critical_tool_history() -> list[dict[str, Any]]:+ """Build OpenAI chat history containing a decisive Stage escalation signal."""+ return [+ {"role": "user", "content": "Fix the failing tests."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {+ "role": "tool",+ "tool_call_id": "call_1",+ "content": "fatal runtime error: out of memory",+ },+ ]+++def test_litellm_preserves_declaration_order_for_stage_candidates() -> None:+ """Lock the pinned LiteLLM contract that gives Stage capable then efficient."""+ router = Router(model_list=model_list(), plugins=[stage_plugin()])+ deployments = router.get_model_list(model_name=MODEL_GROUP)++ assert deployments is not None+ assert [deployment["litellm_params"]["model"] for deployment in deployments] == [+ CAPABLE_MODEL,+ EFFICIENT_MODEL,+ ]+++async def test_stage_plugin_controls_real_litellm_router_pipeline() -> None:+ router = Router(model_list=model_list(), plugins=[stage_plugin()])++ efficient = await router.acompletion(+ model=MODEL_GROUP,+ messages=[{"role": "user", "content": "Say hello."}],+ )+ capable = await router.acompletion(+ model=MODEL_GROUP,+ messages=critical_tool_history(),+ )++ assert efficient.choices[0].message.content == "served by efficient"+ assert capable.choices[0].message.content == "served by capable"+++async def test_seeded_random_plugin_reaches_both_litellm_candidates() -> None:+ router = Router(model_list=model_list(), plugins=[RandomRoutingPlugin(seed=6)])++ served = {+ (+ await router.acompletion(+ model=MODEL_GROUP,+ messages=[{"role": "user", "content": f"Request {index}"}],+ )+ )+ .choices[0]+ .message.content+ for index in range(2)+ }++ assert served == {"served by capable", "served by efficient"}+++class DeploymentRequestRecorder(CustomLogger):+ """Record the request after all deployment callbacks have transformed it."""++ def __init__(self) -> None:+ super().__init__()+ self.requests: list[dict[str, Any]] = []++ async def async_pre_call_deployment_hook(+ self,+ kwargs: dict[str, Any],+ call_type: object,+ ) -> None:+ self.requests.append(kwargs)+++async def test_stage_rewrite_reaches_the_selected_litellm_deployment(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ plugin = StageRoutingPlugin(+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ escalation_note="The efficient tier failed.",+ capable_system_prompt="Use the capable tier.",+ )+ recorder = DeploymentRequestRecorder()+ monkeypatch.setattr(litellm, "callbacks", [plugin, recorder])+ router = Router(model_list=model_list(), plugins=[plugin])++ response = await router.acompletion(+ model=MODEL_GROUP,+ messages=critical_tool_history(),+ temperature=0.2,+ max_tokens=64,+ )++ assert response.choices[0].message.content == "served by capable"+ assert recorder.requests+ deployment_request = recorder.requests[-1]+ assert deployment_request["messages"][0] == {+ "role": "system",+ "content": "Use the capable tier.",+ }+ assert deployment_request["messages"][-1] == {+ "role": "user",+ "content": "The efficient tier failed.",+ }+ assert deployment_request["temperature"] == 0.2+ assert deployment_request["max_tokens"] == 64diff --git a/examples/litellm/tests/unit/test_candidate_routing_plugins.py b/examples/litellm/tests/unit/test_candidate_routing_plugins.pynew file mode 100644index 000000000..0f1cfa1e5--- /dev/null+++ b/examples/litellm/tests/unit/test_candidate_routing_plugins.py@@ -0,0 +1,126 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++from typing import Any++import pytest+import switchyard_litellm+from litellm.integrations.custom_logger import CustomLogger+from litellm.types.router import RoutingContext+++def routing_context(+ messages: list[dict[str, Any]],+ candidates: list[str],+) -> RoutingContext:+ """Build the LiteLLM boundary object passed to candidate-bound plugins."""+ return RoutingContext(+ raw_messages=messages,+ structured_messages=messages,+ candidate_models=candidates,+ metadata={},+ signals={},+ )+++async def test_stage_uses_deployer_candidates_in_capable_then_efficient_order() -> None:+ plugin_type = getattr(switchyard_litellm, "StageRoutingPlugin", None)+ assert plugin_type is not None, "StageRoutingPlugin must be part of the public integration API"+ plugin = plugin_type(+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ )+ assert isinstance(plugin, CustomLogger)+ context = routing_context(+ [{"role": "user", "content": "Say hello."}],+ ["provider/deployer-capable", "provider/deployer-efficient"],+ )++ await plugin.run(context)++ assert context.candidate_models == ["provider/deployer-efficient"]+ assert context.signals["switchyard"] == {+ "selected_model_id": "provider/deployer-efficient",+ "fallback_models": ["provider/deployer-capable"],+ }+++ "candidates",+ [+ ["provider/only-one"],+ ["provider/one", "provider/two", "provider/three"],+ ],+)+async def test_stage_rejects_candidate_pools_without_two_unique_models(+ candidates: list[str],+) -> None:+ plugin = switchyard_litellm.StageRoutingPlugin(+ picker="efficient_first",+ confidence_threshold=0.5,+ )+ context = routing_context([{"role": "user", "content": "Hello."}], candidates)++ with pytest.raises(ValueError, match="exactly two unique LiteLLM candidates"):+ await plugin.run(context)+++async def test_stage_treats_duplicate_deployments_as_one_candidate_model() -> None:+ plugin = switchyard_litellm.StageRoutingPlugin(+ picker="efficient_first",+ confidence_threshold=0.5,+ )+ context = routing_context(+ [{"role": "user", "content": "Hello."}],+ ["provider/capable", "provider/capable", "provider/efficient"],+ )++ await plugin.run(context)++ assert context.candidate_models == ["provider/efficient"]+++async def test_seeded_random_uses_and_retains_the_full_deployer_candidate_pool() -> None:+ plugin_type = getattr(switchyard_litellm, "RandomRoutingPlugin", None)+ assert plugin_type is not None, "RandomRoutingPlugin must be part of the public integration API"+ plugin = plugin_type(seed=6)+ assert isinstance(plugin, CustomLogger)+ candidates = ["provider/alpha", "provider/beta", "provider/gamma"]+ selected: list[str] = []++ for index in range(5):+ context = routing_context(+ [{"role": "user", "content": f"Request {index}."}],+ candidates.copy(),+ )+ await plugin.run(context)+ selected.extend(context.candidate_models)++ assert selected == [+ "provider/beta",+ "provider/gamma",+ "provider/gamma",+ "provider/gamma",+ "provider/alpha",+ ]+++async def test_random_rejects_an_empty_candidate_pool() -> None:+ plugin = switchyard_litellm.RandomRoutingPlugin(seed=6)+ context = routing_context([{"role": "user", "content": "Hello."}], [])++ with pytest.raises(ValueError, match="requires at least one LiteLLM candidate"):+ await plugin.run(context)+++async def test_random_weights_apply_to_unique_candidate_models() -> None:+ plugin = switchyard_litellm.RandomRoutingPlugin(weights=[0.0, 1.0], seed=6)+ context = routing_context(+ [{"role": "user", "content": "Hello."}],+ ["provider/alpha", "provider/alpha", "provider/beta"],+ )++ await plugin.run(context)++ assert context.candidate_models == ["provider/beta"]diff --git a/examples/litellm/tests/unit/test_routing_plugin_configuration.py b/examples/litellm/tests/unit/test_routing_plugin_configuration.pynew file mode 100644index 000000000..123fb7681--- /dev/null+++ b/examples/litellm/tests/unit/test_routing_plugin_configuration.py@@ -0,0 +1,248 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++from pathlib import Path+from typing import Any++import pytest+from litellm.types.router import RoutingContext+from switchyard_litellm.configuration import load_routing_plugin+++def routing_context(+ messages: list[dict[str, Any]],+ candidates: list[str],+) -> RoutingContext:+ """Build the LiteLLM boundary object passed to configured plugins."""+ return RoutingContext(+ raw_messages=messages,+ structured_messages=messages,+ candidate_models=candidates,+ metadata={},+ signals={},+ )+++async def test_loads_stage_picker_and_threshold(tmp_path: Path) -> None:+ config_path = tmp_path / "switchyard.toml"+ config_path.write_text(+ 'algorithm = "stage"\n'+ 'picker = "capable_first"\n'+ "confidence_threshold = 0.5\n"+ "recent_window = 3\n"+ "only_on_wrong_signal_escalation = true\n"+ )+ plugin = load_routing_plugin(config_path)+ context = routing_context(+ [{"role": "user", "content": "Say hello."}],+ ["provider/capable", "provider/efficient"],+ )++ await plugin.run(context)++ assert context.candidate_models == ["provider/capable"]+++async def test_loads_stage_prompt_and_handoff_rewrites(tmp_path: Path) -> None:+ config_path = tmp_path / "switchyard.toml"+ config_path.write_text(+ 'algorithm = "stage"\n'+ 'picker = "capable_first"\n'+ "confidence_threshold = 0.5\n"+ 'escalation_note = "The efficient tier failed."\n'+ 'deescalation_note = "The capable tier recovered."\n'+ 'capable_system_prompt = "Use the capable tier."\n'+ 'efficient_system_prompt = "Use the efficient tier."\n'+ )+ plugin = load_routing_plugin(config_path)+ context = routing_context(+ [{"role": "user", "content": "Say hello."}],+ ["provider/capable", "provider/efficient"],+ )++ await plugin.run(context)+ rewritten = await plugin.async_pre_call_deployment_hook(+ {+ "messages": [{"role": "user", "content": "Say hello."}],+ "metadata": {"routing_plugin_signals": context.signals},+ },+ None,+ )++ assert rewritten is not None+ assert rewritten["messages"][0] == {+ "role": "system",+ "content": "Use the capable tier.",+ }++ settled_messages = [+ {"role": "user", "content": "Implement the change."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "write_1",+ "type": "function",+ "function": {+ "name": "Write",+ "arguments": '{"file_path":"x.py","content":"pass"}',+ },+ }+ ],+ },+ {"role": "tool", "tool_call_id": "write_1", "content": "Wrote x.py"},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "test_1",+ "type": "function",+ "function": {"name": "Bash", "arguments": '{"command":"pytest"}'},+ }+ ],+ },+ {"role": "tool", "tool_call_id": "test_1", "content": "5 passed in 0.12s"},+ ]+ settled_context = routing_context(+ settled_messages,+ ["provider/capable", "provider/efficient"],+ )++ await plugin.run(settled_context)+ settled_rewrite = await plugin.async_pre_call_deployment_hook(+ {+ "messages": settled_messages,+ "metadata": {"routing_plugin_signals": settled_context.signals},+ },+ None,+ )++ assert settled_context.candidate_models == ["provider/efficient"]+ assert settled_rewrite is not None+ assert settled_rewrite["messages"][0] == {+ "role": "system",+ "content": "Use the efficient tier.",+ }+ assert settled_rewrite["messages"][-1] == {+ "role": "user",+ "content": "The capable tier recovered.",+ }+++async def test_loads_random_weights(tmp_path: Path) -> None:+ config_path = tmp_path / "switchyard.toml"+ config_path.write_text('algorithm = "random"\nseed = 6\nweights = [0.0, 1.0]\n')+ plugin = load_routing_plugin(config_path)+ context = routing_context(+ [{"role": "user", "content": "Say hello."}],+ ["provider/alpha", "provider/beta"],+ )++ await plugin.run(context)++ assert context.candidate_models == ["provider/beta"]+++ ("contents", "match"),+ [+ ("", "algorithm"),+ ("algorithm = 7\n", "algorithm"),+ ('algorithm = "unknown"\n', "unsupported algorithm"),+ ('algorithm = "stage"\nconfidence_threshold = 0.5\n', "picker"),+ (+ 'algorithm = "stage"\npicker = "fast"\nconfidence_threshold = 0.5\n',+ "picker",+ ),+ ('algorithm = "stage"\npicker = "efficient_first"\n', "confidence_threshold"),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = true\n',+ "confidence_threshold",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = nan\n',+ "finite",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = -0.1\n',+ "between 0 and 1",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\nconfidence_threshold = 1.1\n',+ "between 0 and 1",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\n'+ "confidence_threshold = 0.5\nrecent_window = true\n",+ "recent_window",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\n'+ "confidence_threshold = 0.5\nrecent_window = -1\n",+ "recent_window",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\n'+ "confidence_threshold = 0.5\nonly_on_wrong_signal_escalation = 1\n",+ "only_on_wrong_signal_escalation",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\n'+ 'confidence_threshold = 0.5\ncapable_system_prompt = ""\n',+ "capable_system_prompt",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\n'+ 'confidence_threshold = 0.5\ndeescalation_note = "Recovered."\n',+ "requires escalation_note",+ ),+ (+ 'algorithm = "stage"\npicker = "efficient_first"\n'+ "confidence_threshold = 0.5\nextra = true\n",+ "unknown keys",+ ),+ ('algorithm = "random"\nextra = true\n', "unknown keys"),+ ('algorithm = "random"\nseed = true\n', "seed"),+ ('algorithm = "random"\nseed = -1\n', "seed"),+ ('algorithm = "random"\nseed = 18446744073709551616\n', "seed"),+ ('algorithm = "random"\nweights = 1.0\n', "weights"),+ ('algorithm = "random"\nweights = []\n', "nonempty"),+ ('algorithm = "random"\nweights = [true]\n', "weights"),+ ('algorithm = "random"\nweights = [-1.0, 2.0]\n', "nonnegative"),+ ('algorithm = "random"\nweights = [nan, 1.0]\n', "finite"),+ ('algorithm = "random"\nweights = [0.0, 0.0]\n', "positive"),+ ],+)+def test_rejects_invalid_static_configuration(+ tmp_path: Path,+ contents: str,+ match: str,+) -> None:+ config_path = tmp_path / "switchyard.toml"+ config_path.write_text(contents)++ with pytest.raises(ValueError, match=match) as error:+ load_routing_plugin(config_path)++ assert str(config_path) in str(error.value)+++def test_wraps_missing_file_with_its_path(tmp_path: Path) -> None:+ config_path = tmp_path / "missing.toml"++ with pytest.raises(ValueError, match="Could not load") as error:+ load_routing_plugin(config_path)++ assert str(config_path) in str(error.value)+++def test_wraps_malformed_toml_with_its_path(tmp_path: Path) -> None:+ config_path = tmp_path / "switchyard.toml"+ config_path.write_text('algorithm = "random"\nweights = [\n')++ with pytest.raises(ValueError, match="Could not load") as error:+ load_routing_plugin(config_path)++ assert str(config_path) in str(error.value)diff --git a/examples/litellm/tests/unit/test_switchyard_routing_plugin.py b/examples/litellm/tests/unit/test_switchyard_routing_plugin.pynew file mode 100644index 000000000..3038f0bb5--- /dev/null+++ b/examples/litellm/tests/unit/test_switchyard_routing_plugin.py@@ -0,0 +1,448 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++from collections.abc import AsyncIterator+from typing import Any++import pytest+from litellm.integrations.custom_logger import CustomLogger+from litellm.types.router import RoutingContext+from switchyard_litellm import SwitchyardRoutingPlugin+from switchyard_litellm.plugins.request_rewrite import build_request_patch+from switchyard_litellm.plugins.switchyard_routing_plugin import _request++from switchyard.libsy import Step, TaskClassifierConfig, algorithms++SOL = "openrouter/openai/gpt-5.6-sol"+TERRA = "openrouter/openai/gpt-5.6-terra"+++def routing_context(+ messages: list[dict[str, Any]],+ *,+ candidates: list[str] | None = None,+) -> RoutingContext:+ """Build the exact LiteLLM boundary object passed to a routing plugin."""+ return RoutingContext(+ raw_messages=messages,+ structured_messages=messages,+ candidate_models=candidates or [SOL, TERRA],+ metadata={"tenant": "test"},+ signals={"earlier-plugin": {"allowed": True}},+ )+++def stage_plugin(**kwargs: object) -> SwitchyardRoutingPlugin:+ """Build the supported signal-only Stage configuration."""+ return SwitchyardRoutingPlugin(+ algorithms.stage_router(+ SOL,+ TERRA,+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ **kwargs,+ )+ )+++async def test_stage_converts_tool_history_and_narrows_to_capable_candidate() -> None:+ context = routing_context(+ [+ {"role": "system", "content": "You are a coding agent."},+ {"role": "developer", "content": [{"type": "text", "text": "Be concise."}]},+ {"role": "user", "content": "Fix the failing tests."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {+ "role": "tool",+ "tool_call_id": "call_1",+ "content": "fatal runtime error: out of memory",+ },+ ],+ candidates=[SOL, TERRA, SOL],+ )++ result = await stage_plugin().run(context)++ assert result is context+ assert result.candidate_models == [SOL, SOL]+ assert result.signals == {+ "earlier-plugin": {"allowed": True},+ "switchyard": {+ "selected_model_id": SOL,+ "fallback_models": [TERRA],+ },+ }+++async def test_litellm_conversion_preserves_stage_tool_signal_input() -> None:+ """Match direct libsy and LiteLLM-plugin routing over the same tool history."""+ litellm_messages = [+ {"role": "user", "content": "Fix the failing tests."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {+ "role": "tool",+ "tool_call_id": "call_1",+ "content": "fatal runtime error: out of memory",+ },+ ]+ original_messages: list[dict[str, object]] = [+ {+ "role": "user",+ "content": [{"type": "text", "text": "Fix the failing tests."}],+ },+ {+ "role": "assistant",+ "content": [+ {+ "type": "tool_call",+ "id": "call_1",+ "name": "Bash",+ "arguments": {"command": "pytest"},+ }+ ],+ },+ {+ "role": "tool",+ "content": [+ {+ "type": "tool_result",+ "tool_call_id": "call_1",+ "content": [+ {+ "type": "text",+ "text": "fatal runtime error: out of memory",+ }+ ],+ "is_error": None,+ }+ ],+ },+ ]+ original_request: dict[str, object] = {+ "model": "auto",+ "messages": original_messages,+ }++ direct_outcome = None+ direct_algorithm = algorithms.stage_router(+ SOL,+ TERRA,+ picker="efficient_first",+ confidence_threshold=0.5,+ recent_window=3,+ )+ async for step in direct_algorithm.run_stream(original_request):+ match step:+ case Step.Done(outcome):+ direct_outcome = outcome++ # ToolSignals reads only normalized messages, so exact equality protects every signal input.+ assert _request(litellm_messages)["messages"] == original_messages+ assert direct_outcome is not None+ assert direct_outcome.selected_model_id == SOL+ assert direct_outcome.fallback_models == [TERRA]++ context = routing_context(litellm_messages)+ await stage_plugin().run(context)++ assert context.signals["switchyard"] == {+ "selected_model_id": direct_outcome.selected_model_id,+ "fallback_models": direct_outcome.fallback_models,+ }+++async def test_stage_selects_efficient_candidate_without_tool_signals() -> None:+ context = routing_context([{"role": "user", "content": "Say hello."}])++ await stage_plugin().run(context)++ assert context.candidate_models == [TERRA]+ assert context.signals["switchyard"] == {+ "selected_model_id": TERRA,+ "fallback_models": [SOL],+ }+++ ("messages", "match"),+ [+ (+ [+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {"name": "Bash", "arguments": "not-json"},+ }+ ],+ }+ ],+ "valid JSON object",+ ),+ (+ [+ {+ "role": "user",+ "content": [+ {"type": "image_url", "image_url": {"url": "https://example.test/x"}}+ ],+ }+ ],+ "content.*not supported",+ ),+ ([{"role": "function", "content": "legacy"}], "role.*not supported"),+ ],+)+async def test_unsupported_structured_messages_fail_closed(+ messages: list[dict[str, Any]],+ match: str,+) -> None:+ with pytest.raises(ValueError, match=match):+ await stage_plugin().run(routing_context(messages))+++async def test_selection_outside_current_litellm_pool_fails_closed() -> None:+ plugin = SwitchyardRoutingPlugin(algorithms.random(["openrouter/openai/not-allowed"]))++ with pytest.raises(ValueError, match="not in LiteLLM's candidate pool"):+ await plugin.run(routing_context([{"role": "user", "content": "hello"}]))+++async def test_classifier_backed_algorithm_fails_on_intermediate_model_call() -> None:+ plugin = SwitchyardRoutingPlugin(+ algorithms.llm_task_classifier(+ "openrouter/openai/gpt-5.6-judge",+ TERRA,+ SOL,+ config=TaskClassifierConfig(0.5),+ )+ )++ with pytest.raises(ValueError, match="intermediate model call"):+ await plugin.run(routing_context([{"role": "user", "content": "hello"}]))+++async def test_algorithm_that_produces_a_response_fails_closed() -> None:+ with pytest.raises(ValueError, match="produced a response while routing"):+ await SwitchyardRoutingPlugin(algorithms.noop()).run(+ routing_context([{"role": "user", "content": "hello"}])+ )+++async def test_stage_request_rewrite_is_applied_after_litellm_selects_a_deployment() -> None:+ messages = [+ {"role": "user", "content": "Fix the failing tests."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {"role": "tool", "tool_call_id": "call_1", "content": "out of memory"},+ ]+ context = routing_context(messages)+ plugin = stage_plugin(+ capable_system_prompt="Use the capable tier.",+ escalation_note="The efficient tier failed.",+ )++ await plugin.run(context)+ rewritten = await plugin.async_pre_call_deployment_hook(+ {+ "messages": messages,+ "temperature": 0.2,+ "tools": [{"type": "function", "function": {"name": "Bash"}}],+ "metadata": {"routing_plugin_signals": context.signals},+ },+ None,+ )++ assert isinstance(plugin, CustomLogger)+ assert rewritten is not None+ assert rewritten["messages"] == [+ {"role": "system", "content": "Use the capable tier."},+ {"role": "user", "content": "Fix the failing tests."},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "Bash",+ "arguments": '{"command":"pytest"}',+ },+ }+ ],+ },+ {"role": "tool", "tool_call_id": "call_1", "content": "out of memory"},+ {"role": "user", "content": "The efficient tier failed."},+ ]+ assert rewritten["temperature"] == 0.2+ assert rewritten["tools"] == [{"type": "function", "function": {"name": "Bash"}}]+ assert "request_patch" not in rewritten["metadata"]["routing_plugin_signals"]["switchyard"]+ assert "request_patch" in context.signals["switchyard"]+++def test_request_patch_maps_supported_switchyard_request_overrides() -> None:+ original = {+ "model": "auto",+ "instructions": [],+ "messages": [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}],+ "tools": [],+ "tool_choice": None,+ "sampling": {"temperature": None, "top_p": None, "top_k": None},+ "output": {"max_output_tokens": None, "response_format": None},+ "reasoning": {"effort": None, "raw": None},+ "stream": False,+ "extensions": {"fields": {}},+ "preservation": {"requests": {}, "responses": {}},+ }+ rewritten = {+ **original,+ "model": SOL,+ "instructions": [+ {+ "role": "developer",+ "content": [{"type": "text", "text": "Use the selected tier."}],+ }+ ],+ "tools": [+ {+ "name": "search",+ "description": "Search documents",+ "parameters": {"type": "object", "properties": {}},+ "strict": True,+ }+ ],+ "tool_choice": {"type": "tool", "data": {"name": "search"}},+ "sampling": {"temperature": 0.3, "top_p": 0.8, "top_k": 20},+ "output": {+ "max_output_tokens": 512,+ "response_format": {"type": "json_object"},+ },+ "reasoning": {"effort": "high", "raw": None},+ "stream": True,+ "extensions": {+ "fields": {+ "parallel_tool_calls": False,+ "stop_sequences": ["END"],+ }+ },+ }++ assert build_request_patch(original, rewritten, selected_model_id=SOL) == {+ "set": {+ "messages": [+ {"role": "developer", "content": "Use the selected tier."},+ {"role": "user", "content": "Hello"},+ ],+ "tools": [+ {+ "type": "function",+ "function": {+ "name": "search",+ "description": "Search documents",+ "parameters": {"type": "object", "properties": {}},+ "strict": True,+ },+ }+ ],+ "tool_choice": {"type": "function", "function": {"name": "search"}},+ "temperature": 0.3,+ "top_p": 0.8,+ "top_k": 20,+ "max_completion_tokens": 512,+ "response_format": {"type": "json_object"},+ "reasoning_effort": "high",+ "stream": True,+ "parallel_tool_calls": False,+ "stop": ["END"],+ },+ "remove": ["max_tokens"],+ }+++ "update",+ [+ {"reasoning": {"effort": None, "raw": {"budget_tokens": 512}}},+ {"extensions": {"fields": {"api_key": "must-not-be-forwarded"}}},+ {"preservation": {"requests": {"openai_chat": {}}, "responses": {}}},+ ],+)+def test_request_patch_rejects_unsafe_or_unrepresentable_overrides(+ update: dict[str, object],+) -> None:+ original = {+ "model": "auto",+ "instructions": [],+ "messages": [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}],+ "tools": [],+ "tool_choice": None,+ "sampling": {"temperature": None, "top_p": None, "top_k": None},+ "output": {"max_output_tokens": None, "response_format": None},+ "reasoning": {"effort": None, "raw": None},+ "stream": False,+ "extensions": {"fields": {}},+ "preservation": {"requests": {}, "responses": {}},+ }+ rewritten = {**original, "model": SOL, **update}++ with pytest.raises(ValueError, match="cannot safely apply"):+ build_request_patch(original, rewritten, selected_model_id=SOL)+++class EmptyAlgorithm:+ """Algorithm-shaped test double whose stream violates the terminal-step contract."""++ async def run_stream(self, request: dict[str, object]) -> AsyncIterator[object]:+ if request:+ return+ yield object()+++async def test_algorithm_stream_without_terminal_outcome_fails_closed() -> None:+ plugin = SwitchyardRoutingPlugin(EmptyAlgorithm()) # type: ignore[arg-type]++ with pytest.raises(ValueError, match="without a routing outcome"):+ await plugin.run(routing_context([{"role": "user", "content": "hello"}]))diff --git a/examples/experimental/litellm/uv.lock b/examples/litellm/uv.locksimilarity index 97%rename from examples/experimental/litellm/uv.lockrename to examples/litellm/uv.lockindex 950a05de7..326c7e84d 100644--- a/examples/experimental/litellm/uv.lock+++ b/examples/litellm/uv.lock@@ -384,7 +384,7 @@ wheels = [[[package]]name = "litellm"-version = "1.92.0"+version = "1.97.0"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "aiohttp" },@@ -396,14 +396,20 @@ dependencies = [{ name = "jsonschema" },{ name = "openai" },{ name = "pydantic" },+ { name = "pydantic-settings" },{ name = "python-dotenv" },{ name = "tiktoken" },{ name = "tokenizers" },]-sdist = { url = "https://files.pythonhosted.org/packages/46/ea/5522be7e0dbcaa7c3fddfd2834f387c6e8eab47c0cc65f2a74657c815af7/litellm-1.92.0.tar.gz", hash = "sha256:773adf5503ee1793289689c899394a83df8122993760d9acd782e32aa798db9d", size = 15098143, upload-time = "2026-07-12T01:15:59.828Z" }+sdist = { url = "https://files.pythonhosted.org/packages/1a/08/db78a9f53e5688ad0f9e50d5c3bfe616e445aac120eafcca907f04634e48/litellm-1.97.0.tar.gz", hash = "sha256:6f7ce326a2e5385ef850e0b0768d41f502ec79278860090a838511cea067b067", size = 17762282, upload-time = "2026-08-16T00:05:44.724Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/cf/26/8061907b67aa04b7cdf2a41cbc4f8aebfbc722c909a7e4b2aea25def7053/litellm-1.92.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:9c7b2849591a35f7039dc7386a81a8e68fccb5ac2fecaa5122192c1172556b29", size = 19291180, upload-time = "2026-07-12T01:15:48.728Z" },- { url = "https://files.pythonhosted.org/packages/ba/6c/dc9b0b580ee8af9117abd729d1de25d74fae487a0029ca77049a09f3ad33/litellm-1.92.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:f17499155366afd1eaaeb6fd0c7b55cbd2239dcd72f2fe1f8071a969cc402fae", size = 19289559, upload-time = "2026-07-12T01:15:51.376Z" },+ { url = "https://files.pythonhosted.org/packages/35/d4/a04f8bda468fb25a0a8a392a6922c9ecf6c122ad9bf9e2f69ebd173e1cac/litellm-1.97.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ff401dd5d66f54b9b474f0652c419fb7bf883fbf5ca64c0bc363acdc98b758b5", size = 24235645, upload-time = "2026-08-16T00:05:20.055Z" },+ { url = "https://files.pythonhosted.org/packages/f1/51/a055bf5df38112f07970937d95416b22379ee84664b739bfe87a8292e098/litellm-1.97.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:2983b40ed5d8b1bcbbfbc0d66fefa21b04db91b87c05dd680ae77cba74e561ef", size = 23893493, upload-time = "2026-08-16T00:05:23.79Z" },+ { url = "https://files.pythonhosted.org/packages/14/28/b1cf429493aec5260ac2737dd82f6200c729fd26b336dc21cf001cfbcd8a/litellm-1.97.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e3a1f70d693716b4e8a8108f0a464b0e2c555e274ddbb8ef89c4c59c80be14ed", size = 24031743, upload-time = "2026-08-16T00:05:27.138Z" },+ { url = "https://files.pythonhosted.org/packages/08/4a/22c49e8bd068bfdab0daba235cc2d2752d76d855ccb8f6dafe7dabf01ca4/litellm-1.97.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:5b56dce7df44a6a9e6caf5379de2578a8cb82831ceabd3d71cc99b370a1015e7", size = 24397362, upload-time = "2026-08-16T00:05:30.89Z" },+ { url = "https://files.pythonhosted.org/packages/0a/22/f60558230969ac7d860bd93aff6bf6a69bcd8a3ef4f35eea211174ef4e81/litellm-1.97.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:6b360ddc3162c2ed39b64d3f9957a7af70cd9c60cf71f7a4dfa355a0bf05bebc", size = 24107078, upload-time = "2026-08-16T00:05:34.463Z" },+ { url = "https://files.pythonhosted.org/packages/5d/e4/6c74ff4b188d9399a036d41e86c4c616d95ce5fd9d9b3c42646bd02f270e/litellm-1.97.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:3c4f1dd45e14127f2303769a7ae79482697823e329ae503513d014ceea4dd704", size = 24493640, upload-time = "2026-08-16T00:05:37.85Z" },+ { url = "https://files.pythonhosted.org/packages/8c/d1/1475c4ecdf43221ab8ee8d0fcc6b469c26d0d80913d0773e18de81cf18f1/litellm-1.97.0-cp310-abi3-win_amd64.whl", hash = "sha256:dce3377207234fc5c5b275a5e234ba056a5051fd178ae8e9a6aeb5d056f12095", size = 24289278, upload-time = "2026-08-16T00:05:41.441Z" },][[package]]@@ -455,7 +461,7 @@ wheels = [[[package]]name = "nemo-switchyard"version = "0.2.0"-source = { editable = "../../../" }+source = { editable = "../../" }[package.metadata]@@ -586,6 +592,20 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" },]+[[package]]+name = "pydantic-settings"+version = "2.15.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "pydantic" },+ { name = "python-dotenv" },+ { name = "typing-inspection" },+]+sdist = { url = "https://files.pythonhosted.org/packages/68/ca/31c57507b13119d7d3cfa1576dad2911a4861e3be07b579395f4e9d393f9/pydantic_settings-2.15.0.tar.gz", hash = "sha256:694b793e84f766ba76a90ebdefc01d0a9a045dab0382bee70393da93712ad117", size = 261253, upload-time = "2026-08-07T09:24:57.419Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/30/a4/2bffa9f8e804325a09867f0e9d30795c80ea9f8d62560bd1b6ad6220eb2f/pydantic_settings-2.15.0-py3-none-any.whl", hash = "sha256:0ba092c291c94baceb5eff768aa0d56400a457585bc0175925a5a5510303da42", size = 69413, upload-time = "2026-08-07T09:24:55.839Z" },+]+[[package]]name = "pygments"version = "2.20.0"@@ -704,18 +724,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" },]-[[package]]-name = "respx"-version = "0.23.1"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "httpx" },-]-sdist = { url = "https://files.pythonhosted.org/packages/43/98/4e55c9c486404ec12373708d015ebce157966965a5ebe7f28ff2c784d41b/respx-0.23.1.tar.gz", hash = "sha256:242dcc6ce6b5b9bf621f5870c82a63997e8e82bc7c947f9ffe272b8f3dd5a780", size = 29243, upload-time = "2026-04-08T14:37:16.008Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/1d/4a/221da6ca167db45693d8d26c7dc79ccfc978a440251bf6721c9aaf251ac0/respx-0.23.1-py2.py3-none-any.whl", hash = "sha256:b18004b029935384bccfa6d7d9d74b4ec9af73a081cc28600fffc0447f4b8c1a", size = 25557, upload-time = "2026-04-08T14:37:14.613Z" },-]-[[package]]name = "rpds-py"version = "2026.6.3"@@ -761,20 +769,18 @@ dependencies = [dev = [{ name = "pytest" },{ name = "pytest-asyncio" },- { name = "respx" },][package.metadata]requires-dist = [- { name = "litellm", specifier = "==1.92.0" },- { name = "nemo-switchyard", editable = "../../../" },+ { name = "litellm", specifier = "==1.97.0" },+ { name = "nemo-switchyard", editable = "../../" },][package.metadata.requires-dev]dev = [{ name = "pytest", specifier = ">=9,<10" },{ name = "pytest-asyncio", specifier = ">=1.3,<2" },- { name = "respx", specifier = ">=0.23,<1" },][[package]]