Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(readme): lead with value, algorithms, and benchmarks (#582)
itayzitbb07f99e999f
diff --git a/README.md b/README.mdindex ded0df8f0..c6537b7e3 100644--- a/README.md+++ b/README.md@@ -4,24 +4,51 @@# Switchyard-Switchyard is a Rust proxy and library for LLM traffic. It routes requests-across providers, translates between OpenAI and Anthropic APIs, records-operational metrics, and provides typed, composable routing algorithms.+**Switchyard routes each LLM call to the cheapest model that can still do the job. Without changing a line of your agent.**-**Why Switchyard?** Point a coding agent such as Claude Code or Codex at an-open-source model. Switchyard translates between the OpenAI Chat, Anthropic-Messages, and OpenAI Responses formats, so the agent keeps speaking its native-API while the request is served by vLLM, NVIDIA NIM, Ollama, or any-OpenAI-compatible endpoint. The same proxy can spread traffic across several-models for A/B benchmarking, apply signal-driven stage routing, or run a custom-algorithm you write yourself.+**[Get started →](#get-started)**-## Features+```mermaid+flowchart LR+ A["Claude Code · Codex CLI<br/>OpenAI / Anthropic SDK clients"]+ SY["Switchyard<br/>routing algorithm + protocol translation"]+ E["Efficient model<br/>GLM, Qwen, your own vLLM"]+ C["Capable model<br/>Opus, GPT, NVIDIA NIM"]++ A -->|"unchanged native API"| SY+ SY -->|"routine turns"| E+ SY -->|"hard turns"| C+```++It has three modes:++**1. A Rust proxy** — run it in front of the agent you already use:++```bash+cargo install --locked switchyard-server+switchyard-server --config routes.toml --port 4000+```++**2. An embeddable Rust library** — call the same algorithms from a gateway you already own:++```bash+cargo add --git https://github.com/NVIDIA-NeMo/Switchyard.git --tag v0.2.0 \+ switchyard-libsy switchyard-protocol+```++**3. A NeMo Relay plugin** — load the same `routes.toml` into a NeMo Relay deployment you already run:-- **Protocol Translation**: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats-- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm-- **Operational Metrics**: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead-- **NeMo Relay Plugin**: run Switchyard-configured routes in NeMo Relay—including any routing algorithm supported by `switchyard-runner`—while Switchyard owns provider HTTP dispatch+```toml+[[plugins.dynamic]]+manifest = "./plugins/switchyard/relay-plugin.toml"++[plugins.dynamic.config]+switchyard_config_path = "/etc/switchyard/routes.toml"+```++Relay runs any routing algorithm `switchyard-runner` supports while Switchyard owns provider HTTP dispatch. See [`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md).++Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at it. Every request keeps its native API format; Switchyard decides per turn which model serves it.## Maturity@@ -35,27 +62,67 @@ Switchyard is pre-alpha software that is evolving rapidly. The API and algorithm> - switchyard-runner: Alpha. Evolving rapidly.> - switchyard-server: Demo server, not for production use.-## Quick Start+## Why Switchyard++Every Switchyard route below pairs the same Opus 4.8 capable tier with a cheaper+efficient tier. The baseline is Opus 4.8 serving every turn.++++| Configuration | Accuracy | Total cost | vs. Opus 4.8 baseline |+|---|---:|---:|---|+| Opus 4.8 baseline | 76.0% | $98.06 | — |+| **[Escalation](#routing-algorithms)** | 75.7% | $85.00 | 99.6% of accuracy, 13.3% cheaper |+| **[Stage](#routing-algorithms)** | 72.7% | $68.19 | 95.7% of accuracy, 30.5% cheaper |+| **[Capability](#routing-algorithms)** | 71.2% | $79.32 | 93.7% of accuracy, 19.1% cheaper |+| Kimi K2.6 alone | 55.8% | $76.28 | |+| GLM 5.2 alone | 52.4% | $16.47 | |+| DeepSeek V4 Pro alone | 48.7% | $96.92 | |+| Ultra 3 alone | 39.0% | $29.66 | |-Choose the server path to run Switchyard as a standalone proxy. Choose the library path to embed-routing in your own Rust application.+## Get Started-### Server Path+Install [Rust with Cargo](https://rust-lang.org/tools/install/), then:-Use this path to install and run the standalone Rust proxy. Install-[Rust with Cargo](https://rust-lang.org/tools/install/), then install the-published binary:+**1. Install the server.**```bashcargo install --locked switchyard-server-switchyard-server --help```-Cargo builds the release binary and installs it into `~/.cargo/bin` by default.+**2. Write `routes.toml`.** A stage router over the same model pair as the+benchmark above: how to reach a provider, which models to use, how to choose+between them. `--config` takes any path; this writes it to the current directory.-Create `routes.toml` using the-[Getting Started guide](docs/getting_started.md#server-path), then validate it-and start the server:+```bash+cat > routes.toml <<'TOML'+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.capable]+id = "anthropic/claude-opus-4.8"+llm_client = "openrouter"++[targets.efficient]+id = "z-ai/glm-5.2"+llm_client = "openrouter"++[routes.switchyard]+id = "switchyard"+type = "stage_router"+capable_target = "capable"+efficient_target = "efficient"+picker = "efficient_first"+confidence_threshold = 0.5+TOML+```++**3. Run it.** `--dry-run` loads the config, prints `server OK:` and the model+IDs it exposes, then exits without starting the server.```bashexport OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret@@ -63,22 +130,39 @@ switchyard-server --config routes.toml --dry-runswitchyard-server --config routes.toml --host 127.0.0.1 --port 4000```-Verify the proxy in another terminal:+**4. Send a request.** The route's `id` is the model name clients ask for.```bash-curl http://localhost:4000/health+curl http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'```-For a complete configuration and a test request, follow-[Getting Started](docs/getting_started.md).+The same route also answers on `/v1/messages` (Anthropic Messages) and+`/v1/responses` (OpenAI Responses). `/v1/stats` reports which target served+what, and `/metrics` exposes Prometheus counters for requests, errors, latency,+tokens, and routing overhead.++### Point a Coding Agent at It++```bash+export ANTHROPIC_BASE_URL="http://localhost:4000"+export ANTHROPIC_MODEL="switchyard"+claude+```++Codex CLI and other OpenAI clients use the OpenAI variables instead:++```bash+export OPENAI_BASE_URL="http://localhost:4000/v1"+```-### Library Path+### Embed It in Your Own Gateway-`switchyard-libsy` embeds the routing algorithms in your own Rust application.-It never calls a model itself: an algorithm decides which target to use and-hands every model call back to you, so it drops into an existing proxy, gateway,-or agent runtime without owning an HTTP stack. Pair it with-`switchyard-llm-client` when you want the calls made for you.+`switchyard-libsy` never calls a model: an algorithm returns the target it chose+and hands the call back to you, so it drops into a proxy, gateway, or agent+runtime you already run. Pair it with `switchyard-llm-client` to have the calls+made for you.```toml[dependencies]@@ -86,52 +170,57 @@ switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tagswitchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" }```-See [Getting Started](docs/getting_started.md#library-path) for setup and the-algorithm list, or the [`switchyard-libsy`](crates/libsy/README.md) crate docs.+See [Getting Started](docs/getting_started.md#library-path) for setup, or the+[`switchyard-libsy`](crates/libsy/README.md) crate docs.-## Routing Strategies+## Routing Algorithms-| Strategy | Use it when | Route `type` |-|---|---|---|-| [LLM Classifier](docs/routing_algorithms/llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm_classifier` |-| [Stage Router](docs/routing_algorithms/stage_router_routing.md) | Signals already in the conversation, such as tool results and errors, should route most turns without an extra model call. | `stage_router` |-| [Escalation Router](docs/routing_algorithms/escalation_router_routing.md) | Every turn runs on the weak tier first, and a judge reads that answer to decide whether to send the same request to the strong tier. | `llm_classifier` with `mode = "escalation"` |-| [Composite](docs/routing_algorithms/composite_routing.md) | Routing algorithms are composed, one setting the configuration of another before handing off. Today an LLM classifier sets the tier a stage router falls open to. | `composite` |-| [Random](docs/routing_algorithms/random_routing.md) | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | `random` |+Most use an LLM as a judge. All of them pick between an **efficient** model and a+**capable** one; what differs is when the decision is made and how.+| Algorithm | How it decides | Route `type` | Benchmark |+|---|---|---|---|+| **[Capability](docs/routing_algorithms/llm_classifier_routing.md)** | The first request is judged by an LLM. | `llm_classifier` | 71.2% at $79.32 |+| **[Stage](docs/routing_algorithms/stage_router_routing.md)** | Tool responses are judged by pattern matching or an LLM. | `stage_router` | 72.7% at $68.19 |+| **[Capability + Stage](docs/routing_algorithms/composite_routing.md)** | Combines the two above. | `composite` | not yet benchmarked |+| **[Escalation](docs/routing_algorithms/escalation_router_routing.md)** | Starts efficient. Responses are judged by an LLM for issues, then escalated. | `llm_classifier` + `mode = "escalation"` | 75.7% at $85.00 |+| **[Advisor Gate](docs/routing_algorithms/advisor_gate_routing.md)** | One model serves every turn; a stronger advisor approves its plans and "done" claims, or sends it back. | `advisor` | lifts a weak executor 43.8% → 54.7% |+| **[Sub-Agent-Aware](docs/routing_algorithms/subagent_routing.md)** | Delegated sub-agent traffic routes separately from the parent agent. | `subagents` on `passthrough` or `stage_router` | not yet benchmarked |+| **[Custom](docs/routing_algorithms/llm_classifier_routing.md#custom-multi-target-routing)** | The first request is judged by an LLM against criteria you define, routing among 2+ of your own models. | `llm_classifier` + `target_selector` policy | not yet benchmarked |+| **[Random](docs/routing_algorithms/random_routing.md)** | Each request is routed at random, uniform or weighted. | `random` | baseline mechanism |++Benchmarks are Terminal-Bench 2.1 against a $98.06 Opus 4.8 baseline at 76.0%.A `passthrough` route registers one target under one model ID with no routingdecision. See the [Routing Overview](docs/routing_algorithms/overview.md) forthe common route shape and self-hosted targets.-## Architecture--```mermaid-flowchart LR- clients["Clients"]- switchyard["Switchyard<br/>routing · translation · fallback"]- backends["Model backends"]-- clients -->|"OpenAI / Anthropic API"| switchyard- switchyard -->|"provider-native format"| backends-```--Clients keep their native OpenAI or Anthropic API format. Switchyard picks a-configured backend, forwards the request in that backend's own format, and-translates the response back into the shape the client expects. The server-accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each-configured LLM client selects one upstream format.-## Documentation- **[Getting Started](docs/getting_started.md)**: complete standalone server walkthrough- **[Core Concepts](docs/core_concepts.md)**: LLM clients, targets, routes, model IDs, and routing algorithms- **[Routing Overview](docs/routing_algorithms/overview.md)**: choose and configure a routing algorithm+- **[TOML Schema](docs/reference/toml_schema.md)**: every configuration key+- **[Architecture](docs/architecture.md)**: how the proxy and library components fit together- **[`switchyard-server`](crates/switchyard-server/README.md)**: server configuration, routing algorithms, and metrics- **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application- **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types- **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation- **[`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md)**: install Switchyard as a native NeMo Relay plugin+## Benchmark Provenance++The numbers in [Why Switchyard](#why-switchyard) are the v0.2.0 Terminal-Bench 2.1+results from [Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/).+Those runs used NVIDIA-internal inference endpoints, so absolute solve rates may+shift on another serving stack; the routing parameters are the ones that ran.++The escalation deployment is checked in at+[`benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml`](benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml),+with OpenRouter targets substituted so it is publicly runnable. To run the+harness, see [`benchmark/README.md`](benchmark/README.md); for latency and+routing overhead rather than task success, see+[Soak Testing](docs/operations/soak_test.md).+## Community- **Issues**: [GitHub Issues](https://github.com/NVIDIA-NeMo/Switchyard/issues)diff --git a/assets/benchmark-accuracy-vs-cost.svg b/assets/benchmark-accuracy-vs-cost.svgnew file mode 100644index 000000000..08d7ccd4d--- /dev/null+++ b/assets/benchmark-accuracy-vs-cost.svg@@ -0,0 +1,94 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 920 580" width="920" height="580" role="img" aria-label="Accuracy versus total cost on Terminal-Bench 2.1. Switchyard routes reach 71 to 76 percent accuracy for 13 to 30 percent less than the Opus 4.8 baseline, while single fixed models stay below 56 percent.">+<style>+svg {+ --sy: #2a78d6; --base: #eb6834; --ref: #898781;+ --ink: #0b0b0b; --ink2: #52514e; --muted: #898781;+ --grid: #e1e0d9; --axis: #c3c2b7; --ring: #ffffff;+ font-family: system-ui, -apple-system, "Segoe UI", Helvetica, Arial, sans-serif;+}+@media (prefers-color-scheme: dark) {+ svg {+ --sy: #3987e5; --base: #d95926; --ref: #898781;+ --ink: #ffffff; --ink2: #c3c2b7; --muted: #898781;+ --grid: #2c2c2a; --axis: #383835; --ring: #0d1117;+ }+}+.t-ink { fill: var(--ink); }+.t-ink2 { fill: var(--ink2); }+.t-muted { fill: var(--muted); }+.t-sy { fill: var(--sy); }+.t-base { fill: var(--base); }+.t-ref { fill: var(--ref); }+.m-sy { fill: var(--sy); }+.m-base { fill: var(--base); }+.m-ref { fill: var(--ref); }+.marker { stroke: var(--ring); stroke-width: 2; }+.grid { stroke: var(--grid); stroke-width: 1; }+.axis { stroke: var(--axis); stroke-width: 1; }+.frontier {+ fill: none; stroke: var(--sy); stroke-width: 2; stroke-opacity: 0.32;+ stroke-linecap: round; stroke-linejoin: round;+}+.num { font-variant-numeric: tabular-nums; }+</style>+<text x="48" y="34" class="t-ink" font-size="21" font-weight="700">Accuracy vs. total cost</text>+<text x="48" y="55" class="t-muted" font-size="12.5">Terminal-Bench 2.1 · Claude Code harness · k=3 runs</text>+<text x="48" y="77" class="t-sy" font-size="13" font-weight="650">Staged routing holds 95.7% of Opus accuracy at 30.5% lower cost</text>+<line x1="74" y1="440.9" x2="890" y2="440.9" class="grid"/>+<text x="62" y="444.9" class="t-muted num" font-size="11.5" text-anchor="end">40%</text>+<line x1="74" y1="354.7" x2="890" y2="354.7" class="grid"/>+<text x="62" y="358.7" class="t-muted num" font-size="11.5" text-anchor="end">50%</text>+<line x1="74" y1="268.4" x2="890" y2="268.4" class="grid"/>+<text x="62" y="272.4" class="t-muted num" font-size="11.5" text-anchor="end">60%</text>+<line x1="74" y1="182.2" x2="890" y2="182.2" class="grid"/>+<text x="62" y="186.2" class="t-muted num" font-size="11.5" text-anchor="end">70%</text>+<line x1="74" y1="96.0" x2="890" y2="96.0" class="grid"/>+<text x="62" y="100.0" class="t-muted num" font-size="11.5" text-anchor="end">80%</text>+<line x1="74.0" y1="96" x2="74.0" y2="484" class="grid"/>+<text x="74.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$0</text>+<line x1="210.0" y1="96" x2="210.0" y2="484" class="grid"/>+<text x="210.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$20</text>+<line x1="346.0" y1="96" x2="346.0" y2="484" class="grid"/>+<text x="346.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$40</text>+<line x1="482.0" y1="96" x2="482.0" y2="484" class="grid"/>+<text x="482.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$60</text>+<line x1="618.0" y1="96" x2="618.0" y2="484" class="grid"/>+<text x="618.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$80</text>+<line x1="754.0" y1="96" x2="754.0" y2="484" class="grid"/>+<text x="754.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$100</text>+<line x1="890.0" y1="96" x2="890.0" y2="484" class="grid"/>+<text x="890.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$120</text>+<line x1="74" y1="484" x2="890" y2="484" class="axis"/>+<line x1="74" y1="96" x2="74" y2="484" class="axis"/>+<text x="482.0" y="529" class="t-muted" font-size="11.5" text-anchor="middle">Total cost across benchmark runs · cheaper ←</text>+<text transform="translate(20,290.0) rotate(-90)" class="t-muted" font-size="11.5" text-anchor="middle">Overall accuracy · better →</text>+<polyline points="186.0,334.0 537.7,158.9 652.0,133.1 740.8,130.5" class="frontier"/>+<circle cx="186.0" cy="334.0" r="5.5" class="m-ref marker"/>+<circle cx="275.7" cy="449.5" r="5.5" class="m-ref marker"/>+<circle cx="537.7" cy="158.9" r="7.5" class="m-sy marker"/>+<circle cx="592.7" cy="304.7" r="5.5" class="m-ref marker"/>+<circle cx="613.4" cy="171.9" r="7.5" class="m-sy marker"/>+<circle cx="652.0" cy="133.1" r="7.5" class="m-sy marker"/>+<circle cx="733.1" cy="365.9" r="5.5" class="m-ref marker"/>+<circle cx="740.8" cy="130.5" r="7.5" class="m-base marker"/>+<text x="186.0" y="320.0" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">GLM 5.2</text>+<text x="275.7" y="435.5" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">Ultra 3</text>+<text x="525.7" y="146.9" class="t-sy" font-size="12.5" font-weight="700" text-anchor="end">SY: Staged</text>+<text x="525.7" y="162.9" class="t-ink2 num" font-size="11.5" text-anchor="end">72.7% · $68.19</text>+<text x="592.7" y="290.7" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">Kimi K2.6</text>+<text x="613.4" y="205.9" class="t-sy" font-size="12.5" font-weight="700" text-anchor="middle">SY: Classifier</text>+<text x="613.4" y="221.9" class="t-ink2 num" font-size="11.5" text-anchor="middle">71.2% · $79.32</text>+<text x="652.0" y="113.1" class="t-sy" font-size="12.5" font-weight="700" text-anchor="middle">SY: Escalation</text>+<text x="652.0" y="129.1" class="t-ink2 num" font-size="11.5" text-anchor="middle">75.7% · $85.00</text>+<text x="733.1" y="387.9" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">DeepSeek V4 Pro</text>+<text x="754.8" y="136.5" class="t-base" font-size="12.5" font-weight="700" text-anchor="start">Opus 4.8 baseline</text>+<text x="754.8" y="152.5" class="t-ink2 num" font-size="11.5" text-anchor="start">76.0% · $98.06</text>+<circle cx="54" cy="550" r="5.5" class="m-sy"/>+<text x="66" y="554" class="t-ink2" font-size="12">Switchyard route</text>+<circle cx="186.4" cy="550" r="5.5" class="m-base"/>+<text x="198.4" y="554" class="t-ink2" font-size="12">Opus 4.8 baseline</text>+<circle cx="325.70000000000005" cy="550" r="5.5" class="m-ref"/>+<text x="337.70000000000005" y="554" class="t-ink2" font-size="12">Single fixed model</text>+<line x1="466.90000000000003" y1="550" x2="486.90000000000003" y2="550" class="frontier"/>+<text x="492.90000000000003" y="554" class="t-ink2" font-size="12">cost–quality frontier</text>+</svg>