Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(readme): lead with value, algorithms, and benchmarks (#582)

itayzitbb07f99e999f
diff --git a/README.md b/README.md
index ded0df8f0..c6537b7e3 100644
--- a/README.md
+++ b/README.md
@@ -4,24 +4,51 @@
# Switchyard
-Switchyard is a Rust proxy and library for LLM traffic. It routes requests
-across providers, translates between OpenAI and Anthropic APIs, records
-operational metrics, and provides typed, composable routing algorithms.
+**Switchyard routes each LLM call to the cheapest model that can still do the job. Without changing a line of your agent.**
-**Why Switchyard?** Point a coding agent such as Claude Code or Codex at an
-open-source model. Switchyard translates between the OpenAI Chat, Anthropic
-Messages, and OpenAI Responses formats, so the agent keeps speaking its native
-API while the request is served by vLLM, NVIDIA NIM, Ollama, or any
-OpenAI-compatible endpoint. The same proxy can spread traffic across several
-models for A/B benchmarking, apply signal-driven stage routing, or run a custom
-algorithm you write yourself.
+**[Get started →](#get-started)**
-## Features
+```mermaid
+flowchart LR
+ A["Claude Code · Codex CLI<br/>OpenAI / Anthropic SDK clients"]
+ SY["Switchyard<br/>routing algorithm + protocol translation"]
+ E["Efficient model<br/>GLM, Qwen, your own vLLM"]
+ C["Capable model<br/>Opus, GPT, NVIDIA NIM"]
+
+ A -->|"unchanged native API"| SY
+ SY -->|"routine turns"| E
+ SY -->|"hard turns"| C
+```
+
+It has three modes:
+
+**1. A Rust proxy** — run it in front of the agent you already use:
+
+```bash
+cargo install --locked switchyard-server
+switchyard-server --config routes.toml --port 4000
+```
+
+**2. An embeddable Rust library** — call the same algorithms from a gateway you already own:
+
+```bash
+cargo add --git https://github.com/NVIDIA-NeMo/Switchyard.git --tag v0.2.0 \
+ switchyard-libsy switchyard-protocol
+```
+
+**3. A NeMo Relay plugin** — load the same `routes.toml` into a NeMo Relay deployment you already run:
-- **Protocol Translation**: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
-- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm
-- **Operational Metrics**: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead
-- **NeMo Relay Plugin**: run Switchyard-configured routes in NeMo Relay—including any routing algorithm supported by `switchyard-runner`—while Switchyard owns provider HTTP dispatch
+```toml
+[[plugins.dynamic]]
+manifest = "./plugins/switchyard/relay-plugin.toml"
+
+[plugins.dynamic.config]
+switchyard_config_path = "/etc/switchyard/routes.toml"
+```
+
+Relay runs any routing algorithm `switchyard-runner` supports while Switchyard owns provider HTTP dispatch. See [`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md).
+
+Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at it. Every request keeps its native API format; Switchyard decides per turn which model serves it.
## Maturity
@@ -35,27 +62,67 @@ Switchyard is pre-alpha software that is evolving rapidly. The API and algorithm
> - switchyard-runner: Alpha. Evolving rapidly.
> - switchyard-server: Demo server, not for production use.
-## Quick Start
+## Why Switchyard
+
+Every Switchyard route below pairs the same Opus 4.8 capable tier with a cheaper
+efficient tier. The baseline is Opus 4.8 serving every turn.
+
+![Accuracy versus total cost on Terminal-Bench 2.1. Switchyard's staged, escalation, and classifier routes reach 71-76% accuracy for 13-30% less than the Opus 4.8 baseline, while single fixed models stay below 56%.](assets/benchmark-accuracy-vs-cost.svg)
+
+| Configuration | Accuracy | Total cost | vs. Opus 4.8 baseline |
+|---|---:|---:|---|
+| Opus 4.8 baseline | 76.0% | $98.06 | — |
+| **[Escalation](#routing-algorithms)** | 75.7% | $85.00 | 99.6% of accuracy, 13.3% cheaper |
+| **[Stage](#routing-algorithms)** | 72.7% | $68.19 | 95.7% of accuracy, 30.5% cheaper |
+| **[Capability](#routing-algorithms)** | 71.2% | $79.32 | 93.7% of accuracy, 19.1% cheaper |
+| Kimi K2.6 alone | 55.8% | $76.28 | |
+| GLM 5.2 alone | 52.4% | $16.47 | |
+| DeepSeek V4 Pro alone | 48.7% | $96.92 | |
+| Ultra 3 alone | 39.0% | $29.66 | |
-Choose the server path to run Switchyard as a standalone proxy. Choose the library path to embed
-routing in your own Rust application.
+## Get Started
-### Server Path
+Install [Rust with Cargo](https://rust-lang.org/tools/install/), then:
-Use this path to install and run the standalone Rust proxy. Install
-[Rust with Cargo](https://rust-lang.org/tools/install/), then install the
-published binary:
+**1. Install the server.**
```bash
cargo install --locked switchyard-server
-switchyard-server --help
```
-Cargo builds the release binary and installs it into `~/.cargo/bin` by default.
+**2. Write `routes.toml`.** A stage router over the same model pair as the
+benchmark above: how to reach a provider, which models to use, how to choose
+between them. `--config` takes any path; this writes it to the current directory.
-Create `routes.toml` using the
-[Getting Started guide](docs/getting_started.md#server-path), then validate it
-and start the server:
+```bash
+cat > routes.toml <<'TOML'
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.capable]
+id = "anthropic/claude-opus-4.8"
+llm_client = "openrouter"
+
+[targets.efficient]
+id = "z-ai/glm-5.2"
+llm_client = "openrouter"
+
+[routes.switchyard]
+id = "switchyard"
+type = "stage_router"
+capable_target = "capable"
+efficient_target = "efficient"
+picker = "efficient_first"
+confidence_threshold = 0.5
+TOML
+```
+
+**3. Run it.** `--dry-run` loads the config, prints `server OK:` and the model
+IDs it exposes, then exits without starting the server.
```bash
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
@@ -63,22 +130,39 @@ switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
```
-Verify the proxy in another terminal:
+**4. Send a request.** The route's `id` is the model name clients ask for.
```bash
-curl http://localhost:4000/health
+curl http://localhost:4000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'
```
-For a complete configuration and a test request, follow
-[Getting Started](docs/getting_started.md).
+The same route also answers on `/v1/messages` (Anthropic Messages) and
+`/v1/responses` (OpenAI Responses). `/v1/stats` reports which target served
+what, and `/metrics` exposes Prometheus counters for requests, errors, latency,
+tokens, and routing overhead.
+
+### Point a Coding Agent at It
+
+```bash
+export ANTHROPIC_BASE_URL="http://localhost:4000"
+export ANTHROPIC_MODEL="switchyard"
+claude
+```
+
+Codex CLI and other OpenAI clients use the OpenAI variables instead:
+
+```bash
+export OPENAI_BASE_URL="http://localhost:4000/v1"
+```
-### Library Path
+### Embed It in Your Own Gateway
-`switchyard-libsy` embeds the routing algorithms in your own Rust application.
-It never calls a model itself: an algorithm decides which target to use and
-hands every model call back to you, so it drops into an existing proxy, gateway,
-or agent runtime without owning an HTTP stack. Pair it with
-`switchyard-llm-client` when you want the calls made for you.
+`switchyard-libsy` never calls a model: an algorithm returns the target it chose
+and hands the call back to you, so it drops into a proxy, gateway, or agent
+runtime you already run. Pair it with `switchyard-llm-client` to have the calls
+made for you.
```toml
[dependencies]
@@ -86,52 +170,57 @@ switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" }
```
-See [Getting Started](docs/getting_started.md#library-path) for setup and the
-algorithm list, or the [`switchyard-libsy`](crates/libsy/README.md) crate docs.
+See [Getting Started](docs/getting_started.md#library-path) for setup, or the
+[`switchyard-libsy`](crates/libsy/README.md) crate docs.
-## Routing Strategies
+## Routing Algorithms
-| Strategy | Use it when | Route `type` |
-|---|---|---|
-| [LLM Classifier](docs/routing_algorithms/llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm_classifier` |
-| [Stage Router](docs/routing_algorithms/stage_router_routing.md) | Signals already in the conversation, such as tool results and errors, should route most turns without an extra model call. | `stage_router` |
-| [Escalation Router](docs/routing_algorithms/escalation_router_routing.md) | Every turn runs on the weak tier first, and a judge reads that answer to decide whether to send the same request to the strong tier. | `llm_classifier` with `mode = "escalation"` |
-| [Composite](docs/routing_algorithms/composite_routing.md) | Routing algorithms are composed, one setting the configuration of another before handing off. Today an LLM classifier sets the tier a stage router falls open to. | `composite` |
-| [Random](docs/routing_algorithms/random_routing.md) | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | `random` |
+Most use an LLM as a judge. All of them pick between an **efficient** model and a
+**capable** one; what differs is when the decision is made and how.
+| Algorithm | How it decides | Route `type` | Benchmark |
+|---|---|---|---|
+| **[Capability](docs/routing_algorithms/llm_classifier_routing.md)** | The first request is judged by an LLM. | `llm_classifier` | 71.2% at $79.32 |
+| **[Stage](docs/routing_algorithms/stage_router_routing.md)** | Tool responses are judged by pattern matching or an LLM. | `stage_router` | 72.7% at $68.19 |
+| **[Capability + Stage](docs/routing_algorithms/composite_routing.md)** | Combines the two above. | `composite` | not yet benchmarked |
+| **[Escalation](docs/routing_algorithms/escalation_router_routing.md)** | Starts efficient. Responses are judged by an LLM for issues, then escalated. | `llm_classifier` + `mode = "escalation"` | 75.7% at $85.00 |
+| **[Advisor Gate](docs/routing_algorithms/advisor_gate_routing.md)** | One model serves every turn; a stronger advisor approves its plans and "done" claims, or sends it back. | `advisor` | lifts a weak executor 43.8% → 54.7% |
+| **[Sub-Agent-Aware](docs/routing_algorithms/subagent_routing.md)** | Delegated sub-agent traffic routes separately from the parent agent. | `subagents` on `passthrough` or `stage_router` | not yet benchmarked |
+| **[Custom](docs/routing_algorithms/llm_classifier_routing.md#custom-multi-target-routing)** | The first request is judged by an LLM against criteria you define, routing among 2+ of your own models. | `llm_classifier` + `target_selector` policy | not yet benchmarked |
+| **[Random](docs/routing_algorithms/random_routing.md)** | Each request is routed at random, uniform or weighted. | `random` | baseline mechanism |
+
+Benchmarks are Terminal-Bench 2.1 against a $98.06 Opus 4.8 baseline at 76.0%.
A `passthrough` route registers one target under one model ID with no routing
decision. See the [Routing Overview](docs/routing_algorithms/overview.md) for
the common route shape and self-hosted targets.
-## Architecture
-
-```mermaid
-flowchart LR
- clients["Clients"]
- switchyard["Switchyard<br/>routing · translation · fallback"]
- backends["Model backends"]
-
- clients -->|"OpenAI / Anthropic API"| switchyard
- switchyard -->|"provider-native format"| backends
-```
-
-Clients keep their native OpenAI or Anthropic API format. Switchyard picks a
-configured backend, forwards the request in that backend's own format, and
-translates the response back into the shape the client expects. The server
-accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each
-configured LLM client selects one upstream format.
-
## Documentation
- **[Getting Started](docs/getting_started.md)**: complete standalone server walkthrough
- **[Core Concepts](docs/core_concepts.md)**: LLM clients, targets, routes, model IDs, and routing algorithms
- **[Routing Overview](docs/routing_algorithms/overview.md)**: choose and configure a routing algorithm
+- **[TOML Schema](docs/reference/toml_schema.md)**: every configuration key
+- **[Architecture](docs/architecture.md)**: how the proxy and library components fit together
- **[`switchyard-server`](crates/switchyard-server/README.md)**: server configuration, routing algorithms, and metrics
- **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application
- **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types
- **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation
- **[`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md)**: install Switchyard as a native NeMo Relay plugin
+## Benchmark Provenance
+
+The numbers in [Why Switchyard](#why-switchyard) are the v0.2.0 Terminal-Bench 2.1
+results from [Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/).
+Those runs used NVIDIA-internal inference endpoints, so absolute solve rates may
+shift on another serving stack; the routing parameters are the ones that ran.
+
+The escalation deployment is checked in at
+[`benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml`](benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml),
+with OpenRouter targets substituted so it is publicly runnable. To run the
+harness, see [`benchmark/README.md`](benchmark/README.md); for latency and
+routing overhead rather than task success, see
+[Soak Testing](docs/operations/soak_test.md).
+
## Community
- **Issues**: [GitHub Issues](https://github.com/NVIDIA-NeMo/Switchyard/issues)
diff --git a/assets/benchmark-accuracy-vs-cost.svg b/assets/benchmark-accuracy-vs-cost.svg
new file mode 100644
index 000000000..08d7ccd4d
--- /dev/null
+++ b/assets/benchmark-accuracy-vs-cost.svg
@@ -0,0 +1,94 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 920 580" width="920" height="580" role="img" aria-label="Accuracy versus total cost on Terminal-Bench 2.1. Switchyard routes reach 71 to 76 percent accuracy for 13 to 30 percent less than the Opus 4.8 baseline, while single fixed models stay below 56 percent.">
+<style>
+svg {
+ --sy: #2a78d6; --base: #eb6834; --ref: #898781;
+ --ink: #0b0b0b; --ink2: #52514e; --muted: #898781;
+ --grid: #e1e0d9; --axis: #c3c2b7; --ring: #ffffff;
+ font-family: system-ui, -apple-system, "Segoe UI", Helvetica, Arial, sans-serif;
+}
+@media (prefers-color-scheme: dark) {
+ svg {
+ --sy: #3987e5; --base: #d95926; --ref: #898781;
+ --ink: #ffffff; --ink2: #c3c2b7; --muted: #898781;
+ --grid: #2c2c2a; --axis: #383835; --ring: #0d1117;
+ }
+}
+.t-ink { fill: var(--ink); }
+.t-ink2 { fill: var(--ink2); }
+.t-muted { fill: var(--muted); }
+.t-sy { fill: var(--sy); }
+.t-base { fill: var(--base); }
+.t-ref { fill: var(--ref); }
+.m-sy { fill: var(--sy); }
+.m-base { fill: var(--base); }
+.m-ref { fill: var(--ref); }
+.marker { stroke: var(--ring); stroke-width: 2; }
+.grid { stroke: var(--grid); stroke-width: 1; }
+.axis { stroke: var(--axis); stroke-width: 1; }
+.frontier {
+ fill: none; stroke: var(--sy); stroke-width: 2; stroke-opacity: 0.32;
+ stroke-linecap: round; stroke-linejoin: round;
+}
+.num { font-variant-numeric: tabular-nums; }
+</style>
+<text x="48" y="34" class="t-ink" font-size="21" font-weight="700">Accuracy vs. total cost</text>
+<text x="48" y="55" class="t-muted" font-size="12.5">Terminal-Bench 2.1 · Claude Code harness · k=3 runs</text>
+<text x="48" y="77" class="t-sy" font-size="13" font-weight="650">Staged routing holds 95.7% of Opus accuracy at 30.5% lower cost</text>
+<line x1="74" y1="440.9" x2="890" y2="440.9" class="grid"/>
+<text x="62" y="444.9" class="t-muted num" font-size="11.5" text-anchor="end">40%</text>
+<line x1="74" y1="354.7" x2="890" y2="354.7" class="grid"/>
+<text x="62" y="358.7" class="t-muted num" font-size="11.5" text-anchor="end">50%</text>
+<line x1="74" y1="268.4" x2="890" y2="268.4" class="grid"/>
+<text x="62" y="272.4" class="t-muted num" font-size="11.5" text-anchor="end">60%</text>
+<line x1="74" y1="182.2" x2="890" y2="182.2" class="grid"/>
+<text x="62" y="186.2" class="t-muted num" font-size="11.5" text-anchor="end">70%</text>
+<line x1="74" y1="96.0" x2="890" y2="96.0" class="grid"/>
+<text x="62" y="100.0" class="t-muted num" font-size="11.5" text-anchor="end">80%</text>
+<line x1="74.0" y1="96" x2="74.0" y2="484" class="grid"/>
+<text x="74.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$0</text>
+<line x1="210.0" y1="96" x2="210.0" y2="484" class="grid"/>
+<text x="210.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$20</text>
+<line x1="346.0" y1="96" x2="346.0" y2="484" class="grid"/>
+<text x="346.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$40</text>
+<line x1="482.0" y1="96" x2="482.0" y2="484" class="grid"/>
+<text x="482.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$60</text>
+<line x1="618.0" y1="96" x2="618.0" y2="484" class="grid"/>
+<text x="618.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$80</text>
+<line x1="754.0" y1="96" x2="754.0" y2="484" class="grid"/>
+<text x="754.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$100</text>
+<line x1="890.0" y1="96" x2="890.0" y2="484" class="grid"/>
+<text x="890.0" y="506" class="t-muted num" font-size="11.5" text-anchor="middle">$120</text>
+<line x1="74" y1="484" x2="890" y2="484" class="axis"/>
+<line x1="74" y1="96" x2="74" y2="484" class="axis"/>
+<text x="482.0" y="529" class="t-muted" font-size="11.5" text-anchor="middle">Total cost across benchmark runs · cheaper &#8592;</text>
+<text transform="translate(20,290.0) rotate(-90)" class="t-muted" font-size="11.5" text-anchor="middle">Overall accuracy · better &#8594;</text>
+<polyline points="186.0,334.0 537.7,158.9 652.0,133.1 740.8,130.5" class="frontier"/>
+<circle cx="186.0" cy="334.0" r="5.5" class="m-ref marker"/>
+<circle cx="275.7" cy="449.5" r="5.5" class="m-ref marker"/>
+<circle cx="537.7" cy="158.9" r="7.5" class="m-sy marker"/>
+<circle cx="592.7" cy="304.7" r="5.5" class="m-ref marker"/>
+<circle cx="613.4" cy="171.9" r="7.5" class="m-sy marker"/>
+<circle cx="652.0" cy="133.1" r="7.5" class="m-sy marker"/>
+<circle cx="733.1" cy="365.9" r="5.5" class="m-ref marker"/>
+<circle cx="740.8" cy="130.5" r="7.5" class="m-base marker"/>
+<text x="186.0" y="320.0" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">GLM 5.2</text>
+<text x="275.7" y="435.5" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">Ultra 3</text>
+<text x="525.7" y="146.9" class="t-sy" font-size="12.5" font-weight="700" text-anchor="end">SY: Staged</text>
+<text x="525.7" y="162.9" class="t-ink2 num" font-size="11.5" text-anchor="end">72.7% · $68.19</text>
+<text x="592.7" y="290.7" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">Kimi K2.6</text>
+<text x="613.4" y="205.9" class="t-sy" font-size="12.5" font-weight="700" text-anchor="middle">SY: Classifier</text>
+<text x="613.4" y="221.9" class="t-ink2 num" font-size="11.5" text-anchor="middle">71.2% · $79.32</text>
+<text x="652.0" y="113.1" class="t-sy" font-size="12.5" font-weight="700" text-anchor="middle">SY: Escalation</text>
+<text x="652.0" y="129.1" class="t-ink2 num" font-size="11.5" text-anchor="middle">75.7% · $85.00</text>
+<text x="733.1" y="387.9" class="t-ink2" font-size="12.5" font-weight="600" text-anchor="middle">DeepSeek V4 Pro</text>
+<text x="754.8" y="136.5" class="t-base" font-size="12.5" font-weight="700" text-anchor="start">Opus 4.8 baseline</text>
+<text x="754.8" y="152.5" class="t-ink2 num" font-size="11.5" text-anchor="start">76.0% · $98.06</text>
+<circle cx="54" cy="550" r="5.5" class="m-sy"/>
+<text x="66" y="554" class="t-ink2" font-size="12">Switchyard route</text>
+<circle cx="186.4" cy="550" r="5.5" class="m-base"/>
+<text x="198.4" y="554" class="t-ink2" font-size="12">Opus 4.8 baseline</text>
+<circle cx="325.70000000000005" cy="550" r="5.5" class="m-ref"/>
+<text x="337.70000000000005" y="554" class="t-ink2" font-size="12">Single fixed model</text>
+<line x1="466.90000000000003" y1="550" x2="486.90000000000003" y2="550" class="frontier"/>
+<text x="492.90000000000003" y="554" class="t-ink2" font-size="12">cost&#8211;quality frontier</text>
+</svg>