Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(readme): trim Get Started to pointers and a minimal embed loop (#670)
itayzit651b7d504223
diff --git a/README.md b/README.mdindex 7d2b6e067..2740c8c85 100644--- a/README.md+++ b/README.md@@ -23,7 +23,7 @@ Switchyard runs inside gateways you may already have.- **NeMo Relay** — a native plugin. Load a `routes.toml` into a Relay deploymentyou already run. [Setup →](#path-1--load-the-nemo-relay-plugin)- **LiteLLM** — a routing plugin for LiteLLM's `Router` and proxy.- [`examples/litellm`](examples/litellm/README.md)+ [Setup →](examples/litellm/README.md)- **More integrations** coming soon.```mermaid@@ -64,16 +64,14 @@ flowchart LR### Run Switchyard as a standalone proxy-A server in front of an agent, when you have no gateway to put Switchyard in:--```bash-cargo install --locked switchyard-server-switchyard-server --config routes.toml --port 4000-```--Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at the proxy.+A server in front of an agent, when you have no gateway to put Switchyard in.+Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at it;Switchyard decides per turn which model serves it.+- Install: `cargo install --locked switchyard-server`+- Then follow [Path 3 — Run the Standalone Proxy](#path-3--run-the-standalone-proxy):+ write `routes.toml`, start the server, point your agent at it.+```mermaidflowchart LRP["Switchyard<br/>standalone proxy"]@@ -98,151 +96,68 @@ releases — pin the version you integrate.## Get Started-Three paths, in the same order as above. Each is self-contained: start at-step 1, stop when you reach the result named under the heading.+Three paths, in the same order as above. Using Claude Code or Codex? Point it+at this README and ask it to set up the path you want.### Path 1 — Load the NeMo Relay PluginYou finish with an existing NeMo Relay deployment routing through Switchyard.-Requires NeMo Relay `>=0.8.0, <1.0.0` and a Rust toolchain to build the plugin.+Requires NeMo Relay `>=0.8.0, <1.0.0` and a Rust toolchain.-**1. Build, package, and register the plugin.** Follow steps 1–3 of the-[install guide in the plugin README](crates/switchyard-nemo-relay-plugin/README.md#install).-They build the shared library, package it into a bundle with a digest-bearing-`relay-plugin.toml`, and register it with `nemo-relay plugins add`.--**2. Write the Switchyard deployment** to `/etc/switchyard/routes.toml` — the-same version-1 TOML the proxy uses. Copy the file from step 2 of Path 3 below.--**3. Point the plugin at the deployment.** Add a `config` table to the-`[[plugins.dynamic]]` entry that `nemo-relay plugins add` wrote, plus the-policy override that lets Relay load the unsigned bundle. Use exactly one-deployment source: a path, as here, or the config nested under-`switchyard_config`.--```toml-[[plugins.dynamic]]-manifest = "./plugins/switchyard/relay-plugin.toml"--[plugins.dynamic.config]-priority = 0-switchyard_config_path = "/etc/switchyard/routes.toml"--[plugins.policy.overrides."nvidia.switchyard"]-attestation = "integrity_only"-```--**4. Enable, validate, and restart Relay.**--```bash-nemo-relay plugins enable nvidia.switchyard-nemo-relay plugins validate nvidia.switchyard-```--Relay now runs any algorithm `switchyard-runner` supports, while Switchyard-owns provider HTTP dispatch.--Details: [`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md)-and the [TOML schema reference](docs/reference/toml_schema.md).+Follow the plugin README's+[Install](crates/switchyard-nemo-relay-plugin/README.md#install) and+[Configure Relay](crates/switchyard-nemo-relay-plugin/README.md#configure-relay)+sections. For the deployment file, use the `routes.toml` from+[Path 3, step 2](#path-3--run-the-standalone-proxy).### Path 2 — Embed the LibraryYou finish with your own harness picking a model per request and still makingevery model call itself. Shown in Python; the Rust API has the same shape.-**1. Install.** The `Step` and `LlmResponse` API below is newer than the-`nemo-switchyard` 0.2.0 release on PyPI, which exposes an older `LlmTarget`-based interface. Until the next release, build from source (requires a Rust-toolchain):+**1. Install.**```bashpip install git+https://github.com/NVIDIA-NeMo/Switchyard.git```-For Rust, the `v0.2.0` tag has the older `run_stream` shape too, so depend on-the repository's `main` branch and pin the `rev` you tested:+The API below is newer than `nemo-switchyard` 0.2.0 on PyPI, so install from+source until the next release. Rust: depend on `switchyard-libsy` and+`switchyard-protocol` from this repository instead. Pin both to the commit you+tested — `@<sha>` for pip, `rev = "<sha>"` for Cargo — before depending on them.-```toml-[dependencies]-async-trait = "0.1"-futures = "0.3"-switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", branch = "main" }-switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", branch = "main" }-tokio = { version = "1", features = ["macros", "rt"] }-```--**2. Construct an algorithm.** Models are supplied when each request runs. This-is the stage router from the benchmark; `random`, `llm_task_classifier`, and-`llm_classifier` are built the same way.+**2. Construct an algorithm.** It selects a category — `efficient` or+`capable` — and you map categories to model IDs when each request runs.```pythonfrom switchyard.libsy import LlmResponse, Stepfrom switchyard.libsy.algorithms import stage_router-algorithm = stage_router(- picker="efficient_first",- confidence_threshold=0.5,-)+algorithm = stage_router(picker="efficient_first", confidence_threshold=0.5)```-**3. Drive it.** `run_stream` takes a normalized Switchyard request dict, not-an OpenAI wire payload: the `Request` shape from-[`switchyard-protocol`](crates/protocol/README.md), with `messages` whose-`content` is a list of typed blocks. It yields steps. A `CallModel` step is a-classifier or judge call — make it with your own client and hand back the-normalized response wrapped in `LlmResponse.Agg`. `Done` carries the pick.+**3. Drive it.** `run_stream` yields steps. Serve each `CallModel` with your own+client — `call.models` is ordered by preference, and `call.fail(error)` reports+a failed call; `Done` carries the pick.```python-async def call_with_fallback(request: dict, models: list[str], clients: dict) -> LlmResponse.Agg:- error: Exception | None = None- for model in models:- try:- return LlmResponse.Agg(await clients[model].call({**request, "model": model}))- except Exception as exc:- error = exc- raise error or RuntimeError("no candidate models")---runtime_models = {- "efficient": ["fast"],- "capable": ["quality"],- "any": ["quality", "fast"],-}---async def route(request: dict, clients: dict) -> LlmResponse.Agg | LlmResponse.Stream:- async for step in algorithm.run_stream(request, runtime_models):- match step:- case Step.CallModel(call):- try:- call.respond(await call_with_fallback(call.request, call.models, clients))- except Exception as error:- call.fail(error)- case Step.Done(outcome):- if outcome.response is not None:- return outcome.response- return await call_with_fallback(- outcome.request, outcome.selected_model_ids, clients- )- raise RuntimeError("algorithm ended without a decision")+models = {"efficient": ["fast"], "capable": ["quality"], "any": ["quality", "fast"]}++async for step in algorithm.run_stream(request, models):+ match step:+ case Step.CallModel(call):+ call.respond(LlmResponse.Agg(await my_client(call.request, call.models[0])))+ case Step.Done(outcome):+ model, request = outcome.selected_model_ids[0], outcome.request```-`runtime_models` groups the model IDs available for this request by category.-`clients` maps each model ID to your existing client; each `call` takes a-normalized request dict and returns a normalized response dict. `call.models`-and `outcome.selected_model_ids` list candidates in order, so the helper tries-each one before giving up. `outcome.request` is the request to send, which may-carry a rewrite the algorithm applied. When `outcome.response` is set, routing-already produced the answer and no further call is needed.--**4. Make the answer call** with your own HTTP client, retries, and-credentials, as `call_with_fallback` does above. A complete runnable version,-including streaming responses, is in [`examples/libsy.py`](examples/libsy.py).+**4. Make the answer call** with `model` and `request`, using your own HTTP+client, retries, and credentials.-Type reference: [`switchyard-libsy`](crates/libsy/README.md) and-[`switchyard-protocol`](crates/protocol/README.md). In Rust the loop is-`Algorithm::run_stream` yielding `Step::CallModel` and `Step::Done`, with-`switchyard-llm-client`'s `run` available to drive it for you.+The complete runnable version — streaming and a working client — is+[`examples/libsy.py`](examples/libsy.py). Types:+[`switchyard-libsy`](crates/libsy/README.md),+[`switchyard-protocol`](crates/protocol/README.md).### Path 3 — Run the Standalone Proxy@@ -351,11 +266,11 @@ the common route shape and self-hosted targets.- **[Routing Overview](docs/routing_algorithms/overview.md)**: choose and configure a routing algorithm- **[TOML Schema](docs/reference/toml_schema.md)**: every configuration key- **[Architecture](docs/architecture.md)**: how the proxy and library components fit together-- **[`switchyard-server`](crates/switchyard-server/README.md)**: server configuration, routing algorithms, and metrics-- **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application-- **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types-- **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation-- **[`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md)**: install Switchyard as a native NeMo Relay plugin+- **[switchyard-server](crates/switchyard-server/README.md)**: server configuration, routing algorithms, and metrics+- **[switchyard-libsy](crates/libsy/README.md)**: embed routing algorithms in a Rust application+- **[switchyard-protocol](crates/protocol/README.md)**: provider-neutral request, response, and streaming types+- **[switchyard-translation](crates/switchyard-translation/README.md)**: request, response, and stream translation+- **[switchyard-nemo-relay-plugin](crates/switchyard-nemo-relay-plugin/README.md)**: install Switchyard as a native NeMo Relay plugin## Benchmark Provenance