Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

chore(docs): massive docs refactor for QA (#230)

Ayush Agarwald6769656c528
diff --git a/AGENTS.md b/AGENTS.md
index 4ce6f7e47..3408ad17b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -354,7 +354,7 @@ uvicorn.run(build_switchyard_app(switchyard), port=4000)
- Export new public classes from `switchyard/__init__.py` with `__all__`.
- Write unit tests for new roles and bug fixes.
- Use `ProxyContext.metadata` for cross-component state within a request.
-- In a new `LLMBackend`, map upstream context-window 4xx to `SwitchyardError::ContextWindowExceeded` (Rust) — the chain executor uses it for evict-and-retry. See [Context-Window Handling](docs/context_window.md).
+- In a new `LLMBackend`, map upstream context-window 4xx to `SwitchyardError::ContextWindowExceeded` (Rust) — the chain executor uses it for evict-and-retry. See [Context-Window Handling](docs/operations/context_window.md).
### Ask first
- Modifying `pyproject.toml` dependencies.
diff --git a/README.md b/README.md
index 70178a336..751ad11c6 100644
--- a/README.md
+++ b/README.md
@@ -19,39 +19,80 @@ algorithm you write yourself.
## Features
- **Protocol Translation**: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
-- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or custom routers
-- **Strong Types**: provider-neutral request, response, and streaming types
-- **Explicit Configuration**: TOML defines LLM clients, targets, and algorithm routes
+- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm
- **Operational Metrics**: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead
## Quick Start
-Install the CLI and server dependencies:
+Choose the launcher path to run Claude Code, Codex CLI, or OpenClaw through
+Switchyard. Choose the server path to run Switchyard as a standalone proxy.
+
+### Launcher Path
+
+Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) if it is
+not already available, then install the published Switchyard tool:
```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+source "$HOME/.local/bin/env"
uv tool install "nemo-switchyard[cli,server]"
-export OPENROUTER_API_KEY="sk-or-..."
```
-Launch a coding agent through the packaged OpenRouter deployment:
+The coding agent you launch must also be installed and on your `PATH`. This does
+not install the standalone `switchyard-server` binary; use the Server Path for
+that.
+
+Set an OpenRouter key and launch against the packaged deployment:
```bash
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
switchyard launch claude --model switchyard
switchyard launch codex --model switchyard
+switchyard launch openclaw --model switchyard
```
-Pass a custom native deployment when needed:
+To use your own native TOML deployment, pass its route ID and configuration:
```bash
switchyard launch claude --model my-route --config routes.toml
```
-The deployment schema belongs in the
-[`switchyard-server` README](crates/switchyard-server/README.md).
+### Server Path
-## Architecture
+Use this path to build and run the standalone Rust proxy. Install
+[Rust with Cargo](https://rust-lang.org/tools/install/), then build the release
+binary from source:
+
+```bash
+git clone https://github.com/NVIDIA-NeMo/Switchyard.git
+cd Switchyard
+cargo build --locked --release -p switchyard-server
+./target/release/switchyard-server --help
+```
+
+Prebuilt binaries are not published yet. `rustup` installs the pinned toolchain
+automatically when you run Cargo from the repository.
+
+Create `routes.toml` using the
+[Getting Started guide](docs/getting_started.md#server-path), then validate it
+and start the server:
+
+```bash
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
+./target/release/switchyard-server --config routes.toml --dry-run
+./target/release/switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
+```
+
+Verify the proxy in another terminal:
+
+```bash
+curl http://localhost:4000/health
+```
-Switchyard sits between your client applications and one or more LLM backends:
+For a complete configuration and a test request, follow
+[Getting Started](docs/getting_started.md).
+
+## Architecture
```mermaid
flowchart LR
@@ -65,26 +106,20 @@ flowchart LR
Clients keep their native OpenAI or Anthropic API format. Switchyard picks a
configured backend, forwards the request in that backend's own format, and
-translates the response back into the shape the client expects.
+translates the response back into the shape the client expects. The server
+accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each
+configured LLM client selects one upstream format.
## Documentation
+- **[Getting Started](docs/getting_started.md)**: complete launcher and standalone server walkthroughs
+- **[Core Concepts](docs/core_concepts.md)**: LLM clients, targets, routes, model IDs, and routing algorithms
+- **[Routing Overview](docs/routing_algorithms/overview.md)**: choose and configure a routing algorithm
- **[`switchyard-server`](crates/switchyard-server/README.md)**: server configuration, routing algorithms, and metrics
- **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application
- **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types
- **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation
-## Supported API Formats
-
-The server exposes:
-
-- OpenAI Chat Completions
-- OpenAI Responses
-- Anthropic Messages
-
-Configured upstream clients support the same formats. The OpenAI Chat format
-also works with compatible servers such as vLLM, NVIDIA NIM, Ollama, and Azure.
-
## Community
- **Issues**: [GitHub Issues](https://github.com/NVIDIA-NeMo/Switchyard/issues)
diff --git a/docs/architecture.md b/docs/architecture.md
index e77dab115..af38d0e8a 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -40,55 +40,23 @@ request signals, or conversation affinity. See the
## Backend Wire Format
-`BackendFormat` controls the upstream endpoint Switchyard calls. Explicit
-formats select an endpoint directly and do not run capability probes.
-
-| Format | Upstream behavior | Use when |
-|---|---|---|
-| `ANTHROPIC` | Always sends to `/v1/messages`. No probe. | You know the upstream is Anthropic-native (Anthropic API, NIM Claude routes). |
-| `RESPONSES` | Always sends to `/v1/responses`. No probe. | You know the upstream supports the OpenAI Responses API. Fails on NIM / non-OpenAI upstreams. |
-| `OPENAI` | Always sends to `/v1/chat/completions`. No probe. | You know the upstream is OpenAI-compatible (NIM, OpenRouter, etc). Safe universal choice. |
-| `AUTO` | Probes at startup and picks the best format. | The upstream is unknown or varies across deployments. |
-| *(omitted)* | Defaults to `OPENAI` with no probe. | Always set `format:` explicitly when the upstream is not OpenAI-compatible. |
-
-### AUTO Decision Tree
-
-```mermaid
-flowchart TB
- auto["BackendFormat.AUTO"]
- chat{"/v1/chat/completions works?"}
- openai_chat["OPENAI<br/>/v1/chat/completions"]
- messages{"/v1/messages works?"}
- anthropic["ANTHROPIC<br/>/v1/messages"]
- responses{"/v1/responses works?"}
- responses_format["RESPONSES<br/>/v1/responses"]
- openai_fallback["OPENAI<br/>/v1/chat/completions fallback"]
-
- auto -->|"Probe /v1/chat/completions"| chat
- chat -->|"Timeout: assume Chat Completions"| openai_chat
- chat -->|"Yes"| openai_chat
- chat -->|"No (fast 404): probe /v1/messages"| messages
- messages -->|"Yes"| anthropic
- messages -->|"No: probe /v1/responses"| responses
- responses -->|"Yes"| responses_format
- responses -->|"No"| openai_fallback
-```
-
-Supported inbound and response formats are handled automatically.
-`TranslationEngine` converts the client's request to the resolved backend
-format and translates the backend response back to the client's expected
-format. When a cross-format conversion is required, both directions decode to
-and re-encode from the neutral conversation IR. This lets Claude Code, Codex,
-OpenClaw, and SDK clients use their native wire format with any supported
-upstream format.
-
-> Prefer an explicit format for controlled deployments. It skips capability
-> probes and makes the upstream contract clear. Use `AUTO` when provider
-> capabilities are unknown or vary across deployments.
->
-> **`AUTO` costs startup latency.** Each probe is a live request to the upstream,
-> so a slow endpoint adds a round-trip per probe, with up to three tried in
-> order. Set `format:` explicitly to remove probing.
+Each LLM client in the deployment file sets `format`. It fixes the upstream
+endpoint and wire format for every target that uses that client.
+
+| `format` | Upstream endpoint |
+|---|---|
+| `openai_chat` | `/v1/chat/completions` |
+| `openai_responses` | `/v1/responses` |
+| `anthropic_messages` | `/v1/messages` |
+
+`format` is required. Switchyard does not probe upstreams or select a format
+automatically.
+
+Switchyard decodes every inbound request into provider-neutral types before
+routing, then encodes it for the selected target's format.
+`switchyard-translation` converts requests, buffered responses, and streaming
+events between these formats, so Claude Code, Codex, OpenClaw, and SDK clients
+keep their native wire format regardless of the upstream a route selects.
## Related Documentation
diff --git a/docs/cli_reference.md b/docs/cli_reference.md
index f0417ddf9..dbe3a11cc 100644
--- a/docs/cli_reference.md
+++ b/docs/cli_reference.md
@@ -1,25 +1,89 @@
-# CLI reference
+# CLI Reference
-The documented CLI workflow launches a coding agent through the native Rust
-server.
+Switchyard has two command-line paths:
-## `switchyard launch`
+- **Launcher Path:** `switchyard launch` starts a supported coding agent and
+ hosts the native Rust server through the packaged PyO3 binding.
+- **Server Path:** `switchyard-server` runs the standalone Rust proxy for API
+ clients and custom deployments.
+
+## Launcher Path: `switchyard launch`
+
+Install the launcher with `uv tool install "nemo-switchyard[cli,server]"`. The
+selected coding agent must also be installed and available on `PATH`.
+
+### Usage
```bash
-export OPENROUTER_API_KEY="sk-or-..."
-switchyard launch claude --model switchyard
+switchyard launch <agent> --model <route-id> [--config <deployment.toml>] [-- <agent args>]
```
Supported agents are `claude`, `codex`, and `openclaw`.
-```text
-switchyard launch <agent> --model <route-id> [--config <deployment.toml>] [-- <agent args>]
+| Option | Required | Purpose |
+|---|:---:|---|
+| `--model ID` | Yes | Route ID from the selected deployment. |
+| `--config PATH` | No | Native TOML deployment. Defaults to the packaged OpenRouter deployment. |
+| `-- ...` | No | Arguments forwarded unchanged to the coding agent. |
+
+### Packaged Deployment
+
+The packaged deployment exposes the route ID `switchyard` and uses OpenRouter:
+
+```bash
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
+switchyard launch claude --model switchyard
+switchyard launch codex --model switchyard
+switchyard launch openclaw --model switchyard
+```
+
+### Custom Deployment
+
+Select a route ID from an explicit native TOML deployment:
+
+```bash
+switchyard launch claude --model my-route --config routes.toml
+```
+
+The launcher owns the server lifecycle and stops it when the coding agent exits.
+This command does not install or run the standalone `switchyard-server` binary.
+
+## Server Path: `switchyard-server`
+
+Build the standalone binary with
+`cargo build --locked --release -p switchyard-server`. It reads the same native
+TOML deployment schema accepted by the launcher.
+
+### Usage
+
+```bash
+switchyard-server --config <deployment.toml> [options]
+```
+
+When built from the repository, invoke it as
+`./target/release/switchyard-server`.
+
+| Option | Default | Purpose |
+|---|---|---|
+| `--config PATH` | Required | TOML file defining LLM clients, targets, and algorithm routes. |
+| `--host HOST` | `0.0.0.0` | Address on which the server listens. |
+| `-p, --port PORT` | `4000` | Port on which the server listens. |
+| `--backlog BACKLOG` | `65535` | TCP listen backlog configured before accepting traffic. |
+| `--dry-run` | Off | Validate the deployment without binding a socket. |
+| `--tls-cert PATH` | None | PEM certificate path; requires `--tls-key`. |
+| `--tls-key PATH` | None | PEM private-key path; requires `--tls-cert`. |
+| `-h, --help` | — | Print command help. |
+| `-V, --version` | — | Print the server version. |
+
+Validate a deployment, then start the proxy:
+
+```bash
+./target/release/switchyard-server --config routes.toml --dry-run
+./target/release/switchyard-server --config routes.toml \
+ --host 127.0.0.1 --port 4000
```
-| Option | Purpose |
-|---|---|
-| `--model ID` | Required route ID from the deployment. |
-| `--config PATH` | TOML deployment. Defaults to the packaged OpenRouter deployment. |
-| `-- ...` | Arguments forwarded to the coding agent. |
+## Related Documentation
-Custom deployment syntax is documented in `crates/switchyard-server/README.md`.
+- [Getting Started](getting_started.md): installation, configuration, and verification for both paths
+- [`switchyard-server`](../crates/switchyard-server/README.md): complete TOML schema, TLS, and metrics
diff --git a/docs/core_concepts.md b/docs/core_concepts.md
index f5cc5cebd..d39fa2b57 100644
--- a/docs/core_concepts.md
+++ b/docs/core_concepts.md
@@ -1,94 +1,111 @@
# Core Concepts
-This page explains the vocabulary used throughout the documentation. For setup,
-see [Getting Started](getting_started.md). For the request lifecycle, see
+Switchyard routes LLM requests through provider-neutral Rust types and exposes
+the result through OpenAI- and Anthropic-compatible APIs. For setup, see
+[Getting Started](getting_started.md). For the complete request lifecycle, see
[Architecture](architecture.md).
-## How it fits together
+## Runtime Surfaces
-Switchyard is a proxy. Clients such as coding agents and SDKs talk to it on one
-side, while hosted providers, private endpoints, and local model servers sit on
-the other. A client sends a model ID, and the route registered under that ID
-decides what runs.
+Switchyard exposes the same Rust routing core through two runtime surfaces:
+
+- **`switchyard-server`** is the standalone HTTP proxy. It loads a native TOML
+ deployment and exposes OpenAI Chat Completions, OpenAI Responses, and
+ Anthropic Messages endpoints.
+- **`switchyard-libsy`** is the embeddable Rust library. Applications construct
+ targets and algorithms directly and can let libsy make calls or fulfill its
+ requested model calls themselves.
+
+The `switchyard launch` command is a launcher for coding agents. It hosts the
+native Rust server through the packaged PyO3 binding and points the selected
+agent at that server.
+
+## Request Flow
+
+The standalone server translates an inbound provider format into the shared
+protocol types before routing. The selected target's LLM client translates the
+request into its upstream format, makes the call, and translates the response
+back to the client's format.
```mermaid
flowchart LR
- client["Client<br/>picks a model ID"]
- route["Route<br/>routing policy"]
+ client["Client<br/>OpenAI or Anthropic API"]
+ request["Decode request<br/>provider-neutral types"]
+ route["Route<br/>algorithm decision"]
target["Target<br/>upstream model"]
- backend["Model backend"]
+ upstream["Encode request<br/>upstream format"]
+ backend["LLM backend"]
+ response["Decode and translate<br/>response or stream"]
- client -->|"model field"| route
- route -->|"selects"| target
- target --> backend
+ client --> request --> route --> target --> upstream --> backend
+ backend --> response --> client
```
-## Routes and targets
+## LLM Clients, Targets, and Routes
+
+A native TOML deployment has three layers:
-The Python CLI loads a YAML bundle with a `routes:` section. Each route has a
-client-facing name and a `type` that chooses its behavior. Depending on that
-type, the route owns one target, two routing tiers, or a larger endpoint list.
+| Layer | Defines |
+|---|---|
+| **LLM client** | Upstream base URL, wire format, credential environment variable, and retry policy. |
+| **Target** | One upstream model ID and the LLM client used to call it. |
+| **Route** | One client-visible model ID and the algorithm that selects or calls targets. |
-Shared `defaults:` can provide a base URL, API key, and backend format without
-repeating them in every target. The [Routing Overview](routing_algorithms/overview.md)
-has a complete runnable bundle.
+These layers keep provider transport separate from routing policy. Several
+targets can share one LLM client, and several routes can reuse the same target.
+Secrets stay outside the TOML: `api_key_env` names the environment variable the
+server reads at startup.
## Model IDs
-Every route name is registered as a model ID and listed on `GET /v1/models`.
-Clients select a route by putting that name in the request's `model` field.
-Some route types also discover a tier's upstream catalog from its
-`GET /v1/models` and register those model IDs directly.
+The table keys in `llm_clients`, `targets`, and `routes` are local references
+inside the TOML file. Their `id` fields have different external meanings:
-## Tiers and routing strategies
+- A target's `id` is the model ID sent to its upstream provider.
+- A route's `id` is the model ID clients send to Switchyard.
-Most routing strategies divide traffic between two tiers: a strong target that
-is more capable and more expensive, and a weak target that is cheaper and
-faster. A tier is a role assigned inside a route, not a fixed property of a
-model.
+The server lists route IDs on `GET /v1/models`. A request selects a route by
+putting that ID in its `model` field. The native Rust server does not discover
+or register additional provider models automatically.
-A route's `type` sets the strategy. `model` calls one target.
-`random_routing` splits traffic on a fixed probability. `deterministic` asks a
-classifier model to pick a tier. `stage_router` uses agent-progress signals,
-with an optional classifier fallback. `escalation_router` starts on the weak
-tier and uses an LLM judge to escalate when needed. The
-[Routing Overview](routing_algorithms/overview.md) explains when to use each.
+## Routing Algorithms
-Session affinity pins a conversation to one tier so later turns reuse it
-instead of being classified again. It belongs to deterministic classifier
-routing and is not a strategy of its own. Random and stage-router routing make
-a decision for every request. [Sticky Routing](routing_algorithms/sticky_routing.md)
-covers affinity in full.
+An algorithm receives the normalized request, publishes a routing decision,
+and serves the selected target. The standalone server supports these primary
+route types:
-## Formats and translation
+| Route type | Behavior |
+|---|---|
+| `passthrough` | Sends every request to one target. |
+| `random` | Selects among targets using optional relative weights. |
+| `llm_classifier` | Uses a classifier target to choose between weak and strong targets. |
+| `stage_router` | Uses tool-result and progress signals to choose an efficient or capable target. |
-Clients reach Switchyard through OpenAI Chat Completions, Anthropic Messages,
-or OpenAI Responses. Each target has a backend format: `openai`, `anthropic`,
-`responses`, or `auto`. When the inbound and backend formats differ,
-Switchyard translates the request on the way out and the response on the way
-back.
+Strong, weak, capable, and efficient are roles within an algorithm, not fixed
+properties of a model. The same upstream model can serve different roles in
+different routes.
-That translation lets Claude Code, which speaks Anthropic Messages, run against
-an OpenAI-compatible model. The [Architecture](architecture.md) page documents
-the supported formats and request lifecycle.
+## Protocol Types and Translation
-## Programmatic Python profiles
+`switchyard-protocol` defines provider-neutral requests, responses, messages,
+content blocks, tool calls, usage, and streaming events. Algorithms operate on
+these types rather than provider SDK objects.
-Embedded Python callers can construct typed profile configs and runtimes
-directly instead of loading a route bundle. `ProfileInput` carries the request
-and request metadata, while the profile protocols define `run`, `process`, and
-`rprocess`. This programmatic API is separate from CLI configuration.
+Each LLM client explicitly selects one upstream format:
-## Rust server configuration
+- `openai_chat`
+- `openai_responses`
+- `anthropic_messages`
-The `switchyard-server` binary is built directly on libsy. Its TOML file
-explicitly defines LLM clients, targets, and algorithm routes; it does not load
-Python route bundles. See the
-[Rust server README](../crates/switchyard-server/README.md) for details.
+`switchyard-translation` converts requests, buffered responses, and streaming
+events between those formats. This lets a client keep its native API while the
+selected target uses a different upstream protocol.
-## Where to go next
+## Where to Go Next
-- [Getting Started](getting_started.md) to install and send a first request.
-- [Routing Overview](routing_algorithms/overview.md) to choose and tune a strategy.
-- [Architecture](architecture.md) to see a request travel end to end.
-- [CLI Reference](cli_reference.md) for flags and environment variables.
+- [Getting Started](getting_started.md): install and run either execution path.
+- [CLI Reference](cli_reference.md): launcher and standalone server arguments.
+- [LLM Classifier Routing](routing_algorithms/llm_classifier_routing.md): configure classifier routing.
+- [Architecture](architecture.md): follow a request through the server and Rust crates.
+- [`switchyard-server`](../crates/switchyard-server/README.md): complete TOML schema, endpoints, and metrics.
+- [`switchyard-libsy`](../crates/libsy/README.md): embed and extend routing algorithms in Rust.
diff --git a/docs/getting_started.md b/docs/getting_started.md
index e48eaf263..05729c1b2 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -1,31 +1,245 @@
-# Getting Started
+# Getting Started with Switchyard
-## Install
+Switchyard has two native Rust execution paths:
+
+- **Launcher path:** install the Python-distributed CLI and launch Claude Code,
+ Codex, or OpenClaw through the packaged Rust server binding.
+- **Server path:** build and run the standalone Rust server for API clients and
+ custom deployments.
+
+## Launcher Path
+
+Use this path when you want Switchyard to start and configure a supported coding
+agent for you.
+
+### Install the CLI
+
+Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) if it is
+not already available:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+source "$HOME/.local/bin/env"
+```
+
+Then install the published Switchyard tool:
```bash
uv tool install "nemo-switchyard[cli,server]"
```
-## Launch a coding agent
+This creates an isolated Python tool environment containing the `switchyard`
+CLI, its CLI and server dependencies, and the packaged PyO3 Rust extension.
+`switchyard launch` starts the native Rust server through that extension; this
+path does not install or run the standalone `switchyard-server` binary.
+
+Install the coding agent you want to launch, then verify both commands are on
+your `PATH`.
-The packaged deployment uses OpenRouter:
+### Launch with the packaged OpenRouter deployment
+
+The packaged deployment exposes the route ID `switchyard`. Export an OpenRouter
+key and choose an agent:
```bash
-export OPENROUTER_API_KEY="sk-or-..."
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
switchyard launch claude --model switchyard
```
-Codex and OpenClaw use the same shape:
+Codex and OpenClaw use the same deployment:
```bash
switchyard launch codex --model switchyard
switchyard launch openclaw --model switchyard
```
-To use another deployment:
+### Launch with a custom deployment
+
+Pass a native TOML deployment and select one of its route IDs:
```bash
-switchyard launch claude --model my-route --config routes.toml
+switchyard launch codex --model my-route --config routes.toml
```
-See `crates/switchyard-server/README.md` for the TOML schema.
+The launcher manages the native server lifecycle and points the selected coding
+agent at it. The server configuration format is the same TOML schema used by the
+standalone server below.
+
+## Server Path
+
+Use this path when you want a standalone proxy for API clients or need to
+operate the Rust server directly.
+
+### Prerequisites
+
+- Git, a native build toolchain, and Rust with Cargo
+- An API key for OpenRouter, OpenAI, Anthropic, or another OpenAI-compatible endpoint.
+ To use OpenRouter, create an account at [openrouter.ai](https://openrouter.ai/)
+ and generate a key from the [OpenRouter keys page](https://openrouter.ai/keys).
+
+On Ubuntu or WSL, install the build prerequisites and Rust with `rustup`:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y build-essential curl git
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+source "$HOME/.cargo/env"
+```
+
+On macOS or native Windows, follow the
+[official Rust installation instructions](https://rust-lang.org/tools/install/).
+The Rust installer includes `rustc`, Cargo, and `rustup`.
+
+Install `uv` for the repository's Python-based tooling and CI checks. It is not
+required to build or run the Rust server:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+If either installer updates your shell configuration, restart the shell before
+continuing. Verify the tools:
+
+```bash
+git --version
+rustc --version
+cargo --version
+uv --version
+```
+
+### Build the server
+
+Build the Rust server from source:
+
+```bash
+git clone https://github.com/NVIDIA-NeMo/Switchyard.git
+cd Switchyard
+cargo build --locked --release -p switchyard-server
+./target/release/switchyard-server --help
+```
+
+The repository pins Rust `1.96.1` in `rust-toolchain.toml`; `rustup` selects and
+installs it automatically when Cargo runs from the repository. Prebuilt Rust
+binaries are not published yet.
+
+### Configure
+
+The Rust server reads an explicit TOML file. It does not use the legacy Python
+CLI's saved configuration or YAML routing profiles.
+
+Create `routes.toml` with an LLM-classifier route:
+
+```toml
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.weak]
+id = "openai/gpt-4o-mini"
+llm_client = "openrouter"
+
+[targets.strong]
+id = "openai/gpt-4o"
+llm_client = "openrouter"
+
+[routes.smart]
+id = "switchyard"
+type = "llm_classifier"
+classifier_target = "weak"
+strong_target = "strong"
+weak_target = "weak"
+base_threshold = 0.5
+```
+
+`format` selects the upstream protocol and must be `openai_chat`,
+`openai_responses`, or `anthropic_messages`. `api_key_env` names the environment
+variable the server reads; the secret does not belong in the TOML file.
+
+### Run the server
+
+Export the provider credential, validate the configuration without binding a
+socket, then start the release binary:
+
+```bash
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
+./target/release/switchyard-server --config routes.toml --dry-run
+./target/release/switchyard-server --config routes.toml \
+ --host 127.0.0.1 --port 4000
+```
+
+Any client that speaks OpenAI Chat Completions, Anthropic Messages, or OpenAI
+Responses API can connect. The route `id` is the model name clients use.
+
+In another terminal:
+
+```bash
+curl http://localhost:4000/health
+curl http://localhost:4000/v1/models
+curl http://localhost:4000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'
+```
+
+### Routing algorithms
+
+#### Choose a route type
+
+This guide uses `llm_classifier`, which asks a classifier target whether each
+request should use the weak or strong target. The Rust server also supports:
+
+| Algorithm | Use it when | Config |
+|---|---|---|
+| [Random](routing_algorithms/random_routing.md) | You need a weighted split for A/B tests or baselines. | `random` |
+| [LLM classifier](routing_algorithms/llm_classifier_routing.md) | Request content should decide whether to use the weak or strong target. | `llm_classifier` |
+| [Stage router](routing_algorithms/stage_router_routing.md) | Tool-result and progress signals should select an efficient or capable target. | `stage_router` |
+
+A single TOML file can declare multiple routes. The table key, such as
+`routes.smart`, is a local configuration name; each route's `id` is exposed as a
+model on `GET /v1/models`.
+
+See [Routing Overview](routing_algorithms/overview.md) to compare strategies,
+and the [`switchyard-server` guide](../crates/switchyard-server/README.md) for
+the complete TOML schema, route options, TLS, and metrics.
+
+### Troubleshooting
+
+**No API key / auth error**
+
+```bash
+test -n "$OPENROUTER_API_KEY" && echo "key is set" || echo "key is missing"
+./target/release/switchyard-server --config routes.toml --dry-run
+```
+
+Confirm that `api_key_env` in `routes.toml` names the environment variable you
+exported. The dry run validates the schema, environment lookup, target
+references, and route construction without starting the server.
+
+**Connection refused**
+
+Check health: `curl http://localhost:4000/health`
+
+**Telemetry header opt-out**
+
+Switchyard adds an `X-Switchyard-Version` header to outbound LLM calls for
+release attribution. No request or response content is included. To disable:
+
+```bash
+export SWITCHYARD_TELEMETRY_OPT_OUT=1
+```
+
+---
+
+### Next steps
+
+- [Core Concepts](core_concepts.md): LLM clients, targets, and routes
+- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration,
+ routing algorithms, TLS, and metrics
+- [`switchyard-libsy`](../crates/libsy/README.md): embed routing algorithms in a
+ Rust application
+- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral
+ request, response, and streaming types
+- [`switchyard-translation`](../crates/switchyard-translation/README.md):
+ request, response, and stream translation
diff --git a/docs/index.md b/docs/index.md
index 70ba539bc..2ada13bc0 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -3,26 +3,29 @@
Switchyard routes and translates LLM traffic for coding agents and API clients.
It supports OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages.
-## Start
+## Choose a Path
-```bash
-uv tool install "nemo-switchyard[cli,server]"
-export OPENROUTER_API_KEY="sk-or-..."
-switchyard launch claude --model switchyard
-```
+| Goal | Path | Start here |
+|---|---|---|
+| Run Claude Code, Codex, or OpenClaw through Switchyard | Launcher Path | [Install and launch an agent](getting_started.md#launcher-path) |
+| Run Switchyard as a standalone proxy for API clients | Server Path | [Build and run the Rust server](getting_started.md#server-path) |
+| Add Switchyard routing to a Rust application | Library Path | [`switchyard-libsy`](../crates/libsy/README.md) |
-Use a custom native deployment when needed:
+The Launcher Path installs the `switchyard` CLI and hosts the native Rust server
+through its packaged PyO3 binding. The Server Path builds and runs the
+standalone `switchyard-server` binary.
-```bash
-switchyard launch codex --model my-route --config routes.toml
-```
+## Explore
-The TOML schema is maintained in `crates/switchyard-server/README.md`.
+- [Core Concepts](core_concepts.md): learn the LLM client, target, and route layers
+- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model
+- [Architecture](architecture.md): understand the proxy and library components
+- [CLI Reference](cli_reference.md): inspect supported launcher commands and options
+- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior
-## Read More
+## Reference
-- [Getting Started](getting_started.md)
-- [CLI Reference](cli_reference.md)
-- [Routing Overview](routing_algorithms/overview.md)
-- [Architecture](architecture.md)
-- [Context-Window Handling](operations/context_window.md)
+- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration, endpoints, and metrics
+- [`switchyard-libsy`](../crates/libsy/README.md): embeddable routing algorithms
+- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral API types
+- [`switchyard-translation`](../crates/switchyard-translation/README.md): protocol translation
diff --git a/docs/internal/subagent_override.md b/docs/internal/subagent_override.md
new file mode 100644
index 000000000..e8b59b268
--- /dev/null
+++ b/docs/internal/subagent_override.md
@@ -0,0 +1,54 @@
+# Sub-Agent Override
+
+Internal note. Not published: `mkdocs.yml` excludes `internal/**` from the site.
+
+`subagent_target` exists only on the Python route bundle loaded by
+`switchyard serve`. The `switchyard-server` TOML schema has no equivalent key.
+
+Any bundle route may name an optional `subagent_target` in its common envelope,
+alongside `type`. A request carrying a recognized sub-agent signal — Claude Code
+agent-lineage headers, Codex delegated-work kinds (`x-openai-subagent:
+collab_spawn` or `review`), or an explicit `x-switchyard-is-subagent: true` —
+bypasses the route's own chain and runs as a direct passthrough to that target:
+
+```yaml
+routes:
+ assistant:
+ type: model
+ target:
+ model: strong-model
+ subagent_target:
+ model: cheaper-worker-model
+```
+
+This keeps a sub-agent loop on one intentional, cache-compatible target instead
+of re-routing every worker turn. The worker may live on a different provider
+entirely — give it its own `base_url` and `api_key` and one route spans two
+upstreams:
+
+```yaml
+ subagent_target:
+ model: my-local-model
+ base_url: http://localhost:8000/v1
+ api_key: dummy
+ format: openai
+```
+
+Detection is the protocol crate's canonical policy, shared with the
+`switchyard-libsy` `SubagentOverride` classifier, so both engines agree on what
+counts as delegated work. Harness-maintenance turns (`compact`,
+`memory_consolidation`) and unrecognized kinds stay on normal routing, as does
+everything else when the field is absent. A worker-target failure surfaces as a
+normal target error — it is never silently re-routed through the route's own
+chain.
+
+To suppress sub-agent routing for a request that carries a recognized signal,
+send `x-switchyard-is-subagent: false`. This explicit header overrides Codex and
+Claude Code lineage signals in either direction: `false` keeps the request on
+normal routing even when delegated-work headers are present, and `true` marks a
+request as a sub-agent even when no harness headers appear.
+
+The key applies to bundle routes of type `model`, `deterministic`,
+`escalation_router`, and `stage_router`. It is accepted on `random_routing`
+routes but has no effect, because they expand into their table entries on a
+separate path that never consumes it.
diff --git a/docs/operations/context_window.md b/docs/operations/context_window.md
index a6504e157..4ade38066 100644
--- a/docs/operations/context_window.md
+++ b/docs/operations/context_window.md
@@ -1,36 +1,64 @@
# Context-Window Handling
When an upstream rejects a request because the prompt exceeds the model's
-context window, Switchyard evicts that target for the current request,
-reroutes to the configured fallback target, and retries once. If the fallback
-also overflows, the request fails with a 400 in the client's inbound wire
-format.
+context window, Switchyard drops that target and calls another target on the
+same route, repeating until a call succeeds or every target has been tried. If
+the request carries `x-switchyard-session-id` or a recognized coding-agent
+session header such as `x-claude-code-session-id`, the target remains excluded
+for the rest of that session. Without a session header, the fallback still
+applies to the current request, but the overflow is not remembered.
+After truncating or resetting context, clients should use a new session ID;
+reusing the old ID preserves its target exclusions.
-Any multi-target route (stage-router, random_routing, or deterministic) supports
-this. Set `fallback_target_on_evict` on the route. Single-target routes
-(`type: model`) have no alternative target, so the original
-overflow propagates unchanged.
+## What counts as an overflow
+
+Switchyard treats an upstream reply as a context overflow only when the status
+is HTTP 400 **and** the body identifies a context-length error: an `error.code`
+of `context_length_exceeded`, or a message containing a phrase such as
+`maximum context length` or `prompt is too long`. An overflow reported any other
+way — HTTP 413 or 422, for example — is not recognized as one. The request
+fails on the spot with the upstream's status code, and the upstream's body is
+returned inside Switchyard's error message.
## Configuration
-`fallback_target_on_evict` is required on every multi-target route and must
-match one of the route's declared target ids:
-
-```yaml
-routes:
- my-stage-router:
- type: stage_router
- picker: capable_first
- fallback_target_on_evict: strong # must match strong.id or weak.id
- strong:
- id: strong
- model: anthropic/claude-opus-4.7
- weak:
- id: weak
- model: moonshotai/kimi-k2.6
+There is no eviction key. A route falls through whenever it has more than one
+target, so the route below needs nothing beyond its two tiers:
+
+```toml
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.strong]
+id = "openai/gpt-4o"
+llm_client = "openrouter"
+
+[targets.weak]
+id = "openai/gpt-4o-mini"
+llm_client = "openrouter"
+
+[routes.stage]
+id = "switchyard/stage"
+type = "stage_router"
+capable_target = "strong"
+efficient_target = "weak"
+picker = "efficient_first"
+confidence_threshold = 0.5
+```
+
+Response headers report where the request actually landed:
+
+```text
+x-model-router-selected-model: openai/gpt-4o-mini
+x-model-router-rationale: openai/gpt-4o exceeded its context window; fell back to openai/gpt-4o-mini
```
-## Scope
+## When every target overflows
-Single eviction + single retry per request. Compaction, cool-down, and
-re-insertion of evicted targets are out of scope.
+Once no untried target is left — including a single-target `passthrough` route,
+which has no alternative from the start — the request fails with HTTP 400 and an
+error `code` of `context_length_exceeded`.
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
new file mode 100644
index 000000000..918e8e3e4
--- /dev/null
+++ b/docs/reference/toml_schema.md
@@ -0,0 +1,143 @@
+# TOML Schema
+
+The native deployment file defines the LLM clients, targets, and routes a
+Switchyard server serves. It is read by `switchyard-server --config` and by
+`switchyard launch --config`.
+
+Validate a file without starting the server:
+
+```bash
+switchyard-server --config routes.toml --dry-run
+```
+
+## Minimal Example
+
+```toml
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.strong]
+id = "anthropic/claude-sonnet-4.5"
+llm_client = "openrouter"
+
+[routes.default]
+id = "switchyard"
+type = "passthrough"
+target = "strong"
+```
+
+`schema_version` must be `1`. Table names under `llm_clients`, `targets`, and
+`routes` are local references; clients send the route's `id` as the model name.
+
+## `[llm_clients.<name>]`
+
+| Key | Required | Default | Meaning |
+|---|:---:|---|---|
+| `format` | Yes | — | `openai_chat`, `openai_responses`, or `anthropic_messages`. |
+| `base_url` | Yes | — | Upstream base URL. |
+| `api_key_env` | No | unset | Name of the environment variable holding the key. Omit to send no authentication. |
+| `extra_headers` | No | `{}` | Extra HTTP headers sent upstream. |
+| `max_retries` | No | `2` | Retry budget, `0`–`10`. |
+
+The TOML never contains the secret itself. `api_key_env` names a variable that
+must exist and be non-empty when the server loads.
+
+## `[targets.<name>]`
+
+| Key | Required | Default | Meaning |
+|---|:---:|---|---|
+| `id` | Yes | — | Exact model ID sent upstream. |
+| `llm_client` | Yes | — | Key under `[llm_clients]`. |
+| `extra_body` | No | `{}` | Values merged into the upstream request when the request does not already set that key. |
+
+## `[routes.<name>]`
+
+Every route takes `id` and `type`, plus the keys for that type.
+
+### `noop`
+
+Returns a buffered assistant response containing `OK` without calling an
+upstream model. Use it for local smoke tests.
+
+### `passthrough`
+
+Sends every request to one target.
+
+| Key | Required | Meaning |
+|---|:---:|---|
+| `target` | Yes | Target every request is sent to. |
+
+### `random`
+
+Splits traffic across targets. See
+[Random Routing](../routing_algorithms/random_routing.md).
+
+| Key | Required | Default | Meaning |
+|---|:---:|---|---|
+| `targets` | Yes | — | Target names to choose from. |
+| `weights` | No | equal | Finite, non-negative relative weights in `targets` order, with at least one positive value. Invalid weights are rejected at load time. |
+| `seed` | No | unset | Reproduces the selection sequence. |
+
+### `llm_classifier`
+
+Classifies each task, then routes to the weak or strong target. See
+[LLM Classifier Routing](../routing_algorithms/llm_classifier_routing.md) for
+tuning.
+
+| Key | Required | Default | Meaning |
+|---|:---:|---|---|
+| `classifier_target` | Yes | — | Target the judge is called through. Not a routing destination. |
+| `strong_target` | Yes | — | Capable tier. |
+| `weak_target` | Yes | — | Efficient tier. |
+| `base_threshold` | Yes | — | Lowest solve probability that routes to the weak target. In `[0, 1]`. |
+| `min_confidence` | No | `0.0` | Lowest judge confidence that permits weak routing. |
+| `capability_elevated_floor` | No | unset | Higher floor for uncertain tasks. Must exceed `base_threshold`. |
+| `session_affinity` | No | `false` | Reuses a session's first decision on later turns. |
+| `message_hash_fallback` | No | `false` | Keys affinity on the first user message. Requires `session_affinity`. |
+| `recent_turn_window` | No | unset | Trailing turns the judge sees. |
+| `escalation` | No | unset | Switches the route to escalation judging. |
+
+Add an `escalation` table to judge each completed turn instead of classifying up
+front. See
+[Escalation-Router Routing](../routing_algorithms/escalation_router_routing.md).
+
+| Key | Required | Default | Meaning |
+|---|:---:|---|---|
+| `confirmations` | No | `2` | Consecutive escalate verdicts required to latch. Above `1` needs a session ID. |
+| `recent_turn_window` | No | `28` | Trailing messages shown to the judge. |
+| `window_message_chars` | No | `500` | Per-message cap inside that window. |
+
+`base_threshold` stays required, but escalation ignores it and the other
+classifier keys.
+
+### `stage_router`
+
+Scores tool signals to pick a tier per turn. See
+[Stage-Router Routing](../routing_algorithms/stage_router_routing.md) for the
+optional `handoff_notes` and `classifier` tables and for tuning.
+
+| Key | Required | Default | Meaning |
+|---|:---:|---|---|
+| `capable_target` | Yes | — | Capable tier. |
+| `efficient_target` | Yes | — | Efficient tier. |
+| `picker` | Yes | — | `capable_first` or `efficient_first`. Tier used when the signals are not confident. |
+| `confidence_threshold` | Yes | — | Corroboration a decisive pick needs. In `[0, 1]`. |
+| `recent_turn_window` | No | `3` | Trailing tool results the signals are computed over. |
+| `capable_system_prompt` | No | unset | System prompt handed to the capable tier. |
+| `efficient_system_prompt` | No | unset | System prompt handed to the efficient tier. |
+
+## Validation Errors
+
+`--dry-run` prefixes configuration failures with
+`invalid server config <path>:`. Within that wrapper, TOML deserialization
+errors start with `failed to parse TOML:`, while errors from validating the
+built configuration retain their inner message unchanged.
+
+## Related Documentation
+
+- [CLI Reference](../cli_reference.md)
+- [Routing Overview](../routing_algorithms/overview.md)
diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md
index 90ee9731a..6b179930c 100644
--- a/docs/routing_algorithms/escalation_router_routing.md
+++ b/docs/routing_algorithms/escalation_router_routing.md
@@ -1,198 +1,166 @@
# Escalation-Router Routing
-Escalation-router routing starts each conversation on a cheaper `weak` model.
-An LLM judge watches how the work progresses and moves the conversation to a
-more capable `strong` model when it detects sustained, recoverable trouble.
-After escalation, the conversation stays on the strong tier for the rest of
-the task.
-
-Use it for multi-turn agent workloads where a weak model can handle routine
-work but may need rescue after repeated errors, loops, drift, false progress,
-or premature completion. Unlike
-[LLM Classifier Routing](llm_classifier_routing.md), which predicts how
-difficult a request looks, escalation routing judges whether the run is
-actually going well.
-
-## How it works
-
-The router keeps one state value per conversation: whether that conversation
-has escalated. Conversations are identified from the stable system prompt and
-first user message, using the same bounded in-memory store described in
-[Sticky Routing](sticky_routing.md).
-
-For each turn, Switchyard:
-
-1. Checks the conversation latch. A latched conversation routes to `strong`
- without calling the judge again.
-2. Routes an unlatched conversation to `weak`. Before `judge.min_turn`
- (default `3`), it skips the judge because there is too little trajectory to
- assess.
-3. Gives the judge a bounded summary containing the system and first-user
- anchors, recent messages, and a coverage note for omitted history.
-4. Parses the judge's structured `escalate` decision. When the configured
- confirmation policy is satisfied, Switchyard pins the conversation to
- `strong` and uses the strong model for the current turn.
-5. Fails open to `weak` when the judge times out, errors, or returns invalid
- output. A judge failure never creates a strong-tier pin.
-
-The routing decision for one turn is:
+Escalation routing starts each conversation on a cheaper weak model. An LLM judge
+reads how the work is going and latches the session to a strong model when it
+detects sustained trouble.
-```mermaid
-%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%
-flowchart LR
- t["turn"] --> p{"strong tier pinned?"}
- p -->|yes| s["route strong; skip judge"]
- p -->|no| m{"turn >= min_turn?"}
- m -->|no| w["route weak"]
- m -->|yes| j["judge recent trajectory"]
- j -->|stay / fail open| w
- j -->|not yet confirmed| w
- j -->|confirmed escalation| l["pin and route strong"]
+Use it for multi-turn agent workloads where a weak model handles routine work but
+may need rescue after repeated errors, loops, or drift. Unlike plain
+[LLM Classifier Routing](llm_classifier_routing.md), which predicts how difficult
+a request looks before running it, escalation judges whether the run is actually
+going well.
- classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;
- class t,p,s,m,w,j,l box;
+## Configure an escalation route
+
+```toml
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.judge]
+id = "google/gemini-3.5-flash"
+llm_client = "openrouter"
+
+[targets.strong]
+id = "anthropic/claude-opus-4.7"
+llm_client = "openrouter"
+
+[targets.weak]
+id = "moonshotai/kimi-k2.6"
+llm_client = "openrouter"
+
+[routes.agent]
+id = "agent"
+type = "llm_classifier"
+classifier_target = "judge"
+strong_target = "strong"
+weak_target = "weak"
+base_threshold = 0.5
+escalation = { confirmations = 2, recent_turn_window = 28, window_message_chars = 500 }
```
-## Confirmation policy
+`classifier_target` is the judge. The route's `id`, `agent`, is the model name
+clients send; the judge is not exposed as a client-selectable model.
-`judge.confirmations` controls how many positive verdicts are required before
-the strong-tier latch fires:
+`base_threshold` is still required, but escalation ignores it, along with
+`min_confidence`, `capability_elevated_floor`, and `session_affinity`.
-- `confirmations: 1` is the default and escalates on the first positive
- verdict.
-- `confirmations: 2` with `confirmation_window: 1` requires positive verdicts
- on consecutive judged turns.
-- A larger `confirmation_window` allows recurring trouble to confirm even when
- negative verdicts occur between positive ones. A positive verdict remains
- live across at most `confirmation_window - 1` intervening negative verdicts.
+## How the decision works
-A judge failure provides no evidence either way: it routes the turn to weak
-without clearing an existing confirmation streak.
+For each turn on an unlatched session, Switchyard:
-## Configure an escalation route
+1. Calls the weak target and buffers its reply.
+2. Appends that reply to the transcript and asks the judge to rule on the
+ completed turn. The judge therefore rates work the weak model actually did,
+ not a prediction about work it might do.
+3. Increments a consecutive-escalate streak on an escalate verdict, and resets it
+ to zero on a decline.
+4. Serves the buffered weak reply when the streak has not yet reached
+ `confirmations` — so a judged turn that does not escalate costs one weak call
+ plus one judge call, and no strong call.
+5. Discards the buffered weak reply and serves the strong target instead once the
+ streak reaches `confirmations`. That turn is billed for a weak call, a judge
+ call, and a strong call.
-Configure escalation routing in the `routes:` bundle loaded by
-`--routing-profiles`.
-
-```yaml
-routes:
- agent-escalation:
- type: escalation_router
- fallback_target_on_evict: strong
- judge:
- model: google/gemini-3.5-flash
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 5.0
- min_turn: 3
- confirmations: 1
- confirmation_window: 1
- recent_turn_window: 14
- weak:
- model: moonshotai/kimi-k2.6
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- strong:
- model: anthropic/claude-opus-4.7
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
-```
+A latched session routes straight to the strong target with no judge call:
-Run the route as a standalone proxy:
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%
+flowchart LR
+ t["turn"] --> p{"streak >= confirmations?"}
+ p -->|yes| s["route strong; skip judge"]
+ p -->|no| c["call weak, buffer reply"]
+ c --> j["judge the completed turn"]
+ j -->|decline: streak = 0| w["serve buffered weak reply"]
+ j -->|escalate, not yet confirmed| w
+ j -->|escalate, confirmed| l["discard weak reply; serve strong"]
-```bash
-switchyard serve --routing-profiles routes.yaml --port 4000
+ classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;
+ class t,p,s,c,j,w,l box;
```
-The route ID (`agent-escalation`) is the model ID clients select to use the
-router. The strong and weak model IDs are also registered as direct
-passthrough choices. The judge is internal to the route and is not exposed as
-a client-selectable model.
+A judge that times out, errors, or returns an unparseable verdict fails open: the
+turn serves the buffered weak reply and the existing streak is held rather than
+cleared. A judge failure never creates a strong-tier latch.
-If the selected tier exceeds its context window, Switchyard retries once on
-`fallback_target_on_evict`, which must match one of the configured tier ids
-(`strong` / `weak` unless the targets set their own `id`). See
-[Context-Window Handling](../operations/context_window.md).
+## Tuning options
-## Useful options
+The judge exposes three settings. Their defaults are the benchmarked
+configuration, so a bare `escalation = {}` is a valid, tuned route:
-| Option | Default | Use it when |
+| Key | Default | Meaning |
|---|---|---|
-| `judge.timeout_secs` | `5.0` | The judge needs a different wall-clock limit. Timeouts fail open to weak. |
-| `judge.min_turn` | `3` | The judge should start earlier or wait for more trajectory evidence. |
-| `judge.confirmations` | `1` | One positive verdict is too eager and escalation should require repeated evidence. |
-| `judge.confirmation_window` | `1` | Intermittent trouble should remain eligible for confirmation across negative verdicts. |
-| `judge.disable_reasoning` | `true` | Set to `false` when a reasoning judge benefits from thinking despite the added latency. The thinking-off hint is only sent to models that accept it (Claude/Bedrock judges never receive it). |
-| `judge.max_completion_tokens` | auto | The judge's completion budget needs an explicit value. The default follows the reasoning mode: `128` when the thinking-off hint is sent, `4096` otherwise (reasoning tokens and the JSON verdict share the budget; a too-small cap silently fails open every turn). Judges that never accept the hint (Claude/Bedrock) also get the larger ceiling — it is a cap, not spend. |
-| `judge.dump_verdicts` | `false` | Benchmark runs need per-turn `escalation_verdict=` stderr lines (includes a first-user `task_hint` snippet). Keep off in production; routing telemetry flows through the stats endpoints. |
-| `judge.recent_turn_window` | `14` | The judge needs a wider or narrower trailing-message window. |
-| `judge.window_message_chars` | `300` | More tool-output detail should survive per-message truncation. |
-| `judge.max_request_chars` | `12000` | The complete judge request needs a different character budget. Oldest recent messages are dropped first. |
-| `judge.prompt` | built-in | A deployment needs a custom escalation rubric inlined in the YAML. |
-| `judge.prompt_path` | built-in | The custom rubric lives in a file (mutually exclusive with `judge.prompt`; relative paths resolve against the server's working directory). |
-| `tier_timeout_s` | `600` | Strong or weak targets without their own `timeout_secs` need a different call timeout. |
-| `enable_stats` | `true` | Set to `false` only when route-level usage statistics are not needed. |
-| `affinity_max_sessions` | `10000` | A long-lived process needs a different in-memory latch capacity. |
-| `affinity_store` | `memory` | Set to `redis` to share the escalation latch across workers and keep it across pod churn; with `memory`, a worker change silently restarts an escalated conversation on weak. Best-effort — a store error never fails a request. |
-| `affinity_store_url` | — | Connection URL for the shared latch store (e.g. `redis://host:6379/0`); required with `affinity_store: redis`. |
-| `affinity_store_ttl_seconds` | `3600` | A shared latch entry should expire on a different horizon. |
-| `affinity_key_prefix` | `swyd:esc:` | Multiple deployments share one Redis and need namespaced keys. |
-| `session_key_depth` | `0` | Repeated benchmark trials with identical task prefixes need separate latches; keep `0` for normal traffic. |
+| `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. |
+| `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. |
+| `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. |
-## Observability
+`confirmations` is the main cost dial. `1` latches sooner and spends more on the
+strong tier. `2` or higher requires a session identity, because the streak is
+retained per session — without one, every turn starts from zero and the route
+never latches. Clients supply it with `x-switchyard-session-id`.
-Read the standard routing stats endpoint:
+Everything else is fixed: anchor and transcript caps, the judge's reply budget,
+and the streak rule that any decline resets to zero.
+
+## Run the route
+
+After building the Rust server, as described in
+[Getting Started](../getting_started.md#build-the-server), export the provider
+credential, validate the configuration, and start the binary:
```bash
-curl -s http://localhost:4000/v1/routing/stats
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
+./target/release/switchyard-server --config routes.toml --dry-run
+./target/release/switchyard-server --config routes.toml \
+ --host 127.0.0.1 --port 4000
```
-The snapshot reports per-model calls, tokens, latency, and cost for the strong
-and weak tiers. Judge calls are recorded in the classifier stats bucket so
-their token cost, latency, and errors remain visible as routing overhead.
+Send a request using the route ID, supplying a session identity so the streak
+persists across turns:
-With `judge.dump_verdicts: true` (off by default), each judged turn also
-writes an `escalation_verdict={...}` JSON line to server stderr. The record
-includes the decision, reason, turn, confirmation state, and judge latency.
-After the latch fires, later turns skip the judge and report the pinned
-routing source in request metadata.
+```bash
+curl http://localhost:4000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -H "x-switchyard-session-id: demo-session" \
+ -d '{"model":"agent","messages":[{"role":"user","content":"hello"}]}'
+```
-## Repeated benchmark trials
+Invalid settings are rejected when the configuration loads rather than on the
+first request, so `--dry-run` catches them.
-By default, the session key is derived from the system prompt and first user
-message. Repeated trials of the same task against one long-lived server can
-therefore share a latch: if the first trial escalates, later trials may start
-on strong.
+## Observability
-Use one of these isolation strategies:
+Read the standard stats endpoint:
-- Start a fresh Switchyard process for each trial set.
-- Set `session_key_depth: N` to extend the key with the first `N` messages
- after the initial user message. This only separates trials when those early
- trajectories differ, so it requires nonzero sampling temperature.
+```bash
+curl -s http://localhost:4000/v1/stats
+```
-Keep `session_key_depth: 0` for normal traffic. Context compaction or other
-mid-session prefix rewrites can change a deep key and lose the existing latch.
+The snapshot reports per-model calls, tokens, latency, and cost for the strong
+and weak tiers. Judge calls are recorded in the classifier stats bucket, so their
+token cost and latency remain visible as routing overhead.
## When not to use escalation routing
-- **One-shot requests.** There is no trajectory to judge. Use
- [LLM Classifier Routing](llm_classifier_routing.md) when the initial request
- should determine the tier.
-- **Fixed traffic experiments.** Use
- [Random Routing](random_routing.md) for A/B splits and gradual traffic ramps.
+- **One-shot requests.** No trajectory to judge. Use
+ [LLM Classifier Routing](llm_classifier_routing.md) without the `escalation`
+ table.
+- **Traffic without session identity.** With `confirmations` above `1`, the route
+ cannot accumulate a streak and never latches.
+- **Fixed traffic experiments.** Use [Random Routing](random_routing.md).
- **Per-turn stage optimization.** Use
- [Stage-Router Routing](stage_router_routing.md) when tool-result signals
- should move individual turns in both directions.
-- **Latency-critical traffic.** Eligible unlatched turns wait for the judge
- before the selected backend call.
-- **Long-range failure cycles.** The judge only sees a bounded recent window;
- cycles longer than that window may be missed.
+ [Stage-Router Routing](stage_router_routing.md) when signals should move
+ individual turns in both directions.
+- **Latency-critical traffic.** An unlatched turn waits for the weak call and then
+ the judge call.
## Related
- [Routing Overview](overview.md): compare all supported routing strategies.
-- [Sticky Routing](sticky_routing.md): session-key derivation and affinity
- behavior.
-- [Architecture](../architecture.md): the end-to-end request lifecycle and
- system boundaries.
+- [LLM Classifier Routing](llm_classifier_routing.md): pick a tier up front
+ instead of judging the run.
+- [Architecture](../architecture.md): the end-to-end request lifecycle and system
+ boundaries.
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index 676e60b73..0f09533cf 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -1,108 +1,130 @@
# LLM Classifier Routing
-LLM classifier routing asks a classifier model to evaluate each request, then
-sends the request to a `weak` or `strong` backend. Use it when routing should
-depend on request content, tool use, context needs, or risk level instead of a
-fixed traffic split.
+LLM classifier routing asks a judge model whether the efficient model is likely
+to complete a request, then routes the request to a weak or strong target.
-The classifier runs before the selected backend. Low-confidence and abstained
-results use the configured default tier. Classifier errors do the same when
-`classifier_fail_open` is enabled, which is the default. The built-in two-tier
-policies default to `strong`.
+## Configure a classifier route
-## Choose a policy
+This example uses the packaged classifier prompt as intended: it estimates
+whether the weak target can complete the task, and keeps the first routing
+decision for later requests in the same conversation.
+
+```toml
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.classifier]
+id = "openai/gpt-4o-mini"
+llm_client = "openrouter"
+
+[targets.strong]
+id = "openai/gpt-4o"
+llm_client = "openrouter"
+
+[targets.weak]
+id = "z-ai/glm-5.2"
+llm_client = "openrouter"
+
+[routes.smart]
+id = "smart"
+type = "llm_classifier"
+classifier_target = "classifier"
+strong_target = "strong"
+weak_target = "weak"
+base_threshold = 0.5
+session_affinity = true
+message_hash_fallback = true
+```
-Set `profile` for the traffic you expect:
+`message_hash_fallback` is best-effort: independent sessions with the same
+first user message share an affinity key. Prefer an explicit
+`x-switchyard-session-id` when repeated opening prompts are possible.
-| `profile` | Use for | Default tier mapping |
-|---|---|---|
-| `general` | Mixed chat or API traffic | `simple` uses `weak`; all higher tiers use `strong`. |
-| `coding_agent` | Claude Code, Codex, Cursor-style agents | `simple` and `medium` use `weak`; `complex` and `reasoning` use `strong`. Tool-planning turns can escalate. |
-| `openclaw` | OpenClaw personal-assistant traffic | `simple` and `medium` use `weak`; `complex` and `reasoning` use `strong`. Tool orchestration and high-risk external actions can escalate. |
+The target table names are local references. Their `id` values are the model
+identifiers sent to the upstream provider. The route's `id`, `smart`, is the
+model name clients send to Switchyard.
-For coding-agent traffic, start with `profile: coding_agent`.
+## How the decision works
-## Configure a classifier route
+The classifier target returns a structured verdict containing:
-Define the strong, weak, and classifier models in a `deterministic` route:
-
-```yaml
-defaults:
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
-
-routes:
- smart:
- type: deterministic
- profile: coding_agent
- classifier:
- model: nvidia/nemotron-3-nano-30b-a3b
- min_confidence: 0.6
- fail_open: true
- recent_turn_window: 4
- strong:
- model: openai/gpt-4o
- weak:
- model: openai/gpt-4o-mini
- fallback_target_on_evict: strong
-```
+- `p_solve`: the estimated probability that the weak model completes the task.
+- `confidence`: confidence in the assessment.
+- `capability_boundary`: `supported`, `uncertain`, `unsupported`, or `unmatched`.
-Start the server with:
+For a usable verdict, Switchyard routes to `weak_target` when `p_solve` is
+greater than or equal to the applicable threshold. Otherwise it routes to
+`strong_target`:
-```bash
-switchyard serve --routing-profiles routes.yaml --port 4000
-```
+- `supported` uses `base_threshold`.
+- `uncertain`, `unsupported`, and `unmatched` use
+ `capability_elevated_floor` when configured, or `base_threshold` otherwise.
-The route ID (`smart`) is the model ID clients select for classifier-based
-routing.
+An abstention, an invalid or unparseable verdict, a judge failure, or confidence
+below `min_confidence` routes to `strong_target`. Raising either threshold sends
+more traffic to the strong model.
-Try the profile with representative requests:
+## Tuning options
-```bash
-# Coding task: expected to use the strong tier.
-curl -X POST http://localhost:4000/v1/chat/completions \
- -H "Authorization: Bearer dummy" -H "Content-Type: application/json" \
- -d '{"model":"smart","messages":[{"role":"user","content":"Plan and implement a multi-file API change."}],"max_tokens":200}'
-
-# Simple question: expected to use the weak tier.
-curl -X POST http://localhost:4000/v1/chat/completions \
- -H "Authorization: Bearer dummy" -H "Content-Type: application/json" \
- -d '{"model":"smart","messages":[{"role":"user","content":"What is 2+2? Reply with just the number."}],"max_tokens":50}'
-```
+| Key | Default | Meaning |
+|---|---|---|
+| `base_threshold` | required | Lowest `p_solve` that routes a supported task to `weak_target`. Must be between `0` and `1`. |
+| `min_confidence` | `0.0` | Lowest judge confidence that permits weak routing. Must be between `0` and `1`. |
+| `capability_elevated_floor` | unset | Higher threshold for uncertain, unsupported, and unmatched tasks. It must be between `0` and `1` and greater than `base_threshold`. |
+| `recent_turn_window` | unset | When unset, the judge sees only the newest user message. When set to `N`, it sees client system/developer instructions, the opening user task, and the last `N` conversation messages after that task. `0` keeps only the instructions and opening task. |
+| `session_affinity` | `false` | Retains the first selected target for a session and reuses it on later requests. |
+| `message_hash_fallback` | `false` | When session metadata is absent, keys affinity from the first user-message text. Requires `session_affinity = true`. |
-Treat these as smoke checks, not fixed test vectors: the classifier model and
-prompt determine the verdict.
+### Classifier calibration
-## Useful options
+The packaged prompt is calibrated for one specific weak model, assumes it
+receives the initial task, and describes a sticky routing decision. It is not a
+model-neutral weak-target evaluator.
-| Configuration path | Use it when |
-|---|---|
-| `classifier.min_confidence` | Low-confidence results should use the default tier instead of the classifier policy. |
-| `classifier.fail_open` | Classifier errors should use the default tier rather than fail the client request. |
-| `classifier.recent_turn_window` | The classifier needs more or less recent conversation and tool context. |
-| `alignment_min_confidence` | A classifier recommendation should only raise the policy tier above this confidence. |
-| `default_tier` | Abstain, low-confidence, and fail-open decisions should use a tier other than the default `strong`. |
-| `tier_mapping` | The four classifier policy tiers need a custom mapping to `weak` or `strong`. |
+Without affinity, the runtime judges every request; by default, it sends the
+newest user message rather than the initial task. The example enables affinity
+and its message-hash fallback so the first request is judged and later requests
+with the same opening task reuse that decision. If you use a different weak
+model or per-request routing, tune the thresholds before treating classifier
+decisions as production policy.
-For a self-hosted strong, weak, or classifier target, configure it like any
-other OpenAI-compatible endpoint. See
-[Self-hosted targets](overview.md#self-hosted-targets).
+Support for supplying your own classifier prompt is coming soon.
## Session affinity
-LLM classifier routing supports optional session affinity through
-`DeterministicRoutingConfig`. Set `session_affinity: true` to share one affinity
-store between the classifier and tier selector. After any configured
-`affinity_warmup_turns`, the first confident verdict pins the tier. Later turns
-reuse that tier before classification, so they skip the classifier call;
-abstain, low-confidence, missing-signal, and fail-open decisions do not pin.
-
-Configure these fields on the `type: deterministic` entry in the `routes:`
-bundle. See [Session Affinity](sticky_routing.md) for YAML and
-[How session affinity composes](overview.md#how-session-affinity-composes) for
-the interaction with routing decisions.
-
-If the per-request classifier cost is too high, use
-[Stage-Router Routing](stage_router_routing.md), which can route many turns from tool and
-agent-progress signals without an extra classifier call.
+With `session_affinity = true`, the first selected target is retained for the
+request's session identity. This includes `strong_target` when it was selected
+as the fallback for an unavailable or unusable judge verdict. There is no
+warmup period. Later requests with the same identity reuse the target before
+classification, so the judge call is skipped.
+
+Affinity is process-local. Clients can send `x-switchyard-session-id`, or enable
+`message_hash_fallback` to key requests without session metadata from the first
+user-message text.
+
+## Run the route
+
+After [building the Rust server](../getting_started.md#build-the-server), export
+the provider credential, validate the configuration, and start the release
+binary:
+
+```bash
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
+./target/release/switchyard-server --config routes.toml --dry-run
+./target/release/switchyard-server --config routes.toml \
+ --host 127.0.0.1 --port 4000
+```
+
+Send a request using the route ID:
+
+```bash
+curl http://localhost:4000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{"model":"smart","messages":[{"role":"user","content":"Explain why the sky appears blue."}]}'
+```
+
+Treat the selected target as model-dependent output, not a fixed test result.
diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.md
index f6fcc8efe..c595cab6f 100644
--- a/docs/routing_algorithms/overview.md
+++ b/docs/routing_algorithms/overview.md
@@ -1,160 +1,134 @@
# Routing Overview
-The Python CLI loads routing policies from a YAML bundle. Each key under
-`routes:` becomes a model ID available through OpenAI Chat Completions,
-Anthropic Messages, and OpenAI Responses requests:
-
-```bash
-switchyard serve --routing-profiles routes.yaml --port 4000
-```
+The `switchyard-server` binary loads a native TOML deployment. Each route's `id`
+becomes a model ID available through OpenAI Chat Completions, Anthropic Messages,
+and OpenAI Responses requests.
Use this page to choose a routing strategy, then open its detailed page for
-configuration and tuning.
+configuration and tuning. For the vocabulary these pages use, see
+[Core Concepts](../core_concepts.md).
## Choose a strategy
| Strategy | Use it when | Route `type` |
|---|---|---|
-| [Random Routing](random_routing.md) | You need a fixed strong/weak split for A/B tests, baselines, or cost experiments. | `random_routing` |
-| [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `deterministic` |
+| [Random Routing](random_routing.md) | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | `random` |
+| [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm_classifier` |
| [Stage-Router Routing](stage_router_routing.md) | Tool-result and agent-progress signals should route most turns without an extra classifier call. | `stage_router` |
-| [Escalation-Router Routing](escalation_router_routing.md) | Start every task on the weak tier and escalate to strong when an LLM judge detects trouble. | `escalation_router` |
-
-[Session Affinity (Sticky Routing)](sticky_routing.md) is an opt-in feature of
-LLM classifier routing, not a standalone routing strategy.
+| [Escalation-Router Routing](escalation_router_routing.md) | Start every task on the weak tier and escalate to strong when an LLM judge detects trouble. | `llm_classifier` with `escalation` |
## Common route shape
-Provider defaults can be shared across all routes, while each route owns its
-target configuration:
-
-```yaml
-defaults:
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
-
-routes:
- fast:
- type: model
- target: openai/gpt-4o-mini
-
- smart:
- type: random_routing
- strong:
- model: openai/gpt-4o
- weak:
- model: openai/gpt-4o-mini
- strong_probability: 0.3
- fallback_target_on_evict: strong
+A deployment has three layers. LLM clients describe how to reach a provider,
+targets name models on those clients, and routes decide which target serves a
+request:
+
+```toml
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.strong]
+id = "openai/gpt-4o"
+llm_client = "openrouter"
+
+[targets.weak]
+id = "openai/gpt-4o-mini"
+llm_client = "openrouter"
+
+[routes.fast]
+id = "fast"
+type = "passthrough"
+target = "weak"
+
+[routes.smart]
+id = "smart"
+type = "random"
+targets = ["strong", "weak"]
+weights = [3, 7]
```
-Use the route name (`fast` or `smart`) as the request's model ID. A single
-bundle can serve multiple routes on the same host and port.
+Clients send a route's `id` (`fast` or `smart`) as the request's model ID. The
+table name in `[routes.smart]` is local to the file; only `id` is visible to
+clients. A single deployment can serve multiple routes on the same host and port.
+
+Credentials stay outside the file: `api_key_env` names an environment variable
+that is read at startup. Omit it for an upstream that needs no credential, such
+as a local model server.
The examples use model IDs from the
[OpenRouter model catalog](https://openrouter.ai/api/v1/models). Select IDs
available to your account before deploying; catalog availability can change.
-## Model routes
+## Direct model routes
-- `type: model` registers one explicit model alias without model discovery.
+A `passthrough` route registers one target under one model ID with no routing
+decision:
-This creates a direct, single-target chain. Use a routing policy when requests
-must be split or classified across targets. There is no catalog auto-discovery,
-so to expose several upstream models, add one `type: model` route per model id.
+```toml
+[routes.fast]
+id = "fast"
+type = "passthrough"
+target = "weak"
+```
+
+Use a routing strategy instead when requests must be split or classified across
+targets. There is no catalog auto-discovery, so to expose several upstream
+models, add one `passthrough` route per model.
## Self-hosted targets
-Any route target can point at an OpenAI-compatible model server you operate.
-For example, start a local vLLM server:
+Any target can point at an OpenAI-compatible model server you operate. For
+example, start a local vLLM server:
```bash
vllm serve ./my-rl-qwen --served-model-name my-rl-qwen --port 8000
```
-Then configure it as a normal route:
+Then declare it as its own LLM client and target. The client needs no
+`api_key_env` when the server does not require a credential:
+
+```toml
+[llm_clients.local_vllm]
+format = "openai_chat"
+base_url = "http://localhost:8000/v1"
-```yaml
-routes:
- local:
- type: model
- target: my-rl-qwen
- base_url: http://localhost:8000/v1
- api_key: dummy
- format: openai
+[targets.local]
+id = "my-rl-qwen"
+llm_client = "local_vllm"
+
+[routes.local]
+id = "local"
+type = "passthrough"
+target = "local"
```
-Switchyard does not start or manage the model server; it only sends requests
-to the configured endpoint.
-
-## Sub-agent override (`subagent_target`)
-
-Any route may name an optional `subagent_target` in its common envelope,
-alongside `type`. A request carrying a recognized sub-agent signal — Claude Code
-agent-lineage headers, Codex delegated-work kinds (`x-openai-subagent:
-collab_spawn` or `review`), or an explicit `x-switchyard-is-subagent: true` —
-bypasses the route's own chain and runs as a direct passthrough to that target:
-
-```yaml
-routes:
- assistant:
- type: model
- target:
- model: strong-model
- subagent_target:
- model: cheaper-worker-model
+Switchyard does not start or manage the model server; it only sends requests to
+the configured endpoint.
+
+## Run a deployment
+
+After building the Rust server, as described in
+[Getting Started](../getting_started.md#build-the-server), export the provider
+credential, validate the configuration, and start the binary:
+
+```bash
+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
+./target/release/switchyard-server --config routes.toml --dry-run
+./target/release/switchyard-server --config routes.toml \
+ --host 127.0.0.1 --port 4000
```
-This keeps a sub-agent loop on one intentional, cache-compatible target instead
-of re-routing every worker turn. The worker may live on a different provider
-entirely — give it its own `base_url` and `api_key` and one route spans two
-upstreams:
-
-```yaml
- subagent_target:
- model: my-local-model
- base_url: http://localhost:8000/v1
- api_key: dummy
- format: openai
+Send a request using a route ID:
+
+```bash
+curl http://localhost:4000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{"model":"smart","messages":[{"role":"user","content":"hello"}]}'
```
-Detection is the protocol crate's canonical policy, shared with the libsy
-`SubagentOverride` classifier, so both engines agree on what counts as
-delegated work. Harness-maintenance turns (`compact`, `memory_consolidation`)
-and unrecognized kinds stay on normal routing, as does everything else when the
-field is absent. A worker-target failure surfaces as a normal target error — it
-is never silently re-routed through the route's own chain.
-
-To suppress sub-agent routing for a request that carries a recognized signal,
-send `x-switchyard-is-subagent: false`. This explicit header overrides Codex and
-Claude Code lineage signals in either direction: `false` keeps the request on
-normal routing even when delegated-work headers are present, and `true` marks a
-request as a sub-agent even when no harness headers appear.
-
-The key applies to `model`, `deterministic`, `escalation_router`,
-and `stage_router` routes. `random_routing` expands into its table entries on a
-separate path and does not consume it.
-
-## How session affinity composes
-
-Session affinity is configured directly on the LLM classifier router. After
-the configured warmup, the first confident policy, tool-planning, or alignment
-verdict pins the tier. Abstain, low-confidence, missing-signal, and fail-open
-decisions never pin. Later turns reuse the tier and skip the classifier call.
-
-Pins use a bounded in-process LRU keyed from the stable conversation prefix.
-They are not shared across workers or restarts. See
-[Sticky Routing](sticky_routing.md) for configuration and key derivation.
-
-Random and stage-router routing make a fresh routing decision for each request.
-The escalation router instead uses affinity as a one-way escalation latch: only
-the strong tier is pinned, and the judge runs until that latch fires.
-
-## Rust server configuration
-
-The `switchyard-server` binary has a separate TOML schema that explicitly
-constructs LLM clients, targets, and libsy algorithms. It does not load Python
-route bundles. See the
-[Rust server README](../../crates/switchyard-server/README.md) for its supported
-configuration.
+For the complete TOML schema and every route option, refer to the
+[TOML Schema](../reference/toml_schema.md).
diff --git a/docs/routing_algorithms/random_routing.md b/docs/routing_algorithms/random_routing.md
index 17da19eb0..3af82f12e 100644
--- a/docs/routing_algorithms/random_routing.md
+++ b/docs/routing_algorithms/random_routing.md
@@ -77,8 +77,9 @@ therefore use different targets.
## Run the route
-After [building the Rust server](../getting_started.md#install), export the
-provider credential, validate the configuration, and start the release binary:
+After [building the Rust server](../getting_started.md#build-the-server), export
+the provider credential, validate the configuration, and start the release
+binary:
```bash
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
diff --git a/docs/routing_algorithms/sticky_routing.md b/docs/routing_algorithms/sticky_routing.md
deleted file mode 100644
index 404d798aa..000000000
--- a/docs/routing_algorithms/sticky_routing.md
+++ /dev/null
@@ -1,80 +0,0 @@
-# Sticky Routing
-
-By default, routing is recomputed on every request, so a single task can hop
-between models mid-conversation. This throws away the upstream prompt/KV cache
-and, on the classifier router, pays for an LLM classifier call every turn.
-
-**Session affinity**, also called sticky routing, is an opt-in feature configured
-directly on the deterministic LLM-classifier router. It pins the routing
-decision for a conversation and reuses it for later turns, so a task stays on
-one model tier. The router can optionally delay that pin for a fixed number of
-warmup turns before committing to a tier.
-
-## How a conversation is identified
-
-No client session ID is required. A conversation is keyed by hashing the stable
-prefix made from the **system prompt + the first user message**. Agent harnesses
-do not rewrite that prefix, so every turn of one task maps to the same key while
-distinct tasks differ. The key is derived per request and the pin store is a bounded
-(least-recently-used) in-process map.
-
-## Configuration
-
-Opt in with `session_affinity: true` on the route. `affinity_max_sessions`
-(default `10000`) caps the number of pinned conversations.
-
-These fields are part of the deterministic router configuration, not a
-standalone route type. Configure them under a `deterministic` entry in the
-`routes:` bundle loaded with `--routing-profiles`.
-
-For deterministic routes, `affinity_warmup_turns` controls how many initial
-turns remain non-sticky. The default is `0`, which preserves the historical
-"pin the first confident verdict" behavior.
-
-### Deterministic (LLM-classifier) router
-
-```yaml
-routes:
- llm-classifier:
- type: deterministic
- profile: coding_agent
- session_affinity: true # pin a confident tier per conversation
- affinity_max_sessions: 10000
- affinity_warmup_turns: 2 # turns 1-2 classify normally; turn 3 can pin
- fallback_target_on_evict: strong
- classifier:
- model: openai/gpt-4o-mini
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- strong:
- model: openai/gpt-4o
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- weak:
- model: openai/gpt-4o-mini
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
-```
-
-The first **confident** classifier verdict after the warmup period is pinned and
-reused. With `affinity_warmup_turns: 2`, turns 1 and 2 route normally and cannot
-read or write a pin; turn 3 is the first turn that can commit a confident tier.
-Once pinned, the classifier **skips its LLM call entirely**. It runs only until
-the route has a committed tier for the task. Fallback decisions (abstain / low
-confidence) are never pinned, so a transient classifier failure can't lock the
-task to the default tier.
-
-## Behavior notes
-
-- **Off by default.** With `session_affinity: false`, routing is recomputed per
- turn as before.
-- **Warmup.** `affinity_warmup_turns: N` means turns
- `1..N` stay non-sticky; the first eligible commit is turn `N + 1`.
-- **`affinity_max_sessions: 0` with affinity on is rejected** at config load. A
- zero-capacity store retains nothing and would silently disable stickiness.
-- **In-process, per-worker.** Pins live in memory and are not shared across
- workers or restarts.
-- **Degrades safe.** Affinity assumes the system + first-user prefix is stable
- across a task's turns. If a harness injects per-turn churn there, the key
- changes each turn and routing simply reverts to per-turn behavior. No lock,
- no error.
diff --git a/mkdocs.yml b/mkdocs.yml
index 74f6cc2f4..0a624c5a0 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -27,13 +27,13 @@ nav:
- Overview: routing_algorithms/overview.md
- Random Routing: routing_algorithms/random_routing.md
- LLM Classifier Routing: routing_algorithms/llm_classifier_routing.md
- - Sticky Routing: routing_algorithms/sticky_routing.md
- Stage-Router Routing: routing_algorithms/stage_router_routing.md
- Escalation-Router Routing: routing_algorithms/escalation_router_routing.md
- Operations:
- Context-Window Handling: operations/context_window.md
- Reference:
- CLI Reference: cli_reference.md
+ - TOML Schema: reference/toml_schema.md
exclude_docs: |
internal/**
diff --git a/tests/getting_started/test_getting_started.py b/tests/getting_started/test_getting_started.py
index 835be30da..7a6b3c1e6 100644
--- a/tests/getting_started/test_getting_started.py
+++ b/tests/getting_started/test_getting_started.py
@@ -1,17 +1,31 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
-"""Drift checks for the concise getting-started guide."""
+"""Drift checks for the Getting Started guide."""
from pathlib import Path
-def test_getting_started_uses_the_current_cli() -> None:
+def test_getting_started_documents_current_paths() -> None:
guide = (
Path(__file__).resolve().parents[2] / "docs" / "getting_started.md"
).read_text()
+
+ assert guide.index("## Launcher Path") < guide.index("## Server Path")
+ assert 'uv tool install "nemo-switchyard[cli,server]"' in guide
assert "switchyard launch claude --model switchyard" in guide
- assert "switchyard launch codex --model switchyard" in guide
- assert "switchyard serve" not in guide
- assert "switchyard configure" not in guide
- assert "switchyard verify" not in guide
+ assert "cargo build --locked --release -p switchyard-server" in guide
+ assert "./target/release/switchyard-server --config routes.toml --dry-run" in guide
+
+ for legacy_command in (
+ "switchyard configure",
+ "switchyard serve",
+ "switchyard verify",
+ "--routing-profiles",
+ "routes.yaml",
+ ):
+ assert legacy_command not in guide
+
+
+# TODO: Add Python snippet coverage when the supported Rust-backed API is finalized.
+# TODO: Add TOML schema coverage if the guide grows beyond one maintained example.
diff --git a/tests/readme/test_readme.py b/tests/readme/test_readme.py
index 240f5232b..7c5b1a3ee 100644
--- a/tests/readme/test_readme.py
+++ b/tests/readme/test_readme.py
@@ -1,15 +1,27 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
-"""README command drift checks."""
+"""Drift checks for the repository README."""
from pathlib import Path
-def test_readme_uses_the_current_cli() -> None:
+def test_readme_documents_current_paths() -> None:
readme = (Path(__file__).resolve().parents[2] / "README.md").read_text()
+
+ assert readme.index("### Launcher Path") < readme.index("### Server Path")
+ assert 'uv tool install "nemo-switchyard[cli,server]"' in readme
assert "switchyard launch claude --model switchyard" in readme
- assert "switchyard launch codex --model switchyard" in readme
- assert "switchyard serve" not in readme
- assert "switchyard configure" not in readme
- assert "switchyard verify" not in readme
+ assert "cargo build --locked --release -p switchyard-server" in readme
+ assert "./target/release/switchyard-server --config routes.toml --dry-run" in readme
+
+ for legacy_command in (
+ "switchyard configure",
+ "switchyard serve",
+ "switchyard verify",
+ ):
+ assert legacy_command not in readme
+
+
+# TODO: Add Python snippet coverage when the supported Rust-backed API is finalized.
+# TODO: Add TOML schema coverage if the README includes a complete server config.
diff --git a/tests/test_cli_reference_docs.py b/tests/test_cli_reference_docs.py
index 3c5af543d..e846404a9 100644
--- a/tests/test_cli_reference_docs.py
+++ b/tests/test_cli_reference_docs.py
@@ -4,6 +4,7 @@
"""CLI reference drift tests."""
import argparse
+import subprocess
from pathlib import Path
from switchyard.cli.switchyard_cli import _build_parser
@@ -29,9 +30,13 @@ def _long_options(parser: argparse.ArgumentParser) -> set[str]:
}
-def test_reference_documents_launch_only() -> None:
+def test_reference_documents_launcher_before_server() -> None:
text = CLI_REFERENCE.read_text()
- assert "## `switchyard launch`" in text
+ launcher = text.index("## Launcher Path: `switchyard launch`")
+ server = text.index("## Server Path: `switchyard-server`")
+ related = text.index("## Related Documentation")
+
+ assert launcher < server < related
assert "switchyard serve" not in text
assert "switchyard configure" not in text
assert "switchyard verify" not in text
@@ -44,3 +49,38 @@ def test_reference_lists_launcher_contract() -> None:
assert _long_options(parser) == {"--model", "--config"}
assert "--model" in text
assert "--config" in text
+
+
+def test_reference_lists_server_contract() -> None:
+ result = subprocess.run(
+ [
+ "cargo",
+ "run",
+ "--quiet",
+ "--locked",
+ "-p",
+ "switchyard-server",
+ "--",
+ "--help",
+ ],
+ cwd=CLI_REFERENCE.parents[1],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ assert result.returncode == 0, result.stderr
+
+ text = CLI_REFERENCE.read_text()
+ for flag in (
+ "--config",
+ "--host",
+ "--port",
+ "--backlog",
+ "--dry-run",
+ "--tls-cert",
+ "--tls-key",
+ "--help",
+ "--version",
+ ):
+ assert flag in result.stdout
+ assert flag in text