Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
chore(docs): massive docs refactor for QA (#230)
Ayush Agarwald6769656c528
diff --git a/AGENTS.md b/AGENTS.mdindex 4ce6f7e47..3408ad17b 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -354,7 +354,7 @@ uvicorn.run(build_switchyard_app(switchyard), port=4000)- Export new public classes from `switchyard/__init__.py` with `__all__`.- Write unit tests for new roles and bug fixes.- Use `ProxyContext.metadata` for cross-component state within a request.-- In a new `LLMBackend`, map upstream context-window 4xx to `SwitchyardError::ContextWindowExceeded` (Rust) — the chain executor uses it for evict-and-retry. See [Context-Window Handling](docs/context_window.md).+- In a new `LLMBackend`, map upstream context-window 4xx to `SwitchyardError::ContextWindowExceeded` (Rust) — the chain executor uses it for evict-and-retry. See [Context-Window Handling](docs/operations/context_window.md).### Ask first- Modifying `pyproject.toml` dependencies.diff --git a/README.md b/README.mdindex 70178a336..751ad11c6 100644--- a/README.md+++ b/README.md@@ -19,39 +19,80 @@ algorithm you write yourself.## Features- **Protocol Translation**: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats-- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or custom routers-- **Strong Types**: provider-neutral request, response, and streaming types-- **Explicit Configuration**: TOML defines LLM clients, targets, and algorithm routes+- **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm- **Operational Metrics**: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead## Quick Start-Install the CLI and server dependencies:+Choose the launcher path to run Claude Code, Codex CLI, or OpenClaw through+Switchyard. Choose the server path to run Switchyard as a standalone proxy.++### Launcher Path++Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) if it is+not already available, then install the published Switchyard tool:```bash+curl -LsSf https://astral.sh/uv/install.sh | sh+source "$HOME/.local/bin/env"uv tool install "nemo-switchyard[cli,server]"-export OPENROUTER_API_KEY="sk-or-..."```-Launch a coding agent through the packaged OpenRouter deployment:+The coding agent you launch must also be installed and on your `PATH`. This does+not install the standalone `switchyard-server` binary; use the Server Path for+that.++Set an OpenRouter key and launch against the packaged deployment:```bash+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secretswitchyard launch claude --model switchyardswitchyard launch codex --model switchyard+switchyard launch openclaw --model switchyard```-Pass a custom native deployment when needed:+To use your own native TOML deployment, pass its route ID and configuration:```bashswitchyard launch claude --model my-route --config routes.toml```-The deployment schema belongs in the-[`switchyard-server` README](crates/switchyard-server/README.md).+### Server Path-## Architecture+Use this path to build and run the standalone Rust proxy. Install+[Rust with Cargo](https://rust-lang.org/tools/install/), then build the release+binary from source:++```bash+git clone https://github.com/NVIDIA-NeMo/Switchyard.git+cd Switchyard+cargo build --locked --release -p switchyard-server+./target/release/switchyard-server --help+```++Prebuilt binaries are not published yet. `rustup` installs the pinned toolchain+automatically when you run Cargo from the repository.++Create `routes.toml` using the+[Getting Started guide](docs/getting_started.md#server-path), then validate it+and start the server:++```bash+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret+./target/release/switchyard-server --config routes.toml --dry-run+./target/release/switchyard-server --config routes.toml --host 127.0.0.1 --port 4000+```++Verify the proxy in another terminal:++```bash+curl http://localhost:4000/health+```-Switchyard sits between your client applications and one or more LLM backends:+For a complete configuration and a test request, follow+[Getting Started](docs/getting_started.md).++## Architecture```mermaidflowchart LR@@ -65,26 +106,20 @@ flowchart LRClients keep their native OpenAI or Anthropic API format. Switchyard picks aconfigured backend, forwards the request in that backend's own format, and-translates the response back into the shape the client expects.+translates the response back into the shape the client expects. The server+accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each+configured LLM client selects one upstream format.## Documentation+- **[Getting Started](docs/getting_started.md)**: complete launcher and standalone server walkthroughs+- **[Core Concepts](docs/core_concepts.md)**: LLM clients, targets, routes, model IDs, and routing algorithms+- **[Routing Overview](docs/routing_algorithms/overview.md)**: choose and configure a routing algorithm- **[`switchyard-server`](crates/switchyard-server/README.md)**: server configuration, routing algorithms, and metrics- **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application- **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types- **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation-## Supported API Formats--The server exposes:--- OpenAI Chat Completions-- OpenAI Responses-- Anthropic Messages--Configured upstream clients support the same formats. The OpenAI Chat format-also works with compatible servers such as vLLM, NVIDIA NIM, Ollama, and Azure.-## Community- **Issues**: [GitHub Issues](https://github.com/NVIDIA-NeMo/Switchyard/issues)diff --git a/docs/architecture.md b/docs/architecture.mdindex e77dab115..af38d0e8a 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -40,55 +40,23 @@ request signals, or conversation affinity. See the## Backend Wire Format-`BackendFormat` controls the upstream endpoint Switchyard calls. Explicit-formats select an endpoint directly and do not run capability probes.--| Format | Upstream behavior | Use when |-|---|---|---|-| `ANTHROPIC` | Always sends to `/v1/messages`. No probe. | You know the upstream is Anthropic-native (Anthropic API, NIM Claude routes). |-| `RESPONSES` | Always sends to `/v1/responses`. No probe. | You know the upstream supports the OpenAI Responses API. Fails on NIM / non-OpenAI upstreams. |-| `OPENAI` | Always sends to `/v1/chat/completions`. No probe. | You know the upstream is OpenAI-compatible (NIM, OpenRouter, etc). Safe universal choice. |-| `AUTO` | Probes at startup and picks the best format. | The upstream is unknown or varies across deployments. |-| *(omitted)* | Defaults to `OPENAI` with no probe. | Always set `format:` explicitly when the upstream is not OpenAI-compatible. |--### AUTO Decision Tree--```mermaid-flowchart TB- auto["BackendFormat.AUTO"]- chat{"/v1/chat/completions works?"}- openai_chat["OPENAI<br/>/v1/chat/completions"]- messages{"/v1/messages works?"}- anthropic["ANTHROPIC<br/>/v1/messages"]- responses{"/v1/responses works?"}- responses_format["RESPONSES<br/>/v1/responses"]- openai_fallback["OPENAI<br/>/v1/chat/completions fallback"]-- auto -->|"Probe /v1/chat/completions"| chat- chat -->|"Timeout: assume Chat Completions"| openai_chat- chat -->|"Yes"| openai_chat- chat -->|"No (fast 404): probe /v1/messages"| messages- messages -->|"Yes"| anthropic- messages -->|"No: probe /v1/responses"| responses- responses -->|"Yes"| responses_format- responses -->|"No"| openai_fallback-```--Supported inbound and response formats are handled automatically.-`TranslationEngine` converts the client's request to the resolved backend-format and translates the backend response back to the client's expected-format. When a cross-format conversion is required, both directions decode to-and re-encode from the neutral conversation IR. This lets Claude Code, Codex,-OpenClaw, and SDK clients use their native wire format with any supported-upstream format.--> Prefer an explicit format for controlled deployments. It skips capability-> probes and makes the upstream contract clear. Use `AUTO` when provider-> capabilities are unknown or vary across deployments.->-> **`AUTO` costs startup latency.** Each probe is a live request to the upstream,-> so a slow endpoint adds a round-trip per probe, with up to three tried in-> order. Set `format:` explicitly to remove probing.+Each LLM client in the deployment file sets `format`. It fixes the upstream+endpoint and wire format for every target that uses that client.++| `format` | Upstream endpoint |+|---|---|+| `openai_chat` | `/v1/chat/completions` |+| `openai_responses` | `/v1/responses` |+| `anthropic_messages` | `/v1/messages` |++`format` is required. Switchyard does not probe upstreams or select a format+automatically.++Switchyard decodes every inbound request into provider-neutral types before+routing, then encodes it for the selected target's format.+`switchyard-translation` converts requests, buffered responses, and streaming+events between these formats, so Claude Code, Codex, OpenClaw, and SDK clients+keep their native wire format regardless of the upstream a route selects.## Related Documentationdiff --git a/docs/cli_reference.md b/docs/cli_reference.mdindex f0417ddf9..dbe3a11cc 100644--- a/docs/cli_reference.md+++ b/docs/cli_reference.md@@ -1,25 +1,89 @@-# CLI reference+# CLI Reference-The documented CLI workflow launches a coding agent through the native Rust-server.+Switchyard has two command-line paths:-## `switchyard launch`+- **Launcher Path:** `switchyard launch` starts a supported coding agent and+ hosts the native Rust server through the packaged PyO3 binding.+- **Server Path:** `switchyard-server` runs the standalone Rust proxy for API+ clients and custom deployments.++## Launcher Path: `switchyard launch`++Install the launcher with `uv tool install "nemo-switchyard[cli,server]"`. The+selected coding agent must also be installed and available on `PATH`.++### Usage```bash-export OPENROUTER_API_KEY="sk-or-..."-switchyard launch claude --model switchyard+switchyard launch <agent> --model <route-id> [--config <deployment.toml>] [-- <agent args>]```Supported agents are `claude`, `codex`, and `openclaw`.-```text-switchyard launch <agent> --model <route-id> [--config <deployment.toml>] [-- <agent args>]+| Option | Required | Purpose |+|---|:---:|---|+| `--model ID` | Yes | Route ID from the selected deployment. |+| `--config PATH` | No | Native TOML deployment. Defaults to the packaged OpenRouter deployment. |+| `-- ...` | No | Arguments forwarded unchanged to the coding agent. |++### Packaged Deployment++The packaged deployment exposes the route ID `switchyard` and uses OpenRouter:++```bash+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret+switchyard launch claude --model switchyard+switchyard launch codex --model switchyard+switchyard launch openclaw --model switchyard+```++### Custom Deployment++Select a route ID from an explicit native TOML deployment:++```bash+switchyard launch claude --model my-route --config routes.toml+```++The launcher owns the server lifecycle and stops it when the coding agent exits.+This command does not install or run the standalone `switchyard-server` binary.++## Server Path: `switchyard-server`++Build the standalone binary with+`cargo build --locked --release -p switchyard-server`. It reads the same native+TOML deployment schema accepted by the launcher.++### Usage++```bash+switchyard-server --config <deployment.toml> [options]+```++When built from the repository, invoke it as+`./target/release/switchyard-server`.++| Option | Default | Purpose |+|---|---|---|+| `--config PATH` | Required | TOML file defining LLM clients, targets, and algorithm routes. |+| `--host HOST` | `0.0.0.0` | Address on which the server listens. |+| `-p, --port PORT` | `4000` | Port on which the server listens. |+| `--backlog BACKLOG` | `65535` | TCP listen backlog configured before accepting traffic. |+| `--dry-run` | Off | Validate the deployment without binding a socket. |+| `--tls-cert PATH` | None | PEM certificate path; requires `--tls-key`. |+| `--tls-key PATH` | None | PEM private-key path; requires `--tls-cert`. |+| `-h, --help` | — | Print command help. |+| `-V, --version` | — | Print the server version. |++Validate a deployment, then start the proxy:++```bash+./target/release/switchyard-server --config routes.toml --dry-run+./target/release/switchyard-server --config routes.toml \+ --host 127.0.0.1 --port 4000```-| Option | Purpose |-|---|---|-| `--model ID` | Required route ID from the deployment. |-| `--config PATH` | TOML deployment. Defaults to the packaged OpenRouter deployment. |-| `-- ...` | Arguments forwarded to the coding agent. |+## Related Documentation-Custom deployment syntax is documented in `crates/switchyard-server/README.md`.+- [Getting Started](getting_started.md): installation, configuration, and verification for both paths+- [`switchyard-server`](../crates/switchyard-server/README.md): complete TOML schema, TLS, and metricsdiff --git a/docs/core_concepts.md b/docs/core_concepts.mdindex f5cc5cebd..d39fa2b57 100644--- a/docs/core_concepts.md+++ b/docs/core_concepts.md@@ -1,94 +1,111 @@# Core Concepts-This page explains the vocabulary used throughout the documentation. For setup,-see [Getting Started](getting_started.md). For the request lifecycle, see+Switchyard routes LLM requests through provider-neutral Rust types and exposes+the result through OpenAI- and Anthropic-compatible APIs. For setup, see+[Getting Started](getting_started.md). For the complete request lifecycle, see[Architecture](architecture.md).-## How it fits together+## Runtime Surfaces-Switchyard is a proxy. Clients such as coding agents and SDKs talk to it on one-side, while hosted providers, private endpoints, and local model servers sit on-the other. A client sends a model ID, and the route registered under that ID-decides what runs.+Switchyard exposes the same Rust routing core through two runtime surfaces:++- **`switchyard-server`** is the standalone HTTP proxy. It loads a native TOML+ deployment and exposes OpenAI Chat Completions, OpenAI Responses, and+ Anthropic Messages endpoints.+- **`switchyard-libsy`** is the embeddable Rust library. Applications construct+ targets and algorithms directly and can let libsy make calls or fulfill its+ requested model calls themselves.++The `switchyard launch` command is a launcher for coding agents. It hosts the+native Rust server through the packaged PyO3 binding and points the selected+agent at that server.++## Request Flow++The standalone server translates an inbound provider format into the shared+protocol types before routing. The selected target's LLM client translates the+request into its upstream format, makes the call, and translates the response+back to the client's format.```mermaidflowchart LR- client["Client<br/>picks a model ID"]- route["Route<br/>routing policy"]+ client["Client<br/>OpenAI or Anthropic API"]+ request["Decode request<br/>provider-neutral types"]+ route["Route<br/>algorithm decision"]target["Target<br/>upstream model"]- backend["Model backend"]+ upstream["Encode request<br/>upstream format"]+ backend["LLM backend"]+ response["Decode and translate<br/>response or stream"]- client -->|"model field"| route- route -->|"selects"| target- target --> backend+ client --> request --> route --> target --> upstream --> backend+ backend --> response --> client```-## Routes and targets+## LLM Clients, Targets, and Routes++A native TOML deployment has three layers:-The Python CLI loads a YAML bundle with a `routes:` section. Each route has a-client-facing name and a `type` that chooses its behavior. Depending on that-type, the route owns one target, two routing tiers, or a larger endpoint list.+| Layer | Defines |+|---|---|+| **LLM client** | Upstream base URL, wire format, credential environment variable, and retry policy. |+| **Target** | One upstream model ID and the LLM client used to call it. |+| **Route** | One client-visible model ID and the algorithm that selects or calls targets. |-Shared `defaults:` can provide a base URL, API key, and backend format without-repeating them in every target. The [Routing Overview](routing_algorithms/overview.md)-has a complete runnable bundle.+These layers keep provider transport separate from routing policy. Several+targets can share one LLM client, and several routes can reuse the same target.+Secrets stay outside the TOML: `api_key_env` names the environment variable the+server reads at startup.## Model IDs-Every route name is registered as a model ID and listed on `GET /v1/models`.-Clients select a route by putting that name in the request's `model` field.-Some route types also discover a tier's upstream catalog from its-`GET /v1/models` and register those model IDs directly.+The table keys in `llm_clients`, `targets`, and `routes` are local references+inside the TOML file. Their `id` fields have different external meanings:-## Tiers and routing strategies+- A target's `id` is the model ID sent to its upstream provider.+- A route's `id` is the model ID clients send to Switchyard.-Most routing strategies divide traffic between two tiers: a strong target that-is more capable and more expensive, and a weak target that is cheaper and-faster. A tier is a role assigned inside a route, not a fixed property of a-model.+The server lists route IDs on `GET /v1/models`. A request selects a route by+putting that ID in its `model` field. The native Rust server does not discover+or register additional provider models automatically.-A route's `type` sets the strategy. `model` calls one target.-`random_routing` splits traffic on a fixed probability. `deterministic` asks a-classifier model to pick a tier. `stage_router` uses agent-progress signals,-with an optional classifier fallback. `escalation_router` starts on the weak-tier and uses an LLM judge to escalate when needed. The-[Routing Overview](routing_algorithms/overview.md) explains when to use each.+## Routing Algorithms-Session affinity pins a conversation to one tier so later turns reuse it-instead of being classified again. It belongs to deterministic classifier-routing and is not a strategy of its own. Random and stage-router routing make-a decision for every request. [Sticky Routing](routing_algorithms/sticky_routing.md)-covers affinity in full.+An algorithm receives the normalized request, publishes a routing decision,+and serves the selected target. The standalone server supports these primary+route types:-## Formats and translation+| Route type | Behavior |+|---|---|+| `passthrough` | Sends every request to one target. |+| `random` | Selects among targets using optional relative weights. |+| `llm_classifier` | Uses a classifier target to choose between weak and strong targets. |+| `stage_router` | Uses tool-result and progress signals to choose an efficient or capable target. |-Clients reach Switchyard through OpenAI Chat Completions, Anthropic Messages,-or OpenAI Responses. Each target has a backend format: `openai`, `anthropic`,-`responses`, or `auto`. When the inbound and backend formats differ,-Switchyard translates the request on the way out and the response on the way-back.+Strong, weak, capable, and efficient are roles within an algorithm, not fixed+properties of a model. The same upstream model can serve different roles in+different routes.-That translation lets Claude Code, which speaks Anthropic Messages, run against-an OpenAI-compatible model. The [Architecture](architecture.md) page documents-the supported formats and request lifecycle.+## Protocol Types and Translation-## Programmatic Python profiles+`switchyard-protocol` defines provider-neutral requests, responses, messages,+content blocks, tool calls, usage, and streaming events. Algorithms operate on+these types rather than provider SDK objects.-Embedded Python callers can construct typed profile configs and runtimes-directly instead of loading a route bundle. `ProfileInput` carries the request-and request metadata, while the profile protocols define `run`, `process`, and-`rprocess`. This programmatic API is separate from CLI configuration.+Each LLM client explicitly selects one upstream format:-## Rust server configuration+- `openai_chat`+- `openai_responses`+- `anthropic_messages`-The `switchyard-server` binary is built directly on libsy. Its TOML file-explicitly defines LLM clients, targets, and algorithm routes; it does not load-Python route bundles. See the-[Rust server README](../crates/switchyard-server/README.md) for details.+`switchyard-translation` converts requests, buffered responses, and streaming+events between those formats. This lets a client keep its native API while the+selected target uses a different upstream protocol.-## Where to go next+## Where to Go Next-- [Getting Started](getting_started.md) to install and send a first request.-- [Routing Overview](routing_algorithms/overview.md) to choose and tune a strategy.-- [Architecture](architecture.md) to see a request travel end to end.-- [CLI Reference](cli_reference.md) for flags and environment variables.+- [Getting Started](getting_started.md): install and run either execution path.+- [CLI Reference](cli_reference.md): launcher and standalone server arguments.+- [LLM Classifier Routing](routing_algorithms/llm_classifier_routing.md): configure classifier routing.+- [Architecture](architecture.md): follow a request through the server and Rust crates.+- [`switchyard-server`](../crates/switchyard-server/README.md): complete TOML schema, endpoints, and metrics.+- [`switchyard-libsy`](../crates/libsy/README.md): embed and extend routing algorithms in Rust.diff --git a/docs/getting_started.md b/docs/getting_started.mdindex e48eaf263..05729c1b2 100644--- a/docs/getting_started.md+++ b/docs/getting_started.md@@ -1,31 +1,245 @@-# Getting Started+# Getting Started with Switchyard-## Install+Switchyard has two native Rust execution paths:++- **Launcher path:** install the Python-distributed CLI and launch Claude Code,+ Codex, or OpenClaw through the packaged Rust server binding.+- **Server path:** build and run the standalone Rust server for API clients and+ custom deployments.++## Launcher Path++Use this path when you want Switchyard to start and configure a supported coding+agent for you.++### Install the CLI++Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) if it is+not already available:++```bash+curl -LsSf https://astral.sh/uv/install.sh | sh+source "$HOME/.local/bin/env"+```++Then install the published Switchyard tool:```bashuv tool install "nemo-switchyard[cli,server]"```-## Launch a coding agent+This creates an isolated Python tool environment containing the `switchyard`+CLI, its CLI and server dependencies, and the packaged PyO3 Rust extension.+`switchyard launch` starts the native Rust server through that extension; this+path does not install or run the standalone `switchyard-server` binary.++Install the coding agent you want to launch, then verify both commands are on+your `PATH`.-The packaged deployment uses OpenRouter:+### Launch with the packaged OpenRouter deployment++The packaged deployment exposes the route ID `switchyard`. Export an OpenRouter+key and choose an agent:```bash-export OPENROUTER_API_KEY="sk-or-..."+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secretswitchyard launch claude --model switchyard```-Codex and OpenClaw use the same shape:+Codex and OpenClaw use the same deployment:```bashswitchyard launch codex --model switchyardswitchyard launch openclaw --model switchyard```-To use another deployment:+### Launch with a custom deployment++Pass a native TOML deployment and select one of its route IDs:```bash-switchyard launch claude --model my-route --config routes.toml+switchyard launch codex --model my-route --config routes.toml```-See `crates/switchyard-server/README.md` for the TOML schema.+The launcher manages the native server lifecycle and points the selected coding+agent at it. The server configuration format is the same TOML schema used by the+standalone server below.++## Server Path++Use this path when you want a standalone proxy for API clients or need to+operate the Rust server directly.++### Prerequisites++- Git, a native build toolchain, and Rust with Cargo+- An API key for OpenRouter, OpenAI, Anthropic, or another OpenAI-compatible endpoint.+ To use OpenRouter, create an account at [openrouter.ai](https://openrouter.ai/)+ and generate a key from the [OpenRouter keys page](https://openrouter.ai/keys).++On Ubuntu or WSL, install the build prerequisites and Rust with `rustup`:++```bash+sudo apt-get update+sudo apt-get install -y build-essential curl git+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh+source "$HOME/.cargo/env"+```++On macOS or native Windows, follow the+[official Rust installation instructions](https://rust-lang.org/tools/install/).+The Rust installer includes `rustc`, Cargo, and `rustup`.++Install `uv` for the repository's Python-based tooling and CI checks. It is not+required to build or run the Rust server:++```bash+curl -LsSf https://astral.sh/uv/install.sh | sh+```++If either installer updates your shell configuration, restart the shell before+continuing. Verify the tools:++```bash+git --version+rustc --version+cargo --version+uv --version+```++### Build the server++Build the Rust server from source:++```bash+git clone https://github.com/NVIDIA-NeMo/Switchyard.git+cd Switchyard+cargo build --locked --release -p switchyard-server+./target/release/switchyard-server --help+```++The repository pins Rust `1.96.1` in `rust-toolchain.toml`; `rustup` selects and+installs it automatically when Cargo runs from the repository. Prebuilt Rust+binaries are not published yet.++### Configure++The Rust server reads an explicit TOML file. It does not use the legacy Python+CLI's saved configuration or YAML routing profiles.++Create `routes.toml` with an LLM-classifier route:++```toml+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.weak]+id = "openai/gpt-4o-mini"+llm_client = "openrouter"++[targets.strong]+id = "openai/gpt-4o"+llm_client = "openrouter"++[routes.smart]+id = "switchyard"+type = "llm_classifier"+classifier_target = "weak"+strong_target = "strong"+weak_target = "weak"+base_threshold = 0.5+```++`format` selects the upstream protocol and must be `openai_chat`,+`openai_responses`, or `anthropic_messages`. `api_key_env` names the environment+variable the server reads; the secret does not belong in the TOML file.++### Run the server++Export the provider credential, validate the configuration without binding a+socket, then start the release binary:++```bash+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret+./target/release/switchyard-server --config routes.toml --dry-run+./target/release/switchyard-server --config routes.toml \+ --host 127.0.0.1 --port 4000+```++Any client that speaks OpenAI Chat Completions, Anthropic Messages, or OpenAI+Responses API can connect. The route `id` is the model name clients use.++In another terminal:++```bash+curl http://localhost:4000/health+curl http://localhost:4000/v1/models+curl http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'+```++### Routing algorithms++#### Choose a route type++This guide uses `llm_classifier`, which asks a classifier target whether each+request should use the weak or strong target. The Rust server also supports:++| Algorithm | Use it when | Config |+|---|---|---|+| [Random](routing_algorithms/random_routing.md) | You need a weighted split for A/B tests or baselines. | `random` |+| [LLM classifier](routing_algorithms/llm_classifier_routing.md) | Request content should decide whether to use the weak or strong target. | `llm_classifier` |+| [Stage router](routing_algorithms/stage_router_routing.md) | Tool-result and progress signals should select an efficient or capable target. | `stage_router` |++A single TOML file can declare multiple routes. The table key, such as+`routes.smart`, is a local configuration name; each route's `id` is exposed as a+model on `GET /v1/models`.++See [Routing Overview](routing_algorithms/overview.md) to compare strategies,+and the [`switchyard-server` guide](../crates/switchyard-server/README.md) for+the complete TOML schema, route options, TLS, and metrics.++### Troubleshooting++**No API key / auth error**++```bash+test -n "$OPENROUTER_API_KEY" && echo "key is set" || echo "key is missing"+./target/release/switchyard-server --config routes.toml --dry-run+```++Confirm that `api_key_env` in `routes.toml` names the environment variable you+exported. The dry run validates the schema, environment lookup, target+references, and route construction without starting the server.++**Connection refused**++Check health: `curl http://localhost:4000/health`++**Telemetry header opt-out**++Switchyard adds an `X-Switchyard-Version` header to outbound LLM calls for+release attribution. No request or response content is included. To disable:++```bash+export SWITCHYARD_TELEMETRY_OPT_OUT=1+```++---++### Next steps++- [Core Concepts](core_concepts.md): LLM clients, targets, and routes+- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration,+ routing algorithms, TLS, and metrics+- [`switchyard-libsy`](../crates/libsy/README.md): embed routing algorithms in a+ Rust application+- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral+ request, response, and streaming types+- [`switchyard-translation`](../crates/switchyard-translation/README.md):+ request, response, and stream translationdiff --git a/docs/index.md b/docs/index.mdindex 70ba539bc..2ada13bc0 100644--- a/docs/index.md+++ b/docs/index.md@@ -3,26 +3,29 @@Switchyard routes and translates LLM traffic for coding agents and API clients.It supports OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages.-## Start+## Choose a Path-```bash-uv tool install "nemo-switchyard[cli,server]"-export OPENROUTER_API_KEY="sk-or-..."-switchyard launch claude --model switchyard-```+| Goal | Path | Start here |+|---|---|---|+| Run Claude Code, Codex, or OpenClaw through Switchyard | Launcher Path | [Install and launch an agent](getting_started.md#launcher-path) |+| Run Switchyard as a standalone proxy for API clients | Server Path | [Build and run the Rust server](getting_started.md#server-path) |+| Add Switchyard routing to a Rust application | Library Path | [`switchyard-libsy`](../crates/libsy/README.md) |-Use a custom native deployment when needed:+The Launcher Path installs the `switchyard` CLI and hosts the native Rust server+through its packaged PyO3 binding. The Server Path builds and runs the+standalone `switchyard-server` binary.-```bash-switchyard launch codex --model my-route --config routes.toml-```+## Explore-The TOML schema is maintained in `crates/switchyard-server/README.md`.+- [Core Concepts](core_concepts.md): learn the LLM client, target, and route layers+- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model+- [Architecture](architecture.md): understand the proxy and library components+- [CLI Reference](cli_reference.md): inspect supported launcher commands and options+- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior-## Read More+## Reference-- [Getting Started](getting_started.md)-- [CLI Reference](cli_reference.md)-- [Routing Overview](routing_algorithms/overview.md)-- [Architecture](architecture.md)-- [Context-Window Handling](operations/context_window.md)+- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration, endpoints, and metrics+- [`switchyard-libsy`](../crates/libsy/README.md): embeddable routing algorithms+- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral API types+- [`switchyard-translation`](../crates/switchyard-translation/README.md): protocol translationdiff --git a/docs/internal/subagent_override.md b/docs/internal/subagent_override.mdnew file mode 100644index 000000000..e8b59b268--- /dev/null+++ b/docs/internal/subagent_override.md@@ -0,0 +1,54 @@+# Sub-Agent Override++Internal note. Not published: `mkdocs.yml` excludes `internal/**` from the site.++`subagent_target` exists only on the Python route bundle loaded by+`switchyard serve`. The `switchyard-server` TOML schema has no equivalent key.++Any bundle route may name an optional `subagent_target` in its common envelope,+alongside `type`. A request carrying a recognized sub-agent signal — Claude Code+agent-lineage headers, Codex delegated-work kinds (`x-openai-subagent:+collab_spawn` or `review`), or an explicit `x-switchyard-is-subagent: true` —+bypasses the route's own chain and runs as a direct passthrough to that target:++```yaml+routes:+ assistant:+ type: model+ target:+ model: strong-model+ subagent_target:+ model: cheaper-worker-model+```++This keeps a sub-agent loop on one intentional, cache-compatible target instead+of re-routing every worker turn. The worker may live on a different provider+entirely — give it its own `base_url` and `api_key` and one route spans two+upstreams:++```yaml+ subagent_target:+ model: my-local-model+ base_url: http://localhost:8000/v1+ api_key: dummy+ format: openai+```++Detection is the protocol crate's canonical policy, shared with the+`switchyard-libsy` `SubagentOverride` classifier, so both engines agree on what+counts as delegated work. Harness-maintenance turns (`compact`,+`memory_consolidation`) and unrecognized kinds stay on normal routing, as does+everything else when the field is absent. A worker-target failure surfaces as a+normal target error — it is never silently re-routed through the route's own+chain.++To suppress sub-agent routing for a request that carries a recognized signal,+send `x-switchyard-is-subagent: false`. This explicit header overrides Codex and+Claude Code lineage signals in either direction: `false` keeps the request on+normal routing even when delegated-work headers are present, and `true` marks a+request as a sub-agent even when no harness headers appear.++The key applies to bundle routes of type `model`, `deterministic`,+`escalation_router`, and `stage_router`. It is accepted on `random_routing`+routes but has no effect, because they expand into their table entries on a+separate path that never consumes it.diff --git a/docs/operations/context_window.md b/docs/operations/context_window.mdindex a6504e157..4ade38066 100644--- a/docs/operations/context_window.md+++ b/docs/operations/context_window.md@@ -1,36 +1,64 @@# Context-Window HandlingWhen an upstream rejects a request because the prompt exceeds the model's-context window, Switchyard evicts that target for the current request,-reroutes to the configured fallback target, and retries once. If the fallback-also overflows, the request fails with a 400 in the client's inbound wire-format.+context window, Switchyard drops that target and calls another target on the+same route, repeating until a call succeeds or every target has been tried. If+the request carries `x-switchyard-session-id` or a recognized coding-agent+session header such as `x-claude-code-session-id`, the target remains excluded+for the rest of that session. Without a session header, the fallback still+applies to the current request, but the overflow is not remembered.+After truncating or resetting context, clients should use a new session ID;+reusing the old ID preserves its target exclusions.-Any multi-target route (stage-router, random_routing, or deterministic) supports-this. Set `fallback_target_on_evict` on the route. Single-target routes-(`type: model`) have no alternative target, so the original-overflow propagates unchanged.+## What counts as an overflow++Switchyard treats an upstream reply as a context overflow only when the status+is HTTP 400 **and** the body identifies a context-length error: an `error.code`+of `context_length_exceeded`, or a message containing a phrase such as+`maximum context length` or `prompt is too long`. An overflow reported any other+way — HTTP 413 or 422, for example — is not recognized as one. The request+fails on the spot with the upstream's status code, and the upstream's body is+returned inside Switchyard's error message.## Configuration-`fallback_target_on_evict` is required on every multi-target route and must-match one of the route's declared target ids:--```yaml-routes:- my-stage-router:- type: stage_router- picker: capable_first- fallback_target_on_evict: strong # must match strong.id or weak.id- strong:- id: strong- model: anthropic/claude-opus-4.7- weak:- id: weak- model: moonshotai/kimi-k2.6+There is no eviction key. A route falls through whenever it has more than one+target, so the route below needs nothing beyond its two tiers:++```toml+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.strong]+id = "openai/gpt-4o"+llm_client = "openrouter"++[targets.weak]+id = "openai/gpt-4o-mini"+llm_client = "openrouter"++[routes.stage]+id = "switchyard/stage"+type = "stage_router"+capable_target = "strong"+efficient_target = "weak"+picker = "efficient_first"+confidence_threshold = 0.5+```++Response headers report where the request actually landed:++```text+x-model-router-selected-model: openai/gpt-4o-mini+x-model-router-rationale: openai/gpt-4o exceeded its context window; fell back to openai/gpt-4o-mini```-## Scope+## When every target overflows-Single eviction + single retry per request. Compaction, cool-down, and-re-insertion of evicted targets are out of scope.+Once no untried target is left — including a single-target `passthrough` route,+which has no alternative from the start — the request fails with HTTP 400 and an+error `code` of `context_length_exceeded`.diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdnew file mode 100644index 000000000..918e8e3e4--- /dev/null+++ b/docs/reference/toml_schema.md@@ -0,0 +1,143 @@+# TOML Schema++The native deployment file defines the LLM clients, targets, and routes a+Switchyard server serves. It is read by `switchyard-server --config` and by+`switchyard launch --config`.++Validate a file without starting the server:++```bash+switchyard-server --config routes.toml --dry-run+```++## Minimal Example++```toml+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.strong]+id = "anthropic/claude-sonnet-4.5"+llm_client = "openrouter"++[routes.default]+id = "switchyard"+type = "passthrough"+target = "strong"+```++`schema_version` must be `1`. Table names under `llm_clients`, `targets`, and+`routes` are local references; clients send the route's `id` as the model name.++## `[llm_clients.<name>]`++| Key | Required | Default | Meaning |+|---|:---:|---|---|+| `format` | Yes | — | `openai_chat`, `openai_responses`, or `anthropic_messages`. |+| `base_url` | Yes | — | Upstream base URL. |+| `api_key_env` | No | unset | Name of the environment variable holding the key. Omit to send no authentication. |+| `extra_headers` | No | `{}` | Extra HTTP headers sent upstream. |+| `max_retries` | No | `2` | Retry budget, `0`–`10`. |++The TOML never contains the secret itself. `api_key_env` names a variable that+must exist and be non-empty when the server loads.++## `[targets.<name>]`++| Key | Required | Default | Meaning |+|---|:---:|---|---|+| `id` | Yes | — | Exact model ID sent upstream. |+| `llm_client` | Yes | — | Key under `[llm_clients]`. |+| `extra_body` | No | `{}` | Values merged into the upstream request when the request does not already set that key. |++## `[routes.<name>]`++Every route takes `id` and `type`, plus the keys for that type.++### `noop`++Returns a buffered assistant response containing `OK` without calling an+upstream model. Use it for local smoke tests.++### `passthrough`++Sends every request to one target.++| Key | Required | Meaning |+|---|:---:|---|+| `target` | Yes | Target every request is sent to. |++### `random`++Splits traffic across targets. See+[Random Routing](../routing_algorithms/random_routing.md).++| Key | Required | Default | Meaning |+|---|:---:|---|---|+| `targets` | Yes | — | Target names to choose from. |+| `weights` | No | equal | Finite, non-negative relative weights in `targets` order, with at least one positive value. Invalid weights are rejected at load time. |+| `seed` | No | unset | Reproduces the selection sequence. |++### `llm_classifier`++Classifies each task, then routes to the weak or strong target. See+[LLM Classifier Routing](../routing_algorithms/llm_classifier_routing.md) for+tuning.++| Key | Required | Default | Meaning |+|---|:---:|---|---|+| `classifier_target` | Yes | — | Target the judge is called through. Not a routing destination. |+| `strong_target` | Yes | — | Capable tier. |+| `weak_target` | Yes | — | Efficient tier. |+| `base_threshold` | Yes | — | Lowest solve probability that routes to the weak target. In `[0, 1]`. |+| `min_confidence` | No | `0.0` | Lowest judge confidence that permits weak routing. |+| `capability_elevated_floor` | No | unset | Higher floor for uncertain tasks. Must exceed `base_threshold`. |+| `session_affinity` | No | `false` | Reuses a session's first decision on later turns. |+| `message_hash_fallback` | No | `false` | Keys affinity on the first user message. Requires `session_affinity`. |+| `recent_turn_window` | No | unset | Trailing turns the judge sees. |+| `escalation` | No | unset | Switches the route to escalation judging. |++Add an `escalation` table to judge each completed turn instead of classifying up+front. See+[Escalation-Router Routing](../routing_algorithms/escalation_router_routing.md).++| Key | Required | Default | Meaning |+|---|:---:|---|---|+| `confirmations` | No | `2` | Consecutive escalate verdicts required to latch. Above `1` needs a session ID. |+| `recent_turn_window` | No | `28` | Trailing messages shown to the judge. |+| `window_message_chars` | No | `500` | Per-message cap inside that window. |++`base_threshold` stays required, but escalation ignores it and the other+classifier keys.++### `stage_router`++Scores tool signals to pick a tier per turn. See+[Stage-Router Routing](../routing_algorithms/stage_router_routing.md) for the+optional `handoff_notes` and `classifier` tables and for tuning.++| Key | Required | Default | Meaning |+|---|:---:|---|---|+| `capable_target` | Yes | — | Capable tier. |+| `efficient_target` | Yes | — | Efficient tier. |+| `picker` | Yes | — | `capable_first` or `efficient_first`. Tier used when the signals are not confident. |+| `confidence_threshold` | Yes | — | Corroboration a decisive pick needs. In `[0, 1]`. |+| `recent_turn_window` | No | `3` | Trailing tool results the signals are computed over. |+| `capable_system_prompt` | No | unset | System prompt handed to the capable tier. |+| `efficient_system_prompt` | No | unset | System prompt handed to the efficient tier. |++## Validation Errors++`--dry-run` prefixes configuration failures with+`invalid server config <path>:`. Within that wrapper, TOML deserialization+errors start with `failed to parse TOML:`, while errors from validating the+built configuration retain their inner message unchanged.++## Related Documentation++- [CLI Reference](../cli_reference.md)+- [Routing Overview](../routing_algorithms/overview.md)diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.mdindex 90ee9731a..6b179930c 100644--- a/docs/routing_algorithms/escalation_router_routing.md+++ b/docs/routing_algorithms/escalation_router_routing.md@@ -1,198 +1,166 @@# Escalation-Router Routing-Escalation-router routing starts each conversation on a cheaper `weak` model.-An LLM judge watches how the work progresses and moves the conversation to a-more capable `strong` model when it detects sustained, recoverable trouble.-After escalation, the conversation stays on the strong tier for the rest of-the task.--Use it for multi-turn agent workloads where a weak model can handle routine-work but may need rescue after repeated errors, loops, drift, false progress,-or premature completion. Unlike-[LLM Classifier Routing](llm_classifier_routing.md), which predicts how-difficult a request looks, escalation routing judges whether the run is-actually going well.--## How it works--The router keeps one state value per conversation: whether that conversation-has escalated. Conversations are identified from the stable system prompt and-first user message, using the same bounded in-memory store described in-[Sticky Routing](sticky_routing.md).--For each turn, Switchyard:--1. Checks the conversation latch. A latched conversation routes to `strong`- without calling the judge again.-2. Routes an unlatched conversation to `weak`. Before `judge.min_turn`- (default `3`), it skips the judge because there is too little trajectory to- assess.-3. Gives the judge a bounded summary containing the system and first-user- anchors, recent messages, and a coverage note for omitted history.-4. Parses the judge's structured `escalate` decision. When the configured- confirmation policy is satisfied, Switchyard pins the conversation to- `strong` and uses the strong model for the current turn.-5. Fails open to `weak` when the judge times out, errors, or returns invalid- output. A judge failure never creates a strong-tier pin.--The routing decision for one turn is:+Escalation routing starts each conversation on a cheaper weak model. An LLM judge+reads how the work is going and latches the session to a strong model when it+detects sustained trouble.-```mermaid-%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%-flowchart LR- t["turn"] --> p{"strong tier pinned?"}- p -->|yes| s["route strong; skip judge"]- p -->|no| m{"turn >= min_turn?"}- m -->|no| w["route weak"]- m -->|yes| j["judge recent trajectory"]- j -->|stay / fail open| w- j -->|not yet confirmed| w- j -->|confirmed escalation| l["pin and route strong"]+Use it for multi-turn agent workloads where a weak model handles routine work but+may need rescue after repeated errors, loops, or drift. Unlike plain+[LLM Classifier Routing](llm_classifier_routing.md), which predicts how difficult+a request looks before running it, escalation judges whether the run is actually+going well.- classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;- class t,p,s,m,w,j,l box;+## Configure an escalation route++```toml+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.judge]+id = "google/gemini-3.5-flash"+llm_client = "openrouter"++[targets.strong]+id = "anthropic/claude-opus-4.7"+llm_client = "openrouter"++[targets.weak]+id = "moonshotai/kimi-k2.6"+llm_client = "openrouter"++[routes.agent]+id = "agent"+type = "llm_classifier"+classifier_target = "judge"+strong_target = "strong"+weak_target = "weak"+base_threshold = 0.5+escalation = { confirmations = 2, recent_turn_window = 28, window_message_chars = 500 }```-## Confirmation policy+`classifier_target` is the judge. The route's `id`, `agent`, is the model name+clients send; the judge is not exposed as a client-selectable model.-`judge.confirmations` controls how many positive verdicts are required before-the strong-tier latch fires:+`base_threshold` is still required, but escalation ignores it, along with+`min_confidence`, `capability_elevated_floor`, and `session_affinity`.-- `confirmations: 1` is the default and escalates on the first positive- verdict.-- `confirmations: 2` with `confirmation_window: 1` requires positive verdicts- on consecutive judged turns.-- A larger `confirmation_window` allows recurring trouble to confirm even when- negative verdicts occur between positive ones. A positive verdict remains- live across at most `confirmation_window - 1` intervening negative verdicts.+## How the decision works-A judge failure provides no evidence either way: it routes the turn to weak-without clearing an existing confirmation streak.+For each turn on an unlatched session, Switchyard:-## Configure an escalation route+1. Calls the weak target and buffers its reply.+2. Appends that reply to the transcript and asks the judge to rule on the+ completed turn. The judge therefore rates work the weak model actually did,+ not a prediction about work it might do.+3. Increments a consecutive-escalate streak on an escalate verdict, and resets it+ to zero on a decline.+4. Serves the buffered weak reply when the streak has not yet reached+ `confirmations` — so a judged turn that does not escalate costs one weak call+ plus one judge call, and no strong call.+5. Discards the buffered weak reply and serves the strong target instead once the+ streak reaches `confirmations`. That turn is billed for a weak call, a judge+ call, and a strong call.-Configure escalation routing in the `routes:` bundle loaded by-`--routing-profiles`.--```yaml-routes:- agent-escalation:- type: escalation_router- fallback_target_on_evict: strong- judge:- model: google/gemini-3.5-flash- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 5.0- min_turn: 3- confirmations: 1- confirmation_window: 1- recent_turn_window: 14- weak:- model: moonshotai/kimi-k2.6- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- strong:- model: anthropic/claude-opus-4.7- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1-```+A latched session routes straight to the strong target with no judge call:-Run the route as a standalone proxy:+```mermaid+%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%+flowchart LR+ t["turn"] --> p{"streak >= confirmations?"}+ p -->|yes| s["route strong; skip judge"]+ p -->|no| c["call weak, buffer reply"]+ c --> j["judge the completed turn"]+ j -->|decline: streak = 0| w["serve buffered weak reply"]+ j -->|escalate, not yet confirmed| w+ j -->|escalate, confirmed| l["discard weak reply; serve strong"]-```bash-switchyard serve --routing-profiles routes.yaml --port 4000+ classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;+ class t,p,s,c,j,w,l box;```-The route ID (`agent-escalation`) is the model ID clients select to use the-router. The strong and weak model IDs are also registered as direct-passthrough choices. The judge is internal to the route and is not exposed as-a client-selectable model.+A judge that times out, errors, or returns an unparseable verdict fails open: the+turn serves the buffered weak reply and the existing streak is held rather than+cleared. A judge failure never creates a strong-tier latch.-If the selected tier exceeds its context window, Switchyard retries once on-`fallback_target_on_evict`, which must match one of the configured tier ids-(`strong` / `weak` unless the targets set their own `id`). See-[Context-Window Handling](../operations/context_window.md).+## Tuning options-## Useful options+The judge exposes three settings. Their defaults are the benchmarked+configuration, so a bare `escalation = {}` is a valid, tuned route:-| Option | Default | Use it when |+| Key | Default | Meaning ||---|---|---|-| `judge.timeout_secs` | `5.0` | The judge needs a different wall-clock limit. Timeouts fail open to weak. |-| `judge.min_turn` | `3` | The judge should start earlier or wait for more trajectory evidence. |-| `judge.confirmations` | `1` | One positive verdict is too eager and escalation should require repeated evidence. |-| `judge.confirmation_window` | `1` | Intermittent trouble should remain eligible for confirmation across negative verdicts. |-| `judge.disable_reasoning` | `true` | Set to `false` when a reasoning judge benefits from thinking despite the added latency. The thinking-off hint is only sent to models that accept it (Claude/Bedrock judges never receive it). |-| `judge.max_completion_tokens` | auto | The judge's completion budget needs an explicit value. The default follows the reasoning mode: `128` when the thinking-off hint is sent, `4096` otherwise (reasoning tokens and the JSON verdict share the budget; a too-small cap silently fails open every turn). Judges that never accept the hint (Claude/Bedrock) also get the larger ceiling — it is a cap, not spend. |-| `judge.dump_verdicts` | `false` | Benchmark runs need per-turn `escalation_verdict=` stderr lines (includes a first-user `task_hint` snippet). Keep off in production; routing telemetry flows through the stats endpoints. |-| `judge.recent_turn_window` | `14` | The judge needs a wider or narrower trailing-message window. |-| `judge.window_message_chars` | `300` | More tool-output detail should survive per-message truncation. |-| `judge.max_request_chars` | `12000` | The complete judge request needs a different character budget. Oldest recent messages are dropped first. |-| `judge.prompt` | built-in | A deployment needs a custom escalation rubric inlined in the YAML. |-| `judge.prompt_path` | built-in | The custom rubric lives in a file (mutually exclusive with `judge.prompt`; relative paths resolve against the server's working directory). |-| `tier_timeout_s` | `600` | Strong or weak targets without their own `timeout_secs` need a different call timeout. |-| `enable_stats` | `true` | Set to `false` only when route-level usage statistics are not needed. |-| `affinity_max_sessions` | `10000` | A long-lived process needs a different in-memory latch capacity. |-| `affinity_store` | `memory` | Set to `redis` to share the escalation latch across workers and keep it across pod churn; with `memory`, a worker change silently restarts an escalated conversation on weak. Best-effort — a store error never fails a request. |-| `affinity_store_url` | — | Connection URL for the shared latch store (e.g. `redis://host:6379/0`); required with `affinity_store: redis`. |-| `affinity_store_ttl_seconds` | `3600` | A shared latch entry should expire on a different horizon. |-| `affinity_key_prefix` | `swyd:esc:` | Multiple deployments share one Redis and need namespaced keys. |-| `session_key_depth` | `0` | Repeated benchmark trials with identical task prefixes need separate latches; keep `0` for normal traffic. |+| `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. |+| `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. |+| `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. |-## Observability+`confirmations` is the main cost dial. `1` latches sooner and spends more on the+strong tier. `2` or higher requires a session identity, because the streak is+retained per session — without one, every turn starts from zero and the route+never latches. Clients supply it with `x-switchyard-session-id`.-Read the standard routing stats endpoint:+Everything else is fixed: anchor and transcript caps, the judge's reply budget,+and the streak rule that any decline resets to zero.++## Run the route++After building the Rust server, as described in+[Getting Started](../getting_started.md#build-the-server), export the provider+credential, validate the configuration, and start the binary:```bash-curl -s http://localhost:4000/v1/routing/stats+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret+./target/release/switchyard-server --config routes.toml --dry-run+./target/release/switchyard-server --config routes.toml \+ --host 127.0.0.1 --port 4000```-The snapshot reports per-model calls, tokens, latency, and cost for the strong-and weak tiers. Judge calls are recorded in the classifier stats bucket so-their token cost, latency, and errors remain visible as routing overhead.+Send a request using the route ID, supplying a session identity so the streak+persists across turns:-With `judge.dump_verdicts: true` (off by default), each judged turn also-writes an `escalation_verdict={...}` JSON line to server stderr. The record-includes the decision, reason, turn, confirmation state, and judge latency.-After the latch fires, later turns skip the judge and report the pinned-routing source in request metadata.+```bash+curl http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -H "x-switchyard-session-id: demo-session" \+ -d '{"model":"agent","messages":[{"role":"user","content":"hello"}]}'+```-## Repeated benchmark trials+Invalid settings are rejected when the configuration loads rather than on the+first request, so `--dry-run` catches them.-By default, the session key is derived from the system prompt and first user-message. Repeated trials of the same task against one long-lived server can-therefore share a latch: if the first trial escalates, later trials may start-on strong.+## Observability-Use one of these isolation strategies:+Read the standard stats endpoint:-- Start a fresh Switchyard process for each trial set.-- Set `session_key_depth: N` to extend the key with the first `N` messages- after the initial user message. This only separates trials when those early- trajectories differ, so it requires nonzero sampling temperature.+```bash+curl -s http://localhost:4000/v1/stats+```-Keep `session_key_depth: 0` for normal traffic. Context compaction or other-mid-session prefix rewrites can change a deep key and lose the existing latch.+The snapshot reports per-model calls, tokens, latency, and cost for the strong+and weak tiers. Judge calls are recorded in the classifier stats bucket, so their+token cost and latency remain visible as routing overhead.## When not to use escalation routing-- **One-shot requests.** There is no trajectory to judge. Use- [LLM Classifier Routing](llm_classifier_routing.md) when the initial request- should determine the tier.-- **Fixed traffic experiments.** Use- [Random Routing](random_routing.md) for A/B splits and gradual traffic ramps.+- **One-shot requests.** No trajectory to judge. Use+ [LLM Classifier Routing](llm_classifier_routing.md) without the `escalation`+ table.+- **Traffic without session identity.** With `confirmations` above `1`, the route+ cannot accumulate a streak and never latches.+- **Fixed traffic experiments.** Use [Random Routing](random_routing.md).- **Per-turn stage optimization.** Use- [Stage-Router Routing](stage_router_routing.md) when tool-result signals- should move individual turns in both directions.-- **Latency-critical traffic.** Eligible unlatched turns wait for the judge- before the selected backend call.-- **Long-range failure cycles.** The judge only sees a bounded recent window;- cycles longer than that window may be missed.+ [Stage-Router Routing](stage_router_routing.md) when signals should move+ individual turns in both directions.+- **Latency-critical traffic.** An unlatched turn waits for the weak call and then+ the judge call.## Related- [Routing Overview](overview.md): compare all supported routing strategies.-- [Sticky Routing](sticky_routing.md): session-key derivation and affinity- behavior.-- [Architecture](../architecture.md): the end-to-end request lifecycle and- system boundaries.+- [LLM Classifier Routing](llm_classifier_routing.md): pick a tier up front+ instead of judging the run.+- [Architecture](../architecture.md): the end-to-end request lifecycle and system+ boundaries.diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.mdindex 676e60b73..0f09533cf 100644--- a/docs/routing_algorithms/llm_classifier_routing.md+++ b/docs/routing_algorithms/llm_classifier_routing.md@@ -1,108 +1,130 @@# LLM Classifier Routing-LLM classifier routing asks a classifier model to evaluate each request, then-sends the request to a `weak` or `strong` backend. Use it when routing should-depend on request content, tool use, context needs, or risk level instead of a-fixed traffic split.+LLM classifier routing asks a judge model whether the efficient model is likely+to complete a request, then routes the request to a weak or strong target.-The classifier runs before the selected backend. Low-confidence and abstained-results use the configured default tier. Classifier errors do the same when-`classifier_fail_open` is enabled, which is the default. The built-in two-tier-policies default to `strong`.+## Configure a classifier route-## Choose a policy+This example uses the packaged classifier prompt as intended: it estimates+whether the weak target can complete the task, and keeps the first routing+decision for later requests in the same conversation.++```toml+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.classifier]+id = "openai/gpt-4o-mini"+llm_client = "openrouter"++[targets.strong]+id = "openai/gpt-4o"+llm_client = "openrouter"++[targets.weak]+id = "z-ai/glm-5.2"+llm_client = "openrouter"++[routes.smart]+id = "smart"+type = "llm_classifier"+classifier_target = "classifier"+strong_target = "strong"+weak_target = "weak"+base_threshold = 0.5+session_affinity = true+message_hash_fallback = true+```-Set `profile` for the traffic you expect:+`message_hash_fallback` is best-effort: independent sessions with the same+first user message share an affinity key. Prefer an explicit+`x-switchyard-session-id` when repeated opening prompts are possible.-| `profile` | Use for | Default tier mapping |-|---|---|---|-| `general` | Mixed chat or API traffic | `simple` uses `weak`; all higher tiers use `strong`. |-| `coding_agent` | Claude Code, Codex, Cursor-style agents | `simple` and `medium` use `weak`; `complex` and `reasoning` use `strong`. Tool-planning turns can escalate. |-| `openclaw` | OpenClaw personal-assistant traffic | `simple` and `medium` use `weak`; `complex` and `reasoning` use `strong`. Tool orchestration and high-risk external actions can escalate. |+The target table names are local references. Their `id` values are the model+identifiers sent to the upstream provider. The route's `id`, `smart`, is the+model name clients send to Switchyard.-For coding-agent traffic, start with `profile: coding_agent`.+## How the decision works-## Configure a classifier route+The classifier target returns a structured verdict containing:-Define the strong, weak, and classifier models in a `deterministic` route:--```yaml-defaults:- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai--routes:- smart:- type: deterministic- profile: coding_agent- classifier:- model: nvidia/nemotron-3-nano-30b-a3b- min_confidence: 0.6- fail_open: true- recent_turn_window: 4- strong:- model: openai/gpt-4o- weak:- model: openai/gpt-4o-mini- fallback_target_on_evict: strong-```+- `p_solve`: the estimated probability that the weak model completes the task.+- `confidence`: confidence in the assessment.+- `capability_boundary`: `supported`, `uncertain`, `unsupported`, or `unmatched`.-Start the server with:+For a usable verdict, Switchyard routes to `weak_target` when `p_solve` is+greater than or equal to the applicable threshold. Otherwise it routes to+`strong_target`:-```bash-switchyard serve --routing-profiles routes.yaml --port 4000-```+- `supported` uses `base_threshold`.+- `uncertain`, `unsupported`, and `unmatched` use+ `capability_elevated_floor` when configured, or `base_threshold` otherwise.-The route ID (`smart`) is the model ID clients select for classifier-based-routing.+An abstention, an invalid or unparseable verdict, a judge failure, or confidence+below `min_confidence` routes to `strong_target`. Raising either threshold sends+more traffic to the strong model.-Try the profile with representative requests:+## Tuning options-```bash-# Coding task: expected to use the strong tier.-curl -X POST http://localhost:4000/v1/chat/completions \- -H "Authorization: Bearer dummy" -H "Content-Type: application/json" \- -d '{"model":"smart","messages":[{"role":"user","content":"Plan and implement a multi-file API change."}],"max_tokens":200}'--# Simple question: expected to use the weak tier.-curl -X POST http://localhost:4000/v1/chat/completions \- -H "Authorization: Bearer dummy" -H "Content-Type: application/json" \- -d '{"model":"smart","messages":[{"role":"user","content":"What is 2+2? Reply with just the number."}],"max_tokens":50}'-```+| Key | Default | Meaning |+|---|---|---|+| `base_threshold` | required | Lowest `p_solve` that routes a supported task to `weak_target`. Must be between `0` and `1`. |+| `min_confidence` | `0.0` | Lowest judge confidence that permits weak routing. Must be between `0` and `1`. |+| `capability_elevated_floor` | unset | Higher threshold for uncertain, unsupported, and unmatched tasks. It must be between `0` and `1` and greater than `base_threshold`. |+| `recent_turn_window` | unset | When unset, the judge sees only the newest user message. When set to `N`, it sees client system/developer instructions, the opening user task, and the last `N` conversation messages after that task. `0` keeps only the instructions and opening task. |+| `session_affinity` | `false` | Retains the first selected target for a session and reuses it on later requests. |+| `message_hash_fallback` | `false` | When session metadata is absent, keys affinity from the first user-message text. Requires `session_affinity = true`. |-Treat these as smoke checks, not fixed test vectors: the classifier model and-prompt determine the verdict.+### Classifier calibration-## Useful options+The packaged prompt is calibrated for one specific weak model, assumes it+receives the initial task, and describes a sticky routing decision. It is not a+model-neutral weak-target evaluator.-| Configuration path | Use it when |-|---|---|-| `classifier.min_confidence` | Low-confidence results should use the default tier instead of the classifier policy. |-| `classifier.fail_open` | Classifier errors should use the default tier rather than fail the client request. |-| `classifier.recent_turn_window` | The classifier needs more or less recent conversation and tool context. |-| `alignment_min_confidence` | A classifier recommendation should only raise the policy tier above this confidence. |-| `default_tier` | Abstain, low-confidence, and fail-open decisions should use a tier other than the default `strong`. |-| `tier_mapping` | The four classifier policy tiers need a custom mapping to `weak` or `strong`. |+Without affinity, the runtime judges every request; by default, it sends the+newest user message rather than the initial task. The example enables affinity+and its message-hash fallback so the first request is judged and later requests+with the same opening task reuse that decision. If you use a different weak+model or per-request routing, tune the thresholds before treating classifier+decisions as production policy.-For a self-hosted strong, weak, or classifier target, configure it like any-other OpenAI-compatible endpoint. See-[Self-hosted targets](overview.md#self-hosted-targets).+Support for supplying your own classifier prompt is coming soon.## Session affinity-LLM classifier routing supports optional session affinity through-`DeterministicRoutingConfig`. Set `session_affinity: true` to share one affinity-store between the classifier and tier selector. After any configured-`affinity_warmup_turns`, the first confident verdict pins the tier. Later turns-reuse that tier before classification, so they skip the classifier call;-abstain, low-confidence, missing-signal, and fail-open decisions do not pin.--Configure these fields on the `type: deterministic` entry in the `routes:`-bundle. See [Session Affinity](sticky_routing.md) for YAML and-[How session affinity composes](overview.md#how-session-affinity-composes) for-the interaction with routing decisions.--If the per-request classifier cost is too high, use-[Stage-Router Routing](stage_router_routing.md), which can route many turns from tool and-agent-progress signals without an extra classifier call.+With `session_affinity = true`, the first selected target is retained for the+request's session identity. This includes `strong_target` when it was selected+as the fallback for an unavailable or unusable judge verdict. There is no+warmup period. Later requests with the same identity reuse the target before+classification, so the judge call is skipped.++Affinity is process-local. Clients can send `x-switchyard-session-id`, or enable+`message_hash_fallback` to key requests without session metadata from the first+user-message text.++## Run the route++After [building the Rust server](../getting_started.md#build-the-server), export+the provider credential, validate the configuration, and start the release+binary:++```bash+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret+./target/release/switchyard-server --config routes.toml --dry-run+./target/release/switchyard-server --config routes.toml \+ --host 127.0.0.1 --port 4000+```++Send a request using the route ID:++```bash+curl http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{"model":"smart","messages":[{"role":"user","content":"Explain why the sky appears blue."}]}'+```++Treat the selected target as model-dependent output, not a fixed test result.diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.mdindex f6fcc8efe..c595cab6f 100644--- a/docs/routing_algorithms/overview.md+++ b/docs/routing_algorithms/overview.md@@ -1,160 +1,134 @@# Routing Overview-The Python CLI loads routing policies from a YAML bundle. Each key under-`routes:` becomes a model ID available through OpenAI Chat Completions,-Anthropic Messages, and OpenAI Responses requests:--```bash-switchyard serve --routing-profiles routes.yaml --port 4000-```+The `switchyard-server` binary loads a native TOML deployment. Each route's `id`+becomes a model ID available through OpenAI Chat Completions, Anthropic Messages,+and OpenAI Responses requests.Use this page to choose a routing strategy, then open its detailed page for-configuration and tuning.+configuration and tuning. For the vocabulary these pages use, see+[Core Concepts](../core_concepts.md).## Choose a strategy| Strategy | Use it when | Route `type` ||---|---|---|-| [Random Routing](random_routing.md) | You need a fixed strong/weak split for A/B tests, baselines, or cost experiments. | `random_routing` |-| [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `deterministic` |+| [Random Routing](random_routing.md) | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | `random` |+| [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm_classifier` || [Stage-Router Routing](stage_router_routing.md) | Tool-result and agent-progress signals should route most turns without an extra classifier call. | `stage_router` |-| [Escalation-Router Routing](escalation_router_routing.md) | Start every task on the weak tier and escalate to strong when an LLM judge detects trouble. | `escalation_router` |--[Session Affinity (Sticky Routing)](sticky_routing.md) is an opt-in feature of-LLM classifier routing, not a standalone routing strategy.+| [Escalation-Router Routing](escalation_router_routing.md) | Start every task on the weak tier and escalate to strong when an LLM judge detects trouble. | `llm_classifier` with `escalation` |## Common route shape-Provider defaults can be shared across all routes, while each route owns its-target configuration:--```yaml-defaults:- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai--routes:- fast:- type: model- target: openai/gpt-4o-mini-- smart:- type: random_routing- strong:- model: openai/gpt-4o- weak:- model: openai/gpt-4o-mini- strong_probability: 0.3- fallback_target_on_evict: strong+A deployment has three layers. LLM clients describe how to reach a provider,+targets name models on those clients, and routes decide which target serves a+request:++```toml+schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.strong]+id = "openai/gpt-4o"+llm_client = "openrouter"++[targets.weak]+id = "openai/gpt-4o-mini"+llm_client = "openrouter"++[routes.fast]+id = "fast"+type = "passthrough"+target = "weak"++[routes.smart]+id = "smart"+type = "random"+targets = ["strong", "weak"]+weights = [3, 7]```-Use the route name (`fast` or `smart`) as the request's model ID. A single-bundle can serve multiple routes on the same host and port.+Clients send a route's `id` (`fast` or `smart`) as the request's model ID. The+table name in `[routes.smart]` is local to the file; only `id` is visible to+clients. A single deployment can serve multiple routes on the same host and port.++Credentials stay outside the file: `api_key_env` names an environment variable+that is read at startup. Omit it for an upstream that needs no credential, such+as a local model server.The examples use model IDs from the[OpenRouter model catalog](https://openrouter.ai/api/v1/models). Select IDsavailable to your account before deploying; catalog availability can change.-## Model routes+## Direct model routes-- `type: model` registers one explicit model alias without model discovery.+A `passthrough` route registers one target under one model ID with no routing+decision:-This creates a direct, single-target chain. Use a routing policy when requests-must be split or classified across targets. There is no catalog auto-discovery,-so to expose several upstream models, add one `type: model` route per model id.+```toml+[routes.fast]+id = "fast"+type = "passthrough"+target = "weak"+```++Use a routing strategy instead when requests must be split or classified across+targets. There is no catalog auto-discovery, so to expose several upstream+models, add one `passthrough` route per model.## Self-hosted targets-Any route target can point at an OpenAI-compatible model server you operate.-For example, start a local vLLM server:+Any target can point at an OpenAI-compatible model server you operate. For+example, start a local vLLM server:```bashvllm serve ./my-rl-qwen --served-model-name my-rl-qwen --port 8000```-Then configure it as a normal route:+Then declare it as its own LLM client and target. The client needs no+`api_key_env` when the server does not require a credential:++```toml+[llm_clients.local_vllm]+format = "openai_chat"+base_url = "http://localhost:8000/v1"-```yaml-routes:- local:- type: model- target: my-rl-qwen- base_url: http://localhost:8000/v1- api_key: dummy- format: openai+[targets.local]+id = "my-rl-qwen"+llm_client = "local_vllm"++[routes.local]+id = "local"+type = "passthrough"+target = "local"```-Switchyard does not start or manage the model server; it only sends requests-to the configured endpoint.--## Sub-agent override (`subagent_target`)--Any route may name an optional `subagent_target` in its common envelope,-alongside `type`. A request carrying a recognized sub-agent signal — Claude Code-agent-lineage headers, Codex delegated-work kinds (`x-openai-subagent:-collab_spawn` or `review`), or an explicit `x-switchyard-is-subagent: true` —-bypasses the route's own chain and runs as a direct passthrough to that target:--```yaml-routes:- assistant:- type: model- target:- model: strong-model- subagent_target:- model: cheaper-worker-model+Switchyard does not start or manage the model server; it only sends requests to+the configured endpoint.++## Run a deployment++After building the Rust server, as described in+[Getting Started](../getting_started.md#build-the-server), export the provider+credential, validate the configuration, and start the binary:++```bash+export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret+./target/release/switchyard-server --config routes.toml --dry-run+./target/release/switchyard-server --config routes.toml \+ --host 127.0.0.1 --port 4000```-This keeps a sub-agent loop on one intentional, cache-compatible target instead-of re-routing every worker turn. The worker may live on a different provider-entirely — give it its own `base_url` and `api_key` and one route spans two-upstreams:--```yaml- subagent_target:- model: my-local-model- base_url: http://localhost:8000/v1- api_key: dummy- format: openai+Send a request using a route ID:++```bash+curl http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{"model":"smart","messages":[{"role":"user","content":"hello"}]}'```-Detection is the protocol crate's canonical policy, shared with the libsy-`SubagentOverride` classifier, so both engines agree on what counts as-delegated work. Harness-maintenance turns (`compact`, `memory_consolidation`)-and unrecognized kinds stay on normal routing, as does everything else when the-field is absent. A worker-target failure surfaces as a normal target error — it-is never silently re-routed through the route's own chain.--To suppress sub-agent routing for a request that carries a recognized signal,-send `x-switchyard-is-subagent: false`. This explicit header overrides Codex and-Claude Code lineage signals in either direction: `false` keeps the request on-normal routing even when delegated-work headers are present, and `true` marks a-request as a sub-agent even when no harness headers appear.--The key applies to `model`, `deterministic`, `escalation_router`,-and `stage_router` routes. `random_routing` expands into its table entries on a-separate path and does not consume it.--## How session affinity composes--Session affinity is configured directly on the LLM classifier router. After-the configured warmup, the first confident policy, tool-planning, or alignment-verdict pins the tier. Abstain, low-confidence, missing-signal, and fail-open-decisions never pin. Later turns reuse the tier and skip the classifier call.--Pins use a bounded in-process LRU keyed from the stable conversation prefix.-They are not shared across workers or restarts. See-[Sticky Routing](sticky_routing.md) for configuration and key derivation.--Random and stage-router routing make a fresh routing decision for each request.-The escalation router instead uses affinity as a one-way escalation latch: only-the strong tier is pinned, and the judge runs until that latch fires.--## Rust server configuration--The `switchyard-server` binary has a separate TOML schema that explicitly-constructs LLM clients, targets, and libsy algorithms. It does not load Python-route bundles. See the-[Rust server README](../../crates/switchyard-server/README.md) for its supported-configuration.+For the complete TOML schema and every route option, refer to the+[TOML Schema](../reference/toml_schema.md).diff --git a/docs/routing_algorithms/random_routing.md b/docs/routing_algorithms/random_routing.mdindex 17da19eb0..3af82f12e 100644--- a/docs/routing_algorithms/random_routing.md+++ b/docs/routing_algorithms/random_routing.md@@ -77,8 +77,9 @@ therefore use different targets.## Run the route-After [building the Rust server](../getting_started.md#install), export the-provider credential, validate the configuration, and start the release binary:+After [building the Rust server](../getting_started.md#build-the-server), export+the provider credential, validate the configuration, and start the release+binary:```bashexport OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secretdiff --git a/docs/routing_algorithms/sticky_routing.md b/docs/routing_algorithms/sticky_routing.mddeleted file mode 100644index 404d798aa..000000000--- a/docs/routing_algorithms/sticky_routing.md+++ /dev/null@@ -1,80 +0,0 @@-# Sticky Routing--By default, routing is recomputed on every request, so a single task can hop-between models mid-conversation. This throws away the upstream prompt/KV cache-and, on the classifier router, pays for an LLM classifier call every turn.--**Session affinity**, also called sticky routing, is an opt-in feature configured-directly on the deterministic LLM-classifier router. It pins the routing-decision for a conversation and reuses it for later turns, so a task stays on-one model tier. The router can optionally delay that pin for a fixed number of-warmup turns before committing to a tier.--## How a conversation is identified--No client session ID is required. A conversation is keyed by hashing the stable-prefix made from the **system prompt + the first user message**. Agent harnesses-do not rewrite that prefix, so every turn of one task maps to the same key while-distinct tasks differ. The key is derived per request and the pin store is a bounded-(least-recently-used) in-process map.--## Configuration--Opt in with `session_affinity: true` on the route. `affinity_max_sessions`-(default `10000`) caps the number of pinned conversations.--These fields are part of the deterministic router configuration, not a-standalone route type. Configure them under a `deterministic` entry in the-`routes:` bundle loaded with `--routing-profiles`.--For deterministic routes, `affinity_warmup_turns` controls how many initial-turns remain non-sticky. The default is `0`, which preserves the historical-"pin the first confident verdict" behavior.--### Deterministic (LLM-classifier) router--```yaml-routes:- llm-classifier:- type: deterministic- profile: coding_agent- session_affinity: true # pin a confident tier per conversation- affinity_max_sessions: 10000- affinity_warmup_turns: 2 # turns 1-2 classify normally; turn 3 can pin- fallback_target_on_evict: strong- classifier:- model: openai/gpt-4o-mini- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- strong:- model: openai/gpt-4o- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- weak:- model: openai/gpt-4o-mini- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1-```--The first **confident** classifier verdict after the warmup period is pinned and-reused. With `affinity_warmup_turns: 2`, turns 1 and 2 route normally and cannot-read or write a pin; turn 3 is the first turn that can commit a confident tier.-Once pinned, the classifier **skips its LLM call entirely**. It runs only until-the route has a committed tier for the task. Fallback decisions (abstain / low-confidence) are never pinned, so a transient classifier failure can't lock the-task to the default tier.--## Behavior notes--- **Off by default.** With `session_affinity: false`, routing is recomputed per- turn as before.-- **Warmup.** `affinity_warmup_turns: N` means turns- `1..N` stay non-sticky; the first eligible commit is turn `N + 1`.-- **`affinity_max_sessions: 0` with affinity on is rejected** at config load. A- zero-capacity store retains nothing and would silently disable stickiness.-- **In-process, per-worker.** Pins live in memory and are not shared across- workers or restarts.-- **Degrades safe.** Affinity assumes the system + first-user prefix is stable- across a task's turns. If a harness injects per-turn churn there, the key- changes each turn and routing simply reverts to per-turn behavior. No lock,- no error.diff --git a/mkdocs.yml b/mkdocs.ymlindex 74f6cc2f4..0a624c5a0 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -27,13 +27,13 @@ nav:- Overview: routing_algorithms/overview.md- Random Routing: routing_algorithms/random_routing.md- LLM Classifier Routing: routing_algorithms/llm_classifier_routing.md- - Sticky Routing: routing_algorithms/sticky_routing.md- Stage-Router Routing: routing_algorithms/stage_router_routing.md- Escalation-Router Routing: routing_algorithms/escalation_router_routing.md- Operations:- Context-Window Handling: operations/context_window.md- Reference:- CLI Reference: cli_reference.md+ - TOML Schema: reference/toml_schema.mdexclude_docs: |internal/**diff --git a/tests/getting_started/test_getting_started.py b/tests/getting_started/test_getting_started.pyindex 835be30da..7a6b3c1e6 100644--- a/tests/getting_started/test_getting_started.py+++ b/tests/getting_started/test_getting_started.py@@ -1,17 +1,31 @@# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.# SPDX-License-Identifier: Apache-2.0-"""Drift checks for the concise getting-started guide."""+"""Drift checks for the Getting Started guide."""from pathlib import Path-def test_getting_started_uses_the_current_cli() -> None:+def test_getting_started_documents_current_paths() -> None:guide = (Path(__file__).resolve().parents[2] / "docs" / "getting_started.md").read_text()++ assert guide.index("## Launcher Path") < guide.index("## Server Path")+ assert 'uv tool install "nemo-switchyard[cli,server]"' in guideassert "switchyard launch claude --model switchyard" in guide- assert "switchyard launch codex --model switchyard" in guide- assert "switchyard serve" not in guide- assert "switchyard configure" not in guide- assert "switchyard verify" not in guide+ assert "cargo build --locked --release -p switchyard-server" in guide+ assert "./target/release/switchyard-server --config routes.toml --dry-run" in guide++ for legacy_command in (+ "switchyard configure",+ "switchyard serve",+ "switchyard verify",+ "--routing-profiles",+ "routes.yaml",+ ):+ assert legacy_command not in guide+++# TODO: Add Python snippet coverage when the supported Rust-backed API is finalized.+# TODO: Add TOML schema coverage if the guide grows beyond one maintained example.diff --git a/tests/readme/test_readme.py b/tests/readme/test_readme.pyindex 240f5232b..7c5b1a3ee 100644--- a/tests/readme/test_readme.py+++ b/tests/readme/test_readme.py@@ -1,15 +1,27 @@# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.# SPDX-License-Identifier: Apache-2.0-"""README command drift checks."""+"""Drift checks for the repository README."""from pathlib import Path-def test_readme_uses_the_current_cli() -> None:+def test_readme_documents_current_paths() -> None:readme = (Path(__file__).resolve().parents[2] / "README.md").read_text()++ assert readme.index("### Launcher Path") < readme.index("### Server Path")+ assert 'uv tool install "nemo-switchyard[cli,server]"' in readmeassert "switchyard launch claude --model switchyard" in readme- assert "switchyard launch codex --model switchyard" in readme- assert "switchyard serve" not in readme- assert "switchyard configure" not in readme- assert "switchyard verify" not in readme+ assert "cargo build --locked --release -p switchyard-server" in readme+ assert "./target/release/switchyard-server --config routes.toml --dry-run" in readme++ for legacy_command in (+ "switchyard configure",+ "switchyard serve",+ "switchyard verify",+ ):+ assert legacy_command not in readme+++# TODO: Add Python snippet coverage when the supported Rust-backed API is finalized.+# TODO: Add TOML schema coverage if the README includes a complete server config.diff --git a/tests/test_cli_reference_docs.py b/tests/test_cli_reference_docs.pyindex 3c5af543d..e846404a9 100644--- a/tests/test_cli_reference_docs.py+++ b/tests/test_cli_reference_docs.py@@ -4,6 +4,7 @@"""CLI reference drift tests."""import argparse+import subprocessfrom pathlib import Pathfrom switchyard.cli.switchyard_cli import _build_parser@@ -29,9 +30,13 @@ def _long_options(parser: argparse.ArgumentParser) -> set[str]:}-def test_reference_documents_launch_only() -> None:+def test_reference_documents_launcher_before_server() -> None:text = CLI_REFERENCE.read_text()- assert "## `switchyard launch`" in text+ launcher = text.index("## Launcher Path: `switchyard launch`")+ server = text.index("## Server Path: `switchyard-server`")+ related = text.index("## Related Documentation")++ assert launcher < server < relatedassert "switchyard serve" not in textassert "switchyard configure" not in textassert "switchyard verify" not in text@@ -44,3 +49,38 @@ def test_reference_lists_launcher_contract() -> None:assert _long_options(parser) == {"--model", "--config"}assert "--model" in textassert "--config" in text+++def test_reference_lists_server_contract() -> None:+ result = subprocess.run(+ [+ "cargo",+ "run",+ "--quiet",+ "--locked",+ "-p",+ "switchyard-server",+ "--",+ "--help",+ ],+ cwd=CLI_REFERENCE.parents[1],+ capture_output=True,+ text=True,+ check=False,+ )+ assert result.returncode == 0, result.stderr++ text = CLI_REFERENCE.read_text()+ for flag in (+ "--config",+ "--host",+ "--port",+ "--backlog",+ "--dry-run",+ "--tls-cert",+ "--tls-key",+ "--help",+ "--version",+ ):+ assert flag in result.stdout+ assert flag in text