Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs: add core concepts page, polish README, and fix stale doc commands (#3)

Ayush Agarwal05a4e9791b6b
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index 0cc20dec5..47a41b1ee 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -16,7 +16,7 @@ Minimal steps to reproduce. Include the command line, the inbound request shape,
```bash
# example
-switchyard passthrough --port 4000
+switchyard serve --config profiles.yaml --port 4000
curl -s http://localhost:4000/v1/chat/completions -d '{"model":"...","messages":[...]}'
```
diff --git a/.secrets.baseline b/.secrets.baseline
index d1dcb81f5..affda6f56 100644
--- a/.secrets.baseline
+++ b/.secrets.baseline
@@ -283,7 +283,7 @@
"filename": "README.md",
"hashed_secret": "ec3810e10fb78db55ce38b9c18d1c3eb1db739e0",
"is_verified": false,
- "line_number": 152
+ "line_number": 165
}
],
"crates/switchyard-components-v2/src/profiles/cascade.rs": [
diff --git a/AGENTS.md b/AGENTS.md
index 87b739feb..47e99dc95 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -247,31 +247,24 @@ and their transitives never appear in downstream vulnerability scans.
export OPENAI_API_KEY="sk-..." # or NVIDIA_API_KEY / ANTHROPIC_API_KEY where supported
export OPENROUTER_API_KEY="sk-or-..." # pass with --api-key or save via configure
-# Passthrough — single OpenAI-compatible backend, all three inbound formats
-switchyard passthrough --port 4000
-switchyard passthrough --inbound anthropic --port 4000
-switchyard passthrough --inbound both --base-url https://... --api-key sk-...
-switchyard passthrough --base-url https://openrouter.ai/api/v1 --api-key "$OPENROUTER_API_KEY"
-
-# Random-routing — weighted strong/weak coin for benchmarks
-switchyard random-routing \
- --strong-model openai/openai/gpt-5.2 \
- --weak-model openai/nvidia/nemotron-3-super \
- --strong-probability 0.3 --port 4000
-
-# One-command launchers
-switchyard launch claude --model nvidia/moonshotai/kimi-k2.5
-switchyard launch claude --preset opus_kimi --strong-probability 0.5
-switchyard launch codex --model nvidia/moonshotai/kimi-k2.5
-switchyard launch codex --model openai/gpt-5.2 \
+# Serve a profile config (passthrough, random-routing, llm-routing, cascade).
+# Endpoints, targets, and profiles live in the YAML; see docs/routing_algorithms/overview.md.
+switchyard serve --config profiles.yaml --port 4000
+switchyard serve --config profiles.yaml --inbound anthropic --port 4000
+
+# One-command launchers — single-model passthrough via --model
+switchyard launch claude --model openai/gpt-4o-mini \
+ --base-url https://openrouter.ai/api/v1 --api-key "$OPENROUTER_API_KEY"
+switchyard launch codex --model openai/gpt-4o-mini \
--base-url https://openrouter.ai/api/v1 --api-key "$OPENROUTER_API_KEY"
-# Built-in routing strategies (zero-config)
-switchyard launch codex --deterministic # LLM-classifier strong/weak
-switchyard launch codex --plan-execute # strong planner + weak executor
+# Launchers default to built-in LLM-classifier routing; tune the tiers:
+switchyard launch claude \
+ --weak-model openai/gpt-4o-mini --classifier-model openai/gpt-4o \
+ --classifier-min-confidence 0.6
-# Verification and status
-switchyard verify --model openai/openai/gpt-5.2
+# Verification and saved config
+switchyard verify --model openai/gpt-4o-mini
switchyard configure --show
```
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 51289aa44..7ec479770 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -10,7 +10,7 @@ Quick start:
```bash
git clone https://github.com/NVIDIA-NeMo/Switchyard.git
-cd switchyard
+cd Switchyard
uv sync
uvx pre-commit install --install-hooks --hook-type pre-commit --hook-type commit-msg
source .venv/bin/activate
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
index 1e6b21bce..12b1245f5 100644
--- a/DEVELOPMENT.md
+++ b/DEVELOPMENT.md
@@ -15,7 +15,7 @@ and dependencies. Install `uv` first if you don't have it
```bash
git clone https://github.com/NVIDIA-NeMo/Switchyard.git
-cd switchyard
+cd Switchyard
uv sync # creates .venv, installs core + dev tooling
uvx pre-commit install --install-hooks --hook-type pre-commit --hook-type commit-msg
diff --git a/README.md b/README.md
index 0ce826d71..0dbb83a39 100644
--- a/README.md
+++ b/README.md
@@ -4,11 +4,24 @@
# Switchyard
-Typed, composable LLM routing and format translation for Python. Route traffic between multiple LLM providers, translate between OpenAI and Anthropic APIs, collect usage statistics, and build profile-backed routing flows with strong typing and minimal boilerplate.
-
-**Why Switchyard?** Point coding agents like **Claude Code** and **Codex** at compatible open-source models. Switchyard transparently translates between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats, so each agent keeps speaking its native API while requests are served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint. The same proxy can also **route across multiple models** for A/B benchmarking splits, signal-driven cascade escalation, or a custom router you wire in.
-
-**Launcher routing is explicit**: launchers default to built-in LLM-classifier routing, which you can tune with `--weak-model`, `--classifier-model`, `--profile`, and `--classifier-min-confidence`; use `--model X` for single-model passthrough. The deprecated `--routing-profiles FILE` path remains for launcher-owned legacy bundles.
+Switchyard is a Python proxy for LLM traffic. It routes requests across
+providers, translates between the OpenAI and Anthropic APIs, collects usage
+statistics, and lets you build typed, profile-backed routing flows with little
+boilerplate.
+
+**Why Switchyard?** Point a coding agent such as Claude Code or Codex at an
+open-source model. Switchyard translates between the OpenAI Chat, Anthropic
+Messages, and OpenAI Responses formats, so the agent keeps speaking its native
+API while the request is served by vLLM, NVIDIA NIM, Ollama, or any
+OpenAI-compatible endpoint. The same proxy can spread traffic across several
+models for A/B benchmarking, signal-driven cascade escalation, or a router you
+write yourself.
+
+**Launcher routing is explicit.** By default, launchers use the built-in
+LLM-classifier router, which you tune with `--weak-model`, `--classifier-model`,
+`--profile`, and `--classifier-min-confidence`. Use `--model X` for
+single-model passthrough. The `--routing-profiles FILE` path is deprecated and
+remains only for launcher-owned legacy bundles.
## Features
@@ -31,7 +44,7 @@ pip install "switchyard[cli,server]"
```bash
git clone [email protected]:NVIDIA-NeMo/Switchyard.git
-cd switchyard
+cd Switchyard
uv tool install --editable '.[server,cli]'
```
@@ -39,7 +52,7 @@ uv tool install --editable '.[server,cli]'
```bash
git clone [email protected]:NVIDIA-NeMo/Switchyard.git
-cd switchyard
+cd Switchyard
uv sync
uv run switchyard ...
```
@@ -167,17 +180,23 @@ asyncio.run(main())
## Architecture
-Switchyard sits between client applications and one or more LLM backends:
+Switchyard sits between your client applications and one or more LLM backends:
+
+```mermaid
+flowchart LR
+ clients["Clients"]
+ switchyard["Switchyard<br/>routing · translation · fallback"]
+ backends["Model backends"]
-```text
-clients --> Switchyard --> model backends
- +--------> routing, translation, and fallback
+ clients -->|"OpenAI / Anthropic API"| switchyard
+ switchyard -->|"provider-native format"| backends
```
-Clients keep their supported OpenAI or Anthropic API format while Switchyard
-selects a configured model endpoint and translates the response back to the
-expected shape. See [Architecture](docs/architecture.md) for system context and
-the end-to-end request flow.
+Clients keep their native OpenAI or Anthropic API format. Switchyard picks a
+configured backend, forwards the request in that backend's own format, and
+translates the response back into the shape the client expects. See
+[Architecture](docs/architecture.md) for the system context and the full
+request flow.
## Installation Options
diff --git a/docs/core_concepts.md b/docs/core_concepts.md
new file mode 100644
index 000000000..b8cbbbd53
--- /dev/null
+++ b/docs/core_concepts.md
@@ -0,0 +1,103 @@
+# Core Concepts
+
+This page explains the vocabulary the rest of the documentation takes for
+granted. Read it once and the configuration examples elsewhere will make sense.
+It is not a setup guide: to install and run Switchyard, see
+[Getting Started](getting_started.md), and to follow a request through the
+system, see [Architecture](architecture.md).
+
+## How it fits together
+
+Switchyard is a proxy. Clients (coding agents, SDKs, your own services) talk to
+it on one side, and model backends (hosted providers, private endpoints, local
+models) sit on the other. A client never asks for a backend by name. It sends a
+model ID, and the configuration behind that ID decides what actually runs.
+
+```mermaid
+flowchart LR
+ client["Client<br/>picks a model ID"]
+ profile["Profile<br/>routing policy"]
+ target["Target<br/>upstream model"]
+ endpoint["Endpoint<br/>provider connection"]
+ backend["Model backend"]
+
+ client -->|"model field"| profile
+ profile -->|"selects"| target
+ target -->|"uses"| endpoint
+ endpoint --> backend
+```
+
+## Endpoints, targets, and profiles
+
+A standalone deployment is described by a profile config: one file with three
+sections that keep provider connectivity, upstream models, and client-facing
+policy apart. Three terms carry most of the weight.
+
+| Term | What it is | YAML section |
+|---|---|---|
+| Endpoint | A provider connection: a `base_url` and its credentials. Many targets can share one. | `endpoints:` |
+| Target | A single upstream model you can call. It names an endpoint, a `model`, and a wire `format`. | `targets:` |
+| Profile | A client-facing routing policy over one or more targets. Its `type` is the routing strategy. | `profiles:` |
+
+The three nest. An endpoint is reused by targets, and targets are referenced by
+profiles. You set credentials in one place, add a model once, and build as many
+policies on top as you need. The [Routing Overview](routing_algorithms/overview.md)
+has a complete, runnable config and the full schema.
+
+## Model IDs
+
+Clients choose what happens through the `model` field on a request. Switchyard
+registers three kinds of model IDs and lists them all on `GET /v1/models`:
+profile IDs, which apply a routing policy; target IDs, which skip routing and
+call that one model; and the upstream model name itself, registered as an alias
+whenever it differs from its target ID. Send a profile ID when you want routing,
+or a target or upstream name when you want to pin a single model.
+
+## Tiers and routing strategies
+
+Most routing strategies divide traffic between two tiers: a strong target that
+is more capable and more expensive, and a weak target that is cheaper and
+faster. A tier is a role you hand to a target rather than a fixed property of it,
+so the same target can be the strong tier in one profile and the weak tier in
+another.
+
+A profile's `type` sets the strategy. `passthrough` sends everything to one
+target with no routing. `random-routing` splits traffic on a fixed probability.
+`llm-routing` asks a classifier model to pick a tier for each turn. `cascade`
+escalates from weak to strong when request signals call for it. The
+[Routing Overview](routing_algorithms/overview.md) covers when to use each and
+how to tune it.
+
+!!! warning "There is no `type: model`"
+ The current profile config does not have a `type: model`. To expose a single
+ model, point clients at a target ID directly, or add a `passthrough` profile
+ when you want a second name for it. `type: model` survives only in the
+ deprecated `--routing-profiles` bundles used by the launcher compatibility
+ path.
+
+Session affinity, or sticky routing, pins a conversation to one tier so later
+turns reuse it instead of being classified again. It belongs to `llm-routing`
+and is not a strategy of its own; random and cascade routing decide every
+request on its own merits. [Sticky Routing](routing_algorithms/sticky_routing.md)
+covers it in full.
+
+## Formats and translation
+
+Clients reach Switchyard in one of three inbound formats: OpenAI Chat
+Completions, Anthropic Messages, or OpenAI Responses. Each target has its own
+backend format, set by its `format:` field, which is one of `openai`,
+`anthropic`, `responses`, or `auto`. When the two differ, Switchyard translates
+the request on the way out and the response on the way back.
+
+That translation is what lets Claude Code, which speaks Anthropic Messages, run
+against an OpenAI-compatible model, and the reverse. The
+[Architecture](architecture.md) page documents every backend format, the `auto`
+probe, and the neutral representation used to convert between them.
+
+## Where to go next
+
+- [Getting Started](getting_started.md) to install and send a first request.
+- [Routing Overview](routing_algorithms/overview.md) to choose and tune a strategy.
+- [Agent Launchers](guides/agent_launchers.md) to run Claude Code, Codex, or OpenClaw.
+- [Architecture](architecture.md) to see a request travel end to end.
+- [CLI Reference](cli_reference.md) for flags and environment variables.
diff --git a/docs/getting_started.md b/docs/getting_started.md
index 461f1df56..2bd23cdad 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -214,7 +214,7 @@ export SWITCHYARD_TELEMETRY_OPT_OUT=1
```bash
git clone https://github.com/NVIDIA-NeMo/Switchyard.git
-cd switchyard
+cd Switchyard
uv sync
source .venv/bin/activate
uv run pytest tests/ -v
diff --git a/mkdocs.yml b/mkdocs.yml
index 482ba691d..d7bdb75b8 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -23,6 +23,7 @@ nav:
- Guides:
- Agent Launchers: guides/agent_launchers.md
- Concepts:
+ - Core Concepts: core_concepts.md
- Architecture: architecture.md
- Routing:
- Overview: routing_algorithms/overview.md