Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
switchyard-server
switchyard-server exposes libsy algorithms through OpenAI Chat Completions, OpenAI Responses,
and Anthropic Messages endpoints. A TOML file explicitly defines the LLM clients, targets, and
algorithm routes served by the process.
The standalone binary listens on 127.0.0.1 unless --host is set. Pass
--host 0.0.0.0 to accept remote clients. The provided Docker image and
dev-server/switchyard.service opt into that address explicitly. A later
--host argument overrides the container's default.
# routes.toml
schema_version = 1
[llm_clients.example]
format = "openai_chat"
base_url = "https://example.com/v1"
api_key_env = "API_KEY"
max_retries = 2
[targets.model_a]
id = "model/a"
llm_client = "example"
system_prompt = "Use the fast path for routine work."
extra_body = { service_tier = "priority" }
[targets.model_b]
id = "model/b"
llm_client = "example"
[routes.general]
id = "switchyard/general"
type = "random"
targets = ["model_a", "model_b"]
weights = [1, 3]
seed = 42
[routes.classified]
id = "switchyard/classified"
type = "llm_classifier"
mode = "capability"
classifier_target = "model_a"
strong_target = "model_a"
weak_target = "model_b"
base_threshold = 0.5
[routes.passthrough]
id = "switchyard/passthrough"
type = "passthrough"
target = "model_a"
[routes.stage]
id = "switchyard/stage"
type = "stage_router"
capable_target = "model_a"
efficient_target = "model_b"
picker = "efficient_first"
confidence_threshold = 0.5
[routes.stage.tool_semantics]
observe = ["lookup_customer"]
mutate = ["update_inventory"]
new = ["send_message"]
export API_KEY="..."
cargo install --locked switchyard-server
switchyard-server --config routes.toml
Ctrl+C and Unix SIGTERM stop new connections and allow active requests to drain for up to
--shutdown-timeout (30 seconds by default) before they are terminated.
The server logs exactly one structured terminal event per LLM request: successful responses at
INFO, 4xx responses at WARN, and 5xx responses at ERROR. Set
RUST_LOG=switchyard_server=debug,libsy=debug to include routing decisions and nested failure
details. A streaming failure is logged separately because it can occur after the response starts.
Target and route table names are local references. A target's id is the exact model ID sent
upstream, and a route's id is the model clients send to select that algorithm.
Each target references an entry under llm_clients. All configured clients use
TranslatingLlmClient; supported formats are openai_chat, openai_responses, and
anthropic_messages. Supported algorithms are noop, random, passthrough,
llm_classifier, stage_router, auto,
composite, and
advisor.
The optional prefill-router feature also enables
the experimental prefill_router. See its
artifact requirements.
An api_key_env value names an environment variable. The TOML never contains the
secret itself. If omitted, the client sends no authentication.
A client can set forward_auth = true instead of api_key_env to send the
caller's credential to the configured upstream. OpenAI clients forward
authorization, chatgpt-account-id, and x-openai-fedramp. Anthropic clients
forward authorization or x-api-key. Enable this only when every forwarding
client's base_url should receive the caller's login. Other application headers
are preserved and may contain provider-specific credentials. A route's
forwarding clients must use one credential family: all OpenAI formats or all
anthropic_messages. The exception is one host that serves both formats, such
as an LLM gateway that accepts each caller's gateway key on every endpoint: a
route may mix the two families when all of its forwarding clients use the same
scheme, host, and port. Such a route serves Chat Completions and Responses
callers and forwards the caller's bearer token to every forwarding client.
The server returns 400 to a caller whose API the route does not serve.
Clients that use api_key_env send the server's own key, so these limits do
not apply to them.
Target-level extra_body values are shallow-merged into the upstream request when
the request does not already contain that key.
Target-level system_prompt values are prepended when that target serves a completion.
Selected and fallback targets are prepared independently.
max_retries defaults to 2 and applies to transport failures, timeouts, HTTP 408/429, and 5xx
responses.
Random-route weights are relative, follow target order, and do not need to sum to one. Omit them
for equal weighting. The optional seed reproduces the selection sequence for the same call order.
Session routing log
Pass --routing-log-file PATH to append one JSON record after each completed routed response.
Streaming responses are recorded after the stream drains. Each record names the client-facing
route_id, routing algorithm, served model, tier, and token usage so routes sharing a backend
remain distinguishable. When enabled,
GET /v1/routing/session-stats?session_id=ID rescans the durable log and returns call and token
totals for that normalized session ID, normally supplied as x-switchyard-session-id, grouped by
served model. The legacy proxy_x_session_id remains a fallback when no normalized session ID is
present. The endpoint returns 404 when the session has no records and is not registered when
routing logging is disabled.
The routing log saves each call's reported charge in USD. When the upstream reports no
charge, the server estimates it using bundled rates.
A reported charge, including zero, takes precedence over the bundled rates.
Bundled rates stay fixed until you install a version with updated rates.
The table includes text-token rates for OpenAI, Claude, Gemini, and DeepSeek models, plus
OpenRouter rates for other popular models. Model IDs match exactly, including provider
prefixes; an unlisted ID has no estimated cost. Direct-provider and OpenRouter rates use
separate IDs, such as gemini-3.8-flash and google/gemini-3.8-flash.
Estimates exclude long-context and service-tier adjustments, OpenAI's 12-hour cache-write
pricing, tool fees, and time-based cache storage. Each model has one rate at all times of
day. Billed charges can differ when a gateway selects another provider or a promotion
ends. The table lists the Gemini promotion end dates.
Use --pricing-file prices.toml to replace the entire bundled table with your own rates.
Use a .toml file with quoted served model IDs and rates in USD per million tokens:
["upstream/model"]
input = 1
output = 5
cache_read = 0.1
cache_write = 2
The .toml extension check ignores case. Other filenames are read as JSON, so existing
JSON overrides still work. TOML and JSON also accept groups of exact IDs that share
one set of rates, as shown in the pi guide.
The bundled table uses this grouped format and can be supplied to --pricing-file.
Session stats include cost totals. Missing usage or rates leave cost unknown; known_usd
keeps the priced subtotal and total_usd is null when any cost is unknown. Rates load once
at startup, and saved costs do not change when prices change. An unreadable or invalid
--pricing-file stops startup; the server does not use bundled rates instead.
The pi guide explains how to supply
rates for your own models. The desktop app uses the same bundled rates by default.
For custom rates, point its pricing_file setting at the same TOML or JSON file as
the server's --pricing-file. The desktop app totals these recorded costs, so its totals
can differ from what your provider bills.
Clients can send x-switchyard-origin: codex-cli (or another client label) to include an
origin field in each routing record. Missing, empty, or non-text header values produce
"origin": null. The value is supplied by the caller; it is not inferred from User-Agent.
Older records without origin remain readable by the session stats endpoint.
A child agent can share its parent's session ID. To tell their calls apart, records also include
the agent identity from the request metadata: agent_id, parent_agent_id, agent_kind,
agent_role, and delegated (true when the request carries delegated sub-agent work). When the client
cannot identify its child agents, the record has "subagent_identity_unsupported": true. Each of
these fields is left out when it is not set, so records without agent identity are unchanged.
An llm_classifier route sends each task to classifier_target for a capability verdict, then
routes to weak_target or strong_target. Beyond the three targets it accepts these keys; only
base_threshold is required, and anything the judge cannot decide routes to strong_target:
| Key | Default | Meaning |
|---|---|---|
base_threshold |
required | Lowest solve probability that routes a task to weak_target. Raise it to send less traffic to the weak model. |
threshold_step |
0.0 |
Finite, non-negative amount added once for uncertain verdicts, unmatched_steps times for unmatched verdicts, and twice for unsupported verdicts. base_threshold + 2 * threshold_step must be at most 1. |
unmatched_steps |
1 |
Threshold steps for an unmatched verdict. 0, 1, or 2; 2 treats it like unsupported. |
classify_trigger |
every_request |
When the judge runs. every_request judges every request including tool continuations, user_turn judges each new user message and holds that target across the tool calls between, new_session judges once and reuses that target for the session. |
message_hash_fallback |
false |
Extends affinity to clients that send no session header, keying on the first user message. Requires classify_trigger = "new_session" or "user_turn". |
Session affinity retains a decision for the process lifetime, including a strong_target
fallback produced while the judge was unreachable. message_hash_fallback keys on request
content rather than a session id, so unrelated callers sending identical text share one
assignment.
A stage_router route scores tool-result and activity signals from recent turns to pick a
tier per turn, without an extra classifier call on every turn. Domain-specific exact tool names
can extend its built-in coding vocabulary through tool_semantics.observe,
tool_semantics.mutate, tool_semantics.plan, and neutral tool_semantics.new. capable_target,
efficient_target, picker (efficient_first or capable_first), and confidence_threshold
are required. All configured semantic names use exact ASCII case-insensitive matching. Optional
handoff notes, per-tier system prompts, and a capability-judge fallback are documented in
Stage-Router Routing.
Model discovery
GET /v1/models returns the standard data list and an empty Codex models list.
Each entry reports the route's declared tool_calling and vision under capabilities.
It reports the route's declared context_window as the top-level context_length field.
OpenAI-compatible clients such as Oh My Pi read context_length when they build their
model list from this endpoint. See
Use Switchyard with Oh My Pi.
Codex keeps its own model catalog and instructions. Select a Switchyard route explicitly
with codex --model route-id; route aliases do not appear automatically in Codex's model
picker. Unknown aliases use Codex's generic defaults and do not receive Switchyard's
route-specific context limits or tool settings.
For registered routes, the server returns HTTP 400 before dispatch when a request
contains inputs disabled by vision = false, reasoning = false, or
tool_calling = false. OpenAI errors use the unsupported_capability code;
Anthropic errors use invalid_request_error. Remove the unsupported input or select
a compatible route. Explicit true and unset declarations do not restrict requests.
Codex keeps its own model settings, so it may still send inputs that the server rejects.
The server preserves caller instructions and does not remove images, reasoning controls,
or tools to make a request fit. These checks apply to registered server routes;
transparent forwarding through fallback_client and direct library calls remain unchanged.
To add instructions for a target, set system_prompt on its [targets.<name>] entry.
Switchyard prepends that text when the selected target serves a completion and retains
the caller's instructions. Omit the setting to add no target instructions.
Codex "Approve for me"
With "Approve for me" on, Codex asks a reviewer model to check each action that
needs approval. Codex sends each review request to its own model provider, so
when Codex points at Switchyard, the review goes to Switchyard. Unless Codex is
logged in with an OpenAI API key, the request names the model ID
codex-auto-review. If no route has that id, the server returns HTTP 404
model_not_found. Codex treats a failed review as a denial, so it declines the
action.
You need the route if you use Codex through Switchyard with "Approve for me" on in the Default or Read Only mode. This includes Codex logged in with ChatGPT. Full Access mode never asks for approval, so it never sends review requests.
Add a route whose id is codex-auto-review. examples/run_codex.sh and
scripts/config/composite.toml forward the review unchanged to Codex's own
reviewer model on the ChatGPT backend:
[llm_clients.chatgpt_backend]
format = "openai_responses"
base_url = "https://chatgpt.com/backend-api/codex"
forward_auth = true
[targets.reviewer]
id = "codex-auto-review"
llm_client = "chatgpt_backend"
[routes.codex_auto_review]
id = "codex-auto-review"
type = "passthrough"
target = "reviewer"
To review with another model, point target at any other target. Use a small,
fast model at low reasoning effort. Codex waits for each review before it runs
the action, so a slow reviewer slows down every step that needs approval.
Codex already asks for low effort in each review request, so don't set a
higher reasoning_effort on the reviewer target. A larger model costs more and
takes longer per review, but OpenAI found that stronger models catch risky
actions more reliably
(Auto-review). The model must also
follow a JSON output schema, because Codex reads the reviewer's final message as
a JSON verdict.
When Codex is logged in with an OpenAI API key, it sends review requests to
gpt-5.6-luna instead. For that login, give the reviewer route
id = "gpt-5.6-luna". That route then receives every Codex request for the
gpt-5.6-luna model ID, not only review requests.
Codex can send reviews to another model ID only through a full replacement model
catalog: set the model_catalog_json config key, and set
auto_review_model_override on the session model's entry. That catalog must
list every model Codex uses and changes with Codex releases, so a Switchyard
route is simpler.
To run every action without a review, use Codex settings instead of a
Switchyard route. With approval_policy = "never" (-a never), Codex never
asks and sends no review requests. Commands still run inside the sandbox, and
anything the sandbox blocks fails back to the model:
approval_policy = "never"
sandbox_mode = "workspace-write"
[sandbox_workspace_write]
network_access = true # only if commands need the network
--dangerously-bypass-approvals-and-sandbox (--yolo, the Full Access preset)
runs everything with no sandbox and no approvals. To skip review only for some
commands, add an experimental prefix_rule(pattern = [...], decision = "allow")
rule to ~/.codex/rules/default.rules. See Codex's
Agent approvals & security
page.
The experimental desktop installer, cargo desktop install, which you run from
experimental/switchyard-desktop, does not overwrite an existing
~/.switchyard/composite.toml.
If you installed Switchyard before this route was added, add the
[targets.reviewer] and [routes.codex_auto_review] blocks to that file by
hand, then restart the server.
Endpoints
| Method | Path | Purpose |
|---|---|---|
POST |
/v1/chat/completions |
OpenAI Chat Completions |
POST |
/v1/messages |
Anthropic Messages |
POST |
/v1/responses |
OpenAI Responses |
POST |
/v1/decision |
Resolve selected and fallback targets without a post-routing answer call |
POST |
/v1/messages/count_tokens |
Token count from a route's Anthropic target |
POST |
/v1/responses/input_tokens |
Token count from a route's OpenAI Responses target |
POST |
/v1/responses/compact |
Compaction through a route's OpenAI Responses target |
ANY |
Any unmatched path | Raw forward through the optional fallback_client |
GET |
/v1/models |
Routes served by this deployment |
GET |
/v1/stats |
Per-model usage plus curated algorithm stats |
POST |
/v1/stats/reset |
Clear accumulated stats |
GET |
/metrics |
Prometheus text, see Metrics |
GET |
/health |
Liveness |
Requests name a route by its id, so POST /v1/chat/completions with "model": "switchyard/general"
routes through the [routes.general] entry above. The server translates between request formats
supported by the selected route and its forwarding clients.
Set the top-level fallback_client to an entry under [llm_clients] to proxy any otherwise
unmatched method and path through that client. The fallback client does not need a target. Requests
and responses are forwarded without translation, including paths, query strings, bodies, and model
identifiers. The caller's end-to-end headers, including authorization, are forwarded; hop-by-hop
headers are removed, and the client's configured API key, extra headers, format, and retry policy
are not applied. Without fallback_client, unmatched paths return 404.
The three model-bearing auxiliary endpoints above remain routed operations: they resolve the route name to a compatible target, rewrite the upstream model ID, and use that target's configured client.
POST /v1/decision accepts {"input_format": "openai_chat", "request": {...}}, where the
nested request names the route in model. It executes required classifier or judge calls, then
returns the selected target and ordered fallbacks with their model, format, base URL, and
extra_body. It does not make a post-routing answer call. Routing-time calls still execute, and
response-dependent algorithms such as escalation and advisor routing may produce an answer while
deciding. When they do, the endpoint includes the buffered answer as response, encoded in
input_format; otherwise the field is omitted. When the algorithm attaches outcome metadata, the
endpoint returns it as metadata, with the outcome ID, the algorithm name and the algorithm's
evidence.
For stage_router, algorithm_stats.stage_router groups routing decisions by source and semantic
target and summarizes its score, confidence, and input-dimension histograms. These values reset
with /v1/stats/reset; the process-lifetime counters on /metrics remain cumulative.
Token counting selects an Anthropic-format completion target, preferring target names or model IDs
containing opus, sonnet, then haiku. Other ties preserve the route's target order. Target
system prompts are applied to answer calls, not token-count requests.
Metrics
GET /metrics exposes Prometheus text from the server's process-wide OpenTelemetry provider.
Routed-call compatibility metrics are:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
switchyard_build_info |
gauge | version |
Constant 1 for this server version |
switchyard_total_requests |
gauge | none | Successful and failed final routed calls |
switchyard_total_errors |
gauge | none | Failed final routed calls |
switchyard_requests_total |
counter | model |
Successful final routed calls |
switchyard_errors_total |
counter | model |
Failed final routed calls |
switchyard_model_call_latency_ms |
histogram | model |
Successful final routed-call latency |
switchyard_llm_calls_total |
counter | algorithm, selected_model, outcome |
Logical routing and terminal model calls |
switchyard_llm_call_duration_ms |
histogram | algorithm, selected_model, outcome |
Logical routing and terminal model-call latency |
switchyard_prompt_tokens_total |
counter | model |
Input tokens, including cached and cache-creation tokens |
switchyard_completion_tokens_total |
counter | model |
Output tokens |
switchyard_cached_tokens_total |
counter | model |
Cached input tokens |
switchyard_cache_creation_tokens_total |
counter | model |
Cache-creation input tokens |
switchyard_reasoning_tokens_total |
counter | model |
Reasoning output tokens |
switchyard_total_latency_ms |
histogram | model |
Full-turn latency for successful routed responses |
switchyard_routing_overhead_ms |
histogram | algorithm |
Time spent producing the terminal routing outcome |
switchyard_classifier_fail_open_total |
counter | judge_model, reason |
Judge failures that made a classifier route without a verdict |
switchyard_client_responses_total |
counter | outcome |
Final LLM-route responses |
switchyard_upstream_attempts_total |
counter | outcome, code |
Actual upstream HTTP attempts |
switchyard_router_retry_recovered_total |
counter | none | Upstream operations recovered by a retry |
switchyard_classifier_fail_open_total counts requests that still reached a target after the
judge call failed. judge_model names the configured judge target, and reason is one of eight
fixed error categories.
switchyard_llm_calls_total and switchyard_llm_call_duration_ms retain the logical call
boundary across routing: libsy records classifier and judge calls, while libsy-llm-client
records the terminal call when the routing outcome requires one. A response already produced by
routing is counted only by its original libsy call. Terminal fallback and backend retries remain
one logical call labeled with the algorithm-selected model.
switchyard_total_latency_ms observes an aggregate when it becomes available or a stream when it
ends cleanly. Its clock starts in a router-wide middleware, before the request body is read and
decoded, so it measures request ingress through response completion. It still excludes connection
accept and TLS handshake, which hyper completes before the server sees the request.
switchyard_routing_overhead_ms records the elapsed time in run from run_started until
drive returns a routing outcome. It includes algorithm execution and any classifier or judge
calls made while driving the algorithm, and it is recorded before a terminal answer call begins.
No duration for the request-serving call is subtracted. If routing itself produced the answer,
that call occurred inside the measured drive interval. The metric carries only algorithm,
since the duration describes the router rather than the target it chose; a routing failure before
an outcome records nothing. Its buckets start at 0.1 ms via a view in the server; the SDK defaults
start at 5 ms.
See CONFIGURATION.md to add an LLM client, target, or algorithm.