Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust
23.2 KB
Raw Blame

switchyard-server

switchyard-server exposes libsy algorithms through OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages endpoints. A TOML file explicitly defines the LLM clients, targets, and algorithm routes served by the process.

The standalone binary listens on 127.0.0.1 unless --host is set. Pass --host 0.0.0.0 to accept remote clients. The provided Docker image and dev-server/switchyard.service opt into that address explicitly. A later --host argument overrides the container's default.

# routes.toml
schema_version = 1

[llm_clients.example]
format = "openai_chat"
base_url = "https://example.com/v1"
api_key_env = "API_KEY"
max_retries = 2

[targets.model_a]
id = "model/a"
llm_client = "example"
system_prompt = "Use the fast path for routine work."
extra_body = { service_tier = "priority" }

[targets.model_b]
id = "model/b"
llm_client = "example"

[routes.general]
id = "switchyard/general"
type = "random"
targets = ["model_a", "model_b"]
weights = [1, 3]
seed = 42

[routes.classified]
id = "switchyard/classified"
type = "llm_classifier"
mode = "capability"
classifier_target = "model_a"
strong_target = "model_a"
weak_target = "model_b"
base_threshold = 0.5

[routes.passthrough]
id = "switchyard/passthrough"
type = "passthrough"
target = "model_a"

[routes.stage]
id = "switchyard/stage"
type = "stage_router"
capable_target = "model_a"
efficient_target = "model_b"
picker = "efficient_first"
confidence_threshold = 0.5

[routes.stage.tool_semantics]
observe = ["lookup_customer"]
mutate = ["update_inventory"]
new = ["send_message"]
export API_KEY="..."
cargo install --locked switchyard-server
switchyard-server --config routes.toml

Ctrl+C and Unix SIGTERM stop new connections and allow active requests to drain for up to --shutdown-timeout (30 seconds by default) before they are terminated.

The server logs exactly one structured terminal event per LLM request: successful responses at INFO, 4xx responses at WARN, and 5xx responses at ERROR. Set RUST_LOG=switchyard_server=debug,libsy=debug to include routing decisions and nested failure details. A streaming failure is logged separately because it can occur after the response starts.

Target and route table names are local references. A target's id is the exact model ID sent upstream, and a route's id is the model clients send to select that algorithm.

Each target references an entry under llm_clients. All configured clients use TranslatingLlmClient; supported formats are openai_chat, openai_responses, and anthropic_messages. Supported algorithms are noop, random, passthrough, llm_classifier, stage_router, auto, composite, and advisor. The optional prefill-router feature also enables the experimental prefill_router. See its artifact requirements. An api_key_env value names an environment variable. The TOML never contains the secret itself. If omitted, the client sends no authentication. A client can set forward_auth = true instead of api_key_env to send the caller's credential to the configured upstream. OpenAI clients forward authorization, chatgpt-account-id, and x-openai-fedramp. Anthropic clients forward authorization or x-api-key. Enable this only when every forwarding client's base_url should receive the caller's login. Other application headers are preserved and may contain provider-specific credentials. A route's forwarding clients must use one credential family: all OpenAI formats or all anthropic_messages. The exception is one host that serves both formats, such as an LLM gateway that accepts each caller's gateway key on every endpoint: a route may mix the two families when all of its forwarding clients use the same scheme, host, and port. Such a route serves Chat Completions and Responses callers and forwards the caller's bearer token to every forwarding client. The server returns 400 to a caller whose API the route does not serve. Clients that use api_key_env send the server's own key, so these limits do not apply to them. Target-level extra_body values are shallow-merged into the upstream request when the request does not already contain that key. Target-level system_prompt values are prepended when that target serves a completion. Selected and fallback targets are prepared independently. max_retries defaults to 2 and applies to transport failures, timeouts, HTTP 408/429, and 5xx responses.

Random-route weights are relative, follow target order, and do not need to sum to one. Omit them for equal weighting. The optional seed reproduces the selection sequence for the same call order.

Session routing log

Pass --routing-log-file PATH to append one JSON record after each completed routed response. Streaming responses are recorded after the stream drains. Each record names the client-facing route_id, routing algorithm, served model, tier, and token usage so routes sharing a backend remain distinguishable. When enabled, GET /v1/routing/session-stats?session_id=ID rescans the durable log and returns call and token totals for that normalized session ID, normally supplied as x-switchyard-session-id, grouped by served model. The legacy proxy_x_session_id remains a fallback when no normalized session ID is present. The endpoint returns 404 when the session has no records and is not registered when routing logging is disabled.

The routing log saves each call's reported charge in USD. When the upstream reports no charge, the server estimates it using bundled rates. A reported charge, including zero, takes precedence over the bundled rates. Bundled rates stay fixed until you install a version with updated rates. The table includes text-token rates for OpenAI, Claude, Gemini, and DeepSeek models, plus OpenRouter rates for other popular models. Model IDs match exactly, including provider prefixes; an unlisted ID has no estimated cost. Direct-provider and OpenRouter rates use separate IDs, such as gemini-3.8-flash and google/gemini-3.8-flash. Estimates exclude long-context and service-tier adjustments, OpenAI's 12-hour cache-write pricing, tool fees, and time-based cache storage. Each model has one rate at all times of day. Billed charges can differ when a gateway selects another provider or a promotion ends. The table lists the Gemini promotion end dates. Use --pricing-file prices.toml to replace the entire bundled table with your own rates. Use a .toml file with quoted served model IDs and rates in USD per million tokens:

["upstream/model"]
input = 1
output = 5
cache_read = 0.1
cache_write = 2

The .toml extension check ignores case. Other filenames are read as JSON, so existing JSON overrides still work. TOML and JSON also accept groups of exact IDs that share one set of rates, as shown in the pi guide. The bundled table uses this grouped format and can be supplied to --pricing-file.

Session stats include cost totals. Missing usage or rates leave cost unknown; known_usd keeps the priced subtotal and total_usd is null when any cost is unknown. Rates load once at startup, and saved costs do not change when prices change. An unreadable or invalid --pricing-file stops startup; the server does not use bundled rates instead.

The pi guide explains how to supply rates for your own models. The desktop app uses the same bundled rates by default. For custom rates, point its pricing_file setting at the same TOML or JSON file as the server's --pricing-file. The desktop app totals these recorded costs, so its totals can differ from what your provider bills.

Clients can send x-switchyard-origin: codex-cli (or another client label) to include an origin field in each routing record. Missing, empty, or non-text header values produce "origin": null. The value is supplied by the caller; it is not inferred from User-Agent. Older records without origin remain readable by the session stats endpoint.

A child agent can share its parent's session ID. To tell their calls apart, records also include the agent identity from the request metadata: agent_id, parent_agent_id, agent_kind, agent_role, and delegated (true when the request carries delegated sub-agent work). When the client cannot identify its child agents, the record has "subagent_identity_unsupported": true. Each of these fields is left out when it is not set, so records without agent identity are unchanged.

An llm_classifier route sends each task to classifier_target for a capability verdict, then routes to weak_target or strong_target. Beyond the three targets it accepts these keys; only base_threshold is required, and anything the judge cannot decide routes to strong_target:

Key Default Meaning
base_threshold required Lowest solve probability that routes a task to weak_target. Raise it to send less traffic to the weak model.
threshold_step 0.0 Finite, non-negative amount added once for uncertain verdicts, unmatched_steps times for unmatched verdicts, and twice for unsupported verdicts. base_threshold + 2 * threshold_step must be at most 1.
unmatched_steps 1 Threshold steps for an unmatched verdict. 0, 1, or 2; 2 treats it like unsupported.
classify_trigger every_request When the judge runs. every_request judges every request including tool continuations, user_turn judges each new user message and holds that target across the tool calls between, new_session judges once and reuses that target for the session.
message_hash_fallback false Extends affinity to clients that send no session header, keying on the first user message. Requires classify_trigger = "new_session" or "user_turn".

Session affinity retains a decision for the process lifetime, including a strong_target fallback produced while the judge was unreachable. message_hash_fallback keys on request content rather than a session id, so unrelated callers sending identical text share one assignment.

A stage_router route scores tool-result and activity signals from recent turns to pick a tier per turn, without an extra classifier call on every turn. Domain-specific exact tool names can extend its built-in coding vocabulary through tool_semantics.observe, tool_semantics.mutate, tool_semantics.plan, and neutral tool_semantics.new. capable_target, efficient_target, picker (efficient_first or capable_first), and confidence_threshold are required. All configured semantic names use exact ASCII case-insensitive matching. Optional handoff notes, per-tier system prompts, and a capability-judge fallback are documented in Stage-Router Routing.

Model discovery

GET /v1/models returns the standard data list and an empty Codex models list. Each entry reports the route's declared tool_calling and vision under capabilities. It reports the route's declared context_window as the top-level context_length field. OpenAI-compatible clients such as Oh My Pi read context_length when they build their model list from this endpoint. See Use Switchyard with Oh My Pi.

Codex keeps its own model catalog and instructions. Select a Switchyard route explicitly with codex --model route-id; route aliases do not appear automatically in Codex's model picker. Unknown aliases use Codex's generic defaults and do not receive Switchyard's route-specific context limits or tool settings.

For registered routes, the server returns HTTP 400 before dispatch when a request contains inputs disabled by vision = false, reasoning = false, or tool_calling = false. OpenAI errors use the unsupported_capability code; Anthropic errors use invalid_request_error. Remove the unsupported input or select a compatible route. Explicit true and unset declarations do not restrict requests.

Codex keeps its own model settings, so it may still send inputs that the server rejects. The server preserves caller instructions and does not remove images, reasoning controls, or tools to make a request fit. These checks apply to registered server routes; transparent forwarding through fallback_client and direct library calls remain unchanged.

To add instructions for a target, set system_prompt on its [targets.<name>] entry. Switchyard prepends that text when the selected target serves a completion and retains the caller's instructions. Omit the setting to add no target instructions.

Codex "Approve for me"

With "Approve for me" on, Codex asks a reviewer model to check each action that needs approval. Codex sends each review request to its own model provider, so when Codex points at Switchyard, the review goes to Switchyard. Unless Codex is logged in with an OpenAI API key, the request names the model ID codex-auto-review. If no route has that id, the server returns HTTP 404 model_not_found. Codex treats a failed review as a denial, so it declines the action.

You need the route if you use Codex through Switchyard with "Approve for me" on in the Default or Read Only mode. This includes Codex logged in with ChatGPT. Full Access mode never asks for approval, so it never sends review requests.

Add a route whose id is codex-auto-review. examples/run_codex.sh and scripts/config/composite.toml forward the review unchanged to Codex's own reviewer model on the ChatGPT backend:

[llm_clients.chatgpt_backend]
format = "openai_responses"
base_url = "https://chatgpt.com/backend-api/codex"
forward_auth = true

[targets.reviewer]
id = "codex-auto-review"
llm_client = "chatgpt_backend"

[routes.codex_auto_review]
id = "codex-auto-review"
type = "passthrough"
target = "reviewer"

To review with another model, point target at any other target. Use a small, fast model at low reasoning effort. Codex waits for each review before it runs the action, so a slow reviewer slows down every step that needs approval. Codex already asks for low effort in each review request, so don't set a higher reasoning_effort on the reviewer target. A larger model costs more and takes longer per review, but OpenAI found that stronger models catch risky actions more reliably (Auto-review). The model must also follow a JSON output schema, because Codex reads the reviewer's final message as a JSON verdict.

When Codex is logged in with an OpenAI API key, it sends review requests to gpt-5.6-luna instead. For that login, give the reviewer route id = "gpt-5.6-luna". That route then receives every Codex request for the gpt-5.6-luna model ID, not only review requests.

Codex can send reviews to another model ID only through a full replacement model catalog: set the model_catalog_json config key, and set auto_review_model_override on the session model's entry. That catalog must list every model Codex uses and changes with Codex releases, so a Switchyard route is simpler.

To run every action without a review, use Codex settings instead of a Switchyard route. With approval_policy = "never" (-a never), Codex never asks and sends no review requests. Commands still run inside the sandbox, and anything the sandbox blocks fails back to the model:

approval_policy = "never"
sandbox_mode = "workspace-write"

[sandbox_workspace_write]
network_access = true  # only if commands need the network

--dangerously-bypass-approvals-and-sandbox (--yolo, the Full Access preset) runs everything with no sandbox and no approvals. To skip review only for some commands, add an experimental prefix_rule(pattern = [...], decision = "allow") rule to ~/.codex/rules/default.rules. See Codex's Agent approvals & security page.

The experimental desktop installer, cargo desktop install, which you run from experimental/switchyard-desktop, does not overwrite an existing ~/.switchyard/composite.toml. If you installed Switchyard before this route was added, add the [targets.reviewer] and [routes.codex_auto_review] blocks to that file by hand, then restart the server.

Endpoints

Method Path Purpose
POST /v1/chat/completions OpenAI Chat Completions
POST /v1/messages Anthropic Messages
POST /v1/responses OpenAI Responses
POST /v1/decision Resolve selected and fallback targets without a post-routing answer call
POST /v1/messages/count_tokens Token count from a route's Anthropic target
POST /v1/responses/input_tokens Token count from a route's OpenAI Responses target
POST /v1/responses/compact Compaction through a route's OpenAI Responses target
ANY Any unmatched path Raw forward through the optional fallback_client
GET /v1/models Routes served by this deployment
GET /v1/stats Per-model usage plus curated algorithm stats
POST /v1/stats/reset Clear accumulated stats
GET /metrics Prometheus text, see Metrics
GET /health Liveness

Requests name a route by its id, so POST /v1/chat/completions with "model": "switchyard/general" routes through the [routes.general] entry above. The server translates between request formats supported by the selected route and its forwarding clients.

Set the top-level fallback_client to an entry under [llm_clients] to proxy any otherwise unmatched method and path through that client. The fallback client does not need a target. Requests and responses are forwarded without translation, including paths, query strings, bodies, and model identifiers. The caller's end-to-end headers, including authorization, are forwarded; hop-by-hop headers are removed, and the client's configured API key, extra headers, format, and retry policy are not applied. Without fallback_client, unmatched paths return 404.

The three model-bearing auxiliary endpoints above remain routed operations: they resolve the route name to a compatible target, rewrite the upstream model ID, and use that target's configured client.

POST /v1/decision accepts {"input_format": "openai_chat", "request": {...}}, where the nested request names the route in model. It executes required classifier or judge calls, then returns the selected target and ordered fallbacks with their model, format, base URL, and extra_body. It does not make a post-routing answer call. Routing-time calls still execute, and response-dependent algorithms such as escalation and advisor routing may produce an answer while deciding. When they do, the endpoint includes the buffered answer as response, encoded in input_format; otherwise the field is omitted. When the algorithm attaches outcome metadata, the endpoint returns it as metadata, with the outcome ID, the algorithm name and the algorithm's evidence.

For stage_router, algorithm_stats.stage_router groups routing decisions by source and semantic target and summarizes its score, confidence, and input-dimension histograms. These values reset with /v1/stats/reset; the process-lifetime counters on /metrics remain cumulative.

Token counting selects an Anthropic-format completion target, preferring target names or model IDs containing opus, sonnet, then haiku. Other ties preserve the route's target order. Target system prompts are applied to answer calls, not token-count requests.

Metrics

GET /metrics exposes Prometheus text from the server's process-wide OpenTelemetry provider. Routed-call compatibility metrics are:

Metric Type Labels Meaning
switchyard_build_info gauge version Constant 1 for this server version
switchyard_total_requests gauge none Successful and failed final routed calls
switchyard_total_errors gauge none Failed final routed calls
switchyard_requests_total counter model Successful final routed calls
switchyard_errors_total counter model Failed final routed calls
switchyard_model_call_latency_ms histogram model Successful final routed-call latency
switchyard_llm_calls_total counter algorithm, selected_model, outcome Logical routing and terminal model calls
switchyard_llm_call_duration_ms histogram algorithm, selected_model, outcome Logical routing and terminal model-call latency
switchyard_prompt_tokens_total counter model Input tokens, including cached and cache-creation tokens
switchyard_completion_tokens_total counter model Output tokens
switchyard_cached_tokens_total counter model Cached input tokens
switchyard_cache_creation_tokens_total counter model Cache-creation input tokens
switchyard_reasoning_tokens_total counter model Reasoning output tokens
switchyard_total_latency_ms histogram model Full-turn latency for successful routed responses
switchyard_routing_overhead_ms histogram algorithm Time spent producing the terminal routing outcome
switchyard_classifier_fail_open_total counter judge_model, reason Judge failures that made a classifier route without a verdict
switchyard_client_responses_total counter outcome Final LLM-route responses
switchyard_upstream_attempts_total counter outcome, code Actual upstream HTTP attempts
switchyard_router_retry_recovered_total counter none Upstream operations recovered by a retry

switchyard_classifier_fail_open_total counts requests that still reached a target after the judge call failed. judge_model names the configured judge target, and reason is one of eight fixed error categories.

switchyard_llm_calls_total and switchyard_llm_call_duration_ms retain the logical call boundary across routing: libsy records classifier and judge calls, while libsy-llm-client records the terminal call when the routing outcome requires one. A response already produced by routing is counted only by its original libsy call. Terminal fallback and backend retries remain one logical call labeled with the algorithm-selected model.

switchyard_total_latency_ms observes an aggregate when it becomes available or a stream when it ends cleanly. Its clock starts in a router-wide middleware, before the request body is read and decoded, so it measures request ingress through response completion. It still excludes connection accept and TLS handshake, which hyper completes before the server sees the request.

switchyard_routing_overhead_ms records the elapsed time in run from run_started until drive returns a routing outcome. It includes algorithm execution and any classifier or judge calls made while driving the algorithm, and it is recorded before a terminal answer call begins. No duration for the request-serving call is subtracted. If routing itself produced the answer, that call occurred inside the measured drive interval. The metric carries only algorithm, since the duration describes the router rather than the target it chose; a routing failure before an outcome records nothing. Its buckets start at 0.1 ms via a view in the server; the SDK defaults start at 5 ms.

See CONFIGURATION.md to add an LLM client, target, or algorithm.