Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
libsy-llm-client
An HTTP client that speaks Switchyard's neutral IR directly. You hand it a
switchyard_protocol::Request and a ModelId; it looks up the configured backend,
encodes the request to that backend's wire format, adds auth and forwards your
headers, makes the call with a shared reqwest::Client, and decodes the reply
back into a switchyard_protocol::Response — buffered or streamed.
It also pairs the client with a libsy algorithm: [run] drives
[Algorithm::run_stream], serves routing-time calls, and consumes the terminal routing outcome.
When routing has not already produced the answer, run makes the terminal call and owns backend
retries plus ordered candidate fallback.
It depends on switchyard-libsy, switchyard-protocol, and
switchyard-translation; no server, no provider SDK.
Concepts
- Target configs. Each
ModelConfighas a target name, a defaultBackend, and optional backends for other wire formats. The target name selects settings.with_provider_modelsupplies the model ID sent upstream. For example, targetsefficientandcapablecan use the same model with different reasoning efforts. - Model ids. A
ModelIdis a newtype overStringthat behaves like one — it derefs tostr, compares against string literals, and converts from&strorStringwith.into(). Anywhere below that takesimpl Into<ModelId>accepts a bare literal; the borrowed positions need aModelIdvalue to reference. - Backends. A
Backendis one ofOpenAiChat,OpenAiResponses, orAnthropic, each wrapping anHttpBackendConfig(base_url,api_key, staticextra_headers, defaultextra_bodyfields, andmax_retries). The variant fixes the URL path and auth scheme (Bearer vsx-api-key+anthropic-version). - Model rewrite. The selected target's provider model ID replaces the request's
modelfield. If no provider model is configured, the target name is used. - Streaming is chosen by the encoded request body. If the body has
stream: trueafterextra_bodyis applied, you getLlmResponse::Stream; otherwiseLlmResponse::Agg. OpenAI Chat streaming requests defaultstream_options.include_usagetotrue; an explicit caller value is preserved.
Add the dependency
Within this workspace:
[dependencies]
futures-util = "0.3"
switchyard-libsy = { path = "../libsy" }
switchyard-llm-client = { path = "../libsy-llm-client" }
switchyard-protocol = { path = "../protocol" }
switchyard-translation = { path = "../switchyard-translation" } # for WireFormat
Quickstart
Build a client
use std::collections::BTreeMap;
use switchyard_llm_client::{
Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient,
};
fn build_client() -> switchyard_llm_client::Result<TranslatingLlmClient> {
let openai = HttpBackendConfig {
base_url: "https://api.openai.com/v1".to_string(),
api_key: std::env::var("OPENAI_API_KEY").ok(),
forward_auth: false,
extra_headers: BTreeMap::new(),
extra_body: BTreeMap::new(),
omit_body_fields: Default::default(),
reasoning_effort: None,
max_retries: 2,
failure_cooldown: std::time::Duration::from_secs(5),
timeout: None,
};
let models = [ModelConfig::new(
"gpt-4o-mini",
Backend::OpenAiChat(openai),
None,
)];
TranslatingLlmClient::new(&models)
}
Buffered call
use switchyard_llm_client::{LlmClientError, TranslatingLlmClient};
use switchyard_protocol::{completion_text, text_request, LlmResponse, ModelId, Request};
async fn ask(client: &TranslatingLlmClient) -> switchyard_llm_client::Result<String> {
let request = Request {
llm_request: text_request(None, "Say hello in five words."),
raw_request: None,
metadata: None,
};
// The explicit target takes precedence over the request model field.
let model = ModelId::from("gpt-4o-mini");
let response = client.call_rewrite_model(request, Some(&model)).await?;
match response.llm_response {
LlmResponse::Agg(agg) => Ok(completion_text(&agg)),
LlmResponse::Stream(_) => Err(LlmClientError::InvalidResponse {
source: "expected a buffered response".into(),
}),
}
}
Streaming call
Set stream on the IR request and drive the returned chunk stream:
use futures_util::StreamExt;
use switchyard_llm_client::TranslatingLlmClient;
use switchyard_protocol::{text_request, LlmResponse, LlmResponseChunk, ModelId, Request};
async fn stream(
client: &TranslatingLlmClient,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut llm_request = text_request(None, "Count to five.");
llm_request.stream = true;
let request = Request { llm_request, raw_request: None, metadata: None };
let model = ModelId::from("gpt-4o-mini");
let response = client.call_rewrite_model(request, Some(&model)).await?;
if let LlmResponse::Stream(mut events) = response.llm_response {
while let Some(item) = events.next().await {
// Each event carries zero or more provider-neutral chunks; the ones
// ignored here are usage, tool-call deltas, and message start/stop.
for chunk in item?.normalized() {
if let LlmResponseChunk::TextDelta { text, .. } = chunk {
print!("{text}");
}
}
}
}
Ok(())
}
Routing an algorithm
[run] takes a libsy algorithm and a [ClientRouter], and returns the algorithm-selected
ModelId plus the final response. Routing-time CallModels are served while the algorithm
runs. The terminal outcome either already contains the answer or supplies the selected model and
ordered fallbacks for the client to try. ClientRouter::single is the single-provider case:
Routing-time calls run concurrently. If an algorithm drops one of its waiting call futures, the corresponding HTTP work is dropped while other calls continue. Ending or dropping the run drops all outstanding call handlers. A returned response stream stays alive until its consumer finishes or drops it. These are local cancellation guarantees; a provider may continue generation after disconnect.
use std::sync::Arc;
use switchyard_libsy::{Algorithm, RuntimeModels};
use switchyard_llm_client::{ClientRouter, TranslatingLlmClient};
use switchyard_protocol::Request;
async fn route(
algorithm: Arc<dyn Algorithm>,
client: Arc<TranslatingLlmClient>,
request: Request,
models: Arc<RuntimeModels>,
) -> switchyard_libsy::Result<String> {
let clients = ClientRouter::single(client);
let (selected_model, _response) =
switchyard_llm_client::run(algorithm, clients, request, models, None).await?;
Ok(selected_model.to_string())
}
When targets are served by different clients — a judge on one provider and the serving
models on another, say — build the router from ModelId -> client with
[ClientRouter::new] instead. A model missing from that map fails with
LlmClientError::Configuration rather than silently going to another provider.
use std::collections::HashMap;
use std::sync::Arc;
use switchyard_llm_client::ClientRouter;
use switchyard_protocol::{ModelId, RoutedLlmClient};
fn split_router(
judge: Arc<dyn RoutedLlmClient>,
serving: Arc<dyn RoutedLlmClient>,
) -> ClientRouter {
ClientRouter::new(HashMap::from([
(ModelId::from("gpt-4o-mini"), judge),
(ModelId::from("claude-sonnet-4-5"), serving),
]))
}
A router is not itself a client: [ClientRouter::route] hands back a
[switchyard_protocol::RoutedLlmClient] and the caller makes the call.
Cross-format translation
The request/response are translated through the neutral IR, so the inbound shape
you build and the backend's wire format are independent. Pointing a
WireFormat::AnthropicMessages backend at an Anthropic endpoint while building
requests with the same helpers works the same way. Set request.metadata.wire_format
to select a non-default backend before calling call_rewrite_model. Register several
formats under one model to serve it over more than one upstream API:
use switchyard_llm_client::{
Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient,
};
fn build_multi_format_client(
openai_chat: HttpBackendConfig,
openai_responses: HttpBackendConfig,
anthropic: HttpBackendConfig,
) -> switchyard_llm_client::Result<TranslatingLlmClient> {
let models = [ModelConfig::new(
"my-model",
Backend::OpenAiChat(openai_chat),
Some(vec![
Backend::OpenAiResponses(openai_responses),
Backend::Anthropic(anthropic),
]),
)];
TranslatingLlmClient::new(&models)
}
Headers & auth
- The
Backendvariant sets auth: OpenAI formats sendAuthorization: Bearer <key>; Anthropic sendsx-api-key: <key>plusanthropic-version. - By default, only
x-request-idfromrequest.metadata.http_headersis forwarded. Withforward_auth, application headers are forwarded, while connection, destination, body, and the selected provider's owned headers are removed or rebuilt. HttpBackendConfig::forward_authuses the caller's credential instead of the backend's configured key and forwards other application headers. OpenAI backends normalizeauthorization,chatgpt-account-id, andx-openai-fedramp. Anthropic backends forwardauthorizationorx-api-key; they also keepoauth-*values fromanthropic-betaand remove other caller-supplied beta values. The forwarding backends in one route must use one credential family (OpenAI or Anthropic) unless they all use the same scheme, host, and port, such as one LLM gateway that accepts the caller's gateway key on every endpoint. Such a route serves Chat Completions and Responses callers and forwards their bearer token to every forwarding backend. Backends that send a configured key are not restricted. Headers owned by other providers are preserved as application headers.- Per-backend custom headers go in
HttpBackendConfig::extra_headers. Set credentials withapi_key. OpenAI backends rejectAuthorization; Anthropic backends rejectx-api-keyandanthropic-version. Header names are case-insensitive. Configured values replace caller values for the same header name. - Per-target top-level request defaults go in
HttpBackendConfig::extra_body. The merge is shallow and fields already present in the request take precedence. - Anthropic reconstruction preserves the caller's native
thinkingsettings, including disabled thinking and manual budgets.output_config.effortis kept separately. Effort implies adaptive thinking only when nativethinkingis absent. To replace either top-level field for a target, list it inomit_body_fieldsand supply its replacement inextra_body. Omission runs before defaults are merged. Targetreasoning_effortoverrides apply to OpenAI backends. HttpBackendConfig::max_retriescontrols additional attempts after retryable transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body transport failures are retried; streaming body failures are not replayed after the response has been returned.- If an OpenAI Chat backend explicitly rejects
chat_template_kwargs = {"enable_thinking": false}with HTTP 400 or 422, the client retries once without that field. This compatibility retry is separate frommax_retriesand shares the original timeout.omit_body_fieldscan remove it from the first attempt. - Codex endpoints whose URL path ends in
/backend-api/codex/responsesreceive stringinputas a message-item list, including preserved requests. Other fields, array input, non-Codex endpoints, and the translation codec's exact preservation remain unchanged. ModelConfig::with_responses_reasoningcontrols Responses reasoning replay. The default,Auto, uses the final outgoing model ID after body overrides, falling back to the configured provider model when the field is omitted. OpenAI targets usePreserveEncrypted: plaintext is removed and encrypted provider state is retained. Stored reasoning IDs are retained unless the request setsstore: false. Other targets usePreservePlaintext:summaryandcontentare retained and opaqueencrypted_contentis removed. Switchyard's translated Anthropic reasoning payload is always removed. SetDropfor a backend that cannot consume provider reasoning. Messages and tool history are retained.HttpBackendConfig::failure_cooldownskips completion calls to a backend briefly after an exhausted transient failure. Zero disables it. State is shared per model within the client. Calls resume when the cooldown expires. Deployment TOML defaultsfailure_cooldown_msto5000(5 seconds); set it to0to disable cooldown. Withforward_auth, HTTP 429 keeps its request-local retries and fallback while leaving shared cooldown state unchanged.HttpBackendConfig::timeoutbounds one complete response, including retries, retry delays, and every stream read. Expiry returnsLlmClientError::Timeout, either from the call or from the returned stream, which then ends.Noneleaves the wait unbounded.
run and decide collect streams used during routing, including answers that an
algorithm must inspect, before returning them to the algorithm. They retain the
provider events for replay. After the configured retries, a client failure stops
run or decide by default. Calls with CallModel::recover_errors enabled return
the error to the algorithm so it can apply its fallback policy. Capability
classifiers and advisor consults default to fail_open = true. Set fail_open = false
to stop the request on their client failures. Recovery includes client errors and
deadlines during stream collection. Custom hosts can honor the same flag when serving calls.
Retries replay the same upstream request to the same model. After routing completes,
non-timeout failures may try another completion candidate. A timeout stops the call;
a connection that times out before it is established is a Transport failure.
A transport failure can duplicate a request that the provider processed but did not
finish returning. Each attempt is counted once; expiry during a retry delay or
after a stream has started does not count another attempt.
Errors
call_rewrite_model returns LlmClientError:
| Variant | When |
|---|---|
InvalidRequest { message } |
the request does not identify a model |
Configuration { message } |
the model or requested wire format has no configured backend |
RequestTranslation(msg) |
decoding the inbound request failed in the translation engine |
RequestEncoding(msg) |
re-encoding an already-decoded request to the wire format failed (internal fault) |
ResponseTranslation(msg) |
response decoding or encoding failed in the translation engine |
Timeout { source } |
request or response body read exceeded its timeout |
Transport { source } |
connection failure, including a connect timeout, or another transport failure |
ContextWindowExceeded { model, message } |
upstream 400 detected as a context overflow (checked before UpstreamHttp, so callers can evict-and-retry) |
UpstreamHttp { status, body } |
any other non-2xx upstream response |
InvalidResponse { source } |
the upstream response could not be decoded |
Other(source) |
a client-specific failure outside the shared categories |