Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust
15.9 KB
Raw Blame

libsy-llm-client

An HTTP client that speaks Switchyard's neutral IR directly. You hand it a switchyard_protocol::Request and a ModelId; it looks up the configured backend, encodes the request to that backend's wire format, adds auth and forwards your headers, makes the call with a shared reqwest::Client, and decodes the reply back into a switchyard_protocol::Response — buffered or streamed.

It also pairs the client with a libsy algorithm: [run] drives [Algorithm::run_stream], serves routing-time calls, and consumes the terminal routing outcome. When routing has not already produced the answer, run makes the terminal call and owns backend retries plus ordered candidate fallback.

It depends on switchyard-libsy, switchyard-protocol, and switchyard-translation; no server, no provider SDK.

Concepts

  • Target configs. Each ModelConfig has a target name, a default Backend, and optional backends for other wire formats. The target name selects settings. with_provider_model supplies the model ID sent upstream. For example, targets efficient and capable can use the same model with different reasoning efforts.
  • Model ids. A ModelId is a newtype over String that behaves like one — it derefs to str, compares against string literals, and converts from &str or String with .into(). Anywhere below that takes impl Into<ModelId> accepts a bare literal; the borrowed positions need a ModelId value to reference.
  • Backends. A Backend is one of OpenAiChat, OpenAiResponses, or Anthropic, each wrapping an HttpBackendConfig (base_url, api_key, static extra_headers, default extra_body fields, and max_retries). The variant fixes the URL path and auth scheme (Bearer vs x-api-key + anthropic-version).
  • Model rewrite. The selected target's provider model ID replaces the request's model field. If no provider model is configured, the target name is used.
  • Streaming is chosen by the encoded request body. If the body has stream: true after extra_body is applied, you get LlmResponse::Stream; otherwise LlmResponse::Agg. OpenAI Chat streaming requests default stream_options.include_usage to true; an explicit caller value is preserved.

Add the dependency

Within this workspace:

[dependencies]
futures-util = "0.3"
switchyard-libsy = { path = "../libsy" }
switchyard-llm-client = { path = "../libsy-llm-client" }
switchyard-protocol = { path = "../protocol" }
switchyard-translation = { path = "../switchyard-translation" }   # for WireFormat

Quickstart

Build a client

use std::collections::BTreeMap;
use switchyard_llm_client::{
    Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient,
};

fn build_client() -> switchyard_llm_client::Result<TranslatingLlmClient> {
    let openai = HttpBackendConfig {
        base_url: "https://api.openai.com/v1".to_string(),
        api_key: std::env::var("OPENAI_API_KEY").ok(),
        forward_auth: false,
        extra_headers: BTreeMap::new(),
        extra_body: BTreeMap::new(),
        omit_body_fields: Default::default(),
        reasoning_effort: None,
        max_retries: 2,
        failure_cooldown: std::time::Duration::from_secs(5),
        timeout: None,
    };

    let models = [ModelConfig::new(
        "gpt-4o-mini",
        Backend::OpenAiChat(openai),
        None,
    )];

    TranslatingLlmClient::new(&models)
}

Buffered call

use switchyard_llm_client::{LlmClientError, TranslatingLlmClient};
use switchyard_protocol::{completion_text, text_request, LlmResponse, ModelId, Request};

async fn ask(client: &TranslatingLlmClient) -> switchyard_llm_client::Result<String> {
    let request = Request {
        llm_request: text_request(None, "Say hello in five words."),
        raw_request: None,
        metadata: None,
    };

    // The explicit target takes precedence over the request model field.
    let model = ModelId::from("gpt-4o-mini");
    let response = client.call_rewrite_model(request, Some(&model)).await?;

    match response.llm_response {
        LlmResponse::Agg(agg) => Ok(completion_text(&agg)),
        LlmResponse::Stream(_) => Err(LlmClientError::InvalidResponse {
            source: "expected a buffered response".into(),
        }),
    }
}

Streaming call

Set stream on the IR request and drive the returned chunk stream:

use futures_util::StreamExt;
use switchyard_llm_client::TranslatingLlmClient;
use switchyard_protocol::{text_request, LlmResponse, LlmResponseChunk, ModelId, Request};

async fn stream(
    client: &TranslatingLlmClient,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let mut llm_request = text_request(None, "Count to five.");
    llm_request.stream = true;
    let request = Request { llm_request, raw_request: None, metadata: None };

    let model = ModelId::from("gpt-4o-mini");
    let response = client.call_rewrite_model(request, Some(&model)).await?;

    if let LlmResponse::Stream(mut events) = response.llm_response {
        while let Some(item) = events.next().await {
            // Each event carries zero or more provider-neutral chunks; the ones
            // ignored here are usage, tool-call deltas, and message start/stop.
            for chunk in item?.normalized() {
                if let LlmResponseChunk::TextDelta { text, .. } = chunk {
                    print!("{text}");
                }
            }
        }
    }
    Ok(())
}

Routing an algorithm

[run] takes a libsy algorithm and a [ClientRouter], and returns the algorithm-selected ModelId plus the final response. Routing-time CallModels are served while the algorithm runs. The terminal outcome either already contains the answer or supplies the selected model and ordered fallbacks for the client to try. ClientRouter::single is the single-provider case:

Routing-time calls run concurrently. If an algorithm drops one of its waiting call futures, the corresponding HTTP work is dropped while other calls continue. Ending or dropping the run drops all outstanding call handlers. A returned response stream stays alive until its consumer finishes or drops it. These are local cancellation guarantees; a provider may continue generation after disconnect.

use std::sync::Arc;
use switchyard_libsy::{Algorithm, RuntimeModels};
use switchyard_llm_client::{ClientRouter, TranslatingLlmClient};
use switchyard_protocol::Request;

async fn route(
    algorithm: Arc<dyn Algorithm>,
    client: Arc<TranslatingLlmClient>,
    request: Request,
    models: Arc<RuntimeModels>,
) -> switchyard_libsy::Result<String> {
    let clients = ClientRouter::single(client);
    let (selected_model, _response) =
        switchyard_llm_client::run(algorithm, clients, request, models, None).await?;
    Ok(selected_model.to_string())
}

When targets are served by different clients — a judge on one provider and the serving models on another, say — build the router from ModelId -> client with [ClientRouter::new] instead. A model missing from that map fails with LlmClientError::Configuration rather than silently going to another provider.

use std::collections::HashMap;
use std::sync::Arc;
use switchyard_llm_client::ClientRouter;
use switchyard_protocol::{ModelId, RoutedLlmClient};

fn split_router(
    judge: Arc<dyn RoutedLlmClient>,
    serving: Arc<dyn RoutedLlmClient>,
) -> ClientRouter {
    ClientRouter::new(HashMap::from([
        (ModelId::from("gpt-4o-mini"), judge),
        (ModelId::from("claude-sonnet-4-5"), serving),
    ]))
}

A router is not itself a client: [ClientRouter::route] hands back a [switchyard_protocol::RoutedLlmClient] and the caller makes the call.

Cross-format translation

The request/response are translated through the neutral IR, so the inbound shape you build and the backend's wire format are independent. Pointing a WireFormat::AnthropicMessages backend at an Anthropic endpoint while building requests with the same helpers works the same way. Set request.metadata.wire_format to select a non-default backend before calling call_rewrite_model. Register several formats under one model to serve it over more than one upstream API:

use switchyard_llm_client::{
    Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient,
};

fn build_multi_format_client(
    openai_chat: HttpBackendConfig,
    openai_responses: HttpBackendConfig,
    anthropic: HttpBackendConfig,
) -> switchyard_llm_client::Result<TranslatingLlmClient> {
    let models = [ModelConfig::new(
        "my-model",
        Backend::OpenAiChat(openai_chat),
        Some(vec![
            Backend::OpenAiResponses(openai_responses),
            Backend::Anthropic(anthropic),
        ]),
    )];

    TranslatingLlmClient::new(&models)
}

Headers & auth

  • The Backend variant sets auth: OpenAI formats send Authorization: Bearer <key>; Anthropic sends x-api-key: <key> plus anthropic-version.
  • By default, only x-request-id from request.metadata.http_headers is forwarded. With forward_auth, application headers are forwarded, while connection, destination, body, and the selected provider's owned headers are removed or rebuilt.
  • HttpBackendConfig::forward_auth uses the caller's credential instead of the backend's configured key and forwards other application headers. OpenAI backends normalize authorization, chatgpt-account-id, and x-openai-fedramp. Anthropic backends forward authorization or x-api-key; they also keep oauth-* values from anthropic-beta and remove other caller-supplied beta values. The forwarding backends in one route must use one credential family (OpenAI or Anthropic) unless they all use the same scheme, host, and port, such as one LLM gateway that accepts the caller's gateway key on every endpoint. Such a route serves Chat Completions and Responses callers and forwards their bearer token to every forwarding backend. Backends that send a configured key are not restricted. Headers owned by other providers are preserved as application headers.
  • Per-backend custom headers go in HttpBackendConfig::extra_headers. Set credentials with api_key. OpenAI backends reject Authorization; Anthropic backends reject x-api-key and anthropic-version. Header names are case-insensitive. Configured values replace caller values for the same header name.
  • Per-target top-level request defaults go in HttpBackendConfig::extra_body. The merge is shallow and fields already present in the request take precedence.
  • Anthropic reconstruction preserves the caller's native thinking settings, including disabled thinking and manual budgets. output_config.effort is kept separately. Effort implies adaptive thinking only when native thinking is absent. To replace either top-level field for a target, list it in omit_body_fields and supply its replacement in extra_body. Omission runs before defaults are merged. Target reasoning_effort overrides apply to OpenAI backends.
  • HttpBackendConfig::max_retries controls additional attempts after retryable transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body transport failures are retried; streaming body failures are not replayed after the response has been returned.
  • If an OpenAI Chat backend explicitly rejects chat_template_kwargs = {"enable_thinking": false} with HTTP 400 or 422, the client retries once without that field. This compatibility retry is separate from max_retries and shares the original timeout. omit_body_fields can remove it from the first attempt.
  • Codex endpoints whose URL path ends in /backend-api/codex/responses receive string input as a message-item list, including preserved requests. Other fields, array input, non-Codex endpoints, and the translation codec's exact preservation remain unchanged.
  • ModelConfig::with_responses_reasoning controls Responses reasoning replay. The default, Auto, uses the final outgoing model ID after body overrides, falling back to the configured provider model when the field is omitted. OpenAI targets use PreserveEncrypted: plaintext is removed and encrypted provider state is retained. Stored reasoning IDs are retained unless the request sets store: false. Other targets use PreservePlaintext: summary and content are retained and opaque encrypted_content is removed. Switchyard's translated Anthropic reasoning payload is always removed. Set Drop for a backend that cannot consume provider reasoning. Messages and tool history are retained.
  • HttpBackendConfig::failure_cooldown skips completion calls to a backend briefly after an exhausted transient failure. Zero disables it. State is shared per model within the client. Calls resume when the cooldown expires. Deployment TOML defaults failure_cooldown_ms to 5000 (5 seconds); set it to 0 to disable cooldown. With forward_auth, HTTP 429 keeps its request-local retries and fallback while leaving shared cooldown state unchanged.
  • HttpBackendConfig::timeout bounds one complete response, including retries, retry delays, and every stream read. Expiry returns LlmClientError::Timeout, either from the call or from the returned stream, which then ends. None leaves the wait unbounded.

run and decide collect streams used during routing, including answers that an algorithm must inspect, before returning them to the algorithm. They retain the provider events for replay. After the configured retries, a client failure stops run or decide by default. Calls with CallModel::recover_errors enabled return the error to the algorithm so it can apply its fallback policy. Capability classifiers and advisor consults default to fail_open = true. Set fail_open = false to stop the request on their client failures. Recovery includes client errors and deadlines during stream collection. Custom hosts can honor the same flag when serving calls.

Retries replay the same upstream request to the same model. After routing completes, non-timeout failures may try another completion candidate. A timeout stops the call; a connection that times out before it is established is a Transport failure. A transport failure can duplicate a request that the provider processed but did not finish returning. Each attempt is counted once; expiry during a retry delay or after a stream has started does not count another attempt.

Errors

call_rewrite_model returns LlmClientError:

Variant When
InvalidRequest { message } the request does not identify a model
Configuration { message } the model or requested wire format has no configured backend
RequestTranslation(msg) decoding the inbound request failed in the translation engine
RequestEncoding(msg) re-encoding an already-decoded request to the wire format failed (internal fault)
ResponseTranslation(msg) response decoding or encoding failed in the translation engine
Timeout { source } request or response body read exceeded its timeout
Transport { source } connection failure, including a connect timeout, or another transport failure
ContextWindowExceeded { model, message } upstream 400 detected as a context overflow (checked before UpstreamHttp, so callers can evict-and-retry)
UpstreamHttp { status, body } any other non-2xx upstream response
InvalidResponse { source } the upstream response could not be decoded
Other(source) a client-specific failure outside the shared categories