Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(llm-client): distinguish targets from provider models (#1275)
ryan-lempkac23e085f0ba3
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.mdindex 23f43b98b..dee3a8081 100644--- a/crates/libsy-llm-client/README.md+++ b/crates/libsy-llm-client/README.md@@ -21,10 +21,10 @@ It depends on `switchyard-libsy`, `switchyard-protocol`, and## Concepts-- **Model configs.** A client is built from [`ModelConfig`] values, each keyed by a- [`ModelId`]. Each model has a default [`Backend`] and can have additional backends- for other wire formats. [`TranslatingLlmClient::call_rewrite_model`] uses the- request's metadata wire format when set, otherwise the model's default backend.+- **Target configs.** Each [`ModelConfig`] has a target name, a default [`Backend`],+ and optional backends for other wire formats. The target name selects settings.+ `with_provider_model` supplies the model ID sent upstream. For example, targets+ `efficient` and `capable` can use the same model with different reasoning efforts.- **Model ids.** A [`ModelId`] is a newtype over `String` that behaves like one — itderefs to `str`, compares against string literals, and converts from `&str` or`String` with `.into()`. Anywhere below that takes `impl Into<ModelId>` accepts a@@ -34,8 +34,8 @@ It depends on `switchyard-libsy`, `switchyard-protocol`, andstatic `extra_headers`, default `extra_body` fields, and `max_retries`). Thevariant fixes the URL path and auth scheme (Bearer vs `x-api-key` +`anthropic-version`).-- **Model rewrite.** The resolved [`ModelId`] is both the map key and the model id- sent upstream — it overwrites whatever `model` the request arrived with.+- **Model rewrite.** The selected target's provider model ID replaces the request's+ `model` field. If no provider model is configured, the target name is used.- **Streaming is chosen by the encoded request body.** If the body has`stream: true` after `extra_body` is applied, you get `LlmResponse::Stream`; otherwise`LlmResponse::Agg`. OpenAI Chat streaming requests default@@ -101,7 +101,7 @@ async fn ask(client: &TranslatingLlmClient) -> switchyard_llm_client::Result<Strmetadata: None,};- // model_name wins over request.llm_request.model; it is also sent upstream.+ // The explicit target takes precedence over the request model field.let model = ModelId::from("gpt-4o-mini");let response = client.call_rewrite_model(request, Some(&model)).await?;@@ -279,9 +279,10 @@ fn build_multi_format_client(fields, array input, non-Codex endpoints, and the translation codec's exactpreservation remain unchanged.- `ModelConfig::with_responses_reasoning` controls Responses reasoning replay.- The default, `Auto`, chooses per request from the target. OpenAI targets use- `PreserveEncrypted`: plaintext is removed and encrypted provider state is- retained. Stored reasoning IDs are retained unless the request sets+ The default, `Auto`, uses the final outgoing model ID after body overrides,+ falling back to the configured provider model when the field is omitted.+ OpenAI targets use `PreserveEncrypted`: plaintext is removed and encrypted+ provider state is retained. Stored reasoning IDs are retained unless the request sets`store: false`. Other targets use `PreservePlaintext`: `summary` and `content`are retained and opaque `encrypted_content` is removed. Switchyard's translatedAnthropic reasoning payload is always removed. Set `Drop` for a backend thatdiff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rsindex 4fa4db687..7962f1ec5 100644--- a/crates/libsy-llm-client/src/client.rs+++ b/crates/libsy-llm-client/src/client.rs@@ -59,11 +59,10 @@ const INITIAL_RETRY_DELAY: Duration = Duration::from_millis(250);const MAX_RETRY_BACKOFF: Duration = Duration::from_secs(2);const MAX_RETRY_AFTER: Duration = Duration::from_secs(60);-/// How one model is served: the `default_backend` used when the request does not-/// pin a wire format, plus any `other_backends` reachable over additional formats.+/// A named target's provider model and backends.#[derive(Clone, Debug)]pub struct ModelConfig {- model_name: ModelId,+ target_name: ModelId,provider_model: Option<ModelId>,default_backend: Backend,other_backends: Option<Vec<Backend>>,@@ -71,15 +70,14 @@ pub struct ModelConfig {}impl ModelConfig {- /// A model named `model_name` served by `default_backend`, optionally reachable- /// over additional wire formats via `other_backends`.+ /// Configures a target. Its name is also the provider model ID unless overridden.pub fn new(- model_name: impl Into<ModelId>,+ target_name: impl Into<ModelId>,default_backend: Backend,other_backends: Option<Vec<Backend>>,) -> Self {Self {- model_name: model_name.into(),+ target_name: target_name.into(),provider_model: None,default_backend,other_backends,@@ -87,15 +85,15 @@ impl ModelConfig {}}- /// Sets how Responses reasoning items are replayed to this model.+ /// Sets how Responses reasoning items are replayed to this target.#[must_use]pub fn with_responses_reasoning(mut self, policy: crate::ResponsesReasoningPolicy) -> Self {self.responses_reasoning = policy;self}- /// Sends `provider_model` upstream in place of `model_name`. The client is still- /// called with `model_name`, so several entries can serve one provider model with+ /// Sends `provider_model` upstream in place of `target_name`. The client is still+ /// called with `target_name`, so several entries can serve one provider model with/// different backend settings.pub fn with_provider_model(mut self, provider_model: impl Into<ModelId>) -> Self {self.provider_model = Some(provider_model.into());@@ -136,16 +134,15 @@ impl AuxiliaryOperation {}}-/// A client that dispatches neutral-IR requests to per-model HTTP backends.+/// A client that dispatches neutral-IR requests to named targets.///-/// Construct it with a list of [`ModelConfig`]s — one per model, each naming a-/// default [`Backend`] and any additional per-format backends. Each call resolves-/// the model and wire format, encodes the request to that backend's wire format,+/// Each [`ModelConfig`] defines a target's provider model and backends.+/// Each call resolves the target and wire format, encodes the request,/// applies auth and forwarded headers, sends the HTTP request with a shared/// [`reqwest::Client`], and decodes the response back to the neutral IR (buffered/// or streamed).pub struct TranslatingLlmClient {- model_to_config: HashMap<ModelId, ModelConfig>,+ configs_by_target: HashMap<ModelId, ModelConfig>,client: reqwest::Client,forward_auth_client: reqwest::Client,is_rate_limit_cooldown_enabled: bool,@@ -164,9 +161,9 @@ impl TranslatingLlmClient {for backend in std::iter::once(&config.default_backend).chain(config.other_backends.iter().flatten()){- backend.validate_configured_headers(&config.model_name)?;+ backend.validate_configured_headers(&config.target_name)?;}- unavailable_until.insert(config.model_name.clone(), AtomicU64::new(0));+ unavailable_until.insert(config.target_name.clone(), AtomicU64::new(0));}let build_client = |builder: reqwest::ClientBuilder| {builder.build().map_err(|error| LlmClientError::Transport {@@ -178,13 +175,13 @@ impl TranslatingLlmClient {// Forwarded credentials are sent only to the configured URL.let forward_auth_client =build_client(reqwest::Client::builder().redirect(reqwest::redirect::Policy::none()))?;- let model_to_config = model_configs+ let configs_by_target = model_configs.iter()- .map(|config| (config.model_name.clone(), config.clone()))+ .map(|config| (config.target_name.clone(), config.clone())).collect();Ok(Self {- model_to_config,+ configs_by_target,client,forward_auth_client,is_rate_limit_cooldown_enabled: true,@@ -199,11 +196,9 @@ impl TranslatingLlmClient {self}- /// The backend serving `model` over `format` — the default backend when its- /// format matches, otherwise a matching entry in `other_backends`; `None` when- /// the model is unknown or has no backend for `format`.- pub fn backend_for(&self, model: &ModelId, format: WireFormat) -> Option<&Backend> {- self.model_to_config.get(model).and_then(|config| {+ /// The target's backend for `format`, or `None` if no matching backend exists.+ pub fn backend_for(&self, target_name: &ModelId, format: WireFormat) -> Option<&Backend> {+ self.configs_by_target.get(target_name).and_then(|config| {if config.default_backend.wire_format() == format {Some(&config.default_backend)} else {@@ -215,18 +210,18 @@ impl TranslatingLlmClient {})}- /// The model id sent upstream for `model`: its provider model when one is set,- /// otherwise `model` itself.- pub fn provider_model<'a>(&'a self, model: &'a ModelId) -> &'a ModelId {- self.model_to_config- .get(model)+ /// The target's provider model ID, falling back to its name when none is configured.+ pub fn provider_model<'a>(&'a self, target_name: &'a ModelId) -> &'a ModelId {+ self.configs_by_target+ .get(target_name).and_then(|config| config.provider_model.as_ref())- .unwrap_or(model)+ .unwrap_or(target_name)}- /// Whether `model` has a backend for `operation`.- pub fn supports_auxiliary(&self, model: &ModelId, operation: AuxiliaryOperation) -> bool {- self.backend_for(model, operation.wire_format()).is_some()+ /// Whether the target has a backend for `operation`.+ pub fn supports_auxiliary(&self, target_name: &ModelId, operation: AuxiliaryOperation) -> bool {+ self.backend_for(target_name, operation.wire_format())+ .is_some()}/// Reads llama.cpp model properties through the backend's configured auth and headers.@@ -234,18 +229,18 @@ impl TranslatingLlmClient {/// Redirects are disabled so credentials cannot move to another origin.pub async fn get_model_properties(&self,- model: &ModelId,+ target_name: &ModelId,format: WireFormat,timeout: Duration,) -> Result<Value> {let backend =- self.backend_for(model, format)+ self.backend_for(target_name, format).ok_or_else(|| LlmClientError::Configuration {- message: format!("model {model} has no backend for {format:?}"),+ message: format!("model {target_name} has no backend for {format:?}"),})?;let mut url = reqwest::Url::parse(backend.base_url()).map_err(|error| {LlmClientError::Configuration {- message: format!("model {model} has an invalid backend URL: {error}"),+ message: format!("model {target_name} has an invalid backend URL: {error}"),}})?;let root = url.path().trim_end_matches('/');@@ -285,29 +280,29 @@ impl TranslatingLlmClient {/// request fails or returns invalid JSON.pub async fn call_auxiliary(&self,- model: &ModelId,+ target_name: &ModelId,request: Request,operation: AuxiliaryOperation,) -> Result<Value> {let wire_format = operation.wire_format();- let backend =- self.backend_for(model, wire_format)- .ok_or_else(|| LlmClientError::Configuration {- message: format!("model {model} has no backend for {operation:?}"),- })?;+ let backend = self.backend_for(target_name, wire_format).ok_or_else(|| {+ LlmClientError::Configuration {+ message: format!("model {target_name} has no backend for {operation:?}"),+ }+ })?;let Request {mut llm_request,metadata,..} = request;- llm_request.model = Some(self.provider_model(model).to_string());+ llm_request.model = Some(self.provider_model(target_name).to_string());let http_response = self.send_encoded(backend,wire_format,llm_request,metadata.as_ref(),- model,+ target_name,UpstreamEndpoint::Auxiliary(operation),).await?;@@ -338,30 +333,29 @@ impl TranslatingLlmClient {wire_format: WireFormat,llm_request: LlmRequest,metadata: Option<&Metadata>,- model: &ModelId,+ target_name: &ModelId,endpoint: UpstreamEndpoint,) -> Result<EncodedResponse> {let cooldown = backend.failure_cooldown();let unavailable_until =if matches!(endpoint, UpstreamEndpoint::Completion) && !cooldown.is_zero() {- self.unavailable_until.get(model)+ self.unavailable_until.get(target_name)} else {None};if let Some(until) = unavailable_until&& until.load(Ordering::Relaxed) > duration_millis(self.cooldown_epoch.elapsed()){- tracing::debug!(model = %model, wire_format = %wire_format, "skipping backend during failure cooldown");+ tracing::debug!(model = %target_name, wire_format = %wire_format, "skipping backend during failure cooldown");return Err(LlmClientError::TemporarilyUnavailable);}let mut body = observed_translation::encode_request(&llm_request, wire_format).map_err(|error| LlmClientError::RequestEncoding(error.to_string()))?;// `encode_request` round-trips a preserved same-format body verbatim,- // which keeps the caller's original `model`; force the resolved model so- // the upstream always sees the provider model.- let upstream_model = self.provider_model(model);- set_json_model(&mut body, upstream_model);+ // which keeps the caller's original `model`. Set the provider model explicitly.+ let provider_model_id = self.provider_model(target_name);+ set_json_model(&mut body, provider_model_id);if matches!(backend, Backend::OpenAiResponses(_)) {sanitize_openai_responses_provider_body(&mut body);}@@ -374,11 +368,16 @@ impl TranslatingLlmClient {omit_configured_body_fields(&mut body, backend.omit_body_fields());merge_extra_body(&mut body, backend.extra_body());if matches!(backend, Backend::OpenAiResponses(_)) {- self.model_to_config- .get(model)+ let outbound_model_id = body+ .get("model")+ .and_then(Value::as_str)+ .unwrap_or(provider_model_id)+ .to_owned();+ self.configs_by_target+ .get(target_name).map(|config| config.responses_reasoning).unwrap_or_default()- .normalize(&mut body, backend.base_url(), model);+ .normalize(&mut body, backend.base_url(), &outbound_model_id);}// After the merge on purpose: the effort override must win over both the caller's// value and any `reasoning` default a target set through `extra_body`. Token counting@@ -402,7 +401,7 @@ impl TranslatingLlmClient {enable_openai_guidance_caching(&mut body,wire_format,- upstream_model,+ provider_model_id,llm_request.temporary_note_count,);}@@ -412,10 +411,10 @@ impl TranslatingLlmClient {let streaming = endpoint.allows_streaming()&& body.get("stream").and_then(Value::as_bool).unwrap_or(false);let url = endpoint.url(backend);- record_gen_ai_request(&url, upstream_model, streaming);+ record_gen_ai_request(&url, provider_model_id, streaming);let result = self- .send_with_retries(&url, backend, &body, metadata, model, streaming)+ .send_with_retries(&url, backend, &body, metadata, target_name, streaming).await;// Keep quota failures request-local when forwarding auth or returning 429 to callers.if let Some(until) = unavailable_until@@ -438,7 +437,7 @@ impl TranslatingLlmClient {backend: &Backend,body: &Value,metadata: Option<&Metadata>,- model: &ModelId,+ target_name: &ModelId,streaming: bool,) -> Result<EncodedResponse> {let max_retries = u64::from(backend.max_retries());@@ -473,7 +472,7 @@ impl TranslatingLlmClient {let span = tracing::debug_span!(target: "libsy","libsy.upstream_attempt",- model = %model,+ model = %target_name,wire_format = %backend.wire_format(),attempt = attempt + 1 + u64::from(template_retry),max_attempts,@@ -497,7 +496,7 @@ impl TranslatingLlmClient {}result = async {attempt_started = true;- self.send_once(url, backend, body.as_ref(), metadata, model, streaming).await+ self.send_once(url, backend, body.as_ref(), metadata, target_name, streaming).await}.instrument(span.clone()) => result,};// The retained handle updates this same attempt span with its outcome.@@ -568,7 +567,7 @@ impl TranslatingLlmClient {backend: &Backend,body: &Value,metadata: Option<&Metadata>,- model: &ModelId,+ target_name: &ModelId,streaming: bool,) -> std::result::Result<EncodedResponse, AttemptFailure> {let client = if backend.is_forwarding_auth() {@@ -605,7 +604,7 @@ impl TranslatingLlmClient {if status.is_success() {if streaming {let upstream_headers = response.headers().clone();- let chunks = match prepare_response_stream(response, backend, model).await {+ let chunks = match prepare_response_stream(response, backend, target_name).await {Ok(chunks) => chunks,Err(error) => {metrics::record_upstream_attempt(None);@@ -661,7 +660,7 @@ impl TranslatingLlmClient {let error =if status == reqwest::StatusCode::BAD_REQUEST && backend.is_context_overflow(&body) {LlmClientError::ContextWindowExceeded {- model: model.clone(),+ model: target_name.clone(),message: body,}} else {@@ -674,18 +673,13 @@ impl TranslatingLlmClient {})}- /// Calls the backend for `model_name` (or the request's own model), over the- /// wire format the request pins in its metadata (else the model's default- /// backend), and returns the neutral response.- ///- /// Resolution: `model_name` wins over `request.llm_request.model`; the- /// resolved name is the outer map key, and its provider model is the model id- /// written into the request before translation. Missing models are invalid requests; unknown- /// models or wire formats are configuration errors.+ /// Calls the selected target using its configured provider model and backend.+ /// `target_name` takes precedence over the request's model field.+ /// Request metadata selects the wire format, falling back to the target's default.pub async fn call_rewrite_model(&self,request: Request,- model_name: Option<&ModelId>,+ target_name: Option<&ModelId>,) -> Result<Response> {let Request {mut llm_request,@@ -693,28 +687,30 @@ impl TranslatingLlmClient {..} = request;- let model_id = model_name+ let selected_target = target_name.cloned().or_else(|| llm_request.model.map(ModelId::from)).ok_or_else(|| LlmClientError::InvalidRequest {message: "no model given".to_string(),})?;- llm_request.model = Some(self.provider_model(&model_id).to_string());+ llm_request.model = Some(self.provider_model(&selected_target).to_string());let orig_format = metadata.as_ref().and_then(|m| m.wire_format);let wire_format = orig_format.unwrap_or(- self.model_to_config- .get(&model_id)+ self.configs_by_target+ .get(&selected_target).map(|config| config.default_backend.wire_format()).ok_or_else(|| LlmClientError::Configuration {- message: format!("no backend configured for model {model_id:?}"),+ message: format!("no backend configured for model {selected_target:?}"),})?,);- let backend = self.backend_for(&model_id, wire_format).ok_or_else(|| {- LlmClientError::Configuration {- message: format!("model {model_id:?} has no backend for format {wire_format}"),- }- })?;+ let backend = self+ .backend_for(&selected_target, wire_format)+ .ok_or_else(|| LlmClientError::Configuration {+ message: format!(+ "model {selected_target:?} has no backend for format {wire_format}"+ ),+ })?;let http_response = self.send_encoded(@@ -722,7 +718,7 @@ impl TranslatingLlmClient {wire_format,llm_request,metadata.as_ref(),- &model_id,+ &selected_target,UpstreamEndpoint::Completion,).await?;@@ -799,19 +795,17 @@ impl TranslatingLlmClient {&self,raw_http_request: Value,http_headers: Option<http::HeaderMap>,- model: Option<&ModelId>,+ target_name: Option<&ModelId>,wire_format: WireFormat,) -> Result<RawResponse> {let llm_request = observed_translation::decode_request(wire_format, &raw_http_request).map_err(|error| LlmClientError::RequestTranslation(error.to_string()))?;let request_extensions = llm_request.extensions.clone();- // The model that serves the call — the rewrite target when the caller pinned- // one, else the request's own model. Mirrors `call_rewrite_model`'s own- // resolution so the response names whoever answered.- let served_model = model+ // Report the provider model for the same target selected by `call_rewrite_model`.+ let served_model = target_name.cloned().or_else(|| llm_request.model.clone().map(ModelId::from))- .map(|model| self.provider_model(&model).to_string());+ .map(|target_name| self.provider_model(&target_name).to_string());let request = Request {llm_request,@@ -827,7 +821,7 @@ impl TranslatingLlmClient {..Default::default()}),};- let response = self.call_rewrite_model(request, model).await?;+ let response = self.call_rewrite_model(request, target_name).await?;match response.llm_response {LlmResponse::Agg(agg) => {@@ -964,7 +958,7 @@ fn stream_with_deadline(async fn prepare_response_stream(response: reqwest::Response,backend: &Backend,- model: &ModelId,+ target_name: &ModelId,) -> Result<LlmResponseStream> {let bytes = response.bytes_stream().map(|chunk| {chunk@@ -982,7 +976,7 @@ async fn prepare_response_stream(// An in-band context overflow skips retries and advances to another candidate.if let Some(message) = first_event_overflow(&first, backend) {return Err(LlmClientError::ContextWindowExceeded {- model: model.clone(),+ model: target_name.clone(),message,});}@@ -2167,7 +2161,7 @@ mod tests {}#[tokio::test]- async fn model_name_arg_wins_over_request_model()+ async fn target_name_arg_wins_over_request_model()-> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {let client = TranslatingLlmClient::new(&[])?;// Arg "b" is looked up (and reported), not the request's "a".diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rsindex 033f40482..e6d781325 100644--- a/crates/switchyard-server/tests/server.rs+++ b/crates/switchyard-server/tests/server.rs@@ -611,7 +611,7 @@ async fn upstream_buffered_responses("model/invalid-error" => response["error"] = json!("invalid error details"),"model/invalid-code" => response["error"]["code"] = json!(42),"model/empty-code" => response["error"]["code"] = json!(""),- "model/fallback" | "model/efficient" => {+ "model/fallback" | "model/efficient" | "openai/gpt-5" => {response["status"] = json!("completed");response["error"] = Value::Null;response["output"] = json!([{@@ -6296,8 +6296,6 @@ async fn upstream_headers_forward_on_streaming_responses() -> TestResult {Ok(())}-// Mirrors the GLM baseline: passthrough to a Responses target with a reasoning effort-// default, called by Codex with `store: false`.#[tokio::test]async fn configured_responses_reasoning_policy_reaches_upstream() -> TestResult {let upstream = MockUpstream::start().await?;@@ -6307,13 +6305,15 @@ async fn configured_responses_reasoning_policy_reaches_upstream() -> TestResultjson!({"type": "reasoning", "id": "rs_2", "summary": [], "encrypted_content": "opaque"});let mut kept_foreign = foreign.clone();kept_foreign["content"] = json!([]);- for (setting, expected_reasoning) in [- ("", vec![plaintext.clone()]),+ for (provider_model_id, setting, expected_reasoning) in [+ ("openai/gpt-5", "", vec![kept_foreign.clone()]),+ ("model/efficient", "", vec![plaintext.clone()]),(+ "model/efficient","responses_reasoning = \"preserve_encrypted\"",vec![kept_foreign],),- ("responses_reasoning = \"drop\"", vec![]),+ ("model/efficient", "responses_reasoning = \"drop\"", vec![]),] {let config = format!(r#"@@ -6322,14 +6322,14 @@ schema_version = 1format = "openai_responses"base_url = "{base_url}"{setting}-[targets.glm]-id = "model/efficient"+[targets.primary]+id = "{provider_model_id}"llm_client = "upstream"extra_body = {{ reasoning = {{ effort = "high" }} }}[routes.switchyard]id = "switchyard"type = "passthrough"-target = "glm"+target = "primary""#);let app = build_switchyard_router(load_test_config(&config)?);@@ -6356,7 +6356,11 @@ target = "glm".filter(|item| item["type"] == "reasoning").cloned().collect();- assert_eq!(reasoning, expected_reasoning, "{setting}");+ assert_eq!(body["model"], json!(provider_model_id));+ assert_eq!(+ reasoning, expected_reasoning,+ "{provider_model_id}, {setting}"+ );assert_eq!(body["reasoning"], json!({"effort": "high"}));}Ok(())