Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(llm-client): fall back when an upstream connect times out (#854)
Elias Shieh4430a512779d
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.mdindex c2c948150..dc8269ee5 100644--- a/crates/libsy-llm-client/README.md+++ b/crates/libsy-llm-client/README.md@@ -290,7 +290,8 @@ to stop the request on their client failures. Recovery includes client errors andeadlines during stream collection. Custom hosts can honor the same flag when serving calls.Retries replay the same upstream request to the same model. After routing completes,-non-timeout failures may try another completion candidate. A timeout stops the call.+non-timeout failures may try another completion candidate. A timeout stops the call;+a connection that times out before it is established is a `Transport` failure.A transport failure can duplicate a request that the provider processed but did notfinish returning. Each attempt is counted once; expiry during a retry delay orafter a stream has started does not count another attempt.@@ -307,7 +308,7 @@ after a stream has started does not count another attempt.| `RequestEncoding(msg)` | re-encoding an already-decoded request to the wire format failed (internal fault) || `ResponseTranslation(msg)` | response decoding or encoding failed in the translation engine || `Timeout { source }` | request or response body read exceeded its timeout |-| `Transport { source }` | non-timeout connection or transport failure |+| `Transport { source }` | connection failure, including a connect timeout, or another transport failure || `ContextWindowExceeded { model, message }` | upstream 400 detected as a context overflow (checked before `UpstreamHttp`, so callers can evict-and-retry) || `UpstreamHttp { status, body }` | any other non-2xx upstream response || `InvalidResponse { source }` | the upstream response could not be decoded |diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rsindex 0265faf99..85062b06f 100644--- a/crates/libsy-llm-client/src/client.rs+++ b/crates/libsy-llm-client/src/client.rs@@ -970,7 +970,14 @@ pub(crate) fn convert_reqwest_error(error: reqwest::Error) -> LlmClientError {// Reqwest labels truncated or otherwise unreadable response bodies as decode// errors, so distinguish them from serde JSON failures at the call site.let error = error.without_url();- if error.is_timeout() {+ // reqwest reports a connect-phase timeout as both `is_connect()` and+ // `is_timeout()`. Check connect first: no request was sent on the failed+ // connection, so it is a transport failure like a refused connection.+ if error.is_connect() {+ LlmClientError::Transport {+ source: Box::new(error),+ }+ } else if error.is_timeout() {LlmClientError::Timeout {source: Box::new(error),}@@ -2943,6 +2950,44 @@ mod tests {Ok(())}+ // A connect that never completes sends nothing, like a refused connection.+ #[cfg(target_os = "linux")]+ #[tokio::test]+ async fn connect_timeout_is_a_transport_error()+ -> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {+ // Linux drops SYNs once a listener's accept queue is full, so later+ // connects hang like an upstream behind a dropping firewall.+ let socket = tokio::net::TcpSocket::new_v4()?;+ socket.bind("127.0.0.1:0".parse()?)?;+ let listener = socket.listen(0)?;+ let addr = listener.local_addr()?;+ let _queued: Vec<_> = (0..2)+ .map(|_| {+ std::net::TcpStream::connect_timeout(&addr, std::time::Duration::from_millis(100))+ })+ .collect();++ let mut client = TranslatingLlmClient::new(&chat_map(&format!("http://{addr}/v1")))?;+ // Stands in for the OS connect timeout (30 s on Linux via reqwest's default+ // TCP_USER_TIMEOUT); both surface as `is_connect() && is_timeout()`.+ client.client = reqwest::Client::builder()+ .connect_timeout(std::time::Duration::from_millis(100))+ .build()?;++ let Err(error) = client.call(request_for(Some("gpt"), false)).await else {+ panic!("expected the connect to fail");+ };+ let LlmClientError::Transport { source } = error else {+ panic!("expected Transport, got {error:?}");+ };+ let Some(source) = source.downcast_ref::<reqwest::Error>() else {+ panic!("expected the reqwest connect-timeout source");+ };+ assert!(source.is_connect());+ assert!(source.is_timeout());+ Ok(())+ }+#[tokio::test]async fn llama_cpp_context_overflow_400_is_mapped()-> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rsindex fd9c6a2fd..16b38d312 100644--- a/crates/libsy-llm-client/src/run.rs+++ b/crates/libsy-llm-client/src/run.rs@@ -2570,6 +2570,58 @@ mod tests {Ok(())}+ #[tokio::test]+ async fn refused_connection_falls_back_to_the_next_candidate() -> Result<()> {+ let server = MockServer::start().await;+ Mock::given(method("POST"))+ .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({+ "id": "answer",+ "model": "strong",+ "choices": [{+ "index": 0,+ "message": {"role": "assistant", "content": "ok"},+ "finish_reason": "stop"+ }],+ "usage": {}+ })))+ .mount(&server)+ .await;++ let backend = |base_url: String| {+ Backend::OpenAiChat(HttpBackendConfig {+ base_url,+ api_key: None,+ forward_auth: false,+ extra_headers: BTreeMap::new(),+ extra_body: BTreeMap::new(),+ omit_body_fields: BTreeSet::new(),+ reasoning_effort: None,+ max_retries: 0,+ failure_cooldown: std::time::Duration::ZERO,+ timeout: None,+ })+ };+ let client = Arc::new(+ TranslatingLlmClient::new(&[+ // Nothing listens on port 1, so the connection is refused.+ ModelConfig::new("weak", backend("http://127.0.0.1:1/v1".to_string()), None),+ ModelConfig::new("strong", backend(format!("{}/v1", server.uri())), None),+ ])+ .map_err(|error| LibsyError::external("building test client", error))?,+ );+ let (_, response) = run(+ Arc::new(CandidateAlgorithm {}),+ ClientRouter::single(client),+ request(),+ to_category_map(&["weak", "strong"]),+ None,+ )+ .await?;++ assert_eq!(response.served_model().map(ModelId::as_str), Some("strong"));+ Ok(())+ }+#[tokio::test]async fn streams_are_outside_the_candidate_fallback_boundary() -> Result<()> {// Receiving a stream handle is a successful call and ends candidate selection.diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex eeeb68214..10718cd73 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -369,7 +369,9 @@ Custom mode validates the judge's JSON against `response_schema`, resolves thepolicy selector, and routes to a runtime model group. A verdict names a group andthe first model in it serves the turn. An eligible non-timeout failure tries therest of that group, then any remaining models in `models.any`. A timeout stops-the request without trying another model.+the request without trying another model. A connection that times out before it+is established counts as a connection failure, not a timeout, so it tries the+next model.The `[routes.<name>.models]` table takes any group name you choose. `any` and`judge` are reserved and required; `capable` and `efficient` are reserved for the