Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(client): enforce response deadlines and stop routing on errors (#702)
Elyas Mehtabuddin1759dfdff95e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0e987ef86..54a2349b7 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).### Added+- **`timeout_ms` on `[llm_clients.<name>]`** — one deadline covers all attempts,+ retry delays, and the complete response, including stream reads. Unset leaves+ the wait unbounded; `0` is rejected. A timeout returns `504` without trying another+ model, or a framed error if the final answer has already started streaming.+ Timed-out attempts are counted in metrics.- **Per-target `reasoning_effort`** — a target can force the reasoning effortof every request it serves, replacing the caller's value (`reasoning.effort`on the Responses wire, `reasoning_effort` on Chat Completions), so a strong@@ -89,6 +94,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).### Changed+- **HTTP client errors stop routing** — after the configured retries, the Rust+ runner stops the request instead of letting the routing algorithm choose a+ fallback. This also applies when `timeout_ms` is unset or an advisor has+ `fail_open = true`. The runner collects streams used during routing and preserves+ provider events for replay. Invalid judge verdicts keep their existing fallback.- **`Algorithm::route` returns `Result<RoutingOutcome>`** — instead of thebare final `Result`, so callers observe the full routing outcome (see #458for the design). (#459)diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.mdindex bad3f86d0..bb3b38caf 100644--- a/crates/libsy-llm-client/README.md+++ b/crates/libsy-llm-client/README.md@@ -36,8 +36,8 @@ It depends on `switchyard-libsy`, `switchyard-protocol`, and`anthropic-version`).- **Model rewrite.** The resolved [`ModelId`] is both the map key and the model idsent upstream — it overwrites whatever `model` the request arrived with.-- **Streaming is chosen by the request.** If the encoded body has `stream: true`- (i.e. `request.llm_request.stream`), you get `LlmResponse::Stream`; otherwise+- **Streaming is chosen by the encoded request body.** If the body has+ `stream: true` after `extra_body` is applied, you get `LlmResponse::Stream`; otherwise`LlmResponse::Agg`. OpenAI Chat streaming requests default`stream_options.include_usage` to `true`; an explicit caller value is preserved.@@ -69,7 +69,9 @@ fn build_client() -> switchyard_llm_client::Result<TranslatingLlmClient> {forward_auth: false,extra_headers: BTreeMap::new(),extra_body: BTreeMap::new(),+ reasoning_effort: None,max_retries: 2,+ timeout: None,};let models = [ModelConfig::new(@@ -247,13 +249,23 @@ fn build_multi_format_client(transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered bodytransport failures are retried; streaming body failures are not replayed afterthe response has been returned.--Retries replay the same upstream request to the same model. Each candidate's-`max_retries` budget is exhausted before candidate fallback advances to the next-model. The worst case is `candidates × (max_retries + 1)` upstream requests, and-total latency includes every candidate's capped `Retry-After` backoff. A transport-failure can duplicate a request that the provider processed but did not finish-returning.+- `HttpBackendConfig::timeout` bounds one complete response, including retries,+ retry delays, and every stream read. Expiry returns `LlmClientError::Timeout`,+ either from the call or from the returned stream, which then ends. `None` leaves+ the wait unbounded.++`run` and `decide` collect streams used during routing, including answers that an+algorithm must inspect, before returning them to the algorithm. They retain the+provider events for replay. After the configured retries, a client failure stops+`run` or `decide` before the algorithm can choose another routing candidate. This+also applies when `timeout` is `None` or an advisor has `fail_open = true`.+`libsy` and custom hosts that drive it directly are unchanged.++Retries replay the same upstream request to the same model. After routing completes,+non-timeout failures may try another completion candidate. A timeout stops the call.+A transport failure can duplicate a request that the provider processed but did not+finish returning. Each attempt is counted once; expiry during a retry delay or+after a stream has started does not count another attempt.## Errorsdiff --git a/crates/libsy-llm-client/src/backend.rs b/crates/libsy-llm-client/src/backend.rsindex da43e2c63..a96a3e1fa 100644--- a/crates/libsy-llm-client/src/backend.rs+++ b/crates/libsy-llm-client/src/backend.rs@@ -3,6 +3,7 @@//! Per-provider backend configuration: wire format, upstream URL, and auth.+use std::time::Duration;use std::{collections::BTreeMap, fmt};use reqwest::RequestBuilder;@@ -62,6 +63,9 @@ pub struct HttpBackendConfig {pub reasoning_effort: Option<String>,/// Additional attempts after the initial upstream request.pub max_retries: u32,+ /// Deadline for one complete response, including retries, retry delays, and stream reads.+ /// `None` leaves the wait unbounded.+ pub timeout: Option<Duration>,}impl fmt::Debug for HttpBackendConfig {@@ -74,6 +78,7 @@ impl fmt::Debug for HttpBackendConfig {.field("extra_body_keys", &self.extra_body.keys()).field("reasoning_effort", &self.reasoning_effort).field("max_retries", &self.max_retries)+ .field("timeout", &self.timeout).finish()}}@@ -256,6 +261,11 @@ impl Backend {self.config().max_retries}+ /// Deadline for all attempts and the complete response; `None` leaves the wait unbounded.+ pub fn timeout(&self) -> Option<Duration> {+ self.config().timeout+ }+/// Whether this backend speaks the Anthropic Messages wire format — the only/// one with a `count_tokens` endpoint.pub fn is_anthropic(&self) -> bool {@@ -348,6 +358,7 @@ mod tests {extra_body: BTreeMap::new(),reasoning_effort: None,max_retries: 0,+ timeout: None,}}diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rsindex f9580a537..dbb69b2ca 100644--- a/crates/libsy-llm-client/src/client.rs+++ b/crates/libsy-llm-client/src/client.rs@@ -224,8 +224,9 @@ impl TranslatingLlmClient {/// forwarded headers plus the backend's static headers and auth, and return the/// successful upstream response. A/// buffered response is fully collected within the retry boundary; a streamed- /// response has its first decoded event checked within that boundary. A non-success- /// status maps to a typed error — a 400 is classified as a context-window+ /// response has its first decoded event checked within that boundary. The backend's+ /// deadline, when set, spans every attempt, retry delay, and subsequent stream read. A+ /// non-success status maps to a typed error — a 400 is classified as a context-window/// overflow via the backend's provider rules. Shared by/// [`call_rewrite_model`](Self::call_rewrite_model) (which POSTs to the/// backend's completion URL and decodes a response) and@@ -269,15 +270,47 @@ impl TranslatingLlmClient {let url = endpoint.url(backend);record_gen_ai_request(&url, model, streaming);+ self.send_with_retries(&url, backend, &body, metadata, model, streaming)+ .await+ }++ // Sends the encoded body, retrying retryable failures within the backend's retry budget.+ async fn send_with_retries(+ &self,+ url: &str,+ backend: &Backend,+ body: &Value,+ metadata: Option<&Metadata>,+ model: &ModelId,+ streaming: bool,+ ) -> Result<EncodedResponse> {let max_retries = u64::from(backend.max_retries());let max_attempts = max_retries + 1;+ let expires_at = backend+ .timeout()+ .map(|timeout| (tokio::time::sleep(timeout).deadline(), timeout));+ let deadline = async {+ match expires_at {+ Some((at, timeout)) => {+ tokio::time::sleep_until(at).await;+ timeout+ }+ None => std::future::pending().await,+ }+ };+ tokio::pin!(deadline);let mut attempt = 0_u64;loop {+ if let Some((at, timeout)) = expires_at+ && tokio::time::Instant::now() >= at+ {+ return Err(deadline_error(timeout));+ }let span = tracing::debug_span!(target: "libsy","libsy.upstream_attempt",model = %model,- wire_format = %wire_format,+ wire_format = %backend.wire_format(),attempt = attempt + 1,max_attempts,retry = attempt > 0,@@ -287,10 +320,22 @@ impl TranslatingLlmClient {will_retry = tracing::field::Empty,retry_delay_ms = tracing::field::Empty,);- let result = self- .send_once(&url, backend, &body, metadata, model, streaming)- .instrument(span.clone())- .await;+ let mut attempt_started = false;+ let result = tokio::select! {+ biased;+ timeout = &mut deadline => {+ if attempt_started {+ metrics::record_upstream_attempt(None);+ }+ span.record("outcome", "error");+ span.record("will_retry", false);+ return Err(deadline_error(timeout));+ }+ result = async {+ attempt_started = true;+ self.send_once(url, backend, body, metadata, model, streaming).await+ }.instrument(span.clone()) => result,+ };// The retained handle updates this same attempt span with its outcome.match result {Ok(response) => {@@ -300,7 +345,18 @@ impl TranslatingLlmClient {if attempt > 0 {metrics::record_retry_recovered();}- return Ok(response);+ return Ok(match response {+ EncodedResponse::Streaming {+ status,+ chunks,+ upstream_headers,+ } => EncodedResponse::Streaming {+ status,+ chunks: stream_with_deadline(chunks, expires_at),+ upstream_headers,+ },+ buffered => buffered,+ });}Err(failure) => {let will_retry = attempt < max_retries && failure.is_retryable();@@ -317,7 +373,11 @@ impl TranslatingLlmClient {span.record("retry_delay_ms", duration_millis(delay));// Close the attempt span before sleeping so backoff is not attempt latency.drop(span);- tokio::time::sleep(delay).await;+ tokio::select! {+ biased;+ timeout = &mut deadline => return Err(deadline_error(timeout)),+ _ = tokio::time::sleep(delay) => {}+ }attempt += 1;}}@@ -653,6 +713,18 @@ struct AttemptFailure {retry_after: Option<Duration>,}+fn deadline_error(timeout: Duration) -> LlmClientError {+ LlmClientError::Timeout {+ source: Box::new(std::io::Error::new(+ std::io::ErrorKind::TimedOut,+ format!(+ "response did not finish within {} ms, retries included",+ duration_millis(timeout)+ ),+ )),+ }+}+impl AttemptFailure {fn is_retryable(&self) -> bool {match &self.error {@@ -665,6 +737,26 @@ impl AttemptFailure {}}+fn stream_with_deadline(+ chunks: LlmResponseStream,+ deadline: Option<(tokio::time::Instant, Duration)>,+) -> LlmResponseStream {+ let Some((at, timeout)) = deadline else {+ return chunks;+ };+ stream::unfold(Some(chunks), move |chunks| async move {+ let mut chunks = chunks?;+ let chunk = tokio::select! {+ biased;+ _ = tokio::time::sleep_until(at) => return Some((Err(deadline_error(timeout)), None)),+ chunk = chunks.next() => chunk?,+ };+ let next = chunk.is_ok().then_some(chunks);+ Some((chunk, next))+ })+ .boxed()+}+async fn prepare_response_stream(response: reqwest::Response,backend: &Backend,@@ -1199,6 +1291,7 @@ mod tests {extra_body: BTreeMap::new(),reasoning_effort: None,max_retries: 0,+ timeout: None,}}@@ -2212,6 +2305,52 @@ mod tests {Ok(())}+ #[tokio::test]+ async fn deadline_expires_before_send_or_stream_poll()+ -> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {+ let server = MockServer::start().await;+ Mock::given(method("POST"))+ .respond_with(ResponseTemplate::new(200).set_body_raw(+ "data: {\"id\":\"test\",\"model\":\"gpt\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n",+ "text/event-stream",+ ))+ .mount(&server)+ .await;+ for timeout in [Duration::ZERO, Duration::from_millis(200)] {+ let mut backend = config(&server.uri());+ backend.timeout = Some(timeout);+ let client = TranslatingLlmClient::new(&[ModelConfig::new(+ "gpt",+ Backend::OpenAiChat(backend),+ None,+ )])?;+ let result = client+ .call_rewrite_model(request_for(Some("gpt"), true), None)+ .await;+ if timeout.is_zero() {+ assert!(matches!(result, Err(LlmClientError::Timeout { .. })));+ assert!(+ server+ .received_requests()+ .await+ .expect("recorded requests")+ .is_empty()+ );+ } else {+ let LlmResponse::Stream(mut chunks) = result?.llm_response else {+ return Err("expected a streaming response".into());+ };+ tokio::time::sleep(Duration::from_millis(300)).await;+ assert!(matches!(+ chunks.next().await,+ Some(Err(LlmClientError::Timeout { .. }))+ ));+ assert!(chunks.next().await.is_none());+ }+ }+ Ok(())+ }+#[tokio::test]async fn timeout_is_retried_before_a_response_is_returned()-> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rsindex c16922c44..2326c951c 100644--- a/crates/libsy-llm-client/src/run.rs+++ b/crates/libsy-llm-client/src/run.rs@@ -10,8 +10,8 @@//!//! libsy owns the stream mechanics; what this module adds is per-target request preparation,//! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each-//! candidate exhausts its backend retry budget before fallback advances, so the worst case is-//! `candidates × (max_retries + 1)` upstream attempts plus every candidate's backoff.+//! completion candidate exhausts its backend retry budget before fallback advances. Routing+//! calls stop on the first candidate's failure. A timeout stops either kind of call.//!//! A Responses continuation can refer to state held by one provider through//! `previous_response_id` or `conversation`. When completion targets use different clients,@@ -22,7 +22,7 @@ use std::collections::HashMap;use std::sync::Arc;use std::time::Instant;-use futures_util::StreamExt;+use futures::{StreamExt, TryStreamExt, stream};use http::StatusCode;use parking_lot::Mutex;use serde_json::{Value, json};@@ -30,8 +30,8 @@ use switchyard_libsy::{Algorithm, CallModel, LibsyError, OutcomeMetadata, Result, RoutingOutcome, RuntimeModels, drive,};use switchyard_protocol::{- LlmClientError, LlmResponse, ModelId, Request, Response, RoutedLlmClient,- RoutingFallbackReason, WireFormat,+ LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStream, ModelId, Request, Response,+ RoutedLlmClient, RoutingFallbackReason, WireFormat,};use switchyard_translation::prepare_request_for_target;@@ -48,8 +48,8 @@ use crate::{metrics, observability};/// a per-call lookup, not one client for the whole run. Use/// [`ClientRouter::single`](ClientRouter::single) when one client serves every target.///-/// Routing-time model failures are forwarded back into the algorithm. Once routing completes,-/// this client exhausts backend retries and then the outcome's ordered candidate fallbacks.+/// Routing calls are buffered; a client failure stops the request before the algorithm continues.+/// Once routing completes, non-timeout failures may try the outcome's ordered fallback candidates.pub async fn run(algorithm: Arc<dyn Algorithm>,clients: ClientRouter,@@ -98,7 +98,6 @@ pub async fn run(&algorithm_name,&outcome.request,&outcome.selected_model_ids,- CallPhase::Completion,&observe,).await;@@ -171,8 +170,7 @@ fn emit_routing_observations(/// Serve one offloaded call and fulfill its promise.///-/// Errors only when the promise itself could not be fulfilled; a call that failed on every-/// candidate is forwarded to the algorithm as an `Err`.+/// A client failure stops the driver before the algorithm can issue another call.async fn serve(clients: ClientRouter,call: CallModel,@@ -183,21 +181,20 @@ async fn serve(observations.lock().push(observation);}};- let result = call_first_available(+ let target = call.models.first().ok_or(LibsyError::NoTargets)?;+ let request = clients.prepare_routing_request(call.request.clone(), target);+ let response = call_one(&clients,+ target,+ request,&call.algorithm,- &call.request,- &call.models,- CallPhase::Routing,&observe,+ 0,+ call.models.len(),+ true,)- .await;- call.respond(result)-}--enum CallPhase {- Routing,- Completion,+ .await?;+ call.respond(Ok(response))}/// Try candidates in order until one succeeds or a failure stops fallback.@@ -206,14 +203,10 @@ async fn call_first_available(algorithm: &str,request: &Request,models: &[ModelId],- phase: CallPhase,observe: &(dyn Fn(LlmCallObservation) + Send + Sync),) -> Result<Response> {for (index, target) in models.iter().enumerate() {- let request = match phase {- CallPhase::Routing => clients.prepare_routing_request(request.clone(), target),- CallPhase::Completion => clients.prepare_completion_request(request.clone(), target),- };+ let request = clients.prepare_completion_request(request.clone(), target);match call_one(clients,target,@@ -222,6 +215,7 @@ async fn call_first_available(observe,index,models.len(),+ false,).await{@@ -291,6 +285,7 @@ async fn call_one(index: usize,// count is for span logcount: usize,+ buffer: bool,) -> Result<Response> {let span = tracing::Span::current();observability::record_gen_ai_request(&span, &request.llm_request);@@ -305,10 +300,14 @@ async fn call_one(// the provider's, so it belongs in the routing overhead.let client = clients.route(model_id);let started = Instant::now();- let result = match client {- Ok(client) => client.call(request).await,- Err(error) => Err(error),+ let result = async {+ let mut response = client?.call(request).await?;+ if buffer && let LlmResponse::Stream(chunks) = response.llm_response {+ response.llm_response = LlmResponse::Stream(buffer_routing_stream(chunks).await?);+ }+ Ok(response)}+ .await.map_err(|source| LibsyError::client_call(model_id.clone(), source));let duration = started.elapsed();@@ -330,6 +329,31 @@ async fn call_one(result}+// Preserve signed provider events while checking the complete routing response.+async fn buffer_routing_stream(+ mut chunks: LlmResponseStream,+) -> std::result::Result<LlmResponseStream, LlmClientError> {+ let mut events = Vec::new();+ while let Some(event) = chunks.try_next().await? {+ for chunk in event.normalized() {+ match chunk {+ LlmResponseChunk::DecodeError { message } => {+ return Err(LlmClientError::ResponseTranslation(message.clone()));+ }+ LlmResponseChunk::StreamError { message } => {+ return Err(LlmClientError::UpstreamHttp {+ status: StatusCode::BAD_GATEWAY,+ body: message.clone(),+ });+ }+ _ => {}+ }+ }+ events.push(event);+ }+ Ok(stream::iter(events.into_iter().map(Ok)).boxed())+}+/// Whether a failed candidate is worth routing around.fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {let LibsyError::ClientCall { source, .. } = error else {@@ -337,9 +361,7 @@ fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {};match source {LlmClientError::ContextWindowExceeded { .. } => Some(RoutingFallbackReason::ContextWindow),- LlmClientError::Transport { .. } | LlmClientError::Timeout { .. } => {- Some(RoutingFallbackReason::Unavailable)- }+ LlmClientError::Transport { .. } => Some(RoutingFallbackReason::Unavailable),LlmClientError::UpstreamHttp { status, .. }if matches!(*status,@@ -967,6 +989,7 @@ mod tests {extra_body: BTreeMap::from([("store".to_string(), json!(store))]),reasoning_effort: None,max_retries: 0,+ timeout: None,};let backend = if responses {Backend::OpenAiResponses(config)@@ -1335,6 +1358,7 @@ mod tests {extra_body: BTreeMap::new(),reasoning_effort: None,max_retries: 2,+ timeout: None,})};let client = Arc::new(@@ -1432,6 +1456,7 @@ mod tests {extra_body: BTreeMap::new(),reasoning_effort: None,max_retries: 0,+ timeout: None,})};let client = Arc::new(diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rsindex 473658281..26757982b 100644--- a/crates/libsy-llm-client/tests/observability.rs+++ b/crates/libsy-llm-client/tests/observability.rs@@ -1380,14 +1380,21 @@ async fn upstream_body_is_redacted_from_the_client_call_span() -> switchyard_libjudge_model: JUDGE.into(),outcome: JudgeOutcome::CallFailure,}) as Arc<dyn RoutedLlmClient>;- run_classifier(+ let result = run_classifier(JUDGE,"redaction-weak","redaction-strong",client,classifier_request(),)- .await?;+ .await;+ assert!(matches!(+ result,+ Err(LibsyError::ClientCall {+ source: LlmClientError::UpstreamHttp { status, body },+ ..+ }) if status == http::StatusCode::INTERNAL_SERVER_ERROR && body.contains(LEAKED_CONTENT)+ ));let spans = store.spans();let client_span = find_span(&spans, "libsy.client_call", "selected_model", JUDGE);@@ -1402,6 +1409,13 @@ async fn upstream_body_is_redacted_from_the_client_call_span() -> switchyard_lib.unwrap_or("");assert!(error.contains("upstream HTTP 500"), "{client_span:?}");assert!(!error.contains(LEAKED_CONTENT), "{client_span:?}");+ assert!(!spans.iter().any(|span| {+ span.name == "libsy.client_call"+ && matches!(+ span.fields.get("selected_model").map(String::as_str),+ Some("redaction-weak" | "redaction-strong")+ )+ }));Ok(())}@@ -1576,22 +1590,19 @@ async fn classifier_metrics_count_routing_and_answer_calls_once() -> switchyard_}#[tokio::test]-async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::Result<()> {+async fn classifier_stops_on_client_errors_and_records_verdict_fallback()+-> switchyard_libsy::Result<()> {let _guard = serialize_test().lock().await;- let (_store, exporter, provider, _, _) = telemetry();+ let (_, exporter, provider, _, _) = telemetry();let cases = [- ("fo-call", JudgeOutcome::CallFailure, Some("upstream_5xx")),+ ("fo-call", JudgeOutcome::CallFailure, None),("fo-parse",JudgeOutcome::Reply("not json at all"),Some("parse_error"),),- (- "fo-stream-decode",- JudgeOutcome::StreamDecodeFailure,- Some("invalid_response"),- ),+ ("fo-stream-decode", JudgeOutcome::StreamDecodeFailure, None),("fo-valid",JudgeOutcome::Reply(@@ -1605,15 +1616,26 @@ async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::let client = Arc::new(JudgeClient {judge_model: judge_model.into(),outcome,- }) as Arc<dyn RoutedLlmClient>;- run_classifier(+ });+ let result = run_classifier(judge_model,"fo-weak","fo-strong",- client,+ client.clone(),classifier_request(),)- .await?;+ .await;+ match client.outcome {+ JudgeOutcome::CallFailure => assert!(result.is_err(), "{judge_model}"),+ JudgeOutcome::StreamDecodeFailure => assert!(matches!(+ result,+ Err(LibsyError::ClientCall {+ source: LlmClientError::ResponseTranslation { .. },+ ..+ })+ )),+ JudgeOutcome::Reply(_) => assert_eq!(result?.0.as_str(), "fo-strong"),+ }let snapshots = flushed_metrics(exporter, provider);match expected_reason {@@ -1633,7 +1655,7 @@ async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::&[("judge_model", judge_model)],),None,- "a valid verdict was counted as a fail-open"+ "client failures and valid verdicts must not increment switchyard.classifier_fail_open"),}}diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rsindex 2d258474f..58a8513ee 100644--- a/crates/switchyard-runner/src/config.rs+++ b/crates/switchyard-runner/src/config.rs@@ -7,6 +7,7 @@ use std::collections::{BTreeMap, HashMap};use std::fs;use std::path::Path;use std::sync::Arc;+use std::time::Duration;use libsy::RuntimeModels;use serde::de::DeserializeOwned;@@ -558,6 +559,8 @@ struct LlmClientConfig {extra_headers: BTreeMap<String, String>,#[serde(default = "default_max_retries")]max_retries: u32,+ /// Deadline in milliseconds for all attempts and the complete response. Unset is unbounded.+ timeout_ms: Option<u64>,}#[derive(Debug, Deserialize)]@@ -634,6 +637,11 @@ fn build_backend("llm client {client_name} max_retries must be at most {MAX_CONFIGURED_RETRIES}")));}+ if config.timeout_ms == Some(0) {+ return Err(RunnerError::configuration(format!(+ "llm client {client_name} timeout_ms must be at least 1"+ )));+ }if config.forward_auth && config.api_key_env.is_some() {return Err(RunnerError::configuration(format!("llm client {client_name} cannot set both forward_auth and api_key_env"@@ -669,6 +677,7 @@ fn build_backend(extra_body: extra_body.clone(),reasoning_effort,max_retries: config.max_retries,+ timeout: config.timeout_ms.map(Duration::from_millis),};let backend = match config.format {ClientFormat::OpenAiChat => Backend::OpenAiChat(http),@@ -1087,17 +1096,22 @@ new = ["send_message"]#[test]fn rejects_invalid_unreferenced_llm_client() {- let invalid = format!(- "{VALID_CONFIG}\n\- [llm_clients.unused]\n\- format = \"openai_chat\"\n\- base_url = \"not a url\"\n"- );- let message = error_message(&invalid);- assert!(- message.contains("base_url must be an absolute HTTP(S) URL"),- "unexpected error: {message}"- );+ for (settings, expected) in [+ (+ "base_url = \"not a url\"",+ "base_url must be an absolute HTTP(S) URL",+ ),+ (+ "base_url = \"https://example.test\"\ntimeout_ms = 0",+ "timeout_ms must be at least 1",+ ),+ ] {+ let invalid = format!(+ "{VALID_CONFIG}\n[llm_clients.unused]\nformat = \"openai_chat\"\n{settings}"+ );+ let message = error_message(&invalid);+ assert!(message.contains(expected), "unexpected error: {message}");+ }}#[test]@@ -1581,6 +1595,27 @@ target = "azure"Ok(())}+ #[test]+ fn client_deadline_defaults_and_rejects_zero() -> RunnerResult<()> {+ for (setting, expected) in [+ ("", None),+ ("timeout_ms = 1500", Some(1500)),+ ("timeout_ms = 0", Some(0)),+ ] {+ let source = format!(+ "format = \"openai_chat\"\nbase_url = \"https://example.test/v1\"\n{setting}"+ );+ let config: LlmClientConfig = toml::from_str(&source).expect("valid deadline config");+ let backend = build_backend("test", &config, &BTreeMap::new(), None);+ if expected == Some(0) {+ assert!(backend.is_err());+ } else {+ assert_eq!(backend?.timeout(), expected.map(Duration::from_millis));+ }+ }+ Ok(())+ }+#[test]fn retry_budget_defaults_and_accepts_an_override() -> RunnerResult<()> {let default: DeploymentConfig = toml::from_str(VALID_CONFIG).map_err(|error| {diff --git a/crates/switchyard-server/tests/client_deadline.rs b/crates/switchyard-server/tests/client_deadline.rsnew file mode 100644index 000000000..58bd06fc3--- /dev/null+++ b/crates/switchyard-server/tests/client_deadline.rs@@ -0,0 +1,328 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++//! Run deadline cases in one test because they share process-wide attempt counters.++use std::convert::Infallible;+use std::error::Error;+use std::io::Write;+use std::sync::Arc;+use std::time::Duration;++use axum::body::{Body, Bytes};+use axum::extract::State;+use axum::http::{Request, StatusCode};+use axum::response::{IntoResponse, Response};+use axum::routing::post;+use axum::{Json, Router};+use http_body_util::BodyExt;+use serde_json::{Value, json};+use switchyard_server::{build_switchyard_router, config::load_server_state};+use tokio::net::TcpListener;+use tokio::sync::Mutex;+use tokio::task::JoinHandle;+use tower::ServiceExt;++type TestResult<T = ()> = Result<T, Box<dyn Error + Send + Sync>>;++// `Upstream::drop` stops the server task even when an assertion fails.+struct Upstream {+ url: String,+ calls: Arc<Mutex<Vec<Value>>>,+ task: JoinHandle<std::io::Result<()>>,+}++impl Upstream {+ async fn start() -> TestResult<Self> {+ let calls = Arc::new(Mutex::new(Vec::new()));+ let app = Router::new()+ .route("/v1/chat/completions", post(upstream))+ .with_state(Arc::clone(&calls));+ let listener = TcpListener::bind("127.0.0.1:0").await?;+ let url = format!("http://{}/v1", listener.local_addr()?);+ let task = tokio::spawn(async move { axum::serve(listener, app).await });+ Ok(Self { url, calls, task })+ }+}++impl Drop for Upstream {+ fn drop(&mut self) {+ self.task.abort();+ }+}++async fn upstream(+ State(calls): State<Arc<Mutex<Vec<Value>>>>,+ Json(body): Json<Value>,+) -> Response {+ let model = body["model"].as_str().unwrap_or_default();+ let attempt = {+ let mut calls = calls.lock().await;+ calls.push(body.clone());+ calls.iter().filter(|call| call["model"] == model).count()+ };+ if model == "backoff" || (model == "retry-body" && attempt == 1) {+ if model == "retry-body" {+ tokio::time::sleep(Duration::from_millis(300)).await;+ }+ let delay = if model == "backoff" { "1" } else { "0" };+ return (+ StatusCode::SERVICE_UNAVAILABLE,+ [("retry-after", delay)],+ "retry",+ )+ .into_response();+ }+ if model == "stalled" {+ tokio::time::sleep(Duration::from_secs(3)).await;+ }+ if body["stream"] == true {+ let model = model.to_string();+ let stream = async_stream::stream! {+ let first = json!({"id":"test", "model":model, "choices":[{"index":0,"delta":{"role":"assistant","content":"PONG"},"finish_reason":null}]});+ yield Ok::<_, Infallible>(Bytes::from(format!("data: {first}\n\n")));+ let delay = match model.as_str() { "slow-body" => 3000, "streaming" => 150, _ => 5 };+ tokio::time::sleep(Duration::from_millis(delay)).await;+ if model == "stream-error" {+ yield Ok(Bytes::from_static(b"data: {\"error\":{\"message\":\"upstream stream failed\"}}\n\ndata: [DONE]\n\n"));+ return;+ }+ let last = json!({"id":"test", "model":model, "choices":[{"index":0,"delta":{},"finish_reason":"stop"}]});+ yield Ok(Bytes::from(format!("data: {last}\n\ndata: [DONE]\n\n")));+ };+ return (+ [("content-type", "text/event-stream")],+ Body::from_stream(stream),+ )+ .into_response();+ }+ let content = if model == "retry-body" {+ r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.9}"#+ } else {+ "PONG"+ };+ let response = json!({"id":"test","object":"chat.completion","model":model,+ "choices":[{"index":0,"message":{"role":"assistant","content":content},"finish_reason":"stop"}],+ "usage":{"prompt_tokens":3,"completion_tokens":1,"total_tokens":4}});+ if model == "slow-body" || model == "retry-body" {+ let delay = if model == "retry-body" { 450 } else { 3000 };+ let stream = async_stream::stream! {+ yield Ok::<_, Infallible>(Bytes::from_static(b" "));+ tokio::time::sleep(Duration::from_millis(delay)).await;+ yield Ok(Bytes::from(response.to_string()));+ };+ return (+ [("content-type", "application/json")],+ Body::from_stream(stream),+ )+ .into_response();+ }+ Json(response).into_response()+}++fn app(+ upstream: &Upstream,+ mode: &str,+ judge: &str,+ weak: &str,+ timeout: u64,+) -> TestResult<Router> {+ let route = match mode {+ "escalation" => {+ "type = \"llm_classifier\"\nmode = \"escalation\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nescalation = { confirmations = 1 }"+ }+ "candidates" => {+ "type = \"random\"\ntargets = [\"weak\", \"strong\"]\nweights = [1000000, 1]\nseed = 17"+ }+ "terminal" => "type = \"passthrough\"\ntarget = \"weak\"",+ _ => {+ "type = \"llm_classifier\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nbase_threshold = 0.5"+ }+ };+ let mut config = tempfile::Builder::new().suffix(".toml").tempfile()?;+ write!(+ config,+ r#"+schema_version = 1+[llm_clients.http]+format = "openai_chat"+base_url = "{}"+max_retries = 1+timeout_ms = {timeout}+[targets]+judge = {{ id = "{judge}", llm_client = "http", extra_body = {{ stream = {judge_stream} }} }}+weak = {{ id = "{weak}", llm_client = "http" }}+strong = {{ id = "strong", llm_client = "http" }}+[routes.test]+id = "test"+{route}+"#,+ upstream.url,+ judge_stream = judge == "stream-error",+ )?;+ Ok(build_switchyard_router(load_server_state(config.path())?))+}++async fn send(app: &Router, path: &str, body: Option<Value>) -> TestResult<(StatusCode, String)> {+ let (method, body) = match body {+ Some(body) => ("POST", Body::from(serde_json::to_vec(&body)?)),+ None => ("GET", Body::empty()),+ };+ let response = tokio::time::timeout(+ Duration::from_secs(2),+ app.clone().oneshot(+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(body)?,+ ),+ )+ .await??;+ let status = response.status();+ let bytes = tokio::time::timeout(Duration::from_secs(2), response.into_body().collect())+ .await??+ .to_bytes();+ Ok((status, String::from_utf8(bytes.to_vec())?))+}++fn metric(text: &str, code: &str) -> f64 {+ text.lines()+ .find(|line| {+ line.starts_with("switchyard_upstream_attempts_total{")+ && line.contains(&format!("code=\"{code}\""))+ })+ .and_then(|line| line.split_whitespace().last())+ .and_then(|value| value.parse().ok())+ .unwrap_or_default()+}++async fn check(+ upstream: &Upstream,+ app: &Router,+ endpoint: &str,+ expected_calls: &[&str],+ expected_counts: [f64; 3],+) -> TestResult<(StatusCode, String)> {+ upstream.calls.lock().await.clear();+ let (_, before) = send(app, "/metrics", None).await?;+ let mut body = if endpoint == "/v1/responses" {+ json!({"model":"test", "input":"Say PONG", "stream":true})+ } else {+ json!({"model":"test", "messages":[{"role":"user","content":"Say PONG"}], "max_tokens":32, "stream":true})+ };+ if endpoint == "/v1/decision" {+ body = json!({"input_format":"openai_chat", "request":body});+ }+ let result = send(app, endpoint, Some(body)).await?;+ let calls = upstream.calls.lock().await.clone();+ let actual: Vec<_> = calls+ .iter()+ .filter_map(|call| call["model"].as_str())+ .collect();+ assert_eq!(actual, expected_calls, "{endpoint}: {}", result.1);+ let (_, after) = send(app, "/metrics", None).await?;+ // Count HTTP 200, HTTP 503, and pre-response failures separately. A stream+ // that times out after HTTP 200 must not count as another attempt.+ let counts = ["200", "503", "none"].map(|code| metric(&after, code) - metric(&before, code));+ assert_eq!(counts, expected_counts, "{endpoint}: {}", result.1);+ Ok(result)+}++#[tokio::test]+async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult {+ let upstream = Upstream::start().await?;+ let stalled = app(&upstream, "classifier", "stalled", "weak", 100)?;+ let streaming = app(&upstream, "terminal", "judge", "streaming", 500)?;+ let slow_stream = app(&upstream, "terminal", "judge", "slow-body", 100)?;+ for endpoint in [+ "/v1/chat/completions",+ "/v1/messages",+ "/v1/responses",+ "/v1/decision",+ ] {+ let (status, body) =+ check(&upstream, &stalled, endpoint, &["stalled"], [0., 0., 1.]).await?;+ assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{body}");+ assert!(body.contains("100 ms"), "{body}");+ if endpoint == "/v1/decision" {+ continue;+ }+ let (status, body) = check(+ &upstream,+ &streaming,+ endpoint,+ &["streaming"],+ [1., 0., 0.],+ )+ .await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert!(body.contains("PONG"), "{body}");+ let success = match endpoint {+ "/v1/messages" => "message_stop",+ "/v1/responses" => "response.completed",+ _ => "[DONE]",+ };+ assert!(body.contains(success), "{body}");+ let (status, body) = check(+ &upstream,+ &slow_stream,+ endpoint,+ &["slow-body"],+ [1., 0., 0.],+ )+ .await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert!(body.contains("PONG") && body.contains("100 ms"), "{body}");+ assert!(!body.contains(success), "{body}");+ assert!(!body.contains("\"finish_reason\":\"stop\""), "{body}");+ }++ // These response and retry failures use the same client code for every endpoint.+ for (judge, expected_calls, counts) in [+ ("slow-body", &["slow-body"][..], [0., 0., 1.]),+ ("backoff", &["backoff"][..], [0., 1., 0.]),+ // The first attempt waits 300 ms and the retry body waits 450 ms.+ // Each fits the 600 ms deadline alone; together they exceed it.+ (+ "retry-body",+ &["retry-body", "retry-body"][..],+ [0., 1., 1.],+ ),+ ] {+ let app = app(&upstream, "classifier", judge, "weak", 600)?;+ let (status, body) = check(+ &upstream,+ &app,+ "/v1/chat/completions",+ expected_calls,+ counts,+ )+ .await?;+ assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{judge}: {body}");+ assert!(body.contains("600 ms"), "{body}");+ }+ let failed_stream = app(&upstream, "classifier", "stream-error", "weak", 100)?;+ let (status, body) = check(+ &upstream,+ &failed_stream,+ "/v1/chat/completions",+ &["stream-error"],+ [1., 0., 0.],+ )+ .await?;+ assert_eq!(status, StatusCode::BAD_GATEWAY, "{body}");+ for (mode, weak, counts) in [+ ("escalation", "slow-body", [1., 0., 0.]),+ ("candidates", "stalled", [0., 0., 1.]),+ ] {+ let app = app(&upstream, mode, "judge", weak, 100)?;+ let (status, body) =+ check(&upstream, &app, "/v1/chat/completions", &[weak], counts).await?;+ assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{mode}: {body}");+ if mode == "escalation" {+ assert_eq!(upstream.calls.lock().await[0]["stream"], true);+ }+ }+ Ok(())+}diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rsindex 00295e7d9..98ea8387a 100644--- a/crates/switchyard-server/tests/server.rs+++ b/crates/switchyard-server/tests/server.rs@@ -730,6 +730,7 @@ fn random_state_with_retries(extra_body: BTreeMap::new(),reasoning_effort: None,max_retries,+ timeout: None,});let target_models = routes.iter()@@ -4635,7 +4636,7 @@ fn gate_count(stats: &Value, path: &[&str]) -> u64 {// process-global, so this is the only test that emits redo / consult-failure// metrics and the only one that may assert their exact counts.#[tokio::test]-async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {+async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {let upstream = MockUpstream::start().await?;let app = build_switchyard_router(advisor_state_no_retry(&upstream.base_url)?);let before = send(&app, "GET", "/v1/stats", None).await?.json()?;@@ -4673,7 +4674,7 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {assert!(feedback.starts_with("A senior reviewer examined your work"));assert!(feedback.ends_with("run the tests"));- // Fail-open: the advisor 503s once (no retries) and the turn still flows.+ // A failed HTTP advisor call stops the request before the algorithm can approve it.let response = send_with_headers(&app,"POST",@@ -4682,8 +4683,8 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {&[("proxy_x_session_id", "fail-flow")],).await?;- assert_eq!(response.status, StatusCode::OK);- assert_eq!(response.json()?["choices"][0]["message"]["content"], "ok");+ assert_eq!(response.status, StatusCode::SERVICE_UNAVAILABLE);+ assert_eq!(response.json()?["error"]["type"], "upstream_error");assert_eq!(upstream.models().await,[@@ -4696,9 +4697,8 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {);let stats = send(&app, "GET", "/v1/stats", None).await?.json()?;- // State-owned accumulator: two client-visible executor answers; the discarded REDO attempt- // is routing work. The failed advisor consult is counted separately.- assert_eq!(stats["models"]["model/executor"]["calls"], 2);+ // Only the successful redo request returns an executor answer.+ assert_eq!(stats["models"]["model/executor"]["calls"], 1);assert_eq!(stats["classifier"]["total_errors"], 1);// Projection deltas for the metrics only this test emits.let redo = gate_count(&stats, &["reviews", "redo", "total"])@@ -4725,10 +4725,10 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {gate_count(&stats, &["discarded", "tokens", "output"]),gate_count(&before, &["discarded", "tokens", "output"]) + 2);- // The 503 maps to the bounded upstream_5xx reason label.+ // The host stops before the algorithm records a fail-open advisor decision.assert_eq!(gate_count(&stats, &["consult_failures", "upstream_5xx"]),- gate_count(&before, &["consult_failures", "upstream_5xx"]) + 1+ gate_count(&before, &["consult_failures", "upstream_5xx"]));// Reset re-baselines the projection: the redo/discard counts this testdiff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex e595e7cef..d5c78c17e 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -54,10 +54,23 @@ route reaches no upstream. A file without a `[targets]` table is rejected with| `forward_auth` | No | `false` | Forward the caller's provider credential and application headers. All backends reachable through the route must use the same provider. || `extra_headers` | No | `{}` | Custom HTTP headers sent to the model server. Set credentials with `api_key_env` or `forward_auth`; the server rejects headers owned by the selected auth mode. Header names are case-insensitive. || `max_retries` | No | `2` | Retry budget, `0`–`10`. |+| `timeout_ms` | No | unset | Deadline in milliseconds for all attempts, retry delays, and the complete response, including stream reads. Must be at least `1`. Unset leaves the wait unbounded. |The TOML never contains the secret itself. `api_key_env` names a variable thatmust exist and be non-empty when the server loads.+`timeout_ms` applies separately to every call through the client, including judge+verdicts and answers. To give a judge a short deadline without limiting the+answering models, put the judge on its own `[llm_clients]` entry; two entries may+share a `base_url`. When the deadline expires, the server returns `504` without+trying another target. If the final answer has already started streaming, the+server sends a framed error and ends the stream without a success marker.++The Rust runner collects streams used during routing before the algorithm+continues, preserving provider events for replay. After the configured retries,+an HTTP client failure stops routing. This also applies when `timeout_ms` is+unset or an advisor has `fail_open = true`.+Set `forward_auth = true` to use each caller's credential instead of aserver-owned key:@@ -222,8 +235,9 @@ Existing configurations that contain `escalation` but omit `mode` remain valid.Custom mode validates the judge's JSON against `response_schema`, resolves thepolicy selector, and routes to a runtime model group. A verdict names a group and-the first model in it serves the turn; if that call fails the client falls through-the rest of that group, then through whatever `models.any` adds.+the first model in it serves the turn. An eligible non-timeout failure tries the+rest of that group, then any remaining models in `models.any`. A timeout stops+the request without trying another model.The `[routes.<name>.models]` table takes any group name you choose. `any` and`judge` are reserved and required; `capable` and `efficient` are reserved for the@@ -232,12 +246,12 @@ how one route chooses between more than two models.| Key | Required | Default | Meaning ||---|:---:|---|---|-| `models.any` | Yes | — | Every selectable completion target, in last-resort fallback order. Every other group's targets must also appear here; one that does not is rejected at configuration load. |-| `models.judge` | Yes | — | One or more ordered judge candidates. Not a completion destination. |-| `models.capable` | No | — | Ordered capable-tier models. A `capable` verdict selects the first and falls through the rest in order. |-| `models.efficient` | No | — | Ordered efficient-tier models. An `efficient` verdict selects the first and falls through the rest in order. |-| `models.<your name>` | No | — | A group you name. A verdict naming it selects its first model and falls through the rest in order. |-| `default_target` | Yes | — | Group used when the judge fails or its verdict cannot be routed. Any group except `judge`, and it must contain at least one target. |+| `models.any` | Yes | — | Every selectable completion target, in fallback order for eligible non-timeout failures. Every other group's targets must also appear here; one that does not is rejected at configuration load. |+| `models.judge` | Yes | — | One or more ordered judge candidates. The Rust runner calls the first and stops on a client error after retries. Not a completion destination. |+| `models.capable` | No | — | Ordered capable-tier models. A `capable` verdict selects the first; eligible non-timeout failures try the rest in order. |+| `models.efficient` | No | — | Ordered efficient-tier models. An `efficient` verdict selects the first; eligible non-timeout failures try the rest in order. |+| `models.<your name>` | No | — | A group you name. A verdict naming it selects its first model; eligible non-timeout failures try the rest in order. |+| `default_target` | Yes | — | Group used when the judge's verdict cannot be parsed or routed. HTTP client failures stop the request after retries. Any group except `judge`, and it must contain at least one target. || `prompt` | Yes | — | Judge system prompt. The configured inner schema is sent separately as structured-output configuration. || `response_schema` | Yes | — | Inner JSON Schema encoded as a TOML string. Switchyard adds the provider wrapper. || `policy` | Yes | — | Policy table. `target_selector` accepts a JSON Pointer such as `/decision/target`. |diff --git a/docs/routing_algorithms/advisor_gate_routing.md b/docs/routing_algorithms/advisor_gate_routing.mdindex 610530cb9..5af0077b3 100644--- a/docs/routing_algorithms/advisor_gate_routing.md+++ b/docs/routing_algorithms/advisor_gate_routing.md@@ -61,15 +61,20 @@ stamp every request of one evaluation with it, sub-agents included, so thebudget means "reviews for this task" even behind a gateway shared by manytasks — then the session id resolved from harness headers (such as`x-switchyard-session-id`), and one scope per server when neither is present.-Failed consults refund the budget and count toward a separate cap of 3, which-bounds consult latency against a down advisor. An unparseable verdict also-refunds and passes the turn through as APPROVE.+For hosts that return client errors to the core algorithm, a failed consult refunds+the review budget and counts toward a separate cap of 3. The Rust runner stops on+HTTP client errors before returning them to the algorithm, so failed HTTP calls+do not refund this budget. An unparseable verdict still refunds the budget and+passes the turn through as APPROVE.Long sessions are truncated middle-out before the consult: the transcript keepsthe task statement at the start and the most recent work at the end, marked`...<middle of the conversation truncated>...`, capped at-`transcript_max_chars`. With `fail_open = true` (default) any advisor failure-degrades to APPROVE; `fail_open = false` surfaces it as a server error instead.+`transcript_max_chars`. `fail_open` controls failures returned to the core+algorithm by its host. The Rust runner stops the request after an HTTP client's+configured retries fail or its deadline expires, even with `fail_open = true`.+HTTP client errors also stop the request when `timeout_ms` is unset. A timeout+before the answer starts returns `504`.Gate behavior is observable at `/v1/stats` under `advisor_gate`: verdicts bytrigger, consult failures by reason, and REDO-discarded turns with their token@@ -109,7 +114,7 @@ gate_min_tool_results = 3| `advisor_max_tokens` | `2048` | Output cap for each advisor consult. || `advisor_temperature` | unset | Sampling temperature for consults; omitted when unset. || `transcript_max_chars` | `200000` | Middle-out cap on the serialized transcript (~50k tokens). |-| `fail_open` | `true` | Advisor failure passes the turn through instead of erroring. |+| `fail_open` | `true` | Failures returned to the core algorithm pass the turn through. The Rust runner stops HTTP client failures regardless of this setting. || `reviewer_system_prompt` | built-in | Overrides the APPROVE/REDO reviewer contract. || `redo_feedback_prefix` | built-in | Overrides the prefix injected before a REDO plan. |diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.mdindex 1ed9e8085..3fecddab6 100644--- a/docs/routing_algorithms/escalation_router_routing.md+++ b/docs/routing_algorithms/escalation_router_routing.md@@ -86,9 +86,12 @@ flowchart LRclass t,p,s,c,j,w,l box;```-A judge that times out, errors, or returns an unparseable verdict fails open: the-turn serves the buffered weak reply and the existing streak is held rather than-cleared. A judge failure never creates a strong-tier latch.+An unparseable verdict serves the buffered weak reply and holds the existing+streak. The Rust runner stops the request if an HTTP model call fails after+retries. To bound both the weak-model response and the judge response, set+`timeout_ms` on each `[llm_clients]` entry they use. The deadline applies separately+to each call, even when both models share a client. Expiry returns `504` without+calling another target or selecting the strong tier for subsequent session turns.## Judge model compatibilitydiff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.mdindex 329ccadc0..c1b3a3fbe 100644--- a/docs/routing_algorithms/llm_classifier_routing.md+++ b/docs/routing_algorithms/llm_classifier_routing.md@@ -67,9 +67,19 @@ greater than or equal to the applicable threshold. Otherwise it routes to- `uncertain` and `unmatched` use `base_threshold + threshold_step`.- `unsupported` uses `base_threshold + 2 * threshold_step`.-An invalid, inconsistent, or unparseable verdict, or a judge failure, routes to+An invalid, inconsistent, or unparseable verdict routes to`strong_target`. Raising either knob sends more traffic to the strong model.+To stop waiting for a judge that accepts the request but never finishes its+response, set `timeout_ms` on the judge's `[llm_clients]` entry+(see the [TOML schema](../reference/toml_schema.md)); it covers the judge's+retries and the complete verdict body. When the deadline expires, the Rust server returns+`504` without calling `strong_target` or `weak_target`. Other HTTP client failures+also stop routing after retries. The deadline applies to every call+through that client. Give the judge its own entry if the answering models need a+different deadline, even when they use the same provider. Without a deadline,+the request can wait indefinitely for the judge.+## Judge model compatibilityThe judge must return complete, schema-valid JSON in normal assistant `content`.