Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

fix(client): enforce response deadlines and stop routing on errors (#702)

Elyas Mehtabuddin1759dfdff95e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0e987ef86..54a2349b7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
### Added
+- **`timeout_ms` on `[llm_clients.<name>]`** — one deadline covers all attempts,
+ retry delays, and the complete response, including stream reads. Unset leaves
+ the wait unbounded; `0` is rejected. A timeout returns `504` without trying another
+ model, or a framed error if the final answer has already started streaming.
+ Timed-out attempts are counted in metrics.
- **Per-target `reasoning_effort`** — a target can force the reasoning effort
of every request it serves, replacing the caller's value (`reasoning.effort`
on the Responses wire, `reasoning_effort` on Chat Completions), so a strong
@@ -89,6 +94,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
### Changed
+- **HTTP client errors stop routing** — after the configured retries, the Rust
+ runner stops the request instead of letting the routing algorithm choose a
+ fallback. This also applies when `timeout_ms` is unset or an advisor has
+ `fail_open = true`. The runner collects streams used during routing and preserves
+ provider events for replay. Invalid judge verdicts keep their existing fallback.
- **`Algorithm::route` returns `Result<RoutingOutcome>`** — instead of the
bare final `Result`, so callers observe the full routing outcome (see #458
for the design). (#459)
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.md
index bad3f86d0..bb3b38caf 100644
--- a/crates/libsy-llm-client/README.md
+++ b/crates/libsy-llm-client/README.md
@@ -36,8 +36,8 @@ It depends on `switchyard-libsy`, `switchyard-protocol`, and
`anthropic-version`).
- **Model rewrite.** The resolved [`ModelId`] is both the map key and the model id
sent upstream — it overwrites whatever `model` the request arrived with.
-- **Streaming is chosen by the request.** If the encoded body has `stream: true`
- (i.e. `request.llm_request.stream`), you get `LlmResponse::Stream`; otherwise
+- **Streaming is chosen by the encoded request body.** If the body has
+ `stream: true` after `extra_body` is applied, you get `LlmResponse::Stream`; otherwise
`LlmResponse::Agg`. OpenAI Chat streaming requests default
`stream_options.include_usage` to `true`; an explicit caller value is preserved.
@@ -69,7 +69,9 @@ fn build_client() -> switchyard_llm_client::Result<TranslatingLlmClient> {
forward_auth: false,
extra_headers: BTreeMap::new(),
extra_body: BTreeMap::new(),
+ reasoning_effort: None,
max_retries: 2,
+ timeout: None,
};
let models = [ModelConfig::new(
@@ -247,13 +249,23 @@ fn build_multi_format_client(
transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body
transport failures are retried; streaming body failures are not replayed after
the response has been returned.
-
-Retries replay the same upstream request to the same model. Each candidate's
-`max_retries` budget is exhausted before candidate fallback advances to the next
-model. The worst case is `candidates × (max_retries + 1)` upstream requests, and
-total latency includes every candidate's capped `Retry-After` backoff. A transport
-failure can duplicate a request that the provider processed but did not finish
-returning.
+- `HttpBackendConfig::timeout` bounds one complete response, including retries,
+ retry delays, and every stream read. Expiry returns `LlmClientError::Timeout`,
+ either from the call or from the returned stream, which then ends. `None` leaves
+ the wait unbounded.
+
+`run` and `decide` collect streams used during routing, including answers that an
+algorithm must inspect, before returning them to the algorithm. They retain the
+provider events for replay. After the configured retries, a client failure stops
+`run` or `decide` before the algorithm can choose another routing candidate. This
+also applies when `timeout` is `None` or an advisor has `fail_open = true`.
+`libsy` and custom hosts that drive it directly are unchanged.
+
+Retries replay the same upstream request to the same model. After routing completes,
+non-timeout failures may try another completion candidate. A timeout stops the call.
+A transport failure can duplicate a request that the provider processed but did not
+finish returning. Each attempt is counted once; expiry during a retry delay or
+after a stream has started does not count another attempt.
## Errors
diff --git a/crates/libsy-llm-client/src/backend.rs b/crates/libsy-llm-client/src/backend.rs
index da43e2c63..a96a3e1fa 100644
--- a/crates/libsy-llm-client/src/backend.rs
+++ b/crates/libsy-llm-client/src/backend.rs
@@ -3,6 +3,7 @@
//! Per-provider backend configuration: wire format, upstream URL, and auth.
+use std::time::Duration;
use std::{collections::BTreeMap, fmt};
use reqwest::RequestBuilder;
@@ -62,6 +63,9 @@ pub struct HttpBackendConfig {
pub reasoning_effort: Option<String>,
/// Additional attempts after the initial upstream request.
pub max_retries: u32,
+ /// Deadline for one complete response, including retries, retry delays, and stream reads.
+ /// `None` leaves the wait unbounded.
+ pub timeout: Option<Duration>,
}
impl fmt::Debug for HttpBackendConfig {
@@ -74,6 +78,7 @@ impl fmt::Debug for HttpBackendConfig {
.field("extra_body_keys", &self.extra_body.keys())
.field("reasoning_effort", &self.reasoning_effort)
.field("max_retries", &self.max_retries)
+ .field("timeout", &self.timeout)
.finish()
}
}
@@ -256,6 +261,11 @@ impl Backend {
self.config().max_retries
}
+ /// Deadline for all attempts and the complete response; `None` leaves the wait unbounded.
+ pub fn timeout(&self) -> Option<Duration> {
+ self.config().timeout
+ }
+
/// Whether this backend speaks the Anthropic Messages wire format — the only
/// one with a `count_tokens` endpoint.
pub fn is_anthropic(&self) -> bool {
@@ -348,6 +358,7 @@ mod tests {
extra_body: BTreeMap::new(),
reasoning_effort: None,
max_retries: 0,
+ timeout: None,
}
}
diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rs
index f9580a537..dbb69b2ca 100644
--- a/crates/libsy-llm-client/src/client.rs
+++ b/crates/libsy-llm-client/src/client.rs
@@ -224,8 +224,9 @@ impl TranslatingLlmClient {
/// forwarded headers plus the backend's static headers and auth, and return the
/// successful upstream response. A
/// buffered response is fully collected within the retry boundary; a streamed
- /// response has its first decoded event checked within that boundary. A non-success
- /// status maps to a typed error — a 400 is classified as a context-window
+ /// response has its first decoded event checked within that boundary. The backend's
+ /// deadline, when set, spans every attempt, retry delay, and subsequent stream read. A
+ /// non-success status maps to a typed error — a 400 is classified as a context-window
/// overflow via the backend's provider rules. Shared by
/// [`call_rewrite_model`](Self::call_rewrite_model) (which POSTs to the
/// backend's completion URL and decodes a response) and
@@ -269,15 +270,47 @@ impl TranslatingLlmClient {
let url = endpoint.url(backend);
record_gen_ai_request(&url, model, streaming);
+ self.send_with_retries(&url, backend, &body, metadata, model, streaming)
+ .await
+ }
+
+ // Sends the encoded body, retrying retryable failures within the backend's retry budget.
+ async fn send_with_retries(
+ &self,
+ url: &str,
+ backend: &Backend,
+ body: &Value,
+ metadata: Option<&Metadata>,
+ model: &ModelId,
+ streaming: bool,
+ ) -> Result<EncodedResponse> {
let max_retries = u64::from(backend.max_retries());
let max_attempts = max_retries + 1;
+ let expires_at = backend
+ .timeout()
+ .map(|timeout| (tokio::time::sleep(timeout).deadline(), timeout));
+ let deadline = async {
+ match expires_at {
+ Some((at, timeout)) => {
+ tokio::time::sleep_until(at).await;
+ timeout
+ }
+ None => std::future::pending().await,
+ }
+ };
+ tokio::pin!(deadline);
let mut attempt = 0_u64;
loop {
+ if let Some((at, timeout)) = expires_at
+ && tokio::time::Instant::now() >= at
+ {
+ return Err(deadline_error(timeout));
+ }
let span = tracing::debug_span!(
target: "libsy",
"libsy.upstream_attempt",
model = %model,
- wire_format = %wire_format,
+ wire_format = %backend.wire_format(),
attempt = attempt + 1,
max_attempts,
retry = attempt > 0,
@@ -287,10 +320,22 @@ impl TranslatingLlmClient {
will_retry = tracing::field::Empty,
retry_delay_ms = tracing::field::Empty,
);
- let result = self
- .send_once(&url, backend, &body, metadata, model, streaming)
- .instrument(span.clone())
- .await;
+ let mut attempt_started = false;
+ let result = tokio::select! {
+ biased;
+ timeout = &mut deadline => {
+ if attempt_started {
+ metrics::record_upstream_attempt(None);
+ }
+ span.record("outcome", "error");
+ span.record("will_retry", false);
+ return Err(deadline_error(timeout));
+ }
+ result = async {
+ attempt_started = true;
+ self.send_once(url, backend, body, metadata, model, streaming).await
+ }.instrument(span.clone()) => result,
+ };
// The retained handle updates this same attempt span with its outcome.
match result {
Ok(response) => {
@@ -300,7 +345,18 @@ impl TranslatingLlmClient {
if attempt > 0 {
metrics::record_retry_recovered();
}
- return Ok(response);
+ return Ok(match response {
+ EncodedResponse::Streaming {
+ status,
+ chunks,
+ upstream_headers,
+ } => EncodedResponse::Streaming {
+ status,
+ chunks: stream_with_deadline(chunks, expires_at),
+ upstream_headers,
+ },
+ buffered => buffered,
+ });
}
Err(failure) => {
let will_retry = attempt < max_retries && failure.is_retryable();
@@ -317,7 +373,11 @@ impl TranslatingLlmClient {
span.record("retry_delay_ms", duration_millis(delay));
// Close the attempt span before sleeping so backoff is not attempt latency.
drop(span);
- tokio::time::sleep(delay).await;
+ tokio::select! {
+ biased;
+ timeout = &mut deadline => return Err(deadline_error(timeout)),
+ _ = tokio::time::sleep(delay) => {}
+ }
attempt += 1;
}
}
@@ -653,6 +713,18 @@ struct AttemptFailure {
retry_after: Option<Duration>,
}
+fn deadline_error(timeout: Duration) -> LlmClientError {
+ LlmClientError::Timeout {
+ source: Box::new(std::io::Error::new(
+ std::io::ErrorKind::TimedOut,
+ format!(
+ "response did not finish within {} ms, retries included",
+ duration_millis(timeout)
+ ),
+ )),
+ }
+}
+
impl AttemptFailure {
fn is_retryable(&self) -> bool {
match &self.error {
@@ -665,6 +737,26 @@ impl AttemptFailure {
}
}
+fn stream_with_deadline(
+ chunks: LlmResponseStream,
+ deadline: Option<(tokio::time::Instant, Duration)>,
+) -> LlmResponseStream {
+ let Some((at, timeout)) = deadline else {
+ return chunks;
+ };
+ stream::unfold(Some(chunks), move |chunks| async move {
+ let mut chunks = chunks?;
+ let chunk = tokio::select! {
+ biased;
+ _ = tokio::time::sleep_until(at) => return Some((Err(deadline_error(timeout)), None)),
+ chunk = chunks.next() => chunk?,
+ };
+ let next = chunk.is_ok().then_some(chunks);
+ Some((chunk, next))
+ })
+ .boxed()
+}
+
async fn prepare_response_stream(
response: reqwest::Response,
backend: &Backend,
@@ -1199,6 +1291,7 @@ mod tests {
extra_body: BTreeMap::new(),
reasoning_effort: None,
max_retries: 0,
+ timeout: None,
}
}
@@ -2212,6 +2305,52 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn deadline_expires_before_send_or_stream_poll()
+ -> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {
+ let server = MockServer::start().await;
+ Mock::given(method("POST"))
+ .respond_with(ResponseTemplate::new(200).set_body_raw(
+ "data: {\"id\":\"test\",\"model\":\"gpt\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n",
+ "text/event-stream",
+ ))
+ .mount(&server)
+ .await;
+ for timeout in [Duration::ZERO, Duration::from_millis(200)] {
+ let mut backend = config(&server.uri());
+ backend.timeout = Some(timeout);
+ let client = TranslatingLlmClient::new(&[ModelConfig::new(
+ "gpt",
+ Backend::OpenAiChat(backend),
+ None,
+ )])?;
+ let result = client
+ .call_rewrite_model(request_for(Some("gpt"), true), None)
+ .await;
+ if timeout.is_zero() {
+ assert!(matches!(result, Err(LlmClientError::Timeout { .. })));
+ assert!(
+ server
+ .received_requests()
+ .await
+ .expect("recorded requests")
+ .is_empty()
+ );
+ } else {
+ let LlmResponse::Stream(mut chunks) = result?.llm_response else {
+ return Err("expected a streaming response".into());
+ };
+ tokio::time::sleep(Duration::from_millis(300)).await;
+ assert!(matches!(
+ chunks.next().await,
+ Some(Err(LlmClientError::Timeout { .. }))
+ ));
+ assert!(chunks.next().await.is_none());
+ }
+ }
+ Ok(())
+ }
+
#[tokio::test]
async fn timeout_is_retried_before_a_response_is_returned()
-> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {
diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rs
index c16922c44..2326c951c 100644
--- a/crates/libsy-llm-client/src/run.rs
+++ b/crates/libsy-llm-client/src/run.rs
@@ -10,8 +10,8 @@
//!
//! libsy owns the stream mechanics; what this module adds is per-target request preparation,
//! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each
-//! candidate exhausts its backend retry budget before fallback advances, so the worst case is
-//! `candidates × (max_retries + 1)` upstream attempts plus every candidate's backoff.
+//! completion candidate exhausts its backend retry budget before fallback advances. Routing
+//! calls stop on the first candidate's failure. A timeout stops either kind of call.
//!
//! A Responses continuation can refer to state held by one provider through
//! `previous_response_id` or `conversation`. When completion targets use different clients,
@@ -22,7 +22,7 @@ use std::collections::HashMap;
use std::sync::Arc;
use std::time::Instant;
-use futures_util::StreamExt;
+use futures::{StreamExt, TryStreamExt, stream};
use http::StatusCode;
use parking_lot::Mutex;
use serde_json::{Value, json};
@@ -30,8 +30,8 @@ use switchyard_libsy::{
Algorithm, CallModel, LibsyError, OutcomeMetadata, Result, RoutingOutcome, RuntimeModels, drive,
};
use switchyard_protocol::{
- LlmClientError, LlmResponse, ModelId, Request, Response, RoutedLlmClient,
- RoutingFallbackReason, WireFormat,
+ LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStream, ModelId, Request, Response,
+ RoutedLlmClient, RoutingFallbackReason, WireFormat,
};
use switchyard_translation::prepare_request_for_target;
@@ -48,8 +48,8 @@ use crate::{metrics, observability};
/// a per-call lookup, not one client for the whole run. Use
/// [`ClientRouter::single`](ClientRouter::single) when one client serves every target.
///
-/// Routing-time model failures are forwarded back into the algorithm. Once routing completes,
-/// this client exhausts backend retries and then the outcome's ordered candidate fallbacks.
+/// Routing calls are buffered; a client failure stops the request before the algorithm continues.
+/// Once routing completes, non-timeout failures may try the outcome's ordered fallback candidates.
pub async fn run(
algorithm: Arc<dyn Algorithm>,
clients: ClientRouter,
@@ -98,7 +98,6 @@ pub async fn run(
&algorithm_name,
&outcome.request,
&outcome.selected_model_ids,
- CallPhase::Completion,
&observe,
)
.await;
@@ -171,8 +170,7 @@ fn emit_routing_observations(
/// Serve one offloaded call and fulfill its promise.
///
-/// Errors only when the promise itself could not be fulfilled; a call that failed on every
-/// candidate is forwarded to the algorithm as an `Err`.
+/// A client failure stops the driver before the algorithm can issue another call.
async fn serve(
clients: ClientRouter,
call: CallModel,
@@ -183,21 +181,20 @@ async fn serve(
observations.lock().push(observation);
}
};
- let result = call_first_available(
+ let target = call.models.first().ok_or(LibsyError::NoTargets)?;
+ let request = clients.prepare_routing_request(call.request.clone(), target);
+ let response = call_one(
&clients,
+ target,
+ request,
&call.algorithm,
- &call.request,
- &call.models,
- CallPhase::Routing,
&observe,
+ 0,
+ call.models.len(),
+ true,
)
- .await;
- call.respond(result)
-}
-
-enum CallPhase {
- Routing,
- Completion,
+ .await?;
+ call.respond(Ok(response))
}
/// Try candidates in order until one succeeds or a failure stops fallback.
@@ -206,14 +203,10 @@ async fn call_first_available(
algorithm: &str,
request: &Request,
models: &[ModelId],
- phase: CallPhase,
observe: &(dyn Fn(LlmCallObservation) + Send + Sync),
) -> Result<Response> {
for (index, target) in models.iter().enumerate() {
- let request = match phase {
- CallPhase::Routing => clients.prepare_routing_request(request.clone(), target),
- CallPhase::Completion => clients.prepare_completion_request(request.clone(), target),
- };
+ let request = clients.prepare_completion_request(request.clone(), target);
match call_one(
clients,
target,
@@ -222,6 +215,7 @@ async fn call_first_available(
observe,
index,
models.len(),
+ false,
)
.await
{
@@ -291,6 +285,7 @@ async fn call_one(
index: usize,
// count is for span log
count: usize,
+ buffer: bool,
) -> Result<Response> {
let span = tracing::Span::current();
observability::record_gen_ai_request(&span, &request.llm_request);
@@ -305,10 +300,14 @@ async fn call_one(
// the provider's, so it belongs in the routing overhead.
let client = clients.route(model_id);
let started = Instant::now();
- let result = match client {
- Ok(client) => client.call(request).await,
- Err(error) => Err(error),
+ let result = async {
+ let mut response = client?.call(request).await?;
+ if buffer && let LlmResponse::Stream(chunks) = response.llm_response {
+ response.llm_response = LlmResponse::Stream(buffer_routing_stream(chunks).await?);
+ }
+ Ok(response)
}
+ .await
.map_err(|source| LibsyError::client_call(model_id.clone(), source));
let duration = started.elapsed();
@@ -330,6 +329,31 @@ async fn call_one(
result
}
+// Preserve signed provider events while checking the complete routing response.
+async fn buffer_routing_stream(
+ mut chunks: LlmResponseStream,
+) -> std::result::Result<LlmResponseStream, LlmClientError> {
+ let mut events = Vec::new();
+ while let Some(event) = chunks.try_next().await? {
+ for chunk in event.normalized() {
+ match chunk {
+ LlmResponseChunk::DecodeError { message } => {
+ return Err(LlmClientError::ResponseTranslation(message.clone()));
+ }
+ LlmResponseChunk::StreamError { message } => {
+ return Err(LlmClientError::UpstreamHttp {
+ status: StatusCode::BAD_GATEWAY,
+ body: message.clone(),
+ });
+ }
+ _ => {}
+ }
+ }
+ events.push(event);
+ }
+ Ok(stream::iter(events.into_iter().map(Ok)).boxed())
+}
+
/// Whether a failed candidate is worth routing around.
fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {
let LibsyError::ClientCall { source, .. } = error else {
@@ -337,9 +361,7 @@ fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {
};
match source {
LlmClientError::ContextWindowExceeded { .. } => Some(RoutingFallbackReason::ContextWindow),
- LlmClientError::Transport { .. } | LlmClientError::Timeout { .. } => {
- Some(RoutingFallbackReason::Unavailable)
- }
+ LlmClientError::Transport { .. } => Some(RoutingFallbackReason::Unavailable),
LlmClientError::UpstreamHttp { status, .. }
if matches!(
*status,
@@ -967,6 +989,7 @@ mod tests {
extra_body: BTreeMap::from([("store".to_string(), json!(store))]),
reasoning_effort: None,
max_retries: 0,
+ timeout: None,
};
let backend = if responses {
Backend::OpenAiResponses(config)
@@ -1335,6 +1358,7 @@ mod tests {
extra_body: BTreeMap::new(),
reasoning_effort: None,
max_retries: 2,
+ timeout: None,
})
};
let client = Arc::new(
@@ -1432,6 +1456,7 @@ mod tests {
extra_body: BTreeMap::new(),
reasoning_effort: None,
max_retries: 0,
+ timeout: None,
})
};
let client = Arc::new(
diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rs
index 473658281..26757982b 100644
--- a/crates/libsy-llm-client/tests/observability.rs
+++ b/crates/libsy-llm-client/tests/observability.rs
@@ -1380,14 +1380,21 @@ async fn upstream_body_is_redacted_from_the_client_call_span() -> switchyard_lib
judge_model: JUDGE.into(),
outcome: JudgeOutcome::CallFailure,
}) as Arc<dyn RoutedLlmClient>;
- run_classifier(
+ let result = run_classifier(
JUDGE,
"redaction-weak",
"redaction-strong",
client,
classifier_request(),
)
- .await?;
+ .await;
+ assert!(matches!(
+ result,
+ Err(LibsyError::ClientCall {
+ source: LlmClientError::UpstreamHttp { status, body },
+ ..
+ }) if status == http::StatusCode::INTERNAL_SERVER_ERROR && body.contains(LEAKED_CONTENT)
+ ));
let spans = store.spans();
let client_span = find_span(&spans, "libsy.client_call", "selected_model", JUDGE);
@@ -1402,6 +1409,13 @@ async fn upstream_body_is_redacted_from_the_client_call_span() -> switchyard_lib
.unwrap_or("");
assert!(error.contains("upstream HTTP 500"), "{client_span:?}");
assert!(!error.contains(LEAKED_CONTENT), "{client_span:?}");
+ assert!(!spans.iter().any(|span| {
+ span.name == "libsy.client_call"
+ && matches!(
+ span.fields.get("selected_model").map(String::as_str),
+ Some("redaction-weak" | "redaction-strong")
+ )
+ }));
Ok(())
}
@@ -1576,22 +1590,19 @@ async fn classifier_metrics_count_routing_and_answer_calls_once() -> switchyard_
}
#[tokio::test]
-async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::Result<()> {
+async fn classifier_stops_on_client_errors_and_records_verdict_fallback()
+-> switchyard_libsy::Result<()> {
let _guard = serialize_test().lock().await;
- let (_store, exporter, provider, _, _) = telemetry();
+ let (_, exporter, provider, _, _) = telemetry();
let cases = [
- ("fo-call", JudgeOutcome::CallFailure, Some("upstream_5xx")),
+ ("fo-call", JudgeOutcome::CallFailure, None),
(
"fo-parse",
JudgeOutcome::Reply("not json at all"),
Some("parse_error"),
),
- (
- "fo-stream-decode",
- JudgeOutcome::StreamDecodeFailure,
- Some("invalid_response"),
- ),
+ ("fo-stream-decode", JudgeOutcome::StreamDecodeFailure, None),
(
"fo-valid",
JudgeOutcome::Reply(
@@ -1605,15 +1616,26 @@ async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::
let client = Arc::new(JudgeClient {
judge_model: judge_model.into(),
outcome,
- }) as Arc<dyn RoutedLlmClient>;
- run_classifier(
+ });
+ let result = run_classifier(
judge_model,
"fo-weak",
"fo-strong",
- client,
+ client.clone(),
classifier_request(),
)
- .await?;
+ .await;
+ match client.outcome {
+ JudgeOutcome::CallFailure => assert!(result.is_err(), "{judge_model}"),
+ JudgeOutcome::StreamDecodeFailure => assert!(matches!(
+ result,
+ Err(LibsyError::ClientCall {
+ source: LlmClientError::ResponseTranslation { .. },
+ ..
+ })
+ )),
+ JudgeOutcome::Reply(_) => assert_eq!(result?.0.as_str(), "fo-strong"),
+ }
let snapshots = flushed_metrics(exporter, provider);
match expected_reason {
@@ -1633,7 +1655,7 @@ async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::
&[("judge_model", judge_model)],
),
None,
- "a valid verdict was counted as a fail-open"
+ "client failures and valid verdicts must not increment switchyard.classifier_fail_open"
),
}
}
diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rs
index 2d258474f..58a8513ee 100644
--- a/crates/switchyard-runner/src/config.rs
+++ b/crates/switchyard-runner/src/config.rs
@@ -7,6 +7,7 @@ use std::collections::{BTreeMap, HashMap};
use std::fs;
use std::path::Path;
use std::sync::Arc;
+use std::time::Duration;
use libsy::RuntimeModels;
use serde::de::DeserializeOwned;
@@ -558,6 +559,8 @@ struct LlmClientConfig {
extra_headers: BTreeMap<String, String>,
#[serde(default = "default_max_retries")]
max_retries: u32,
+ /// Deadline in milliseconds for all attempts and the complete response. Unset is unbounded.
+ timeout_ms: Option<u64>,
}
#[derive(Debug, Deserialize)]
@@ -634,6 +637,11 @@ fn build_backend(
"llm client {client_name} max_retries must be at most {MAX_CONFIGURED_RETRIES}"
)));
}
+ if config.timeout_ms == Some(0) {
+ return Err(RunnerError::configuration(format!(
+ "llm client {client_name} timeout_ms must be at least 1"
+ )));
+ }
if config.forward_auth && config.api_key_env.is_some() {
return Err(RunnerError::configuration(format!(
"llm client {client_name} cannot set both forward_auth and api_key_env"
@@ -669,6 +677,7 @@ fn build_backend(
extra_body: extra_body.clone(),
reasoning_effort,
max_retries: config.max_retries,
+ timeout: config.timeout_ms.map(Duration::from_millis),
};
let backend = match config.format {
ClientFormat::OpenAiChat => Backend::OpenAiChat(http),
@@ -1087,17 +1096,22 @@ new = ["send_message"]
#[test]
fn rejects_invalid_unreferenced_llm_client() {
- let invalid = format!(
- "{VALID_CONFIG}\n\
- [llm_clients.unused]\n\
- format = \"openai_chat\"\n\
- base_url = \"not a url\"\n"
- );
- let message = error_message(&invalid);
- assert!(
- message.contains("base_url must be an absolute HTTP(S) URL"),
- "unexpected error: {message}"
- );
+ for (settings, expected) in [
+ (
+ "base_url = \"not a url\"",
+ "base_url must be an absolute HTTP(S) URL",
+ ),
+ (
+ "base_url = \"https://example.test\"\ntimeout_ms = 0",
+ "timeout_ms must be at least 1",
+ ),
+ ] {
+ let invalid = format!(
+ "{VALID_CONFIG}\n[llm_clients.unused]\nformat = \"openai_chat\"\n{settings}"
+ );
+ let message = error_message(&invalid);
+ assert!(message.contains(expected), "unexpected error: {message}");
+ }
}
#[test]
@@ -1581,6 +1595,27 @@ target = "azure"
Ok(())
}
+ #[test]
+ fn client_deadline_defaults_and_rejects_zero() -> RunnerResult<()> {
+ for (setting, expected) in [
+ ("", None),
+ ("timeout_ms = 1500", Some(1500)),
+ ("timeout_ms = 0", Some(0)),
+ ] {
+ let source = format!(
+ "format = \"openai_chat\"\nbase_url = \"https://example.test/v1\"\n{setting}"
+ );
+ let config: LlmClientConfig = toml::from_str(&source).expect("valid deadline config");
+ let backend = build_backend("test", &config, &BTreeMap::new(), None);
+ if expected == Some(0) {
+ assert!(backend.is_err());
+ } else {
+ assert_eq!(backend?.timeout(), expected.map(Duration::from_millis));
+ }
+ }
+ Ok(())
+ }
+
#[test]
fn retry_budget_defaults_and_accepts_an_override() -> RunnerResult<()> {
let default: DeploymentConfig = toml::from_str(VALID_CONFIG).map_err(|error| {
diff --git a/crates/switchyard-server/tests/client_deadline.rs b/crates/switchyard-server/tests/client_deadline.rs
new file mode 100644
index 000000000..58bd06fc3
--- /dev/null
+++ b/crates/switchyard-server/tests/client_deadline.rs
@@ -0,0 +1,328 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Run deadline cases in one test because they share process-wide attempt counters.
+
+use std::convert::Infallible;
+use std::error::Error;
+use std::io::Write;
+use std::sync::Arc;
+use std::time::Duration;
+
+use axum::body::{Body, Bytes};
+use axum::extract::State;
+use axum::http::{Request, StatusCode};
+use axum::response::{IntoResponse, Response};
+use axum::routing::post;
+use axum::{Json, Router};
+use http_body_util::BodyExt;
+use serde_json::{Value, json};
+use switchyard_server::{build_switchyard_router, config::load_server_state};
+use tokio::net::TcpListener;
+use tokio::sync::Mutex;
+use tokio::task::JoinHandle;
+use tower::ServiceExt;
+
+type TestResult<T = ()> = Result<T, Box<dyn Error + Send + Sync>>;
+
+// `Upstream::drop` stops the server task even when an assertion fails.
+struct Upstream {
+ url: String,
+ calls: Arc<Mutex<Vec<Value>>>,
+ task: JoinHandle<std::io::Result<()>>,
+}
+
+impl Upstream {
+ async fn start() -> TestResult<Self> {
+ let calls = Arc::new(Mutex::new(Vec::new()));
+ let app = Router::new()
+ .route("/v1/chat/completions", post(upstream))
+ .with_state(Arc::clone(&calls));
+ let listener = TcpListener::bind("127.0.0.1:0").await?;
+ let url = format!("http://{}/v1", listener.local_addr()?);
+ let task = tokio::spawn(async move { axum::serve(listener, app).await });
+ Ok(Self { url, calls, task })
+ }
+}
+
+impl Drop for Upstream {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+async fn upstream(
+ State(calls): State<Arc<Mutex<Vec<Value>>>>,
+ Json(body): Json<Value>,
+) -> Response {
+ let model = body["model"].as_str().unwrap_or_default();
+ let attempt = {
+ let mut calls = calls.lock().await;
+ calls.push(body.clone());
+ calls.iter().filter(|call| call["model"] == model).count()
+ };
+ if model == "backoff" || (model == "retry-body" && attempt == 1) {
+ if model == "retry-body" {
+ tokio::time::sleep(Duration::from_millis(300)).await;
+ }
+ let delay = if model == "backoff" { "1" } else { "0" };
+ return (
+ StatusCode::SERVICE_UNAVAILABLE,
+ [("retry-after", delay)],
+ "retry",
+ )
+ .into_response();
+ }
+ if model == "stalled" {
+ tokio::time::sleep(Duration::from_secs(3)).await;
+ }
+ if body["stream"] == true {
+ let model = model.to_string();
+ let stream = async_stream::stream! {
+ let first = json!({"id":"test", "model":model, "choices":[{"index":0,"delta":{"role":"assistant","content":"PONG"},"finish_reason":null}]});
+ yield Ok::<_, Infallible>(Bytes::from(format!("data: {first}\n\n")));
+ let delay = match model.as_str() { "slow-body" => 3000, "streaming" => 150, _ => 5 };
+ tokio::time::sleep(Duration::from_millis(delay)).await;
+ if model == "stream-error" {
+ yield Ok(Bytes::from_static(b"data: {\"error\":{\"message\":\"upstream stream failed\"}}\n\ndata: [DONE]\n\n"));
+ return;
+ }
+ let last = json!({"id":"test", "model":model, "choices":[{"index":0,"delta":{},"finish_reason":"stop"}]});
+ yield Ok(Bytes::from(format!("data: {last}\n\ndata: [DONE]\n\n")));
+ };
+ return (
+ [("content-type", "text/event-stream")],
+ Body::from_stream(stream),
+ )
+ .into_response();
+ }
+ let content = if model == "retry-body" {
+ r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.9}"#
+ } else {
+ "PONG"
+ };
+ let response = json!({"id":"test","object":"chat.completion","model":model,
+ "choices":[{"index":0,"message":{"role":"assistant","content":content},"finish_reason":"stop"}],
+ "usage":{"prompt_tokens":3,"completion_tokens":1,"total_tokens":4}});
+ if model == "slow-body" || model == "retry-body" {
+ let delay = if model == "retry-body" { 450 } else { 3000 };
+ let stream = async_stream::stream! {
+ yield Ok::<_, Infallible>(Bytes::from_static(b" "));
+ tokio::time::sleep(Duration::from_millis(delay)).await;
+ yield Ok(Bytes::from(response.to_string()));
+ };
+ return (
+ [("content-type", "application/json")],
+ Body::from_stream(stream),
+ )
+ .into_response();
+ }
+ Json(response).into_response()
+}
+
+fn app(
+ upstream: &Upstream,
+ mode: &str,
+ judge: &str,
+ weak: &str,
+ timeout: u64,
+) -> TestResult<Router> {
+ let route = match mode {
+ "escalation" => {
+ "type = \"llm_classifier\"\nmode = \"escalation\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nescalation = { confirmations = 1 }"
+ }
+ "candidates" => {
+ "type = \"random\"\ntargets = [\"weak\", \"strong\"]\nweights = [1000000, 1]\nseed = 17"
+ }
+ "terminal" => "type = \"passthrough\"\ntarget = \"weak\"",
+ _ => {
+ "type = \"llm_classifier\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nbase_threshold = 0.5"
+ }
+ };
+ let mut config = tempfile::Builder::new().suffix(".toml").tempfile()?;
+ write!(
+ config,
+ r#"
+schema_version = 1
+[llm_clients.http]
+format = "openai_chat"
+base_url = "{}"
+max_retries = 1
+timeout_ms = {timeout}
+[targets]
+judge = {{ id = "{judge}", llm_client = "http", extra_body = {{ stream = {judge_stream} }} }}
+weak = {{ id = "{weak}", llm_client = "http" }}
+strong = {{ id = "strong", llm_client = "http" }}
+[routes.test]
+id = "test"
+{route}
+"#,
+ upstream.url,
+ judge_stream = judge == "stream-error",
+ )?;
+ Ok(build_switchyard_router(load_server_state(config.path())?))
+}
+
+async fn send(app: &Router, path: &str, body: Option<Value>) -> TestResult<(StatusCode, String)> {
+ let (method, body) = match body {
+ Some(body) => ("POST", Body::from(serde_json::to_vec(&body)?)),
+ None => ("GET", Body::empty()),
+ };
+ let response = tokio::time::timeout(
+ Duration::from_secs(2),
+ app.clone().oneshot(
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(body)?,
+ ),
+ )
+ .await??;
+ let status = response.status();
+ let bytes = tokio::time::timeout(Duration::from_secs(2), response.into_body().collect())
+ .await??
+ .to_bytes();
+ Ok((status, String::from_utf8(bytes.to_vec())?))
+}
+
+fn metric(text: &str, code: &str) -> f64 {
+ text.lines()
+ .find(|line| {
+ line.starts_with("switchyard_upstream_attempts_total{")
+ && line.contains(&format!("code=\"{code}\""))
+ })
+ .and_then(|line| line.split_whitespace().last())
+ .and_then(|value| value.parse().ok())
+ .unwrap_or_default()
+}
+
+async fn check(
+ upstream: &Upstream,
+ app: &Router,
+ endpoint: &str,
+ expected_calls: &[&str],
+ expected_counts: [f64; 3],
+) -> TestResult<(StatusCode, String)> {
+ upstream.calls.lock().await.clear();
+ let (_, before) = send(app, "/metrics", None).await?;
+ let mut body = if endpoint == "/v1/responses" {
+ json!({"model":"test", "input":"Say PONG", "stream":true})
+ } else {
+ json!({"model":"test", "messages":[{"role":"user","content":"Say PONG"}], "max_tokens":32, "stream":true})
+ };
+ if endpoint == "/v1/decision" {
+ body = json!({"input_format":"openai_chat", "request":body});
+ }
+ let result = send(app, endpoint, Some(body)).await?;
+ let calls = upstream.calls.lock().await.clone();
+ let actual: Vec<_> = calls
+ .iter()
+ .filter_map(|call| call["model"].as_str())
+ .collect();
+ assert_eq!(actual, expected_calls, "{endpoint}: {}", result.1);
+ let (_, after) = send(app, "/metrics", None).await?;
+ // Count HTTP 200, HTTP 503, and pre-response failures separately. A stream
+ // that times out after HTTP 200 must not count as another attempt.
+ let counts = ["200", "503", "none"].map(|code| metric(&after, code) - metric(&before, code));
+ assert_eq!(counts, expected_counts, "{endpoint}: {}", result.1);
+ Ok(result)
+}
+
+#[tokio::test]
+async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult {
+ let upstream = Upstream::start().await?;
+ let stalled = app(&upstream, "classifier", "stalled", "weak", 100)?;
+ let streaming = app(&upstream, "terminal", "judge", "streaming", 500)?;
+ let slow_stream = app(&upstream, "terminal", "judge", "slow-body", 100)?;
+ for endpoint in [
+ "/v1/chat/completions",
+ "/v1/messages",
+ "/v1/responses",
+ "/v1/decision",
+ ] {
+ let (status, body) =
+ check(&upstream, &stalled, endpoint, &["stalled"], [0., 0., 1.]).await?;
+ assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{body}");
+ assert!(body.contains("100 ms"), "{body}");
+ if endpoint == "/v1/decision" {
+ continue;
+ }
+ let (status, body) = check(
+ &upstream,
+ &streaming,
+ endpoint,
+ &["streaming"],
+ [1., 0., 0.],
+ )
+ .await?;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert!(body.contains("PONG"), "{body}");
+ let success = match endpoint {
+ "/v1/messages" => "message_stop",
+ "/v1/responses" => "response.completed",
+ _ => "[DONE]",
+ };
+ assert!(body.contains(success), "{body}");
+ let (status, body) = check(
+ &upstream,
+ &slow_stream,
+ endpoint,
+ &["slow-body"],
+ [1., 0., 0.],
+ )
+ .await?;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert!(body.contains("PONG") && body.contains("100 ms"), "{body}");
+ assert!(!body.contains(success), "{body}");
+ assert!(!body.contains("\"finish_reason\":\"stop\""), "{body}");
+ }
+
+ // These response and retry failures use the same client code for every endpoint.
+ for (judge, expected_calls, counts) in [
+ ("slow-body", &["slow-body"][..], [0., 0., 1.]),
+ ("backoff", &["backoff"][..], [0., 1., 0.]),
+ // The first attempt waits 300 ms and the retry body waits 450 ms.
+ // Each fits the 600 ms deadline alone; together they exceed it.
+ (
+ "retry-body",
+ &["retry-body", "retry-body"][..],
+ [0., 1., 1.],
+ ),
+ ] {
+ let app = app(&upstream, "classifier", judge, "weak", 600)?;
+ let (status, body) = check(
+ &upstream,
+ &app,
+ "/v1/chat/completions",
+ expected_calls,
+ counts,
+ )
+ .await?;
+ assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{judge}: {body}");
+ assert!(body.contains("600 ms"), "{body}");
+ }
+ let failed_stream = app(&upstream, "classifier", "stream-error", "weak", 100)?;
+ let (status, body) = check(
+ &upstream,
+ &failed_stream,
+ "/v1/chat/completions",
+ &["stream-error"],
+ [1., 0., 0.],
+ )
+ .await?;
+ assert_eq!(status, StatusCode::BAD_GATEWAY, "{body}");
+ for (mode, weak, counts) in [
+ ("escalation", "slow-body", [1., 0., 0.]),
+ ("candidates", "stalled", [0., 0., 1.]),
+ ] {
+ let app = app(&upstream, mode, "judge", weak, 100)?;
+ let (status, body) =
+ check(&upstream, &app, "/v1/chat/completions", &[weak], counts).await?;
+ assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{mode}: {body}");
+ if mode == "escalation" {
+ assert_eq!(upstream.calls.lock().await[0]["stream"], true);
+ }
+ }
+ Ok(())
+}
diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs
index 00295e7d9..98ea8387a 100644
--- a/crates/switchyard-server/tests/server.rs
+++ b/crates/switchyard-server/tests/server.rs
@@ -730,6 +730,7 @@ fn random_state_with_retries(
extra_body: BTreeMap::new(),
reasoning_effort: None,
max_retries,
+ timeout: None,
});
let target_models = routes
.iter()
@@ -4635,7 +4636,7 @@ fn gate_count(stats: &Value, path: &[&str]) -> u64 {
// process-global, so this is the only test that emits redo / consult-failure
// metrics and the only one that may assert their exact counts.
#[tokio::test]
-async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {
+async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {
let upstream = MockUpstream::start().await?;
let app = build_switchyard_router(advisor_state_no_retry(&upstream.base_url)?);
let before = send(&app, "GET", "/v1/stats", None).await?.json()?;
@@ -4673,7 +4674,7 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {
assert!(feedback.starts_with("A senior reviewer examined your work"));
assert!(feedback.ends_with("run the tests"));
- // Fail-open: the advisor 503s once (no retries) and the turn still flows.
+ // A failed HTTP advisor call stops the request before the algorithm can approve it.
let response = send_with_headers(
&app,
"POST",
@@ -4682,8 +4683,8 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {
&[("proxy_x_session_id", "fail-flow")],
)
.await?;
- assert_eq!(response.status, StatusCode::OK);
- assert_eq!(response.json()?["choices"][0]["message"]["content"], "ok");
+ assert_eq!(response.status, StatusCode::SERVICE_UNAVAILABLE);
+ assert_eq!(response.json()?["error"]["type"], "upstream_error");
assert_eq!(
upstream.models().await,
[
@@ -4696,9 +4697,8 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {
);
let stats = send(&app, "GET", "/v1/stats", None).await?.json()?;
- // State-owned accumulator: two client-visible executor answers; the discarded REDO attempt
- // is routing work. The failed advisor consult is counted separately.
- assert_eq!(stats["models"]["model/executor"]["calls"], 2);
+ // Only the successful redo request returns an executor answer.
+ assert_eq!(stats["models"]["model/executor"]["calls"], 1);
assert_eq!(stats["classifier"]["total_errors"], 1);
// Projection deltas for the metrics only this test emits.
let redo = gate_count(&stats, &["reviews", "redo", "total"])
@@ -4725,10 +4725,10 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult {
gate_count(&stats, &["discarded", "tokens", "output"]),
gate_count(&before, &["discarded", "tokens", "output"]) + 2
);
- // The 503 maps to the bounded upstream_5xx reason label.
+ // The host stops before the algorithm records a fail-open advisor decision.
assert_eq!(
gate_count(&stats, &["consult_failures", "upstream_5xx"]),
- gate_count(&before, &["consult_failures", "upstream_5xx"]) + 1
+ gate_count(&before, &["consult_failures", "upstream_5xx"])
);
// Reset re-baselines the projection: the redo/discard counts this test
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
index e595e7cef..d5c78c17e 100644
--- a/docs/reference/toml_schema.md
+++ b/docs/reference/toml_schema.md
@@ -54,10 +54,23 @@ route reaches no upstream. A file without a `[targets]` table is rejected with
| `forward_auth` | No | `false` | Forward the caller's provider credential and application headers. All backends reachable through the route must use the same provider. |
| `extra_headers` | No | `{}` | Custom HTTP headers sent to the model server. Set credentials with `api_key_env` or `forward_auth`; the server rejects headers owned by the selected auth mode. Header names are case-insensitive. |
| `max_retries` | No | `2` | Retry budget, `0`–`10`. |
+| `timeout_ms` | No | unset | Deadline in milliseconds for all attempts, retry delays, and the complete response, including stream reads. Must be at least `1`. Unset leaves the wait unbounded. |
The TOML never contains the secret itself. `api_key_env` names a variable that
must exist and be non-empty when the server loads.
+`timeout_ms` applies separately to every call through the client, including judge
+verdicts and answers. To give a judge a short deadline without limiting the
+answering models, put the judge on its own `[llm_clients]` entry; two entries may
+share a `base_url`. When the deadline expires, the server returns `504` without
+trying another target. If the final answer has already started streaming, the
+server sends a framed error and ends the stream without a success marker.
+
+The Rust runner collects streams used during routing before the algorithm
+continues, preserving provider events for replay. After the configured retries,
+an HTTP client failure stops routing. This also applies when `timeout_ms` is
+unset or an advisor has `fail_open = true`.
+
Set `forward_auth = true` to use each caller's credential instead of a
server-owned key:
@@ -222,8 +235,9 @@ Existing configurations that contain `escalation` but omit `mode` remain valid.
Custom mode validates the judge's JSON against `response_schema`, resolves the
policy selector, and routes to a runtime model group. A verdict names a group and
-the first model in it serves the turn; if that call fails the client falls through
-the rest of that group, then through whatever `models.any` adds.
+the first model in it serves the turn. An eligible non-timeout failure tries the
+rest of that group, then any remaining models in `models.any`. A timeout stops
+the request without trying another model.
The `[routes.<name>.models]` table takes any group name you choose. `any` and
`judge` are reserved and required; `capable` and `efficient` are reserved for the
@@ -232,12 +246,12 @@ how one route chooses between more than two models.
| Key | Required | Default | Meaning |
|---|:---:|---|---|
-| `models.any` | Yes | — | Every selectable completion target, in last-resort fallback order. Every other group's targets must also appear here; one that does not is rejected at configuration load. |
-| `models.judge` | Yes | — | One or more ordered judge candidates. Not a completion destination. |
-| `models.capable` | No | — | Ordered capable-tier models. A `capable` verdict selects the first and falls through the rest in order. |
-| `models.efficient` | No | — | Ordered efficient-tier models. An `efficient` verdict selects the first and falls through the rest in order. |
-| `models.<your name>` | No | — | A group you name. A verdict naming it selects its first model and falls through the rest in order. |
-| `default_target` | Yes | — | Group used when the judge fails or its verdict cannot be routed. Any group except `judge`, and it must contain at least one target. |
+| `models.any` | Yes | — | Every selectable completion target, in fallback order for eligible non-timeout failures. Every other group's targets must also appear here; one that does not is rejected at configuration load. |
+| `models.judge` | Yes | — | One or more ordered judge candidates. The Rust runner calls the first and stops on a client error after retries. Not a completion destination. |
+| `models.capable` | No | — | Ordered capable-tier models. A `capable` verdict selects the first; eligible non-timeout failures try the rest in order. |
+| `models.efficient` | No | — | Ordered efficient-tier models. An `efficient` verdict selects the first; eligible non-timeout failures try the rest in order. |
+| `models.<your name>` | No | — | A group you name. A verdict naming it selects its first model; eligible non-timeout failures try the rest in order. |
+| `default_target` | Yes | — | Group used when the judge's verdict cannot be parsed or routed. HTTP client failures stop the request after retries. Any group except `judge`, and it must contain at least one target. |
| `prompt` | Yes | — | Judge system prompt. The configured inner schema is sent separately as structured-output configuration. |
| `response_schema` | Yes | — | Inner JSON Schema encoded as a TOML string. Switchyard adds the provider wrapper. |
| `policy` | Yes | — | Policy table. `target_selector` accepts a JSON Pointer such as `/decision/target`. |
diff --git a/docs/routing_algorithms/advisor_gate_routing.md b/docs/routing_algorithms/advisor_gate_routing.md
index 610530cb9..5af0077b3 100644
--- a/docs/routing_algorithms/advisor_gate_routing.md
+++ b/docs/routing_algorithms/advisor_gate_routing.md
@@ -61,15 +61,20 @@ stamp every request of one evaluation with it, sub-agents included, so the
budget means "reviews for this task" even behind a gateway shared by many
tasks — then the session id resolved from harness headers (such as
`x-switchyard-session-id`), and one scope per server when neither is present.
-Failed consults refund the budget and count toward a separate cap of 3, which
-bounds consult latency against a down advisor. An unparseable verdict also
-refunds and passes the turn through as APPROVE.
+For hosts that return client errors to the core algorithm, a failed consult refunds
+the review budget and counts toward a separate cap of 3. The Rust runner stops on
+HTTP client errors before returning them to the algorithm, so failed HTTP calls
+do not refund this budget. An unparseable verdict still refunds the budget and
+passes the turn through as APPROVE.
Long sessions are truncated middle-out before the consult: the transcript keeps
the task statement at the start and the most recent work at the end, marked
`...<middle of the conversation truncated>...`, capped at
-`transcript_max_chars`. With `fail_open = true` (default) any advisor failure
-degrades to APPROVE; `fail_open = false` surfaces it as a server error instead.
+`transcript_max_chars`. `fail_open` controls failures returned to the core
+algorithm by its host. The Rust runner stops the request after an HTTP client's
+configured retries fail or its deadline expires, even with `fail_open = true`.
+HTTP client errors also stop the request when `timeout_ms` is unset. A timeout
+before the answer starts returns `504`.
Gate behavior is observable at `/v1/stats` under `advisor_gate`: verdicts by
trigger, consult failures by reason, and REDO-discarded turns with their token
@@ -109,7 +114,7 @@ gate_min_tool_results = 3
| `advisor_max_tokens` | `2048` | Output cap for each advisor consult. |
| `advisor_temperature` | unset | Sampling temperature for consults; omitted when unset. |
| `transcript_max_chars` | `200000` | Middle-out cap on the serialized transcript (~50k tokens). |
-| `fail_open` | `true` | Advisor failure passes the turn through instead of erroring. |
+| `fail_open` | `true` | Failures returned to the core algorithm pass the turn through. The Rust runner stops HTTP client failures regardless of this setting. |
| `reviewer_system_prompt` | built-in | Overrides the APPROVE/REDO reviewer contract. |
| `redo_feedback_prefix` | built-in | Overrides the prefix injected before a REDO plan. |
diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md
index 1ed9e8085..3fecddab6 100644
--- a/docs/routing_algorithms/escalation_router_routing.md
+++ b/docs/routing_algorithms/escalation_router_routing.md
@@ -86,9 +86,12 @@ flowchart LR
class t,p,s,c,j,w,l box;
```
-A judge that times out, errors, or returns an unparseable verdict fails open: the
-turn serves the buffered weak reply and the existing streak is held rather than
-cleared. A judge failure never creates a strong-tier latch.
+An unparseable verdict serves the buffered weak reply and holds the existing
+streak. The Rust runner stops the request if an HTTP model call fails after
+retries. To bound both the weak-model response and the judge response, set
+`timeout_ms` on each `[llm_clients]` entry they use. The deadline applies separately
+to each call, even when both models share a client. Expiry returns `504` without
+calling another target or selecting the strong tier for subsequent session turns.
## Judge model compatibility
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index 329ccadc0..c1b3a3fbe 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -67,9 +67,19 @@ greater than or equal to the applicable threshold. Otherwise it routes to
- `uncertain` and `unmatched` use `base_threshold + threshold_step`.
- `unsupported` uses `base_threshold + 2 * threshold_step`.
-An invalid, inconsistent, or unparseable verdict, or a judge failure, routes to
+An invalid, inconsistent, or unparseable verdict routes to
`strong_target`. Raising either knob sends more traffic to the strong model.
+To stop waiting for a judge that accepts the request but never finishes its
+response, set `timeout_ms` on the judge's `[llm_clients]` entry
+(see the [TOML schema](../reference/toml_schema.md)); it covers the judge's
+retries and the complete verdict body. When the deadline expires, the Rust server returns
+`504` without calling `strong_target` or `weak_target`. Other HTTP client failures
+also stop routing after retries. The deadline applies to every call
+through that client. Give the judge its own entry if the answering models need a
+different deadline, even when they use the same provider. Without a deadline,
+the request can wait indefinitely for the judge.
+
## Judge model compatibility
The judge must return complete, schema-valid JSON in normal assistant `content`.