Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(libsy): let routing judges fail open on client errors and deadlines (#888)
Graham King8a2195279d93
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.mdindex 8b0ef430d..9f028be2b 100644--- a/crates/libsy-llm-client/README.md+++ b/crates/libsy-llm-client/README.md@@ -266,9 +266,11 @@ fn build_multi_format_client(`run` and `decide` collect streams used during routing, including answers that analgorithm must inspect, before returning them to the algorithm. They retain theprovider events for replay. After the configured retries, a client failure stops-`run` or `decide` before the algorithm can choose another routing candidate. This-also applies when `timeout` is `None` or an advisor has `fail_open = true`.-`libsy` and custom hosts that drive it directly are unchanged.+`run` or `decide` by default. Calls with `CallModel::recover_errors` enabled return+the error to the algorithm so it can apply its fallback policy. Capability+classifiers and advisor consults default to `fail_open = true`. Set `fail_open = false`+to stop the request on their client failures. Recovery includes client errors and+deadlines during stream collection. Custom hosts can honor the same flag when serving calls.Retries replay the same upstream request to the same model. After routing completes,non-timeout failures may try another completion candidate. A timeout stops the call.diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rsindex 691e169cf..1ff16bc3a 100644--- a/crates/libsy-llm-client/src/run.rs+++ b/crates/libsy-llm-client/src/run.rs@@ -11,7 +11,8 @@//! libsy owns the stream mechanics; what this module adds is per-target request preparation,//! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each//! completion candidate exhausts its backend retry budget before fallback advances. Routing-//! calls stop on the first candidate's failure. A timeout stops either kind of call.+//! calls stop on the first candidate's failure unless the algorithm enables error recovery.+//! A timeout stops completion calls.//!//! A Responses continuation can refer to state through `previous_response_id` or `conversation`.//! [`ClientRouter`] sends native Responses state back to its provider; for Chat or Anthropic@@ -49,7 +50,7 @@ use crate::{metrics, observability};/// a per-call lookup, not one client for the whole run. Use/// [`ClientRouter::single`](ClientRouter::single) when one client serves every target.///-/// Routing calls are buffered; a client failure stops the request before the algorithm continues.+/// Routing calls are buffered; client failures stop the request unless the call enables recovery./// Once routing completes, non-timeout failures may try the outcome's ordered fallback candidates.pub async fn run(algorithm: Arc<dyn Algorithm>,@@ -172,7 +173,7 @@ fn emit_routing_observations(/// Serve one offloaded call and fulfill its promise.///-/// A client failure stops the driver before the algorithm can issue another call.+/// A client failure stops the driver unless this call opts into algorithm-level recovery.async fn serve(clients: ClientRouter,call: CallModel,@@ -198,6 +199,7 @@ async fn serve(.await{Ok(response) => call.respond(Ok(response)),+ Err(error) if call.recover_errors => call.respond(Err(error)),Err(error) => call.fail(error),}}diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rsindex 93c12adb5..30adc407c 100644--- a/crates/libsy-llm-client/tests/observability.rs+++ b/crates/libsy-llm-client/tests/observability.rs@@ -607,6 +607,7 @@ fn classifier_router() -> switchyard_libsy::Result<Arc<dyn Algorithm>> {LlmClassifierConfig::Capability {config: TaskClassifierConfig {base_threshold: 0.5,+ fail_open: false,..TaskClassifierConfig::default()},},@@ -1855,6 +1856,101 @@ async fn classifier_stops_on_client_errors_and_records_verdict_fallback()Ok(())}+#[tokio::test]+async fn classifier_can_fail_open_on_judge_client_errors() -> switchyard_libsy::Result<()> {+ let _guard = serialize_test().lock().await;+ assert!(TaskClassifierConfig::default().fail_open);+ for (judge, reason) in [+ (+ Arc::new(TimeoutClient) as Arc<dyn RoutedLlmClient>,+ "timeout",+ ),+ (+ Arc::new(JudgeClient {+ judge_model: "judge".into(),+ outcome: JudgeOutcome::CallFailure,+ }),+ "upstream_5xx",+ ),+ (+ Arc::new(JudgeClient {+ judge_model: "judge".into(),+ outcome: JudgeOutcome::StreamDecodeFailure,+ }),+ "invalid_response",+ ),+ ] {+ let algorithm = |fail_open: Option<bool>| -> switchyard_libsy::Result<Arc<dyn Algorithm>> {+ let mut config = serde_json::json!({"base_threshold": 0.5});+ if let Some(fail_open) = fail_open {+ config["fail_open"] = fail_open.into();+ }+ Ok(Arc::new(LlmTaskClassifier::new(+ LlmClassifierConfig::Capability {+ config: serde_json::from_value(config).unwrap(),+ },+ )?))+ };+ let clients = ClientRouter::new(std::collections::HashMap::from([+ (ModelId::from("judge"), judge),+ (+ ModelId::from("capable"),+ Arc::new(JudgeClient {+ judge_model: "judge".into(),+ outcome: JudgeOutcome::CallFailure,+ }) as Arc<dyn RoutedLlmClient>,+ ),+ ]));+ let models = classifier_models("judge", "efficient", "capable");+ let stopped = switchyard_llm_client::decide(+ algorithm(Some(false))?,+ clients.clone(),+ classifier_request(),+ models.clone(),+ )+ .await;+ assert!(+ matches!(stopped, Err(LibsyError::ClientCall { .. })),+ "{reason}"+ );++ let outcome = switchyard_llm_client::decide(+ algorithm(None)?,+ clients.clone(),+ classifier_request(),+ models.clone(),+ )+ .await?;+ assert_eq!(outcome.selected_model_id()?.as_str(), "capable");+ assert_eq!(+ outcome.metadata.and_then(|metadata| metadata.evidence),+ Some(serde_json::json!({"source": "fail_open", "reason_code": reason})),+ );+ let (selected, _) = switchyard_llm_client::run(+ algorithm(Some(true))?,+ clients,+ classifier_request(),+ models.clone(),+ None,+ )+ .await?;+ assert_eq!(selected.as_str(), "capable");++ let answer_failure = switchyard_llm_client::run(+ algorithm(None)?,+ ClientRouter::single(Arc::new(TimeoutClient)),+ classifier_request(),+ models,+ None,+ )+ .await;+ assert!(matches!(answer_failure, Err(LibsyError::ClientCall {+ target, source: LlmClientError::Timeout { .. },+ }) if target.as_str() == "capable"));+ }+ Ok(())+}+#[tokio::test]async fn in_flight_gauge_reads_a_run_parked_on_an_unanswered_routing_call()-> switchyard_libsy::Result<()> {diff --git a/crates/libsy/src/algorithms/advisor_gate.rs b/crates/libsy/src/algorithms/advisor_gate.rsindex 4551ac703..9f0c35ef2 100644--- a/crates/libsy/src/algorithms/advisor_gate.rs+++ b/crates/libsy/src/algorithms/advisor_gate.rs@@ -408,7 +408,14 @@ impl AdvisorGate {Ok(advisor) => {let advisor = advisor.clone();let advisor_models = driver.models_for(&Category::Judge).to_vec();- match driver.call_model(consult_request, advisor_models).await {+ match driver+ .call_model_with_error_recovery(+ consult_request,+ advisor_models,+ self.config.fail_open,+ )+ .await+ {Ok(response) => {let served_advisor = response.served_model()diff --git a/crates/libsy/src/algorithms/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rsindex 488bc6585..40d9ad3d9 100644--- a/crates/libsy/src/algorithms/llm_class.rs+++ b/crates/libsy/src/algorithms/llm_class.rs@@ -300,6 +300,8 @@ fn capability_evidence(#[derive(Clone, Debug)]/// Settings that control capability classifier prompting and routing.pub struct TaskClassifierConfig {+ /// Routes to the capable tier on judge client failures and deadlines. Defaults to true.+ pub fail_open: bool,/// Lowest solve probability that routes a supported task to the efficient target.pub base_threshold: f64,/// Amount added per capability-boundary step.@@ -329,6 +331,8 @@ pub struct TaskClassifierConfig {#[serde(deny_unknown_fields)]struct TaskClassifierConfigWire {base_threshold: f64,+ #[serde(default = "default_fail_open")]+ fail_open: bool,#[serde(default)]threshold_step: f64,#[serde(default)]@@ -358,6 +362,7 @@ impl<'de> Deserialize<'de> for TaskClassifierConfig {contract = contract.with_response_format_type(wire.response_format_type);Ok(Self {base_threshold: wire.base_threshold,+ fail_open: wire.fail_open,threshold_step: wire.threshold_step,classify_trigger: wire.classify_trigger,message_hash_fallback: wire.message_hash_fallback,@@ -368,6 +373,10 @@ impl<'de> Deserialize<'de> for TaskClassifierConfig {}}+const fn default_fail_open() -> bool {+ true+}+const fn default_judge_max_output_tokens() -> u64 {DEFAULT_JUDGE_MAX_OUTPUT_TOKENS}@@ -376,6 +385,7 @@ impl Default for TaskClassifierConfig {fn default() -> Self {Self {base_threshold: 0.0,+ fail_open: default_fail_open(),threshold_step: 0.0,classify_trigger: ClassifyTrigger::default(),message_hash_fallback: false,@@ -644,6 +654,7 @@ impl LlmTaskClassifier {),TaskClassifierPolicy::new(&config),)+ .with_error_recovery(config.fail_open).with_evidence(capability_evidence),);Self::from_classifier(diff --git a/crates/libsy/src/algorithms/util/llm_judge.rs b/crates/libsy/src/algorithms/util/llm_judge.rsindex cb8277dae..d3fbe34e0 100644--- a/crates/libsy/src/algorithms/util/llm_judge.rs+++ b/crates/libsy/src/algorithms/util/llm_judge.rs@@ -216,6 +216,7 @@ where{judge: J,policy: P,+ recover_errors: bool,evidence: Option<EvidenceFn<J::Verdict, P>>,}@@ -229,10 +230,17 @@ whereSelf {judge,policy,+ recover_errors: false,evidence: None,}}+ /// Lets the judge policy recover from client failures, including deadlines.+ pub(crate) fn with_error_recovery(mut self, enabled: bool) -> Self {+ self.recover_errors = enabled;+ self+ }+/// Enables bounded evidence for built-in judges without widening the public policy trait.pub(crate) fn with_evidence(mut self, evidence: EvidenceFn<J::Verdict, P>) -> Self {self.evidence = Some(evidence);@@ -256,11 +264,9 @@ where/// Consults the judge, yielding `None` when it is unavailable or unintelligible.///- /// A judge is an optimization, not a dependency: failing the caller's request because the- /// judge is down would be worse than routing without it, so every failure — transport,- /// mid-stream, or unparseable reply — is logged and folded into `None` for the policy's- /// fallback branch. A closed driver stream is folded too; the algorithm's next driver- /// call surfaces it, so nothing is masked.+ /// Errors delivered by the host and invalid verdicts are logged and folded into `None`+ /// for the policy's fallback branch. The HTTP driver delivers client errors here when+ /// error recovery is enabled; otherwise it stops the run.pub(crate) async fn verdict(&self,state: &mut State,@@ -272,9 +278,10 @@ wheretracing::info!(target = judge_model, "consulting llm judge");let response = driver- .call_model(+ .call_model_with_error_recovery(self.judge.build_request(state, request),judge_models.to_vec(),+ self.recover_errors,).await.inspect_err(|error| {diff --git a/crates/libsy/src/core/algorithm.rs b/crates/libsy/src/core/algorithm.rsindex 959621db3..5c3ebda92 100644--- a/crates/libsy/src/core/algorithm.rs+++ b/crates/libsy/src/core/algorithm.rs@@ -109,6 +109,8 @@ pub struct CallModel {pub request: Request,/// Candidate models, tried in order until one answers. Never empty.pub models: Vec<ModelId>,+ /// Return client errors to the algorithm so it can apply its fallback policy.+ pub recover_errors: bool,/// How to send the response back to the algorithm. `None` once the call is recorded.reply: Option<oneshot::Sender<Result<Response>>>,started: Instant,@@ -275,6 +277,12 @@ impl Driver {/// through [`CallModel::respond`] or [`CallModel::fail`]; outcome and token usage/// are recorded on the span when the promise resolves. The provider call itself is the/// host's, and is instrumented by whoever makes it.+ pub async fn call_model(&self, request: Request, models: Vec<ModelId>) -> Result<Response> {+ self.call_model_with_error_recovery(request, models, false)+ .await+ }++ /// Allows a routing policy to handle a failed call when recovery is enabled.#[tracing::instrument(target = "libsy",name = "libsy.llm_call",@@ -290,7 +298,12 @@ impl Driver {reasoning_tokens = tracing::field::Empty,))]- pub async fn call_model(&self, mut request: Request, models: Vec<ModelId>) -> Result<Response> {+ pub(crate) async fn call_model_with_error_recovery(+ &self,+ mut request: Request,+ models: Vec<ModelId>,+ recover_errors: bool,+ ) -> Result<Response> {let Some(selected_model_id) = models.first() else {return Err(LibsyError::NoTargets);};@@ -301,6 +314,7 @@ impl Driver {algorithm: self.algorithm.clone(),request,models,+ recover_errors,reply: Some(reply),started,};diff --git a/crates/switchyard-py/src/libsy_bindings.rs b/crates/switchyard-py/src/libsy_bindings.rsindex 23c90d7ad..e0d86d365 100644--- a/crates/switchyard-py/src/libsy_bindings.rs+++ b/crates/switchyard-py/src/libsy_bindings.rs@@ -328,6 +328,7 @@ impl PyTaskClassifierConfig {) -> PyResult<Self> {Ok(Self {inner: TaskClassifierConfig {+ fail_open: true,base_threshold,threshold_step,classify_trigger: classify_trigger(session_affinity),diff --git a/crates/switchyard-runner/src/algorithm.rs b/crates/switchyard-runner/src/algorithm.rsindex e742d48ba..acb2644c8 100644--- a/crates/switchyard-runner/src/algorithm.rs+++ b/crates/switchyard-runner/src/algorithm.rs@@ -99,6 +99,7 @@ enum LlmClassifierModeConfig {#[derive(Clone, Debug)]struct CapabilityClassifierRouteConfig {classifier_target: String,+ fail_open: bool,strong_target: String,weak_target: String,base_threshold: f64,@@ -237,6 +238,9 @@ pub struct LlmClassifierRouteConfig {/// Capability mode: how much to raise the threshold when the judge is/// uncertain. Added once for an uncertain verdict and twice for unsupported.pub threshold_step: Option<f64>,+ /// Capability mode: routes to the capable tier on judge client failures and deadlines.+ /// Defaults to true when capability mode is selected.+ pub fail_open: Option<bool>,/// How often the judge runs: every request, once per user turn, or once per session.pub classify_trigger: ClassifyTrigger,/// Reuses the session's target by hashing the first user message when no@@ -522,6 +526,7 @@ pub struct StageTierConfig {impl StageClassifierConfig {fn task_classifier_config(&self) -> TaskClassifierConfig {TaskClassifierConfig {+ fail_open: true,base_threshold: self.base_threshold,threshold_step: self.threshold_step,classify_trigger: self.classify_trigger,@@ -878,6 +883,7 @@ impl LlmClassifierRouteConfig {weak_target,base_threshold,threshold_step,+ fail_open,classify_trigger,message_hash_fallback,recent_turn_window,@@ -897,6 +903,12 @@ impl LlmClassifierRouteConfig {(None, false) => ClassifierMode::Capability,};+ if !matches!(selected_mode, ClassifierMode::Capability) && *fail_open == Some(true) {+ return Err(AlgorithmConfigError::new(format!(+ "llm_classifier route {route_name}: fail_open = true requires capability mode"+ )));+ }+match selected_mode {ClassifierMode::Capability => {if escalation.is_some() {@@ -917,6 +929,7 @@ impl LlmClassifierRouteConfig {Ok(LlmClassifierModeConfig::Capability(CapabilityClassifierRouteConfig {classifier_target: classifier_target.clone(),+ fail_open: fail_open.unwrap_or(true),strong_target: required_classifier_field(route_name,"strong_target",@@ -1229,6 +1242,7 @@ fn build_algorithm(let algorithm = match mode {LlmClassifierModeConfig::Capability(config) => {let classifier_config = TaskClassifierConfig {+ fail_open: config.fail_open,base_threshold: config.base_threshold,threshold_step: config.threshold_step,classify_trigger: config.classify_trigger,diff --git a/crates/switchyard-server/tests/client_deadline.rs b/crates/switchyard-server/tests/client_deadline.rsindex 58bd06fc3..395f07d8a 100644--- a/crates/switchyard-server/tests/client_deadline.rs+++ b/crates/switchyard-server/tests/client_deadline.rs@@ -139,6 +139,11 @@ fn app("type = \"llm_classifier\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nbase_threshold = 0.5"}};+ let route = if mode == "classifier" {+ format!("{route}\nfail_open = false")+ } else {+ route.to_owned()+ };let mut config = tempfile::Builder::new().suffix(".toml").tempfile()?;write!(config,@@ -233,6 +238,7 @@ async fn check(async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult {let upstream = Upstream::start().await?;let stalled = app(&upstream, "classifier", "stalled", "weak", 100)?;+ let fail_open = app(&upstream, "classifier-default", "stalled", "weak", 100)?;let streaming = app(&upstream, "terminal", "judge", "streaming", 500)?;let slow_stream = app(&upstream, "terminal", "judge", "slow-body", 100)?;for endpoint in [@@ -245,6 +251,13 @@ async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult {check(&upstream, &stalled, endpoint, &["stalled"], [0., 0., 1.]).await?;assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{body}");assert!(body.contains("100 ms"), "{body}");+ let (calls, counts) = if endpoint == "/v1/decision" {+ (&["stalled"][..], [0., 0., 1.])+ } else {+ (&["stalled", "strong"][..], [1., 0., 1.])+ };+ let (status, body) = check(&upstream, &fail_open, endpoint, calls, counts).await?;+ assert_eq!(status, StatusCode::OK, "{body}");if endpoint == "/v1/decision" {continue;}diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rsindex 3c2656257..5236ff559 100644--- a/crates/switchyard-server/tests/server.rs+++ b/crates/switchyard-server/tests/server.rs@@ -4977,7 +4977,7 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {assert!(feedback.starts_with("A senior reviewer examined your work"));assert!(feedback.ends_with("run the tests"));- // A failed HTTP advisor call stops the request before the algorithm can approve it.+ // Fail-open returns the buffered executor turn when the HTTP advisor call fails.let response = send_with_headers(&app,"POST",@@ -4986,8 +4986,8 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {&[("proxy_x_session_id", "fail-flow")],).await?;- assert_eq!(response.status, StatusCode::SERVICE_UNAVAILABLE);- assert_eq!(response.json()?["error"]["type"], "upstream_error");+ assert_eq!(response.status, StatusCode::OK);+ assert_eq!(response.json()?["choices"][0]["message"]["content"], "ok");assert_eq!(upstream.models().await,[@@ -5000,8 +5000,8 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {);let stats = send(&app, "GET", "/v1/stats", None).await?.json()?;- // Only the successful redo request returns an executor answer.- assert_eq!(stats["models"]["model/executor"]["calls"], 1);+ // Both requests return an executor answer.+ assert_eq!(stats["models"]["model/executor"]["calls"], 2);assert_eq!(stats["classifier"]["total_errors"], 1);// Projection deltas for the metrics only this test emits.let redo = gate_count(&stats, &["reviews", "redo", "total"])@@ -5028,10 +5028,9 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {gate_count(&stats, &["discarded", "tokens", "output"]),gate_count(&before, &["discarded", "tokens", "output"]) + 2);- // The host stops before the algorithm records a fail-open advisor decision.assert_eq!(gate_count(&stats, &["consult_failures", "upstream_5xx"]),- gate_count(&before, &["consult_failures", "upstream_5xx"])+ gate_count(&before, &["consult_failures", "upstream_5xx"]) + 1);// Reset re-baselines the projection: the redo/discard counts this testdiff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex 4b51a7918..bbd03af38 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -62,14 +62,18 @@ must exist and be non-empty when the server loads.`timeout_ms` applies separately to every call through the client, including judgeverdicts and answers. To give a judge a short deadline without limiting theanswering models, put the judge on its own `[llm_clients]` entry; two entries may-share a `base_url`. When the deadline expires, the server returns `504` without-trying another target. If the final answer has already started streaming, the+share a `base_url`. When an answer deadline expires, the server returns `504`.+If the final answer has already started streaming, theserver sends a framed error and ends the stream without a success marker.The Rust runner collects streams used during routing before the algorithmcontinues, preserving provider events for replay. After the configured retries,-an HTTP client failure stops routing. This also applies when `timeout_ms` is-unset or an advisor has `fail_open = true`.+an HTTP client failure stops routing unless the call enables recovery.+Capability classifiers and advisor gates default to `fail_open = true`.+A capability classifier routes to the capable tier on judge failures, including+deadlines. An advisor returns the buffered executor turn when its consult fails.+Set `fail_open = false` to stop the request on these failures.+Answer calls keep their usual error behavior.Set `forward_auth = true` to use each caller's credential instead of aserver-owned key:@@ -239,6 +243,7 @@ Capability mode classifies before serving. See| `strong_target` | Yes | — | Capable tier. || `weak_target` | Yes | — | Efficient tier. || `base_threshold` | Yes | — | Lowest solve probability that routes to the weak target. In `[0, 1]`. |+| `fail_open` | No | `true` | Capability mode only. When `true`, judge client failures and deadlines route to the capable tier and record fail-open evidence. When `false`, they stop the request. Invalid verdicts use the capable tier with either setting. || `threshold_step` | No | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. || `classify_trigger` | No | `every_request` | When the judge runs. `every_request` judges every request, tool continuations included. `user_turn` judges each new user message and retains that target across intervening tool calls only when requests carry a session ID; without a session ID, it behaves like `every_request`. `new_session` judges once and reuses that target for the session. || `message_hash_fallback` | No | `false` | Retains the target against a hash of the first user message when a request carries no session ID. Requires `classify_trigger = "new_session"` or `"user_turn"`. |