Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(benchmark): preserve terminal failure pressure (#768)
Alex Fournier1c374121fb77
diff --git a/crates/switchyard-soak/examples/mock_server.rs b/crates/switchyard-soak/examples/mock_server.rsindex c92578fd6..2168168a3 100644--- a/crates/switchyard-soak/examples/mock_server.rs+++ b/crates/switchyard-soak/examples/mock_server.rs@@ -72,6 +72,10 @@ fn classifier_content(marker: Option<&str>) -> &'static str {r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":1.0}"#}+fn should_inject_failure(name: &str, attempt: u64) -> bool {+ name == "upstream_500" || attempt <= 2+}+fn completion(model: &str, content: &str) -> Value {json!({"id": "chatcmpl-switchyard-soak",@@ -192,9 +196,11 @@ async fn chat(State(state): State<BackendState>, Json(body): Json<Value>) -> Res*attempt += 1;*attempt};- if attempt <= 2 {+ if should_inject_failure(name, attempt) {+ // Keep retry-exhaustion scenarios fast enough for local benchmarks.return (status,+ [("retry-after", "0")],Json(json!({"error": {"message": format!("injected {name} attempt {attempt}")}}),),@@ -255,7 +261,7 @@ async fn run(args: Args) -> Result<(), String> {mod tests {use serde_json::json;- use super::requested_output_tokens;+ use super::{requested_output_tokens, should_inject_failure};#[test]fn output_tokens_follow_openai_limits_and_stay_bounded() {@@ -266,6 +272,13 @@ mod tests {);assert_eq!(requested_output_tokens(&json!({})), 2);}++ #[test]+ fn failure_pressure_recovers_429_but_not_500() {+ assert!(should_inject_failure("upstream_429", 2));+ assert!(!should_inject_failure("upstream_429", 3));+ assert!(should_inject_failure("upstream_500", 11));+ }}#[tokio::main]diff --git a/crates/switchyard-soak/src/scenarios/failure_pressure.rs b/crates/switchyard-soak/src/scenarios/failure_pressure.rsindex a1606b065..be34126b3 100644--- a/crates/switchyard-soak/src/scenarios/failure_pressure.rs+++ b/crates/switchyard-soak/src/scenarios/failure_pressure.rs@@ -28,7 +28,7 @@ pub fn build(options: ScenarioOptions<'_>) -> Scenario {Scenario::chat("failure-pressure",ScenarioGroup::Resilience,- "Bounded 429, 500, malformed-classifier, and truncated-stream injections.",+ "Transient 429, persistent 500, malformed-classifier, and truncated-stream injections.","Retries recover transient failures; terminal failures remain explicit and bounded.",ErrorExpectation::MIXED,sessions,diff --git a/docs/operations/soak_test.md b/docs/operations/soak_test.mdindex c0ce76cb4..ee384b7ae 100644--- a/docs/operations/soak_test.md+++ b/docs/operations/soak_test.md@@ -34,7 +34,7 @@ The scenario catalog covers these distinct pressure angles:| `stage-transitions` | One growing history across exploration, critical failure, and productive work | Stage-router tier changes and scorer output || `classifier-mix` | Deterministic 80/20 then 50/50 easy/hard requests | Target share, classifier calls/errors, and classifier latency || `context-overflow` | One target rejects a near-window request | Fallback to another eligible target |-| `failure-pressure` | Bounded 429, 500, malformed verdict, and truncated stream | Retry recovery, explicit terminal errors, and connection health |+| `failure-pressure` | Transient 429, persistent 500, malformed verdict, and truncated stream | Retry recovery, explicit terminal errors, and connection health || `client-cancellation` | A client timeout during a delayed response | Teardown and recovery of later traffic |`standard` includes the core and agentic rows. Run `resilience` separately because expected@@ -239,7 +239,7 @@ python3.12 scripts/benchmark_routing_algorithms.py \--profile-runs 1```-`context-overflow` checks target fallback, `failure-pressure` injects bounded 429, 500, malformed+`context-overflow` checks target fallback, `failure-pressure` injects transient 429, persistent 500, malformedclassifier, and truncated-stream cases, and `client-cancellation` uses a one-second client timeoutagainst a delayed response. Their expected error-rate ranges appear in the report's Resiliencesection, and the command fails after writing the report when any row misses its range. The local