Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

fix(benchmark): preserve terminal failure pressure (#768)

Alex Fournier1c374121fb77
diff --git a/crates/switchyard-soak/examples/mock_server.rs b/crates/switchyard-soak/examples/mock_server.rs
index c92578fd6..2168168a3 100644
--- a/crates/switchyard-soak/examples/mock_server.rs
+++ b/crates/switchyard-soak/examples/mock_server.rs
@@ -72,6 +72,10 @@ fn classifier_content(marker: Option<&str>) -> &'static str {
r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":1.0}"#
}
+fn should_inject_failure(name: &str, attempt: u64) -> bool {
+ name == "upstream_500" || attempt <= 2
+}
+
fn completion(model: &str, content: &str) -> Value {
json!({
"id": "chatcmpl-switchyard-soak",
@@ -192,9 +196,11 @@ async fn chat(State(state): State<BackendState>, Json(body): Json<Value>) -> Res
*attempt += 1;
*attempt
};
- if attempt <= 2 {
+ if should_inject_failure(name, attempt) {
+ // Keep retry-exhaustion scenarios fast enough for local benchmarks.
return (
status,
+ [("retry-after", "0")],
Json(
json!({"error": {"message": format!("injected {name} attempt {attempt}")}}),
),
@@ -255,7 +261,7 @@ async fn run(args: Args) -> Result<(), String> {
mod tests {
use serde_json::json;
- use super::requested_output_tokens;
+ use super::{requested_output_tokens, should_inject_failure};
#[test]
fn output_tokens_follow_openai_limits_and_stay_bounded() {
@@ -266,6 +272,13 @@ mod tests {
);
assert_eq!(requested_output_tokens(&json!({})), 2);
}
+
+ #[test]
+ fn failure_pressure_recovers_429_but_not_500() {
+ assert!(should_inject_failure("upstream_429", 2));
+ assert!(!should_inject_failure("upstream_429", 3));
+ assert!(should_inject_failure("upstream_500", 11));
+ }
}
#[tokio::main]
diff --git a/crates/switchyard-soak/src/scenarios/failure_pressure.rs b/crates/switchyard-soak/src/scenarios/failure_pressure.rs
index a1606b065..be34126b3 100644
--- a/crates/switchyard-soak/src/scenarios/failure_pressure.rs
+++ b/crates/switchyard-soak/src/scenarios/failure_pressure.rs
@@ -28,7 +28,7 @@ pub fn build(options: ScenarioOptions<'_>) -> Scenario {
Scenario::chat(
"failure-pressure",
ScenarioGroup::Resilience,
- "Bounded 429, 500, malformed-classifier, and truncated-stream injections.",
+ "Transient 429, persistent 500, malformed-classifier, and truncated-stream injections.",
"Retries recover transient failures; terminal failures remain explicit and bounded.",
ErrorExpectation::MIXED,
sessions,
diff --git a/docs/operations/soak_test.md b/docs/operations/soak_test.md
index c0ce76cb4..ee384b7ae 100644
--- a/docs/operations/soak_test.md
+++ b/docs/operations/soak_test.md
@@ -34,7 +34,7 @@ The scenario catalog covers these distinct pressure angles:
| `stage-transitions` | One growing history across exploration, critical failure, and productive work | Stage-router tier changes and scorer output |
| `classifier-mix` | Deterministic 80/20 then 50/50 easy/hard requests | Target share, classifier calls/errors, and classifier latency |
| `context-overflow` | One target rejects a near-window request | Fallback to another eligible target |
-| `failure-pressure` | Bounded 429, 500, malformed verdict, and truncated stream | Retry recovery, explicit terminal errors, and connection health |
+| `failure-pressure` | Transient 429, persistent 500, malformed verdict, and truncated stream | Retry recovery, explicit terminal errors, and connection health |
| `client-cancellation` | A client timeout during a delayed response | Teardown and recovery of later traffic |
`standard` includes the core and agentic rows. Run `resilience` separately because expected
@@ -239,7 +239,7 @@ python3.12 scripts/benchmark_routing_algorithms.py \
--profile-runs 1
```
-`context-overflow` checks target fallback, `failure-pressure` injects bounded 429, 500, malformed
+`context-overflow` checks target fallback, `failure-pressure` injects transient 429, persistent 500, malformed
classifier, and truncated-stream cases, and `client-cancellation` uses a one-second client timeout
against a delayed response. Their expected error-rate ranges appear in the report's Resilience
section, and the command fails after writing the report when any row misses its range. The local