Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(benchmarks): hide DeepSWE results pending validation (#1054)
nachiketb-nvidia9537e07259cd
diff --git a/README.md b/README.mdindex d4e94b441..519c27ca5 100644--- a/README.md+++ b/README.md@@ -101,7 +101,7 @@ The [routing overview](docs/routing_algorithms/overview.md) retains the full cat## Evaluation and reference-[Compare benchmark results and configs](docs/benchmarks.md) for Terminal-Bench 2.1 and DeepSWE.+[Compare benchmark results and configs](docs/benchmarks.md) for Terminal-Bench 2.1.For latency and routing overhead testing, see [Soak Testing](docs/operations/soak_test.md).### Further readingdiff --git a/docs/benchmarks.md b/docs/benchmarks.mdindex 4a63d0e9d..50c772bc6 100644--- a/docs/benchmarks.md+++ b/docs/benchmarks.md@@ -2,7 +2,7 @@Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY`.-[Terminal-Bench 2.1](#terminal-bench-21) · [DeepSWE](#deepswe) · [Run a benchmark](#run-a-benchmark)+[Terminal-Bench 2.1](#terminal-bench-21) · [Run a benchmark](#run-a-benchmark)## Terminal-Bench 2.1@@ -19,6 +19,8 @@ Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY| Opus 4.8 only | 77.15% | $148.83 | — || GLM 5.2 only | 74.16% | $71.31 | — |+<!-- DeepSWE results are hidden while run validity is being reviewed.+## DeepSWE**v1.1 · 113 tasks · Codex CLI · GPT-5.6 Luna + Sol**@@ -35,6 +37,8 @@ Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEYLuna: one run; others: three-run means. [Evaluation settings](../benchmark/DEEPSWE_V11_QUALIFICATION.md).+-->+## Run a benchmark- **Terminal-Bench 2.1:** [Setup](../benchmark/README.md#one-time-setup) · [Run](../benchmark/README.md#run-with-switchyard-routing)diff --git a/docs/index.md b/docs/index.mdindex 67c220ea0..a71c97193 100644--- a/docs/index.md+++ b/docs/index.md@@ -18,7 +18,7 @@ The Server Path builds and runs the standalone `switchyard-server` binary.## Explore-- [Benchmarks](benchmarks.md): compare Terminal-Bench 2.1 and DeepSWE results and open their routing configs+- [Benchmarks](benchmarks.md): compare Terminal-Bench 2.1 results and open their routing configs- [Core Concepts](core_concepts.md): learn the LLM client, target, and route layers- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model- [Architecture](architecture.md): understand the proxy and library componentsdiff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.mdindex 519942617..b625e232f 100644--- a/docs/routing_algorithms/model_selection.md+++ b/docs/routing_algorithms/model_selection.md@@ -129,8 +129,8 @@ behave differently on different endpoints. See## Measured pairs-See [Benchmarks](../benchmarks.md) for consolidated Terminal-Bench 2.1 and DeepSWE results,-config links, and evaluation settings.+See [Benchmarks](../benchmarks.md) for consolidated Terminal-Bench 2.1 results+and config links.These pairs have results from checked-in profiles or a public report. Each rowdescribes one pair on one workload. The thresholds and results do not transfer@@ -146,7 +146,7 @@ serving stack.What the rows show about model choice:- The gap between the two single models sets the room for routing. Compare the- fixed-model baselines with the routing results, as in the [DeepSWE studies](../benchmarks.md#deepswe).+ fixed-model baselines with the routing results, as in the [Terminal-Bench 2.1 results](../benchmarks.md#terminal-bench-21).- When the efficient model is close to the capable model, escalation keeps mostof the accuracy at a fraction of the cost. On the LangChain suite the twosingle models were 8 points apart. Routing kept 80 of the 86 points at about