Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(benchmarks): hide DeepSWE results pending validation (#1054)

nachiketb-nvidia9537e07259cd
diff --git a/README.md b/README.md
index d4e94b441..519c27ca5 100644
--- a/README.md
+++ b/README.md
@@ -101,7 +101,7 @@ The [routing overview](docs/routing_algorithms/overview.md) retains the full cat
## Evaluation and reference
-[Compare benchmark results and configs](docs/benchmarks.md) for Terminal-Bench 2.1 and DeepSWE.
+[Compare benchmark results and configs](docs/benchmarks.md) for Terminal-Bench 2.1.
For latency and routing overhead testing, see [Soak Testing](docs/operations/soak_test.md).
### Further reading
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index 4a63d0e9d..50c772bc6 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -2,7 +2,7 @@
Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY`.
-[Terminal-Bench 2.1](#terminal-bench-21) · [DeepSWE](#deepswe) · [Run a benchmark](#run-a-benchmark)
+[Terminal-Bench 2.1](#terminal-bench-21) · [Run a benchmark](#run-a-benchmark)
## Terminal-Bench 2.1
@@ -19,6 +19,8 @@ Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY
| Opus 4.8 only | 77.15% | $148.83 | — |
| GLM 5.2 only | 74.16% | $71.31 | — |
+<!-- DeepSWE results are hidden while run validity is being reviewed.
+
## DeepSWE
**v1.1 · 113 tasks · Codex CLI · GPT-5.6 Luna + Sol**
@@ -35,6 +37,8 @@ Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY
Luna: one run; others: three-run means. [Evaluation settings](../benchmark/DEEPSWE_V11_QUALIFICATION.md).
+-->
+
## Run a benchmark
- **Terminal-Bench 2.1:** [Setup](../benchmark/README.md#one-time-setup) · [Run](../benchmark/README.md#run-with-switchyard-routing)
diff --git a/docs/index.md b/docs/index.md
index 67c220ea0..a71c97193 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -18,7 +18,7 @@ The Server Path builds and runs the standalone `switchyard-server` binary.
## Explore
-- [Benchmarks](benchmarks.md): compare Terminal-Bench 2.1 and DeepSWE results and open their routing configs
+- [Benchmarks](benchmarks.md): compare Terminal-Bench 2.1 results and open their routing configs
- [Core Concepts](core_concepts.md): learn the LLM client, target, and route layers
- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model
- [Architecture](architecture.md): understand the proxy and library components
diff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.md
index 519942617..b625e232f 100644
--- a/docs/routing_algorithms/model_selection.md
+++ b/docs/routing_algorithms/model_selection.md
@@ -129,8 +129,8 @@ behave differently on different endpoints. See
## Measured pairs
-See [Benchmarks](../benchmarks.md) for consolidated Terminal-Bench 2.1 and DeepSWE results,
-config links, and evaluation settings.
+See [Benchmarks](../benchmarks.md) for consolidated Terminal-Bench 2.1 results
+and config links.
These pairs have results from checked-in profiles or a public report. Each row
describes one pair on one workload. The thresholds and results do not transfer
@@ -146,7 +146,7 @@ serving stack.
What the rows show about model choice:
- The gap between the two single models sets the room for routing. Compare the
- fixed-model baselines with the routing results, as in the [DeepSWE studies](../benchmarks.md#deepswe).
+ fixed-model baselines with the routing results, as in the [Terminal-Bench 2.1 results](../benchmarks.md#terminal-bench-21).
- When the efficient model is close to the capable model, escalation keeps most
of the accuracy at a fraction of the cost. On the LangChain suite the two
single models were 8 points apart. Routing kept 80 of the 86 points at about