Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(benchmarks): consolidate results and configs (#1044)

nachiketb-nvidiac39d38def7db
diff --git a/README.md b/README.md
index 000c7cb3c..d4e94b441 100644
--- a/README.md
+++ b/README.md
@@ -20,6 +20,7 @@ Use Switchyard through a gateway integration, try it with a local proxy, or embe
<p align="center">
<a href="docs/getting_started.md">Quickstart</a> &emsp;·&emsp;
<a href="https://nvidia-nemo.github.io/Switchyard/">Documentation</a> &emsp;·&emsp;
+ <a href="docs/benchmarks.md">Benchmarks</a> &emsp;·&emsp;
<a href="#whats-new">What’s new</a> &emsp;·&emsp;
<a href="https://github.com/NVIDIA-NeMo/Switchyard/issues/1026">Roadmap</a> &emsp;·&emsp;
<a href="https://github.com/NVIDIA-NeMo/Switchyard/releases">Releases</a> &emsp;·&emsp;
@@ -100,9 +101,7 @@ The [routing overview](docs/routing_algorithms/overview.md) retains the full cat
## Evaluation and reference
-![Task completion versus cost for Switchyard classification, stage, escalation, and Jev task routing, compared with Opus 4.8 and GLM 5.2 single-model baselines.](assets/switchyard-cost-accuracy.svg)
-
-Results depend on the benchmark, model pool, serving stack, and routing configuration.
+[Compare benchmark results and configs](docs/benchmarks.md) for Terminal-Bench 2.1 and DeepSWE.
For latency and routing overhead testing, see [Soak Testing](docs/operations/soak_test.md).
### Further reading
diff --git a/assets/switchyard-cost-accuracy.png b/assets/switchyard-cost-accuracy.png
deleted file mode 100644
index 09b541bed..000000000
Binary files a/assets/switchyard-cost-accuracy.png and /dev/null differ
diff --git a/benchmark/README.md b/benchmark/README.md
index dc9a2e243..e77e1e5b5 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -3,6 +3,8 @@
# Harbor Benchmarks
+For recorded results and their routing configs, see [Benchmarks](../docs/benchmarks.md).
+
Use this guide to run Harbor Terminal-Bench Lite from a fresh Switchyard clone.
It covers the two smoke paths most people need first:
@@ -338,7 +340,8 @@ DeepSWE uses Harbor's task format but its own runner, [Pier](https://github.com/
(required since v1.1 for the separate-verifier/collect-hook pattern). It is not part of
`run-baseline.sh`.
-See [DeepSWE v1.1 qualification settings](DEEPSWE_V11_QUALIFICATION.md) for the exact
+See [DeepSWE results](../docs/benchmarks.md#deepswe) for task completion and cost comparisons,
+and [DeepSWE v1.1 qualification settings](DEEPSWE_V11_QUALIFICATION.md) for the exact
versions, timeouts, scoring rules, and routing profiles used for qualification.
```bash
@@ -380,12 +383,7 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
instead. Binding port 443 needs elevated privileges on most Linux hosts (`sudo`, or
`setcap 'cap_net_bind_service=+ep'` on the binary).
-Advisor-gate routing (GPT-5.6 Luna executes; GPT-5.6 Sol reviews its "done" claims, up to
-three per task) measured 57.5% +/- 3.2 on the full 113 tasks (k=3, closed-book). The profile
-keeps the routing parameters exactly as run and reaches both models through OpenRouter, so it
-needs only `OPENROUTER_API_KEY`. The measured runs had Codex at reasoning effort `max`
-(`model_reasoning_effort = "max"` in the agent's Codex config); the profile forces `max` on
-both models server-side regardless:
+Run the Advisor profile. It sets reasoning effort to `max` on both models:
```bash
export OPENROUTER_API_KEY="..."
@@ -398,11 +396,7 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
--ae OPENAI_API_KEY=unused
```
-Plan/execute routing uses GPT-5.6 Sol for repository inspection and planning, then hands the
-full trajectory to GPT-5.6 Luna after the first mutation. It measured 60.8% +/- 5.3 at
-$180.51 +/- 9.43 per run on the full closed-book benchmark (k=3). The profile preserves the
-exact planning prompt and routing policy, with provider settings adapted from NVIDIA Inference
-Hub to OpenRouter:
+Run the Plan/Execute profile:
```bash
export OPENROUTER_API_KEY="..."
@@ -416,11 +410,7 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
--ae OPENAI_API_KEY=unused
```
-The qualified stage-router profile (GPT-5.6 Luna efficient tier, GPT-5.6 Sol capable tier) solved
-76/113 tasks (67.3% strict) on the full closed-book benchmark. Its routing policy is published
-with OpenRouter provider settings so the file runs as-is with `OPENROUTER_API_KEY`. The header
-records the Switchyard commit, model ids, harness inputs, run id, and the one crashed task. The
-profile's public route id is `gpt-5.6-luna`, matching the qualification run:
+Run the Stage profile. Its public route ID is `gpt-5.6-luna`:
```bash
export OPENROUTER_API_KEY="..."
diff --git a/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml b/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml
index c2c13a0dd..b128d6c3a 100644
--- a/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml
+++ b/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml
@@ -1,7 +1,7 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
-# Escalation-router deployment behind the v0.2.0 Terminal-Bench 2.1 efficiency results
+# Escalation-router deployment behind the Terminal-Bench 2.1 efficiency results
# (https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/).
#
# This is the configuration as benchmarked, with one change: the published runs were
@@ -12,9 +12,7 @@
# the weak tier ran with reasoning effort `high`, and the judge ran with thinking
# disabled — mirrored below with OpenRouter's `reasoning` controls.
#
-# The `escalation` block is part of the v0.2.0 server config schema only; it is not
-# accepted by the current `switchyard-server`. Run from the tag:
-# git checkout v0.2.0-rc.1
+# Run with the current switchyard-server:
# switchyard-server --config benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml --port 4000
schema_version = 1
diff --git a/benchmark/routing-profiles/tb21-jev-opus-glm.toml b/benchmark/routing-profiles/tb21-jev-opus-glm.toml
index 0ca28fc21..897a1928d 100644
--- a/benchmark/routing-profiles/tb21-jev-opus-glm.toml
+++ b/benchmark/routing-profiles/tb21-jev-opus-glm.toml
@@ -6,11 +6,11 @@
# Switchyard uses its default advantage/no_advantage Choice question and sends
# task messages and candidate metadata. Criteria below are part of the instructions.
# The 0.05 cutoff is a starting value, not calibrated for this adapted question.
-# Completion endpoints use OpenRouter rather than the experiment's
-# Bedrock/Inference Hub baselines. Re-evaluate before comparing with the results.
-# Recorded replay results are shown in the repository README.
+# Jev and completion calls use OpenRouter. Completion endpoints differ from the
+# experiment's Bedrock/Inference Hub baselines. Re-evaluate before comparing with
+# the recorded replay results in docs/benchmarks.md#terminal-bench-21.
#
-# Set TYPESAFE_API_KEY and OPENROUTER_API_KEY, then run from the repository root:
+# Set OPENROUTER_API_KEY, then run from the repository root:
# switchyard-server --config benchmark/routing-profiles/tb21-jev-opus-glm.toml
# Send a unique x-switchyard-session-id per task.
@@ -23,12 +23,12 @@ api_key_env = "OPENROUTER_API_KEY"
[decision_clients.jev]
format = "system_one"
-endpoint = "https://api.typesafe.ai/v1/systemone"
-api_key_env = "TYPESAFE_API_KEY"
+endpoint = "https://openrouter.ai/api/alpha/decisions"
+api_key_env = "OPENROUTER_API_KEY"
timeout_ms = 10000
[decision_targets.judge]
-id = "jev-1.13.0"
+id = "typesafe/jev-1.13"
decision_client = "jev"
[targets.opus]
diff --git a/docs/assets/deepswe-cost-accuracy.svg b/docs/assets/deepswe-cost-accuracy.svg
new file mode 100644
index 000000000..c19e75d4e
--- /dev/null
+++ b/docs/assets/deepswe-cost-accuracy.svg
@@ -0,0 +1,251 @@
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1250pt" height="560pt" viewBox="0 0 1250 560" xmlns="http://www.w3.org/2000/svg" version="1.1" role="img" aria-labelledby="title description">
+ <title id="title">DeepSWE v1.1 task completion versus cost</title>
+ <desc id="description">DeepSWE v1.1 on 113 tasks using Codex CLI and GPT-5.6 Luna and Sol. Plan/Execute: 60.8% ± 5.3 at $180.51 ± 9.43 per run. Advisor: 57.5% ± 3.2 at $208 ± 22 per run. Stage: 61.36% ± 6.02 at $466.28 ± $8.83 per run, averaged over 3 runs. One baseline per model, selected by highest reported completion score, with lower cost breaking ties: Sol: 62.8% ± 3.1 at $720.95 ± 19.47, from the Plan/Execute study; Luna: 52.2% at about $49, the better of two valid runs from the Advisor study. Sol and the routed points are three-run means; Luna is an individual run. Reported ± values are sample standard deviations.</desc>
+ <metadata>
+ <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
+ <cc:Work>
+ <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
+ <dc:title>DeepSWE v1.1 task completion versus cost</dc:title>
+ <dc:format>image/svg+xml</dc:format>
+ <dc:creator>
+ <cc:Agent>
+ <dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
+ </cc:Agent>
+ </dc:creator>
+ </cc:Work>
+ </rdf:RDF>
+ </metadata>
+ <defs>
+ <linearGradient id="frontier" gradientUnits="userSpaceOnUse" x1="157" y1="24" x2="309" y2="176">
+ <stop offset="0" stop-color="#5f8f15"/>
+ <stop offset="0.2" stop-color="#92b75f"/>
+ <stop offset="0.45" stop-color="#c5df9d"/>
+ <stop offset="0.7" stop-color="#eaf2d8"/>
+ <stop offset="0.9" stop-color="#fbfdf7"/>
+ <stop offset="1" stop-color="#ffffff"/>
+ </linearGradient>
+ <style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
+ </defs>
+ <g id="figure_1">
+ <g id="patch_1">
+ <path d="M 0 560
+L 1250 560
+L 1250 0
+L 0 0
+z
+" style="fill: #ffffff"/>
+ </g>
+ <rect x="157" y="24" width="304" height="304" fill="url(#frontier)"/>
+ <g id="axes_1">
+ <g id="line2d_1">
+ <path d="M 165.5 415.5
+L 1082.5 415.5
+" clip-path="url(#p9201e5e723)" style="fill: none; stroke: #d9d9d9; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_2">
+ <path d="M 165.5 264.5
+L 1082.5 264.5
+" clip-path="url(#p9201e5e723)" style="fill: none; stroke: #d9d9d9; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_3">
+ <path d="M 165.5 113.5
+L 1082.5 113.5
+" clip-path="url(#p9201e5e723)" style="fill: none; stroke: #d9d9d9; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_4">
+ <defs>
+ <path id="m0288455ae3" d="M 0 5.2
+C 1.379056 5.2 2.701815 4.652095 3.676955 3.676955
+C 4.652095 2.701815 5.2 1.379056 5.2 0
+C 5.2 -1.379056 4.652095 -2.701815 3.676955 -3.676955
+C 2.701815 -4.652095 1.379056 -5.2 0 -5.2
+C -1.379056 -5.2 -2.701815 -4.652095 -3.676955 -3.676955
+C -4.652095 -2.701815 -5.2 -1.379056 -5.2 0
+C -5.2 1.379056 -4.652095 2.701815 -3.676955 3.676955
+C -2.701815 4.652095 -1.379056 5.2 0 5.2
+z
+"/>
+ </defs>
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#m0288455ae3" x="370.4" y="516.7" style="fill: #76b900"/>
+ </g>
+ </g>
+ <g id="line2d_5">
+ <defs>
+ <path id="m460a3c8150" d="M 0 5.2
+C 1.379056 5.2 2.701815 4.652095 3.676955 3.676955
+C 4.652095 2.701815 5.2 1.379056 5.2 0
+C 5.2 -1.379056 4.652095 -2.701815 3.676955 -3.676955
+C 2.701815 -4.652095 1.379056 -5.2 0 -5.2
+C -1.379056 -5.2 -2.701815 -4.652095 -3.676955 -3.676955
+C -4.652095 -2.701815 -5.2 -1.379056 -5.2 0
+C -5.2 1.379056 -4.652095 2.701815 -3.676955 3.676955
+C -2.701815 4.652095 -1.379056 5.2 0 5.2
+z
+"/>
+ </defs>
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#m460a3c8150" x="709.2" y="516.7" style="fill: #ffc000"/>
+ </g>
+ </g>
+ <g id="text_1">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: end" x="145" y="421" transform="rotate(-0 145 421)">45</text>
+ </g>
+ <g id="text_2">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: end" x="145" y="270" transform="rotate(-0 145 270)">55</text>
+ </g>
+ <g id="text_3">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: end" x="145" y="119" transform="rotate(-0 145 119)">65</text>
+ </g>
+ <g id="text_4">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="165.5" y="448" transform="rotate(-0 165.5 448)">$0</text>
+ </g>
+ <g id="text_5">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="394.75" y="448" transform="rotate(-0 394.75 448)">$200</text>
+ </g>
+ <g id="text_6">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="624" y="448" transform="rotate(-0 624 448)">$400</text>
+ </g>
+ <g id="text_7">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="853.25" y="448" transform="rotate(-0 853.25 448)">$600</text>
+ </g>
+ <g id="text_8">
+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="1082.5" y="448" transform="rotate(-0 1082.5 448)">$800</text>
+ </g>
+ <g id="text_9">
+ <text style="font-weight: 700; font-size: 18px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="586" y="482" transform="rotate(-0 586 482)">Cost per run ($)</text>
+ </g>
+ <g id="text_10">
+ <text style="font-weight: 700; font-size: 16px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif" transform="translate(54.648438 276.367188) rotate(-90)">% Tasks Completed</text>
+ </g>
+ <g id="text_11">
+ <text style="font-weight: 700; font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="173" y="59" transform="rotate(-0 173 59)">Efficiency</text>
+ </g>
+ <g id="text_12">
+ <text style="font-weight: 700; font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="173" y="85" transform="rotate(-0 173 85)">Frontier</text>
+ </g>
+ <g id="plan-execute" data-cost="180.51" data-score="60.8">
+ <defs>
+ <path id="m38f0431c9c" d="M 0 9
+C 2.386828 9 4.676219 8.051703 6.363961 6.363961
+C 8.051703 4.676219 9 2.386828 9 0
+C 9 -2.386828 8.051703 -4.676219 6.363961 -6.363961
+C 4.676219 -8.051703 2.386828 -9 0 -9
+C -2.386828 -9 -4.676219 -8.051703 -6.363961 -6.363961
+C -8.051703 -4.676219 -9 -2.386828 -9 0
+C -9 2.386828 -8.051703 4.676219 -6.363961 6.363961
+C -4.676219 8.051703 -2.386828 9 0 9
+z
+"/>
+ </defs>
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#m38f0431c9c" x="372.409587" y="176.92" style="fill: #76b900"/>
+ </g>
+ </g>
+ <g id="advisor" data-cost="208" data-score="57.5">
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#m38f0431c9c" x="403.92" y="226.75" style="fill: #76b900"/>
+ </g>
+ </g>
+ <g id="stage" data-cost="466.28" data-score="61.36">
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#m38f0431c9c" x="699.97345" y="168.464" style="fill: #76b900"/>
+ </g>
+ </g>
+ <g id="plan-sol" data-cost="720.95" data-score="62.8">
+ <defs>
+ <path id="mabb4518413" d="M 0 8
+C 2.121625 8 4.156639 7.15707 5.656854 5.656854
+C 7.15707 4.156639 8 2.121625 8 0
+C 8 -2.121625 7.15707 -4.156639 5.656854 -5.656854
+C 4.156639 -7.15707 2.121625 -8 0 -8
+C -2.121625 -8 -4.156639 -7.15707 -5.656854 -5.656854
+C -7.15707 -4.156639 -8 -2.121625 -8 0
+C -8 2.121625 -7.15707 4.156639 -5.656854 5.656854
+C -4.156639 7.15707 -2.121625 8 0 8
+z
+"/>
+ </defs>
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#mabb4518413" x="991.888937" y="146.72" style="fill: #ffc000"/>
+ </g>
+ </g>
+ <g id="advisor-luna-2" data-cost="49" data-score="52.2">
+ <g clip-path="url(#p9201e5e723)">
+ <use xlink:href="#mabb4518413" x="221.66625" y="306.78" style="fill: #ffc000"/>
+ </g>
+ </g>
+ <g id="text_13">
+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="430" y="169" transform="rotate(-0 430 169)">Plan/Execute Router</text>
+ </g>
+ <g id="text_14">
+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="430" y="188" transform="rotate(-0 430 188)">Sol + Luna · 60.8% · $180.51</text>
+ </g>
+ <g id="text_15">
+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="740" y="140" transform="rotate(-0 740 140)">Agent Stage Router</text>
+ </g>
+ <g id="text_16">
+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="740" y="159" transform="rotate(-0 740 159)">Sol + Luna · 61.36% · $466.28</text>
+ </g>
+ <g id="text_17">
+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="450" y="240" transform="rotate(-0 450 240)">Advisor Router</text>
+ </g>
+ <g id="text_18">
+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="450" y="259" transform="rotate(-0 450 259)">Luna + Sol · 57.5% · $208</text>
+ </g>
+ <g id="text_19">
+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="992" y="197" transform="rotate(-0 992 197)">GPT-5.6 Sol Only</text>
+ </g>
+ <g id="text_20">
+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle; fill: #777777" x="992" y="216" transform="rotate(-0 992 216)">62.8% · $720.95</text>
+ </g>
+ <g id="text_21">
+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="285" y="314" transform="rotate(-0 285 314)">GPT-5.6 Luna Only</text>
+ </g>
+ <g id="text_22">
+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="285" y="333" transform="rotate(-0 285 333)">52.2% · ≈$49</text>
+ </g>
+ <g id="patch_2">
+ <path d="M 425 177
+Q 407 177 390.118034 177
+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>
+ <path d="M 397.518034 180.7
+L 390.118034 177
+L 397.518034 173.3
+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>
+ </g>
+ <g id="patch_3">
+ <path d="M 444 232
+Q 431.5 230 420.103992 228.176639
+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>
+ <path d="M 426.826488 232.999299
+L 420.103992 228.176639
+L 427.995618 225.692238
+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>
+ </g>
+ <g id="patch_4">
+ <path d="M 275 307
+Q 256.5 307 239.118034 307
+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>
+ <path d="M 246.518034 310.7
+L 239.118034 307
+L 246.518034 303.3
+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>
+ </g>
+ <g id="text_23">
+ <text style="font-size: 18px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="379.2" y="522" transform="rotate(-0 379.2 522)">NeMo Switchyard Routing Algorithm</text>
+ </g>
+ <g id="text_24">
+ <text style="font-size: 18px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="718" y="522" transform="rotate(-0 718 522)">Single Model</text>
+ </g>
+ </g>
+ </g>
+ <defs>
+ <clipPath id="p9201e5e723">
+ <rect x="0" y="0" width="1250" height="560"/>
+ </clipPath>
+ </defs>
+</svg>
diff --git a/assets/switchyard-cost-accuracy.svg b/docs/assets/switchyard-cost-accuracy.svg
similarity index 87%
rename from assets/switchyard-cost-accuracy.svg
rename to docs/assets/switchyard-cost-accuracy.svg
index 03e9a3a34..0114048c6 100644
--- a/assets/switchyard-cost-accuracy.svg
+++ b/docs/assets/switchyard-cost-accuracy.svg
@@ -1,6 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1250 560" width="1250" height="560" role="img" aria-labelledby="title description">
<title id="title">Switchyard task completion versus cost</title>
- <desc id="description">The original classification, stage, escalation, Opus 4.8, and GLM 5.2 points preserve the positions shown in switchyard-cost-accuracy.png. Jev task routing is a separate exploratory recorded-run replay: 73 of 89 tasks completed (82.02%) at an estimated model cost of $92.12, including estimated Jev calls. The combined set includes 62 training tasks. On the reused 27-task test split, Jev solved 22 tasks versus 23 for its paired Opus baseline. The replay used a different evaluation setup and cost accounting from the other plotted runs; its adapted TOML profile needs a fresh evaluation.</desc>
+ <desc id="description">The original classification, stage, escalation, Opus 4.8, and GLM 5.2 points preserve the positions shown in the published comparison. Jev task routing is a separate exploratory recorded-run replay: 73 of 89 tasks completed (82.02%) at an estimated model cost of $92.12, including estimated Jev calls. The combined set includes 62 training tasks. On the reused 27-task test split, Jev solved 22 tasks versus 23 for its paired Opus baseline. The replay used a different evaluation setup and cost accounting from the other plotted runs; its adapted TOML profile needs a fresh evaluation.</desc>
<defs>
<linearGradient id="frontier" gradientUnits="userSpaceOnUse" x1="157" y1="24" x2="309" y2="176">
<stop offset="0" stop-color="#5f8f15"/>
@@ -59,7 +59,7 @@
<g id="jev-replay" data-total-cost="92.116074918" data-successes="73" data-tasks="89">
<circle cx="563.754" cy="52.421" r="9" class="route"/>
- <text x="583" y="45" class="label">Jev Task Router</text>
+ <text x="583" y="45" class="label">Jev Task Router (replay)</text>
<text x="583" y="64" class="models">Opus 4.8 and GLM 5.2</text>
</g>
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
new file mode 100644
index 000000000..4a63d0e9d
--- /dev/null
+++ b/docs/benchmarks.md
@@ -0,0 +1,41 @@
+# Benchmarks
+
+Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY`.
+
+[Terminal-Bench 2.1](#terminal-bench-21) · [DeepSWE](#deepswe) · [Run a benchmark](#run-a-benchmark)
+
+## Terminal-Bench 2.1
+
+**Claude Opus 4.8 + GLM 5.2**
+
+![Terminal-Bench 2.1 task completion versus cost.](assets/switchyard-cost-accuracy.svg)
+
+| Strategy | Tasks solved | Cost per run | Config |
+| --- | ---: | ---: | --- |
+| LLM classifier | 79.40% | $109.04 | — |
+| Stage | 79.03% | $109.81 | — |
+| Escalation | 79.80% | $122.63 | [TOML](../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml) |
+| Jev (replay) | 82.02% | ≈$92.12 | [TOML](../benchmark/routing-profiles/tb21-jev-opus-glm.toml) |
+| Opus 4.8 only | 77.15% | $148.83 | — |
+| GLM 5.2 only | 74.16% | $71.31 | — |
+
+## DeepSWE
+
+**v1.1 · 113 tasks · Codex CLI · GPT-5.6 Luna + Sol**
+
+![DeepSWE task completion versus cost.](assets/deepswe-cost-accuracy.svg)
+
+| Strategy | Tasks solved | Cost per run | Config |
+| --- | ---: | ---: | --- |
+| Plan/Execute | 60.8% | $180.51 | [TOML](../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml) |
+| Advisor | 57.5% | $208.00 | [TOML](../benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml) |
+| Stage | 61.36% | $466.28 | [TOML](../benchmark/routing-profiles/deepswe-v11-stage-router-luna-sol.toml) |
+| Sol only | 62.8% | $720.95 | — |
+| Luna only | 52.2% | ≈$49 | — |
+
+Luna: one run; others: three-run means. [Evaluation settings](../benchmark/DEEPSWE_V11_QUALIFICATION.md).
+
+## Run a benchmark
+
+- **Terminal-Bench 2.1:** [Setup](../benchmark/README.md#one-time-setup) · [Run](../benchmark/README.md#run-with-switchyard-routing)
+- **DeepSWE:** [Run with Pier](../benchmark/README.md#deepswe-v11)
diff --git a/docs/index.md b/docs/index.md
index e456b6b98..67c220ea0 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -18,6 +18,7 @@ The Server Path builds and runs the standalone `switchyard-server` binary.
## Explore
+- [Benchmarks](benchmarks.md): compare Terminal-Bench 2.1 and DeepSWE results and open their routing configs
- [Core Concepts](core_concepts.md): learn the LLM client, target, and route layers
- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model
- [Architecture](architecture.md): understand the proxy and library components
diff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.md
index bb4c83d5c..519942617 100644
--- a/docs/routing_algorithms/model_selection.md
+++ b/docs/routing_algorithms/model_selection.md
@@ -129,6 +129,9 @@ behave differently on different endpoints. See
## Measured pairs
+See [Benchmarks](../benchmarks.md) for consolidated Terminal-Bench 2.1 and DeepSWE results,
+config links, and evaluation settings.
+
These pairs have results from checked-in profiles or a public report. Each row
describes one pair on one workload. The thresholds and results do not transfer
to another pair or another workload. The profile headers give the serving stack,
@@ -137,17 +140,13 @@ serving stack.
| Efficient tier | Capable tier | Strategy | Workload | Result | Single-model anchors |
|---|---|---|---|---|---|
-| GPT-5.6 Luna | GPT-5.6 Sol | [`stage_router`](../../benchmark/routing-profiles/deepswe-v11-stage-router-luna-sol.toml): `efficient_first`, threshold 0.5, window 5, hold 2, handoff note | DeepSWE v1.1, 113 tasks, Codex CLI, k=1 | 76/113 solved (67.3% strict) | None in the same run |
-| GPT-5.6 Luna, executes | GPT-5.6 Sol, plans | [`plan_execute`](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml), both at effort `max` | DeepSWE v1.1, 113 tasks, Codex CLI, k=3 | 60.8% ± 5.3 at $180.51 ± 9.43 per run | Sol alone 62.8% ± 3.1 at $720.95. Luna alone 51.0% ± 3.7 at $50.02. |
-| GPT-5.6 Luna, executes | GPT-5.6 Sol, reviews | [`advisor`](../../benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml), both at effort `max` | DeepSWE v1.1, 113 tasks, Codex CLI, k=3 | 57.5% ± 3.2 at $208 ± 22 per run | Sol alone 62.8% ± 3.1 at $743. Luna alone 46.9% and 52.2% at about $49. |
| Qwen3.6-35B-A3B, also the judge | Claude Opus 4.7 | `llm_classifier` custom, [balanced](../../benchmark/routing-profiles/tau2-telecom-custom-opus-qwen-balanced.toml) and [aggressive](../../benchmark/routing-profiles/tau2-telecom-custom-opus-qwen-aggressive.toml) | tau2-bench telecom, 114 tasks | 0.903 ± 0.071 solve rate with 45% of turns on the weak tier. 0.891 ± 0.029 with about 85%. | Within noise of the strong tier, per the profile headers |
| Nemotron 3.5 Lightning | Claude Opus 4.8, with Gemini 3.1 Flash-Lite as judge | `llm_classifier` escalation, `confirmations = 2` | LangChain Deep Agents suite, 145 tasks, 5 runs ([report](https://www.langchain.com/blog/switchyard-agent-routing-benchmark)) | 80.0% at $3.00 per run, with 7% of calls on Opus | Opus alone 86.0% at $11.45. Nemotron alone 77.7% at $0.72. |
What the rows show about model choice:
-- The gap between the two single models sets the room for routing. On DeepSWE,
- Sol alone beat Luna alone by about 11 to 16 points. The plan/execute and
- advisor arms landed between them at about a quarter of Sol's cost.
+- The gap between the two single models sets the room for routing. Compare the
+ fixed-model baselines with the routing results, as in the [DeepSWE studies](../benchmarks.md#deepswe).
- When the efficient model is close to the capable model, escalation keeps most
of the accuracy at a fraction of the cost. On the LangChain suite the two
single models were 8 points apart. Routing kept 80 of the 86 points at about
@@ -160,10 +159,6 @@ Other checked-in configurations:
- [`tb-lite-llm-classifier-opus-kimi-gemini.toml`](../../benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml)
(Claude Opus 4.7 strong, Kimi K2.7 code weak, Gemini 3.5 Flash judge) is a
smoke test. It has no published result.
-- [`tb21-escalation-opus-glm-deepseek.toml`](../../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml)
- (Claude Opus 4.8 strong, GLM 5.2 weak, DeepSeek V4 Flash judge) is the
- escalation configuration used for the v0.2.0 Terminal-Bench 2.1 runs. Its
- header links the post about those runs.
## Counter-examples
diff --git a/mkdocs.yml b/mkdocs.yml
index 6c5309ffc..8d6f51518 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -20,6 +20,7 @@ nav:
- Get Started:
- Getting Started: getting_started.md
- Known Issues: known_issues.md
+ - Benchmarks: benchmarks.md
- Concepts:
- Core Concepts: core_concepts.md
- Architecture: architecture.md