Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(benchmarks): consolidate results and configs (#1044)
nachiketb-nvidiac39d38def7db
diff --git a/README.md b/README.mdindex 000c7cb3c..d4e94b441 100644--- a/README.md+++ b/README.md@@ -20,6 +20,7 @@ Use Switchyard through a gateway integration, try it with a local proxy, or embe<p align="center"><a href="docs/getting_started.md">Quickstart</a>  · <a href="https://nvidia-nemo.github.io/Switchyard/">Documentation</a>  · + <a href="docs/benchmarks.md">Benchmarks</a>  · <a href="#whats-new">What’s new</a>  · <a href="https://github.com/NVIDIA-NeMo/Switchyard/issues/1026">Roadmap</a>  · <a href="https://github.com/NVIDIA-NeMo/Switchyard/releases">Releases</a>  · @@ -100,9 +101,7 @@ The [routing overview](docs/routing_algorithms/overview.md) retains the full cat## Evaluation and reference---Results depend on the benchmark, model pool, serving stack, and routing configuration.+[Compare benchmark results and configs](docs/benchmarks.md) for Terminal-Bench 2.1 and DeepSWE.For latency and routing overhead testing, see [Soak Testing](docs/operations/soak_test.md).### Further readingdiff --git a/assets/switchyard-cost-accuracy.png b/assets/switchyard-cost-accuracy.pngdeleted file mode 100644index 09b541bed..000000000Binary files a/assets/switchyard-cost-accuracy.png and /dev/null differdiff --git a/benchmark/README.md b/benchmark/README.mdindex dc9a2e243..e77e1e5b5 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -3,6 +3,8 @@# Harbor Benchmarks+For recorded results and their routing configs, see [Benchmarks](../docs/benchmarks.md).+Use this guide to run Harbor Terminal-Bench Lite from a fresh Switchyard clone.It covers the two smoke paths most people need first:@@ -338,7 +340,8 @@ DeepSWE uses Harbor's task format but its own runner, [Pier](https://github.com/(required since v1.1 for the separate-verifier/collect-hook pattern). It is not part of`run-baseline.sh`.-See [DeepSWE v1.1 qualification settings](DEEPSWE_V11_QUALIFICATION.md) for the exact+See [DeepSWE results](../docs/benchmarks.md#deepswe) for task completion and cost comparisons,+and [DeepSWE v1.1 qualification settings](DEEPSWE_V11_QUALIFICATION.md) for the exactversions, timeouts, scoring rules, and routing profiles used for qualification.```bash@@ -380,12 +383,7 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \instead. Binding port 443 needs elevated privileges on most Linux hosts (`sudo`, or`setcap 'cap_net_bind_service=+ep'` on the binary).-Advisor-gate routing (GPT-5.6 Luna executes; GPT-5.6 Sol reviews its "done" claims, up to-three per task) measured 57.5% +/- 3.2 on the full 113 tasks (k=3, closed-book). The profile-keeps the routing parameters exactly as run and reaches both models through OpenRouter, so it-needs only `OPENROUTER_API_KEY`. The measured runs had Codex at reasoning effort `max`-(`model_reasoning_effort = "max"` in the agent's Codex config); the profile forces `max` on-both models server-side regardless:+Run the Advisor profile. It sets reasoning effort to `max` on both models:```bashexport OPENROUTER_API_KEY="..."@@ -398,11 +396,7 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \--ae OPENAI_API_KEY=unused```-Plan/execute routing uses GPT-5.6 Sol for repository inspection and planning, then hands the-full trajectory to GPT-5.6 Luna after the first mutation. It measured 60.8% +/- 5.3 at-$180.51 +/- 9.43 per run on the full closed-book benchmark (k=3). The profile preserves the-exact planning prompt and routing policy, with provider settings adapted from NVIDIA Inference-Hub to OpenRouter:+Run the Plan/Execute profile:```bashexport OPENROUTER_API_KEY="..."@@ -416,11 +410,7 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \--ae OPENAI_API_KEY=unused```-The qualified stage-router profile (GPT-5.6 Luna efficient tier, GPT-5.6 Sol capable tier) solved-76/113 tasks (67.3% strict) on the full closed-book benchmark. Its routing policy is published-with OpenRouter provider settings so the file runs as-is with `OPENROUTER_API_KEY`. The header-records the Switchyard commit, model ids, harness inputs, run id, and the one crashed task. The-profile's public route id is `gpt-5.6-luna`, matching the qualification run:+Run the Stage profile. Its public route ID is `gpt-5.6-luna`:```bashexport OPENROUTER_API_KEY="..."diff --git a/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml b/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.tomlindex c2c13a0dd..b128d6c3a 100644--- a/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml+++ b/benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml@@ -1,7 +1,7 @@# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.# SPDX-License-Identifier: Apache-2.0#-# Escalation-router deployment behind the v0.2.0 Terminal-Bench 2.1 efficiency results+# Escalation-router deployment behind the Terminal-Bench 2.1 efficiency results# (https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/).## This is the configuration as benchmarked, with one change: the published runs were@@ -12,9 +12,7 @@# the weak tier ran with reasoning effort `high`, and the judge ran with thinking# disabled — mirrored below with OpenRouter's `reasoning` controls.#-# The `escalation` block is part of the v0.2.0 server config schema only; it is not-# accepted by the current `switchyard-server`. Run from the tag:-# git checkout v0.2.0-rc.1+# Run with the current switchyard-server:# switchyard-server --config benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml --port 4000schema_version = 1diff --git a/benchmark/routing-profiles/tb21-jev-opus-glm.toml b/benchmark/routing-profiles/tb21-jev-opus-glm.tomlindex 0ca28fc21..897a1928d 100644--- a/benchmark/routing-profiles/tb21-jev-opus-glm.toml+++ b/benchmark/routing-profiles/tb21-jev-opus-glm.toml@@ -6,11 +6,11 @@# Switchyard uses its default advantage/no_advantage Choice question and sends# task messages and candidate metadata. Criteria below are part of the instructions.# The 0.05 cutoff is a starting value, not calibrated for this adapted question.-# Completion endpoints use OpenRouter rather than the experiment's-# Bedrock/Inference Hub baselines. Re-evaluate before comparing with the results.-# Recorded replay results are shown in the repository README.+# Jev and completion calls use OpenRouter. Completion endpoints differ from the+# experiment's Bedrock/Inference Hub baselines. Re-evaluate before comparing with+# the recorded replay results in docs/benchmarks.md#terminal-bench-21.#-# Set TYPESAFE_API_KEY and OPENROUTER_API_KEY, then run from the repository root:+# Set OPENROUTER_API_KEY, then run from the repository root:# switchyard-server --config benchmark/routing-profiles/tb21-jev-opus-glm.toml# Send a unique x-switchyard-session-id per task.@@ -23,12 +23,12 @@ api_key_env = "OPENROUTER_API_KEY"[decision_clients.jev]format = "system_one"-endpoint = "https://api.typesafe.ai/v1/systemone"-api_key_env = "TYPESAFE_API_KEY"+endpoint = "https://openrouter.ai/api/alpha/decisions"+api_key_env = "OPENROUTER_API_KEY"timeout_ms = 10000[decision_targets.judge]-id = "jev-1.13.0"+id = "typesafe/jev-1.13"decision_client = "jev"[targets.opus]diff --git a/docs/assets/deepswe-cost-accuracy.svg b/docs/assets/deepswe-cost-accuracy.svgnew file mode 100644index 000000000..c19e75d4e--- /dev/null+++ b/docs/assets/deepswe-cost-accuracy.svg@@ -0,0 +1,251 @@+<?xml version="1.0" encoding="utf-8" standalone="no"?>+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">+<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1250pt" height="560pt" viewBox="0 0 1250 560" xmlns="http://www.w3.org/2000/svg" version="1.1" role="img" aria-labelledby="title description">+ <title id="title">DeepSWE v1.1 task completion versus cost</title>+ <desc id="description">DeepSWE v1.1 on 113 tasks using Codex CLI and GPT-5.6 Luna and Sol. Plan/Execute: 60.8% ± 5.3 at $180.51 ± 9.43 per run. Advisor: 57.5% ± 3.2 at $208 ± 22 per run. Stage: 61.36% ± 6.02 at $466.28 ± $8.83 per run, averaged over 3 runs. One baseline per model, selected by highest reported completion score, with lower cost breaking ties: Sol: 62.8% ± 3.1 at $720.95 ± 19.47, from the Plan/Execute study; Luna: 52.2% at about $49, the better of two valid runs from the Advisor study. Sol and the routed points are three-run means; Luna is an individual run. Reported ± values are sample standard deviations.</desc>+ <metadata>+ <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">+ <cc:Work>+ <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>+ <dc:title>DeepSWE v1.1 task completion versus cost</dc:title>+ <dc:format>image/svg+xml</dc:format>+ <dc:creator>+ <cc:Agent>+ <dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>+ </cc:Agent>+ </dc:creator>+ </cc:Work>+ </rdf:RDF>+ </metadata>+ <defs>+ <linearGradient id="frontier" gradientUnits="userSpaceOnUse" x1="157" y1="24" x2="309" y2="176">+ <stop offset="0" stop-color="#5f8f15"/>+ <stop offset="0.2" stop-color="#92b75f"/>+ <stop offset="0.45" stop-color="#c5df9d"/>+ <stop offset="0.7" stop-color="#eaf2d8"/>+ <stop offset="0.9" stop-color="#fbfdf7"/>+ <stop offset="1" stop-color="#ffffff"/>+ </linearGradient>+ <style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>+ </defs>+ <g id="figure_1">+ <g id="patch_1">+ <path d="M 0 560+L 1250 560+L 1250 0+L 0 0+z+" style="fill: #ffffff"/>+ </g>+ <rect x="157" y="24" width="304" height="304" fill="url(#frontier)"/>+ <g id="axes_1">+ <g id="line2d_1">+ <path d="M 165.5 415.5+L 1082.5 415.5+" clip-path="url(#p9201e5e723)" style="fill: none; stroke: #d9d9d9; stroke-linecap: square"/>+ </g>+ <g id="line2d_2">+ <path d="M 165.5 264.5+L 1082.5 264.5+" clip-path="url(#p9201e5e723)" style="fill: none; stroke: #d9d9d9; stroke-linecap: square"/>+ </g>+ <g id="line2d_3">+ <path d="M 165.5 113.5+L 1082.5 113.5+" clip-path="url(#p9201e5e723)" style="fill: none; stroke: #d9d9d9; stroke-linecap: square"/>+ </g>+ <g id="line2d_4">+ <defs>+ <path id="m0288455ae3" d="M 0 5.2+C 1.379056 5.2 2.701815 4.652095 3.676955 3.676955+C 4.652095 2.701815 5.2 1.379056 5.2 0+C 5.2 -1.379056 4.652095 -2.701815 3.676955 -3.676955+C 2.701815 -4.652095 1.379056 -5.2 0 -5.2+C -1.379056 -5.2 -2.701815 -4.652095 -3.676955 -3.676955+C -4.652095 -2.701815 -5.2 -1.379056 -5.2 0+C -5.2 1.379056 -4.652095 2.701815 -3.676955 3.676955+C -2.701815 4.652095 -1.379056 5.2 0 5.2+z+"/>+ </defs>+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#m0288455ae3" x="370.4" y="516.7" style="fill: #76b900"/>+ </g>+ </g>+ <g id="line2d_5">+ <defs>+ <path id="m460a3c8150" d="M 0 5.2+C 1.379056 5.2 2.701815 4.652095 3.676955 3.676955+C 4.652095 2.701815 5.2 1.379056 5.2 0+C 5.2 -1.379056 4.652095 -2.701815 3.676955 -3.676955+C 2.701815 -4.652095 1.379056 -5.2 0 -5.2+C -1.379056 -5.2 -2.701815 -4.652095 -3.676955 -3.676955+C -4.652095 -2.701815 -5.2 -1.379056 -5.2 0+C -5.2 1.379056 -4.652095 2.701815 -3.676955 3.676955+C -2.701815 4.652095 -1.379056 5.2 0 5.2+z+"/>+ </defs>+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#m460a3c8150" x="709.2" y="516.7" style="fill: #ffc000"/>+ </g>+ </g>+ <g id="text_1">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: end" x="145" y="421" transform="rotate(-0 145 421)">45</text>+ </g>+ <g id="text_2">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: end" x="145" y="270" transform="rotate(-0 145 270)">55</text>+ </g>+ <g id="text_3">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: end" x="145" y="119" transform="rotate(-0 145 119)">65</text>+ </g>+ <g id="text_4">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="165.5" y="448" transform="rotate(-0 165.5 448)">$0</text>+ </g>+ <g id="text_5">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="394.75" y="448" transform="rotate(-0 394.75 448)">$200</text>+ </g>+ <g id="text_6">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="624" y="448" transform="rotate(-0 624 448)">$400</text>+ </g>+ <g id="text_7">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="853.25" y="448" transform="rotate(-0 853.25 448)">$600</text>+ </g>+ <g id="text_8">+ <text style="font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="1082.5" y="448" transform="rotate(-0 1082.5 448)">$800</text>+ </g>+ <g id="text_9">+ <text style="font-weight: 700; font-size: 18px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="586" y="482" transform="rotate(-0 586 482)">Cost per run ($)</text>+ </g>+ <g id="text_10">+ <text style="font-weight: 700; font-size: 16px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif" transform="translate(54.648438 276.367188) rotate(-90)">% Tasks Completed</text>+ </g>+ <g id="text_11">+ <text style="font-weight: 700; font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="173" y="59" transform="rotate(-0 173 59)">Efficiency</text>+ </g>+ <g id="text_12">+ <text style="font-weight: 700; font-size: 22px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="173" y="85" transform="rotate(-0 173 85)">Frontier</text>+ </g>+ <g id="plan-execute" data-cost="180.51" data-score="60.8">+ <defs>+ <path id="m38f0431c9c" d="M 0 9+C 2.386828 9 4.676219 8.051703 6.363961 6.363961+C 8.051703 4.676219 9 2.386828 9 0+C 9 -2.386828 8.051703 -4.676219 6.363961 -6.363961+C 4.676219 -8.051703 2.386828 -9 0 -9+C -2.386828 -9 -4.676219 -8.051703 -6.363961 -6.363961+C -8.051703 -4.676219 -9 -2.386828 -9 0+C -9 2.386828 -8.051703 4.676219 -6.363961 6.363961+C -4.676219 8.051703 -2.386828 9 0 9+z+"/>+ </defs>+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#m38f0431c9c" x="372.409587" y="176.92" style="fill: #76b900"/>+ </g>+ </g>+ <g id="advisor" data-cost="208" data-score="57.5">+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#m38f0431c9c" x="403.92" y="226.75" style="fill: #76b900"/>+ </g>+ </g>+ <g id="stage" data-cost="466.28" data-score="61.36">+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#m38f0431c9c" x="699.97345" y="168.464" style="fill: #76b900"/>+ </g>+ </g>+ <g id="plan-sol" data-cost="720.95" data-score="62.8">+ <defs>+ <path id="mabb4518413" d="M 0 8+C 2.121625 8 4.156639 7.15707 5.656854 5.656854+C 7.15707 4.156639 8 2.121625 8 0+C 8 -2.121625 7.15707 -4.156639 5.656854 -5.656854+C 4.156639 -7.15707 2.121625 -8 0 -8+C -2.121625 -8 -4.156639 -7.15707 -5.656854 -5.656854+C -7.15707 -4.156639 -8 -2.121625 -8 0+C -8 2.121625 -7.15707 4.156639 -5.656854 5.656854+C -4.156639 7.15707 -2.121625 8 0 8+z+"/>+ </defs>+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#mabb4518413" x="991.888937" y="146.72" style="fill: #ffc000"/>+ </g>+ </g>+ <g id="advisor-luna-2" data-cost="49" data-score="52.2">+ <g clip-path="url(#p9201e5e723)">+ <use xlink:href="#mabb4518413" x="221.66625" y="306.78" style="fill: #ffc000"/>+ </g>+ </g>+ <g id="text_13">+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="430" y="169" transform="rotate(-0 430 169)">Plan/Execute Router</text>+ </g>+ <g id="text_14">+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="430" y="188" transform="rotate(-0 430 188)">Sol + Luna · 60.8% · $180.51</text>+ </g>+ <g id="text_15">+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="740" y="140" transform="rotate(-0 740 140)">Agent Stage Router</text>+ </g>+ <g id="text_16">+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="740" y="159" transform="rotate(-0 740 159)">Sol + Luna · 61.36% · $466.28</text>+ </g>+ <g id="text_17">+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="450" y="240" transform="rotate(-0 450 240)">Advisor Router</text>+ </g>+ <g id="text_18">+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="450" y="259" transform="rotate(-0 450 259)">Luna + Sol · 57.5% · $208</text>+ </g>+ <g id="text_19">+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle" x="992" y="197" transform="rotate(-0 992 197)">GPT-5.6 Sol Only</text>+ </g>+ <g id="text_20">+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: middle; fill: #777777" x="992" y="216" transform="rotate(-0 992 216)">62.8% · $720.95</text>+ </g>+ <g id="text_21">+ <text style="font-weight: 700; font-size: 18.5px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="285" y="314" transform="rotate(-0 285 314)">GPT-5.6 Luna Only</text>+ </g>+ <g id="text_22">+ <text style="font-size: 14px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start; fill: #777777" x="285" y="333" transform="rotate(-0 285 333)">52.2% · ≈$49</text>+ </g>+ <g id="patch_2">+ <path d="M 425 177+Q 407 177 390.118034 177+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>+ <path d="M 397.518034 180.7+L 390.118034 177+L 397.518034 173.3+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>+ </g>+ <g id="patch_3">+ <path d="M 444 232+Q 431.5 230 420.103992 228.176639+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>+ <path d="M 426.826488 232.999299+L 420.103992 228.176639+L 427.995618 225.692238+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>+ </g>+ <g id="patch_4">+ <path d="M 275 307+Q 256.5 307 239.118034 307+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>+ <path d="M 246.518034 310.7+L 239.118034 307+L 246.518034 303.3+" style="fill: none; stroke-dasharray: 2,2; stroke-dashoffset: 0; stroke: #a6a6a6; stroke-linecap: round"/>+ </g>+ <g id="text_23">+ <text style="font-size: 18px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="379.2" y="522" transform="rotate(-0 379.2 522)">NeMo Switchyard Routing Algorithm</text>+ </g>+ <g id="text_24">+ <text style="font-size: 18px; font-family: 'Arial', 'Helvetica', 'Liberation Sans', 'DejaVu Sans', sans-serif; text-anchor: start" x="718" y="522" transform="rotate(-0 718 522)">Single Model</text>+ </g>+ </g>+ </g>+ <defs>+ <clipPath id="p9201e5e723">+ <rect x="0" y="0" width="1250" height="560"/>+ </clipPath>+ </defs>+</svg>diff --git a/assets/switchyard-cost-accuracy.svg b/docs/assets/switchyard-cost-accuracy.svgsimilarity index 87%rename from assets/switchyard-cost-accuracy.svgrename to docs/assets/switchyard-cost-accuracy.svgindex 03e9a3a34..0114048c6 100644--- a/assets/switchyard-cost-accuracy.svg+++ b/docs/assets/switchyard-cost-accuracy.svg@@ -1,6 +1,6 @@<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1250 560" width="1250" height="560" role="img" aria-labelledby="title description"><title id="title">Switchyard task completion versus cost</title>- <desc id="description">The original classification, stage, escalation, Opus 4.8, and GLM 5.2 points preserve the positions shown in switchyard-cost-accuracy.png. Jev task routing is a separate exploratory recorded-run replay: 73 of 89 tasks completed (82.02%) at an estimated model cost of $92.12, including estimated Jev calls. The combined set includes 62 training tasks. On the reused 27-task test split, Jev solved 22 tasks versus 23 for its paired Opus baseline. The replay used a different evaluation setup and cost accounting from the other plotted runs; its adapted TOML profile needs a fresh evaluation.</desc>+ <desc id="description">The original classification, stage, escalation, Opus 4.8, and GLM 5.2 points preserve the positions shown in the published comparison. Jev task routing is a separate exploratory recorded-run replay: 73 of 89 tasks completed (82.02%) at an estimated model cost of $92.12, including estimated Jev calls. The combined set includes 62 training tasks. On the reused 27-task test split, Jev solved 22 tasks versus 23 for its paired Opus baseline. The replay used a different evaluation setup and cost accounting from the other plotted runs; its adapted TOML profile needs a fresh evaluation.</desc><defs><linearGradient id="frontier" gradientUnits="userSpaceOnUse" x1="157" y1="24" x2="309" y2="176"><stop offset="0" stop-color="#5f8f15"/>@@ -59,7 +59,7 @@<g id="jev-replay" data-total-cost="92.116074918" data-successes="73" data-tasks="89"><circle cx="563.754" cy="52.421" r="9" class="route"/>- <text x="583" y="45" class="label">Jev Task Router</text>+ <text x="583" y="45" class="label">Jev Task Router (replay)</text><text x="583" y="64" class="models">Opus 4.8 and GLM 5.2</text></g>diff --git a/docs/benchmarks.md b/docs/benchmarks.mdnew file mode 100644index 000000000..4a63d0e9d--- /dev/null+++ b/docs/benchmarks.md@@ -0,0 +1,41 @@+# Benchmarks++Task completion, cost, and runnable configs. All configs use `OPENROUTER_API_KEY`.++[Terminal-Bench 2.1](#terminal-bench-21) · [DeepSWE](#deepswe) · [Run a benchmark](#run-a-benchmark)++## Terminal-Bench 2.1++**Claude Opus 4.8 + GLM 5.2**++++| Strategy | Tasks solved | Cost per run | Config |+| --- | ---: | ---: | --- |+| LLM classifier | 79.40% | $109.04 | — |+| Stage | 79.03% | $109.81 | — |+| Escalation | 79.80% | $122.63 | [TOML](../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml) |+| Jev (replay) | 82.02% | ≈$92.12 | [TOML](../benchmark/routing-profiles/tb21-jev-opus-glm.toml) |+| Opus 4.8 only | 77.15% | $148.83 | — |+| GLM 5.2 only | 74.16% | $71.31 | — |++## DeepSWE++**v1.1 · 113 tasks · Codex CLI · GPT-5.6 Luna + Sol**++++| Strategy | Tasks solved | Cost per run | Config |+| --- | ---: | ---: | --- |+| Plan/Execute | 60.8% | $180.51 | [TOML](../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml) |+| Advisor | 57.5% | $208.00 | [TOML](../benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml) |+| Stage | 61.36% | $466.28 | [TOML](../benchmark/routing-profiles/deepswe-v11-stage-router-luna-sol.toml) |+| Sol only | 62.8% | $720.95 | — |+| Luna only | 52.2% | ≈$49 | — |++Luna: one run; others: three-run means. [Evaluation settings](../benchmark/DEEPSWE_V11_QUALIFICATION.md).++## Run a benchmark++- **Terminal-Bench 2.1:** [Setup](../benchmark/README.md#one-time-setup) · [Run](../benchmark/README.md#run-with-switchyard-routing)+- **DeepSWE:** [Run with Pier](../benchmark/README.md#deepswe-v11)diff --git a/docs/index.md b/docs/index.mdindex e456b6b98..67c220ea0 100644--- a/docs/index.md+++ b/docs/index.md@@ -18,6 +18,7 @@ The Server Path builds and runs the standalone `switchyard-server` binary.## Explore+- [Benchmarks](benchmarks.md): compare Terminal-Bench 2.1 and DeepSWE results and open their routing configs- [Core Concepts](core_concepts.md): learn the LLM client, target, and route layers- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model- [Architecture](architecture.md): understand the proxy and library componentsdiff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.mdindex bb4c83d5c..519942617 100644--- a/docs/routing_algorithms/model_selection.md+++ b/docs/routing_algorithms/model_selection.md@@ -129,6 +129,9 @@ behave differently on different endpoints. See## Measured pairs+See [Benchmarks](../benchmarks.md) for consolidated Terminal-Bench 2.1 and DeepSWE results,+config links, and evaluation settings.+These pairs have results from checked-in profiles or a public report. Each rowdescribes one pair on one workload. The thresholds and results do not transferto another pair or another workload. The profile headers give the serving stack,@@ -137,17 +140,13 @@ serving stack.| Efficient tier | Capable tier | Strategy | Workload | Result | Single-model anchors ||---|---|---|---|---|---|-| GPT-5.6 Luna | GPT-5.6 Sol | [`stage_router`](../../benchmark/routing-profiles/deepswe-v11-stage-router-luna-sol.toml): `efficient_first`, threshold 0.5, window 5, hold 2, handoff note | DeepSWE v1.1, 113 tasks, Codex CLI, k=1 | 76/113 solved (67.3% strict) | None in the same run |-| GPT-5.6 Luna, executes | GPT-5.6 Sol, plans | [`plan_execute`](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml), both at effort `max` | DeepSWE v1.1, 113 tasks, Codex CLI, k=3 | 60.8% ± 5.3 at $180.51 ± 9.43 per run | Sol alone 62.8% ± 3.1 at $720.95. Luna alone 51.0% ± 3.7 at $50.02. |-| GPT-5.6 Luna, executes | GPT-5.6 Sol, reviews | [`advisor`](../../benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml), both at effort `max` | DeepSWE v1.1, 113 tasks, Codex CLI, k=3 | 57.5% ± 3.2 at $208 ± 22 per run | Sol alone 62.8% ± 3.1 at $743. Luna alone 46.9% and 52.2% at about $49. || Qwen3.6-35B-A3B, also the judge | Claude Opus 4.7 | `llm_classifier` custom, [balanced](../../benchmark/routing-profiles/tau2-telecom-custom-opus-qwen-balanced.toml) and [aggressive](../../benchmark/routing-profiles/tau2-telecom-custom-opus-qwen-aggressive.toml) | tau2-bench telecom, 114 tasks | 0.903 ± 0.071 solve rate with 45% of turns on the weak tier. 0.891 ± 0.029 with about 85%. | Within noise of the strong tier, per the profile headers || Nemotron 3.5 Lightning | Claude Opus 4.8, with Gemini 3.1 Flash-Lite as judge | `llm_classifier` escalation, `confirmations = 2` | LangChain Deep Agents suite, 145 tasks, 5 runs ([report](https://www.langchain.com/blog/switchyard-agent-routing-benchmark)) | 80.0% at $3.00 per run, with 7% of calls on Opus | Opus alone 86.0% at $11.45. Nemotron alone 77.7% at $0.72. |What the rows show about model choice:-- The gap between the two single models sets the room for routing. On DeepSWE,- Sol alone beat Luna alone by about 11 to 16 points. The plan/execute and- advisor arms landed between them at about a quarter of Sol's cost.+- The gap between the two single models sets the room for routing. Compare the+ fixed-model baselines with the routing results, as in the [DeepSWE studies](../benchmarks.md#deepswe).- When the efficient model is close to the capable model, escalation keeps mostof the accuracy at a fraction of the cost. On the LangChain suite the twosingle models were 8 points apart. Routing kept 80 of the 86 points at about@@ -160,10 +159,6 @@ Other checked-in configurations:- [`tb-lite-llm-classifier-opus-kimi-gemini.toml`](../../benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml)(Claude Opus 4.7 strong, Kimi K2.7 code weak, Gemini 3.5 Flash judge) is asmoke test. It has no published result.-- [`tb21-escalation-opus-glm-deepseek.toml`](../../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml)- (Claude Opus 4.8 strong, GLM 5.2 weak, DeepSeek V4 Flash judge) is the- escalation configuration used for the v0.2.0 Terminal-Bench 2.1 runs. Its- header links the post about those runs.## Counter-examplesdiff --git a/mkdocs.yml b/mkdocs.ymlindex 6c5309ffc..8d6f51518 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -20,6 +20,7 @@ nav:- Get Started:- Getting Started: getting_started.md- Known Issues: known_issues.md+ - Benchmarks: benchmarks.md- Concepts:- Core Concepts: core_concepts.md- Architecture: architecture.md