Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(benchmark): add DeepSWE v1.1 advisor-gate routing profile (#811)

eric-liu-nvidia3f32fabb16d2
diff --git a/benchmark/README.md b/benchmark/README.md
index 9d32b55fd..1afef7220 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -350,6 +350,24 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
instead. Binding port 443 needs elevated privileges on most Linux hosts (`sudo`, or
`setcap 'cap_net_bind_service=+ep'` on the binary).
+Advisor-gate routing (GPT-5.6 Luna executes; GPT-5.6 Sol reviews its "done" claims, up to
+three per task) measured 57.5% +/- 3.2 on the full 113 tasks (k=3, closed-book). The profile
+keeps the routing parameters exactly as run and reaches both models through OpenRouter, so it
+needs only `OPENROUTER_API_KEY`. The measured runs had Codex at reasoning effort `max`
+(`model_reasoning_effort = "max"` in the agent's Codex config); the profile forces `max` on
+both models server-side regardless:
+
+```bash
+export OPENROUTER_API_KEY="..."
+switchyard-server --config benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml \
+ --host 0.0.0.0 --port 443
+
+pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
+ --model openai/switchyard \
+ --ae OPENAI_BASE_URL=http://host.docker.internal:443/v1 \
+ --ae OPENAI_API_KEY=unused
+```
+
Smoke subset:
```bash
diff --git a/benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml b/benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml
new file mode 100644
index 000000000..85df8b4cb
--- /dev/null
+++ b/benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml
@@ -0,0 +1,65 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Advisor-gate routing for DeepSWE v1.1: a fast executor serves every turn. Once the
+# executor has at least three tool results, a stronger reviewer checks its "done" claims,
+# up to three per task, and either approves or sends it back with a concrete redo plan.
+# See docs/routing_algorithms/advisor_gate_routing.md.
+#
+# Measured on:
+# Switchyard main at 832374ac, config schema version 1
+# Executor OpenAI GPT-5.6 Luna, reasoning effort `max`
+# Advisor OpenAI GPT-5.6 Sol, reasoning effort `max`
+# Benchmark DeepSWE v1.1 (github.com/datacurve-ai/deep-swe), all 113 tasks,
+# closed-book: sandbox egress limited to this server, 3-hour agent
+# budget, no step or cost limit
+# Agent Codex CLI 0.154.0 with `model_reasoning_effort = "max"`, calling
+# this server with model name `switchyard`; Harbor 0.13.2 ran the
+# sandboxes, 10 tasks at a time
+# Scoring strict: tasks with reward 1 over all 113, so a sandbox failure counts
+# as unsolved; k=3 independent runs, mean +/- sample standard deviation
+# Result 57.5% +/- 3.2 solved at $208 +/- 22 per run (about $3.20 per solve);
+# 8 of 339 attempts failed, all sandbox out-of-memory, none timed out
+# Anchors same harness and conditions, k=3: Sol alone 62.8% +/- 3.1 at $743 per
+# run; Luna alone 46.9% and 52.2% (two valid runs) at about $49 per run
+# Pricing per-token list prices frozen 2026-09-09
+#
+# The routing parameters below are exactly as run. The model wiring is not: the measured
+# runs reached the same two models through a different serving stack, replaced here with
+# OpenRouter so the file runs as written with an OpenRouter key. Treat the figures above
+# as this pair's operating point, not a guarantee for every serving stack.
+#
+# switchyard-server --config benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml --port 4000
+
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_responses"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+max_retries = 2
+
+[targets.executor]
+id = "openai/gpt-5.6-luna"
+llm_client = "openrouter"
+reasoning_effort = "max"
+
+[targets.advisor]
+id = "openai/gpt-5.6-sol"
+llm_client = "openrouter"
+reasoning_effort = "max"
+
+# Luna serves every client-visible turn. When it stops calling tools (after at least three
+# tool results) or grinds for 30 turns, Sol reviews the transcript, trimmed from the middle
+# past 200k characters, and either approves or returns a redo plan that Luna must act on;
+# at most three reviews per session.
+[routes.switchyard]
+id = "switchyard"
+type = "advisor"
+executor_target = "executor"
+advisor_target = "advisor"
+max_reviews = 3
+gate_stall_turns = 30
+gate_min_tool_results = 3
+advisor_max_tokens = 32768
+fail_open = false