Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(benchmark): add DeepSWE plan-execute profile (#824)

ryan-lempkaa30ee2c8b37b
diff --git a/benchmark/DEEPSWE_V11_QUALIFICATION.md b/benchmark/DEEPSWE_V11_QUALIFICATION.md
index 245b2b3f6..b0755ef5d 100644
--- a/benchmark/DEEPSWE_V11_QUALIFICATION.md
+++ b/benchmark/DEEPSWE_V11_QUALIFICATION.md
@@ -3,6 +3,7 @@
Routing configurations:
- [Advisor gate: Luna executor with Sol reviews](routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml)
+- [Plan/execute: Sol planner with Luna executor](routing-profiles/deepswe-v11-plan-execute-luna-sol.toml)
- [Stage router: Luna efficient tier with Sol capable tier](routing-profiles/deepswe-v11-stage-router-luna-sol.toml)
## Shared settings
@@ -21,23 +22,23 @@ Routing configurations:
| Infrastructure retries | 2 |
| Step and cost limits | Disabled |
| Strict scoring | Reward `1` divided by 113; failures and missing results score zero |
-| Pricing snapshot | OpenRouter list prices from 2026-09-09 |
+| Pricing snapshot | OpenRouter standard list prices from 2026-09-09, promotional pricing excluded |
| Pricing SHA-256 | `945490e47f9a4d5a234a7ee95b2db7a74bf88d32ced68d3d664dcb36a94bef19` |
## Routing-specific settings
-| Setting | Advisor gate | Stage router |
-| --- | --- | --- |
-| Switchyard revision | `832374ac` | `6a98efdbb63304ff1625f09a240a32381e396d28` |
-| Configuration schema | 1 | 1 |
-| Public route ID | `switchyard` | `gpt-5.6-luna` |
-| Luna role | Executor, reasoning effort `max` | Efficient tier |
-| Sol role | Advisor, reasoning effort `max` | Capable tier |
-| Concurrency | 10 tasks | 5 tasks |
-| Independent runs | 3 | 1 |
-| Reported score | Mean and sample standard deviation | One strict score |
+| Setting | Advisor gate | Plan/execute | Stage router |
+| --- | --- | --- | --- |
+| Switchyard revision | `832374ac` | `08136d7d2ed4f862e5b3dd83488b3f120457d0ac` | `6a98efdbb63304ff1625f09a240a32381e396d28` |
+| Configuration schema | 1 | 1 | 1 |
+| Public route ID | `switchyard` | `switchyard` | `gpt-5.6-luna` |
+| Luna role | Executor, reasoning effort `max` | Executor, reasoning effort `max` | Efficient tier |
+| Sol role | Advisor, reasoning effort `max` | Planner, reasoning effort `max` | Capable tier |
+| Concurrency | 10 tasks | 5, 5, and 8 tasks | 5 tasks |
+| Independent runs | 3 | 3 | 1 |
+| Reported score | Mean and sample standard deviation | Mean and sample standard deviation | One strict score |
-The linked files contain the exact routing thresholds, turn windows, review limits, handoff
-settings, and model IDs. Their provider blocks use OpenRouter so they can run outside the original
-evaluation environment. The qualification runs used a different serving stack, so absolute results
-can differ even when the routing settings are unchanged.
+The linked files contain the exact routing parameters, prompts, and model IDs. Their provider
+blocks use OpenRouter so they can run outside the original evaluation environment. The qualification
+runs used a different serving stack, so absolute results can differ even when the routing settings
+are unchanged.
diff --git a/benchmark/README.md b/benchmark/README.md
index 8ee1ba9df..0876f73d0 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -371,6 +371,24 @@ pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
--ae OPENAI_API_KEY=unused
```
+Plan/execute routing uses GPT-5.6 Sol for repository inspection and planning, then hands the
+full trajectory to GPT-5.6 Luna after the first mutation. It measured 60.8% +/- 5.3 at
+$180.51 +/- 9.43 per run on the full closed-book benchmark (k=3). The profile preserves the
+exact planning prompt and routing policy, with provider settings adapted from NVIDIA Inference
+Hub to OpenRouter:
+
+```bash
+export OPENROUTER_API_KEY="..."
+switchyard-server \
+ --config benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml \
+ --host 0.0.0.0 --port 443
+
+pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
+ --model openai/switchyard \
+ --ae OPENAI_BASE_URL=http://host.docker.internal:443/v1 \
+ --ae OPENAI_API_KEY=unused
+```
+
The qualified stage-router profile (GPT-5.6 Luna efficient tier, GPT-5.6 Sol capable tier) solved
76/113 tasks (67.3% strict) on the full closed-book benchmark. Its routing policy is published
with OpenRouter provider settings so the file runs as-is with `OPENROUTER_API_KEY`. The header
diff --git a/benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml b/benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml
new file mode 100644
index 000000000..24197ac62
--- /dev/null
+++ b/benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml
@@ -0,0 +1,56 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Plan/execute routing for DeepSWE v1.1: GPT-5.6 Sol inspects the repository and
+# plans through read-only turns. Its first mutation hands the full trajectory to
+# GPT-5.6 Luna, which executes the plan for the rest of the session.
+# See docs/routing_algorithms/plan_execute_routing.md.
+#
+# Measured on:
+# Switchyard commit 08136d7d2ed4f862e5b3dd83488b3f120457d0ac,
+# config schema version 1
+# Planner OpenAI GPT-5.6 Sol, reasoning effort `max`
+# Executor OpenAI GPT-5.6 Luna, reasoning effort `max`
+# Benchmark DeepSWE v1.1 (github.com/datacurve-ai/deep-swe), all 113 tasks,
+# closed-book with sandbox network policy `default_deny`
+# Agent Codex CLI 0.154.0 calling this route with model name
+# `switchyard`; Harbor 0.13.2 ran 5, 5, and 8 tasks at a time
+# Scoring strict: reward 1 over all 113 tasks regardless of evaluation
+# status. k=3 independent runs, mean +/- sample standard deviation
+# Result 60.8% +/- 5.3 solved at $180.51 +/- 9.43 per run
+# Anchors same harness and conditions, k=3: Sol alone 62.8% +/- 3.1 at
+# $720.95 +/- 19.47 per run. Luna alone 51.0% +/- 3.7 at
+# $50.02 +/- 1.04 per run
+# Pricing OpenRouter standard per-token list prices frozen 2026-09-09,
+# promotional pricing excluded
+#
+# The routing parameters below are exactly as run. The measured runs reached the
+# same models through NVIDIA Inference Hub. This profile uses OpenRouter so it can
+# run as written with an `OPENROUTER_API_KEY`. Absolute results may vary by serving
+# stack.
+#
+# switchyard-server --config benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml --port 4000
+
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_responses"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.planner]
+id = "openai/gpt-5.6-sol"
+llm_client = "openrouter"
+reasoning_effort = "max"
+
+[targets.executor]
+id = "openai/gpt-5.6-luna"
+llm_client = "openrouter"
+reasoning_effort = "max"
+
+[routes.switchyard]
+id = "switchyard"
+type = "plan_execute"
+capable_target = "planner"
+efficient_target = "executor"
+planning_prompt = "You are in the planning phase. Inspect the task and the relevant code, then form a concrete implementation plan before modifying any files. Use read-only tools as needed. Do not make an edit until the plan is complete. After completing the plan, immediately make the first edit in the same task to hand execution off. Do not finish or return a final response with only the plan."