Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(benchmark): publish DeepSWE qualification settings (#814)
Bhuvan Agrawal06e88c017a43
diff --git a/benchmark/DEEPSWE_V11_QUALIFICATION.md b/benchmark/DEEPSWE_V11_QUALIFICATION.mdnew file mode 100644index 000000000..245b2b3f6--- /dev/null+++ b/benchmark/DEEPSWE_V11_QUALIFICATION.md@@ -0,0 +1,43 @@+# DeepSWE v1.1 qualification settings++Routing configurations:++- [Advisor gate: Luna executor with Sol reviews](routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml)+- [Stage router: Luna efficient tier with Sol capable tier](routing-profiles/deepswe-v11-stage-router-luna-sol.toml)++## Shared settings++| Setting | Value |+| --- | --- |+| Workload | DeepSWE v1.1, all 113 tasks, benchmark revision 1 |+| Agent | Codex CLI 0.154.0 |+| Harness | Harbor 0.13.2 |+| Protocol | OpenAI Responses |+| Network | Closed book with public network access denied |+| Agent budget | 3 hours |+| Command timeout | 10,800 seconds |+| Sandbox timeout | 8,400 seconds |+| Lifecycle timeout | 21,600 seconds |+| Infrastructure retries | 2 |+| Step and cost limits | Disabled |+| Strict scoring | Reward `1` divided by 113; failures and missing results score zero |+| Pricing snapshot | OpenRouter list prices from 2026-09-09 |+| Pricing SHA-256 | `945490e47f9a4d5a234a7ee95b2db7a74bf88d32ced68d3d664dcb36a94bef19` |++## Routing-specific settings++| Setting | Advisor gate | Stage router |+| --- | --- | --- |+| Switchyard revision | `832374ac` | `6a98efdbb63304ff1625f09a240a32381e396d28` |+| Configuration schema | 1 | 1 |+| Public route ID | `switchyard` | `gpt-5.6-luna` |+| Luna role | Executor, reasoning effort `max` | Efficient tier |+| Sol role | Advisor, reasoning effort `max` | Capable tier |+| Concurrency | 10 tasks | 5 tasks |+| Independent runs | 3 | 1 |+| Reported score | Mean and sample standard deviation | One strict score |++The linked files contain the exact routing thresholds, turn windows, review limits, handoff+settings, and model IDs. Their provider blocks use OpenRouter so they can run outside the original+evaluation environment. The qualification runs used a different serving stack, so absolute results+can differ even when the routing settings are unchanged.diff --git a/benchmark/README.md b/benchmark/README.mdindex 1d3893e45..8ee1ba9df 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -311,6 +311,9 @@ DeepSWE uses Harbor's task format but its own runner, [Pier](https://github.com/(required since v1.1 for the separate-verifier/collect-hook pattern). It is not part of`run-baseline.sh`.+See [DeepSWE v1.1 qualification settings](DEEPSWE_V11_QUALIFICATION.md) for the exact+versions, timeouts, scoring rules, and routing profiles used for qualification.+```bashgit clone https://github.com/datacurve-ai/deep-swe benchmark/datasets/deep-sweuv tool install 'datacurve-pier>0.3.0'