Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(benchmark): add DeepSWE v1.1 Harbor/Pier workflow (#701)

ryan-lempka3807ff0f4bd0
diff --git a/benchmark/README.md b/benchmark/README.md
index dfaa36a52..7a04e7efd 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -302,6 +302,60 @@ SWITCHYARD_DOCKER_BUILD=0 bash benchmark/run-baseline.sh ...
Only reuse the image when you know it already contains the current Rust `switchyard-server` binary.
+## DeepSWE v1.1
+
+DeepSWE uses Harbor's task format but its own runner, [Pier](https://github.com/datacurve-ai/pier)
+(required since v1.1 for the separate-verifier/collect-hook pattern). It is not part of
+`run-baseline.sh`.
+
+```bash
+git clone https://github.com/datacurve-ai/deep-swe benchmark/datasets/deep-swe
+uv tool install 'datacurve-pier>0.3.0'
+```
+
+Direct upstream:
+
+```bash
+export OPENAI_API_KEY="..."
+pier run -p benchmark/datasets/deep-swe/tasks --agent mini-swe-agent --model openai/gpt-5.5
+```
+
+Switchyard routing: start the server, then point the agent's OpenAI client at it through `--ae`.
+Pier's Docker environment routes all agent traffic through a policy proxy whose `Safe_ports` ACL
+only allows ports 80/443, so bind Switchyard to 443:
+
+```bash
+switchyard-server --config benchmark/server-configs/deepswe-single.toml \
+ --host 0.0.0.0 --port 443
+
+pier run -p benchmark/datasets/deep-swe/tasks --agent mini-swe-agent \
+ --model openai/deepswe-single \
+ --ae OPENAI_BASE_URL=http://host.docker.internal:443/v1 \
+ --ae OPENAI_API_KEY=unused
+```
+
+`--agent codex` reads the same `OPENAI_BASE_URL`/`OPENAI_API_KEY` pair through `--ae`:
+
+```bash
+pier run -p benchmark/datasets/deep-swe/tasks --agent codex \
+ --model openai/deepswe-single \
+ --ae OPENAI_BASE_URL=http://host.docker.internal:443/v1 \
+ --ae OPENAI_API_KEY=unused
+```
+
+`host.docker.internal` requires Docker Desktop; on Linux, pass the host's Docker-bridge address
+instead. Binding port 443 needs elevated privileges on most Linux hosts (`sudo`, or
+`setcap 'cap_net_bind_service=+ep'` on the binary).
+
+Smoke subset:
+
+```bash
+pier run -p benchmark/datasets/deep-swe/tasks --agent mini-swe-agent \
+ --model openai/gpt-5.5 --n-tasks 1 --sample-seed 0
+```
+
+Results land under `jobs/<job-name>/`, per Pier's own layout.
+
## Troubleshooting
If the runner reports that the current Harbor patch is not applied cleanly, recreate or reinstall the
diff --git a/benchmark/server-configs/deepswe-single.toml b/benchmark/server-configs/deepswe-single.toml
new file mode 100644
index 000000000..4356c29c2
--- /dev/null
+++ b/benchmark/server-configs/deepswe-single.toml
@@ -0,0 +1,18 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_responses"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.primary]
+id = "openai/gpt-5.5"
+llm_client = "openrouter"
+
+[routes.primary]
+id = "deepswe-single"
+type = "passthrough"
+target = "primary"