Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(benchmark): add terminal-bench 2.1 dataset support (#45)
Ayush Agarwal8fba1be9093d
diff --git a/benchmark/DATASETS.md b/benchmark/DATASETS.mdindex 1cc277be9..a60d6b7ea 100644--- a/benchmark/DATASETS.md+++ b/benchmark/DATASETS.md@@ -12,6 +12,21 @@ benchmark/datasets/openthoughts-tblite-closed-bookUse this page when you want to prepare a different Harbor dataset for the sameSwitchyard benchmark runner.+## Supported Datasets++These Harbor datasets are supported out of the box — `prepare_harbor_dataset.py` recognizes each+one and generates the right closed-book proxy allowlist automatically:++| Dataset | Harbor id | Closed-book allowlist |+|---------|-----------|-----------------------|+| TB Lite (default) | `[email protected]` | none beyond the closed-book base |+| Terminal-Bench 2.0 | `terminal-bench/terminal-bench-2` | TB2 Oracle package/data sources |+| Terminal-Bench 2.1 | `terminal-bench/terminal-bench-2-1` | shares the TB2 Oracle allowlist |+| SWE-Bench Pro | `cais/swebenchpro` | none beyond the closed-book base |++Any other Harbor dataset that matches the [supported shape](#supported-shape) can still be prepared;+it just receives no dataset-specific allowlist. Generate commands for each are below.+## Supported Shape`prepare_harbor_dataset.py` is a repo-local dataset rewriter, not a published Harbor adapter. It can@@ -36,6 +51,16 @@ uv run --no-sync python benchmark/prepare_harbor_dataset.py \--overwrite```+Terminal-Bench 2.1 is the verified iteration of 2.0. It shares the same task family and Oracle+egress, so the generated proxy allowlist matches 2.0:++```bash+uv run --no-sync python benchmark/prepare_harbor_dataset.py \+ --source-dataset terminal-bench/terminal-bench-2-1 \+ --output-dir benchmark/datasets/terminal-bench-2-1-closed-book \+ --overwrite+```+Download and rewrite SWE-Bench Pro from Harbor. The Harbor dataset id is `cais/swebenchpro`; thegenerated dataset keeps the closed-book proxy topology without opening dataset-specific agentegress:diff --git a/benchmark/README.md b/benchmark/README.mdindex 834b32f63..dd6bff538 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -119,6 +119,16 @@ uv run --no-sync python benchmark/prepare_harbor_dataset.py \--overwrite```+Terminal-Bench 2.1 (the verified iteration of 2.0) is supported the same way and shares the 2.0+Oracle allowlist:++```bash+uv run --no-sync python benchmark/prepare_harbor_dataset.py \+ --source-dataset terminal-bench/terminal-bench-2-1 \+ --output-dir benchmark/datasets/terminal-bench-2-1-closed-book \+ --overwrite+```+SWE-Bench Pro is supported with the Harbor dataset `cais/swebenchpro`. The generated dataset usesthe same pinned-agent and closed-book proxy path without opening dataset-specific agent egress.diff --git a/benchmark/prepare_harbor_dataset.py b/benchmark/prepare_harbor_dataset.pyindex 9e5b73457..d87e8f1d7 100644--- a/benchmark/prepare_harbor_dataset.py+++ b/benchmark/prepare_harbor_dataset.py@@ -30,6 +30,9 @@ AGENT_VERSIONS_FILE = SCRIPT_DIR / "agent-versions.env"PROXY_ASSET_DIR = SCRIPT_DIR / "closed_book_proxy" / "proxy"AGENT_ENTRYPOINT = "switchyard-agent-entrypoint.sh"TERMINAL_BENCH_2_SOURCE_DATASET = "terminal-bench/terminal-bench-2"+TERMINAL_BENCH_2_1_SOURCE_DATASET = "terminal-bench/terminal-bench-2-1"+# Shared across the TB2 family (2.0 + the 2.1 verified iteration): 2.1 tweaks+# timeouts/resources on existing tasks, so its Oracle solutions reach the same hosts.TERMINAL_BENCH_2_PROXY_ALLOWLIST_HOSTS = ("api.github.com","api.launchpad.net",@@ -128,7 +131,7 @@ def _exported_dataset_candidates(download_root: Path, source_dataset: str) -> lidef _proxy_allowlist_hosts_for_dataset(source_dataset: str) -> tuple[str, ...]:source_name = source_dataset.split("@", 1)[0]- if source_name == TERMINAL_BENCH_2_SOURCE_DATASET:+ if source_name in (TERMINAL_BENCH_2_SOURCE_DATASET, TERMINAL_BENCH_2_1_SOURCE_DATASET):return TERMINAL_BENCH_2_PROXY_ALLOWLIST_HOSTSreturn ()diff --git a/tests/test_prepare_harbor_dataset.py b/tests/test_prepare_harbor_dataset.pyindex 11f3c36ac..e30048b33 100644--- a/tests/test_prepare_harbor_dataset.py+++ b/tests/test_prepare_harbor_dataset.py@@ -116,6 +116,29 @@ def test_terminal_bench_2_dataset_adds_proxy_allowlist_hosts(tmp_path: Path) ->assert host in manifest["closed_book"]["proxy_allowlist_hosts"]+def test_terminal_bench_2_1_dataset_reuses_terminal_bench_2_allowlist(tmp_path: Path) -> None:+ source = tmp_path / "source"+ _write_task(source, "tb21-task", "[environment]\n", "FROM ubuntu:22.04\n")++ output = _prepare(+ tmp_path,+ source,+ source_dataset="terminal-bench/terminal-bench-2-1",+ )+ allowlist = (output / "tb21-task" / "environment" / "proxy" / "allowlist-base.txt").read_text()+ manifest = json.loads((output / "switchyard_dataset_manifest.json").read_text())++ for host in (+ "archive.ubuntu.com",+ "pypi.org",+ "github.com",+ "huggingface.co",+ "download.pytorch.org",+ ):+ assert host in allowlist+ assert host in manifest["closed_book"]["proxy_allowlist_hosts"]++def test_swe_bench_pro_dataset_keeps_agent_proxy_allowlist_empty(tmp_path: Path) -> None:source = tmp_path / "source"_write_task(source, "swe-task", "[environment]\n", "FROM ubuntu:22.04\n")