Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(benchmark): add terminal-bench 2.1 dataset support (#45)

Ayush Agarwal8fba1be9093d
diff --git a/benchmark/DATASETS.md b/benchmark/DATASETS.md
index 1cc277be9..a60d6b7ea 100644
--- a/benchmark/DATASETS.md
+++ b/benchmark/DATASETS.md
@@ -12,6 +12,21 @@ benchmark/datasets/openthoughts-tblite-closed-book
Use this page when you want to prepare a different Harbor dataset for the same
Switchyard benchmark runner.
+## Supported Datasets
+
+These Harbor datasets are supported out of the box — `prepare_harbor_dataset.py` recognizes each
+one and generates the right closed-book proxy allowlist automatically:
+
+| Dataset | Harbor id | Closed-book allowlist |
+|---------|-----------|-----------------------|
+| TB Lite (default) | `[email protected]` | none beyond the closed-book base |
+| Terminal-Bench 2.0 | `terminal-bench/terminal-bench-2` | TB2 Oracle package/data sources |
+| Terminal-Bench 2.1 | `terminal-bench/terminal-bench-2-1` | shares the TB2 Oracle allowlist |
+| SWE-Bench Pro | `cais/swebenchpro` | none beyond the closed-book base |
+
+Any other Harbor dataset that matches the [supported shape](#supported-shape) can still be prepared;
+it just receives no dataset-specific allowlist. Generate commands for each are below.
+
## Supported Shape
`prepare_harbor_dataset.py` is a repo-local dataset rewriter, not a published Harbor adapter. It can
@@ -36,6 +51,16 @@ uv run --no-sync python benchmark/prepare_harbor_dataset.py \
--overwrite
```
+Terminal-Bench 2.1 is the verified iteration of 2.0. It shares the same task family and Oracle
+egress, so the generated proxy allowlist matches 2.0:
+
+```bash
+uv run --no-sync python benchmark/prepare_harbor_dataset.py \
+ --source-dataset terminal-bench/terminal-bench-2-1 \
+ --output-dir benchmark/datasets/terminal-bench-2-1-closed-book \
+ --overwrite
+```
+
Download and rewrite SWE-Bench Pro from Harbor. The Harbor dataset id is `cais/swebenchpro`; the
generated dataset keeps the closed-book proxy topology without opening dataset-specific agent
egress:
diff --git a/benchmark/README.md b/benchmark/README.md
index 834b32f63..dd6bff538 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -119,6 +119,16 @@ uv run --no-sync python benchmark/prepare_harbor_dataset.py \
--overwrite
```
+Terminal-Bench 2.1 (the verified iteration of 2.0) is supported the same way and shares the 2.0
+Oracle allowlist:
+
+```bash
+uv run --no-sync python benchmark/prepare_harbor_dataset.py \
+ --source-dataset terminal-bench/terminal-bench-2-1 \
+ --output-dir benchmark/datasets/terminal-bench-2-1-closed-book \
+ --overwrite
+```
+
SWE-Bench Pro is supported with the Harbor dataset `cais/swebenchpro`. The generated dataset uses
the same pinned-agent and closed-book proxy path without opening dataset-specific agent egress.
diff --git a/benchmark/prepare_harbor_dataset.py b/benchmark/prepare_harbor_dataset.py
index 9e5b73457..d87e8f1d7 100644
--- a/benchmark/prepare_harbor_dataset.py
+++ b/benchmark/prepare_harbor_dataset.py
@@ -30,6 +30,9 @@ AGENT_VERSIONS_FILE = SCRIPT_DIR / "agent-versions.env"
PROXY_ASSET_DIR = SCRIPT_DIR / "closed_book_proxy" / "proxy"
AGENT_ENTRYPOINT = "switchyard-agent-entrypoint.sh"
TERMINAL_BENCH_2_SOURCE_DATASET = "terminal-bench/terminal-bench-2"
+TERMINAL_BENCH_2_1_SOURCE_DATASET = "terminal-bench/terminal-bench-2-1"
+# Shared across the TB2 family (2.0 + the 2.1 verified iteration): 2.1 tweaks
+# timeouts/resources on existing tasks, so its Oracle solutions reach the same hosts.
TERMINAL_BENCH_2_PROXY_ALLOWLIST_HOSTS = (
"api.github.com",
"api.launchpad.net",
@@ -128,7 +131,7 @@ def _exported_dataset_candidates(download_root: Path, source_dataset: str) -> li
def _proxy_allowlist_hosts_for_dataset(source_dataset: str) -> tuple[str, ...]:
source_name = source_dataset.split("@", 1)[0]
- if source_name == TERMINAL_BENCH_2_SOURCE_DATASET:
+ if source_name in (TERMINAL_BENCH_2_SOURCE_DATASET, TERMINAL_BENCH_2_1_SOURCE_DATASET):
return TERMINAL_BENCH_2_PROXY_ALLOWLIST_HOSTS
return ()
diff --git a/tests/test_prepare_harbor_dataset.py b/tests/test_prepare_harbor_dataset.py
index 11f3c36ac..e30048b33 100644
--- a/tests/test_prepare_harbor_dataset.py
+++ b/tests/test_prepare_harbor_dataset.py
@@ -116,6 +116,29 @@ def test_terminal_bench_2_dataset_adds_proxy_allowlist_hosts(tmp_path: Path) ->
assert host in manifest["closed_book"]["proxy_allowlist_hosts"]
+def test_terminal_bench_2_1_dataset_reuses_terminal_bench_2_allowlist(tmp_path: Path) -> None:
+ source = tmp_path / "source"
+ _write_task(source, "tb21-task", "[environment]\n", "FROM ubuntu:22.04\n")
+
+ output = _prepare(
+ tmp_path,
+ source,
+ source_dataset="terminal-bench/terminal-bench-2-1",
+ )
+ allowlist = (output / "tb21-task" / "environment" / "proxy" / "allowlist-base.txt").read_text()
+ manifest = json.loads((output / "switchyard_dataset_manifest.json").read_text())
+
+ for host in (
+ "archive.ubuntu.com",
+ "pypi.org",
+ "github.com",
+ "huggingface.co",
+ "download.pytorch.org",
+ ):
+ assert host in allowlist
+ assert host in manifest["closed_book"]["proxy_allowlist_hosts"]
+
+
def test_swe_bench_pro_dataset_keeps_agent_proxy_allowlist_empty(tmp_path: Path) -> None:
source = tmp_path / "source"
_write_task(source, "swe-task", "[environment]\n", "FROM ubuntu:22.04\n")