Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

benchmark: a search tool is not the network, outside is everything but the workspace, and the grader's engine gets a profile per call

richard-epsillaac11a0280a67
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 77174c4b..bd0ab8a3 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -43,6 +43,9 @@ jobs:
# against the spec schema. It stays out of requirements.txt so it never ships in the image.
- run: pip install -r gateway/requirements.txt pytest pytest-asyncio jsonschema
- run: python -m pytest gateway/tests -q
+ # the benchmark suite's rules are pinned by its tests; they read the gateway's vendor tables
+ # from source (scripts/support-matrix/samemodel.py) and need nothing the gateway job lacks
+ - run: python -m pytest scripts/benchmark -q
runner:
runs-on: ubuntu-latest
diff --git a/docs/benchmark.md b/docs/benchmark.md
index ce657124..b187f25b 100644
--- a/docs/benchmark.md
+++ b/docs/benchmark.md
@@ -2,16 +2,16 @@
How a row is measured, and the rules that decide it, are in [scripts/benchmark/README.md](../scripts/benchmark/README.md). One task is one session; the pack's own grader decides; every number is read from the turn record.
-## Provider: deepseek — pack: spreadsheetbench
+## Provider: deepseek, pack: spreadsheetbench
| Harness | Model | Tasks | Resolved | Reward | Wall (sum) | Median wall | Tool calls | Calls failed | Fresh in | Cached in | Output | Served by | Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
-| cline | deepseek-v4.1-flash | 47 | 36 (77%) | 0.77 | 6882 s | 108 s | 736 | 121 (16%) | 1.20M | 19.74M | 1.21M | deepseek | 2 runs the pack could not grade (the task fails its own grader), left out ; served model unreported on 50 of 50 runs (rule 2 unverifiable there) ; 1 runs are findings, not counted |
+| cline | deepseek-v4.1-flash | 47 | 36 (77%) | 0.77 | 6882 s | 108 s | 736 | 121 (16%) | 1.20M | 19.74M | 1.21M | deepseek | 2 runs the pack could not grade (the task fails its own grader), left out ; served model unreported on 50 of 50 runs (rule 2 unverifiable there) ; 1 run is a finding, not counted |
| dsh | deepseek-v4.1-flash | 46 | 36 (78%) | 0.78 | 6506 s | 100 s | 721 | 113 (16%) | 761k | 12.32M | 752k | deepseek | 2 runs the pack could not grade (the task fails its own grader), left out ; served model unreported on 50 of 50 runs (rule 2 unverifiable there) ; 2 runs are findings, not counted |
| opencode | deepseek-v4.1-flash | 48 | 39 (81%) | 0.81 | 5515 s | 75 s | 563 | 67 (12%) | 666k | 11.64M | 709k | deepseek | 2 runs the pack could not grade (the task fails its own grader), left out ; served model unreported on 1 of 50 runs (rule 2 unverifiable there) |
-| pi | deepseek-v4.1-flash | 47 | 40 (85%) | 0.85 | 3605 s | 48 s | 526 | 89 (17%) | 484k | 7.53M | 546k | deepseek | 2 runs the pack could not grade (the task fails its own grader), left out ; served model unreported on 1 of 50 runs (rule 2 unverifiable there) ; 1 runs are findings, not counted |
+| pi | deepseek-v4.1-flash | 47 | 40 (85%) | 0.85 | 3605 s | 48 s | 526 | 89 (17%) | 484k | 7.53M | 546k | deepseek | 2 runs the pack could not grade (the task fails its own grader), left out ; served model unreported on 1 of 50 runs (rule 2 unverifiable there) ; 1 run is a finding, not counted |
-Findings, runs served by another connection, as another model, or that reached the network or looked for the task outside the workspace — listed, not scored:
+Findings, runs served by another connection, as another model, or that reached the network or looked for the task outside the workspace: listed, not scored.
- cline x deepseek-v4.1-flash 290-1: reached the network: fetch_web_content; fetch_web_content
- dsh x deepseek-v4.1-flash 79-7: looked for the task outside the workspace: bash: grep -rl "Where the answer goes" /opt/harnessrouter /data/agent-tools 2>/dev/null | head -20
diff --git a/scripts/benchmark/README.md b/scripts/benchmark/README.md
index 4bdcb8e4..49711004 100644
--- a/scripts/benchmark/README.md
+++ b/scripts/benchmark/README.md
@@ -21,7 +21,7 @@ grader stay outside this repository, at `PACK_ROOT`, under the suite's own licen
| Pack | Suite | Grading | Needs on the grading machine |
|---|---|---|---|
-| `spreadsheetbench` | [SpreadsheetBench Verified](https://github.com/RUCKBReasoning/SpreadsheetBench) (400 real spreadsheet-manipulation questions, NeurIPS 2024 D&B) | the suite's `evaluation.py`, cell by cell over the answer range | `openpyxl`; LibreOffice (`soffice`) so a workbook saved with formulas has values to compare |
+| `spreadsheetbench` | [SpreadsheetBench Verified](https://github.com/RUCKBReasoning/SpreadsheetBench) (400 real spreadsheet-manipulation questions, NeurIPS 2024 D&B) | the suite's `evaluation.py`, cell by cell over the answer range | `openpyxl`; LibreOffice (`soffice`) so a workbook saved with formulas has values to compare. Each grade opens the engine with a profile of its own, so parallel workers do not refuse each other's lock; a workbook the engine could not evaluate is graded as saved and its record says "formulas unevaluated" and why |
## The rules that decide a row
@@ -43,7 +43,11 @@ web tool; `curl`, `wget`, `git clone`, `urllib.request`, `requests`, `httpx`, `a
shell or script) or searched the machine for the task (a recursive `grep`, `rg` or `find`
rooted outside the workspace; a package fetched under a suite's name) is a finding, never a
score. Package installs are the sandbox's normal traffic and do not count; a URL merely spelled
-in code (an xlsx's XML namespaces) does not either; the workspace is the agent's to search.
+in code (an xlsx's XML namespaces) does not either; the workspace is the agent's to search, and
+so is `/tmp`, its scratch; every other absolute path is outside, the product's own code under
+`/app` included. A tool is network use by its name when the name says it goes out (web, fetch,
+browse, a URL, the internet, a search engine by name), not merely because it searches: cline's
+`search_files` and a codebase search search the workspace and are not findings.
Disable the harness's web tools as well, on the harness or the connection: the rule catches what
the switch missed — and on a harness whose switch is an instruction to the model (cline, dsh)
it did, once in two hundred runs, fifteen minutes into a task the run did not finish. A
diff --git a/scripts/benchmark/packs/spreadsheetbench.py b/scripts/benchmark/packs/spreadsheetbench.py
index 49e941be..8291d191 100644
--- a/scripts/benchmark/packs/spreadsheetbench.py
+++ b/scripts/benchmark/packs/spreadsheetbench.py
@@ -16,6 +16,7 @@ import os
import shutil
import subprocess
import sys
+import tempfile
NAME = "spreadsheetbench"
@@ -55,19 +56,31 @@ def stage(task: dict, root: str) -> dict:
return {"prompt": prompt, "files": [(INPUT, data)]}
-def _recalculate(path: str, workdir: str) -> bool:
+def _recalculate(path: str, workdir: str) -> str:
+ """Open the workbook in a spreadsheet engine so its formulas have values, in place. Returns ""
+ when it did, else why it did not: no engine on this machine, or the conversion failed or hung.
+ Two LibreOffice processes sharing one user profile refuse each other's lock and one of them
+ converts nothing, silently; with WORKERS above one that is the ordinary case, so every call
+ gets a profile of its own."""
soffice = shutil.which("soffice") or shutil.which("libreoffice")
if not soffice:
- return False
+ return "no spreadsheet engine on the grading machine"
tmp = os.path.join(workdir, "_recalc")
os.makedirs(tmp, exist_ok=True)
- subprocess.run([soffice, "--headless", "--calc", "--convert-to", "xlsx", "--outdir", tmp, path],
- stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=180)
+ profile = tempfile.mkdtemp(prefix="lo-profile-")
+ try:
+ subprocess.run([soffice, f"-env:UserInstallation=file://{profile}", "--headless", "--calc",
+ "--convert-to", "xlsx", "--outdir", tmp, path],
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=180)
+ except subprocess.TimeoutExpired:
+ return "the spreadsheet engine hung on the workbook"
+ finally:
+ shutil.rmtree(profile, ignore_errors=True)
converted = os.path.join(tmp, os.path.basename(path))
if not os.path.exists(converted):
- return False
+ return "the spreadsheet engine did not convert the workbook"
shutil.move(converted, path)
- return True
+ return ""
_GRADABLE: dict[str, str] = {}
@@ -103,10 +116,12 @@ def grade(task: dict, root: str, produced: dict[str, str], workdir: str) -> dict
path = others[0] if len(others) == 1 else None
if path is None:
return {"reward": 0.0, "resolved": False, "detail": "no workbook produced"}
- recalculated = _recalculate(path, workdir)
+ unevaluated = _recalculate(path, workdir)
try:
ok, detail = compare_workbooks(golden, path, task["instruction_type"], task["answer_position"])
except Exception as e: # noqa: BLE001 — a workbook the grader cannot open is a fail with its reason
return {"reward": 0.0, "resolved": False, "detail": f"grader: {e!r}"[:300]}
+ # A grade taken on unevaluated formulas is a grade of the grading machine; the record says so
+ # either way, so a column can tell a wrong answer from an answer nobody computed.
return {"reward": 1.0 if ok else 0.0, "resolved": bool(ok),
- "detail": (str(detail)[:300] if detail else "") + ("" if recalculated else " (no soffice: formulas unevaluated)")}
+ "detail": (str(detail)[:300] if detail else "") + (f" (formulas unevaluated: {unevaluated})" if unevaluated else "")}
diff --git a/scripts/benchmark/render.py b/scripts/benchmark/render.py
index d11861d4..86c3ec87 100644
--- a/scripts/benchmark/render.py
+++ b/scripts/benchmark/render.py
@@ -26,6 +26,10 @@ def fmt_k(n):
return "-" if n is None else (f"{n / 1e6:.2f}M" if n >= 1e6 else f"{n / 1e3:.0f}k")
+def runs(n: int) -> str:
+ return "1 run" if n == 1 else f"{n} runs"
+
+
def render(records: list[dict]) -> str:
by = collections.defaultdict(list)
for r in records:
@@ -35,7 +39,7 @@ def render(records: list[dict]) -> str:
"[scripts/benchmark/README.md](../scripts/benchmark/README.md). One task is one session; "
"the pack's own grader decides; every number is read from the turn record.", ""]
for (prov, pack), rows in sorted(by.items()):
- out += [f"## Provider: {prov} — pack: {pack}", "",
+ out += [f"## Provider: {prov}, pack: {pack}", "",
"| Harness | Model | Tasks | Resolved | Reward | Wall (sum) | Median wall | Tool calls | Calls failed | Fresh in | Cached in | Output | Served by | Notes |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
findings, errors = [], []
@@ -63,15 +67,15 @@ def render(records: list[dict]) -> str:
else:
clean.append(r)
if ungraded:
- notes.append(f"{ungraded} runs the pack could not grade (the task fails its own grader), left out")
+ notes.append(f"{runs(ungraded)} the pack could not grade (the task fails its own grader), left out")
capped = sum(1 for r in clean if r.get("capped"))
if capped:
- notes.append(f"{capped} runs hit the time cap (counted as failures)")
+ notes.append(f"{runs(capped)} hit the time cap (counted as failures)")
unreported = sum(1 for r in ran if r.get("served_unreported"))
if unreported:
notes.append(f"served model unreported on {unreported} of {len(ran)} runs (rule 2 unverifiable there)")
if len(clean) + ungraded < len(ran):
- notes.append(f"{len(ran) - len(clean) - ungraded} runs are findings, not counted")
+ notes.append(f"{runs(len(ran) - len(clean) - ungraded)} {'is a finding' if len(ran) - len(clean) - ungraded == 1 else 'are findings'}, not counted")
u = [r.get("usage") or {} for r in clean]
served = sorted({str(r.get("connection") or "").replace("integration:", "") for r in ran if r.get("connection")}) or ["?"]
resolved = sum(1 for r in clean if r.get("resolved"))
@@ -97,7 +101,7 @@ def render(records: list[dict]) -> str:
f"| {h} | {m} | 0 | - | - | - | - | - | - | - | - | - | {', '.join(served)} | {' ; '.join(notes) or 'no counted runs'} |")
out.append("")
if findings:
- out += ["Findings, runs served by another connection, as another model, or that reached the network or looked for the task outside the workspace — listed, not scored:", ""]
+ out += ["Findings, runs served by another connection, as another model, or that reached the network or looked for the task outside the workspace: listed, not scored.", ""]
out += [f"- {f}" for f in findings] + [""]
if errors:
out += ["Runner errors (the run never produced a record to judge; re-run them):", ""] + [f"- {e}" for e in errors] + [""]
diff --git a/scripts/benchmark/run.py b/scripts/benchmark/run.py
index 0060ef78..d2657002 100644
--- a/scripts/benchmark/run.py
+++ b/scripts/benchmark/run.py
@@ -61,7 +61,12 @@ def api(method: str, path: str, body=None, raw: bool = False, timeout: int = 180
# The first pilot's third task fetched its own ground truth from the suite's public data because
# the task id was in a filename and the sandbox has internet. Packs anonymise; this reads the tool
# calls the runner stored for the turn and names any that went out for anything but pip.
-WEB_TOOL = re.compile(r"web|fetch|search|browse|http", re.I)
+# A tool's NAME says it goes out when it names the web, a fetch, a browse, a URL or the internet.
+# "search" alone does not: cline's search_files, codex's codebase search and a grep-shaped tool
+# search the workspace, and a column that called them network use would turn its cleanest runs
+# into findings. A search is network use when the name says where it searches: the web, the
+# internet, online, or a search engine by name.
+WEB_TOOL = re.compile(r"web|fetch|browse|http|url|internet|online|google|bing|duckduckgo|brave|tavily|perplexity|exa\b", re.I)
# What makes a request from a shell or a script, not what merely spells a URL: the XML namespaces
# inside an xlsx are http:// strings, and a turn parsing one with zipfile went nowhere. Package
# installs (pip, npm) are the sandbox's normal traffic and are not the task's answer.
@@ -73,7 +78,11 @@ NET_IN_SHELL = re.compile(r"\b(curl|wget|git\s+clone)\b|\burllib\.request\b|\bre
# vocabulary grepped the whole filesystem for its data and tried to download it by name
# (measured 2026-09-18). A recursive grep or find rooted outside the workspace, or a package
# fetched under a suite's name, is a lookup; the workspace itself (/data/workspaces/…) is not.
-OUTSIDE = r"(?:/|/(?:data|root|home|usr|opt|srv|var|etc)(?!/workspaces/)\S*)"
+# Inside is the workspace (/data/workspaces/... on this image) and the scratch directory an agent
+# may use for its own files; every other absolute path is outside, the product's own code under
+# /app and the base images' trees included. A list of outside roots was the first cut, and a
+# search of /app for the suite's vocabulary passed it.
+OUTSIDE = r"/(?!data/workspaces/|tmp(?:/|\s|$))\S*"
LOOKUP_IN_SHELL = re.compile(r"\bgrep\b[^|;&\n]*\s-[A-Za-z]*[rR][A-Za-z]*\b[^|;&\n]*\s" + OUTSIDE + r"(?:\s|$)"
r"|\brg\b[^|;&\n]*\s" + OUTSIDE + r"(?:\s|$)"
r"|\bfind\s+" + OUTSIDE + r"(?:\s|$)"
@@ -258,7 +267,7 @@ def run_task(pack, root: str, task: dict, label: str, harness_id: str, model: st
# the request waits as long as the task may; past that, the session is cancelled below
resp = api("POST", "/v1/responses", body, timeout=TASK_CAP_S)
rid = resp.get("id")
- while resp.get("status") in ("in_progress", "queued"):
+ while resp.get("status") in ("in_progress", "queued", "running"):
time.sleep(5)
resp = api("GET", f"/v1/responses/{rid}")
rec.update(status=resp.get("status"), wall_s=round(time.time() - t0, 1), sid=(resp.get("metadata") or {}).get("session_id"),
diff --git a/scripts/benchmark/test_run.py b/scripts/benchmark/test_run.py
index 069c004d..96a06c33 100644
--- a/scripts/benchmark/test_run.py
+++ b/scripts/benchmark/test_run.py
@@ -60,6 +60,16 @@ def test_a_web_tool_or_a_shell_that_reaches_out_is_network_use():
assert network_use([{"name": "bash", "arguments": json.dumps({"command": "wget http://h/x"})}])
+def test_a_tool_that_searches_the_workspace_is_not_network_use():
+ """cline's search_files, a codebase search and a grep-shaped tool search the workspace; a
+ column that read their names as network use would turn its cleanest runs into findings. A
+ search is network use when its name says where: the web, the internet, a search engine."""
+ for name in ("search_files", "codebase_search", "file_search", "grep_search", "search", "Grep", "list_code_definition_names"):
+ assert network_use([{"name": name, "arguments": '{"regex": "TOTAL"}'}]) == [], name
+ for name in ("web_search", "WebSearch", "search_web", "internet_search", "google_search", "brave_search", "online_search"):
+ assert network_use([{"name": name, "arguments": '{"query": "x"}'}]) == [f"{name}: x"], name
+
+
def test_pip_and_local_work_are_not_network_use():
assert network_use([{"name": "bash", "arguments": json.dumps({"command": "pip install openpyxl pandas"})}]) == []
assert network_use([{"name": "bash", "arguments": json.dumps({"command": "python3 -m pip install openpyxl && python3 solve.py"})}]) == []
@@ -88,7 +98,11 @@ def test_searching_the_machine_for_the_task_is_a_lookup():
assert lookup_use(sh("find / -name '*golden*.xlsx' 2>/dev/null"))
assert lookup_use(sh("cd /tmp && timeout 60 pip download spreadsheetbench -d /tmp/sb --no-deps"))
assert lookup_use(sh("rg 'answer_position' /data 2>/dev/null"))
- # the workspace is the agent's to search; pip for a library is not a lookup
+ # the product's own code and the base images' trees are outside the workspace too
+ assert lookup_use(sh("grep -rl 'answer_position' /app 2>/dev/null"))
+ assert lookup_use(sh("find /opt/harnessrouter -name 'dataset.json'"))
+ # the workspace is the agent's to search, and so is its scratch directory; pip for a library is not a lookup
+ assert lookup_use(sh("find /tmp -name '*.xlsx'")) == []
assert lookup_use(sh("grep -rn 'TOTAL' /data/workspaces/hsess1234/ | head")) == []
assert lookup_use(sh("find /data/workspaces/hsess1234 -name '*.xlsx'")) == []
assert lookup_use(sh("grep -r 'Sheet1' . && pip install openpyxl pandas")) == []
@@ -133,6 +147,7 @@ def test_a_finding_is_listed_and_left_out_of_the_score():
row = next(line for line in md.splitlines() if line.startswith("| opencode |"))
assert "| 1 | 1 (100%) |" in row, row # one counted run of four
assert "3 runs are findings, not counted" in row
+ assert "1 run is a finding, not counted" in render([_rec(task="a"), _rec(task="b", network=["webfetch"])])
assert "reached the network: webfetch" in md and "served by integration:other" in md and "served as gpt-5.5" in md
md = render([_rec(task="e", lookup=["bash: grep -rl x /"])])
assert "looked for the task outside the workspace: bash: grep -rl x /" in md
@@ -141,7 +156,7 @@ def test_a_finding_is_listed_and_left_out_of_the_score():
def test_a_capped_run_is_a_failure_and_the_row_says_how_many():
md = render([_rec(task="a"), _rec(task="b", resolved=False, reward=0.0, capped=900, detail="time cap 900 s; no workbook produced")])
row = next(line for line in md.splitlines() if line.startswith("| opencode |"))
- assert "| 2 | 1 (50%) |" in row and "1 runs hit the time cap (counted as failures)" in row, row
+ assert "| 2 | 1 (50%) |" in row and "1 run hit the time cap (counted as failures)" in row, row
def test_a_task_the_grader_cannot_decide_is_left_out_not_failed():
@@ -150,7 +165,7 @@ def test_a_task_the_grader_cannot_decide_is_left_out_not_failed():
md = render([_rec(task="a"), _rec(task="b", resolved=None, reward=None, detail="ungradable: the grader raises on the golden workbook: ValueError('b2b is not a valid coordinate or range')")])
row = next(line for line in md.splitlines() if line.startswith("| opencode |"))
assert "| 1 | 1 (100%) |" in row, row
- assert "1 runs the pack could not grade (the task fails its own grader), left out" in row and "findings" not in row
+ assert "1 run the pack could not grade (the task fails its own grader), left out" in row and "findings" not in row
def test_an_unreported_served_model_is_noted_not_scored_against():