HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
Python agent-apiagent-frameworkagent-harnessai-agentsai-infrastructurecoding-agentconformance-testingdeepseek-harnessharness-engineeringhermesllmopen-protocolopen-sourceopen-standardpi-coding-agentself-hosteduhpunified-apiunified-harness-protocolunified-interface
benchmark: plain-words prompts, a lookup on the machine is a finding, and a time cap per task
Hao Lu2e6d85edadf5
diff --git a/scripts/benchmark/README.md b/scripts/benchmark/README.mdindex 71243241..614da71c 100644--- a/scripts/benchmark/README.md+++ b/scripts/benchmark/README.md@@ -33,14 +33,19 @@ record names another connection is a finding, never a score. The matrix's rule 1suffix is the same model, another family, number or tier is not. A harness whose turn recordcarries no served model is scored, and the row says on how many runs rule 2 could not be checked.-**3. It did the task, not a lookup.** The sandbox has internet. The first pilot's third task-fetched its own ground truth from the suite's public data, because the task id was in a filename.-So a pack names nothing — the workbook is `input.xlsx`, the answer `output.xlsx`, the prompt-carries no id — and a run whose tool calls made a request (a web tool; `curl`, `wget`, `git clone`,-`urllib.request`, `requests`, `httpx`, `aiohttp` in a shell or script) is a finding, never a+**3. It did the task, not a lookup.** The sandbox has internet and a filesystem. The first+pilot's third task fetched its own ground truth from the suite's public data, because the task id+was in a filename; two later turns recognised the suite from its field names in the prompt and+spent half an hour grepping the whole filesystem for its data and trying to `pip download` it.+So a pack names nothing — the workbook is `input.xlsx`, the answer `output.xlsx`, the prompt is+plain words with none of the suite's vocabulary — and a run whose tool calls made a request (a+web tool; `curl`, `wget`, `git clone`, `urllib.request`, `requests`, `httpx`, `aiohttp` in a+shell or script) or searched the machine for the task (a recursive `grep`, `rg` or `find`+rooted outside the workspace; a package fetched under a suite's name) is a finding, never ascore. Package installs are the sandbox's normal traffic and do not count; a URL merely spelled-in code (an xlsx's XML namespaces) does not either. Disable the harness's web tools as well, on-the harness or the connection: the rule catches what the switch missed.+in code (an xlsx's XML namespaces) does not either; the workspace is the agent's to search.+Disable the harness's web tools as well, on the harness or the connection: the rule catches what+the switch missed.**4. Tokens on one convention.** Fresh input, cached input and output are reported apart andnever summed: a cached read is the same prompt read again, not new input. The runner's contract@@ -50,6 +55,10 @@ instance older than #209 reports cline's input gross, and its rows say so in the**5. No judge.** A suite whose grading needs a model's opinion is not a pack. Findings andrunner errors are listed under the table, not scored.+Every task has a time cap, `TASK_CAP_S` (900 s by default): a turn still running at the cap is+cancelled through the API and graded as it stands, a failure, not a finding — the cap is part of+the task. The row says how many runs hit it.+## Running it```diff --git a/scripts/benchmark/packs/spreadsheetbench.py b/scripts/benchmark/packs/spreadsheetbench.pyindex 05e80948..42f03539 100644--- a/scripts/benchmark/packs/spreadsheetbench.py+++ b/scripts/benchmark/packs/spreadsheetbench.py@@ -22,27 +22,23 @@ NAME = "spreadsheetbench"INPUT = "input.xlsx"OUTPUT = "output.xlsx"-# The suite's own framing (inference/prompt_format.py), addressed to an agent with a workspace-# instead of a code generator: the file is there, the answer is a file, and the answer goes only-# where the instruction says. VALUES rather than formulas keeps the grade about the task, not-# about whether the grading machine's spreadsheet engine evaluates a function the same way.-PROMPT = """You are working in a workspace that contains a spreadsheet file. Solve the spreadsheet manipulation task below by editing the workbook and saving the result to output_path inside the workspace. Python 3 is available; install packages with pip if you need them (openpyxl, pandas). Do not use the internet for anything other than pip: no web search, no fetching pages, no looking the task up. Write cell VALUES, not formulas, unless the instruction asks for formulas. Only modify cells inside answer_position on the answer sheet; leave everything else exactly as it is (other sheets, formatting, column widths). When the file is saved, reply with one line: done.+# The suite's framing (inference/prompt_format.py) in plain words, addressed to an agent with a+# workspace: the file is there, the answer is a file, and the answer goes only where the task+# says. None of the suite's own field names appear: a turn that saw "instruction_type:+# Sheet-Level Manipulation" recognised the suite and spent twenty minutes grepping the whole+# filesystem for its data and trying to `pip download` it (measured 2026-09-18) — the vocabulary+# was the search key. VALUES rather than formulas keeps the grade about the task, not about+# whether the grading machine's spreadsheet engine evaluates a function the same way.+PROMPT = """You are working in a workspace that contains a spreadsheet, {spreadsheet_path}. Do the task below by editing that workbook and saving the result as {output_path} in the workspace.-### instruction-{instruction}+The task: {instruction}-### spreadsheet_path-{spreadsheet_path}+Where the answer goes: sheet "{answer_sheet}", cells {answer_position}. {scope}-### instruction_type-{instruction_type}--### answer_position-{answer_sheet}!{answer_position}--### output_path-{output_path}+Python 3 is available; install packages with pip if you need them (openpyxl, pandas). Do not use the internet, and do not look for this task or its answer anywhere else on this machine: everything you need is in the workspace. Write cell values, not formulas, unless the task asks for formulas. Leave everything outside the answer cells exactly as it is (other sheets, formatting, column widths). When {output_path} is saved, reply with one line: done."""+SCOPE = {"Cell-Level Manipulation": "Only those cells change.",+ "Sheet-Level Manipulation": "Everything the task asks for lands inside that range, and nothing outside it changes."}def load(root: str) -> list[dict]:@@ -53,8 +49,8 @@ def stage(task: dict, root: str) -> dict:init = os.path.join(root, "spreadsheet", task["id"], f"1_{task['id']}_init.xlsx")with open(init, "rb") as f:data = f.read()- prompt = PROMPT.format(instruction=task["instruction"], spreadsheet_path=INPUT,- instruction_type=task["instruction_type"], answer_sheet=task["answer_sheet"],+ prompt = PROMPT.format(instruction=task["instruction"].strip(), spreadsheet_path=INPUT,+ scope=SCOPE.get(task["instruction_type"], ""), answer_sheet=task["answer_sheet"],answer_position=task["answer_position"], output_path=OUTPUT)return {"prompt": prompt, "files": [(INPUT, data)]}diff --git a/scripts/benchmark/render.py b/scripts/benchmark/render.pyindex 7919852b..1d010a9b 100644--- a/scripts/benchmark/render.py+++ b/scripts/benchmark/render.py@@ -54,10 +54,15 @@ def render(records: list[dict]) -> str:why.append(f"served as {r.get('served')}")if r.get("network"):why.append("reached the network: " + "; ".join(str(x) for x in r["network"])[:120])+ if r.get("lookup"):+ why.append("looked for the task outside the workspace: " + "; ".join(str(x) for x in r["lookup"])[:120])if why:findings.append(f"{h} x {m} {r.get('task')}: " + ", ".join(why))else:clean.append(r)+ capped = sum(1 for r in clean if r.get("capped"))+ if capped:+ notes.append(f"{capped} runs hit the time cap (counted as failures)")unreported = sum(1 for r in ran if r.get("served_unreported"))if unreported:notes.append(f"served model unreported on {unreported} of {len(ran)} runs (rule 2 unverifiable there)")@@ -88,7 +93,7 @@ def render(records: list[dict]) -> str:f"| {h} | {m} | 0 | - | - | - | - | - | - | - | - | - | {', '.join(served)} | {' ; '.join(notes) or 'no counted runs'} |")out.append("")if findings:- out += ["Findings, runs served by another connection, as another model, or that reached the network — listed, not scored:", ""]+ out += ["Findings, runs served by another connection, as another model, or that reached the network or looked for the task outside the workspace — listed, not scored:", ""]out += [f"- {f}" for f in findings] + [""]if errors:out += ["Runner errors (the run never produced a record to judge; re-run them):", ""] + [f"- {e}" for e in errors] + [""]diff --git a/scripts/benchmark/run.py b/scripts/benchmark/run.pyindex ce36c4ed..59849c62 100644--- a/scripts/benchmark/run.py+++ b/scripts/benchmark/run.py@@ -68,25 +68,44 @@ NET_IN_SHELL = re.compile(r"\b(curl|wget|git\s+clone)\b|\burllib\.request\b|\brer"|\bhttpx\.|\baiohttp\b", re.I)+# The other way to look a task up: on the machine. Two turns that recognised the suite from its+# vocabulary grepped the whole filesystem for its data and tried to download it by name+# (measured 2026-09-18). A recursive grep or find rooted outside the workspace, or a package+# fetched under a suite's name, is a lookup; the workspace itself (/data/workspaces/…) is not.+OUTSIDE = r"(?:/|/(?:data|root|home|usr|opt|srv|var|etc)(?!/workspaces/)\S*)"+LOOKUP_IN_SHELL = re.compile(r"\bgrep\b[^|;&\n]*\s-[A-Za-z]*[rR][A-Za-z]*\b[^|;&\n]*\s" + OUTSIDE + r"(?:\s|$)"+ r"|\brg\b[^|;&\n]*\s" + OUTSIDE + r"(?:\s|$)"+ r"|\bfind\s+" + OUTSIDE + r"(?:\s|$)"+ r"|\bpip3?\s+(?:download|install)\b[^\n]*\b\S*(?:bench|dataset)\S*", re.I)+++def _command_of(t: dict) -> str:+ args = t.get("arguments") or ""+ if isinstance(args, str) and args.startswith("{"):+ try:+ return str(json.loads(args).get("command", ""))+ except ValueError:+ return args+ return args if isinstance(args, str) else ""++def network_use(tools: list[dict]) -> list[str]:hits = []for t in tools or []:name = str(t.get("name") or "")- args = t.get("arguments") or ""if WEB_TOOL.search(name):hits.append(name)continue- cmd = args- if isinstance(args, str) and args.startswith("{"):- try:- cmd = json.loads(args).get("command", "")- except ValueError:- cmd = args- if isinstance(cmd, str) and NET_IN_SHELL.search(cmd):+ cmd = _command_of(t)+ if NET_IN_SHELL.search(cmd):hits.append(f"{name}: {cmd[:120]}")return hits+def lookup_use(tools: list[dict]) -> list[str]:+ return [f"{t.get('name')}: {_command_of(t)[:120]}" for t in tools or [] if LOOKUP_IN_SHELL.search(_command_of(t))]++# ── usage on one convention ───────────────────────────────────────────────────────────────────────# The runner's contract is input_tokens = fresh input, cache reads and writes beside it (codex,# gemini and, since #209, cline are netted to it). The table reports fresh, cached and output@@ -147,14 +166,19 @@ def prompt_matches(stored: str, prompt: str) -> bool:return any(p and p in prompt for p in probes)-def recover_session(before: set[str], harness_id: str, prompt: str, t0: float, wait_s: int = 1800) -> dict | None:+TASK_CAP_S = int(os.environ.get("TASK_CAP_S", "900"))+++def recover_session(before: set[str], harness_id: str, prompt: str, t0: float, cap_s: int = TASK_CAP_S) -> dict | None:"""The session this harness opened for this prompt since t0 that was not there before, onceit has finished: an instance whose console proxy cuts a synchronous request at five minutes(fixed by #214) still ran the turn, and the record is the record. With several workers theprompt tells the sessions apart; a session whose stored prompt cannot be read is taken only- when it is the sole new one."""- deadline = t0 + wait_s- while time.time() < deadline:+ when it is the sole new one. A turn still running at the task's time cap is cancelled and+ returned as it stands: the cap is part of the task, and two turns that grepped the whole+ filesystem for their answer ran for half an hour before this existed."""+ cancelled = False+ while True:new = [s for s in api("GET", "/v1/sessions?limit=100").get("sessions", [])if s["session_id"] not in before and harness_id in (s.get("harness_id"), s.get("backend"))]mine = [s for s in new if prompt_matches(str(s.get("user_prompt") or ""), prompt)]@@ -162,10 +186,21 @@ def recover_session(before: set[str], harness_id: str, prompt: str, t0: float, wmine = newif not mine:return None- if mine[0].get("status") not in ("running", "queued", None):- return mine[0]+ sess = mine[0]+ if sess.get("status") not in ("running", "queued", "starting", None):+ if cancelled:+ sess["capped"] = cap_s+ return sess+ if time.time() - t0 >= cap_s and not cancelled:+ try:+ api("POST", f"/v1/sessions/{sess['session_id']}/cancel", {})+ except urllib.error.HTTPError:+ pass+ cancelled = True+ if cancelled and time.time() - t0 >= cap_s + 180:+ sess["capped"] = cap_s # the runner did not confirm the stop; the record is what it is+ return sesstime.sleep(10)- return Nonedef run_task(pack, root: str, task: dict, label: str, harness_id: str, model: str, provider: str, workdir: str) -> dict:@@ -182,7 +217,8 @@ def run_task(pack, root: str, task: dict, label: str, harness_id: str, model: stt0 = time.time()sid = rid = Nonetry:- resp = api("POST", "/v1/responses", body)+ # the request waits as long as the task may; past that, the session is cancelled below+ resp = api("POST", "/v1/responses", body, timeout=TASK_CAP_S)rid = resp.get("id")while resp.get("status") in ("in_progress", "queued"):time.sleep(5)@@ -198,7 +234,7 @@ def run_task(pack, root: str, task: dict, label: str, harness_id: str, model: streturn recrec.update(status="completed" if sess.get("status") == "done" else sess.get("status"), recovered=err,wall_s=sess.get("elapsed") or round(time.time() - t0, 1), sid=sess["session_id"],- usage=usage_of(sess.get("usage")), turn_error=None)+ usage=usage_of(sess.get("usage")), turn_error=None, capped=sess.get("capped"))sid, rid = sess["session_id"], sess.get("last_response_id")if not sid:rec["error"] = "no session id on the response"@@ -211,7 +247,8 @@ def run_task(pack, root: str, task: dict, label: str, harness_id: str, model: stsubstituted=bool(served) and not alias_of(model, served),served_unreported=not served,tool_calls=len(tools), tools=sorted({str(t.get("name")) for t in tools}),- network=network_use(tools), files=[f.get("filename") for f in (turn.get("files") or [])],+ network=network_use(tools), lookup=lookup_use(tools),+ files=[f.get("filename") for f in (turn.get("files") or [])],assistant=(turn.get("assistant") or "")[:200])if rec.get("turn_error") is None and turn.get("error"):rec["turn_error"] = turn.get("error")@@ -237,6 +274,8 @@ def run_task(pack, root: str, task: dict, label: str, harness_id: str, model: stproduced[name] = pathg = pack.grade(task, root, produced, tdir)rec.update(reward=g["reward"], resolved=bool(g["resolved"]), detail=g.get("detail", ""))+ if rec.get("capped") and not rec["resolved"]:+ rec["detail"] = f"time cap {rec['capped']} s; " + rec["detail"]if os.environ.get("KEEP") != "1":try:api("DELETE", f"/v1/sessions/{sid}")@@ -299,8 +338,8 @@ def main() -> None:rec["foreign"] = rec["connection"]save(rec)verdict = ("ERROR " + rec["error"][:80]) if rec.get("error") else \- ("FINDING network" if rec.get("network") else "FINDING foreign" if rec.get("foreign") else- "FINDING substituted" if rec.get("substituted") else+ ("FINDING network" if rec.get("network") else "FINDING lookup" if rec.get("lookup") else+ "FINDING foreign" if rec.get("foreign") else "FINDING substituted" if rec.get("substituted") else("pass" if rec.get("resolved") else f"FAIL {rec.get('detail', '')[:60]}"))log(f"TASK {label} x {model} {t['id']}: {verdict} {rec.get('wall_s', '?')}s tools={rec.get('tool_calls', '?')} "f"served={rec.get('served') or 'unreported'} tokens={json.dumps(rec.get('usage') or {})}")diff --git a/scripts/benchmark/test_run.py b/scripts/benchmark/test_run.pyindex bb381bf1..5de5d060 100644--- a/scripts/benchmark/test_run.py+++ b/scripts/benchmark/test_run.py@@ -10,7 +10,7 @@ import syssys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))from render import render # noqa: E402-from run import network_use, prompt_matches, tool_outcomes, usage_of # noqa: E402+from run import lookup_use, network_use, prompt_matches, tool_outcomes, usage_of # noqa: E402def test_a_truncated_stored_prompt_still_identifies_its_session():@@ -51,6 +51,22 @@ def test_a_url_spelled_in_code_is_not_network_use():assert network_use([{"name": "bash", "arguments": json.dumps({"command": "python3 -c 'import requests; requests.get(\"http://x\")'"})}])+def test_searching_the_machine_for_the_task_is_a_lookup():+ """Measured 2026-09-18: two turns recognised the suite and went looking for its data — a+ recursive grep of the whole filesystem for its vocabulary, a pip download by its name."""+ sh = lambda c: [{"name": "bash", "arguments": json.dumps({"command": c})}]+ assert lookup_use(sh('grep -rl "Sheet-Level Manipulation" / --include=*.json 2>/dev/null | head'))+ assert lookup_use(sh("grep -rl 'delete rows in an Excel worksheet' / 2>/dev/null | head -20"))+ assert lookup_use(sh("find / -name '*golden*.xlsx' 2>/dev/null"))+ assert lookup_use(sh("cd /tmp && timeout 60 pip download spreadsheetbench -d /tmp/sb --no-deps"))+ assert lookup_use(sh("rg 'answer_position' /data 2>/dev/null"))+ # the workspace is the agent's to search; pip for a library is not a lookup+ assert lookup_use(sh("grep -rn 'TOTAL' /data/workspaces/hsess1234/ | head")) == []+ assert lookup_use(sh("find /data/workspaces/hsess1234 -name '*.xlsx'")) == []+ assert lookup_use(sh("grep -r 'Sheet1' . && pip install openpyxl pandas")) == []+ assert lookup_use(sh("ls -la /data/workspaces/hsess1234")) == []++def test_usage_keeps_fresh_cached_and_output_apart():u = usage_of({"input_tokens": 14847, "output_tokens": 3623, "cache_read_tokens": 288000})assert u == {"fresh_input": 14847, "cached_input": 288000, "cache_write": 0, "output": 3623, "prompt_total": 302847}@@ -90,6 +106,14 @@ def test_a_finding_is_listed_and_left_out_of_the_score():assert "| 1 | 1 (100%) |" in row, row # one counted run of fourassert "3 runs are findings, not counted" in rowassert "reached the network: webfetch" in md and "served by integration:other" in md and "served as gpt-5.5" in md+ md = render([_rec(task="e", lookup=["bash: grep -rl x /"])])+ assert "looked for the task outside the workspace: bash: grep -rl x /" in md+++def test_a_capped_run_is_a_failure_and_the_row_says_how_many():+ md = render([_rec(task="a"), _rec(task="b", resolved=False, reward=0.0, capped=900, detail="time cap 900 s; no workbook produced")])+ row = next(line for line in md.splitlines() if line.startswith("| opencode |"))+ assert "| 2 | 1 (50%) |" in row and "1 runs hit the time cap (counted as failures)" in row, rowdef test_an_unreported_served_model_is_noted_not_scored_against():