The open-source agent harness - the runtime layer that turns an LLM into a working agent.
benchmark(tfy): ship the validated optimized config + replication robustness (#212)
shubhamagarwal-create0e40e60c604c
diff --git a/benchmark/.env.example b/benchmark/.env.exampleindex 4da6e1568..294de33da 100644--- a/benchmark/.env.example+++ b/benchmark/.env.example@@ -31,11 +31,17 @@ JUDGE_MODEL=claude-opus-4-8# ---- TrueForge arm ----------------------------------------------------------# Point at your running TrueForge server's HTTP API. It needs no auth by default. In-# TrueForge: configure a provider for MODEL_TFY, and register the MCP_CONFIG servers as-# Connectors under the same names used in mcp_config.json.+# TrueForge you must configure THREE things (see README "TrueForge setup"):+# 1. a model provider for MODEL_TFY;+# 2. the MCP_CONFIG servers registered as Connectors under the same names;+# 3. a SANDBOX PROVIDER (e.g. Daytona). The benchmark config offloads oversized tool+# results to a sandbox file (large_tool_response); without a sandbox the data-heavy+# tasks pile everything into context and abort with "max_tokens breached".+# The rest of the agent config (sub-agents on, compaction, etc.) is sent inline by the+# adapter and matches the validated setup — you don't set it here.TFY_BASE_URL=http://localhost:8790MODEL_TFY=anthropic/claude-opus-4-8 # provider/model string as configured in TrueForge-TFY_ITERATION_LIMIT=100 # operational cap on tool-use iterations per turn+TFY_ITERATION_LIMIT=500 # tool-use iterations per turn (validated default)# ---- Claude Managed Agents arm ---------------------------------------------# ANTHROPIC_API_KEY above is reused. cma_ids.json is created by setup.py.diff --git a/benchmark/README.md b/benchmark/README.mdindex a1188edb8..7b9f82ca4 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -63,10 +63,10 @@ granularity, and use all the related data — with no task-specific detail.## How it works1. **`setup.py`** — creates the CMA environment/agent/vault with the shipped system- prompt and the MCP servers. TrueForge and deepagents need no setup step: TrueForge+ prompt and the MCP servers. TrueForge and deepagents need no setup step here: TrueForgesends an inline agent spec when it creates each session, and deepagents attaches the- MCP tools at run time. (TrueForge does need, on its side, a model provider for- `MODEL_TFY` and the MCP servers registered as Connectors — see `.env.example`.)+ MCP tools at run time. (TrueForge does need three things configured on its side — a model+ provider, the MCP Connectors, and a **sandbox provider** — see **TrueForge setup** below.)2. **`bench_matrix.py run-all`** — runs every `{arm × task × trial}` cell. Each taskruns as a subprocess with a hard wall-clock timeout and is retried ontimeout / crash / empty answer. Fully resumable — rerun to continue.@@ -78,6 +78,45 @@ granularity, and use all the related data — with no task-specific detail.4. **`aggregate.py`** — rolls the matrix up into per-arm accuracy, cost, and latency.`Solved / 14` is the mean number of tasks passed per trial. Writes `results/summary.csv`.+### TrueForge setup (required to reproduce)++The `tfy` arm talks to a running TrueForge server (`TFY_BASE_URL`). Before running it,+configure three things **inside TrueForge** — miss any one and the data-heavy tasks fail:++1. **A model provider** for `MODEL_TFY` (e.g. `anthropic/claude-opus-4-8`, `zai/glm-5-2`).+2. **The MCP servers** from `mcp_config.json`, registered as Connectors **under the same+ names**.+3. **A sandbox provider** (e.g. Daytona). This is the one people miss. The benchmark config+ enables `large_tool_response`, which **offloads oversized query results to a sandbox+ file** instead of letting them pile into the model's context. Several of these tasks scan+ tens of thousands of records; **without a sandbox the context blows past the token budget+ and the turn aborts with `max_tokens breached`.**++The agent config is sent **inline by the adapter** (`run_tfy` in `bench_matrix.py`) — you do+not set it in TrueForge — and is the validated setup that produced the results below:++| Setting | Value | Why |+| ---------------------------------------- | ----------------------------- | -------------------------------------------------------- |+| `iteration_limit` | **500** | enough tool-use steps to fully work a cross-system task |+| `dynamic_sub_agents` | **enabled** | decompose big sub-tasks to sub-agents with fresh context |+| `sandbox` | **enabled** | needed for `large_tool_response` offload (see #3 above) |+| `context_management.compaction` | **enabled, threshold 60 000** | summarize old history so long turns stay under budget |+| `context_management.large_tool_response` | **enabled** | offload oversized tool results to the sandbox |+| `generative_ui` | disabled | not needed for a headless benchmark |+| `ask_user_questions` | disabled | fully autonomous; never pause a turn for input |++**Long turns.** The heaviest tasks run for several minutes. TrueForge streams turn events+over SSE, and on a long turn that stream can close before the turn finishes (the turn keeps+running server-side). The adapter handles this: if the stream ends without a terminal+`turn.done`, it **polls the turn to its terminal state** rather than recording an empty+answer. If you write your own client, do the same, or long tasks will look like spurious+failures.++**Strict MCP servers.** The MCP spec requires clients to send+`Accept: application/json, text/event-stream`. If your MCP server enforces this strictly and+TrueForge's connector doesn't satisfy it, tool listing fails; use an MCP server (or a thin+proxy) that accepts the header TrueForge sends.+### Cost modelCost is computed from token counts at provider list prices (`$/1M`), not estimated:diff --git a/benchmark/bench_matrix.py b/benchmark/bench_matrix.pyindex e2b7a7eb6..9f1cbe82e 100644--- a/benchmark/bench_matrix.py+++ b/benchmark/bench_matrix.py@@ -50,10 +50,27 @@ def run_tfy(task, prompt, cap):"instructions": SYSTEM_PROMPT,"mcp_servers": [{"name": n, "enable_tools": ["@all"], "require_approval_for_tools": []}for n in mcp],- # Fully autonomous: no approval gates (above) and no clarifying-question pauses,- # either of which would end a turn early with incomplete work.- "config": {"iteration_limit": int(os.environ.get("TFY_ITERATION_LIMIT", "100")),- "ask_user_questions": {"enabled": False}},+ # Validated benchmark config (the setup that produced the published results):+ # - fully autonomous: no approval gates (above) and no clarifying-question pauses,+ # either of which would end a turn early with incomplete work;+ # - dynamic sub-agents ON and a 500-iteration budget, so the agent can fully work+ # the data-heavy cross-system tasks (decompose big sub-tasks, take enough steps);+ # - compaction + large_tool_response keep long turns under the token budget:+ # large_tool_response OFFLOADS oversized query results to a sandbox file instead+ # of piling them into context. This REQUIRES a sandbox provider configured in+ # TrueForge (see README "Sandbox"); without one the offload can't happen and the+ # heaviest tasks abort with "max_tokens breached".+ "config": {+ "iteration_limit": int(os.environ.get("TFY_ITERATION_LIMIT", "500")),+ "ask_user_questions": {"enabled": False},+ "dynamic_sub_agents": {"enabled": True},+ "sandbox": {"enabled": True},+ "context_management": {+ "compaction": {"enabled": True, "compaction_threshold_tokens": 60000},+ "large_tool_response": {"enabled": True},+ },+ "generative_ui": {"enabled": False},+ },}def _post(path, body, stream=False):@@ -62,6 +79,10 @@ def run_tfy(task, prompt, cap):"Accept": "text/event-stream" if stream else "application/json"}, method="POST")return urllib.request.urlopen(req, timeout=TASK_TIMEOUT)+ def _get(path):+ req = urllib.request.Request(base + path, headers={"Accept": "application/json"})+ return json.load(urllib.request.urlopen(req, timeout=60))+def text_of(content):if isinstance(content, str):return content@@ -71,7 +92,7 @@ def run_tfy(task, prompt, cap):sid = json.load(_post("/api/v1/sessions", {"agent": {"spec": spec}}))["data"]["id"]tot = {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0}- tools = 0; answer = ""; last_msg = ""; cost = None+ tools = 0; answer = ""; last_msg = ""; cost = None; terminal = Falseresp = _post(f"/api/v1/sessions/{sid}/turns",{"input": [{"type": "user.message", "content": prompt}],"previous_turn_id": "none", "stream": True}, stream=True)@@ -100,7 +121,33 @@ def run_tfy(task, prompt, cap):cost = m.get("total_cost_in_usd")out = state.get("output") or {}answer = text_of(out.get("content") if isinstance(out, dict) else None)+ terminal = Truebreak+ # On the data-heavy tasks a turn can run for several minutes, and the SSE stream may+ # close before the turn finishes (the turn keeps running server-side). Don't treat the+ # stream ending as "done" — poll the turn to its terminal state so we capture the real+ # answer/metrics instead of an empty result. This is what keeps the heaviest tasks from+ # being scored as spurious failures.+ if not terminal:+ try:+ turns = _get(f"/api/v1/sessions/{sid}/turns").get("data", []) or []+ tid = (turns[0] if turns else {}).get("id")+ deadline = time.time() + TASK_TIMEOUT+ while tid and time.time() < deadline:+ st = (_get(f"/api/v1/sessions/{sid}/turns/{tid}").get("data", {}) or {}).get("state", {}) or {}+ if st.get("status") in ("done", "error", "cancelled"):+ m = st.get("metrics") or {}+ tot = {"input": m.get("total_input_tokens", 0) or 0,+ "output": m.get("total_output_tokens", 0) or 0,+ "cache_read": m.get("total_cache_read_tokens", 0) or 0,+ "cache_write": m.get("total_cache_write_tokens", 0) or 0}+ cost = m.get("total_cost_in_usd")+ out = st.get("output") or {}+ answer = text_of(out.get("content") if isinstance(out, dict) else None) or answer+ break+ time.sleep(6)+ except Exception:+ passif not answer:answer = last_msgres = {"session_id": sid, "tokens": tot, "tool_calls": tools, "answer": answer}diff --git a/benchmark/setup.py b/benchmark/setup.pyindex 6e3307fb0..0a4b9d956 100644--- a/benchmark/setup.py+++ b/benchmark/setup.py@@ -65,8 +65,11 @@ if __name__ == "__main__":if "cma" in harnesses:setup_cma()if "tfy" in harnesses:- print("[tfy] no setup step here — in TrueForge, configure a provider for "- "MODEL_TFY and register the MCP_CONFIG servers as Connectors under the "- "same names; the agent spec is sent inline when each session is created.",- flush=True)+ print("[tfy] no setup step here, but TrueForge itself needs THREE things "+ "configured (see README): (1) a model provider for MODEL_TFY, (2) the "+ "MCP_CONFIG servers registered as Connectors under the same names, and "+ "(3) a SANDBOX PROVIDER (e.g. Daytona) — required so oversized tool results "+ "offload to a file instead of aborting the heaviest tasks. The agent spec "+ "(sub-agents, compaction, sandbox on, 500 iterations) is sent inline when "+ "each session is created.", flush=True)print("[setup] done", flush=True)