Skip to content

The open-source agent harness - the runtime layer that turns an LLM into a working agent.

benchmark(tfy): ship the validated optimized config + replication robustness (#212)

shubhamagarwal-create0e40e60c604c
diff --git a/benchmark/.env.example b/benchmark/.env.example
index 4da6e1568..294de33da 100644
--- a/benchmark/.env.example
+++ b/benchmark/.env.example
@@ -31,11 +31,17 @@ JUDGE_MODEL=claude-opus-4-8
# ---- TrueForge arm ----------------------------------------------------------
# Point at your running TrueForge server's HTTP API. It needs no auth by default. In
-# TrueForge: configure a provider for MODEL_TFY, and register the MCP_CONFIG servers as
-# Connectors under the same names used in mcp_config.json.
+# TrueForge you must configure THREE things (see README "TrueForge setup"):
+# 1. a model provider for MODEL_TFY;
+# 2. the MCP_CONFIG servers registered as Connectors under the same names;
+# 3. a SANDBOX PROVIDER (e.g. Daytona). The benchmark config offloads oversized tool
+# results to a sandbox file (large_tool_response); without a sandbox the data-heavy
+# tasks pile everything into context and abort with "max_tokens breached".
+# The rest of the agent config (sub-agents on, compaction, etc.) is sent inline by the
+# adapter and matches the validated setup — you don't set it here.
TFY_BASE_URL=http://localhost:8790
MODEL_TFY=anthropic/claude-opus-4-8 # provider/model string as configured in TrueForge
-TFY_ITERATION_LIMIT=100 # operational cap on tool-use iterations per turn
+TFY_ITERATION_LIMIT=500 # tool-use iterations per turn (validated default)
# ---- Claude Managed Agents arm ---------------------------------------------
# ANTHROPIC_API_KEY above is reused. cma_ids.json is created by setup.py.
diff --git a/benchmark/README.md b/benchmark/README.md
index a1188edb8..7b9f82ca4 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -63,10 +63,10 @@ granularity, and use all the related data — with no task-specific detail.
## How it works
1. **`setup.py`** — creates the CMA environment/agent/vault with the shipped system
- prompt and the MCP servers. TrueForge and deepagents need no setup step: TrueForge
+ prompt and the MCP servers. TrueForge and deepagents need no setup step here: TrueForge
sends an inline agent spec when it creates each session, and deepagents attaches the
- MCP tools at run time. (TrueForge does need, on its side, a model provider for
- `MODEL_TFY` and the MCP servers registered as Connectors — see `.env.example`.)
+ MCP tools at run time. (TrueForge does need three things configured on its side — a model
+ provider, the MCP Connectors, and a **sandbox provider** — see **TrueForge setup** below.)
2. **`bench_matrix.py run-all`** — runs every `{arm × task × trial}` cell. Each task
runs as a subprocess with a hard wall-clock timeout and is retried on
timeout / crash / empty answer. Fully resumable — rerun to continue.
@@ -78,6 +78,45 @@ granularity, and use all the related data — with no task-specific detail.
4. **`aggregate.py`** — rolls the matrix up into per-arm accuracy, cost, and latency.
`Solved / 14` is the mean number of tasks passed per trial. Writes `results/summary.csv`.
+### TrueForge setup (required to reproduce)
+
+The `tfy` arm talks to a running TrueForge server (`TFY_BASE_URL`). Before running it,
+configure three things **inside TrueForge** — miss any one and the data-heavy tasks fail:
+
+1. **A model provider** for `MODEL_TFY` (e.g. `anthropic/claude-opus-4-8`, `zai/glm-5-2`).
+2. **The MCP servers** from `mcp_config.json`, registered as Connectors **under the same
+ names**.
+3. **A sandbox provider** (e.g. Daytona). This is the one people miss. The benchmark config
+ enables `large_tool_response`, which **offloads oversized query results to a sandbox
+ file** instead of letting them pile into the model's context. Several of these tasks scan
+ tens of thousands of records; **without a sandbox the context blows past the token budget
+ and the turn aborts with `max_tokens breached`.**
+
+The agent config is sent **inline by the adapter** (`run_tfy` in `bench_matrix.py`) — you do
+not set it in TrueForge — and is the validated setup that produced the results below:
+
+| Setting | Value | Why |
+| ---------------------------------------- | ----------------------------- | -------------------------------------------------------- |
+| `iteration_limit` | **500** | enough tool-use steps to fully work a cross-system task |
+| `dynamic_sub_agents` | **enabled** | decompose big sub-tasks to sub-agents with fresh context |
+| `sandbox` | **enabled** | needed for `large_tool_response` offload (see #3 above) |
+| `context_management.compaction` | **enabled, threshold 60 000** | summarize old history so long turns stay under budget |
+| `context_management.large_tool_response` | **enabled** | offload oversized tool results to the sandbox |
+| `generative_ui` | disabled | not needed for a headless benchmark |
+| `ask_user_questions` | disabled | fully autonomous; never pause a turn for input |
+
+**Long turns.** The heaviest tasks run for several minutes. TrueForge streams turn events
+over SSE, and on a long turn that stream can close before the turn finishes (the turn keeps
+running server-side). The adapter handles this: if the stream ends without a terminal
+`turn.done`, it **polls the turn to its terminal state** rather than recording an empty
+answer. If you write your own client, do the same, or long tasks will look like spurious
+failures.
+
+**Strict MCP servers.** The MCP spec requires clients to send
+`Accept: application/json, text/event-stream`. If your MCP server enforces this strictly and
+TrueForge's connector doesn't satisfy it, tool listing fails; use an MCP server (or a thin
+proxy) that accepts the header TrueForge sends.
+
### Cost model
Cost is computed from token counts at provider list prices (`$/1M`), not estimated:
diff --git a/benchmark/bench_matrix.py b/benchmark/bench_matrix.py
index e2b7a7eb6..9f1cbe82e 100644
--- a/benchmark/bench_matrix.py
+++ b/benchmark/bench_matrix.py
@@ -50,10 +50,27 @@ def run_tfy(task, prompt, cap):
"instructions": SYSTEM_PROMPT,
"mcp_servers": [{"name": n, "enable_tools": ["@all"], "require_approval_for_tools": []}
for n in mcp],
- # Fully autonomous: no approval gates (above) and no clarifying-question pauses,
- # either of which would end a turn early with incomplete work.
- "config": {"iteration_limit": int(os.environ.get("TFY_ITERATION_LIMIT", "100")),
- "ask_user_questions": {"enabled": False}},
+ # Validated benchmark config (the setup that produced the published results):
+ # - fully autonomous: no approval gates (above) and no clarifying-question pauses,
+ # either of which would end a turn early with incomplete work;
+ # - dynamic sub-agents ON and a 500-iteration budget, so the agent can fully work
+ # the data-heavy cross-system tasks (decompose big sub-tasks, take enough steps);
+ # - compaction + large_tool_response keep long turns under the token budget:
+ # large_tool_response OFFLOADS oversized query results to a sandbox file instead
+ # of piling them into context. This REQUIRES a sandbox provider configured in
+ # TrueForge (see README "Sandbox"); without one the offload can't happen and the
+ # heaviest tasks abort with "max_tokens breached".
+ "config": {
+ "iteration_limit": int(os.environ.get("TFY_ITERATION_LIMIT", "500")),
+ "ask_user_questions": {"enabled": False},
+ "dynamic_sub_agents": {"enabled": True},
+ "sandbox": {"enabled": True},
+ "context_management": {
+ "compaction": {"enabled": True, "compaction_threshold_tokens": 60000},
+ "large_tool_response": {"enabled": True},
+ },
+ "generative_ui": {"enabled": False},
+ },
}
def _post(path, body, stream=False):
@@ -62,6 +79,10 @@ def run_tfy(task, prompt, cap):
"Accept": "text/event-stream" if stream else "application/json"}, method="POST")
return urllib.request.urlopen(req, timeout=TASK_TIMEOUT)
+ def _get(path):
+ req = urllib.request.Request(base + path, headers={"Accept": "application/json"})
+ return json.load(urllib.request.urlopen(req, timeout=60))
+
def text_of(content):
if isinstance(content, str):
return content
@@ -71,7 +92,7 @@ def run_tfy(task, prompt, cap):
sid = json.load(_post("/api/v1/sessions", {"agent": {"spec": spec}}))["data"]["id"]
tot = {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0}
- tools = 0; answer = ""; last_msg = ""; cost = None
+ tools = 0; answer = ""; last_msg = ""; cost = None; terminal = False
resp = _post(f"/api/v1/sessions/{sid}/turns",
{"input": [{"type": "user.message", "content": prompt}],
"previous_turn_id": "none", "stream": True}, stream=True)
@@ -100,7 +121,33 @@ def run_tfy(task, prompt, cap):
cost = m.get("total_cost_in_usd")
out = state.get("output") or {}
answer = text_of(out.get("content") if isinstance(out, dict) else None)
+ terminal = True
break
+ # On the data-heavy tasks a turn can run for several minutes, and the SSE stream may
+ # close before the turn finishes (the turn keeps running server-side). Don't treat the
+ # stream ending as "done" — poll the turn to its terminal state so we capture the real
+ # answer/metrics instead of an empty result. This is what keeps the heaviest tasks from
+ # being scored as spurious failures.
+ if not terminal:
+ try:
+ turns = _get(f"/api/v1/sessions/{sid}/turns").get("data", []) or []
+ tid = (turns[0] if turns else {}).get("id")
+ deadline = time.time() + TASK_TIMEOUT
+ while tid and time.time() < deadline:
+ st = (_get(f"/api/v1/sessions/{sid}/turns/{tid}").get("data", {}) or {}).get("state", {}) or {}
+ if st.get("status") in ("done", "error", "cancelled"):
+ m = st.get("metrics") or {}
+ tot = {"input": m.get("total_input_tokens", 0) or 0,
+ "output": m.get("total_output_tokens", 0) or 0,
+ "cache_read": m.get("total_cache_read_tokens", 0) or 0,
+ "cache_write": m.get("total_cache_write_tokens", 0) or 0}
+ cost = m.get("total_cost_in_usd")
+ out = st.get("output") or {}
+ answer = text_of(out.get("content") if isinstance(out, dict) else None) or answer
+ break
+ time.sleep(6)
+ except Exception:
+ pass
if not answer:
answer = last_msg
res = {"session_id": sid, "tokens": tot, "tool_calls": tools, "answer": answer}
diff --git a/benchmark/setup.py b/benchmark/setup.py
index 6e3307fb0..0a4b9d956 100644
--- a/benchmark/setup.py
+++ b/benchmark/setup.py
@@ -65,8 +65,11 @@ if __name__ == "__main__":
if "cma" in harnesses:
setup_cma()
if "tfy" in harnesses:
- print("[tfy] no setup step here — in TrueForge, configure a provider for "
- "MODEL_TFY and register the MCP_CONFIG servers as Connectors under the "
- "same names; the agent spec is sent inline when each session is created.",
- flush=True)
+ print("[tfy] no setup step here, but TrueForge itself needs THREE things "
+ "configured (see README): (1) a model provider for MODEL_TFY, (2) the "
+ "MCP_CONFIG servers registered as Connectors under the same names, and "
+ "(3) a SANDBOX PROVIDER (e.g. Daytona) — required so oversized tool results "
+ "offload to a file instead of aborting the heaviest tasks. The agent spec "
+ "(sub-agents, compaction, sandbox on, 500 iterations) is sent inline when "
+ "each session is created.", flush=True)
print("[setup] done", flush=True)