The open-source agent harness - the runtime layer that turns an LLM into a working agent.
Add agent-harness cost/accuracy benchmark (#124)
shubhamagarwal-createdde81f592dd5
diff --git a/README.md b/README.mdindex 84a517112..0ede8176b 100644--- a/README.md+++ b/README.md@@ -186,6 +186,10 @@ flowchart LR- [API reference](#)- [Deploying to Kubernetes](#)+## Benchmarks++We compare TrueForge against other agent frameworks (Claude Managed Agents, deepagents) on the same tasks, the same model, and the same prompt, and report accuracy alongside cost and latency. The harness, the exact prompt, the blind grader, and our results all live in [`benchmark/`](benchmark/) - and it is built to be re-run against your own dataset and models.+## ContributingWe love contributions! Whether it's a bug report, a new feature, or a docs fix - see [CONTRIBUTING.md](CONTRIBUTING.md) for how to set up a development environment and open a pull request. Please also read our [Code of Conduct](CODE_OF_CONDUCT.md).diff --git a/benchmark/.env.example b/benchmark/.env.examplenew file mode 100644index 000000000..4da6e1568--- /dev/null+++ b/benchmark/.env.example@@ -0,0 +1,57 @@+# ---------------------------------------------------------------------------+# Copy to `.env` and fill in. `source .env` (or use direnv) before running.+# Nothing here is committed — .env is gitignored.+# ---------------------------------------------------------------------------++# ---- Shared -----------------------------------------------------------------+# The task suite. Each task is a dir: <task>/prompt.txt and <task>/tests/criteria.yaml.+# Point this at your local copy of the DevRev Enterprise-Bench L1-L2 tasks+# (see README — we do not redistribute the dataset).+DATASET_DIR=./dataset++# The single system prompt used by every harness. Keep it uniform across arms.+SYSTEM_PROMPT=./prompts/system.md++# MCP servers exposed to the agents, as a JSON file: {"pm": "https://...", ...}.+MCP_CONFIG=./mcp_config.json++# Which arms to run / set up (comma-separated): tfy, cma, deepagents+HARNESSES=tfy,cma,deepagents++# Trials per (harness, task); wall-clock kill per task (s); attempts before giving up.+N_TRIALS=3+TASK_TIMEOUT=1500+MAX_ATTEMPTS=3++OUT_DIR=./results++# ---- Judge (Anthropic) ------------------------------------------------------+ANTHROPIC_API_KEY=sk-ant-...+JUDGE_MODEL=claude-opus-4-8++# ---- TrueForge arm ----------------------------------------------------------+# Point at your running TrueForge server's HTTP API. It needs no auth by default. In+# TrueForge: configure a provider for MODEL_TFY, and register the MCP_CONFIG servers as+# Connectors under the same names used in mcp_config.json.+TFY_BASE_URL=http://localhost:8790+MODEL_TFY=anthropic/claude-opus-4-8 # provider/model string as configured in TrueForge+TFY_ITERATION_LIMIT=100 # operational cap on tool-use iterations per turn++# ---- Claude Managed Agents arm ---------------------------------------------+# ANTHROPIC_API_KEY above is reused. cma_ids.json is created by setup.py.+CMA_IDS=./results/cma_ids.json+MODEL_CMA=claude-opus-4-8+# Only if your MCP servers require a bearer token (leave unset for open servers):+# MCP_BEARER=<token>++# ---- deepagents arm (OpenAI-compatible endpoint for the model) --------------+DA_MODEL=<model-id>+DA_BASE_URL=https://<openai-compatible-endpoint>/v1+DA_API_KEY=<key>+DA_RECURSION=300++# ---- Cost rates ($/1M tokens) ----------------------------------------------+# Defaults are Opus 4.8 ({"in":5,"out":25,"cache_read":0.5,"cache_write":6.25}).+# Override globally with RATES, or per-arm with RATES_TFY / RATES_CMA / RATES_DEEPAGENTS.+# Example: GLM-5.2 on the TFY arm:+# RATES_TFY={"in":0.7266,"out":2.284,"cache_read":0.18,"cache_write":0.9}diff --git a/benchmark/.gitignore b/benchmark/.gitignorenew file mode 100644index 000000000..4c8d5e1e2--- /dev/null+++ b/benchmark/.gitignore@@ -0,0 +1,6 @@+.env+results/+mcp_config.json+dataset/+__pycache__/+*.pycdiff --git a/benchmark/Dockerfile b/benchmark/Dockerfilenew file mode 100644index 000000000..1144deddf--- /dev/null+++ b/benchmark/Dockerfile@@ -0,0 +1,13 @@+# Runs the core arms (TrueForge + CMA) and the judge/aggregate pipeline.+# The deepagents arm has conflicting deps — run it in a separate venv/image+# (see README). Config comes from a mounted .env and mcp_config.json.+FROM python:3.11-slim++WORKDIR /bench+COPY requirements.txt .+RUN pip install --no-cache-dir -r requirements.txt+COPY . .++# Full pipeline: set up agents, run the matrix, grade, aggregate.+# Override HARNESSES in the env to run a subset.+CMD ["bash", "-lc", "python setup.py && python bench_matrix.py run-all && python judge.py && python aggregate.py"]diff --git a/benchmark/README.md b/benchmark/README.mdnew file mode 100644index 000000000..6cebbea71--- /dev/null+++ b/benchmark/README.md@@ -0,0 +1,179 @@+# Agent-harness benchmark: cost & accuracy++A small, self-contained harness for running the **same agent task suite** through+three agent frameworks on the model of your choice, grading the answers with a+**blind LLM judge**, and reporting **accuracy, cost, and latency** side by side.++Built by TrueFoundry to compare [TrueForge](https://github.com/truefoundry/trueforge) (our+open-source agent harness) against **Claude Managed Agents (CMA)** on a suite of+cross-system enterprise tasks; a **deepagents** (LangGraph) arm is included as well.+Everything here is what we run — no hidden scoring, no per-task hints, one uniform+prompt for every framework. Point it at your own dataset and models and you should+reproduce our shape of results.++---++## What it measures++The suite is a set of **cross-system data tasks** over three backend services+exposed as MCP servers:++- **pm** — a Jira-style project-management server (issues, comments, components, wiki)+- **crm** — a Salesforce-style CRM (SOQL-like query, records, schema describe)+- **file-server** — a Drive-style document store (search, read, metadata)++Each task asks a question that requires **joining data across two or three of+these systems** (for example: tie open support tickets to the CRM accounts they+belong to and the engineering components behind them). The agent has to discover+the schemas, figure out the joins, pull the data, and write a grounded answer.++### The dataset++We run against the **L1-L2 tasks from [DevRev's Enterprise-Bench](https://x.com/devrev/status/2075815567458734468)**,+which DevRev open-sourced — dataset, grading criteria, and their+[Harbor](https://github.com/harbor-framework/harbor) evaluation harness (the dataset+is published on Harbor Hub). We do **not** redistribute their data here — get it from+DevRev's release and point `DATASET_DIR` at it. The harness expects each task as:++```+dataset/+ <task-id>/+ prompt.txt # the task instruction given to the agent+ tests/criteria.yaml # the required criteria the judge grades against+```++Any suite in that shape works — this harness is not DevRev-specific.++---++## The three arms++| Arm | Framework | How the agent runs | Tools |+| ------------ | ------------------------------------- | ------------------------------------------------ | --------------------------------------------- |+| `tfy` | TrueForge (TrueFoundry Agent Harness) | Session via TrueForge's own HTTP API (`/api/v1`) | MCP servers registered in TrueForge (by name) |+| `cma` | Claude Managed Agents (beta) | Anthropic-hosted session + sandbox | `mcp_toolset` attached to the agent |+| `deepagents` | deepagents (LangGraph) | Local `create_deep_agent` loop | MCP tools via `MultiServerMCPClient` |++Every arm gets the **same system prompt** (`prompts/system.md`) and the **same task+prompts**. The prompt is generic operating guidance — how to approach a cross-system+task, be thorough with sources, ground claims, respect scope.++---++## How it works++1. **`setup.py`** — creates the CMA environment/agent/vault with the shipped system+ prompt and the MCP servers. TrueForge and deepagents need no setup step: TrueForge+ sends an inline agent spec when it creates each session, and deepagents attaches the+ MCP tools at run time. (TrueForge does need, on its side, a model provider for+ `MODEL_TFY` and the MCP servers registered as Connectors — see `.env.example`.)+2. **`bench_matrix.py run-all`** — runs every `{arm × task × trial}` cell. Each task+ runs as a subprocess with a hard wall-clock timeout and is retried on+ timeout / crash / empty answer. Fully resumable — rerun to continue.+ Answers land in `results/<arm>/t<trial>/<task>.md`; metrics in `results/matrix.jsonl`.+3. **`judge.py`** — grades every answer against its `criteria.yaml`. The judge sees+ only the criteria and the answer, never a reference value and never which arm+ produced it. A task PASSES only if **every** required criterion is met (no+ partial credit). Writes `results/grades.jsonl`.+4. **`aggregate.py`** — rolls the matrix up into per-arm accuracy, cost, and latency.+ `Solved / 14` is the mean number of tasks passed per trial. Writes `results/summary.csv`.++### Cost model++Cost is computed from token counts at provider list prices (`$/1M`), not estimated:++```+cost = (uncached_input * in_rate+ + output * out_rate+ + cache_read * cache_read_rate+ + cache_write * cache_write_rate) / 1e6+```++Token semantics differ by framework, so `aggregate.py` normalizes them+(`INPUT_IS_TOTAL`): for TrueForge and deepagents, `input` is the **total** prompt+and cache reads/writes are a subset (`uncached = input - cache_read - cache_write`);+for CMA, `input` is already the uncached count. Rates default to Opus 4.8; override+per arm with `RATES_<ARM>` (see `.env.example`) when an arm runs a different model.++---++## Run it++```bash+# 1. install (core arms + judge)+python -m venv .venv && source .venv/bin/activate+pip install -r requirements.txt++# 2. configure+cp .env.example .env # fill in keys, hosts, models+cp mcp_config.example.json mcp_config.json # your three MCP server URLs+source .env++# 3. run the full pipeline+python setup.py+python bench_matrix.py run-all+python judge.py+python aggregate.py # prints the table, writes results/summary.csv+```++Run a single arm with `HARNESSES=tfy python bench_matrix.py run-all` (or `run tfy`).++The **deepagents** arm pulls in LangGraph/LangChain, which can conflict with the+core stack. If pip complains, create a second venv from+`requirements-deepagents.txt` and run `HARNESSES=deepagents python bench_matrix.py run tfy`+there; the `results/` directory is shared, so `judge.py` / `aggregate.py` pick it up.++Docker: `docker build -t trueforge-bench . && docker run --env-file .env -v $PWD/results:/bench/results trueforge-bench`+(runs the core arms; see above for deepagents).++---++## Our results++The DevRev L1-L2 suite is **14 tasks**, each run in a fresh session over **n = 3+trials**. `Solved / 14` is the mean number of tasks passed per trial (blind judge,+all-or-nothing); cost is per run at provider list prices.++| Configuration | n | Solved / 14 | Cost / run | Tokens / run |+| -------------------------------- | :-: | :---------: | :--------: | :----------: |+| Claude Managed Agents · Opus 4.8 | 3 | 10.7 | $11.8 | 10.0M |+| TrueForge · Opus 4.8 | 3 | 10.7 | $8.6 | 3.7M |+| TrueForge · GLM-5.2 | 3 | 11.7 | $3.0 | 3.8M |++On the **same model (Opus 4.8)**, TrueForge and Claude Managed Agents solve the same+number of tasks (10.7 each) — matched accuracy — while TrueForge uses far fewer tokens+per run (3.7M vs 10.0M) and so costs less ($8.6 vs $11.8, ~27% lower). Running+TrueForge on the open model **GLM-5.2** solves 11.7 / 14 at $3.0 per run — about 75%+below Claude Managed Agents. Numbers move a task or two run to run, so reproduce the+shape, not a single cell, and plug in your own models and rates.++A **deepagents** (LangGraph) arm is also included in the kit and can be run the same way.++---++## Files++```+prompts/system.md the single system prompt (identical for every arm)+mcp_config.example.json template for your three MCP server URLs+setup.py create the CMA env/agent/vault (TrueForge + deepagents need none)+bench_matrix.py run {arm × task × trial}; resumable, timeout-guarded+judge.py blind criteria-only grader+aggregate.py accuracy + cost + latency -> summary.csv+requirements.txt core arms + judge+requirements-deepagents.txt deepagents arm (separate venv)+Dockerfile core pipeline+.env.example all configuration+```++## Method++- **Same prompt, same tasks.** Every arm gets the identical system prompt and task+ prompts, and each answer is graded from the arm's reply.+- **Blind, strict judge.** The judge sees only the criteria and the answer — never the+ reference values or which arm produced it; a task passes only if every required+ criterion is met (no partial credit).+- **List-price cost.** Computed per cell from token counts at provider list prices,+ normalized for each framework's token reporting.+- **Dataset.** DevRev's Enterprise-Bench; we point at their release and do not ship it.+- **Trials.** Report the mean across trials and compare distributions, not a single cell.diff --git a/benchmark/aggregate.py b/benchmark/aggregate.pynew file mode 100644index 000000000..f87b1f069--- /dev/null+++ b/benchmark/aggregate.py@@ -0,0 +1,110 @@+#!/usr/bin/env python3+"""Aggregate the matrix into per-harness accuracy, cost, and latency.++Reads results/matrix.jsonl (per-cell tokens/latency/tool-calls) and results/grades.jsonl+(per-cell PASS/FAIL), and writes results/summary.csv plus a printed table.++`Solved / 14` is the mean number of tasks passed per trial (each answer graded+all-or-nothing by the blind judge). Cost is computed per cell from token counts and+per-harness rates, then averaged per run.++ python aggregate.py++Token accounting differs by harness (see INPUT_IS_TOTAL): some report `input` as the+TOTAL prompt with cache reads/writes as a subset (uncached = input - cache_read -+cache_write); others report `input` as already-uncached. Rates are $/1M tokens; override+per harness with RATES / RATES_<HARNESS> env vars (see .env.example).+"""+import os, csv, json, pathlib+from collections import defaultdict++BENCH = pathlib.Path(__file__).parent+OUT = pathlib.Path(os.environ.get("OUT_DIR", str(BENCH / "results")))++# True -> `input` is the total prompt, cache_read/cache_write are a subset of it.+# False -> `input` already excludes cached tokens (cache_read/write are separate).+INPUT_IS_TOTAL = {"tfy": True, "deepagents": True, "cma": False}++DEFAULT_RATES = {"in": 5.0, "out": 25.0, "cache_read": 0.5, "cache_write": 6.25} # $/1M (Opus 4.8)+++def rates_for(harness):+ r = dict(DEFAULT_RATES)+ if os.environ.get("RATES"):+ r.update(json.loads(os.environ["RATES"]))+ key = f"RATES_{harness.upper()}"+ if os.environ.get(key):+ r.update(json.loads(os.environ[key]))+ return r+++def cell_cost(harness, tk):+ r = rates_for(harness)+ cr, cw = tk.get("cache_read", 0), tk.get("cache_write", 0)+ uncached = (tk["input"] - cr - cw) if INPUT_IS_TOTAL.get(harness, True) else tk["input"]+ uncached = max(0, uncached)+ return (uncached * r["in"] + tk["output"] * r["out"] + cr * r["cache_read"] + cw * r["cache_write"]) / 1e6+++def load(path):+ rows = []+ p = OUT / path+ if p.exists():+ rows = [json.loads(l) for l in p.read_text().splitlines() if l.strip()]+ return rows+++def main():+ all_rows = load("matrix.jsonl") # both "ok" and "failed" cells+ metrics = {(r["harness"], r["trial"], r["task"]): r for r in all_rows if r.get("status") == "ok"}+ verdicts = {(r["harness"], r["trial"], r["task"]): r["verdict"] for r in load("grades.jsonl")}++ harnesses = sorted({r["harness"] for r in all_rows})+ tasks = sorted({r["task"] for r in all_rows})+ total_tasks = len(tasks)++ # Seed every ATTEMPTED (harness, trial) to zero passes — including a trial whose cells+ # all hard-failed (no "ok" rows). Otherwise that trial would be absent and the mean+ # would divide by fewer trials, overstating accuracy after a full-trial failure.+ solved_per_trial = defaultdict(dict) # solved_per_trial[harness][trial] -> #tasks passed+ for r in all_rows:+ solved_per_trial[r["harness"]].setdefault(r["trial"], 0)+ for (h, tr, t), v in verdicts.items():+ if v == "PASS":+ solved_per_trial[h][tr] = solved_per_trial[h].get(tr, 0) + 1++ per_task = defaultdict(lambda: defaultdict(list)) # per_task[harness][task] -> [PASS/FAIL...]+ per_cell = defaultdict(lambda: {"cost": [], "lat": [], "tok": [], "tools": []})+ for (h, tr, t), m in metrics.items():+ per_task[h][t].append(verdicts.get((h, tr, t), "UNGRADED"))+ per_cell[h]["cost"].append(cell_cost(h, m["tokens"]))+ per_cell[h]["lat"].append(m.get("latency_s", 0))+ per_cell[h]["tok"].append(m["tokens"]["input"] + m["tokens"]["output"])+ per_cell[h]["tools"].append(m.get("tool_calls", 0))++ def mean(xs):+ return sum(xs) / len(xs) if xs else 0.0++ # summary.csv: per (harness, task) pass count across trials+ with open(OUT / "summary.csv", "w", newline="") as f:+ w = csv.writer(f)+ w.writerow(["harness", "task", "trials", "passes"])+ for h in harnesses:+ for t in tasks:+ vs = per_task[h].get(t, [])+ w.writerow([h, t, len(vs), sum(1 for v in vs if v == "PASS")])++ hdr = f"{'harness':22} {'n':>2} {'solved/'+str(total_tasks):>10} {'$/run':>8} {'tokens':>9} {'tools':>6} {'lat_s':>6}"+ print("\n" + hdr); print("-" * len(hdr))+ for h in harnesses:+ trials = solved_per_trial[h]+ solved = mean(list(trials.values())) # mean tasks passed per trial+ c = per_cell[h]+ print(f"{h:22} {len(trials):>2} {solved:>10.1f} "+ f"{'$'+format(mean(c['cost']),'.2f'):>8} {mean(c['tok'])/1e6:>8.2f}M "+ f"{mean(c['tools']):>6.1f} {mean(c['lat']):>6.0f}")+ print(f"\nwrote {OUT/'summary.csv'}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/bench_matrix.py b/benchmark/bench_matrix.pynew file mode 100644index 000000000..e2b7a7eb6--- /dev/null+++ b/benchmark/bench_matrix.py@@ -0,0 +1,269 @@+#!/usr/bin/env python3+"""TrueForge benchmark runner: {harness} x {task} x {trial}.++Runs the same task suite through one or more agent harnesses (TrueForge, Claude+Managed Agents, deepagents) on a model of your choice, one fresh session per task,+N trials each. Resumable by (harness, trial, task) — rerun to continue.++Pipeline:+ python bench_matrix.py run-all # every configured harness, N trials+ python judge.py # blind grade every captured answer+ python aggregate.py # per-task pass-rate + cost/latency -> summary.csv++Robustness (each task runs as a subprocess with a hard wall-clock timeout; retries on+timeout / crash / empty answer). Config is entirely via environment — see .env.example.+"""+import os, sys, json, time, pathlib, subprocess++BENCH = pathlib.Path(__file__).parent+OUT = pathlib.Path(os.environ.get("OUT_DIR", str(BENCH / "results")))+MATRIX = OUT / "matrix.jsonl"+DATASET_DIR = pathlib.Path(os.environ.get("DATASET_DIR", str(BENCH / "dataset")))+SYSTEM_PROMPT = pathlib.Path(os.environ.get("SYSTEM_PROMPT", str(BENCH / "prompts" / "system.md"))).read_text()++N_TRIALS = int(os.environ.get("N_TRIALS", "3"))+TASK_TIMEOUT = int(os.environ.get("TASK_TIMEOUT", "1500")) # hard per-task kill (seconds)+MAX_ATTEMPTS = int(os.environ.get("MAX_ATTEMPTS", "3"))+# Which harnesses to run, comma-separated. Any of: tfy, cma, deepagents.+HARNESSES = [h.strip() for h in os.environ.get("HARNESSES", "tfy,cma,deepagents").split(",") if h.strip()]+++def tasks():+ """Every task in DATASET_DIR: a subdir containing tests/criteria.yaml and prompt.txt."""+ return sorted(p.name for p in DATASET_DIR.iterdir()+ if (p / "tests" / "criteria.yaml").exists() and (p / "prompt.txt").exists())+++def prompt_for(task):+ return (DATASET_DIR / task / "prompt.txt").read_text().strip()+++# ---------------- TrueForge adapter (native /api/v1 HTTP API, stdlib only) ----------------+def run_tfy(task, prompt, cap):+ import urllib.request+ base = os.environ["TFY_BASE_URL"].rstrip("/")+ mcp = json.loads(pathlib.Path(os.environ["MCP_CONFIG"]).read_text()) # {name: url}+ # Inline agent spec: the shared model + system prompt and the MCP connectors (by+ # the names they are registered under in TrueForge), autonomous (no approval gates).+ spec = {+ "model": {"name": os.environ["MODEL_TFY"]},+ "instructions": SYSTEM_PROMPT,+ "mcp_servers": [{"name": n, "enable_tools": ["@all"], "require_approval_for_tools": []}+ for n in mcp],+ # Fully autonomous: no approval gates (above) and no clarifying-question pauses,+ # either of which would end a turn early with incomplete work.+ "config": {"iteration_limit": int(os.environ.get("TFY_ITERATION_LIMIT", "100")),+ "ask_user_questions": {"enabled": False}},+ }++ def _post(path, body, stream=False):+ req = urllib.request.Request(base + path, data=json.dumps(body).encode(),+ headers={"Content-Type": "application/json",+ "Accept": "text/event-stream" if stream else "application/json"}, method="POST")+ return urllib.request.urlopen(req, timeout=TASK_TIMEOUT)++ def text_of(content):+ if isinstance(content, str):+ return content+ if isinstance(content, list):+ return "\n".join(b.get("text", "") for b in content if isinstance(b, dict))+ return ""++ sid = json.load(_post("/api/v1/sessions", {"agent": {"spec": spec}}))["data"]["id"]+ tot = {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0}+ tools = 0; answer = ""; last_msg = ""; cost = None+ resp = _post(f"/api/v1/sessions/{sid}/turns",+ {"input": [{"type": "user.message", "content": prompt}],+ "previous_turn_id": "none", "stream": True}, stream=True)+ for raw in resp: # Server-Sent Events, one JSON object per `data:` line+ line = raw.decode("utf-8", "ignore").strip()+ if not line.startswith("data:"):+ continue+ try:+ ev = json.loads(line[5:].strip())+ except Exception:+ continue+ et = ev.get("type", "")+ if et == "tool.response":+ tools += 1+ elif et == "model.message":+ t = text_of(ev.get("content"))+ if t.strip():+ last_msg = t+ elif et == "turn.done":+ state = ev.get("state", {}) or {}+ m = state.get("metrics") or {}+ tot = {"input": m.get("total_input_tokens", 0) or 0,+ "output": m.get("total_output_tokens", 0) or 0,+ "cache_read": m.get("total_cache_read_tokens", 0) or 0,+ "cache_write": m.get("total_cache_write_tokens", 0) or 0}+ cost = m.get("total_cost_in_usd")+ out = state.get("output") or {}+ answer = text_of(out.get("content") if isinstance(out, dict) else None)+ break+ if not answer:+ answer = last_msg+ res = {"session_id": sid, "tokens": tot, "tool_calls": tools, "answer": answer}+ if cost is not None:+ res["cost_usd"] = cost # harness-reported cost, kept for cross-check+ cap.put(res)+++# ---------------- Claude Managed Agents adapter ----------------+def run_cma(task, prompt, cap):+ import anthropic+ ids = json.loads(pathlib.Path(os.environ["CMA_IDS"]).read_text()) # {agent_id, env_id, vault_id?}+ client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])+ kw = {"agent": ids["agent_id"], "environment_id": ids["env_id"], "title": f"bench {task}"}+ if ids.get("vault_id"):+ kw["vault_ids"] = [ids["vault_id"]]+ s = client.beta.sessions.create(**kw)+ ans = ""; tools = 0+ with client.beta.sessions.events.stream(s.id) as stream:+ client.beta.sessions.events.send(s.id, events=[{"type": "user.message",+ "content": [{"type": "text", "text": prompt}]}])+ for e in stream:+ et = getattr(e, "type", "")+ if et == "agent.message":+ txt = "\n".join([getattr(b, "text", "") or "" for b in getattr(e, "content", []) or []])+ if txt:+ ans = txt+ elif et in ("agent.tool_use", "agent.mcp_tool_use"):+ tools += 1+ elif et == "session.status_idle":+ break+ u = getattr(client.beta.sessions.retrieve(s.id), "usage", None)+ tot = {"input": getattr(u, "input_tokens", 0) or 0, "output": getattr(u, "output_tokens", 0) or 0,+ "cache_read": getattr(u, "cache_read_input_tokens", 0) or 0,+ "cache_write": getattr(u, "cache_creation_input_tokens", 0) or 0}+ cap.put({"session_id": s.id, "tokens": tot, "tool_calls": tools, "answer": ans})+++# ---------------- deepagents (LangGraph) adapter ----------------+def run_deepagents(task, prompt, cap):+ import asyncio+ from langchain_mcp_adapters.client import MultiServerMCPClient+ from langchain_openai import ChatOpenAI+ from deepagents import create_deep_agent+ mcp = json.loads(pathlib.Path(os.environ["MCP_CONFIG"]).read_text())++ async def _run():+ client = MultiServerMCPClient({n: {"url": u, "transport": "streamable_http"} for n, u in mcp.items()})+ tools = await client.get_tools()+ model = ChatOpenAI(model=os.environ["DA_MODEL"], base_url=os.environ["DA_BASE_URL"],+ api_key=os.environ["DA_API_KEY"], timeout=180, max_retries=5)+ agent = create_deep_agent(model=model, tools=tools, system_prompt=SYSTEM_PROMPT)+ res = await agent.ainvoke({"messages": [{"role": "user", "content": prompt}]},+ {"recursion_limit": int(os.environ.get("DA_RECURSION", "300"))})+ tot = {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0}; tools_n = 0; answer = ""+ for m in res["messages"]:+ if m.__class__.__name__ != "AIMessage":+ continue+ um = getattr(m, "usage_metadata", None) or {}+ tot["input"] += int(um.get("input_tokens", 0) or 0); tot["output"] += int(um.get("output_tokens", 0) or 0)+ det = um.get("input_token_details", {}) or {}+ tot["cache_read"] += int(det.get("cache_read", 0) or 0); tot["cache_write"] += int(det.get("cache_creation", 0) or 0)+ tools_n += len(getattr(m, "tool_calls", []) or [])+ c = getattr(m, "content", None)+ if isinstance(c, str) and c.strip():+ answer = c+ elif isinstance(c, list):+ txt = "\n".join(b.get("text", "") for b in c if isinstance(b, dict) and b.get("type") in (None, "text"))+ if txt.strip():+ answer = txt+ # Grade the reply, exactly like the other arms. Fallback only: if the agent left+ # its answer in its virtual filesystem instead of the reply, grade those files so+ # deepagents is not under-graded — this never adds to any other arm's answer.+ if not answer.strip():+ parts = []+ for p, fd in (res.get("files") or {}).items():+ if "large_tool_results" in p: # offloaded raw tool dumps — scratch, not deliverable+ continue+ content = fd.get("content") if isinstance(fd, dict) else getattr(fd, "content", None)+ if isinstance(content, list):+ content = "\n".join(str(x) for x in content)+ if content and str(content).strip():+ parts.append(f"### FILE: {p}\n{content}")+ answer = "\n\n".join(parts)+ return {"session_id": f"da-{task}", "tokens": tot, "tool_calls": tools_n, "answer": answer}++ cap.put(asyncio.run(_run()))+++ADAPTER = {"tfy": run_tfy, "cma": run_cma, "deepagents": run_deepagents}+++def _done_cells():+ d = set()+ if MATRIX.exists():+ for line in MATRIX.read_text().splitlines():+ if not line.strip():+ continue+ r = json.loads(line)+ if r.get("status") == "ok" and r.get("answer_chars", 0) > 0:+ d.add((r["harness"], r["trial"], r["task"]))+ return d+++def run_cell(harness, trial, task):+ outdir = OUT / harness / f"t{trial}"; outdir.mkdir(parents=True, exist_ok=True)+ for attempt in range(1, MAX_ATTEMPTS + 1):+ tmp = OUT / f".cell_{harness}_t{trial}_{task}.json"+ if tmp.exists():+ tmp.unlink()+ t0 = time.time()+ try:+ subprocess.run([sys.executable, str(BENCH / "bench_matrix.py"), "_one", harness, task, str(tmp)],+ timeout=TASK_TIMEOUT, env=os.environ, cwd=str(BENCH))+ except subprocess.TimeoutExpired:+ print(f" [{harness} t{trial} {task}] TIMEOUT attempt {attempt}", flush=True); time.sleep(3); continue+ if not tmp.exists():+ print(f" [{harness} t{trial} {task}] crashed attempt {attempt}", flush=True); time.sleep(5 * attempt); continue+ try:+ res = json.loads(tmp.read_text()); tmp.unlink()+ except Exception:+ print(f" [{harness} t{trial} {task}] bad-output attempt {attempt}", flush=True); time.sleep(5 * attempt); continue+ if not res.get("answer"):+ print(f" [{harness} t{trial} {task}] empty answer attempt {attempt}", flush=True); time.sleep(5 * attempt); continue+ (outdir / f"{task}.md").write_text(res["answer"])+ row = {"harness": harness, "trial": trial, "task": task, "status": "ok",+ "session_id": res.get("session_id"), "latency_s": round(time.time() - t0, 1),+ "tool_calls": res["tool_calls"], "tokens": res["tokens"], "answer_chars": len(res["answer"])}+ with open(MATRIX, "a") as f:+ f.write(json.dumps(row) + "\n")+ print(f" [{harness} t{trial} {task}] ok {row['latency_s']}s tools={row['tool_calls']}", flush=True)+ return+ with open(MATRIX, "a") as f:+ f.write(json.dumps({"harness": harness, "trial": trial, "task": task, "status": "failed", "answer_chars": 0}) + "\n")+ print(f" [{harness} t{trial} {task}] FAILED after {MAX_ATTEMPTS} attempts", flush=True)+++def run_harness(harness, trials):+ done = _done_cells()+ for trial in range(1, trials + 1):+ for task in tasks():+ if (harness, trial, task) in done:+ print(f"skip {harness} t{trial} {task}", flush=True); continue+ run_cell(harness, trial, task)+++if __name__ == "__main__":+ OUT.mkdir(parents=True, exist_ok=True)+ mode = sys.argv[1]+ if mode == "_one":+ # private single-cell worker: run one adapter, write result JSON, exit+ harness, task, outpath = sys.argv[2], sys.argv[3], sys.argv[4]+ class _Cap:+ val = None+ def put(self, x): self.val = x+ cap = _Cap()+ ADAPTER[harness](task, prompt_for(task), cap)+ pathlib.Path(outpath).write_text(json.dumps(cap.val))+ sys.exit(0)+ elif mode == "run":+ run_harness(sys.argv[2], N_TRIALS)+ elif mode == "run-all":+ for h in HARNESSES:+ run_harness(h, N_TRIALS)+ else:+ print(__doc__); sys.exit(1)diff --git a/benchmark/judge.py b/benchmark/judge.pynew file mode 100644index 000000000..d2168a74d--- /dev/null+++ b/benchmark/judge.py@@ -0,0 +1,78 @@+#!/usr/bin/env python3+"""Blind LLM judge.++Grades every captured answer against its task's required criteria. The judge sees+ONLY the criteria and the answer — never the reference values, never which harness+produced the answer. A task is PASS only if every required criterion is satisfied+(no partial credit). Resumable: rerun to grade any answers not yet graded.++ python judge.py++Config (see .env.example): ANTHROPIC_API_KEY, JUDGE_MODEL (default claude-opus-4-8),+OUT_DIR, DATASET_DIR.+"""+import os, re, json, time, pathlib+import anthropic++BENCH = pathlib.Path(__file__).parent+OUT = pathlib.Path(os.environ.get("OUT_DIR", str(BENCH / "results")))+DATASET_DIR = pathlib.Path(os.environ.get("DATASET_DIR", str(BENCH / "dataset")))+GRADES = OUT / "grades.jsonl"+JUDGE_MODEL = os.environ.get("JUDGE_MODEL", "claude-opus-4-8")++# Criteria-only rubric. No reference answers, no per-task hints — the judge reasons+# from the criteria alone so the grade reflects the answer, not a leaked key.+SYSTEM = (+ "You grade a benchmark answer against its REQUIRED criteria. ALL required criteria "+ "must be satisfied to PASS; be strict and do not award partial credit. Work through "+ "each required criterion briefly against the answer, then end with a final line that "+ "is EXACTLY 'VERDICT: PASS' or 'VERDICT: FAIL'."+)++client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])+++def grade(criteria, answer):+ for attempt in range(5):+ try:+ msg = client.messages.create(+ model=JUDGE_MODEL, max_tokens=4000, system=SYSTEM,+ messages=[{"role": "user",+ "content": f"REQUIRED CRITERIA:\n{criteria}\n\nANSWER:\n{answer[:60000]}"}])+ txt = "".join(b.text for b in msg.content if getattr(b, "type", "") == "text")+ m = re.findall(r"VERDICT:\s*(PASS|FAIL)", txt, re.I) or re.findall(r"\b(PASS|FAIL)\b", txt, re.I)+ return (m[-1].upper() if m else "UNKNOWN")+ except Exception as e:+ last = e; time.sleep(5 * (attempt + 1))+ raise last+++def criteria_for(task):+ return (DATASET_DIR / task / "tests" / "criteria.yaml").read_text()+++def main():+ done = set()+ if GRADES.exists():+ for line in GRADES.read_text().splitlines():+ if line.strip():+ r = json.loads(line); done.add((r["harness"], r["trial"], r["task"]))+ answers = sorted(OUT.glob("*/t*/*.md"))+ with open(GRADES, "a") as out:+ for p in answers:+ task = p.stem+ trial = int(p.parent.name[1:]) # t1 -> 1+ harness = p.parent.parent.name+ if (harness, trial, task) in done:+ continue+ if not (DATASET_DIR / task / "tests" / "criteria.yaml").exists():+ print(f"skip {harness} t{trial} {task}: no criteria", flush=True); continue+ v = grade(criteria_for(task), p.read_text())+ out.write(json.dumps({"harness": harness, "trial": trial, "task": task, "verdict": v}) + "\n")+ out.flush()+ print(f"{harness} t{trial} {task}: {v}", flush=True)+ print("[judge] done", flush=True)+++if __name__ == "__main__":+ main()diff --git a/benchmark/mcp_config.example.json b/benchmark/mcp_config.example.jsonnew file mode 100644index 000000000..01a962240--- /dev/null+++ b/benchmark/mcp_config.example.json@@ -0,0 +1,5 @@+{+ "pm": "https://<host>/mcp/pm/server",+ "crm": "https://<host>/mcp/crm/server",+ "file-server": "https://<host>/mcp/file-server/server"+}diff --git a/benchmark/prompts/system.md b/benchmark/prompts/system.mdnew file mode 100644index 000000000..fdf1c1b7c--- /dev/null+++ b/benchmark/prompts/system.md@@ -0,0 +1,54 @@+# Bench task agent guide++You are solving ONE cross-system data task over three backend services, exposed+to you as MCP tools. There is no scoring here; produce the most correct,+complete, well-organized answer you can.++## How to call the tools++The three servers below are attached as MCP tools you can call directly. Before+you use a server, list its tools once to read the exact tool names, parameters,+and query syntax; do not guess a signature. The servers are `pm`, `crm`, and+`file-server`.++### Servers & tools++- **pm** (Jira-style): `pm_search_issues` (query language: project/status/assignee/+ reporter/priority/component etc.), `pm_get_issue`, `pm_get_comments`, `pm_get_user`,+ `pm_list_components`, `wiki_search`, `wiki_get_page`, `wiki_list_pages`.+- **crm** (Salesforce-style): `crm_query` (SOQL: SELECT/WHERE/LIKE/ORDER BY/LIMIT),+ `crm_get_record`, `crm_describe` (object schema), `crm_list_objects`, `crm_search`.+ Objects: Account (42), Opportunity (8704), Case (32768), User (289).+- **file-server** (Drive-style): `search_files`, `get_file_metadata`,+ `read_file_content`, `list_recent_files`. Holds company docs, transcripts, MSAs.++## Method++1. `crm_describe` / `pm_list_components` / `crm_list_objects` to learn the schema+ BEFORE querying. Do not guess field names.+2. Figure out how the systems join (e.g. tickets = CRM Cases; product component or+ area links Cases, PM issues, and components by PART-ID; account ARR on Account).+3. Pull the data with targeted queries; paginate if a result is truncated.+4. Do the analysis/aggregation carefully. State any assumptions you had to make+ (e.g. what counts as "open", how you mapped component to product area).+5. If the task names a person, deal, or time window, filter to exactly that.++## Output++Reply with your COMPLETE answer (do not rely on writing files — put everything in+your reply). Include:++- A short "Approach & assumptions" note (which objects/fields/joins you used).+- The requested table(s) or analysis as the main deliverable.+- If data was missing or ambiguous, say so explicitly rather than inventing values.++## Guidance for good answers++1. Granularity: answer at the level of specificity the question asks for. Do not+ summarize detailed results into broader aggregates unless the user asks for a summary.+2. Be thorough with sources: consult all the data available to you before concluding;+ one system rarely has the whole picture.+3. Ground your claims: base conclusions on specific evidence you actually retrieved,+ not on high-level fields or assumptions alone.+4. Respect scope: if the question names a person, entity, or time period, constrain+ the analysis to exactly that.diff --git a/benchmark/requirements-deepagents.txt b/benchmark/requirements-deepagents.txtnew file mode 100644index 000000000..c4b316772--- /dev/null+++ b/benchmark/requirements-deepagents.txt@@ -0,0 +1,5 @@+# deepagents arm only — install in a separate venv from the core stack.+deepagents>=0.1+langgraph>=0.2+langchain-openai>=0.2+langchain-mcp-adapters>=0.1diff --git a/benchmark/requirements.txt b/benchmark/requirements.txtnew file mode 100644index 000000000..baa3f71f3--- /dev/null+++ b/benchmark/requirements.txt@@ -0,0 +1,8 @@+# Core arms (TrueForge + CMA) and the judge.+# - TrueForge arm calls its own /api/v1 HTTP API over the stdlib (no extra dep).+# - CMA arm and the judge use the Anthropic SDK.+anthropic>=0.40++# The deepagents arm pulls in LangGraph + LangChain, which can conflict with the+# core stack. If pip resolves a conflict, install the deepagents arm in its own+# venv from requirements-deepagents.txt and run only that arm there.diff --git a/benchmark/setup.py b/benchmark/setup.pynew file mode 100644index 000000000..6e3307fb0--- /dev/null+++ b/benchmark/setup.py@@ -0,0 +1,72 @@+#!/usr/bin/env python3+"""One-time setup for the CMA arm.++- CMA: creates a cloud environment, an agent (model = MODEL_CMA, system = the shipped+ system prompt, with the MCP servers attached as always-allow toolsets), and — if the+ servers need a bearer token — a vault with one static-bearer credential per server+ URL. Writes results/cma_ids.json {env_id, agent_id, vault_id}. Skips if that file+ already exists, so it is safe to rerun.++The TrueForge and deepagents arms need no setup step here:+- TrueForge attaches an inline agent spec when it creates each session (see run_tfy in+ bench_matrix.py). Its prerequisites live in TrueForge itself: a model provider for+ MODEL_TFY, and the MCP servers registered as Connectors under the names in MCP_CONFIG.+- deepagents attaches the MCP tools at run time.++ python setup.py++Config: MCP_CONFIG (JSON file {name: url}), SYSTEM_PROMPT, ANTHROPIC_API_KEY, MODEL_CMA,+and optional MCP_BEARER. See .env.example.+"""+import os, json, pathlib++BENCH = pathlib.Path(__file__).parent+OUT = pathlib.Path(os.environ.get("OUT_DIR", str(BENCH / "results")))+SYSTEM = pathlib.Path(os.environ.get("SYSTEM_PROMPT", str(BENCH / "prompts" / "system.md"))).read_text()+MCP = json.loads(pathlib.Path(os.environ["MCP_CONFIG"]).read_text()) # {name: url}+MODEL_CMA = os.environ.get("MODEL_CMA", "claude-opus-4-8")+++def setup_cma():+ ids_path = OUT / "cma_ids.json"+ if ids_path.exists():+ print("[cma] cma_ids.json exists, skipping", flush=True); return+ import anthropic+ c = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])+ env = c.beta.environments.create(name="ebench-env",+ config={"type": "cloud", "networking": {"type": "unrestricted"}})+ mcp_servers = [{"type": "url", "name": n, "url": u} for n, u in MCP.items()]+ tools = [{"type": "mcp_toolset", "mcp_server_name": n,+ "default_config": {"permission_policy": {"type": "always_allow"}}} for n in MCP]+ agent = c.beta.agents.create(name="ebench-cma", model=MODEL_CMA,+ system=SYSTEM, mcp_servers=mcp_servers, tools=tools)+ # If the MCP servers sit behind auth, attach a bearer vault. Otherwise leave it null.+ vault_id = None+ bearer = os.environ.get("MCP_BEARER")+ if bearer:+ try:+ vault = c.beta.vaults.create(display_name="ebench-mcp-bearer")+ for u in MCP.values():+ c.beta.vaults.credentials.create(vault_id=vault.id,+ auth={"type": "static_bearer", "mcp_server_url": u, "token": bearer})+ vault_id = vault.id+ print(f"[cma] vault {vault_id} with {len(MCP)} bearer creds", flush=True)+ except Exception as e:+ print(f"[cma] WARN vault creation failed ({type(e).__name__}: {e}); "+ f"MCP auth may fail — check the first trial", flush=True)+ OUT.mkdir(parents=True, exist_ok=True)+ ids_path.write_text(json.dumps({"env_id": env.id, "agent_id": agent.id, "vault_id": vault_id}, indent=2))+ print(f"[cma] agent {agent.id} / env {env.id} ready", flush=True)+++if __name__ == "__main__":+ OUT.mkdir(parents=True, exist_ok=True)+ harnesses = os.environ.get("HARNESSES", "tfy,cma,deepagents")+ if "cma" in harnesses:+ setup_cma()+ if "tfy" in harnesses:+ print("[tfy] no setup step here — in TrueForge, configure a provider for "+ "MODEL_TFY and register the MCP_CONFIG servers as Connectors under the "+ "same names; the agent spec is sent inline when each session is created.",+ flush=True)+ print("[setup] done", flush=True)diff --git a/docs/docs.json b/docs/docs.jsonindex 6431e7b25..a6eb335ed 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -33,7 +33,7 @@"groups": [{"group": "Getting Started",- "pages": ["introduction", "quickstart"]+ "pages": ["introduction", "quickstart", "evaluation"]},{"group": "Capabilities",diff --git a/docs/evaluation.mdx b/docs/evaluation.mdxnew file mode 100644index 000000000..2d10eb1a8--- /dev/null+++ b/docs/evaluation.mdx@@ -0,0 +1,45 @@+---+title: 'Evaluation'+sidebarTitle: 'Evaluation'+description: 'How TrueForge compares to other agent harnesses on the same tasks, model, and prompt — and how to reproduce the benchmark yourself.'+---++TrueForge is a vendor-neutral harness you run on your own infrastructure with any model. To check that the harness itself does not cost you accuracy — and to see where the tokens and dollars actually go — we run the same agent tasks through TrueForge and other harnesses under identical conditions and grade the answers blind.++## What we measured++We use the **L1-L2 tasks from [DevRev's Enterprise-Bench](https://x.com/devrev/status/2075815567458734468)** — 14 cross-system tasks that read like real B2B operations work across engineering, sales, and support. Each task makes the agent join data across three MCP servers: a CRM (Salesforce-style), a project tracker (Jira-style), and a document store (Drive-style).++Every harness gets the **same model, the same three MCP servers, and the same system prompt**, and runs each task in a fresh session. Answers are graded by an independent LLM judge that sees only the task's criteria and the answer — never which harness produced it — and a task passes only if it meets every required criterion.++## Results++`n = 3` trials per configuration. `Solved / 14` is the mean number of tasks passed per trial; cost is per run at provider list prices.++| Configuration | Solved / 14 | Cost / run | Tokens / run |+| -------------------------------- | :---------: | :--------: | :----------: |+| Claude Managed Agents · Opus 4.8 | 10.7 | $11.8 | 10.0M |+| TrueForge · Opus 4.8 | 10.7 | $8.6 | 3.7M |+| TrueForge · GLM-5.2 | 11.7 | $3.0 | 3.8M |++**Same model, fewer tokens.** On Opus 4.8, TrueForge and Claude Managed Agents solve the same number of tasks (10.7 / 14) — the harness costs you no accuracy — while TrueForge reaches those answers on far fewer tokens (3.7M vs 10.0M per run), so each run costs about 27% less.++**Open model, same quality, ~75% less.** Because TrueForge runs any model, you can swap Opus 4.8 for an open model like GLM-5.2 and keep quality in the same band (11.7 / 14) at $3.0 per run — roughly 75% below Claude Managed Agents on Opus.++## How it's measured++- **Same prompt, same tasks.** Every harness gets the identical system prompt and task prompts, and each answer is graded from the harness's reply.+- **Blind, strict judge.** The judge sees only the criteria and the answer, never which harness produced it; a task passes only if every required criterion is met (no partial credit).+- **List-price cost.** Computed per run from token counts at each provider's published prices, cache-aware.+- **Fresh sessions, multiple trials.** Each task runs in a new session with nothing carried over; we report the mean across trials.++## Reproduce it++The full harness — the runners for each arm, the exact system prompt, the blind grader, and the cost aggregator — is open in the TrueForge repo. Point it at your own dataset and models and you should reproduce the same shape of results.++<Card title="Benchmark kit on GitHub" icon="github" href="https://github.com/truefoundry/trueforge/tree/main/benchmark">+ A reproducible harness comparing TrueForge, Claude Managed Agents, and deepagents on the same tasks, model, and+ prompt — with setup, the system prompt, the judge, and the cost aggregator.+</Card>++The dataset itself comes from DevRev's open Enterprise-Bench release — the queries, judging criteria, and their [Harbor](https://github.com/harbor-framework/harbor) harness. We point at their release rather than redistributing it, so your run scores against the canonical rubric.