graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff vs Claude Code, OpenCode 2 and pi on a gateway model
justrachfb5ab10b139d
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 60969393..a3ecd5dc 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -15,8 +15,10 @@ grok-4.6 SuperGrok) isgraff against the Codex app server and pi on the same model and ChatGPT account(21 coding and MCP tasks, every run's data) is[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md),-and graff against OpenCode 2 and pi on one model through the Codegraff gateway is-[evals/graff-vs-opencode-and-pi/RESULTS.md](../evals/graff-vs-opencode-and-pi/RESULTS.md).+graff against Claude Code, OpenCode 2 and pi on one model through the Codegraff+gateway is [evals/graff-vs-claude-code/RESULTS.md](../evals/graff-vs-claude-code/RESULTS.md)+(an earlier run without Claude Code:+[evals/graff-vs-opencode-and-pi/RESULTS.md](../evals/graff-vs-opencode-and-pi/RESULTS.md)).The same 21 tasks against the Grok CLI on the same SuperGrok account are[evals/graff-vs-grok-cli/RESULTS.md](../evals/graff-vs-grok-cli/RESULTS.md),and what both mean for an engineering budget isdiff --git a/evals/graff-vs-claude-code/RESULTS.md b/evals/graff-vs-claude-code/RESULTS.mdnew file mode 100644index 00000000..6eeb8139--- /dev/null+++ b/evals/graff-vs-claude-code/RESULTS.md@@ -0,0 +1,106 @@+# graff vs Claude Code, OpenCode 2 and pi on a gateway model++Measured 2026-10-01. One model, served through the Codegraff gateway, ran the same 21 tasks in each harness, side by side, on one machine and one gateway key.++| Arm | What ran |+| --- | --- |+| graff | The ADR 0231 build (#1449) as a scripted `graff repl` (the `graff-gateway-repl` harness). |+| Claude Code | Claude Code 2.1.286: `claude -p --output-format stream-json --verbose --dangerously-skip-permissions`, with the task's `.mcp.json` as its only MCP config, through [`graff-evals/claude_code_run.py`](../../graff-evals/claude_code_run.py). The gateway takes chat completions only, so Claude Code reached it through a local LiteLLM 1.103.2 proxy that turns Anthropic Messages requests into chat completions. |+| OpenCode 2 | `@opencode/cli` 2.0.21: `opencode run --standalone --auto --format json`, through [`graff-evals/opencode2_run.py`](../../graff-evals/opencode2_run.py). |+| pi | pi 0.84.2 (the `pi-codegraff` harness). pi has no MCP client, so it ran the 11 coding tasks only. |++All four send the same model id to the same gateway with the same key, and each runs at its own defaults. Claude Code's background requests for a small model went to the same model.++## Results++All 21 tasks (pi: the 11 coding tasks), mean per task over 3 runs of each:++| | Pass | Wall per task | Input read from cache | Output tokens per task | Model calls per task |+| --- | ---: | ---: | ---: | ---: | ---: |+| graff (ADR 0231 build) | 63/63 | 8.7s | 95% | 573 | 3.5 |+| Claude Code | 63/63 | 13.2s | 60% | 926 | 3.5 |+| OpenCode 2 | 63/63 | 14.1s | 86% | 978 | 4.3 |+| pi | 33/33 | 6.0s | 76% | 262 | 2.8 |++By suite:++| Suite | graff (ADR 0231 build) | Claude Code | OpenCode 2 | pi |+| --- | ---: | ---: | ---: | ---: |+| core (11 coding tasks), wall | 7.0s | 6.0s | 6.8s | 6.0s |+| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 | 33/33 |+| mcp (10 MCP tasks), wall | 10.6s | 21.2s | 22.1s | — |+| mcp (10 MCP tasks), pass | 30/30 | 30/30 | 30/30 | — |++**Against Claude Code.** graff is 34% faster per task over all 21 tasks (8.7s against 13.2s). The gap is the MCP tasks, which graff finishes in half the time (10.6s against 21.2s). Claude Code keeps every MCP result whole in its context: 179k prompt tokens per MCP task against graff's 44k, about 40k per call against 10k, and it writes nearly twice the output (1,645 tokens against 882). graff slims large list results and often fetches, reduces and writes the report in one `rlm` script. On the coding tasks Claude Code is faster (6.0s against 7.0s); one 11.7-second gateway response on graff's one-call `exact-reply` accounts for about 0.3s of that.++**Against OpenCode 2 and pi.** graff takes 38% less time than OpenCode 2 over all tasks (8.7s against 14.1s) and about half on the MCP tasks (10.6s against 22.1s); OpenCode 2 is slightly faster on the coding tasks (6.8s against 7.0s). pi ties Claude Code as the fastest on the coding tasks (6.0s), with the smallest prompt.++## Per task++Wall time is the mean of 3 runs.++| Task | Suite | graff (ADR 0231 build) | Claude Code | OpenCode 2 | pi |+| --- | --- | ---: | ---: | ---: | ---: |+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 8.4s | 6.9s | 5.9s | 4.7s |+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 5.3s | 6.0s | 6.8s | 7.8s |+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 6.2s | 1.7s | 1.7s | 1.9s |+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 6.8s | 6.0s | 5.8s | 3.5s |+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 8.7s | 6.4s | 7.9s | 12.0s |+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 4.1s | 4.6s | 6.3s | 7.1s |+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 5.8s | 6.7s | 6.3s | 5.0s |+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 8.8s | 18.6s | 22.9s | — |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 11.1s | 20.1s | 18.7s | — |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 10.2s | 19.3s | 23.0s | — |+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 9.2s | 28.0s | 21.2s | — |+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 9.8s | 19.3s | 21.4s | — |+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 11.5s | 21.7s | 22.8s | — |+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 10.1s | 23.5s | 19.2s | — |+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 11.2s | 21.5s | 22.3s | — |+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 15.6s | 20.3s | 26.5s | — |+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 8.8s | 19.8s | 23.4s | — |+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 3.5s | 3.7s | 5.7s | 3.5s |+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 10.7s | 6.5s | 10.6s | 7.9s |+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 10.5s | 7.8s | 8.7s | 6.6s |+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 7.0s | 9.3s | 9.2s | 6.4s |++`results.json` has every run: pass, wall time, model calls, and input, cached input and output tokens. An earlier run of graff, OpenCode 2 and pi on the same model is [graff-vs-opencode-and-pi](../graff-vs-opencode-and-pi/RESULTS.md).++## Method++- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). Each task passes or fails on its own check.+- **Order.** 3 runs per task and arm. graff, Claude Code and OpenCode 2 were interleaved task by task (`run.py --interleave`), 6 runs at a time; pi ran its coding tasks afterwards. Each arm made one unscored request first.+- **Wall time.** The runner's clock around each process, start to exit.+- **Tokens.** Prompt tokens include cached ones in every arm. Claude Code's come from its stream's final result (session totals; Anthropic's `input_tokens` leaves out cache reads and writes, so the wrapper adds them back); its model calls are the distinct API responses. No list cost: the gateway's prices are not public list prices.++## Caveats++- **Claude Code ran through a translation proxy.** The proxy itself added no measurable time: a small request took 1.10s median through it against 1.61s sent straight to the gateway, because it keeps its connection to the gateway open. But Claude Code's own prompt-cache breakpoints do not survive the translation; the gateway applies its own caching, as it does for the other three. Only 49% of Claude Code's MCP input was read from cache (graff's: 96%), so against an endpoint that takes Anthropic Messages directly, Claude Code's MCP times would likely be better. The MCP gap here is an upper bound. Claude Code's token-count requests also fail through the proxy; it carried on without them.+- **pi runs only the coding tasks.** It has no MCP client.+- **One model.** Harness differences can look different on another model; the [Codex comparison](../graff-vs-codex-app-server/RESULTS.md) runs graff and pi on a different one.+- **Small sample.** 3 runs per task, one machine, one key. A second or two per task is within run-to-run noise.++## Reproduce++```sh+cd graff-evals+# CODEGRAFF_API_KEY set. Claude Code needs an Anthropic Messages endpoint; with LiteLLM in front+# of the gateway's chat completions:+# model_list:+# - model_name: "*"+# litellm_params: {model: "openai/<gateway model>", api_base: "https://gateway.codegraff.com/v1",+# api_key: "os.environ/CODEGRAFF_API_KEY"}+# litellm_settings: {use_chat_completions_url_for_anthropic_messages: true}+# litellm --config config.yaml --host 127.0.0.1 --port 4000+# then ANTHROPIC_BASE_URL=http://127.0.0.1:4000 and any ANTHROPIC_AUTH_TOKEN for the claude-code arm.+# OpenCode 2 and pi set up as in evals/graff-vs-opencode-and-pi. Keep --output-root outside any+# repository, with scripts/linear_fixture_mcp.py copied to <parent of output-root>/scripts/.+TASKS="--task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm"+python3 run.py --harness graff-gateway-repl,claude-code,opencode2 --model <gateway model> --suite core,mcp $TASKS \+ --reps 3 -j 6 --interleave --output-root /path/outside/repo/gateway+python3 run.py --harness pi-codegraff --model <gateway model> --suite core $TASKS \+ --reps 3 -j 6 --output-root /path/outside/repo/gateway-pi+```diff --git a/evals/graff-vs-claude-code/results.json b/evals/graff-vs-claude-code/results.jsonnew file mode 100644index 00000000..01b2d1d9--- /dev/null+++ b/evals/graff-vs-claude-code/results.json@@ -0,0 +1,3504 @@+{+ "model": "one gateway model (same id in every arm)",+ "tasks": 21,+ "reps": 3,+ "arms": {+ "graff": "graff (ADR 0231 build)",+ "claude-code": "Claude Code",+ "opencode2": "OpenCode 2",+ "pi": "pi"+ },+ "summary": {+ "all": {+ "graff": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 8.7,+ "cached_input_share": 0.953,+ "output_tokens_per_task": 573,+ "model_calls_per_task": 3.5+ },+ "claude-code": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 13.2,+ "cached_input_share": 0.599,+ "output_tokens_per_task": 926,+ "model_calls_per_task": 3.5+ },+ "opencode2": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 14.1,+ "cached_input_share": 0.86,+ "output_tokens_per_task": 978,+ "model_calls_per_task": 4.3+ },+ "pi": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 6.0,+ "cached_input_share": 0.765,+ "output_tokens_per_task": 262,+ "model_calls_per_task": 2.8+ }+ },+ "core": {+ "graff": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 7.0,+ "cached_input_share": 0.944,+ "output_tokens_per_task": 293,+ "model_calls_per_task": 2.8+ },+ "claude-code": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 6.0,+ "cached_input_share": 0.888,+ "output_tokens_per_task": 272,+ "model_calls_per_task": 2.6+ },+ "opencode2": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 6.8,+ "cached_input_share": 0.868,+ "output_tokens_per_task": 363,+ "model_calls_per_task": 3.2+ },+ "pi": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 6.0,+ "cached_input_share": 0.765,+ "output_tokens_per_task": 262,+ "model_calls_per_task": 2.8+ }+ },+ "mcp": {+ "graff": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 10.6,+ "cached_input_share": 0.96,+ "output_tokens_per_task": 882,+ "model_calls_per_task": 4.2+ },+ "claude-code": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 21.2,+ "cached_input_share": 0.493,+ "output_tokens_per_task": 1645,+ "model_calls_per_task": 4.5+ },+ "opencode2": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 22.1,+ "cached_input_share": 0.856,+ "output_tokens_per_task": 1654,+ "model_calls_per_task": 5.5+ }+ }+ },+ "per_task": {+ "csv-sum": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.4,+ "model_calls": 3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.9,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.9,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.7,+ "model_calls": 3+ }+ },+ "dead-code": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.3,+ "model_calls": 3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.0,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.8,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.8,+ "model_calls": 3+ }+ },+ "exact-reply": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.2,+ "model_calls": 1+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 1.7,+ "model_calls": 1+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 1.7,+ "model_calls": 1+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 1.9,+ "model_calls": 1+ }+ },+ "file-ops": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.8,+ "model_calls": 2+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.0,+ "model_calls": 2.7+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.8,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 3.5,+ "model_calls": 2+ }+ },+ "fix-fib": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.7,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.4,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.9,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 12.0,+ "model_calls": 4+ }+ },+ "git-ops": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.1,+ "model_calls": 2+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.6,+ "model_calls": 2+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.3,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.1,+ "model_calls": 2.3+ }+ },+ "json-transform": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.8,+ "model_calls": 3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.7,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.3,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.0,+ "model_calls": 3+ }+ },+ "linear-issues": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.8,+ "model_calls": 3.3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.6,+ "model_calls": 3.7+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.9,+ "model_calls": 5.7+ }+ },+ "linear-nohint": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 11.1,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.1,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.7,+ "model_calls": 5.7+ }+ },+ "linear-nohint-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.2,+ "model_calls": 3.3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.3,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 23.0,+ "model_calls": 6+ }+ },+ "linear-quiet": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.2,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 28.0,+ "model_calls": 5+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.2,+ "model_calls": 5.7+ }+ },+ "linear-quiet-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.8,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.3,+ "model_calls": 4.3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.4,+ "model_calls": 5.3+ }+ },+ "linear-reduce": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 11.5,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.7,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.8,+ "model_calls": 6+ }+ },+ "linear-reduce-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.1,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 23.5,+ "model_calls": 4.7+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.2,+ "model_calls": 5+ }+ },+ "linear-sidecar": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 11.2,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.5,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.3,+ "model_calls": 6+ }+ },+ "linear-split": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 15.6,+ "model_calls": 7.3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.3,+ "model_calls": 7+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 26.5,+ "model_calls": 3+ }+ },+ "linear-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.8,+ "model_calls": 4+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.8,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 23.4,+ "model_calls": 6.3+ }+ },+ "recall-noise": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 3.5,+ "model_calls": 2+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 3.7,+ "model_calls": 2+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.7,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 3.5,+ "model_calls": 2+ }+ },+ "refactor-rename": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.7,+ "model_calls": 5+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.5,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.6,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.9,+ "model_calls": 4+ }+ },+ "regex-count": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.5,+ "model_calls": 3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.8,+ "model_calls": 3.3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.7,+ "model_calls": 3.3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.6,+ "model_calls": 3.3+ }+ },+ "write-tests": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.0,+ "model_calls": 3+ },+ "claude-code": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.3,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.2,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.4,+ "model_calls": 3+ }+ }+ },+ "runs": [+ {+ "arm": "claude-code",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.73,+ "model_calls": 3,+ "input_tokens": 67670,+ "cached_input_tokens": 60999,+ "output_tokens": 230+ },+ {+ "arm": "claude-code",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.09,+ "model_calls": 3,+ "input_tokens": 67671,+ "cached_input_tokens": 60999,+ "output_tokens": 236+ },+ {+ "arm": "claude-code",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.02,+ "model_calls": 3,+ "input_tokens": 67663,+ "cached_input_tokens": 60999,+ "output_tokens": 227+ },+ {+ "arm": "claude-code",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.32,+ "model_calls": 3,+ "input_tokens": 67857,+ "cached_input_tokens": 60993,+ "output_tokens": 176+ },+ {+ "arm": "claude-code",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.27,+ "model_calls": 3,+ "input_tokens": 67851,+ "cached_input_tokens": 60993,+ "output_tokens": 170+ },+ {+ "arm": "claude-code",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.43,+ "model_calls": 3,+ "input_tokens": 67980,+ "cached_input_tokens": 60993,+ "output_tokens": 213+ },+ {+ "arm": "claude-code",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.79,+ "model_calls": 1,+ "input_tokens": 22408,+ "cached_input_tokens": 17105,+ "output_tokens": 4+ },+ {+ "arm": "claude-code",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.81,+ "model_calls": 1,+ "input_tokens": 22409,+ "cached_input_tokens": 17105,+ "output_tokens": 4+ },+ {+ "arm": "claude-code",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.62,+ "model_calls": 1,+ "input_tokens": 22408,+ "cached_input_tokens": 17105,+ "output_tokens": 4+ },+ {+ "arm": "claude-code",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.44,+ "model_calls": 3,+ "input_tokens": 67824,+ "cached_input_tokens": 60993,+ "output_tokens": 278+ },+ {+ "arm": "claude-code",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.18,+ "model_calls": 3,+ "input_tokens": 67818,+ "cached_input_tokens": 60993,+ "output_tokens": 277+ },+ {+ "arm": "claude-code",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.45,+ "model_calls": 2,+ "input_tokens": 45024,+ "cached_input_tokens": 39038,+ "output_tokens": 227+ },+ {+ "arm": "claude-code",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.29,+ "model_calls": 3,+ "input_tokens": 68239,+ "cached_input_tokens": 60999,+ "output_tokens": 371+ },+ {+ "arm": "claude-code",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.64,+ "model_calls": 3,+ "input_tokens": 68239,+ "cached_input_tokens": 60999,+ "output_tokens": 372+ },+ {+ "arm": "claude-code",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.35,+ "model_calls": 3,+ "input_tokens": 68233,+ "cached_input_tokens": 60999,+ "output_tokens": 371+ },+ {+ "arm": "claude-code",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.69,+ "model_calls": 2,+ "input_tokens": 45117,+ "cached_input_tokens": 39038,+ "output_tokens": 226+ },+ {+ "arm": "claude-code",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.95,+ "model_calls": 2,+ "input_tokens": 45098,+ "cached_input_tokens": 39038,+ "output_tokens": 276+ },+ {+ "arm": "claude-code",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.16,+ "model_calls": 2,+ "input_tokens": 45100,+ "cached_input_tokens": 39038,+ "output_tokens": 239+ },+ {+ "arm": "claude-code",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.31,+ "model_calls": 3,+ "input_tokens": 67944,+ "cached_input_tokens": 60993,+ "output_tokens": 339+ },+ {+ "arm": "claude-code",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.74,+ "model_calls": 3,+ "input_tokens": 67953,+ "cached_input_tokens": 60993,+ "output_tokens": 318+ },+ {+ "arm": "claude-code",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.94,+ "model_calls": 3,+ "input_tokens": 67944,+ "cached_input_tokens": 60993,+ "output_tokens": 325+ },+ {+ "arm": "claude-code",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.82,+ "model_calls": 4,+ "input_tokens": 169851,+ "cached_input_tokens": 83758,+ "output_tokens": 1556+ },+ {+ "arm": "claude-code",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.87,+ "model_calls": 3,+ "input_tokens": 144910,+ "cached_input_tokens": 61584,+ "output_tokens": 1366+ },+ {+ "arm": "claude-code",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.25,+ "model_calls": 4,+ "input_tokens": 170065,+ "cached_input_tokens": 83758,+ "output_tokens": 1678+ },+ {+ "arm": "claude-code",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.68,+ "model_calls": 4,+ "input_tokens": 169849,+ "cached_input_tokens": 83767,+ "output_tokens": 1674+ },+ {+ "arm": "claude-code",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.8,+ "model_calls": 4,+ "input_tokens": 169408,+ "cached_input_tokens": 83767,+ "output_tokens": 1382+ },+ {+ "arm": "claude-code",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.79,+ "model_calls": 4,+ "input_tokens": 205410,+ "cached_input_tokens": 83767,+ "output_tokens": 1417+ },+ {+ "arm": "claude-code",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.48,+ "model_calls": 4,+ "input_tokens": 169460,+ "cached_input_tokens": 83794,+ "output_tokens": 1503+ },+ {+ "arm": "claude-code",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.62,+ "model_calls": 4,+ "input_tokens": 206396,+ "cached_input_tokens": 83794,+ "output_tokens": 1824+ },+ {+ "arm": "claude-code",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.79,+ "model_calls": 4,+ "input_tokens": 205770,+ "cached_input_tokens": 83794,+ "output_tokens": 1473+ },+ {+ "arm": "claude-code",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.49,+ "model_calls": 4,+ "input_tokens": 169392,+ "cached_input_tokens": 83767,+ "output_tokens": 1526+ },+ {+ "arm": "claude-code",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.45,+ "model_calls": 6,+ "input_tokens": 293424,+ "cached_input_tokens": 128184,+ "output_tokens": 2361+ },+ {+ "arm": "claude-code",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.17,+ "model_calls": 5,+ "input_tokens": 231177,+ "cached_input_tokens": 105965,+ "output_tokens": 1863+ },+ {+ "arm": "claude-code",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.64,+ "model_calls": 5,+ "input_tokens": 231129,+ "cached_input_tokens": 106001,+ "output_tokens": 1933+ },+ {+ "arm": "claude-code",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.69,+ "model_calls": 4,+ "input_tokens": 169413,+ "cached_input_tokens": 83794,+ "output_tokens": 1402+ },+ {+ "arm": "claude-code",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.51,+ "model_calls": 4,+ "input_tokens": 169464,+ "cached_input_tokens": 83794,+ "output_tokens": 1492+ },+ {+ "arm": "claude-code",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.47,+ "model_calls": 4,+ "input_tokens": 170042,+ "cached_input_tokens": 83767,+ "output_tokens": 1456+ },+ {+ "arm": "claude-code",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.0,+ "model_calls": 4,+ "input_tokens": 170234,+ "cached_input_tokens": 83767,+ "output_tokens": 1748+ },+ {+ "arm": "claude-code",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.49,+ "model_calls": 4,+ "input_tokens": 169913,+ "cached_input_tokens": 83767,+ "output_tokens": 1555+ },+ {+ "arm": "claude-code",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.94,+ "model_calls": 5,+ "input_tokens": 232235,+ "cached_input_tokens": 106001,+ "output_tokens": 2258+ },+ {+ "arm": "claude-code",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.32,+ "model_calls": 5,+ "input_tokens": 231251,+ "cached_input_tokens": 106001,+ "output_tokens": 1915+ },+ {+ "arm": "claude-code",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.24,+ "model_calls": 4,+ "input_tokens": 170289,+ "cached_input_tokens": 83794,+ "output_tokens": 1792+ },+ {+ "arm": "claude-code",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.81,+ "model_calls": 4,+ "input_tokens": 170071,+ "cached_input_tokens": 83776,+ "output_tokens": 1629+ },+ {+ "arm": "claude-code",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.52,+ "model_calls": 4,+ "input_tokens": 169813,+ "cached_input_tokens": 83776,+ "output_tokens": 1443+ },+ {+ "arm": "claude-code",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.23,+ "model_calls": 4,+ "input_tokens": 170174,+ "cached_input_tokens": 83776,+ "output_tokens": 1679+ },+ {+ "arm": "claude-code",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.53,+ "model_calls": 6,+ "input_tokens": 100734,+ "cached_input_tokens": 83749,+ "output_tokens": 1528+ },+ {+ "arm": "claude-code",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.18,+ "model_calls": 9,+ "input_tokens": 131230,+ "cached_input_tokens": 107157,+ "output_tokens": 1627+ },+ {+ "arm": "claude-code",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.33,+ "model_calls": 6,+ "input_tokens": 100546,+ "cached_input_tokens": 83749,+ "output_tokens": 1494+ },+ {+ "arm": "claude-code",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.95,+ "model_calls": 4,+ "input_tokens": 169963,+ "cached_input_tokens": 83758,+ "output_tokens": 1534+ },+ {+ "arm": "claude-code",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.16,+ "model_calls": 4,+ "input_tokens": 169806,+ "cached_input_tokens": 83758,+ "output_tokens": 1394+ },+ {+ "arm": "claude-code",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.18,+ "model_calls": 4,+ "input_tokens": 170101,+ "cached_input_tokens": 83758,+ "output_tokens": 1847+ },+ {+ "arm": "claude-code",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.76,+ "model_calls": 2,+ "input_tokens": 45330,+ "cached_input_tokens": 39041,+ "output_tokens": 104+ },+ {+ "arm": "claude-code",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.15,+ "model_calls": 2,+ "input_tokens": 45330,+ "cached_input_tokens": 39041,+ "output_tokens": 104+ },+ {+ "arm": "claude-code",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.25,+ "model_calls": 2,+ "input_tokens": 45328,+ "cached_input_tokens": 39041,+ "output_tokens": 104+ },+ {+ "arm": "claude-code",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.14,+ "model_calls": 3,+ "input_tokens": 67833,+ "cached_input_tokens": 61005,+ "output_tokens": 254+ },+ {+ "arm": "claude-code",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.98,+ "model_calls": 3,+ "input_tokens": 67827,+ "cached_input_tokens": 61005,+ "output_tokens": 248+ },+ {+ "arm": "claude-code",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.3,+ "model_calls": 3,+ "input_tokens": 67827,+ "cached_input_tokens": 61005,+ "output_tokens": 251+ },+ {+ "arm": "claude-code",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.11,+ "model_calls": 3,+ "input_tokens": 67999,+ "cached_input_tokens": 60993,+ "output_tokens": 399+ },+ {+ "arm": "claude-code",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.21,+ "model_calls": 4,+ "input_tokens": 90726,+ "cached_input_tokens": 82969,+ "output_tokens": 396+ },+ {+ "arm": "claude-code",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.19,+ "model_calls": 3,+ "input_tokens": 67999,+ "cached_input_tokens": 60993,+ "output_tokens": 358+ },+ {+ "arm": "claude-code",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.28,+ "model_calls": 3,+ "input_tokens": 68157,+ "cached_input_tokens": 60993,+ "output_tokens": 636+ },+ {+ "arm": "claude-code",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.78,+ "model_calls": 3,+ "input_tokens": 68166,+ "cached_input_tokens": 60993,+ "output_tokens": 618+ },+ {+ "arm": "claude-code",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.88,+ "model_calls": 3,+ "input_tokens": 68160,+ "cached_input_tokens": 60993,+ "output_tokens": 637+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.05,+ "model_calls": 3,+ "input_tokens": 27862,+ "cached_input_tokens": 22546,+ "output_tokens": 393+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.82,+ "model_calls": 3,+ "input_tokens": 27839,+ "cached_input_tokens": 27756,+ "output_tokens": 303+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.27,+ "model_calls": 3,+ "input_tokens": 27927,+ "cached_input_tokens": 27756,+ "output_tokens": 341+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.31,+ "model_calls": 3,+ "input_tokens": 28218,+ "cached_input_tokens": 22632,+ "output_tokens": 162+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.86,+ "model_calls": 3,+ "input_tokens": 28228,+ "cached_input_tokens": 27748,+ "output_tokens": 165+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.6,+ "model_calls": 3,+ "input_tokens": 28218,+ "cached_input_tokens": 28210,+ "output_tokens": 160+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.27,+ "model_calls": 1,+ "input_tokens": 9121,+ "cached_input_tokens": 9117,+ "output_tokens": 51+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.74,+ "model_calls": 1,+ "input_tokens": 9121,+ "cached_input_tokens": 9117,+ "output_tokens": 51+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.73,+ "model_calls": 1,+ "input_tokens": 9121,+ "cached_input_tokens": 9117,+ "output_tokens": 51+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.5,+ "model_calls": 2,+ "input_tokens": 18395,+ "cached_input_tokens": 18186,+ "output_tokens": 162+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.2,+ "model_calls": 2,+ "input_tokens": 18395,+ "cached_input_tokens": 18389,+ "output_tokens": 145+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 13.55,+ "model_calls": 2,+ "input_tokens": 18395,+ "cached_input_tokens": 18389,+ "output_tokens": 148+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.33,+ "model_calls": 4,+ "input_tokens": 38157,+ "cached_input_tokens": 32356,+ "output_tokens": 386+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.79,+ "model_calls": 4,+ "input_tokens": 38157,+ "cached_input_tokens": 37477,+ "output_tokens": 388+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.91,+ "model_calls": 4,+ "input_tokens": 38157,+ "cached_input_tokens": 37477,+ "output_tokens": 388+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.47,+ "model_calls": 2,+ "input_tokens": 18597,+ "cached_input_tokens": 13279,+ "output_tokens": 311+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.13,+ "model_calls": 2,+ "input_tokens": 18594,+ "cached_input_tokens": 18398,+ "output_tokens": 266+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.71,+ "model_calls": 2,+ "input_tokens": 18590,+ "cached_input_tokens": 18398,+ "output_tokens": 240+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.04,+ "model_calls": 3,+ "input_tokens": 27943,+ "cached_input_tokens": 27935,+ "output_tokens": 240+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.62,+ "model_calls": 3,+ "input_tokens": 27943,+ "cached_input_tokens": 27935,+ "output_tokens": 273+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.63,+ "model_calls": 3,+ "input_tokens": 27943,+ "cached_input_tokens": 27935,+ "output_tokens": 252+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.63,+ "model_calls": 3,+ "input_tokens": 30825,+ "cached_input_tokens": 29264,+ "output_tokens": 500+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.04,+ "model_calls": 4,+ "input_tokens": 41510,+ "cached_input_tokens": 40531,+ "output_tokens": 723+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.73,+ "model_calls": 3,+ "input_tokens": 30744,+ "cached_input_tokens": 30536,+ "output_tokens": 637+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.56,+ "model_calls": 4,+ "input_tokens": 41221,+ "cached_input_tokens": 39503,+ "output_tokens": 856+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.37,+ "model_calls": 4,+ "input_tokens": 41244,+ "cached_input_tokens": 40545,+ "output_tokens": 868+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.3,+ "model_calls": 4,+ "input_tokens": 41221,+ "cached_input_tokens": 40925,+ "output_tokens": 846+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.75,+ "model_calls": 4,+ "input_tokens": 42475,+ "cached_input_tokens": 40513,+ "output_tokens": 798+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.19,+ "model_calls": 3,+ "input_tokens": 32072,+ "cached_input_tokens": 30981,+ "output_tokens": 959+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.52,+ "model_calls": 3,+ "input_tokens": 31797,+ "cached_input_tokens": 31503,+ "output_tokens": 683+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.71,+ "model_calls": 4,+ "input_tokens": 41002,+ "cached_input_tokens": 40808,+ "output_tokens": 613+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.96,+ "model_calls": 4,+ "input_tokens": 41013,+ "cached_input_tokens": 40808,+ "output_tokens": 587+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.84,+ "model_calls": 4,+ "input_tokens": 41002,+ "cached_input_tokens": 40299,+ "output_tokens": 514+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.61,+ "model_calls": 4,+ "input_tokens": 43108,+ "cached_input_tokens": 40900,+ "output_tokens": 744+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.05,+ "model_calls": 4,+ "input_tokens": 43108,+ "cached_input_tokens": 43098,+ "output_tokens": 683+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.86,+ "model_calls": 4,+ "input_tokens": 43108,+ "cached_input_tokens": 43098,+ "output_tokens": 645+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.76,+ "model_calls": 4,+ "input_tokens": 41266,+ "cached_input_tokens": 39561,+ "output_tokens": 765+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.79,+ "model_calls": 4,+ "input_tokens": 41308,+ "cached_input_tokens": 40370,+ "output_tokens": 869+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.84,+ "model_calls": 4,+ "input_tokens": 41229,+ "cached_input_tokens": 40477,+ "output_tokens": 701+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.32,+ "model_calls": 4,+ "input_tokens": 42800,+ "cached_input_tokens": 40737,+ "output_tokens": 729+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.21,+ "model_calls": 4,+ "input_tokens": 42819,+ "cached_input_tokens": 41903,+ "output_tokens": 751+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.7,+ "model_calls": 4,+ "input_tokens": 42818,+ "cached_input_tokens": 42613,+ "output_tokens": 588+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.15,+ "model_calls": 5,+ "input_tokens": 63150,+ "cached_input_tokens": 58966,+ "output_tokens": 1302+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.74,+ "model_calls": 4,+ "input_tokens": 49882,+ "cached_input_tokens": 48906,+ "output_tokens": 1173+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.66,+ "model_calls": 3,+ "input_tokens": 37282,+ "cached_input_tokens": 36266,+ "output_tokens": 833+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.27,+ "model_calls": 7,+ "input_tokens": 63761,+ "cached_input_tokens": 46131,+ "output_tokens": 1874+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.49,+ "model_calls": 7,+ "input_tokens": 63887,+ "cached_input_tokens": 60554,+ "output_tokens": 2014+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.96,+ "model_calls": 8,+ "input_tokens": 77784,+ "cached_input_tokens": 73972,+ "output_tokens": 2184+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.36,+ "model_calls": 4,+ "input_tokens": 43475,+ "cached_input_tokens": 41215,+ "output_tokens": 741+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.46,+ "model_calls": 4,+ "input_tokens": 43085,+ "cached_input_tokens": 42111,+ "output_tokens": 590+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.55,+ "model_calls": 4,+ "input_tokens": 43098,+ "cached_input_tokens": 42654,+ "output_tokens": 684+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.74,+ "model_calls": 2,+ "input_tokens": 18747,+ "cached_input_tokens": 13253,+ "output_tokens": 110+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.27,+ "model_calls": 2,+ "input_tokens": 18747,+ "cached_input_tokens": 18741,+ "output_tokens": 110+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.4,+ "model_calls": 2,+ "input_tokens": 18747,+ "cached_input_tokens": 18741,+ "output_tokens": 110+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.42,+ "model_calls": 5,+ "input_tokens": 48198,+ "cached_input_tokens": 42190,+ "output_tokens": 665+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.7,+ "model_calls": 5,+ "input_tokens": 48174,+ "cached_input_tokens": 47635,+ "output_tokens": 614+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.1,+ "model_calls": 5,+ "input_tokens": 48194,+ "cached_input_tokens": 47316,+ "output_tokens": 656+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.27,+ "model_calls": 3,+ "input_tokens": 28160,+ "cached_input_tokens": 22709,+ "output_tokens": 423+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.78,+ "model_calls": 3,+ "input_tokens": 28085,+ "cached_input_tokens": 27811,+ "output_tokens": 327+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.54,+ "model_calls": 3,+ "input_tokens": 28146,+ "cached_input_tokens": 27829,+ "output_tokens": 435+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.09,+ "model_calls": 3,+ "input_tokens": 28080,+ "cached_input_tokens": 22556,+ "output_tokens": 440+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.75,+ "model_calls": 3,+ "input_tokens": 28080,+ "cached_input_tokens": 28072,+ "output_tokens": 482+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.03,+ "model_calls": 3,+ "input_tokens": 28080,+ "cached_input_tokens": 28072,+ "output_tokens": 415+ },+ {+ "arm": "opencode2",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.36,+ "model_calls": 3,+ "input_tokens": 23865,+ "cached_input_tokens": 20630,+ "output_tokens": 282+ },+ {+ "arm": "opencode2",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.79,+ "model_calls": 3,+ "input_tokens": 23865,+ "cached_input_tokens": 20671,+ "output_tokens": 321+ },+ {+ "arm": "opencode2",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.48,+ "model_calls": 3,+ "input_tokens": 23865,+ "cached_input_tokens": 20671,+ "output_tokens": 324+ },+ {+ "arm": "opencode2",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.29,+ "model_calls": 4,+ "input_tokens": 32305,+ "cached_input_tokens": 28619,+ "output_tokens": 289+ },+ {+ "arm": "opencode2",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.09,+ "model_calls": 4,+ "input_tokens": 32305,+ "cached_input_tokens": 28671,+ "output_tokens": 289+ },+ {+ "arm": "opencode2",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.03,+ "model_calls": 4,+ "input_tokens": 32301,+ "cached_input_tokens": 28669,+ "output_tokens": 287+ },+ {+ "arm": "opencode2",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.87,+ "model_calls": 1,+ "input_tokens": 7484,+ "cached_input_tokens": 4644,+ "output_tokens": 4+ },+ {+ "arm": "opencode2",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.57,+ "model_calls": 1,+ "input_tokens": 7484,+ "cached_input_tokens": 4644,+ "output_tokens": 4+ },+ {+ "arm": "opencode2",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.79,+ "model_calls": 1,+ "input_tokens": 7484,+ "cached_input_tokens": 4644,+ "output_tokens": 4+ },+ {+ "arm": "opencode2",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.24,+ "model_calls": 3,+ "input_tokens": 23744,+ "cached_input_tokens": 20568,+ "output_tokens": 286+ },+ {+ "arm": "opencode2",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.35,+ "model_calls": 3,+ "input_tokens": 23744,+ "cached_input_tokens": 20614,+ "output_tokens": 276+ },+ {+ "arm": "opencode2",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.7,+ "model_calls": 3,+ "input_tokens": 23744,+ "cached_input_tokens": 20614,+ "output_tokens": 311+ },+ {+ "arm": "opencode2",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.52,+ "model_calls": 4,+ "input_tokens": 32294,+ "cached_input_tokens": 28798,+ "output_tokens": 355+ },+ {+ "arm": "opencode2",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.35,+ "model_calls": 4,+ "input_tokens": 32294,+ "cached_input_tokens": 28857,+ "output_tokens": 344+ },+ {+ "arm": "opencode2",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.76,+ "model_calls": 4,+ "input_tokens": 32294,+ "cached_input_tokens": 28857,+ "output_tokens": 358+ },+ {+ "arm": "opencode2",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.21,+ "model_calls": 3,+ "input_tokens": 24080,+ "cached_input_tokens": 20623,+ "output_tokens": 357+ },+ {+ "arm": "opencode2",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.95,+ "model_calls": 3,+ "input_tokens": 24075,+ "cached_input_tokens": 20691,+ "output_tokens": 340+ },+ {+ "arm": "opencode2",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.76,+ "model_calls": 3,+ "input_tokens": 24084,+ "cached_input_tokens": 20691,+ "output_tokens": 332+ },+ {+ "arm": "opencode2",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.34,+ "model_calls": 3,+ "input_tokens": 23966,+ "cached_input_tokens": 20665,+ "output_tokens": 383+ },+ {+ "arm": "opencode2",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.81,+ "model_calls": 3,+ "input_tokens": 23966,+ "cached_input_tokens": 20719,+ "output_tokens": 344+ },+ {+ "arm": "opencode2",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.66,+ "model_calls": 3,+ "input_tokens": 23966,+ "cached_input_tokens": 20719,+ "output_tokens": 384+ },+ {+ "arm": "opencode2",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.4,+ "model_calls": 6,+ "input_tokens": 65814,+ "cached_input_tokens": 57075,+ "output_tokens": 1818+ },+ {+ "arm": "opencode2",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.99,+ "model_calls": 6,+ "input_tokens": 65767,+ "cached_input_tokens": 57326,+ "output_tokens": 1745+ },+ {+ "arm": "opencode2",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.43,+ "model_calls": 5,+ "input_tokens": 53853,+ "cached_input_tokens": 45358,+ "output_tokens": 1783+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.09,+ "model_calls": 5,+ "input_tokens": 57856,+ "cached_input_tokens": 46884,+ "output_tokens": 1634+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.06,+ "model_calls": 6,+ "input_tokens": 65561,+ "cached_input_tokens": 56903,+ "output_tokens": 1742+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.06,+ "model_calls": 6,+ "input_tokens": 68449,+ "cached_input_tokens": 58710,+ "output_tokens": 1726+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.95,+ "model_calls": 6,+ "input_tokens": 66382,+ "cached_input_tokens": 57487,+ "output_tokens": 1624+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.19,+ "model_calls": 6,+ "input_tokens": 65088,+ "cached_input_tokens": 56446,+ "output_tokens": 1658+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.98,+ "model_calls": 6,+ "input_tokens": 66345,+ "cached_input_tokens": 57604,+ "output_tokens": 1644+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.61,+ "model_calls": 5,+ "input_tokens": 45917,+ "cached_input_tokens": 40113,+ "output_tokens": 1443+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.39,+ "model_calls": 6,+ "input_tokens": 58273,+ "cached_input_tokens": 51693,+ "output_tokens": 1867+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.64,+ "model_calls": 6,+ "input_tokens": 57350,+ "cached_input_tokens": 50964,+ "output_tokens": 1849+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.65,+ "model_calls": 5,+ "input_tokens": 47358,+ "cached_input_tokens": 41341,+ "output_tokens": 1761+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.65,+ "model_calls": 5,+ "input_tokens": 47471,+ "cached_input_tokens": 41447,+ "output_tokens": 1974+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.93,+ "model_calls": 6,+ "input_tokens": 58086,+ "cached_input_tokens": 51508,+ "output_tokens": 1783+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.34,+ "model_calls": 6,+ "input_tokens": 56123,+ "cached_input_tokens": 49789,+ "output_tokens": 1798+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.47,+ "model_calls": 6,+ "input_tokens": 59512,+ "cached_input_tokens": 52199,+ "output_tokens": 1901+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.54,+ "model_calls": 6,+ "input_tokens": 57388,+ "cached_input_tokens": 50882,+ "output_tokens": 1990+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.16,+ "model_calls": 6,+ "input_tokens": 57297,+ "cached_input_tokens": 51036,+ "output_tokens": 1853+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.99,+ "model_calls": 4,+ "input_tokens": 36639,+ "cached_input_tokens": 31328,+ "output_tokens": 1681+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.47,+ "model_calls": 5,+ "input_tokens": 47319,+ "cached_input_tokens": 41186,+ "output_tokens": 1897+ },+ {+ "arm": "opencode2",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.03,+ "model_calls": 6,+ "input_tokens": 67258,+ "cached_input_tokens": 57985,+ "output_tokens": 1889+ },+ {+ "arm": "opencode2",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.89,+ "model_calls": 6,+ "input_tokens": 67495,+ "cached_input_tokens": 58046,+ "output_tokens": 1789+ },+ {+ "arm": "opencode2",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.86,+ "model_calls": 6,+ "input_tokens": 66217,+ "cached_input_tokens": 57489,+ "output_tokens": 1838+ },+ {+ "arm": "opencode2",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.72,+ "model_calls": 3,+ "input_tokens": 21547,+ "cached_input_tokens": 13793,+ "output_tokens": 594+ },+ {+ "arm": "opencode2",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.44,+ "model_calls": 3,+ "input_tokens": 20988,+ "cached_input_tokens": 16802,+ "output_tokens": 406+ },+ {+ "arm": "opencode2",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.32,+ "model_calls": 3,+ "input_tokens": 32210,+ "cached_input_tokens": 22190,+ "output_tokens": 666+ },+ {+ "arm": "opencode2",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.66,+ "model_calls": 6,+ "input_tokens": 69118,+ "cached_input_tokens": 59552,+ "output_tokens": 1774+ },+ {+ "arm": "opencode2",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.66,+ "model_calls": 6,+ "input_tokens": 66240,+ "cached_input_tokens": 57633,+ "output_tokens": 1691+ },+ {+ "arm": "opencode2",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.87,+ "model_calls": 7,+ "input_tokens": 271673,+ "cached_input_tokens": 224144,+ "output_tokens": 1799+ },+ {+ "arm": "opencode2",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.36,+ "model_calls": 3,+ "input_tokens": 24029,+ "cached_input_tokens": 20531,+ "output_tokens": 167+ },+ {+ "arm": "opencode2",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.66,+ "model_calls": 3,+ "input_tokens": 24029,+ "cached_input_tokens": 20567,+ "output_tokens": 167+ },+ {+ "arm": "opencode2",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.97,+ "model_calls": 3,+ "input_tokens": 24029,+ "cached_input_tokens": 20567,+ "output_tokens": 167+ },+ {+ "arm": "opencode2",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.86,+ "model_calls": 4,+ "input_tokens": 32910,+ "cached_input_tokens": 29006,+ "output_tokens": 692+ },+ {+ "arm": "opencode2",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.67,+ "model_calls": 4,+ "input_tokens": 32878,+ "cached_input_tokens": 29060,+ "output_tokens": 679+ },+ {+ "arm": "opencode2",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.22,+ "model_calls": 4,+ "input_tokens": 32966,+ "cached_input_tokens": 29104,+ "output_tokens": 716+ },+ {+ "arm": "opencode2",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.44,+ "model_calls": 3,+ "input_tokens": 24257,+ "cached_input_tokens": 20810,+ "output_tokens": 519+ },+ {+ "arm": "opencode2",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.48,+ "model_calls": 4,+ "input_tokens": 32509,+ "cached_input_tokens": 28973,+ "output_tokens": 539+ },+ {+ "arm": "opencode2",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.06,+ "model_calls": 3,+ "input_tokens": 24324,+ "cached_input_tokens": 20935,+ "output_tokens": 454+ },+ {+ "arm": "opencode2",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.76,+ "model_calls": 4,+ "input_tokens": 32026,+ "cached_input_tokens": 28378,+ "output_tokens": 710+ },+ {+ "arm": "opencode2",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.63,+ "model_calls": 4,+ "input_tokens": 31983,+ "cached_input_tokens": 28440,+ "output_tokens": 628+ },+ {+ "arm": "opencode2",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.35,+ "model_calls": 4,+ "input_tokens": 32019,+ "cached_input_tokens": 28438,+ "output_tokens": 671+ },+ {+ "arm": "pi",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.57,+ "model_calls": 3,+ "input_tokens": 7198,+ "cached_input_tokens": 5896,+ "output_tokens": 224+ },+ {+ "arm": "pi",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.53,+ "model_calls": 3,+ "input_tokens": 7223,+ "cached_input_tokens": 5896,+ "output_tokens": 230+ },+ {+ "arm": "pi",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.99,+ "model_calls": 3,+ "input_tokens": 7223,+ "cached_input_tokens": 5896,+ "output_tokens": 204+ },+ {+ "arm": "pi",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.56,+ "model_calls": 3,+ "input_tokens": 7558,+ "cached_input_tokens": 5984,+ "output_tokens": 231+ },+ {+ "arm": "pi",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.37,+ "model_calls": 3,+ "input_tokens": 7536,+ "cached_input_tokens": 5973,+ "output_tokens": 220+ },+ {+ "arm": "pi",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.41,+ "model_calls": 3,+ "input_tokens": 7572,+ "cached_input_tokens": 5991,+ "output_tokens": 238+ },+ {+ "arm": "pi",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 2.13,+ "model_calls": 1,+ "input_tokens": 2271,+ "cached_input_tokens": 0,+ "output_tokens": 4+ },+ {+ "arm": "pi",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.73,+ "model_calls": 1,+ "input_tokens": 2271,+ "cached_input_tokens": 0,+ "output_tokens": 4+ },+ {+ "arm": "pi",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.73,+ "model_calls": 1,+ "input_tokens": 2271,+ "cached_input_tokens": 0,+ "output_tokens": 4+ },+ {+ "arm": "pi",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.34,+ "model_calls": 2,+ "input_tokens": 4746,+ "cached_input_tokens": 3489,+ "output_tokens": 168+ },+ {+ "arm": "pi",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.71,+ "model_calls": 2,+ "input_tokens": 4746,+ "cached_input_tokens": 3489,+ "output_tokens": 164+ },+ {+ "arm": "pi",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.37,+ "model_calls": 2,+ "input_tokens": 4746,+ "cached_input_tokens": 3489,+ "output_tokens": 168+ },+ {+ "arm": "pi",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.83,+ "model_calls": 4,+ "input_tokens": 10457,+ "cached_input_tokens": 8722,+ "output_tokens": 432+ },+ {+ "arm": "pi",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.32,+ "model_calls": 4,+ "input_tokens": 10449,+ "cached_input_tokens": 7532,+ "output_tokens": 417+ },+ {+ "arm": "pi",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.91,+ "model_calls": 4,+ "input_tokens": 10434,+ "cached_input_tokens": 7532,+ "output_tokens": 464+ },+ {+ "arm": "pi",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.9,+ "model_calls": 2,+ "input_tokens": 5016,+ "cached_input_tokens": 3507,+ "output_tokens": 286+ },+ {+ "arm": "pi",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.01,+ "model_calls": 3,+ "input_tokens": 7451,+ "cached_input_tokens": 5897,+ "output_tokens": 330+ },+ {+ "arm": "pi",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.27,+ "model_calls": 2,+ "input_tokens": 5009,+ "cached_input_tokens": 3507,+ "output_tokens": 274+ },+ {+ "arm": "pi",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.74,+ "model_calls": 3,+ "input_tokens": 7341,+ "cached_input_tokens": 5927,+ "output_tokens": 281+ },+ {+ "arm": "pi",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.47,+ "model_calls": 3,+ "input_tokens": 7298,+ "cached_input_tokens": 5927,+ "output_tokens": 238+ },+ {+ "arm": "pi",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.7,+ "model_calls": 3,+ "input_tokens": 7357,+ "cached_input_tokens": 5934,+ "output_tokens": 240+ },+ {+ "arm": "pi",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.6,+ "model_calls": 2,+ "input_tokens": 4979,+ "cached_input_tokens": 3480,+ "output_tokens": 61+ },+ {+ "arm": "pi",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.68,+ "model_calls": 2,+ "input_tokens": 4979,+ "cached_input_tokens": 3480,+ "output_tokens": 61+ },+ {+ "arm": "pi",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.19,+ "model_calls": 2,+ "input_tokens": 4979,+ "cached_input_tokens": 3480,+ "output_tokens": 61+ },+ {+ "arm": "pi",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.65,+ "model_calls": 4,+ "input_tokens": 10136,+ "cached_input_tokens": 8537,+ "output_tokens": 538+ },+ {+ "arm": "pi",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.25,+ "model_calls": 4,+ "input_tokens": 10226,+ "cached_input_tokens": 8601,+ "output_tokens": 384+ },+ {+ "arm": "pi",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.75,+ "model_calls": 4,+ "input_tokens": 10230,+ "cached_input_tokens": 8579,+ "output_tokens": 541+ },+ {+ "arm": "pi",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.21,+ "model_calls": 3,+ "input_tokens": 7399,+ "cached_input_tokens": 6012,+ "output_tokens": 257+ },+ {+ "arm": "pi",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.97,+ "model_calls": 4,+ "input_tokens": 10428,+ "cached_input_tokens": 8756,+ "output_tokens": 518+ },+ {+ "arm": "pi",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.51,+ "model_calls": 3,+ "input_tokens": 7389,+ "cached_input_tokens": 5993,+ "output_tokens": 316+ },+ {+ "arm": "pi",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.82,+ "model_calls": 3,+ "input_tokens": 7324,+ "cached_input_tokens": 5903,+ "output_tokens": 386+ },+ {+ "arm": "pi",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.09,+ "model_calls": 3,+ "input_tokens": 7324,+ "cached_input_tokens": 5903,+ "output_tokens": 357+ },+ {+ "arm": "pi",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.22,+ "model_calls": 3,+ "input_tokens": 7324,+ "cached_input_tokens": 5903,+ "output_tokens": 350+ }+ ]+}\ No newline at end of filediff --git a/graff-evals/README.md b/graff-evals/README.mdindex e1af4f1c..19e0d2a4 100644--- a/graff-evals/README.md+++ b/graff-evals/README.md@@ -56,9 +56,10 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co# Results: evals/graff-vs-codex-app-server/RESULTS.md./run.py --suite mcp --harness graff-repl,codex-app-server -j 8 --interleave./run.py --suite core --harness graff-repl,codex-app-server,pi-codex -j 8 --interleave-# one gateway model in graff, OpenCode 2 and pi (set CODEGRAFF_API_KEY; OPENCODE2_BIN-# when `opencode` on PATH is 1.x). Results: evals/graff-vs-opencode-and-pi/RESULTS.md-./run.py --suite core,mcp --harness graff-gateway-repl,opencode2 --model <gateway model> -j 6 --interleave+# one gateway model in graff, Claude Code, OpenCode 2 and pi (set CODEGRAFF_API_KEY; OPENCODE2_BIN+# when `opencode` on PATH is 1.x; Claude Code needs an Anthropic-format proxy in front of the+# gateway, see the write-up). Results: evals/graff-vs-claude-code/RESULTS.md+./run.py --suite core,mcp --harness graff-gateway-repl,claude-code,opencode2 --model <gateway model> -j 6 --interleave./run.py --suite core --harness pi-codegraff --model <gateway model> -j 6```diff --git a/graff-evals/claude_code_run.py b/graff-evals/claude_code_run.pynew file mode 100644index 00000000..eef9b14c--- /dev/null+++ b/graff-evals/claude_code_run.py@@ -0,0 +1,58 @@+#!/usr/bin/env python3+"""Run Claude Code headless on one eval task and report like graff does.++usage: claude_code_run.py MODEL PROMPT (cwd = the task sandbox)++`claude -p --output-format stream-json --verbose`, permissions skipped, with+the task's `.mcp.json` as its only MCP config. The final result text goes to+stdout; a graff-format `[usage]` line goes to stderr. Calls are the API+responses (one message id each); tokens are the result event's session totals.+Anthropic's `input_tokens` leaves out cache reads and writes, so they are+added back to make the total prompt tokens. stdin is empty. Point Claude Code at an endpoint with the+usual environment (ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN).+"""+import json, os, subprocess, sys++CLAUDE = os.environ.get("CLAUDE_CODE_BIN", "claude")+model, prompt = sys.argv[1], sys.argv[2]++cmd = [CLAUDE, "-p", prompt, "--model", model, "--dangerously-skip-permissions",+ "--output-format", "stream-json", "--verbose"]+if os.path.exists(".mcp.json"):+ cmd += ["--mcp-config", ".mcp.json", "--strict-mcp-config"]++r = subprocess.run(cmd, stdin=subprocess.DEVNULL, capture_output=True, text=True)+per_message, answer, result = {}, "", None+for line in r.stdout.splitlines():+ try:+ ev = json.loads(line)+ except ValueError:+ continue+ if ev.get("type") == "assistant":+ msg = ev.get("message") or {}+ u = msg.get("usage") or {}+ key = msg.get("id") or f"anon-{len(per_message)}"+ prev = per_message.get(key, {})+ # Content blocks of one response arrive as separate events with the same id.+ per_message[key] = {k: max(int(u.get(k) or 0), int(prev.get(k) or 0))+ for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens", "output_tokens")}+ elif ev.get("type") == "result":+ result = ev+ answer = ev.get("result") or answer+print(answer)+sys.stderr.write(r.stderr)+if result is not None and result.get("is_error"):+ sys.stderr.write(f"claude_code_run: result is_error ({result.get('subtype')})\n")+calls = len(per_message) or int((result or {}).get("num_turns") or 0)+# The result event carries the session's totals; per-message usage can be empty+# when a gateway streams it only at the end.+totals = (result or {}).get("usage") or {}+if totals:+ tin, tread, twrite, tout = (int(totals.get(k) or 0) for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens", "output_tokens"))+else:+ tin = sum(m["input_tokens"] for m in per_message.values())+ tread = sum(m["cache_read_input_tokens"] for m in per_message.values())+ twrite = sum(m["cache_creation_input_tokens"] for m in per_message.values())+ tout = sum(m["output_tokens"] for m in per_message.values())+sys.stderr.write(f"\n[usage] {calls} api call(s) · {tin + tread + twrite} in ({tread} cached, {twrite} cache writes) + {tout} out tokens\n")+sys.exit(r.returncode)diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.jsonindex 803e44a7..96da5052 100644--- a/graff-evals/harnesses.json+++ b/graff-evals/harnesses.json@@ -367,5 +367,12 @@"usage": "graff-stderr","model_prefix": "codegraff","note": "OpenCode 2 (@opencode/cli 2.x; OPENCODE2_BIN when `opencode` on PATH is 1.x) with a codegraff provider in its config; pass --model. The wrapper maps .mcp.json into opencode.json and reads usage from OpenCode's session database."+ },+ "claude-code": {+ "cmd": ["python3", "{repo}/graff-evals/claude_code_run.py", "{model}", "{prompt}"],+ "answer": "stdout",+ "usage": "graff-stderr",+ "env": {"DISABLE_TELEMETRY": "1", "DISABLE_AUTOUPDATER": "1", "DISABLE_ERROR_REPORTING": "1", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"},+ "note": "Claude Code (`claude -p`, stream-json) with the task's .mcp.json as its only MCP config. Set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN for the endpoint; a chat-completions-only endpoint needs an Anthropic-format proxy in front (evals/graff-vs-claude-code). Pass --model."}}