graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff vs the Codex app server on gpt-6-astra, with sub-agents
justrach266941abc25c
diff --git a/docs/adr/0232-sub-agents-at-parity-with-the-codex-route.md b/docs/adr/0232-sub-agents-at-parity-with-the-codex-route.mdindex 84ee9c59..8f3c169e 100644--- a/docs/adr/0232-sub-agents-at-parity-with-the-codex-route.md+++ b/docs/adr/0232-sub-agents-at-parity-with-the-codex-route.md@@ -72,7 +72,7 @@ child keeps its own smaller prompt and catalog, which its siblings alreadyshare. Nor resident-thread eviction: graff frees a child's conversation whenthe child finishes and keeps only its report, and a run with four childrenstayed an order of magnitude below the Codex app server process in resident-memory and CPU time.+memory.## Consequencesdiff --git a/evals/graff-vs-codex-app-server/ASTRA.md b/evals/graff-vs-codex-app-server/ASTRA.mdnew file mode 100644index 00000000..561d253a--- /dev/null+++ b/evals/graff-vs-codex-app-server/ASTRA.md@@ -0,0 +1,67 @@+# graff vs the Codex app server on gpt-6-astra, with sub-agents++Measured 2026-10-01. The same model (`gpt-6-astra`), ChatGPT account and machine ran the same tasks in each harness, interleaved.++| Arm | What ran |+| --- | --- |+| graff | This PR's build: [ADR 0230](../../docs/adr/0230-mcp-connects-in-the-background.md) and [ADR 0232](../../docs/adr/0232-sub-agents-at-parity-with-the-codex-route.md) on top of v0.0.302.13, as a scripted `graff repl` over the Codex Responses WebSocket. |+| Codex app server | `codex app-server` from codex-cli 0.159.0, through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py). |++Both run at the model catalog's defaults: `medium` effort, `low` verbosity. graff delegates with its `subagent` tool; the app server with its multi-agent tools, which run at most three children at a time.++## Results++Mean per task:++| Suite | Runs | Pass (graff / Codex) | Wall per task | List cost per task | Model calls per task |+| --- | ---: | ---: | ---: | ---: | ---: |+| core (11 coding tasks) | 2 | 22/22 / 22/22 | **15.2s** / 18.9s | **$0.077** / $0.091 | 2.9 / 3.2 |+| mcp (10 MCP tasks) | 2 | 20/20 / 20/20 | **27.2s** / 38.7s | **$0.117** / $0.436 | 3.2 / 6.0 |+| subagents (5 delegation tasks) | 3 | 15/15 / 15/15 | **71.0s** / 86.1s | **$0.427** / $1.586 | 16.3 / 24.4 |+| subagents-solo (the same 5, no delegation) | 2 | 10/10 / 10/10 | **47.9s** / 50.8s | **$0.144** / $0.192 | 4.1 / 4.0 |++graff is faster on every suite: 20% on the coding tasks, 30% on the MCP tasks, 18% when the tasks ask for sub-agents, and 6% when they do not. It costs 16%, 73%, 73% and 25% less, on the app server's own reported usage. Every arm passed every run.++The `subagents` suite is new ([graff-evals/README.md](../../graff-evals/README.md)): three packages fixed in parallel, a census of four services' logs, an API rename across three packages, a coverage merge of two inventories, and a docs sidecar beside a fix, each in a git repository with a held-out check. `subagents-solo` asks for the same work without asking for sub-agents.++## Where sub-agent time goes++Means over the `subagents` runs, from [`graff-evals/subagent_split.py`](../../graff-evals/subagent_split.py):++| | graff | Codex app server |+| --- | ---: | ---: |+| first child started | 10.3s | 22.3s |+| first to last spawn | 10.6s | 50.8s |+| first child start to last child finish | 47.1s | 48.6s |+| slowest child | 39.9s | 40.9s |+| last child finish to the end of the run | 13.2s | 12.0s |+| parent model calls | 6.7 | 13.2 |+| child model calls, all children | 9.6 | 11.2 |++The children themselves take about as long in both. graff starts them sooner and closer together: the app server spawns one child per model call and holds a fourth until a slot frees. graff's parent makes half as many calls; one `agent_output` call with `ids` collects every report.++## Did the children pay for themselves?++Not at this size, in either harness. Asked to delegate, graff took 23.1s (48%) longer than doing the same work itself, and the app server 35.3s (69%) longer. A child starts from a fresh context, so it re-reads what the parent already had, and the parent spends calls starting and collecting it. The suite measures that orchestration cost; work long enough for parallel children to win back their start-up time is not in it.++## Resource use++In one census run with four children, graff's own process peaked at 30 MB resident (53 MB with the Python a child was running at that moment), and the whole process tree used 0.9 s of CPU over 67 s of wall time. The app server's own process held about 270 MB, not counting its code-mode host or the MCP servers it started.++## Method++Each round interleaved the arms task by task (`--interleave`, 6 or 8 workers). Costs are list prices from graff's pricing table (`gpt-6-astra`: $10 per million input tokens, $1 cached, $50 output) applied to each harness's reported usage. Per-run data: [`results-astra.json`](results-astra.json).++## Caveats++Two or three runs per task: a difference of a few seconds on one task is noise. The app server ran with a signed-in Codex home, which can start MCP servers of its own; graff ran with none but the task's. The `subagents` tasks are short, so they show orchestration overhead, not the payoff of parallel work on long tasks.++## Reproduce++```sh+cd graff-evals+./run.py --suite core --harness graff-repl,codex-app-server --model gpt-6-astra -j 8 --interleave+./run.py --suite mcp --harness graff-repl,codex-app-server --model gpt-6-astra -j 8 --interleave+EVAL_TRACE_DIR=<out>-codex-traces ./run.py --suite subagents,subagents-solo --harness graff-repl,codex-app-server --model gpt-6-astra -j 6 --interleave --output-root <out>+python3 subagent_split.py <out>+```diff --git a/evals/graff-vs-codex-app-server/RESULTS.md b/evals/graff-vs-codex-app-server/RESULTS.mdindex ff6a8f35..348d8f25 100644--- a/evals/graff-vs-codex-app-server/RESULTS.md+++ b/evals/graff-vs-codex-app-server/RESULTS.md@@ -2,6 +2,8 @@Measured 2026-10-01. The same model (`gpt-6.1-sol`), ChatGPT account and machine ran the same 21 tasks in each harness, side by side.+On `gpt-6-astra`, and on tasks that ask for sub-agents: [ASTRA.md](ASTRA.md).+| Arm | What ran || --- | --- || graff | The ADR 0231 build (#1449: the `perf: drop round trips the model does not need` commit on top of v0.0.302.12), as a scripted `graff repl` over the Codex Responses WebSocket. |diff --git a/evals/graff-vs-codex-app-server/results-astra.json b/evals/graff-vs-codex-app-server/results-astra.jsonnew file mode 100644index 00000000..c3b6852f--- /dev/null+++ b/evals/graff-vs-codex-app-server/results-astra.json@@ -0,0 +1,2379 @@+{+ "measured": "2026-10-01",+ "model": "gpt-6-astra",+ "arms": {+ "graff": "graff (this PR: ADR 0230 + ADR 0232 on v0.0.302.13)",+ "codex": "Codex app server (codex-cli 0.159.0)"+ },+ "suites": {+ "core": {+ "reps": 2,+ "tasks": 11,+ "summary": {+ "graff": {+ "runs": 22,+ "passed": 22,+ "wall_s_per_task": 15.2,+ "list_usd_per_task": 0.0767,+ "cached_input_share": 0.86,+ "output_tokens_per_task": 178,+ "model_calls_per_task": 2.9+ },+ "codex": {+ "runs": 22,+ "passed": 22,+ "wall_s_per_task": 18.9,+ "list_usd_per_task": 0.0912,+ "cached_input_share": 0.918,+ "output_tokens_per_task": 224,+ "model_calls_per_task": 3.2+ }+ }+ },+ "mcp": {+ "reps": 2,+ "tasks": 10,+ "summary": {+ "graff": {+ "runs": 20,+ "passed": 20,+ "wall_s_per_task": 27.2,+ "list_usd_per_task": 0.1168,+ "cached_input_share": 0.842,+ "output_tokens_per_task": 561,+ "model_calls_per_task": 3.2+ },+ "codex": {+ "runs": 20,+ "passed": 20,+ "wall_s_per_task": 38.7,+ "list_usd_per_task": 0.4364,+ "cached_input_share": 0.845,+ "output_tokens_per_task": 673,+ "model_calls_per_task": 6+ }+ }+ },+ "subagents": {+ "reps": 3,+ "tasks": 5,+ "summary": {+ "graff": {+ "runs": 15,+ "passed": 15,+ "wall_s_per_task": 71.0,+ "list_usd_per_task": 0.4265,+ "cached_input_share": 0.914,+ "output_tokens_per_task": 2539,+ "model_calls_per_task": 16.3+ },+ "codex": {+ "runs": 15,+ "passed": 15,+ "wall_s_per_task": 86.1,+ "list_usd_per_task": 1.5858,+ "cached_input_share": 0.927,+ "output_tokens_per_task": 2788,+ "model_calls_per_task": 24.4+ }+ }+ },+ "subagents-solo": {+ "reps": 2,+ "tasks": 5,+ "summary": {+ "graff": {+ "runs": 10,+ "passed": 10,+ "wall_s_per_task": 47.9,+ "list_usd_per_task": 0.1437,+ "cached_input_share": 0.916,+ "output_tokens_per_task": 1187,+ "model_calls_per_task": 4.1+ },+ "codex": {+ "runs": 10,+ "passed": 10,+ "wall_s_per_task": 50.8,+ "list_usd_per_task": 0.192,+ "cached_input_share": 0.876,+ "output_tokens_per_task": 1200,+ "model_calls_per_task": 4+ }+ }+ }+ },+ "per_task": {+ "csv-sum": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 17.5,+ "list_usd": 0.0859,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 26.6,+ "list_usd": 0.1183,+ "model_calls": 3.5+ }+ },+ "dead-code": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 9.1,+ "list_usd": 0.0617,+ "model_calls": 2+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 18.0,+ "list_usd": 0.1108,+ "model_calls": 4+ }+ },+ "exact-reply": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 4.0,+ "list_usd": 0.0287,+ "model_calls": 1+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 7.3,+ "list_usd": 0.0314,+ "model_calls": 1+ }+ },+ "file-ops": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 12.8,+ "list_usd": 0.0627,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 12.0,+ "list_usd": 0.0623,+ "model_calls": 2+ }+ },+ "fix-fib": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 21.3,+ "list_usd": 0.125,+ "model_calls": 4+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 23.9,+ "list_usd": 0.1203,+ "model_calls": 4+ }+ },+ "git-ops": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 21.6,+ "list_usd": 0.0947,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 18.0,+ "list_usd": 0.0888,+ "model_calls": 3+ }+ },+ "json-transform": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 15.0,+ "list_usd": 0.0645,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 19.3,+ "list_usd": 0.0958,+ "model_calls": 4+ }+ },+ "recall-noise": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 8.8,+ "list_usd": 0.0436,+ "model_calls": 2+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 11.8,+ "list_usd": 0.0607,+ "model_calls": 2+ }+ },+ "refactor-rename": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 23.1,+ "list_usd": 0.1193,+ "model_calls": 4+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 26.5,+ "list_usd": 0.1197,+ "model_calls": 4+ }+ },+ "regex-count": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 13.6,+ "list_usd": 0.0574,+ "model_calls": 2.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 18.3,+ "list_usd": 0.0828,+ "model_calls": 3.5+ }+ },+ "write-tests": {+ "suite": "core",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 20.1,+ "list_usd": 0.0998,+ "model_calls": 4+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 26.6,+ "list_usd": 0.1125,+ "model_calls": 4+ }+ },+ "linear-issues": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 19.4,+ "list_usd": 0.0926,+ "model_calls": 2+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 31.8,+ "list_usd": 0.2447,+ "model_calls": 4.5+ }+ },+ "linear-nohint": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 26.4,+ "list_usd": 0.1052,+ "model_calls": 3.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 29.0,+ "list_usd": 0.2821,+ "model_calls": 4+ }+ },+ "linear-nohint-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 24.2,+ "list_usd": 0.0846,+ "model_calls": 2.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 42.4,+ "list_usd": 0.3946,+ "model_calls": 6.5+ }+ },+ "linear-quiet": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 20.3,+ "list_usd": 0.0651,+ "model_calls": 2+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 36.3,+ "list_usd": 0.2398,+ "model_calls": 5+ }+ },+ "linear-quiet-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 25.9,+ "list_usd": 0.0786,+ "model_calls": 2.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 35.1,+ "list_usd": 0.2471,+ "model_calls": 4+ }+ },+ "linear-reduce": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 23.0,+ "list_usd": 0.0811,+ "model_calls": 2.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 42.4,+ "list_usd": 0.3101,+ "model_calls": 6+ }+ },+ "linear-reduce-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 33.5,+ "list_usd": 0.1268,+ "model_calls": 3.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 36.7,+ "list_usd": 0.2201,+ "model_calls": 5+ }+ },+ "linear-sidecar": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 27.1,+ "list_usd": 0.1339,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 34.1,+ "list_usd": 0.281,+ "model_calls": 5+ }+ },+ "linear-split": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 47.3,+ "list_usd": 0.2842,+ "model_calls": 8+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 64.5,+ "list_usd": 1.8366,+ "model_calls": 16+ }+ },+ "linear-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 25.3,+ "list_usd": 0.1155,+ "model_calls": 2.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 35.1,+ "list_usd": 0.3079,+ "model_calls": 4+ }+ },+ "sa-census": {+ "suite": "subagents",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 60.9,+ "list_usd": 0.3775,+ "model_calls": 15+ },+ "codex": {+ "passed": 3,+ "runs": 3,+ "wall_s": 83.1,+ "list_usd": 1.6704,+ "model_calls": 24.7+ }+ },+ "sa-coverage": {+ "suite": "subagents",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 57.4,+ "list_usd": 0.2819,+ "model_calls": 12+ },+ "codex": {+ "passed": 3,+ "runs": 3,+ "wall_s": 72.5,+ "list_usd": 1.2064,+ "model_calls": 18.3+ }+ },+ "sa-fix-three": {+ "suite": "subagents",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 61.9,+ "list_usd": 0.4618,+ "model_calls": 19+ },+ "codex": {+ "passed": 3,+ "runs": 3,+ "wall_s": 68.8,+ "list_usd": 1.5096,+ "model_calls": 23.7+ }+ },+ "sa-rename-api": {+ "suite": "subagents",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 79.4,+ "list_usd": 0.631,+ "model_calls": 23.7+ },+ "codex": {+ "passed": 3,+ "runs": 3,+ "wall_s": 92.5,+ "list_usd": 1.9411,+ "model_calls": 31.3+ }+ },+ "sa-sidecar": {+ "suite": "subagents",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 95.2,+ "list_usd": 0.3802,+ "model_calls": 12+ },+ "codex": {+ "passed": 3,+ "runs": 3,+ "wall_s": 113.4,+ "list_usd": 1.6014,+ "model_calls": 24+ }+ },+ "solo-census": {+ "suite": "subagents-solo",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 32.0,+ "list_usd": 0.0957,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 24.5,+ "list_usd": 0.1057,+ "model_calls": 3+ }+ },+ "solo-coverage": {+ "suite": "subagents-solo",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 33.5,+ "list_usd": 0.0966,+ "model_calls": 3+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 39.6,+ "list_usd": 0.1642,+ "model_calls": 4+ }+ },+ "solo-fix-three": {+ "suite": "subagents-solo",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 42.1,+ "list_usd": 0.1422,+ "model_calls": 4+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 47.6,+ "list_usd": 0.2228,+ "model_calls": 4.5+ }+ },+ "solo-rename-api": {+ "suite": "subagents-solo",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 62.7,+ "list_usd": 0.1874,+ "model_calls": 5.5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 59.6,+ "list_usd": 0.2188,+ "model_calls": 4+ }+ },+ "solo-sidecar": {+ "suite": "subagents-solo",+ "graff": {+ "passed": 2,+ "runs": 2,+ "wall_s": 69.2,+ "list_usd": 0.1967,+ "model_calls": 5+ },+ "codex": {+ "passed": 2,+ "runs": 2,+ "wall_s": 82.7,+ "list_usd": 0.2484,+ "model_calls": 4.5+ }+ }+ },+ "runs": [+ {+ "arm": "codex",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "wall_s": 20.48,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.101418+ },+ {+ "arm": "codex",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "wall_s": 32.76,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.13514+ },+ {+ "arm": "codex",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "wall_s": 17.44,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.112574+ },+ {+ "arm": "codex",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "wall_s": 18.47,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.109052+ },+ {+ "arm": "codex",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "wall_s": 7.72,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.031358+ },+ {+ "arm": "codex",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "wall_s": 6.92,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.031358+ },+ {+ "arm": "codex",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "wall_s": 12.8,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.0552+ },+ {+ "arm": "codex",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "wall_s": 11.26,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.069436+ },+ {+ "arm": "codex",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "wall_s": 22.57,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.120582+ },+ {+ "arm": "codex",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "wall_s": 25.26,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.120112+ },+ {+ "arm": "codex",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "wall_s": 16.53,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.079326+ },+ {+ "arm": "codex",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "wall_s": 19.5,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.098286+ },+ {+ "arm": "codex",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "wall_s": 19.3,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.095332+ },+ {+ "arm": "codex",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "wall_s": 19.29,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.096332+ },+ {+ "arm": "codex",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "wall_s": 12.63,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.05321+ },+ {+ "arm": "codex",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "wall_s": 11.04,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.068186+ },+ {+ "arm": "codex",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "wall_s": 25.4,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.112108+ },+ {+ "arm": "codex",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "wall_s": 27.51,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.127296+ },+ {+ "arm": "codex",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "wall_s": 19.2,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.090792+ },+ {+ "arm": "codex",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "wall_s": 17.44,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.074886+ },+ {+ "arm": "codex",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "wall_s": 24.96,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.102202+ },+ {+ "arm": "codex",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "wall_s": 28.3,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.122862+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "wall_s": 17.01,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.085944+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "wall_s": 18.04,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.085904+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "wall_s": 8.69,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.062302+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "wall_s": 9.47,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.06115+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "wall_s": 4.65,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.04536+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "wall_s": 3.32,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.011952+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "wall_s": 14.02,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.080728+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "wall_s": 11.64,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.044596+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "wall_s": 21.2,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.141274+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "wall_s": 21.47,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.108728+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "wall_s": 25.22,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.132182+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "wall_s": 18.01,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.057288+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "wall_s": 14.42,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.081472+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "wall_s": 15.57,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.047442+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "wall_s": 8.3,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.061432+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "wall_s": 9.29,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.02572+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "wall_s": 22.52,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.101912+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "wall_s": 23.73,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.136692+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "wall_s": 13.85,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.080842+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "wall_s": 13.37,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.034004+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "wall_s": 20.98,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.099982+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "wall_s": 19.16,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.099676+ },+ {+ "arm": "codex",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 30.13,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.245844+ },+ {+ "arm": "codex",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 33.37,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.243532+ },+ {+ "arm": "codex",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 30.78,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.242624+ },+ {+ "arm": "codex",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 27.29,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.321548+ },+ {+ "arm": "codex",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 40.9,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.34047+ },+ {+ "arm": "codex",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 43.93,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.44878+ },+ {+ "arm": "codex",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 33.86,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.15112+ },+ {+ "arm": "codex",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 38.73,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.328526+ },+ {+ "arm": "codex",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 36.97,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.239926+ },+ {+ "arm": "codex",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 33.25,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.254226+ },+ {+ "arm": "codex",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 40.3,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.177872+ },+ {+ "arm": "codex",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 44.44,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.442334+ },+ {+ "arm": "codex",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 32.58,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.148088+ },+ {+ "arm": "codex",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 40.77,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.292126+ },+ {+ "arm": "codex",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 35.11,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.284694+ },+ {+ "arm": "codex",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 33.14,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.277336+ },+ {+ "arm": "codex",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 65.18,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.871251+ },+ {+ "arm": "codex",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 63.8,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.801867+ },+ {+ "arm": "codex",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 36.74,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.350268+ },+ {+ "arm": "codex",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 33.51,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.265518+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 20.3,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.119484+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 18.5,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.065644+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 25.28,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.097686+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 27.45,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.11265+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 22.96,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.075644+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 25.54,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.093596+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 19.61,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.065464+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 20.92,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.064736+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 28.84,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.091352+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 22.91,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.065872+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 22.61,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.070524+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 23.44,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.091716+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 36.65,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.139448+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 30.29,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.114196+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 27.91,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.167028+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 26.33,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.100852+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 48.9,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.32965+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 45.71,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.238804+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "wall_s": 23.42,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.128654+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "wall_s": 27.26,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.102316+ },+ {+ "arm": "codex",+ "task": "sa-census",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 78.53,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.583248+ },+ {+ "arm": "codex",+ "task": "sa-census",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 91.1,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 2.009003+ },+ {+ "arm": "codex",+ "task": "sa-census",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 79.64,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.418813+ },+ {+ "arm": "codex",+ "task": "sa-coverage",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 72.84,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.166416+ },+ {+ "arm": "codex",+ "task": "sa-coverage",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 69.51,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.240408+ },+ {+ "arm": "codex",+ "task": "sa-coverage",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 75.27,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.212459+ },+ {+ "arm": "codex",+ "task": "sa-fix-three",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 66.16,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.527472+ },+ {+ "arm": "codex",+ "task": "sa-fix-three",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 73.49,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.669447+ },+ {+ "arm": "codex",+ "task": "sa-fix-three",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 66.76,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.331797+ },+ {+ "arm": "codex",+ "task": "sa-rename-api",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 79.28,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.709732+ },+ {+ "arm": "codex",+ "task": "sa-rename-api",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 95.87,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 2.103197+ },+ {+ "arm": "codex",+ "task": "sa-rename-api",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 102.25,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 2.010445+ },+ {+ "arm": "codex",+ "task": "sa-sidecar",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 117.93,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.57212+ },+ {+ "arm": "codex",+ "task": "sa-sidecar",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 119.55,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.871558+ },+ {+ "arm": "codex",+ "task": "sa-sidecar",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 102.86,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 1.360411+ },+ {+ "arm": "graff",+ "task": "sa-census",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 59.98,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.366188+ },+ {+ "arm": "graff",+ "task": "sa-census",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 60.41,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.386332+ },+ {+ "arm": "graff",+ "task": "sa-census",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 62.4,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.379838+ },+ {+ "arm": "graff",+ "task": "sa-coverage",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 59.71,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.296528+ },+ {+ "arm": "graff",+ "task": "sa-coverage",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 55.3,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.284584+ },+ {+ "arm": "graff",+ "task": "sa-coverage",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 57.29,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.264662+ },+ {+ "arm": "graff",+ "task": "sa-fix-three",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 67.06,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.436854+ },+ {+ "arm": "graff",+ "task": "sa-fix-three",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 60.99,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.489968+ },+ {+ "arm": "graff",+ "task": "sa-fix-three",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 57.56,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.458562+ },+ {+ "arm": "graff",+ "task": "sa-rename-api",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 88.74,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.572352+ },+ {+ "arm": "graff",+ "task": "sa-rename-api",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 72.19,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.63029+ },+ {+ "arm": "graff",+ "task": "sa-rename-api",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 77.13,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.690314+ },+ {+ "arm": "graff",+ "task": "sa-sidecar",+ "suite": "subagents",+ "rep": 1,+ "wall_s": 102.72,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.397278+ },+ {+ "arm": "graff",+ "task": "sa-sidecar",+ "suite": "subagents",+ "rep": 2,+ "wall_s": 78.35,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.348168+ },+ {+ "arm": "graff",+ "task": "sa-sidecar",+ "suite": "subagents",+ "rep": 3,+ "wall_s": 104.55,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.395174+ },+ {+ "arm": "codex",+ "task": "solo-census",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 25.16,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.115036+ },+ {+ "arm": "codex",+ "task": "solo-census",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 23.86,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.096294+ },+ {+ "arm": "codex",+ "task": "solo-coverage",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 38.14,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.178592+ },+ {+ "arm": "codex",+ "task": "solo-coverage",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 40.99,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.149784+ },+ {+ "arm": "codex",+ "task": "solo-fix-three",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 45.83,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.170968+ },+ {+ "arm": "codex",+ "task": "solo-fix-three",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 49.33,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.274538+ },+ {+ "arm": "codex",+ "task": "solo-rename-api",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 60.79,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.231488+ },+ {+ "arm": "codex",+ "task": "solo-rename-api",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 58.51,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.206136+ },+ {+ "arm": "codex",+ "task": "solo-sidecar",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 74.44,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.21682+ },+ {+ "arm": "codex",+ "task": "solo-sidecar",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 90.9,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.279966+ },+ {+ "arm": "graff",+ "task": "solo-census",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 29.85,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.091356+ },+ {+ "arm": "graff",+ "task": "solo-census",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 34.21,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.100012+ },+ {+ "arm": "graff",+ "task": "solo-coverage",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 34.0,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.097426+ },+ {+ "arm": "graff",+ "task": "solo-coverage",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 32.94,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.095838+ },+ {+ "arm": "graff",+ "task": "solo-fix-three",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 42.0,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.142392+ },+ {+ "arm": "graff",+ "task": "solo-fix-three",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 42.21,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.14204+ },+ {+ "arm": "graff",+ "task": "solo-rename-api",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 64.4,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.19823+ },+ {+ "arm": "graff",+ "task": "solo-rename-api",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 60.94,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.17649+ },+ {+ "arm": "graff",+ "task": "solo-sidecar",+ "suite": "subagents-solo",+ "rep": 1,+ "wall_s": 70.28,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.211152+ },+ {+ "arm": "graff",+ "task": "solo-sidecar",+ "suite": "subagents-solo",+ "rep": 2,+ "wall_s": 68.06,+ "outcome_ok": null,+ "tok_calls": null,+ "tok_in": null,+ "tok_cached": null,+ "tok_out": null,+ "list_usd": 0.182304+ }+ ]+}\ No newline at end of file