graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff vs the Codex app server, same model and account
justrach45c0e4b0d4fb
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 9b5d63fe..379ba320 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -12,6 +12,10 @@ grok-4.6 SuperGrok) is[graff-live-12-report.html](graff-live-12-report.html) plus[artifacts/graff-evals-live/RECEIPT.md](../artifacts/graff-evals-live/RECEIPT.md).+graff against the Codex app server on the same model and ChatGPT account (21+coding and MCP tasks, every run's data) is+[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md).+## Prerequisites- `graff`, `claude`, and `codex` on your PATH, each authenticated.diff --git a/evals/graff-vs-codex-app-server/RESULTS.md b/evals/graff-vs-codex-app-server/RESULTS.mdnew file mode 100644index 00000000..6d4bdcd2--- /dev/null+++ b/evals/graff-vs-codex-app-server/RESULTS.md@@ -0,0 +1,101 @@+# graff vs the Codex app server++Measured 2026-10-01. The same model (`gpt-6.1-sol`), ChatGPT account and machine ran the same 21 tasks in each harness, side by side.++| Arm | What ran |+| --- | --- |+| graff v0.0.302.11 | The published release asset, checked against the release's `SHA256SUMS`. |+| graff main | Main at `900e62d9`: v0.0.302.11 plus #1438, #1439 and #1440 (ADR 0226, 0227, 0228). |+| Codex app server | `codex app-server` from codex-cli 0.159.0. |++graff runs as a scripted `graff repl` over the Codex Responses WebSocket, the transport the app server uses (the `graff-codex-repl` harness). The app server runs through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py), which drives its JSON-RPC and reads the token usage it reports. Each harness uses its own defaults. For this model that means `low` effort and `low` verbosity for graff main and for the app server (the model catalog's defaults), and `medium` effort for v0.0.302.11.++## Results++All 21 tasks, mean per task over 3 runs of each:++| | Pass | Wall per task | List cost per task | Input read from cache | Output tokens per task | Model calls per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| graff v0.0.302.11 | 63/63 | 31.2s | $0.0189 | 92% | 598 | 4.7 |+| graff main | 63/63 | 28.2s | $0.0164 | 93% | 490 | 4.7 |+| Codex app server | 63/63 | 25.0s | $0.0278 | 86% | 382 | 4.3 |++By suite:++| Suite | graff v0.0.302.11 | graff main | Codex app server |+| --- | ---: | ---: | ---: |+| core (11 coding tasks), wall | 17.3s | 17.5s | 17.6s |+| core (11 coding tasks), cost | $0.0078 | $0.0081 | $0.0148 |+| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 |+| mcp (10 MCP tasks), wall | 46.5s | 40.0s | 33.0s |+| mcp (10 MCP tasks), cost | $0.0311 | $0.0256 | $0.0436 |+| mcp (10 MCP tasks), pass | 30/30 | 30/30 | 30/30 |++**Against the release.** graff main is 10% faster and 13% cheaper per task than v0.0.302.11. The gain is in the MCP suite (46.5s to 40.0s per task, $0.0311 to $0.0256), most of it on `linear-split` (114.6s to 65.7s), where sub-agents now call the tools their root loaded (ADR 0227, 0228). The core suite is unchanged.++**Against the Codex app server.** graff main ties on the core suite (17.5s against 17.6s) and is slower on the MCP suite (40.0s against 33.0s). Per MCP task it makes 6.1 model calls to the app server's 5.8 and writes 789 output tokens to its 575; `linear-sidecar` shows it most, 6 to 7 calls against 4. Over all 21 tasks the app server takes 11% less time per task (25.0s against 28.2s).++graff main costs 41% less per task on the app server's own reported usage ($0.0164 against $0.0278), and 43% to 69% less once the app server's unreported warm-up request is counted (see Caveats). graff reads more of its input from cache (93% against 86%); part of that gap is the same warm-up, which caches the app server's first real request without appearing in its usage.++Every arm passed all 63 runs.++## Per task++Wall time is the mean of 3 runs.++| Task | Suite | graff v0.0.302.11 | graff main | Codex app server |+| --- | --- | ---: | ---: | ---: |+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 16.8s | 15.2s | 22.1s |+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 8.9s | 9.0s | 18.9s |+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 4.4s | 4.3s | 6.4s |+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 22.4s | 21.2s | 11.8s |+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 20.9s | 20.5s | 21.6s |+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 24.4s | 32.7s | 17.9s |+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 20.0s | 20.8s | 20.5s |+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 33.7s | 36.7s | 27.1s |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 42.6s | 43.7s | 31.1s |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 43.3s | 39.5s | 31.1s |+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 35.5s | 33.8s | 29.6s |+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 34.0s | 33.6s | 29.1s |+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 38.2s | 36.9s | 34.6s |+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 39.5s | 34.5s | 36.2s |+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 49.0s | 46.8s | 27.1s |+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 114.6s | 65.7s | 58.0s |+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 35.0s | 28.9s | 26.2s |+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.2s | 7.4s | 10.2s |+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 24.3s | 20.7s | 26.9s |+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 18.8s | 19.1s | 14.1s |+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 22.0s | 21.6s | 23.6s |++`results.json` has every run: pass, wall time, model calls, input, cached input and output tokens, and list cost.++## Method++- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The fixture declares its two list tools read-only (`readOnlyHint`), as the hosted server it imitates does. Each task passes or fails on its own check.+- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 8 runs at a time. Each arm made one unscored request first, so no arm started with a prompt cache the others had warmed.+- **Wall time.** The runner's clock around each process, start to exit.+- **Cost.** List price of the tokens each harness reports: $2 per 1M input, $0.10 per 1M cached input and $10 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). A ChatGPT plan is not billed per token; this makes the two harnesses comparable, not a bill.++## Caveats++- **Codex's reported usage leaves out one request.** Before a thread's first turn the app server sends a warm-up request (`generate: false`) with its instructions and tools, about 12k tokens. Its usage is not reported, so the Codex cost above is a floor. Counting it adds between about $0.001 per task (fully cached) and $0.025 (uncached).+- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in either harness.+- **`file-ops` has a newline in its prompt.** The scripted repl sends each line as a turn, so graff runs that task as two turns and the app server as one.+- **Small sample.** 3 runs per task, one machine, one account, one 12-minute run. A few seconds per task is within run-to-run noise.++## Reproduce++```sh+cd graff-evals+# graff-codex-repl runs {repo}/zig-out/bin/graff; --binary points it elsewhere.+# The app server needs `codex` on PATH and CODEX_HOME at a signed-in Codex home.+# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py+# copied to <parent of output-root>/scripts/ for the mcp suite.+python3 run.py --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol \+ --suite core,mcp --reps 3 -j 8 --interleave --output-root /path/outside/repo/run \+ --task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm+```diff --git a/evals/graff-vs-codex-app-server/results.json b/evals/graff-vs-codex-app-server/results.jsonnew file mode 100644index 00000000..6d8c1090--- /dev/null+++ b/evals/graff-vs-codex-app-server/results.json@@ -0,0 +1,3190 @@+{+ "model": "gpt-6.1-sol",+ "tasks": 21,+ "reps": 3,+ "arms": {+ "graff-v302-11": "graff v0.0.302.11 (release)",+ "graff-next": "graff main (ADR 0226, 0227, 0228)",+ "codex-app-server": "Codex app server"+ },+ "summary": {+ "all": {+ "graff-v302-11": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 31.2,+ "list_usd_per_task": 0.0189,+ "cached_input_share": 0.922,+ "output_tokens_per_task": 598,+ "model_calls_per_task": 4.7+ },+ "graff-next": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 28.2,+ "list_usd_per_task": 0.0164,+ "cached_input_share": 0.933,+ "output_tokens_per_task": 490,+ "model_calls_per_task": 4.7+ },+ "codex-app-server": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 25.0,+ "list_usd_per_task": 0.0278,+ "cached_input_share": 0.857,+ "output_tokens_per_task": 382,+ "model_calls_per_task": 4.3+ }+ },+ "core": {+ "graff-v302-11": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 17.3,+ "list_usd_per_task": 0.0078,+ "cached_input_share": 0.964,+ "output_tokens_per_task": 218,+ "model_calls_per_task": 3.2+ },+ "graff-next": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 17.5,+ "list_usd_per_task": 0.0081,+ "cached_input_share": 0.964,+ "output_tokens_per_task": 219,+ "model_calls_per_task": 3.3+ },+ "codex-app-server": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 17.6,+ "list_usd_per_task": 0.0148,+ "cached_input_share": 0.9,+ "output_tokens_per_task": 206,+ "model_calls_per_task": 3.0+ }+ },+ "mcp": {+ "graff-v302-11": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 46.5,+ "list_usd_per_task": 0.0311,+ "cached_input_share": 0.901,+ "output_tokens_per_task": 1015,+ "model_calls_per_task": 6.3+ },+ "graff-next": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 40.0,+ "list_usd_per_task": 0.0256,+ "cached_input_share": 0.917,+ "output_tokens_per_task": 789,+ "model_calls_per_task": 6.1+ },+ "codex-app-server": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 33.0,+ "list_usd_per_task": 0.0436,+ "cached_input_share": 0.84,+ "output_tokens_per_task": 575,+ "model_calls_per_task": 5.8+ }+ }+ },+ "per_task": {+ "csv-sum": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 16.8,+ "list_usd": 0.0071+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 15.2,+ "list_usd": 0.007+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.1,+ "list_usd": 0.0153+ }+ },+ "dead-code": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.9,+ "list_usd": 0.0036+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.0,+ "list_usd": 0.0035+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.9,+ "list_usd": 0.0146+ }+ },+ "exact-reply": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.4,+ "list_usd": 0.0037+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.3,+ "list_usd": 0.0014+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.4,+ "list_usd": 0.0052+ }+ },+ "file-ops": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.4,+ "list_usd": 0.0098+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.2,+ "list_usd": 0.0089+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 11.8,+ "list_usd": 0.0105+ }+ },+ "fix-fib": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.9,+ "list_usd": 0.0108+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.5,+ "list_usd": 0.0106+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.6,+ "list_usd": 0.0183+ }+ },+ "git-ops": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 24.4,+ "list_usd": 0.0106+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 32.7,+ "list_usd": 0.0151+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 17.9,+ "list_usd": 0.0166+ }+ },+ "json-transform": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.0,+ "list_usd": 0.0095+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.8,+ "list_usd": 0.0096+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.5,+ "list_usd": 0.0181+ }+ },+ "linear-issues": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 33.7,+ "list_usd": 0.0206+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 36.7,+ "list_usd": 0.0207+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 27.1,+ "list_usd": 0.0532+ }+ },+ "linear-nohint": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 42.6,+ "list_usd": 0.0244+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 43.7,+ "list_usd": 0.0245+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 31.1,+ "list_usd": 0.0524+ }+ },+ "linear-nohint-warm": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 43.3,+ "list_usd": 0.0233+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 39.5,+ "list_usd": 0.0251+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 31.1,+ "list_usd": 0.0583+ }+ },+ "linear-quiet": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 35.5,+ "list_usd": 0.0164+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 33.8,+ "list_usd": 0.0163+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 29.6,+ "list_usd": 0.0281+ }+ },+ "linear-quiet-warm": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 34.0,+ "list_usd": 0.0173+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 33.6,+ "list_usd": 0.0167+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 29.1,+ "list_usd": 0.0282+ }+ },+ "linear-reduce": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 38.2,+ "list_usd": 0.0207+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 36.9,+ "list_usd": 0.0188+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 34.6,+ "list_usd": 0.0489+ }+ },+ "linear-reduce-warm": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 39.5,+ "list_usd": 0.0204+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 34.5,+ "list_usd": 0.0207+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 36.2,+ "list_usd": 0.033+ }+ },+ "linear-sidecar": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 49.0,+ "list_usd": 0.0293+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 46.8,+ "list_usd": 0.0251+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 27.1,+ "list_usd": 0.0499+ }+ },+ "linear-split": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 114.6,+ "list_usd": 0.1186+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 65.7,+ "list_usd": 0.0705+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 58.0,+ "list_usd": 0+ }+ },+ "linear-warm": {+ "suite": "mcp",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 35.0,+ "list_usd": 0.0204+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 28.9,+ "list_usd": 0.0178+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 26.2,+ "list_usd": 0.0404+ }+ },+ "recall-noise": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.2,+ "list_usd": 0.0033+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.4,+ "list_usd": 0.0054+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.2,+ "list_usd": 0.0104+ }+ },+ "refactor-rename": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 24.3,+ "list_usd": 0.0099+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.7,+ "list_usd": 0.0101+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 26.9,+ "list_usd": 0.0224+ }+ },+ "regex-count": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.8,+ "list_usd": 0.0076+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.1,+ "list_usd": 0.0074+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 14.1,+ "list_usd": 0.0136+ }+ },+ "write-tests": {+ "suite": "core",+ "graff-v302-11": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.0,+ "list_usd": 0.0101+ },+ "graff-next": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.6,+ "list_usd": 0.0097+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 23.6,+ "list_usd": 0.0177+ }+ }+ },+ "runs": [+ {+ "arm": "codex-app-server",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.08,+ "model_calls": 3,+ "input_tokens": 43326,+ "cached_input_tokens": 40576,+ "output_tokens": 308,+ "list_usd": 0.012638+ },+ {+ "arm": "codex-app-server",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.12,+ "model_calls": 3,+ "input_tokens": 43418,+ "cached_input_tokens": 40576,+ "output_tokens": 404,+ "list_usd": 0.013782+ },+ {+ "arm": "codex-app-server",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.99,+ "model_calls": 3,+ "input_tokens": 43332,+ "cached_input_tokens": 36864,+ "output_tokens": 286,+ "list_usd": 0.019482+ },+ {+ "arm": "graff-next",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.06,+ "model_calls": 3,+ "input_tokens": 31281,+ "cached_input_tokens": 30336,+ "output_tokens": 216,+ "list_usd": 0.007084+ },+ {+ "arm": "graff-next",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.83,+ "model_calls": 3,+ "input_tokens": 31270,+ "cached_input_tokens": 30464,+ "output_tokens": 203,+ "list_usd": 0.006688+ },+ {+ "arm": "graff-next",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.83,+ "model_calls": 3,+ "input_tokens": 31286,+ "cached_input_tokens": 30336,+ "output_tokens": 219,+ "list_usd": 0.007124+ },+ {+ "arm": "graff-v302-11",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.82,+ "model_calls": 3,+ "input_tokens": 31286,+ "cached_input_tokens": 30720,+ "output_tokens": 223,+ "list_usd": 0.006434+ },+ {+ "arm": "graff-v302-11",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.23,+ "model_calls": 3,+ "input_tokens": 31286,+ "cached_input_tokens": 30336,+ "output_tokens": 223,+ "list_usd": 0.007164+ },+ {+ "arm": "graff-v302-11",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.44,+ "model_calls": 3,+ "input_tokens": 31301,+ "cached_input_tokens": 30336,+ "output_tokens": 272,+ "list_usd": 0.007684+ },+ {+ "arm": "codex-app-server",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.97,+ "model_calls": 4,+ "input_tokens": 58039,+ "cached_input_tokens": 55040,+ "output_tokens": 156,+ "list_usd": 0.013062+ },+ {+ "arm": "codex-app-server",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.93,+ "model_calls": 4,+ "input_tokens": 57949,+ "cached_input_tokens": 54912,+ "output_tokens": 150,+ "list_usd": 0.013065+ },+ {+ "arm": "codex-app-server",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.81,+ "model_calls": 4,+ "input_tokens": 58134,+ "cached_input_tokens": 52864,+ "output_tokens": 171,+ "list_usd": 0.017536+ },+ {+ "arm": "graff-next",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.81,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20352,+ "output_tokens": 56,+ "list_usd": 0.003513+ },+ {+ "arm": "graff-next",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.12,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20352,+ "output_tokens": 56,+ "list_usd": 0.003513+ },+ {+ "arm": "graff-next",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.99,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20352,+ "output_tokens": 56,+ "list_usd": 0.003513+ },+ {+ "arm": "graff-v302-11",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.63,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20352,+ "output_tokens": 70,+ "list_usd": 0.003653+ },+ {+ "arm": "graff-v302-11",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.58,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20352,+ "output_tokens": 56,+ "list_usd": 0.003513+ },+ {+ "arm": "graff-v302-11",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.56,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20352,+ "output_tokens": 56,+ "list_usd": 0.003513+ },+ {+ "arm": "codex-app-server",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.6,+ "model_calls": 1,+ "input_tokens": 14234,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005171+ },+ {+ "arm": "codex-app-server",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.4,+ "model_calls": 1,+ "input_tokens": 14233,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005169+ },+ {+ "arm": "codex-app-server",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.12,+ "model_calls": 1,+ "input_tokens": 14233,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005169+ },+ {+ "arm": "graff-next",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.46,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001379+ },+ {+ "arm": "graff-next",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.19,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001379+ },+ {+ "arm": "graff-next",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.32,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001379+ },+ {+ "arm": "graff-v302-11",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.29,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 6400,+ "output_tokens": 5,+ "list_usd": 0.008432+ },+ {+ "arm": "graff-v302-11",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.54,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001379+ },+ {+ "arm": "graff-v302-11",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.28,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001379+ },+ {+ "arm": "codex-app-server",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.73,+ "model_calls": 2,+ "input_tokens": 28647,+ "cached_input_tokens": 24576,+ "output_tokens": 99,+ "list_usd": 0.01159+ },+ {+ "arm": "codex-app-server",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.56,+ "model_calls": 2,+ "input_tokens": 28653,+ "cached_input_tokens": 24576,+ "output_tokens": 104,+ "list_usd": 0.011652+ },+ {+ "arm": "codex-app-server",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 13.23,+ "model_calls": 2,+ "input_tokens": 28649,+ "cached_input_tokens": 26368,+ "output_tokens": 104,+ "list_usd": 0.008239+ },+ {+ "arm": "graff-next",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.16,+ "model_calls": 5,+ "input_tokens": 51990,+ "cached_input_tokens": 50688,+ "output_tokens": 219,+ "list_usd": 0.009863+ },+ {+ "arm": "graff-next",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.6,+ "model_calls": 5,+ "input_tokens": 51990,+ "cached_input_tokens": 50944,+ "output_tokens": 217,+ "list_usd": 0.009356+ },+ {+ "arm": "graff-next",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.95,+ "model_calls": 4,+ "input_tokens": 41443,+ "cached_input_tokens": 40576,+ "output_tokens": 158,+ "list_usd": 0.007372+ },+ {+ "arm": "graff-v302-11",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.16,+ "model_calls": 4,+ "input_tokens": 41391,+ "cached_input_tokens": 40192,+ "output_tokens": 125,+ "list_usd": 0.007667+ },+ {+ "arm": "graff-v302-11",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.82,+ "model_calls": 5,+ "input_tokens": 52064,+ "cached_input_tokens": 50560,+ "output_tokens": 288,+ "list_usd": 0.010944+ },+ {+ "arm": "graff-v302-11",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.23,+ "model_calls": 5,+ "input_tokens": 52073,+ "cached_input_tokens": 50688,+ "output_tokens": 299,+ "list_usd": 0.010829+ },+ {+ "arm": "codex-app-server",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.37,+ "model_calls": 3,+ "input_tokens": 43831,+ "cached_input_tokens": 38656,+ "output_tokens": 259,+ "list_usd": 0.016806+ },+ {+ "arm": "codex-app-server",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.93,+ "model_calls": 4,+ "input_tokens": 58587,+ "cached_input_tokens": 53120,+ "output_tokens": 287,+ "list_usd": 0.019116+ },+ {+ "arm": "codex-app-server",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.64,+ "model_calls": 4,+ "input_tokens": 58583,+ "cached_input_tokens": 53248,+ "output_tokens": 286,+ "list_usd": 0.018855+ },+ {+ "arm": "graff-next",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.24,+ "model_calls": 4,+ "input_tokens": 42525,+ "cached_input_tokens": 40448,+ "output_tokens": 261,+ "list_usd": 0.010809+ },+ {+ "arm": "graff-next",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.3,+ "model_calls": 4,+ "input_tokens": 42277,+ "cached_input_tokens": 40704,+ "output_tokens": 247,+ "list_usd": 0.009686+ },+ {+ "arm": "graff-next",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.05,+ "model_calls": 4,+ "input_tokens": 42561,+ "cached_input_tokens": 40448,+ "output_tokens": 289,+ "list_usd": 0.011161+ },+ {+ "arm": "graff-v302-11",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.05,+ "model_calls": 4,+ "input_tokens": 42551,+ "cached_input_tokens": 40832,+ "output_tokens": 283,+ "list_usd": 0.010351+ },+ {+ "arm": "graff-v302-11",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.92,+ "model_calls": 4,+ "input_tokens": 42547,+ "cached_input_tokens": 40448,+ "output_tokens": 277,+ "list_usd": 0.011013+ },+ {+ "arm": "graff-v302-11",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.69,+ "model_calls": 4,+ "input_tokens": 42549,+ "cached_input_tokens": 40448,+ "output_tokens": 276,+ "list_usd": 0.011007+ },+ {+ "arm": "codex-app-server",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.11,+ "model_calls": 3,+ "input_tokens": 43523,+ "cached_input_tokens": 38912,+ "output_tokens": 206,+ "list_usd": 0.015173+ },+ {+ "arm": "codex-app-server",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.36,+ "model_calls": 3,+ "input_tokens": 43519,+ "cached_input_tokens": 38656,+ "output_tokens": 203,+ "list_usd": 0.015622+ },+ {+ "arm": "codex-app-server",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.25,+ "model_calls": 3,+ "input_tokens": 43551,+ "cached_input_tokens": 36864,+ "output_tokens": 193,+ "list_usd": 0.01899+ },+ {+ "arm": "graff-next",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.24,+ "model_calls": 6,+ "input_tokens": 64159,+ "cached_input_tokens": 61056,+ "output_tokens": 573,+ "list_usd": 0.018042+ },+ {+ "arm": "graff-next",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.31,+ "model_calls": 3,+ "input_tokens": 31714,+ "cached_input_tokens": 30336,+ "output_tokens": 344,+ "list_usd": 0.00923+ },+ {+ "arm": "graff-next",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 35.56,+ "model_calls": 5,+ "input_tokens": 53790,+ "cached_input_tokens": 50560,+ "output_tokens": 648,+ "list_usd": 0.017996+ },+ {+ "arm": "graff-v302-11",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.04,+ "model_calls": 3,+ "input_tokens": 31814,+ "cached_input_tokens": 30336,+ "output_tokens": 448,+ "list_usd": 0.01047+ },+ {+ "arm": "graff-v302-11",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.79,+ "model_calls": 4,+ "input_tokens": 42655,+ "cached_input_tokens": 40832,+ "output_tokens": 467,+ "list_usd": 0.012399+ },+ {+ "arm": "graff-v302-11",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.45,+ "model_calls": 3,+ "input_tokens": 31706,+ "cached_input_tokens": 30336,+ "output_tokens": 316,+ "list_usd": 0.008934+ },+ {+ "arm": "codex-app-server",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.71,+ "model_calls": 4,+ "input_tokens": 57997,+ "cached_input_tokens": 52864,+ "output_tokens": 277,+ "list_usd": 0.018322+ },+ {+ "arm": "codex-app-server",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.08,+ "model_calls": 4,+ "input_tokens": 58013,+ "cached_input_tokens": 52992,+ "output_tokens": 275,+ "list_usd": 0.018091+ },+ {+ "arm": "codex-app-server",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.67,+ "model_calls": 4,+ "input_tokens": 57990,+ "cached_input_tokens": 52992,+ "output_tokens": 247,+ "list_usd": 0.017765+ },+ {+ "arm": "graff-next",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.79,+ "model_calls": 4,+ "input_tokens": 41937,+ "cached_input_tokens": 40704,+ "output_tokens": 271,+ "list_usd": 0.009246+ },+ {+ "arm": "graff-next",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.3,+ "model_calls": 4,+ "input_tokens": 41975,+ "cached_input_tokens": 40704,+ "output_tokens": 295,+ "list_usd": 0.009562+ },+ {+ "arm": "graff-next",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.35,+ "model_calls": 4,+ "input_tokens": 41976,+ "cached_input_tokens": 40448,+ "output_tokens": 294,+ "list_usd": 0.010041+ },+ {+ "arm": "graff-v302-11",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.67,+ "model_calls": 4,+ "input_tokens": 41919,+ "cached_input_tokens": 40576,+ "output_tokens": 290,+ "list_usd": 0.009644+ },+ {+ "arm": "graff-v302-11",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.69,+ "model_calls": 4,+ "input_tokens": 41889,+ "cached_input_tokens": 40576,+ "output_tokens": 270,+ "list_usd": 0.009384+ },+ {+ "arm": "graff-v302-11",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.75,+ "model_calls": 4,+ "input_tokens": 41907,+ "cached_input_tokens": 40576,+ "output_tokens": 280,+ "list_usd": 0.00952+ },+ {+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.29,+ "model_calls": 4,+ "input_tokens": 82398,+ "cached_input_tokens": 66688,+ "output_tokens": 388,+ "list_usd": 0.041969+ },+ {+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.73,+ "model_calls": 5,+ "input_tokens": 131306,+ "cached_input_tokens": 109312,+ "output_tokens": 506,+ "list_usd": 0.059979+ },+ {+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.31,+ "model_calls": 4,+ "input_tokens": 78755,+ "cached_input_tokens": 54656,+ "output_tokens": 410,+ "list_usd": 0.057764+ },+ {+ "arm": "graff-next",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.44,+ "model_calls": 5,+ "input_tokens": 57133,+ "cached_input_tokens": 54144,+ "output_tokens": 599,+ "list_usd": 0.017382+ },+ {+ "arm": "graff-next",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 35.36,+ "model_calls": 5,+ "input_tokens": 57541,+ "cached_input_tokens": 52736,+ "output_tokens": 613,+ "list_usd": 0.021014+ },+ {+ "arm": "graff-next",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.27,+ "model_calls": 5,+ "input_tokens": 57293,+ "cached_input_tokens": 51200,+ "output_tokens": 631,+ "list_usd": 0.023616+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 35.54,+ "model_calls": 5,+ "input_tokens": 57185,+ "cached_input_tokens": 51200,+ "output_tokens": 575,+ "list_usd": 0.02284+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 31.41,+ "model_calls": 4,+ "input_tokens": 45746,+ "cached_input_tokens": 40960,+ "output_tokens": 580,+ "list_usd": 0.019468+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 34.06,+ "model_calls": 4,+ "input_tokens": 45436,+ "cached_input_tokens": 40960,+ "output_tokens": 632,+ "list_usd": 0.019368+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.64,+ "model_calls": 4,+ "input_tokens": 102531,+ "cached_input_tokens": 80128,+ "output_tokens": 443,+ "list_usd": 0.057249+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.92,+ "model_calls": 4,+ "input_tokens": 105527,+ "cached_input_tokens": 83840,+ "output_tokens": 442,+ "list_usd": 0.056178+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.74,+ "model_calls": 5,+ "input_tokens": 115163,+ "cached_input_tokens": 100864,+ "output_tokens": 504,+ "list_usd": 0.043724+ },+ {+ "arm": "graff-next",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 50.7,+ "model_calls": 7,+ "input_tokens": 80135,+ "cached_input_tokens": 75136,+ "output_tokens": 873,+ "list_usd": 0.026242+ },+ {+ "arm": "graff-next",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.2,+ "model_calls": 6,+ "input_tokens": 67932,+ "cached_input_tokens": 64384,+ "output_tokens": 752,+ "list_usd": 0.021054+ },+ {+ "arm": "graff-next",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 41.31,+ "model_calls": 6,+ "input_tokens": 67909,+ "cached_input_tokens": 61568,+ "output_tokens": 739,+ "list_usd": 0.026229+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.35,+ "model_calls": 6,+ "input_tokens": 67928,+ "cached_input_tokens": 62080,+ "output_tokens": 753,+ "list_usd": 0.025434+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 41.26,+ "model_calls": 6,+ "input_tokens": 67928,+ "cached_input_tokens": 63616,+ "output_tokens": 754,+ "list_usd": 0.022526+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 46.34,+ "model_calls": 6,+ "input_tokens": 68431,+ "cached_input_tokens": 63872,+ "output_tokens": 982,+ "list_usd": 0.025325+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.35,+ "model_calls": 4,+ "input_tokens": 102526,+ "cached_input_tokens": 80128,+ "output_tokens": 428,+ "list_usd": 0.057089+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.49,+ "model_calls": 4,+ "input_tokens": 104964,+ "cached_input_tokens": 81664,+ "output_tokens": 449,+ "list_usd": 0.059256+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.41,+ "model_calls": 4,+ "input_tokens": 103865,+ "cached_input_tokens": 80896,+ "output_tokens": 456,+ "list_usd": 0.058588+ },+ {+ "arm": "graff-next",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.95,+ "model_calls": 6,+ "input_tokens": 69113,+ "cached_input_tokens": 63616,+ "output_tokens": 721,+ "list_usd": 0.024566+ },+ {+ "arm": "graff-next",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.71,+ "model_calls": 6,+ "input_tokens": 69164,+ "cached_input_tokens": 63616,+ "output_tokens": 795,+ "list_usd": 0.025408+ },+ {+ "arm": "graff-next",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.69,+ "model_calls": 6,+ "input_tokens": 69249,+ "cached_input_tokens": 63616,+ "output_tokens": 767,+ "list_usd": 0.025298+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 43.51,+ "model_calls": 6,+ "input_tokens": 69306,+ "cached_input_tokens": 63360,+ "output_tokens": 796,+ "list_usd": 0.026188+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 46.2,+ "model_calls": 6,+ "input_tokens": 69673,+ "cached_input_tokens": 65664,+ "output_tokens": 923,+ "list_usd": 0.023814+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.09,+ "model_calls": 6,+ "input_tokens": 69145,+ "cached_input_tokens": 66304,+ "output_tokens": 760,+ "list_usd": 0.019912+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.32,+ "model_calls": 4,+ "input_tokens": 63308,+ "cached_input_tokens": 52864,+ "output_tokens": 479,+ "list_usd": 0.030964+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.73,+ "model_calls": 5,+ "input_tokens": 79606,+ "cached_input_tokens": 68864,+ "output_tokens": 514,+ "list_usd": 0.03351+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.88,+ "model_calls": 4,+ "input_tokens": 63293,+ "cached_input_tokens": 58496,+ "output_tokens": 435,+ "list_usd": 0.019794+ },+ {+ "arm": "graff-next",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.02,+ "model_calls": 3,+ "input_tokens": 33539,+ "cached_input_tokens": 31232,+ "output_tokens": 508,+ "list_usd": 0.012817+ },+ {+ "arm": "graff-next",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 44.37,+ "model_calls": 4,+ "input_tokens": 45830,+ "cached_input_tokens": 41728,+ "output_tokens": 893,+ "list_usd": 0.021307+ },+ {+ "arm": "graff-next",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.12,+ "model_calls": 3,+ "input_tokens": 33616,+ "cached_input_tokens": 30720,+ "output_tokens": 606,+ "list_usd": 0.014924+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.88,+ "model_calls": 3,+ "input_tokens": 33819,+ "cached_input_tokens": 30720,+ "output_tokens": 764,+ "list_usd": 0.01691+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.42,+ "model_calls": 3,+ "input_tokens": 33831,+ "cached_input_tokens": 30720,+ "output_tokens": 841,+ "list_usd": 0.017704+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.18,+ "model_calls": 3,+ "input_tokens": 33635,+ "cached_input_tokens": 30720,+ "output_tokens": 564,+ "list_usd": 0.014542+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.01,+ "model_calls": 5,+ "input_tokens": 79608,+ "cached_input_tokens": 70656,+ "output_tokens": 460,+ "list_usd": 0.02957+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.91,+ "model_calls": 5,+ "input_tokens": 84598,+ "cached_input_tokens": 73344,+ "output_tokens": 540,+ "list_usd": 0.035242+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.39,+ "model_calls": 4,+ "input_tokens": 63291,+ "cached_input_tokens": 58496,+ "output_tokens": 443,+ "list_usd": 0.01987+ },+ {+ "arm": "graff-next",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 35.13,+ "model_calls": 4,+ "input_tokens": 46171,+ "cached_input_tokens": 42496,+ "output_tokens": 656,+ "list_usd": 0.01816+ },+ {+ "arm": "graff-next",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.18,+ "model_calls": 3,+ "input_tokens": 34078,+ "cached_input_tokens": 30720,+ "output_tokens": 585,+ "list_usd": 0.015638+ },+ {+ "arm": "graff-next",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.51,+ "model_calls": 3,+ "input_tokens": 34235,+ "cached_input_tokens": 31488,+ "output_tokens": 764,+ "list_usd": 0.016283+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 41.66,+ "model_calls": 4,+ "input_tokens": 46759,+ "cached_input_tokens": 42624,+ "output_tokens": 742,+ "list_usd": 0.019952+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.96,+ "model_calls": 4,+ "input_tokens": 46403,+ "cached_input_tokens": 43264,+ "output_tokens": 561,+ "list_usd": 0.016214+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.51,+ "model_calls": 3,+ "input_tokens": 34085,+ "cached_input_tokens": 30720,+ "output_tokens": 581,+ "list_usd": 0.015612+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.76,+ "model_calls": 5,+ "input_tokens": 107419,+ "cached_input_tokens": 89344,+ "output_tokens": 608,+ "list_usd": 0.051164+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 33.79,+ "model_calls": 5,+ "input_tokens": 107311,+ "cached_input_tokens": 92928,+ "output_tokens": 626,+ "list_usd": 0.044319+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.32,+ "model_calls": 6,+ "input_tokens": 133180,+ "cached_input_tokens": 116480,+ "output_tokens": 626,+ "list_usd": 0.051308+ },+ {+ "arm": "graff-next",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.1,+ "model_calls": 5,+ "input_tokens": 56886,+ "cached_input_tokens": 52224,+ "output_tokens": 653,+ "list_usd": 0.021076+ },+ {+ "arm": "graff-next",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.92,+ "model_calls": 4,+ "input_tokens": 45136,+ "cached_input_tokens": 41984,+ "output_tokens": 614,+ "list_usd": 0.016642+ },+ {+ "arm": "graff-next",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.74,+ "model_calls": 5,+ "input_tokens": 56686,+ "cached_input_tokens": 53248,+ "output_tokens": 645,+ "list_usd": 0.018651+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.49,+ "model_calls": 4,+ "input_tokens": 45075,+ "cached_input_tokens": 41984,+ "output_tokens": 539,+ "list_usd": 0.01577+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 33.75,+ "model_calls": 4,+ "input_tokens": 45113,+ "cached_input_tokens": 40960,+ "output_tokens": 581,+ "list_usd": 0.018212+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 51.42,+ "model_calls": 6,+ "input_tokens": 69541,+ "cached_input_tokens": 63488,+ "output_tokens": 966,+ "list_usd": 0.028115+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 35.38,+ "model_calls": 6,+ "input_tokens": 96202,+ "cached_input_tokens": 86784,+ "output_tokens": 590,+ "list_usd": 0.033414+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 34.92,+ "model_calls": 6,+ "input_tokens": 96157,+ "cached_input_tokens": 87040,+ "output_tokens": 549,+ "list_usd": 0.032428+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.31,+ "model_calls": 6,+ "input_tokens": 96155,+ "cached_input_tokens": 86784,+ "output_tokens": 568,+ "list_usd": 0.0331+ },+ {+ "arm": "graff-next",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.54,+ "model_calls": 5,+ "input_tokens": 57658,+ "cached_input_tokens": 52480,+ "output_tokens": 559,+ "list_usd": 0.021194+ },+ {+ "arm": "graff-next",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.58,+ "model_calls": 5,+ "input_tokens": 57972,+ "cached_input_tokens": 52480,+ "output_tokens": 707,+ "list_usd": 0.023302+ },+ {+ "arm": "graff-next",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.26,+ "model_calls": 4,+ "input_tokens": 45909,+ "cached_input_tokens": 42240,+ "output_tokens": 594,+ "list_usd": 0.017502+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.23,+ "model_calls": 4,+ "input_tokens": 45833,+ "cached_input_tokens": 42240,+ "output_tokens": 561,+ "list_usd": 0.01702+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.59,+ "model_calls": 5,+ "input_tokens": 57909,+ "cached_input_tokens": 52608,+ "output_tokens": 656,+ "list_usd": 0.022423+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 45.54,+ "model_calls": 6,+ "input_tokens": 69948,+ "cached_input_tokens": 66176,+ "output_tokens": 747,+ "list_usd": 0.021632+ },+ {+ "arm": "codex-app-server",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.2,+ "model_calls": 4,+ "input_tokens": 78016,+ "cached_input_tokens": 65664,+ "output_tokens": 420,+ "list_usd": 0.03547+ },+ {+ "arm": "codex-app-server",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.82,+ "model_calls": 4,+ "input_tokens": 105531,+ "cached_input_tokens": 82048,+ "output_tokens": 417,+ "list_usd": 0.059341+ },+ {+ "arm": "codex-app-server",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.4,+ "model_calls": 4,+ "input_tokens": 104169,+ "cached_input_tokens": 83072,+ "output_tokens": 432,+ "list_usd": 0.054821+ },+ {+ "arm": "graff-next",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 49.54,+ "model_calls": 7,+ "input_tokens": 80485,+ "cached_input_tokens": 76160,+ "output_tokens": 967,+ "list_usd": 0.025936+ },+ {+ "arm": "graff-next",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 52.58,+ "model_calls": 7,+ "input_tokens": 80592,+ "cached_input_tokens": 75776,+ "output_tokens": 979,+ "list_usd": 0.027+ },+ {+ "arm": "graff-next",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.4,+ "model_calls": 6,+ "input_tokens": 68164,+ "cached_input_tokens": 64000,+ "output_tokens": 763,+ "list_usd": 0.022358+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 47.53,+ "model_calls": 6,+ "input_tokens": 68696,+ "cached_input_tokens": 63360,+ "output_tokens": 986,+ "list_usd": 0.026868+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 49.91,+ "model_calls": 7,+ "input_tokens": 80552,+ "cached_input_tokens": 73472,+ "output_tokens": 1000,+ "list_usd": 0.031507+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 49.5,+ "model_calls": 7,+ "input_tokens": 80632,+ "cached_input_tokens": 74368,+ "output_tokens": 960,+ "list_usd": 0.029565+ },+ {+ "arm": "codex-app-server",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 60.23,+ "model_calls": 16,+ "input_tokens": 366660,+ "cached_input_tokens": 309632,+ "output_tokens": 1384,+ "list_usd": null+ },+ {+ "arm": "codex-app-server",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 57.52,+ "model_calls": 16,+ "input_tokens": 367431,+ "cached_input_tokens": 312704,+ "output_tokens": 1416,+ "list_usd": null+ },+ {+ "arm": "codex-app-server",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 56.25,+ "model_calls": 18,+ "input_tokens": 439318,+ "cached_input_tokens": 377344,+ "output_tokens": 1495,+ "list_usd": null+ },+ {+ "arm": "graff-next",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 67.91,+ "model_calls": 18,+ "input_tokens": 190207,+ "cached_input_tokens": 167680,+ "output_tokens": 1688,+ "list_usd": 0.078702+ },+ {+ "arm": "graff-next",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 58.66,+ "model_calls": 17,+ "input_tokens": 176523,+ "cached_input_tokens": 160384,+ "output_tokens": 1628,+ "list_usd": 0.064596+ },+ {+ "arm": "graff-next",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 70.64,+ "model_calls": 18,+ "input_tokens": 190968,+ "cached_input_tokens": 174720,+ "output_tokens": 1823,+ "list_usd": 0.068198+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 140.58,+ "model_calls": 21,+ "input_tokens": 256931,+ "cached_input_tokens": 213760,+ "output_tokens": 4107,+ "list_usd": 0.148788+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 107.78,+ "model_calls": 21,+ "input_tokens": 226902,+ "cached_input_tokens": 199808,+ "output_tokens": 3823,+ "list_usd": 0.112399+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 95.44,+ "model_calls": 17,+ "input_tokens": 202662,+ "cached_input_tokens": 177664,+ "output_tokens": 2686,+ "list_usd": 0.094622+ },+ {+ "arm": "codex-app-server",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.83,+ "model_calls": 4,+ "input_tokens": 82420,+ "cached_input_tokens": 66688,+ "output_tokens": 407,+ "list_usd": 0.042203+ },+ {+ "arm": "codex-app-server",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.12,+ "model_calls": 4,+ "input_tokens": 78746,+ "cached_input_tokens": 64256,+ "output_tokens": 409,+ "list_usd": 0.039496+ },+ {+ "arm": "codex-app-server",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.51,+ "model_calls": 4,+ "input_tokens": 78756,+ "cached_input_tokens": 64256,+ "output_tokens": 416,+ "list_usd": 0.039586+ },+ {+ "arm": "graff-next",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.57,+ "model_calls": 3,+ "input_tokens": 33957,+ "cached_input_tokens": 30720,+ "output_tokens": 513,+ "list_usd": 0.014676+ },+ {+ "arm": "graff-next",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.41,+ "model_calls": 4,+ "input_tokens": 46236,+ "cached_input_tokens": 40960,+ "output_tokens": 537,+ "list_usd": 0.020018+ },+ {+ "arm": "graff-next",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.59,+ "model_calls": 4,+ "input_tokens": 46376,+ "cached_input_tokens": 41728,+ "output_tokens": 511,+ "list_usd": 0.018579+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.73,+ "model_calls": 4,+ "input_tokens": 46192,+ "cached_input_tokens": 40960,+ "output_tokens": 523,+ "list_usd": 0.01979+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 35.51,+ "model_calls": 5,+ "input_tokens": 58221,+ "cached_input_tokens": 54016,+ "output_tokens": 599,+ "list_usd": 0.019802+ },+ {+ "arm": "graff-v302-11",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.91,+ "model_calls": 4,+ "input_tokens": 46634,+ "cached_input_tokens": 42496,+ "output_tokens": 914,+ "list_usd": 0.021666+ },+ {+ "arm": "codex-app-server",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.81,+ "model_calls": 2,+ "input_tokens": 28817,+ "cached_input_tokens": 24576,+ "output_tokens": 57,+ "list_usd": 0.01151+ },+ {+ "arm": "codex-app-server",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.09,+ "model_calls": 2,+ "input_tokens": 28814,+ "cached_input_tokens": 24576,+ "output_tokens": 57,+ "list_usd": 0.011504+ },+ {+ "arm": "codex-app-server",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.69,+ "model_calls": 2,+ "input_tokens": 28808,+ "cached_input_tokens": 26368,+ "output_tokens": 52,+ "list_usd": 0.008037+ },+ {+ "arm": "graff-next",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.4,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-next",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.69,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 16768,+ "output_tokens": 29,+ "list_usd": 0.010149+ },+ {+ "arm": "graff-next",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.1,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-v302-11",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.56,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-v302-11",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.68,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-v302-11",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.38,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20224,+ "output_tokens": 43,+ "list_usd": 0.003722+ },+ {+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.72,+ "model_calls": 4,+ "input_tokens": 58425,+ "cached_input_tokens": 50944,+ "output_tokens": 365,+ "list_usd": 0.023706+ },+ {+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.76,+ "model_calls": 4,+ "input_tokens": 58340,+ "cached_input_tokens": 50944,+ "output_tokens": 374,+ "list_usd": 0.023626+ },+ {+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.34,+ "model_calls": 4,+ "input_tokens": 58471,+ "cached_input_tokens": 52992,+ "output_tokens": 375,+ "list_usd": 0.020007+ },+ {+ "arm": "graff-next",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.81,+ "model_calls": 4,+ "input_tokens": 42164,+ "cached_input_tokens": 40576,+ "output_tokens": 304,+ "list_usd": 0.010274+ },+ {+ "arm": "graff-next",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.8,+ "model_calls": 4,+ "input_tokens": 42152,+ "cached_input_tokens": 40832,+ "output_tokens": 310,+ "list_usd": 0.009823+ },+ {+ "arm": "graff-next",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.5,+ "model_calls": 4,+ "input_tokens": 42164,+ "cached_input_tokens": 40704,+ "output_tokens": 316,+ "list_usd": 0.01015+ },+ {+ "arm": "graff-v302-11",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.19,+ "model_calls": 4,+ "input_tokens": 42142,+ "cached_input_tokens": 40704,+ "output_tokens": 305,+ "list_usd": 0.009996+ },+ {+ "arm": "graff-v302-11",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.0,+ "model_calls": 4,+ "input_tokens": 42190,+ "cached_input_tokens": 40832,+ "output_tokens": 332,+ "list_usd": 0.010119+ },+ {+ "arm": "graff-v302-11",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.71,+ "model_calls": 4,+ "input_tokens": 42186,+ "cached_input_tokens": 41088,+ "output_tokens": 327,+ "list_usd": 0.009575+ },+ {+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.33,+ "model_calls": 2,+ "input_tokens": 28686,+ "cached_input_tokens": 26368,+ "output_tokens": 111,+ "list_usd": 0.008383+ },+ {+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.72,+ "model_calls": 3,+ "input_tokens": 43218,+ "cached_input_tokens": 36864,+ "output_tokens": 172,+ "list_usd": 0.018114+ },+ {+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.27,+ "model_calls": 3,+ "input_tokens": 43241,+ "cached_input_tokens": 38656,+ "output_tokens": 128,+ "list_usd": 0.014316+ },+ {+ "arm": "graff-next",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.4,+ "model_calls": 4,+ "input_tokens": 41918,+ "cached_input_tokens": 40704,+ "output_tokens": 230,+ "list_usd": 0.008798+ },+ {+ "arm": "graff-next",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.32,+ "model_calls": 2,+ "input_tokens": 20863,+ "cached_input_tokens": 20224,+ "output_tokens": 224,+ "list_usd": 0.00554+ },+ {+ "arm": "graff-next",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.68,+ "model_calls": 3,+ "input_tokens": 31380,+ "cached_input_tokens": 30336,+ "output_tokens": 280,+ "list_usd": 0.007922+ },+ {+ "arm": "graff-v302-11",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.81,+ "model_calls": 3,+ "input_tokens": 31390,+ "cached_input_tokens": 30208,+ "output_tokens": 206,+ "list_usd": 0.007445+ },+ {+ "arm": "graff-v302-11",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.79,+ "model_calls": 3,+ "input_tokens": 31340,+ "cached_input_tokens": 30464,+ "output_tokens": 240,+ "list_usd": 0.007198+ },+ {+ "arm": "graff-v302-11",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.94,+ "model_calls": 3,+ "input_tokens": 31387,+ "cached_input_tokens": 30208,+ "output_tokens": 287,+ "list_usd": 0.008249+ },+ {+ "arm": "codex-app-server",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.95,+ "model_calls": 4,+ "input_tokens": 57983,+ "cached_input_tokens": 54912,+ "output_tokens": 348,+ "list_usd": 0.015113+ },+ {+ "arm": "codex-app-server",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.81,+ "model_calls": 4,+ "input_tokens": 58008,+ "cached_input_tokens": 52992,+ "output_tokens": 356,+ "list_usd": 0.018891+ },+ {+ "arm": "codex-app-server",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.03,+ "model_calls": 4,+ "input_tokens": 58015,+ "cached_input_tokens": 52864,+ "output_tokens": 361,+ "list_usd": 0.019198+ },+ {+ "arm": "graff-next",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.07,+ "model_calls": 4,+ "input_tokens": 41938,+ "cached_input_tokens": 40320,+ "output_tokens": 280,+ "list_usd": 0.010068+ },+ {+ "arm": "graff-next",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.39,+ "model_calls": 4,+ "input_tokens": 41909,+ "cached_input_tokens": 40576,+ "output_tokens": 269,+ "list_usd": 0.009414+ },+ {+ "arm": "graff-next",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.42,+ "model_calls": 4,+ "input_tokens": 41933,+ "cached_input_tokens": 40576,+ "output_tokens": 280,+ "list_usd": 0.009572+ },+ {+ "arm": "graff-v302-11",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.26,+ "model_calls": 4,+ "input_tokens": 41950,+ "cached_input_tokens": 40192,+ "output_tokens": 298,+ "list_usd": 0.010515+ },+ {+ "arm": "graff-v302-11",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.2,+ "model_calls": 4,+ "input_tokens": 41960,+ "cached_input_tokens": 40320,+ "output_tokens": 297,+ "list_usd": 0.010282+ },+ {+ "arm": "graff-v302-11",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.44,+ "model_calls": 4,+ "input_tokens": 41912,+ "cached_input_tokens": 40576,+ "output_tokens": 270,+ "list_usd": 0.00943+ }+ ]+}diff --git a/graff-evals/README.md b/graff-evals/README.mdindex b9a66cb4..8eabb2a3 100644--- a/graff-evals/README.md+++ b/graff-evals/README.md@@ -52,6 +52,7 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co# login without its refresh token). Keep --output-root outside any repo so neither# harness reads repo docs; the mcp suite then needs scripts/linear_fixture_mcp.py# copied next to that root (the setup looks for it in a parent directory).+# Results: evals/graff-vs-codex-app-server/RESULTS.md./run.py --suite core,mcp --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol -j 6```