graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff and Pi 1.0 with and without their code modes
justrachb3190e9c5f19
diff --git a/evals/graff-vs-pi-codemode/RESULTS.md b/evals/graff-vs-pi-codemode/RESULTS.mdnew file mode 100644index 00000000..0489472c--- /dev/null+++ b/evals/graff-vs-pi-codemode/RESULTS.md@@ -0,0 +1,169 @@+# graff and Pi 1.0, with and without their code modes++Measured 2026-10-02 on two models: MiMo v2.6 Pro (every arm sent the same model id to the same endpoint with the same key) and gpt-6-astra on a ChatGPT plan. One machine, arms interleaved task by task, 3 runs of each task.++Pi 1.0 added `codemode`: the model writes JavaScript that calls tools as async functions in a QuickJS sandbox, and only what the script prints reaches the model. graff's code mode is `rlm`, a call-only script ([ADR 0029](../../docs/adr/0029-mcp-inside-rlm-and-return-shapes.md)). This compares both harnesses with their code modes on and off, on tasks that need more than the ids and titles graff's MCP result trimming keeps.++## Arms++| Arm | What ran |+| --- | --- |+| graff v0.0.302.16 | The published `graff-aarch64-macos` release binary as a scripted `graff repl` at the default effort (MiMo: thinking off, [ADR 0235](../../docs/adr/0235-mimo-thinks-only-at-high-effort.md); gpt-6-astra: medium reasoning), rlm on, as it ships. `graff-gateway-repl` / `graff-repl`. |+| graff, rlm off | The same binary with `GRAFF_RLM=0`: the structured-only catalog (`graff-gateway-repl-norlm`). |+| graff + ADR 0238 | v0.0.302.16 plus [ADR 0238](../../docs/adr/0238-rlm-binds-keep-the-whole-mcp-result.md): an rlm bind keeps the whole MCP result and only `print()` shows the slim view. |+| Pi 1.0 + codemode | `@earendil-works/pi-coding-agent` 1.0.0, `pi -p --mode json` through [`graff-evals/pi1_run.py`](../../graff-evals/pi1_run.py), with codemode added to the default tools (`"defaultTools": ["+codemode"]`). |+| Pi 1.0 | The same with Pi's default tools. Pi still reaches MCP tools through codemode, its default MCP exposure, and switches codemode on when an MCP server connects. |+| Pi 1.0, MCP direct | Pi's default tools with the task's MCP server at `exposure: "direct"`: MCP tools are declared as ordinary tools and codemode never starts. |++Pi ran at the same thinking setting as graff: `--thinking off` on MiMo (sent as `thinking: {"type": "disabled"}`), `--thinking medium` on gpt-6-astra. On gpt-6-astra both harnesses used a copy of one ChatGPT sign-in with its refresh token removed. Each run gets a private home directory with only the task's MCP server.++## Tasks++| Kind | Tasks | What they need |+| --- | --- | --- |+| MCP reports | `linear-nohint`, `linear-nohint-warm` | Fetch 8 issues and every issue's comments from the Linear-shaped fixture server, write a report of ids, titles, comment counts and latest authors. |+| MCP joins | `xref-issues`, `triage-rollup`, `stale-triage` | Fields the trimmed view drops (priority, estimate) or a join with local files or git history: count issue keys in `src/` with decoys, roll issues up by priority, find priority 1 and 2 issues no commit authored since a date mentions. |+| Commit digest | `commit-digest` | Count commits per top-level directory over a pinned slice of this repository's history (271 commits by author date), in one script. |+| Multi-file coding | `solo-fix-three`, `solo-census`, `solo-rename-api`, `solo-coverage`, `solo-sidecar` | The `subagents-solo` suite: per-package bugfixes, a log census over plain and gzipped files, an API rename, an endpoint-coverage merge, a parser fix with a format doc. |++Every task has a held-out check; none is graded by a model.++## MiMo v2.6 Pro++### Session 1: all 11 tasks++| | Pass | Wall per task | Model calls per task | Input tokens per task | Read from cache | Output tokens per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| graff v0.0.302.16 | 27/33 | 81.5s | 8.3 | 75.9k | 90% | 1,505 |+| graff, rlm off | 30/33 | 55.1s | 8.3 | 79.1k | 90% | 1,807 |+| Pi 1.0 + codemode | 31/33 | 72.0s | 10.3 | 91.1k | 90% | 2,158 |+| Pi 1.0 | 32/33 | 92.8s | 12.4 | 110.7k | 90% | 2,753 |++By kind of task (wall per task, passes, input tokens per task):++| Kind of task | graff v0.0.302.16 | graff, rlm off | Pi 1.0 + codemode | Pi 1.0 |+| --- | ---: | ---: | ---: | ---: |+| MCP reports | 36.0s (6/6, 54k in) | 27.4s (6/6, 28k in) | 43.2s (6/6, 33k in) | 66.3s (6/6, 45k in) |+| MCP joins | 87.5s (8/9, 91k in) | 48.9s (8/9, 87k in) | 47.3s (9/9, 59k in) | 48.6s (9/9, 68k in) |+| Commit digest | 128.6s (2/3, 62k in) | 41.2s (3/3, 89k in) | 56.0s (2/3, 59k in) | 92.3s (3/3, 102k in) |+| Multi-file coding | 86.6s (11/15, 78k in) | 72.7s (13/15, 93k in) | 101.5s (14/15, 140k in) | 130.1s (14/15, 164k in) |++graff was fastest with rlm off (55.1s against 81.5s with it on). In 17 runs without an MCP server MiMo never called Pi's codemode, so Pi's two arms differ on those tasks only by noise and by codemode's tool description; on the MCP tasks both reach MCP through codemode.++| Task | Kind | graff v0.0.302.16 | graff, rlm off | Pi 1.0 + codemode | Pi 1.0 |+| --- | --- | ---: | ---: | ---: | ---: |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | MCP reports | 47.9s | 22.1s | 36.2s | 76.8s |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | MCP reports | 24.1s | 32.7s | 50.2s | 55.7s |+| [`xref-issues`](../../graff-evals/tasks/91-xref-issues.json) | MCP joins | 30.5s | 27.0s (2/3) | 51.2s | 42.7s |+| [`triage-rollup`](../../graff-evals/tasks/92-triage-rollup.json) | MCP joins | 171.9s (2/3) | 66.0s | 38.6s | 34.5s |+| [`stale-triage`](../../graff-evals/tasks/93-stale-triage.json) | MCP joins | 60.1s | 53.8s | 52.0s | 68.5s |+| [`commit-digest`](../../graff-evals/tasks/90-commit-digest.json) | Commit digest | 128.6s (2/3) | 41.2s | 56.0s (2/3) | 92.3s |+| [`solo-fix-three`](../../graff-evals/tasks/67-solo-fix-three.json) | Multi-file coding | 76.0s | 51.5s | 45.5s | 79.5s |+| [`solo-census`](../../graff-evals/tasks/68-solo-census.json) | Multi-file coding | 54.2s | 71.6s | 91.4s | 102.5s |+| [`solo-rename-api`](../../graff-evals/tasks/69-solo-rename-api.json) | Multi-file coding | 63.1s | 58.4s | 47.7s | 36.7s |+| [`solo-coverage`](../../graff-evals/tasks/70-solo-coverage.json) | Multi-file coding | 29.6s (0/3) | 44.0s (1/3) | 26.6s | 48.9s |+| [`solo-sidecar`](../../graff-evals/tasks/71-solo-sidecar.json) | Multi-file coding | 210.0s (2/3) | 138.2s | 296.1s (2/3) | 383.1s (2/3) |++### Session 2: ADR 0238 on the five MCP tasks++| | Pass | Wall per task | Model calls per task | Input tokens per task | Read from cache | Output tokens per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| graff + ADR 0238 | 14/15 | 31.0s | 8.3 | 72.7k | 92% | 1,012 |+| graff v0.0.302.16 | 13/15 | 37.1s | 8.6 | 82.5k | 92% | 1,128 |+| graff, rlm off | 15/15 | 38.3s | 6.7 | 69.9k | 89% | 1,032 |+| Pi 1.0 + codemode | 15/15 | 39.3s | 9.1 | 45.5k | 84% | 1,412 |++| Kind of task | graff + ADR 0238 | graff v0.0.302.16 | graff, rlm off | Pi 1.0 + codemode |+| --- | ---: | ---: | ---: | ---: |+| MCP reports | 26.2s (5/6, 54k in) | 32.9s (5/6, 55k in) | 18.3s (6/6, 28k in) | 37.4s (6/6, 44k in) |+| MCP joins | 34.1s (9/9, 85k in) | 39.9s (8/9, 101k in) | 51.7s (9/9, 98k in) | 40.6s (9/9, 46k in) |++| Task | Kind | graff + ADR 0238 | graff v0.0.302.16 | graff, rlm off | Pi 1.0 + codemode |+| --- | --- | ---: | ---: | ---: | ---: |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | MCP reports | 37.0s | 42.2s | 17.2s | 27.9s |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | MCP reports | 15.4s (2/3) | 23.7s (2/3) | 19.3s | 46.9s |+| [`xref-issues`](../../graff-evals/tasks/91-xref-issues.json) | MCP joins | 15.6s | 32.2s | 31.4s | 39.2s |+| [`triage-rollup`](../../graff-evals/tasks/92-triage-rollup.json) | MCP joins | 40.0s | 56.0s | 62.7s | 40.8s |+| [`stale-triage`](../../graff-evals/tasks/93-stale-triage.json) | MCP joins | 46.8s | 31.5s (2/3) | 60.9s | 42.0s |++### Session 3: Pi with and without codemode on the five MCP tasks++| | Pass | Wall per task | Model calls per task | Input tokens per task | Read from cache | Output tokens per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| Pi 1.0 + codemode | 14/15 | 34.3s | 8.7 | 47.6k | 84% | 1,318 |+| Pi 1.0 | 15/15 | 37.8s | 8.8 | 44.8k | 84% | 1,399 |+| Pi 1.0, MCP direct | 15/15 | 30.5s | 6.9 | 66.9k | 79% | 1,229 |++| Kind of task | Pi 1.0 + codemode | Pi 1.0 | Pi 1.0, MCP direct |+| --- | ---: | ---: | ---: |+| MCP reports | 33.9s (6/6, 35k in) | 38.3s (6/6, 41k in) | 31.6s (6/6, 77k in) |+| MCP joins | 34.5s (8/9, 56k in) | 37.4s (9/9, 47k in) | 29.8s (9/9, 60k in) |++Without codemode Pi was quicker on both kinds of MCP task; codemode read about half the input tokens on the report tasks.++## gpt-6-astra++### Session 1: all 11 tasks++| | Pass | Wall per task | Model calls per task | Input tokens per task | Read from cache | Output tokens per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| graff v0.0.302.16 | 33/33 | 54.1s | 4.5 | 49.4k | 89% | 788 |+| graff, rlm off | 33/33 | 57.3s | 4.6 | 52.2k | 89% | 927 |+| graff + ADR 0238 | 33/33 | 48.3s | 3.8 | 44.0k | 89% | 835 |+| Pi 1.0 + codemode | 33/33 | 41.7s | 4.8 | 20.9k | 54% | 647 |+| Pi 1.0 | 33/33 | 48.1s | 5.2 | 21.9k | 57% | 704 |++By kind of task (wall per task, passes, input tokens per task):++| Kind of task | graff v0.0.302.16 | graff, rlm off | graff + ADR 0238 | Pi 1.0 + codemode | Pi 1.0 |+| --- | ---: | ---: | ---: | ---: | ---: |+| MCP reports | 31.5s (6/6, 39k in) | 38.5s (6/6, 45k in) | 31.4s (6/6, 41k in) | 30.5s (6/6, 26k in) | 30.9s (6/6, 26k in) |+| MCP joins | 51.4s (9/9, 83k in) | 55.8s (9/9, 78k in) | 37.7s (9/9, 46k in) | 34.8s (9/9, 34k in) | 36.7s (9/9, 38k in) |+| Commit digest | 42.2s (3/3, 35k in) | 48.4s (3/3, 45k in) | 34.7s (3/3, 30k in) | 32.7s (3/3, 10k in) | 29.8s (3/3, 9k in) |+| Multi-file coding | 67.2s (15/15, 37k in) | 67.5s (15/15, 41k in) | 64.2s (15/15, 47k in) | 52.1s (15/15, 13k in) | 65.3s (15/15, 13k in) |++Every arm passed every run. With codemode in its default tools Pi used it in 8 of 12 runs without an MCP server (reading files and running commands from a script), and was fastest on the multi-file tasks. rlm helped graff a little (54.1s against 57.3s); astra's rlm scripts were never refused. ADR 0238 took graff from 54.1s to 48.3s per task and from 51.4s to 37.7s on the join tasks; no run paged a stored result with `read_tool_result`, against 6 of 33 for v0.0.302.16.++| Task | Kind | graff v0.0.302.16 | graff, rlm off | graff + ADR 0238 | Pi 1.0 + codemode | Pi 1.0 |+| --- | --- | ---: | ---: | ---: | ---: | ---: |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | MCP reports | 34.3s | 38.5s | 31.4s | 30.8s | 28.9s |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | MCP reports | 28.8s | 38.5s | 31.4s | 30.3s | 32.9s |+| [`xref-issues`](../../graff-evals/tasks/91-xref-issues.json) | MCP joins | 37.7s | 38.3s | 38.1s | 36.9s | 43.2s |+| [`triage-rollup`](../../graff-evals/tasks/92-triage-rollup.json) | MCP joins | 60.1s | 76.3s | 36.3s | 32.5s | 31.9s |+| [`stale-triage`](../../graff-evals/tasks/93-stale-triage.json) | MCP joins | 56.5s | 52.9s | 38.8s | 34.9s | 35.1s |+| [`commit-digest`](../../graff-evals/tasks/90-commit-digest.json) | Commit digest | 42.2s | 48.4s | 34.7s | 32.7s | 29.8s |+| [`solo-fix-three`](../../graff-evals/tasks/67-solo-fix-three.json) | Multi-file coding | 53.0s | 53.1s | 52.5s | 50.6s | 93.6s |+| [`solo-census`](../../graff-evals/tasks/68-solo-census.json) | Multi-file coding | 36.4s | 42.3s | 40.1s | 25.1s | 25.5s |+| [`solo-rename-api`](../../graff-evals/tasks/69-solo-rename-api.json) | Multi-file coding | 114.8s | 105.3s | 92.7s | 63.6s | 84.5s |+| [`solo-coverage`](../../graff-evals/tasks/70-solo-coverage.json) | Multi-file coding | 43.3s | 49.9s | 47.9s | 37.8s | 39.4s |+| [`solo-sidecar`](../../graff-evals/tasks/71-solo-sidecar.json) | Multi-file coding | 88.5s | 86.8s | 87.8s | 83.2s | 83.8s |++### Session 2: Pi with and without codemode on the five MCP tasks++| | Pass | Wall per task | Model calls per task | Input tokens per task | Read from cache | Output tokens per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| Pi 1.0 + codemode | 15/15 | 41.2s | 5.3 | 29.9k | 50% | 388 |+| Pi 1.0 | 15/15 | 35.6s | 5.7 | 33.7k | 55% | 402 |+| Pi 1.0, MCP direct | 15/15 | 40.6s | 5.6 | 41.7k | 63% | 477 |++| Kind of task | Pi 1.0 + codemode | Pi 1.0 | Pi 1.0, MCP direct |+| --- | ---: | ---: | ---: |+| MCP reports | 32.3s (6/6, 26k in) | 34.2s (6/6, 29k in) | 55.8s (6/6, 63k in) |+| MCP joins | 47.1s (9/9, 33k in) | 36.5s (9/9, 37k in) | 30.5s (9/9, 27k in) |++On the report tasks codemode was quicker and read less than half the input tokens; on the join tasks the direct arm was quicker and read fewer tokens.++## Where the time went++- Pi reached MCP through codemode in every MCP run on both models and fetched and transformed data in one script in 28 of 30 runs on MiMo and 18 of 30 on gpt-6-astra. MCP tools are not listed in codemode's description, so a run spent about two calls on `searchTools()` / `describeTool()` before its first fetch.+- graff v0.0.302.16 bound the trimmed cut inside rlm (ids and titles; comment lists as a count and the latest author), so tasks that needed other fields left the script and paged the stored result with `read_tool_result`, whose slices end in a `[a..b of N bytes]` marker that broke the JSON when joined. ADR 0238 binds the whole result.+- MiMo used rlm in 13 of 33 runs (session 1) and hit a refusal in 5 (Python-shaped scripts: loops, comprehensions, multi-argument calls). gpt-6-astra's rlm was never refused.+- On `solo-coverage` graff's model wrote `coverage.json` by hand in all six MiMo runs and five failed (unsorted or a dropped endpoint); Pi computed it with a script in all six and passed.+- graff never names the working directory in its prompt; MiMo sometimes opens with `cd` into an invented path such as `/Users/dev/workspace-…` (6 graff runs in session 1). gpt-6-astra never did.+- graff's first request is about 10,500 input tokens on gpt-6-astra (about 6,200 on MiMo), Pi's about 1,600 (about 2,000 on MiMo), so graff reads roughly twice Pi's input per task; graff's cache hit rate is higher (89% against 54-57% on astra).++## Notes++- Three runs per task is a small sample, and an endpoint's latency swings a lot over a day: compare arms only within a session.+- Wall time is the runner's clock around each process. Model calls and tokens are each harness's own usage report; input includes cached tokens.+- `commit-digest` fetches a pinned commit of this repository from GitHub in its setup; the counts come from that history.diff --git a/evals/graff-vs-pi-codemode/results.json b/evals/graff-vs-pi-codemode/results.jsonnew file mode 100644index 00000000..0f744fdc--- /dev/null+++ b/evals/graff-vs-pi-codemode/results.json@@ -0,0 +1,2384 @@+{+ "measured": "2026-10-02",+ "note": "wall = mean seconds per task; input = all input tokens incl. cached; calls = model calls per task; 3 runs of each task per arm",+ "mimo-v2.6-pro": {+ "session_1_all_tasks": {+ "overall": {+ "graff v0.0.302.16": {+ "runs": 33,+ "pass": 27,+ "wall": 81.5,+ "calls": 8.3,+ "input": 75869,+ "cached_pct": 90,+ "output": 1505,+ "timeouts": 3+ },+ "graff, rlm off": {+ "runs": 33,+ "pass": 30,+ "wall": 55.1,+ "calls": 8.3,+ "input": 79114,+ "cached_pct": 90,+ "output": 1807,+ "timeouts": 0+ },+ "Pi 1.0 + codemode": {+ "runs": 33,+ "pass": 31,+ "wall": 72.0,+ "calls": 10.3,+ "input": 91132,+ "cached_pct": 90,+ "output": 2158,+ "timeouts": 1+ },+ "Pi 1.0": {+ "runs": 33,+ "pass": 32,+ "wall": 92.8,+ "calls": 12.4,+ "input": 110690,+ "cached_pct": 90,+ "output": 2753,+ "timeouts": 1+ }+ },+ "families": {+ "graff v0.0.302.16": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 36.0,+ "calls": 7.2,+ "input": 54086,+ "cached_pct": 89,+ "output": 654,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 8,+ "wall": 87.5,+ "calls": 9.7,+ "input": 91371,+ "cached_pct": 92,+ "output": 1191,+ "timeouts": 1+ },+ "Commit digest": {+ "runs": 3,+ "pass": 2,+ "wall": 128.6,+ "calls": 5.3,+ "input": 61552,+ "cached_pct": 95,+ "output": 1109,+ "timeouts": 1+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 11,+ "wall": 86.6,+ "calls": 8.6,+ "input": 78144,+ "cached_pct": 89,+ "output": 2113,+ "timeouts": 1+ }+ },+ "graff, rlm off": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 27.4,+ "calls": 4,+ "input": 27503,+ "cached_pct": 90,+ "output": 833,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 8,+ "wall": 48.9,+ "calls": 7.3,+ "input": 86764,+ "cached_pct": 88,+ "output": 1367,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 41.2,+ "calls": 8,+ "input": 88800,+ "cached_pct": 96,+ "output": 1235,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 13,+ "wall": 72.7,+ "calls": 10.6,+ "input": 93231,+ "cached_pct": 90,+ "output": 2576,+ "timeouts": 0+ }+ },+ "Pi 1.0 + codemode": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 43.2,+ "calls": 8.8,+ "input": 32554,+ "cached_pct": 82,+ "output": 959,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 47.3,+ "calls": 10.7,+ "input": 58597,+ "cached_pct": 85,+ "output": 1480,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 2,+ "wall": 56.0,+ "calls": 9,+ "input": 59232,+ "cached_pct": 87,+ "output": 1979,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 14,+ "wall": 101.5,+ "calls": 11,+ "input": 140464,+ "cached_pct": 93,+ "output": 3081,+ "timeouts": 1+ }+ },+ "Pi 1.0": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 66.3,+ "calls": 10,+ "input": 44883,+ "cached_pct": 83,+ "output": 1506,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 48.6,+ "calls": 9.1,+ "input": 68387,+ "cached_pct": 84,+ "output": 1459,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 92.3,+ "calls": 12,+ "input": 101775,+ "cached_pct": 89,+ "output": 3436,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 14,+ "wall": 130.1,+ "calls": 15.5,+ "input": 164177,+ "cached_pct": 92,+ "output": 3891,+ "timeouts": 1+ }+ }+ },+ "tasks": {+ "graff v0.0.302.16": {+ "commit-digest": {+ "runs": 3,+ "pass": 2,+ "wall": 128.6,+ "calls": 5.3,+ "input": 61552,+ "cached_pct": 95,+ "output": 1109,+ "timeouts": 1+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 47.9,+ "calls": 10.3,+ "input": 79448,+ "cached_pct": 90,+ "output": 807,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 24.1,+ "calls": 4,+ "input": 28725,+ "cached_pct": 85,+ "output": 501,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 54.2,+ "calls": 7.7,+ "input": 69557,+ "cached_pct": 86,+ "output": 1967,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 0,+ "wall": 29.6,+ "calls": 5,+ "input": 36797,+ "cached_pct": 89,+ "output": 1125,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 76.0,+ "calls": 9,+ "input": 79479,+ "cached_pct": 86,+ "output": 1968,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 63.1,+ "calls": 8.3,+ "input": 70443,+ "cached_pct": 86,+ "output": 2143,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 2,+ "wall": 210.0,+ "calls": 13,+ "input": 134446,+ "cached_pct": 95,+ "output": 3364,+ "timeouts": 1+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 60.1,+ "calls": 11.3,+ "input": 123479,+ "cached_pct": 91,+ "output": 1819,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 2,+ "wall": 171.9,+ "calls": 11,+ "input": 101045,+ "cached_pct": 91,+ "output": 942,+ "timeouts": 1+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 30.5,+ "calls": 6.7,+ "input": 49589,+ "cached_pct": 94,+ "output": 812,+ "timeouts": 0+ }+ },+ "graff, rlm off": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 41.2,+ "calls": 8,+ "input": 88800,+ "cached_pct": 96,+ "output": 1235,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 22.1,+ "calls": 4,+ "input": 27077,+ "cached_pct": 92,+ "output": 735,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 32.7,+ "calls": 4,+ "input": 27929,+ "cached_pct": 87,+ "output": 931,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 71.6,+ "calls": 8,+ "input": 70961,+ "cached_pct": 90,+ "output": 2048,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 1,+ "wall": 44.0,+ "calls": 5.7,+ "input": 39586,+ "cached_pct": 90,+ "output": 1120,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 51.5,+ "calls": 10.7,+ "input": 91271,+ "cached_pct": 94,+ "output": 1959,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 58.4,+ "calls": 9,+ "input": 74199,+ "cached_pct": 86,+ "output": 1966,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 3,+ "wall": 138.2,+ "calls": 19.7,+ "input": 190140,+ "cached_pct": 91,+ "output": 5784,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 53.8,+ "calls": 8.3,+ "input": 125712,+ "cached_pct": 86,+ "output": 2088,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 66.0,+ "calls": 8.3,+ "input": 96209,+ "cached_pct": 87,+ "output": 1081,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 2,+ "wall": 27.0,+ "calls": 5.3,+ "input": 38371,+ "cached_pct": 94,+ "output": 933,+ "timeouts": 0+ }+ },+ "Pi 1.0 + codemode": {+ "commit-digest": {+ "runs": 3,+ "pass": 2,+ "wall": 56.0,+ "calls": 9,+ "input": 59232,+ "cached_pct": 87,+ "output": 1979,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 36.2,+ "calls": 8,+ "input": 26259,+ "cached_pct": 82,+ "output": 896,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 50.2,+ "calls": 9.7,+ "input": 38848,+ "cached_pct": 81,+ "output": 1022,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 91.4,+ "calls": 10,+ "input": 52467,+ "cached_pct": 85,+ "output": 2937,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 26.6,+ "calls": 5.3,+ "input": 20306,+ "cached_pct": 72,+ "output": 1322,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 45.5,+ "calls": 8,+ "input": 36100,+ "cached_pct": 85,+ "output": 1645,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 47.7,+ "calls": 7.3,+ "input": 27157,+ "cached_pct": 81,+ "output": 1607,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 2,+ "wall": 296.1,+ "calls": 24.3,+ "input": 566293,+ "cached_pct": 95,+ "output": 7894,+ "timeouts": 1+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 52.0,+ "calls": 11.7,+ "input": 91915,+ "cached_pct": 86,+ "output": 1920,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 38.6,+ "calls": 9.3,+ "input": 32572,+ "cached_pct": 83,+ "output": 948,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 51.2,+ "calls": 11,+ "input": 51303,+ "cached_pct": 84,+ "output": 1571,+ "timeouts": 0+ }+ },+ "Pi 1.0": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 92.3,+ "calls": 12,+ "input": 101775,+ "cached_pct": 89,+ "output": 3436,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 76.8,+ "calls": 10.7,+ "input": 53272,+ "cached_pct": 85,+ "output": 1707,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 55.7,+ "calls": 9.3,+ "input": 36494,+ "cached_pct": 81,+ "output": 1305,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 102.5,+ "calls": 13,+ "input": 99316,+ "cached_pct": 85,+ "output": 3801,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 48.9,+ "calls": 8.7,+ "input": 35297,+ "cached_pct": 82,+ "output": 1965,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 79.5,+ "calls": 13.7,+ "input": 63737,+ "cached_pct": 86,+ "output": 2457,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 36.7,+ "calls": 6.7,+ "input": 21156,+ "cached_pct": 82,+ "output": 1545,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 2,+ "wall": 383.1,+ "calls": 35.3,+ "input": 601381,+ "cached_pct": 95,+ "output": 9689,+ "timeouts": 1+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 68.5,+ "calls": 10,+ "input": 112817,+ "cached_pct": 84,+ "output": 2125,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 34.5,+ "calls": 8.7,+ "input": 35606,+ "cached_pct": 83,+ "output": 1012,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 42.7,+ "calls": 8.7,+ "input": 56739,+ "cached_pct": 85,+ "output": 1241,+ "timeouts": 0+ }+ }+ }+ },+ "session_2_adr0238_mcp": {+ "overall": {+ "graff + ADR 0238": {+ "runs": 15,+ "pass": 14,+ "wall": 31.0,+ "calls": 8.3,+ "input": 72655,+ "cached_pct": 92,+ "output": 1012,+ "timeouts": 0+ },+ "graff v0.0.302.16": {+ "runs": 15,+ "pass": 13,+ "wall": 37.1,+ "calls": 8.6,+ "input": 82459,+ "cached_pct": 92,+ "output": 1128,+ "timeouts": 0+ },+ "graff, rlm off": {+ "runs": 15,+ "pass": 15,+ "wall": 38.3,+ "calls": 6.7,+ "input": 69919,+ "cached_pct": 89,+ "output": 1032,+ "timeouts": 0+ },+ "Pi 1.0 + codemode": {+ "runs": 15,+ "pass": 15,+ "wall": 39.3,+ "calls": 9.1,+ "input": 45517,+ "cached_pct": 84,+ "output": 1412,+ "timeouts": 0+ }+ },+ "families": {+ "graff + ADR 0238": {+ "MCP reports": {+ "runs": 6,+ "pass": 5,+ "wall": 26.2,+ "calls": 7.3,+ "input": 54407,+ "cached_pct": 93,+ "output": 803,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 34.1,+ "calls": 9,+ "input": 84820,+ "cached_pct": 92,+ "output": 1151,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "graff v0.0.302.16": {+ "MCP reports": {+ "runs": 6,+ "pass": 5,+ "wall": 32.9,+ "calls": 7.3,+ "input": 55381,+ "cached_pct": 92,+ "output": 917,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 8,+ "wall": 39.9,+ "calls": 9.4,+ "input": 100511,+ "cached_pct": 92,+ "output": 1268,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "graff, rlm off": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 18.3,+ "calls": 4,+ "input": 28365,+ "cached_pct": 88,+ "output": 742,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 51.7,+ "calls": 8.4,+ "input": 97622,+ "cached_pct": 89,+ "output": 1226,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "Pi 1.0 + codemode": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 37.4,+ "calls": 9.5,+ "input": 44381,+ "cached_pct": 84,+ "output": 1452,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 40.6,+ "calls": 8.9,+ "input": 46274,+ "cached_pct": 84,+ "output": 1385,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ }+ },+ "tasks": {+ "graff + ADR 0238": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 37.0,+ "calls": 11,+ "input": 82405,+ "cached_pct": 92,+ "output": 1002,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 2,+ "wall": 15.4,+ "calls": 3.7,+ "input": 26409,+ "cached_pct": 95,+ "output": 605,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 46.8,+ "calls": 10.7,+ "input": 119176,+ "cached_pct": 92,+ "output": 1826,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 40.0,+ "calls": 12,+ "input": 103868,+ "cached_pct": 94,+ "output": 1007,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 15.6,+ "calls": 4.3,+ "input": 31416,+ "cached_pct": 87,+ "output": 621,+ "timeouts": 0+ }+ },+ "graff v0.0.302.16": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 42.2,+ "calls": 10,+ "input": 76658,+ "cached_pct": 94,+ "output": 1030,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 2,+ "wall": 23.7,+ "calls": 4.7,+ "input": 34105,+ "cached_pct": 87,+ "output": 804,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 2,+ "wall": 31.5,+ "calls": 8,+ "input": 79210,+ "cached_pct": 88,+ "output": 1209,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 56.0,+ "calls": 14,+ "input": 173117,+ "cached_pct": 93,+ "output": 1603,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 32.2,+ "calls": 6.3,+ "input": 49206,+ "cached_pct": 94,+ "output": 994,+ "timeouts": 0+ }+ },+ "graff, rlm off": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 17.2,+ "calls": 4,+ "input": 28810,+ "cached_pct": 90,+ "output": 720,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 19.3,+ "calls": 4,+ "input": 27920,+ "cached_pct": 86,+ "output": 763,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 60.9,+ "calls": 8,+ "input": 121527,+ "cached_pct": 87,+ "output": 1570,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 62.7,+ "calls": 12,+ "input": 134049,+ "cached_pct": 89,+ "output": 1281,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 31.4,+ "calls": 5.3,+ "input": 37291,+ "cached_pct": 91,+ "output": 827,+ "timeouts": 0+ }+ },+ "Pi 1.0 + codemode": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 27.9,+ "calls": 7.3,+ "input": 40192,+ "cached_pct": 82,+ "output": 1176,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 46.9,+ "calls": 11.7,+ "input": 48570,+ "cached_pct": 86,+ "output": 1727,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 42.0,+ "calls": 8.3,+ "input": 68227,+ "cached_pct": 82,+ "output": 1938,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 40.8,+ "calls": 9.3,+ "input": 35285,+ "cached_pct": 85,+ "output": 1012,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 39.2,+ "calls": 9,+ "input": 35310,+ "cached_pct": 85,+ "output": 1207,+ "timeouts": 0+ }+ }+ }+ },+ "session_3_pi_codemode_mcp": {+ "overall": {+ "Pi 1.0 + codemode": {+ "runs": 15,+ "pass": 14,+ "wall": 34.3,+ "calls": 8.7,+ "input": 47594,+ "cached_pct": 84,+ "output": 1318,+ "timeouts": 0+ },+ "Pi 1.0": {+ "runs": 15,+ "pass": 15,+ "wall": 37.8,+ "calls": 8.8,+ "input": 44825,+ "cached_pct": 84,+ "output": 1399,+ "timeouts": 0+ },+ "Pi 1.0, MCP direct": {+ "runs": 15,+ "pass": 15,+ "wall": 30.5,+ "calls": 6.9,+ "input": 66934,+ "cached_pct": 79,+ "output": 1229,+ "timeouts": 0+ }+ },+ "families": {+ "Pi 1.0 + codemode": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 33.9,+ "calls": 8.2,+ "input": 34803,+ "cached_pct": 82,+ "output": 1273,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 8,+ "wall": 34.5,+ "calls": 9.1,+ "input": 56122,+ "cached_pct": 84,+ "output": 1348,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "Pi 1.0": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 38.3,+ "calls": 9.2,+ "input": 41386,+ "cached_pct": 84,+ "output": 1325,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 37.4,+ "calls": 8.6,+ "input": 47117,+ "cached_pct": 84,+ "output": 1449,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "Pi 1.0, MCP direct": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 31.6,+ "calls": 7.2,+ "input": 77038,+ "cached_pct": 79,+ "output": 1251,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 29.8,+ "calls": 6.7,+ "input": 60199,+ "cached_pct": 80,+ "output": 1214,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ }+ },+ "tasks": {+ "Pi 1.0 + codemode": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 33.8,+ "calls": 8.3,+ "input": 39962,+ "cached_pct": 82,+ "output": 1155,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 34.1,+ "calls": 8,+ "input": 29644,+ "cached_pct": 82,+ "output": 1391,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 2,+ "wall": 40.1,+ "calls": 9.3,+ "input": 84018,+ "cached_pct": 84,+ "output": 1650,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 30.2,+ "calls": 10.3,+ "input": 35832,+ "cached_pct": 84,+ "output": 1106,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 33.3,+ "calls": 7.7,+ "input": 48516,+ "cached_pct": 84,+ "output": 1288,+ "timeouts": 0+ }+ },+ "Pi 1.0": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 38.4,+ "calls": 9.3,+ "input": 50237,+ "cached_pct": 84,+ "output": 1432,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 38.2,+ "calls": 9,+ "input": 32536,+ "cached_pct": 85,+ "output": 1218,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 48.3,+ "calls": 9.3,+ "input": 76889,+ "cached_pct": 84,+ "output": 2224,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 29.8,+ "calls": 9.3,+ "input": 34437,+ "cached_pct": 87,+ "output": 932,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 34.1,+ "calls": 7,+ "input": 30025,+ "cached_pct": 80,+ "output": 1190,+ "timeouts": 0+ }+ },+ "Pi 1.0, MCP direct": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 32.0,+ "calls": 7.3,+ "input": 77615,+ "cached_pct": 79,+ "output": 1248,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 31.2,+ "calls": 7,+ "input": 76461,+ "cached_pct": 78,+ "output": 1255,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 37.7,+ "calls": 7.3,+ "input": 70618,+ "cached_pct": 81,+ "output": 1636,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 25.7,+ "calls": 7,+ "input": 74733,+ "cached_pct": 79,+ "output": 899,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 26.1,+ "calls": 5.7,+ "input": 35246,+ "cached_pct": 78,+ "output": 1106,+ "timeouts": 0+ }+ }+ }+ }+ },+ "gpt-6-astra": {+ "session_1_all_tasks": {+ "overall": {+ "graff v0.0.302.16": {+ "runs": 33,+ "pass": 33,+ "wall": 54.1,+ "calls": 4.5,+ "input": 49367,+ "cached_pct": 89,+ "output": 788,+ "timeouts": 0+ },+ "graff, rlm off": {+ "runs": 33,+ "pass": 33,+ "wall": 57.3,+ "calls": 4.6,+ "input": 52177,+ "cached_pct": 89,+ "output": 927,+ "timeouts": 0+ },+ "graff + ADR 0238": {+ "runs": 33,+ "pass": 33,+ "wall": 48.3,+ "calls": 3.8,+ "input": 44041,+ "cached_pct": 89,+ "output": 835,+ "timeouts": 0+ },+ "Pi 1.0 + codemode": {+ "runs": 33,+ "pass": 33,+ "wall": 41.7,+ "calls": 4.8,+ "input": 20941,+ "cached_pct": 54,+ "output": 647,+ "timeouts": 0+ },+ "Pi 1.0": {+ "runs": 33,+ "pass": 33,+ "wall": 48.1,+ "calls": 5.2,+ "input": 21887,+ "cached_pct": 57,+ "output": 704,+ "timeouts": 0+ }+ },+ "families": {+ "graff v0.0.302.16": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 31.5,+ "calls": 3.3,+ "input": 38882,+ "cached_pct": 91,+ "output": 523,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 51.4,+ "calls": 6.4,+ "input": 82622,+ "cached_pct": 89,+ "output": 823,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 42.2,+ "calls": 2.3,+ "input": 34864,+ "cached_pct": 88,+ "output": 756,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 15,+ "wall": 67.2,+ "calls": 4.1,+ "input": 36508,+ "cached_pct": 88,+ "output": 880,+ "timeouts": 0+ }+ },+ "graff, rlm off": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 38.5,+ "calls": 4,+ "input": 45116,+ "cached_pct": 92,+ "output": 674,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 55.8,+ "calls": 6.1,+ "input": 78441,+ "cached_pct": 90,+ "output": 917,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 48.4,+ "calls": 3,+ "input": 44539,+ "cached_pct": 83,+ "output": 892,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 15,+ "wall": 67.5,+ "calls": 4.2,+ "input": 40771,+ "cached_pct": 87,+ "output": 1042,+ "timeouts": 0+ }+ },+ "graff + ADR 0238": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 31.4,+ "calls": 3.5,+ "input": 41050,+ "cached_pct": 86,+ "output": 486,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 37.7,+ "calls": 3.9,+ "input": 45901,+ "cached_pct": 90,+ "output": 654,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 34.7,+ "calls": 2,+ "input": 29762,+ "cached_pct": 87,+ "output": 646,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 15,+ "wall": 64.2,+ "calls": 4.3,+ "input": 46977,+ "cached_pct": 90,+ "output": 1121,+ "timeouts": 0+ }+ },+ "Pi 1.0 + codemode": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 30.5,+ "calls": 5.2,+ "input": 26302,+ "cached_pct": 47,+ "output": 368,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 34.8,+ "calls": 5.6,+ "input": 33909,+ "cached_pct": 59,+ "output": 409,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 32.7,+ "calls": 2,+ "input": 9596,+ "cached_pct": 15,+ "output": 507,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 15,+ "wall": 52.1,+ "calls": 4.7,+ "input": 13285,+ "cached_pct": 59,+ "output": 930,+ "timeouts": 0+ }+ },+ "Pi 1.0": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 30.9,+ "calls": 5.2,+ "input": 26285,+ "cached_pct": 55,+ "output": 364,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 36.7,+ "calls": 5.9,+ "input": 38228,+ "cached_pct": 61,+ "output": 407,+ "timeouts": 0+ },+ "Commit digest": {+ "runs": 3,+ "pass": 3,+ "wall": 29.8,+ "calls": 2,+ "input": 8837,+ "cached_pct": 30,+ "output": 464,+ "timeouts": 0+ },+ "Multi-file coding": {+ "runs": 15,+ "pass": 15,+ "wall": 65.3,+ "calls": 5.3,+ "input": 12933,+ "cached_pct": 55,+ "output": 1066,+ "timeouts": 0+ }+ }+ },+ "tasks": {+ "graff v0.0.302.16": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 42.2,+ "calls": 2.3,+ "input": 34864,+ "cached_pct": 88,+ "output": 756,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 34.3,+ "calls": 4,+ "input": 46287,+ "cached_pct": 92,+ "output": 549,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 28.8,+ "calls": 2.7,+ "input": 31476,+ "cached_pct": 88,+ "output": 496,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 36.4,+ "calls": 3.3,+ "input": 36910,+ "cached_pct": 90,+ "output": 682,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 43.3,+ "calls": 3.3,+ "input": 37807,+ "cached_pct": 88,+ "output": 860,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 53.0,+ "calls": 4,+ "input": 47862,+ "cached_pct": 84,+ "output": 1027,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 114.8,+ "calls": 5,+ "input": 0,+ "cached_pct": 0,+ "output": 0,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 3,+ "wall": 88.5,+ "calls": 5,+ "input": 59962,+ "cached_pct": 91,+ "output": 1830,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 56.5,+ "calls": 6.7,+ "input": 88613,+ "cached_pct": 87,+ "output": 927,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 60.1,+ "calls": 8,+ "input": 106090,+ "cached_pct": 89,+ "output": 924,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 37.7,+ "calls": 4.7,+ "input": 53164,+ "cached_pct": 91,+ "output": 618,+ "timeouts": 0+ }+ },+ "graff, rlm off": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 48.4,+ "calls": 3,+ "input": 44539,+ "cached_pct": 83,+ "output": 892,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 38.5,+ "calls": 4,+ "input": 44602,+ "cached_pct": 93,+ "output": 677,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 38.5,+ "calls": 4,+ "input": 45630,+ "cached_pct": 91,+ "output": 671,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 42.3,+ "calls": 3.3,+ "input": 36529,+ "cached_pct": 86,+ "output": 822,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 49.9,+ "calls": 3.3,+ "input": 37349,+ "cached_pct": 85,+ "output": 1030,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 53.1,+ "calls": 4,+ "input": 47147,+ "cached_pct": 83,+ "output": 1023,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 105.3,+ "calls": 5.3,+ "input": 23579,+ "cached_pct": 88,+ "output": 473,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 3,+ "wall": 86.8,+ "calls": 5,+ "input": 59253,+ "cached_pct": 91,+ "output": 1860,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 52.9,+ "calls": 6,+ "input": 80263,+ "cached_pct": 85,+ "output": 880,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 76.3,+ "calls": 8,+ "input": 106288,+ "cached_pct": 91,+ "output": 1236,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 38.3,+ "calls": 4.3,+ "input": 48773,+ "cached_pct": 93,+ "output": 635,+ "timeouts": 0+ }+ },+ "graff + ADR 0238": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 34.7,+ "calls": 2,+ "input": 29762,+ "cached_pct": 87,+ "output": 646,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 31.4,+ "calls": 3.7,+ "input": 42609,+ "cached_pct": 82,+ "output": 486,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 31.4,+ "calls": 3.3,+ "input": 39490,+ "cached_pct": 90,+ "output": 487,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 40.1,+ "calls": 3.3,+ "input": 37168,+ "cached_pct": 89,+ "output": 723,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 47.9,+ "calls": 3.3,+ "input": 38019,+ "cached_pct": 86,+ "output": 969,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 52.5,+ "calls": 4,+ "input": 47858,+ "cached_pct": 93,+ "output": 1025,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 92.7,+ "calls": 5.7,+ "input": 48070,+ "cached_pct": 90,+ "output": 1079,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 3,+ "wall": 87.8,+ "calls": 5.3,+ "input": 63772,+ "cached_pct": 89,+ "output": 1810,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 38.8,+ "calls": 4,+ "input": 47587,+ "cached_pct": 92,+ "output": 671,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 36.3,+ "calls": 4,+ "input": 46797,+ "cached_pct": 90,+ "output": 616,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 38.1,+ "calls": 3.7,+ "input": 43320,+ "cached_pct": 88,+ "output": 675,+ "timeouts": 0+ }+ },+ "Pi 1.0 + codemode": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 32.7,+ "calls": 2,+ "input": 9596,+ "cached_pct": 15,+ "output": 507,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 30.8,+ "calls": 5.3,+ "input": 27067,+ "cached_pct": 56,+ "output": 383,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 30.3,+ "calls": 5,+ "input": 25536,+ "cached_pct": 38,+ "output": 353,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 25.1,+ "calls": 3,+ "input": 5728,+ "cached_pct": 45,+ "output": 363,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 37.8,+ "calls": 4.3,+ "input": 10647,+ "cached_pct": 55,+ "output": 579,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 50.6,+ "calls": 4.7,+ "input": 13777,+ "cached_pct": 51,+ "output": 878,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 63.6,+ "calls": 5.7,+ "input": 17203,+ "cached_pct": 63,+ "output": 1171,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 3,+ "wall": 83.2,+ "calls": 6,+ "input": 19069,+ "cached_pct": 67,+ "output": 1661,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 34.9,+ "calls": 5,+ "input": 30783,+ "cached_pct": 57,+ "output": 417,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 32.5,+ "calls": 5.3,+ "input": 31110,+ "cached_pct": 49,+ "output": 363,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 36.9,+ "calls": 6.3,+ "input": 39833,+ "cached_pct": 69,+ "output": 445,+ "timeouts": 0+ }+ },+ "Pi 1.0": {+ "commit-digest": {+ "runs": 3,+ "pass": 3,+ "wall": 29.8,+ "calls": 2,+ "input": 8837,+ "cached_pct": 30,+ "output": 464,+ "timeouts": 0+ },+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 28.9,+ "calls": 5,+ "input": 26403,+ "cached_pct": 58,+ "output": 366,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 32.9,+ "calls": 5.3,+ "input": 26166,+ "cached_pct": 52,+ "output": 361,+ "timeouts": 0+ },+ "solo-census": {+ "runs": 3,+ "pass": 3,+ "wall": 25.5,+ "calls": 3.3,+ "input": 5303,+ "cached_pct": 31,+ "output": 369,+ "timeouts": 0+ },+ "solo-coverage": {+ "runs": 3,+ "pass": 3,+ "wall": 39.4,+ "calls": 4.7,+ "input": 9892,+ "cached_pct": 63,+ "output": 639,+ "timeouts": 0+ },+ "solo-fix-three": {+ "runs": 3,+ "pass": 3,+ "wall": 93.6,+ "calls": 5.7,+ "input": 14255,+ "cached_pct": 51,+ "output": 951,+ "timeouts": 0+ },+ "solo-rename-api": {+ "runs": 3,+ "pass": 3,+ "wall": 84.5,+ "calls": 6.7,+ "input": 19078,+ "cached_pct": 55,+ "output": 1677,+ "timeouts": 0+ },+ "solo-sidecar": {+ "runs": 3,+ "pass": 3,+ "wall": 83.8,+ "calls": 6.3,+ "input": 16137,+ "cached_pct": 63,+ "output": 1692,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 35.1,+ "calls": 5,+ "input": 30766,+ "cached_pct": 42,+ "output": 408,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 31.9,+ "calls": 6,+ "input": 38528,+ "cached_pct": 68,+ "output": 385,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 43.2,+ "calls": 6.7,+ "input": 45391,+ "cached_pct": 68,+ "output": 427,+ "timeouts": 0+ }+ }+ }+ },+ "session_2_pi_codemode_mcp": {+ "overall": {+ "Pi 1.0 + codemode": {+ "runs": 15,+ "pass": 15,+ "wall": 41.2,+ "calls": 5.3,+ "input": 29917,+ "cached_pct": 50,+ "output": 388,+ "timeouts": 0+ },+ "Pi 1.0": {+ "runs": 15,+ "pass": 15,+ "wall": 35.6,+ "calls": 5.7,+ "input": 33670,+ "cached_pct": 55,+ "output": 402,+ "timeouts": 0+ },+ "Pi 1.0, MCP direct": {+ "runs": 15,+ "pass": 15,+ "wall": 40.6,+ "calls": 5.6,+ "input": 41667,+ "cached_pct": 63,+ "output": 477,+ "timeouts": 0+ }+ },+ "families": {+ "Pi 1.0 + codemode": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 32.3,+ "calls": 5,+ "input": 25566,+ "cached_pct": 44,+ "output": 363,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 47.1,+ "calls": 5.6,+ "input": 32817,+ "cached_pct": 54,+ "output": 405,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "Pi 1.0": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 34.2,+ "calls": 5.5,+ "input": 28689,+ "cached_pct": 49,+ "output": 384,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 36.5,+ "calls": 5.8,+ "input": 36991,+ "cached_pct": 58,+ "output": 413,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ },+ "Pi 1.0, MCP direct": {+ "MCP reports": {+ "runs": 6,+ "pass": 6,+ "wall": 55.8,+ "calls": 7.5,+ "input": 63423,+ "cached_pct": 68,+ "output": 673,+ "timeouts": 0+ },+ "MCP joins": {+ "runs": 9,+ "pass": 9,+ "wall": 30.5,+ "calls": 4.3,+ "input": 27163,+ "cached_pct": 56,+ "output": 346,+ "timeouts": 0+ },+ "Commit digest": null,+ "Multi-file coding": null+ }+ },+ "tasks": {+ "Pi 1.0 + codemode": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 31.4,+ "calls": 5,+ "input": 25541,+ "cached_pct": 53,+ "output": 356,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 33.3,+ "calls": 5,+ "input": 25590,+ "cached_pct": 36,+ "output": 370,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 33.2,+ "calls": 5,+ "input": 30796,+ "cached_pct": 48,+ "output": 418,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 68.9,+ "calls": 5.7,+ "input": 30468,+ "cached_pct": 60,+ "output": 376,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 39.1,+ "calls": 6,+ "input": 37188,+ "cached_pct": 53,+ "output": 420,+ "timeouts": 0+ }+ },+ "Pi 1.0": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 31.5,+ "calls": 5,+ "input": 26229,+ "cached_pct": 41,+ "output": 365,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 36.8,+ "calls": 6,+ "input": 31150,+ "cached_pct": 56,+ "output": 403,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 34.6,+ "calls": 5,+ "input": 30766,+ "cached_pct": 45,+ "output": 407,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 34.2,+ "calls": 6,+ "input": 38598,+ "cached_pct": 66,+ "output": 383,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 40.8,+ "calls": 6.3,+ "input": 41611,+ "cached_pct": 60,+ "output": 449,+ "timeouts": 0+ }+ },+ "Pi 1.0, MCP direct": {+ "linear-nohint": {+ "runs": 3,+ "pass": 3,+ "wall": 59.4,+ "calls": 9.7,+ "input": 87475,+ "cached_pct": 73,+ "output": 690,+ "timeouts": 0+ },+ "linear-nohint-warm": {+ "runs": 3,+ "pass": 3,+ "wall": 52.2,+ "calls": 5.3,+ "input": 39371,+ "cached_pct": 56,+ "output": 657,+ "timeouts": 0+ },+ "stale-triage": {+ "runs": 3,+ "pass": 3,+ "wall": 22.1,+ "calls": 4,+ "input": 22003,+ "cached_pct": 51,+ "output": 209,+ "timeouts": 0+ },+ "triage-rollup": {+ "runs": 3,+ "pass": 3,+ "wall": 39.3,+ "calls": 5,+ "input": 41136,+ "cached_pct": 61,+ "output": 457,+ "timeouts": 0+ },+ "xref-issues": {+ "runs": 3,+ "pass": 3,+ "wall": 30.1,+ "calls": 4,+ "input": 18349,+ "cached_pct": 53,+ "output": 373,+ "timeouts": 0+ }+ }+ }+ }+ }+}diff --git a/graff-evals/README.md b/graff-evals/README.mdindex e4cdeba3..b9493787 100644--- a/graff-evals/README.md+++ b/graff-evals/README.md@@ -26,6 +26,8 @@ harness under test spends model calls.| `bg` | Commands that outlast the 15s `-p` foreground wait, so each run must collect background results. For A/Bs of how the harness words and delivers them. Opt-in. || `subagents` | Work the prompt asks to hand to sub-agents in parallel: per-package bugfixes, a per-service log census, an API rename across three packages, an endpoint-coverage merge of two inventories, and a docs sidecar beside a fix. Fixtures are git repos; held-out checks. Opt-in. || `subagents-solo` | The same five tasks with the delegation wording removed: the counterfactual that says whether the children paid for themselves. Opt-in. |+| `mcp-hard` | Linear-shaped fixture MCP tasks that need more than ids and titles: cross-reference issue keys against `src/` (with decoy keys), roll issues up by priority, and join issues with git history by author date. Opt-in. |+| `aggregate` | Aggregate local data in one script: a per-directory commit digest over a pinned slice of this repository's history, fetched from GitHub in setup. Opt-in. |Terminal-Bench and other Harbor datasets run graff through the adapter in [harbor/](harbor/README.md).@@ -66,6 +68,15 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co# gateway, see the write-up). Results: evals/graff-vs-claude-code/RESULTS.md./run.py --suite core,mcp --harness graff-gateway-repl,claude-code,opencode2 --model <gateway model> -j 6 --interleave./run.py --suite core --harness pi-codegraff --model <gateway model> -j 6+# graff and Pi 1.0, each with and without its code mode (set CODEGRAFF_API_KEY for the gateway; PI1_BIN+# when `pi` on PATH is older than 1.0, PI1_MODELS / PI1_AUTH for Pi's provider and sign-in, PI1_THINKING to+# match graff's effort). pi1-direct is Pi with no codemode at all (MCP tools declared directly).+# Results: evals/graff-vs-pi-codemode/RESULTS.md+./run.py --suite mcp,mcp-hard,aggregate,subagents-solo --task linear-nohint --task linear-nohint-warm \+ --task xref-issues --task triage-rollup --task stale-triage --task commit-digest --task solo-fix-three \+ --task solo-census --task solo-rename-api --task solo-coverage --task solo-sidecar \+ --harness graff-gateway-repl,graff-gateway-repl-norlm,pi1-codemode,pi1 --model mimo-v2.6-pro --reps 3 -j 8 --interleave+./run.py --suite mcp,mcp-hard --harness pi1-codemode,pi1,pi1-direct --model mimo-v2.6-pro --reps 3 -j 8 --interleave```## Run itdiff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.jsonindex b70122b4..7a1fb6ba 100644--- a/graff-evals/harnesses.json+++ b/graff-evals/harnesses.json@@ -376,5 +376,9 @@"note": "Claude Code (`claude -p`, stream-json) with the task's .mcp.json as its only MCP config. Set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN for the endpoint; a chat-completions-only endpoint needs an Anthropic-format proxy in front (evals/graff-vs-claude-code). Pass --model."},"graff-deepseek": {"cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"], "stdin": "\u001b[200~{prompt}\u001b[201~\n", "answer": "stdout", "usage": "graff-stderr", "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"}, "model_prefix": "deepseek", "default_model": "deepseek-v4-pro", "note": "scripted graff repl on DeepSeek's own API (DEEPSEEK_API_KEY)"},- "deepseek-harness": {"cmd": ["python3", "{repo}/graff-evals/dsh_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "env": {"DSH_PERMISSION_MODE": "danger-full-access"}, "default_model": "deepseek-v4-pro", "note": "deepseek-harness (@deepseek-ai/dsh, DSH_BIN when dsh is not on PATH) headless on DeepSeek's own API; the model and the task's MCP servers go in a patch layer"}+ "deepseek-harness": {"cmd": ["python3", "{repo}/graff-evals/dsh_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "env": {"DSH_PERMISSION_MODE": "danger-full-access"}, "default_model": "deepseek-v4-pro", "note": "deepseek-harness (@deepseek-ai/dsh, DSH_BIN when dsh is not on PATH) headless on DeepSeek's own API; the model and the task's MCP servers go in a patch layer"},+ "graff-gateway-repl-norlm": {"cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"], "stdin": "\u001b[200~{prompt}\u001b[201~\n", "answer": "stdout", "usage": "graff-stderr", "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1", "GRAFF_RLM": "0"}, "model_prefix": "codegraff", "note": "graff-gateway-repl with rlm off (GRAFF_RLM=0: the structured-only catalog), the counterfactual for code-mode comparisons; pass --model"},+ "pi1": {"cmd": ["python3", "{repo}/graff-evals/pi1_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "model_prefix": "codegraff", "env": {"PI1_CODEMODE": "0"}, "note": "Pi 1.0 (@earendil-works/pi-coding-agent 1.x) headless with its default tools; MCP tools are reached through codemode, Pi's default MCP exposure. PI1_MODELS / PI1_AUTH copy a models.json / auth.json into the run's private ~/.pi/agent, PI1_THINKING passes --thinking (match the other arm's effort), PI1_BIN picks Pi 1.x; pass --model provider/model for a provider other than codegraff"},+ "pi1-codemode": {"cmd": ["python3", "{repo}/graff-evals/pi1_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "model_prefix": "codegraff", "env": {"PI1_CODEMODE": "1"}, "note": "Pi 1.0 with codemode added to the default tools (settings defaultTools +codemode, not --tools, which drops MCP tools). PI1_MODELS / PI1_AUTH copy a models.json / auth.json into the run's private ~/.pi/agent, PI1_THINKING passes --thinking (match the other arm's effort), PI1_BIN picks Pi 1.x; pass --model provider/model for a provider other than codegraff"},+ "pi1-direct": {"cmd": ["python3", "{repo}/graff-evals/pi1_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "model_prefix": "codegraff", "env": {"PI1_CODEMODE": "0", "PI1_MCP_EXPOSURE": "direct"}, "note": "Pi 1.0 default tools with every MCP server at exposure direct: no codemode at all, the counterfactual for codemode on MCP tasks. PI1_MODELS / PI1_AUTH copy a models.json / auth.json into the run's private ~/.pi/agent, PI1_THINKING passes --thinking (match the other arm's effort), PI1_BIN picks Pi 1.x; pass --model provider/model for a provider other than codegraff"}}diff --git a/graff-evals/hidden/check_rollup.py b/graff-evals/hidden/check_rollup.pynew file mode 100644index 00000000..a72c4fd7--- /dev/null+++ b/graff-evals/hidden/check_rollup.py@@ -0,0 +1,18 @@+import json, subprocess, sys, collections+# Ground truth straight from the fixture the task used.+reqs = [{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}},+ {"jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": {"name": "list_issues", "arguments": {}}}]+reqs += [{"jsonrpc": "2.0", "id": 10 + i, "method": "tools/call", "params": {"name": "list_comments", "arguments": {"id": f"ISS-{i}"}}} for i in range(1, 9)]+out = subprocess.run(["python3", "linear_fixture_mcp.py"], input="".join(json.dumps(r) + "\n" for r in reqs), capture_output=True, text=True, timeout=30).stdout+res = {json.loads(l)["id"]: json.loads(l)["result"] for l in out.splitlines() if l.strip()}+def body(r):+ data = json.loads(r["content"][0]["text"])+ return data if isinstance(data, list) else (data.get("issues") or data.get("comments") or data.get("nodes"))+issues = body(res[2])+comments = {f"ISS-{i}": len(body(res[10 + i])) for i in range(1, 9)}+exp = collections.defaultdict(lambda: {"issues": 0, "estimate": 0, "comments": 0})+for it in issues:+ e = exp[str(it["priority"])]+ e["issues"] += 1; e["estimate"] += it["estimate"]; e["comments"] += comments[it["id"]]+got = {str(k): {kk: int(vv) for kk, vv in v.items() if kk in ("issues", "estimate", "comments")} for k, v in json.load(open("rollup.json")).items()}+sys.exit(0 if got == dict(exp) else f"rollup.json {got} != {dict(exp)}")diff --git a/graff-evals/hidden/check_stale.py b/graff-evals/hidden/check_stale.pynew file mode 100644index 00000000..38c2c3e4--- /dev/null+++ b/graff-evals/hidden/check_stale.py@@ -0,0 +1,4 @@+import json, sys+got = json.load(open("stale.json"))+want = ["ENG-104", "ENG-105"]+sys.exit(0 if got == want else f"stale.json {got} != {want}")diff --git a/graff-evals/hidden/check_xref.py b/graff-evals/hidden/check_xref.pynew file mode 100644index 00000000..44d1894f--- /dev/null+++ b/graff-evals/hidden/check_xref.py@@ -0,0 +1,5 @@+import json, sys+expected = {"ENG-101": 3, "ENG-102": 0, "ENG-103": 2, "ENG-104": 6, "ENG-105": 1, "ENG-106": 0, "ENG-107": 3, "ENG-108": 2}+got = json.load(open('xref.json'))+got = {k: int(v) for k, v in got.items()}+sys.exit(0 if got == expected else f'xref.json {got} != {expected}')diff --git a/graff-evals/hidden/stale-gen.py b/graff-evals/hidden/stale-gen.pynew file mode 100644index 00000000..24d81707--- /dev/null+++ b/graff-evals/hidden/stale-gen.py@@ -0,0 +1,46 @@+"""Build the stale-triage history: a repo whose commit messages mention ENG keys.+Traps: a commit authored before the cutoff but committed after it, a longer key (ENG-1040),+a lowercase key (eng-105), and a key that appears only in a commit body."""+import os, subprocess+git = ["git", "-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null"]+subprocess.run(git + ["init", "-q", "-b", "main"], check=True)+with open(".git/info/exclude", "a") as f:+ f.write(".graff/\n.eval-*\n.pi/\n")+def commit(subject, adate, cdate=None, body=None, n=[0]):+ n[0] += 1+ with open("CHANGES.txt", "a") as f:+ f.write(f"{n[0]}: {subject}\n")+ env = dict(os.environ, GIT_AUTHOR_NAME="dev", GIT_AUTHOR_EMAIL="[email protected]",+ GIT_COMMITTER_NAME="dev", GIT_COMMITTER_EMAIL="[email protected]",+ GIT_AUTHOR_DATE=adate + " +0000", GIT_COMMITTER_DATE=(cdate or adate) + " +0000")+ subprocess.run(git + ["add", "-A"], check=True)+ msg = subject if body is None else subject + "\n\n" + body+ subprocess.run(git + ["commit", "-q", "-m", msg], env=env, check=True)+noise = ["Tidy imports", "Bump dependencies", "Refactor config loader", "Fix flaky clock test", "Rename helper",+ "Update README", "Lint fixes", "Speed up CI cache", "Drop dead code", "Log request ids"]+story = {+ 2: ("ENG-106: board filter keeps assignee", None, None),+ 4: ("Fix tax rounding (ENG-102)", None, None),+ 6: ("ENG-101: first pass at session resume", None, None),+ 9: ("ENG-104 retry 429s with backoff", None, None),+ 12: ("ENG-104: honour Retry-After", "2026-08-13T09:00:00", None),+ 15: ("Search index lag metrics (ENG-103)", None, None),+ 18: ("ENG-105 strip EXIF on upload", None, None),+ 21: ("ENG-105: strip GPS tags too", None, None),+ 26: ("Digest dedupe", None, "Key digests on the message id; fixes ENG-108."),+ 29: ("Follow-up on eng-105 review comments", None, None),+ 31: ("ENG-107 keep UTF-8 in CSV export", None, None),+ 35: ("ENG-101: resume keeps the session alive", None, None),+ 38: ("ENG-103 shard rebuild", None, None),+ 42: ("Migrate ENG-1040 tables", None, None),+ 45: ("Link the ENG-1010 runbook", None, None),+}+# 48 commits, two per half-day from 2026-08-01 to 2026-08-20.+for i in range(48):+ day, half = 1 + i * 20 // 48, i % 2+ adate = f"2026-08-{day:02d}T{9 + 6 * half:02d}:{i % 60:02d}:00"+ if i in story:+ subject, cdate, body = story[i]+ commit(subject, adate, cdate, body)+ else:+ commit(noise[i % len(noise)], adate)diff --git a/graff-evals/pi1_run.py b/graff-evals/pi1_run.pynew file mode 100755index 00000000..b301d78f--- /dev/null+++ b/graff-evals/pi1_run.py@@ -0,0 +1,74 @@+#!/usr/bin/env python3+"""Run Pi 1.0 headless (--mode json) on one eval task and report like graff does.++usage: pi1_run.py PROVIDER/MODEL PROMPT (cwd = the task sandbox)++Each run gets a private HOME whose ~/.pi/agent/mcp.json holds the task's+.mcp.json servers, so parallel runs never share config. PI1_CODEMODE=1 turns+codemode on through settings ("defaultTools": ["+codemode"]); --tools is not+used because it replaces the whole tool selection and drops MCP tools.+PI1_MCP_EXPOSURE sets every MCP server's exposure: Pi's default, codemode,+reaches MCP tools only from codemode scripts; direct declares them as ordinary+tools, so codemode never starts. PI1_MODELS copies a models.json (a provider+definition) and PI1_AUTH an auth.json (a sign-in) into that HOME, and+PI1_THINKING passes --thinking. PI1_BIN picks the binary when `pi` on PATH is+an older release. The final assistant text goes to stdout; a graff-format+[usage] line goes to stderr, summed over assistant messages (input counts+cache reads and writes, so it is the whole prompt).+"""+import json, os, shutil, subprocess, sys, tempfile++PI = os.environ.get("PI1_BIN", "pi")+model, prompt = sys.argv[1], sys.argv[2]+provider, _, name = model.partition("/")+codemode = os.environ.get("PI1_CODEMODE") == "1"+cwd = os.getcwd()++home = tempfile.mkdtemp(prefix="pi1-home-")+agent = os.path.join(home, ".pi", "agent")+os.makedirs(agent)+if os.path.exists(".mcp.json"):+ servers = {}+ for server, spec in json.load(open(".mcp.json")).get("mcpServers", {}).items():+ args = [os.path.join(cwd, a) if os.path.exists(os.path.join(cwd, a)) else a for a in spec.get("args", [])]+ servers[server] = {k: v for k, v in {"command": spec["command"], "args": args, "env": spec.get("env"), "cwd": cwd,+ "exposure": os.environ.get("PI1_MCP_EXPOSURE")}.items() if v}+ json.dump({"mcpServers": servers}, open(os.path.join(agent, "mcp.json"), "w"))+if codemode:+ json.dump({"defaultTools": ["+codemode"]}, open(os.path.join(agent, "settings.json"), "w"))+if os.environ.get("PI1_MODELS"):+ shutil.copy(os.environ["PI1_MODELS"], os.path.join(agent, "models.json"))+if os.environ.get("PI1_AUTH"):+ shutil.copy(os.environ["PI1_AUTH"], os.path.join(agent, "auth.json"))+thinking = ["--thinking", os.environ["PI1_THINKING"]] if os.environ.get("PI1_THINKING") else []+env = dict(os.environ, HOME=home)+try:+ r = subprocess.run([PI, "-p", "--mode", "json", "--provider", provider, "--model", name,+ "--no-session", *thinking, prompt], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env)+finally:+ shutil.rmtree(home, ignore_errors=True)++answer, calls = "", 0+tin = cached = writes = tout = 0+for line in r.stdout.splitlines():+ try:+ ev = json.loads(line)+ except ValueError:+ continue+ msg = ev.get("message") or {}+ if ev.get("type") == "message_end" and msg.get("role") == "assistant":+ calls += 1+ u = msg.get("usage") or {}+ tin += (u.get("input") or 0) + (u.get("cacheRead") or 0) + (u.get("cacheWrite") or 0)+ cached += u.get("cacheRead") or 0+ writes += u.get("cacheWrite") or 0+ tout += u.get("output") or 0+ text = "".join(c.get("text", "") for c in msg.get("content") or [] if c.get("type") == "text")+ if text.strip():+ answer = text+print(answer)+sys.stderr.write(r.stderr)+sys.stderr.write(f"\n[usage] {calls} api call(s) · {tin} in ({cached} cached, {writes} cache writes) + {tout} out tokens\n")+with open(".eval-pi1-events.jsonl", "w") as f:+ f.write(r.stdout)+sys.exit(r.returncode)diff --git a/graff-evals/tasks/90-commit-digest.json b/graff-evals/tasks/90-commit-digest.jsonnew file mode 100644index 00000000..f730e5d0--- /dev/null+++ b/graff-evals/tasks/90-commit-digest.json@@ -0,0 +1,16 @@+{+ "id": "commit-digest",+ "suite": "aggregate",+ "category": "aggregate",+ "prompt": "This directory is a git repository. Look at the non-merge commits whose author date, read in the author's own timezone (what `git log --date=short --format=%ad` prints), is 2026-09-25 or later. Count, for each top-level directory, how many of those commits touch at least one file under it; files at the repository root count under \"(root)\". Do the counting inside a single script so the raw git log never reaches you. Write the counts to digest.json as one JSON object mapping directory name to count, then reply with the five top-level directories with the most commits and one line on the main theme of the week.",+ "files": {},+ "setup": [+ "rm -rf .git && git init -q -b main .",+ "git fetch -q --shallow-since=2026-09-20 https://github.com/justrach/codegraff.git 5f9aa8225e43ba774dbb89064f2b3e16ec38c85c",+ "git checkout -q -B main FETCH_HEAD"+ ],+ "setup_timeout_s": 180,+ "strict_setup": true,+ "check": "python3 -c \"import json,subprocess,collections; out=subprocess.run(['git','log','--no-merges','--date=short','--name-only','--pretty=format:@@%ad'],capture_output=True,text=True).stdout; c=collections.Counter(); [c.update({(f.split('/')[0] if '/' in f else '(root)') for f in b.split(chr(10))[1:] if f.strip()}) for b in out.split('@@') if b.strip() and b.split(chr(10))[0].strip()>='2026-09-25']; got=json.load(open('digest.json')); exit(0 if {k:int(v) for k,v in got.items() if int(v)}==dict(c) else 1)\"",+ "timeout_s": 300+}diff --git a/graff-evals/tasks/91-xref-issues.json b/graff-evals/tasks/91-xref-issues.jsonnew file mode 100644index 00000000..d43dc49e--- /dev/null+++ b/graff-evals/tasks/91-xref-issues.json@@ -0,0 +1,29 @@+{+ "id": "xref-issues",+ "suite": "mcp-hard",+ "category": "mcp-aggregate",+ "prompt": "This workspace has a Linear-shaped MCP server named linear. For every issue it lists, count how many files under src/ contain that issue's key (its identifier, such as ENG-101) as an exact, case-sensitive match that is not part of a longer key such as ENG-1010. A file counts once per key. Write xref.json as one object mapping every issue key to its count, including keys with a count of 0, then reply with the keys that no file under src/ mentions.",+ "files": {+ ".mcp.json": "{\"mcpServers\":{\"linear\":{\"command\":\"python3\",\"args\":[\"linear_fixture_mcp.py\"]}}}\n",+ "src/auth/session.py": "# ENG-101: resume drops the session\n# see ENG-101 again for the retry\nRESUME = 'ENG-104 adjacent'\n",+ "src/auth/tokens.py": "# Token refresh, tracked in ENG-101\n",+ "src/billing/tax.py": "# Rounding fix (ENG-104); unrelated migration ENG-1040\n",+ "src/billing/invoice.py": "INVOICE_NOTE = 'ENG-104'\n",+ "src/search/index.py": "# ENG-103 index rebuild; ENG-107 shard\n",+ "src/search/query.py": "# ENG-107: query timeout\n",+ "src/mailer/send.py": "# ENG-108 bounce handling; old note: eng-105\n",+ "src/mailer/queue.py": "# backlog from ENG-108\n",+ "src/api/routes.py": "# ENG-101 login route, ENG-104 tax route, ENG-1010 is a different ticket\n",+ "src/api/errors.py": "# error mapping for ENG-105\n",+ "src/util/retry.py": "# retries: ENG-107, ENG-104\n",+ "src/util/clock.py": "# clock skew (ENG-104)\n",+ "src/util/log.py": "# ENG-1040 migration only\n",+ "src/README.md": "Search work lives under ENG-103.\n",+ "docs/notes.md": "Planning: ENG-102 and ENG-106 are not in code yet.\n"+ },+ "setup": [+ "python3 -c \"import pathlib,shutil; p=pathlib.Path('.').resolve(); root=next(x for x in p.parents if (x/'scripts'/'linear_fixture_mcp.py').exists()); shutil.copy(root/'scripts'/'linear_fixture_mcp.py','.')\""+ ],+ "check": "python3 \"$TASK_ROOT/hidden/check_xref.py\"",+ "timeout_s": 300+}diff --git a/graff-evals/tasks/92-triage-rollup.json b/graff-evals/tasks/92-triage-rollup.jsonnew file mode 100644index 00000000..445299f7--- /dev/null+++ b/graff-evals/tasks/92-triage-rollup.json@@ -0,0 +1,14 @@+{+ "id": "triage-rollup",+ "suite": "mcp-hard",+ "category": "mcp-aggregate",+ "prompt": "This workspace has a Linear-shaped MCP server named linear. Group its issues by priority. For each priority, count the issues, sum their estimates, and count every comment on those issues. Write rollup.json as one object keyed by priority, {\"1\": {\"issues\": n, \"estimate\": e, \"comments\": c}, ...}, then reply with the priority whose issues have the most comments.",+ "files": {+ ".mcp.json": "{\"mcpServers\":{\"linear\":{\"command\":\"python3\",\"args\":[\"linear_fixture_mcp.py\"]}}}\n"+ },+ "setup": [+ "python3 -c \"import pathlib,shutil; p=pathlib.Path('.').resolve(); root=next(x for x in p.parents if (x/'scripts'/'linear_fixture_mcp.py').exists()); shutil.copy(root/'scripts'/'linear_fixture_mcp.py','.')\""+ ],+ "check": "python3 \"$TASK_ROOT/hidden/check_rollup.py\"",+ "timeout_s": 300+}diff --git a/graff-evals/tasks/93-stale-triage.json b/graff-evals/tasks/93-stale-triage.jsonnew file mode 100644index 00000000..d4eb920b--- /dev/null+++ b/graff-evals/tasks/93-stale-triage.json@@ -0,0 +1,15 @@+{+ "id": "stale-triage",+ "suite": "mcp-hard",+ "category": "mcp-aggregate",+ "prompt": "This workspace is a git repository, and it has a Linear-shaped MCP server named linear. An issue is stale when its priority is 1 or 2 and no commit authored on or after 2026-08-10 (by author date) mentions its key anywhere in the commit message, subject or body. A mention is the issue's identifier, such as ENG-101, as an exact, case-sensitive match that is not part of a longer key such as ENG-1010. Write stale.json as a sorted JSON list of the stale issue keys, then reply with how many there are.",+ "files": {+ ".mcp.json": "{\"mcpServers\":{\"linear\":{\"command\":\"python3\",\"args\":[\"linear_fixture_mcp.py\"]}}}\n"+ },+ "setup": [+ "python3 -c \"import pathlib,shutil; p=pathlib.Path('.').resolve(); root=next(x for x in p.parents if (x/'scripts'/'linear_fixture_mcp.py').exists()); shutil.copy(root/'scripts'/'linear_fixture_mcp.py','.')\"",+ "python3 \"$TASK_ROOT/hidden/stale-gen.py\""+ ],+ "check": "python3 \"$TASK_ROOT/hidden/check_stale.py\"",+ "timeout_s": 300+}