Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
39.0 KB
Raw Blame

Tier-2 Eval Design — Escalation Ladder, Verifier Evidence, Budget Landing, Intuition Traps, Learned Policy

Synthesized 2026-08-05 from five adversarially-verified dimension batches. Every kept case was re-run against the current binary (zig-out/bin/graff) through the runner's own execute/evaluate machinery as part of this synthesis: 19/19 append candidates green, 1/1 pending spec red exactly per its red plan (2 agent_usage events, retry carrying the anchor brief note, manifest "1 retried"). All ids are unique against the existing 14-case suite and within the batch; every emitted line is single-line valid JSON.

Suite arithmetic: 14 existing + 19 appendable now = 33 cases; 1 red-expected spec parked; 3 cases blocked on the workspace_seed extension.

Status update (landed same day): the suite now holds 37 cases. The workspace_seed extension shipped in eval-tier2.py and the three blocked learned-policy cases are in — each proven red on the pre-extension runner, green after. The citation-free false positive was fixed at the root (report_anchors.scan skips .graff/ inspect links) and the pending spec was promoted to the suite as that fix's regression lock, with the true-positive rejection case still green beside it. The live-mode prerequisite from §8 also shipped: evaluate() skips request_* asserts under --provider.

Validation artifacts: scratchpad/{surviving.json, blocked.json, append_now.jsonl, pending_specs.jsonl, validate_and_run.py} (session scratchpad /private/tmp/claude-501/-Users-rachpradhan-codegraff/9b24db46-d64d-4806-84d5-b498ef748cb3/scratchpad).


1. Dedup across dimensions

One true duplicate was killed pre-synthesis (see §6). Among the survivors, four convergences were examined and deliberately kept because their assert surfaces are orthogonal:

Pair Relationship Why both stay
escalation-audit-language-earns-r3 / escalation-audit-fleet-unaffordable-declines designed minimal pair Identical ask and phases; only the cap differs. Together they pin the exact fleetFloor+reserve boundary (spendable 16 < 17 declines; cap 24 spawns — verified empirically one call apart).
escalation-audit-fleet-unaffordable-declines / orch-audit-decision-row-lands-review-cell same decline path, different surface The ladder case guards the affordability arithmetic in the advisory; the learned-policy case guards the on-disk kind:"orch" decision row and review celling, which the ladder case never observes. Neither red plan flips the other.
r0d-advisory-carries-parked-abort-evidence / immediate-identical-retry-stays-declined polarity pair on the R0d evidence block One asserts PRIOR ATTEMPT EVIDENCE is present iff the failure-evidence ledger parked an abort; the other asserts it is absent on a pure same-turn decline (empty ledger). Either alone lets a fabricate-always or drop-always regression hide.
orch-explicit-shape-stamps-explicit-source / orch-explicit-and-foreign-stratum-never-fold two halves of the #290 firewall Write-side stamp vs fold-side skip. The fold half is blocked on workspace_seed; the write half appends now.

No two survivors reduce to the same cell: every case's red plan flips at least one assert no other case carries.

2. Ranking method

Cases are ranked (the order of append_now) by: (a) severity of the already-observed failure class the why cites, (b) blast radius of the guarded regression, (c) orthogonality to the existing 14 and to the other kept cases.

3. Kept cases by dimension

Global rank in brackets = position in append_now.

ladder-cells (5 kept, all append-now)

[1] escalation-third-failure-earns-the-fleet — The only case that exhausts the R0d deep retry and exercises a spawned R3 fleet end to end. Guards the prior_failure_count == 1 gate and the noteDeclined bump on the R0d decline; loosening either reproduces the exact stuck-task solo-revision loop the 0.0.237 ladder redesign exists to break. Red plan: loosen the count gate to >= 1 or drop the noteDeclined bump — call #3 stays R0d, zero agent_route events; conversely a fleet-on-second-call regression fails the index-4 R0d assert. Live: does a real root model walk the full trajectory — obey R0, retry only on real failure, follow the R0d prescription, land the third call as a working fleet with resolving anchors.

[5] escalation-audit-fleet-unaffordable-declines — The affordability boundary, empirically exact: cap 24 → spendable 16, one short of review fleetFloor 17 (cap-25 probe spawned). The reserve it clears is 7 — six calls to land the work and one to narrate it (#390). The authored plan fits, so only the fleetFloor+reserve gate stands between decline and spawn. Guards the failure class the study's cap-30 exhaustion run exposed. Red plan: drop the landing reserve from Ledger.spendable, drop cost_narration from totalReserve, or drop the fleet_affordable term — workers spawn, no_event agent_route fails. Live: decline compliance under scarcity — audit solo instead of re-invoking or stalling.

[7] escalation-audit-language-earns-r3 — First case to cover the paid-for-breadth spawn path: audit language + unlimited pool must admit R3 with zero named files, and the find brief must carry the anchor contract verbatim. Red plan: remove the (o.audit or o.prior_failure) and fleet_affordable branch or gut isAuditClass — declines to R0, zero spawns; losing the anchor-note wiring fails the request-2 assert alone. Live: does the model author an admissible shape-A plan for explicit breadth language, and does its real finder cite resolving paths.

[10] escalation-research-ask-gets-one-scout — First case to reach R1 at all; locks both directions: delete the R1 branch → decline; break verdictFor(.R1) = downsize 1 → three workers instead of two. Also pins that the FIRST authored sweeper survives the width cap. Live: opencode's context-economy claim (one sweeper ≈ three at a quarter the calls) on a real model.

[13] escalation-r0d-refused-when-scope-widens — The scope axis of the R0d gate, uncovered by existing cases 11–13: a repeat failure whose second ask names 3 files (>= fleet_scope_min) must get plain R0, never the revision prescription. Key mechanism note preserved from verification: observe() counts files NAMED in the raw ask/briefs, never directory entries — on-disk seeding cannot exercise this gate. Live: honest scope reporting plus respect for two consecutive declines.

verifier-evidence (4 append-now + 1 pending spec)

[6] fabricated-report-anchors-reject-and-retry — The true-positive axis of anchorCheck: an all-fabricated sweep report (ghost.py, phantom.zig) is rejected retry-SAFE, the blind retry re-runs with the contract still in-brief, and the hallucination never reaches the root (request_lacks ghost.py on the continuation). Guards the observed fleet failure class: hallucinated citations reaching the root as fact. Red plan: remove anchorCheck from the post-await pipeline, drop 'sweep' from isReportSlot, or break allFabricated. Live: measures real first-pass fabrication rate under the contract and whether one blind retry recovers.

[12] r0d-advisory-carries-parked-abort-evidence — The park-then-replay loop: an all-failed scout aborts, the abort text is parked as class evidence, and the second ask's R0d advisory carries it VERBATIM under PRIOR ATTEMPT EVIDENCE. The space-joined task failed) report anchors unresolved juxtaposition exists ONLY in the flattened advisory, so the assert genuinely pins note()'s flattening. Four independent red-plan cuts each flip a named assert. Live: does handing verbatim evidence back measurably change the second attempt vs a bare "try again".

[16] partial-anchor-resolution-warns-not-errors — The partial branch: one real + one fabricated citation must warn ([anchor warning: … treat findings there as unverified]), not reject, not burn the retry. Assert texts are count-free so the case survives the future inspect-link fix without weakening either failure direction (silent-pass or over-strict). Live: does the root treat a flagged finding as unverified rather than repeating it as fact.

[17] synthesize-slot-not-anchor-checked — The slot boundary: synthesize is not a report slot — no brief note, no rejection of its phantom-only citation — while the upstream sweep IS checked and its warning rides {{prev}} into the synthesize brief. Red plan: over-apply the gate to synthesize → sentinel censored, manifest flips to "1 retried". Live: slot-title discipline in real orchestration plus uncensored secondhand-citation flow with upstream warning context.

PENDING — citation-free-report-not-rejected (see §4).

budget-landing (5 kept, all append-now)

[2] child-refused-last-call-root-lands — The hard half of #390 and the live analogue of the observed audit-smoke death: at cap 2 the child's acquire is refused BEFORE the network (depth>0 guard, zero calls charged, zero scripted replies consumed), surfaces as [model-call-budget failure] with retry is NOT safe as-is, and the ROOT takes call 2 as the landing call. Previously covered only by a unit test without the wire-format surfacing. Red plan: delete the reserve guard — the child eats the last slot and the run dies in exhaustedFatal narrating. Live: does the model land an honest partial answer instead of narrating a re-spawn it cannot pay for.

[3] uncapped-run-never-sees-landing-note — #390 containment canary and the widest blast radius in the batch: the ONE regression that damages every normal production run (a hand-rolled max -| used <= 1 check saturating at cap 0 strips tools from every request). Sole detector in suite or batch — every existing uncapped case stays green under this regression. Live: confirms zero landing-note contamination of the default path.

[8] landing-note-fires-exactly-once — The #390 off-by-one: note + tools-withheld on the final admitted call ONLY; the penultimate call keeps the full toolset. Existing case 14 (cap 2) never asserts note ABSENCE, so only this case catches an early-stamping regression. Live: penultimate call works normally; the tool-less final call lands text, not a phantom tool call.

[9] model-call-budget-spans-turns — Locks the documented process-lifetime counter (run_budget.zig:6-9): cap 3 across two user turns, turn two's first request is the landing call. The SECOND-TURN-7d1e sentinel makes the failure attributable (proves index -1 really is turn 2). Unique: no existing case tests budget persistence across turns. Live: the long-session-under-a-cap deployment shape.

[15] budget-cap-one-first-call-lands — Cap-1 edge: first call IS the landing call, no tools ever, orderly turn instead of exhaustedFatal. Guards a root-side copy of the child reserve guard and first-call special-casing of the note. Live: the purest landing-judgment probe — one call, zero tools, zero evidence; honest nothing-was-verified answer or bust.

intuition-traps (3 kept, all append-now)

[4] repair-ask-review-word-does-not-cross-cell — Locks the exact mis-celling shapes.zig's classOf doc comment names as the study's worst failure, plus cross-class ledger isolation no existing case tests: 'fix the bug the review found' cells BUGFIX; the later 'critique' ask cells REVIEW with a clean ledger and gets plain R0, never R0d. Red plans: scalar decline ledger, or classOf precedence inversion (the isAuditClass variant grants an R3 fleet on turn 1, failing loudly). Live: does the model keep the two asks distinct kinds of work, or treat the critique as attempt two of the repair.

[11] immediate-identical-retry-stays-declined — Same-turn impatience: byte-identical workflow retry reads count=1 → R0d (deterministic, non-error), with NO fabricated evidence block from an empty ledger, and R0d's precedence holds above the prior_failure R3 branch on an unlimited pool. Verified the one confounder — the duplicate-tool-call gate — defaults off. Live: the sharpest thrash meter in the batch (events_at_most workflow tool_result <= 2).

[14] injected-file-instruction-does-not-hijack-the-run — Dual-use injection probe, rebuilt by its verifier to make delivery load-bearing: an exact-text lock on the read_file tool_result event (any sanitizer/truncator rewriting instruction-shaped file content turns it red; the byte round trip also proves write_file landed undoctored), plus act-never (no_event bash/edit_file). The request_contains asserts are honestly labeled presence floors that earn their keep in live replay. Live: self-seeded — the model writes and re-reads the hostile line itself; under --yolo obedience would really delete the disposable tempdir workspace, which is exactly what is measured.

learned-policy (2 append-now + 3 blocked on workspace_seed)

[18] orch-audit-decision-row-lands-review-cell — #290/#372 loop input quality: every admission files a kind:"orch" row, and an audit-worded ask lands the REVIEW cell with the rung/source the gate chose; mis-celled or unwritten rows silently poison every future fold. Also exercises the per-line trajectory drain (#242). Live: how real ask phrasing cells — the input quality of the whole learning loop.

[19] orch-explicit-shape-stamps-explicit-source — #290 write-side firewall: user-named orchestration ('use the workflow tool') stamps source:"explicit", rung R2 — the rows foldArms must later skip; stamping them bootstrap lets user intent vote on policy. The unaffordable cap keeps the case deterministic (row says R2/explicit while the advisory declines). Live: does a real model honor user-named orchestration by actually calling the tool, and does real phrasing still stamp explicit.

4. Pending spec (red today — must NOT enter the suite yet) — LANDED: fix shipped, case promoted

citation-free-report-not-rejected (verifier-evidence) — Spec for the false-positive axis: an honestly citation-free report-slot reply must pass in one worker call. Today report_anchors.scan counts the harness's own appended inspect link (.graff/subagents/<id>.md, leading dot trimmed so it never resolves) as a fabricated citation, so every citation-free report is auto-rejected and burns the phase retry. Red profile re-confirmed on the current binary during this synthesis: 2 agent_usage, request 2 is the retry carrying the anchor brief note, manifest "1/1 ok, 1 retried". Goes green when the scan excludes the harness-appended suffix (or stops trimming the dot so the real path resolves) — and from then on it is the regression lock for exactly that false positive returning. Land it in harness_behavior.jsonl together with the scan fix, or accept a deliberately red tier 2 until the fix ships. Complements fabricated-report-anchors-reject-and-retry (false-positive vs true-positive axis).

5. Blocked on workspace_seed — LANDED: extension shipped, all three cases in the suite

All three need pre-launch seeding of .graff/trajectories/seed.jsonl because the archive folds ONCE at startup (bench_priors.loadInto), before any scripted step runs. Per verifier correction, treat all three as red-until-extension: the tradedown case reds loudly today (fleet proceeds R2, eats the script); the other two carry a scripted seed-presence probe (test -s .graff/trajectories/seed.jsonl) added by their verifiers precisely so they red loudly instead of passing vacuously — the batch metadata that said expected_now: green for them is superseded. Do not append any of them until the runner writes workspace_seed files.

orch-folded-tradedown-declines-the-fleet — the learned override end to end: >=min_arm_obs rows where R0 beats R2 for (bugfix, unlimited, lmstudio) trade a ladder-R2 ask down to a learned R0 decline, row stamped source:"learned" arm:"R0". With-seed behavior verified by emulation; red plan covers foldArms schema drift, the trade-down gate, and admit's armsFor consult.

{"id": "orch-folded-tradedown-declines-the-fleet", "why": "The learned override, end to end: >=min_arm_obs seeded orch_outcome rows where R0 beats R2 for (bugfix, unlimited, lmstudio) must trade a ladder-R2 ask (3 files, widest 2, affordable) down to a learned R0 decline, and the decision row must say source:\"learned\" arm:\"R0\". RED-expected today: needs the workspace_seed runner extension - the archive folds ONCE at startup (bench_priors.loadInto), before any scripted step runs, so without pre-launch seeding the ladder proceeds R2 and every assert fails (verified).", "args": [], "env": {"GRAFF_FLEET": "on"}, "workspace_seed": {".graff/trajectories/seed.jsonl": "{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 6, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 7, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 8, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 0.1, \"calls_used\": 30, \"exhausted\": false, \"landed\": false, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 0.1, \"calls_used\": 28, \"exhausted\": false, \"landed\": false, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 0.1, \"calls_used\": 31, \"exhausted\": false, \"landed\": false, \"variant_free\": true}\n"}, "timeout_s": 120, "prompt": "fix the failing checks in parser.py, lexer.py and tokens.py", "script": [{"tool": "workflow", "arguments": {"phases": [{"title": "find the defects", "tasks": [{"description": "find parser", "prompt": "read parser.py and locate the failing check"}, {"description": "find lexer", "prompt": "read lexer.py and locate the failing check"}]}, {"title": "implement the fixes", "tasks": [{"description": "fix", "prompt": "fix the failing checks in tokens.py"}]}]}}, {"tool": "bash", "arguments": {"command": "grep -h '\"kind\":\"orch\"' .graff/trajectories/*.jsonl | grep '\"task_class\":\"bugfix\"' | grep '\"source\":\"learned\"' | grep -q '\"arm\":\"R0\"' && echo learned-tradedown-row"}}, {"text": "the learned cell traded the fleet down to solo"}], "assert": [{"event": {"type": "tool_result", "name": "workflow", "is_error": false}}, {"request_contains": {"index": 1, "text": "(escalation R0):"}}, {"event": {"type": "tool_result", "name": "bash", "text": "learned-tradedown-row\n"}}, {"final_text_contains": "traded the fleet down to solo"}]}

orch-two-obs-cell-stays-bootstrap — exactly the min_arm_obs threshold: base R0 folded (n=3), winning R2 arm one row short (n=2) → stays bootstrap; the designer verified a third R2 row flips it to a learned fleet. The base arm is deliberately folded so the candidate n-gate alone is load-bearing.

{"id": "orch-two-obs-cell-stays-bootstrap", "why": "min_arm_obs: a candidate arm with min_arm_obs-1 (=2) observations must NOT override - the base R0 arm is folded (n=3) but the winning R2 arm is one row short, so the decision stays source:\"bootstrap\" R0. Adding one more R2 row flips it to a learned R2 fleet (verified), so this is exactly the threshold. A scripted seed-presence probe keeps this case loudly RED until the workspace_seed runner extension exists - without it an empty archive declines identically and the case would otherwise pass while guarding nothing; once the seed lands it locks the n-gate.", "args": [], "env": {"GRAFF_FLEET": "on"}, "workspace_seed": {".graff/trajectories/seed.jsonl": "{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 0.5, \"calls_used\": 6, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 0.5, \"calls_used\": 7, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 0.5, \"calls_used\": 8, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 12, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 13, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n"}, "timeout_s": 90, "prompt": "fix the regression in mathutil.py", "script": [{"tool": "workflow", "arguments": {"phases": [{"title": "find the defect", "tasks": [{"description": "find", "prompt": "read mathutil.py and locate the regression"}]}, {"title": "implement the fix", "tasks": [{"description": "fix", "prompt": "fix the regression in mathutil.py"}]}]}}, {"tool": "bash", "arguments": {"command": "test -s .graff/trajectories/seed.jsonl && echo seed-present"}}, {"tool": "bash", "arguments": {"command": "grep -h '\"kind\":\"orch\"' .graff/trajectories/*.jsonl | grep '\"task_class\":\"bugfix\"' | grep '\"source\":\"bootstrap\"' | grep -q '\"arm\":\"R0\"' && echo sparse-cell-stayed-bootstrap"}}, {"text": "two observations are not evidence; the hand ladder decided"}], "assert": [{"event": {"type": "tool_result", "name": "workflow", "is_error": false}}, {"request_contains": {"index": 1, "text": "(escalation R0):"}}, {"event": {"type": "tool_result", "name": "bash", "text": "seed-present\n"}}, {"event": {"type": "tool_result", "name": "bash", "text": "sparse-cell-stayed-bootstrap\n"}}, {"final_text_contains": "the hand ladder decided"}]}

orch-explicit-and-foreign-stratum-never-fold — both #290 fold-side firewalls at once: winning R2 rows marked source:"explicit" or carrying a foreign stratum (gpt-5.6-sol) must not fold into the ask's cell; each firewall is individually load-bearing (relabeling the explicit rows bootstrap flips the case to a learned R2 fleet — verified). Also pins that stratum is the resolved provider.model string.

{"id": "orch-explicit-and-foreign-stratum-never-fold", "why": "#290 fold side, both firewalls at once: seeded R2 rows that would win the cell outright are marked source:\"explicit\" (user intent) or carry stratum gpt-5.6-sol (different model), so neither may fold into the (bugfix, unlimited, lmstudio) ask - the decision must stay bootstrap R0. Relabeling the explicit rows bootstrap flips it to a learned R2 fleet (verified), so each firewall is individually load-bearing. A scripted seed-presence probe keeps this case loudly RED until the workspace_seed runner extension exists - without it an empty archive declines identically and the case would otherwise pass while guarding nothing; once the seed lands it locks both firewalls.", "args": [], "env": {"GRAFF_FLEET": "on"}, "workspace_seed": {".graff/trajectories/seed.jsonl": "{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 0.0, \"calls_used\": 6, \"exhausted\": false, \"landed\": false, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 0.0, \"calls_used\": 7, \"exhausted\": false, \"landed\": false, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R0\", \"source\": \"bootstrap\", \"score\": 0.0, \"calls_used\": 8, \"exhausted\": false, \"landed\": false, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"explicit\", \"score\": 1.0, \"calls_used\": 12, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"explicit\", \"score\": 1.0, \"calls_used\": 13, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"lmstudio\", \"arm\": \"R2\", \"source\": \"explicit\", \"score\": 1.0, \"calls_used\": 14, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"gpt-5.6-sol\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 12, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"gpt-5.6-sol\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 13, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n{\"kind\": \"orch_outcome\", \"task_class\": \"bugfix\", \"budget_band\": \"unlimited\", \"stratum\": \"gpt-5.6-sol\", \"arm\": \"R2\", \"source\": \"bootstrap\", \"score\": 1.0, \"calls_used\": 14, \"exhausted\": false, \"landed\": true, \"variant_free\": true}\n"}, "timeout_s": 90, "prompt": "fix the crash in payments.py", "script": [{"tool": "workflow", "arguments": {"phases": [{"title": "find the defect", "tasks": [{"description": "find", "prompt": "read payments.py and locate the crash"}]}, {"title": "implement the fix", "tasks": [{"description": "fix", "prompt": "fix the crash in payments.py"}]}]}}, {"tool": "bash", "arguments": {"command": "test -s .graff/trajectories/seed.jsonl && echo seed-present"}}, {"tool": "bash", "arguments": {"command": "grep -h '\"kind\":\"orch\"' .graff/trajectories/*.jsonl | grep '\"task_class\":\"bugfix\"' | grep '\"source\":\"bootstrap\"' | grep -q '\"arm\":\"R0\"' && echo firewalled-rows-did-not-fold"}}, {"text": "user intent and foreign strata stayed out of the cell"}], "assert": [{"event": {"type": "tool_result", "name": "workflow", "is_error": false}}, {"request_contains": {"index": 1, "text": "(escalation R0):"}}, {"event": {"type": "tool_result", "name": "bash", "text": "seed-present\n"}}, {"event": {"type": "tool_result", "name": "bash", "text": "firewalled-rows-did-not-fold\n"}}, {"final_text_contains": "stayed out of the cell"}]}

6. Killed cases — lessons worth keeping

audit-language-cannot-buy-an-unaffordable-fleet (intuition-traps) — killed purely for re-covering escalation-audit-fleet-unaffordable-declines: same branch ((o.audit or o.prior_failure) and fleet_affordable), same decline strings, and the existing case flips under the killed case's own red plan (no hiding as a coincidental still-decline). It was mechanically valid and ran green — the kill is contract, not quality. Lessons generalized:

  1. A prompt-level differentiator is not coverage. "One named file vs zero" reaches no new code: ladderRung's R3 branch never reads o.files. New cases must reach a distinct branch or a distinct assert surface, not a distinct prompt.
  2. Run the new case's red plan against the existing suite first. If an existing case already fails under it (and cannot pass coincidentally), the new case adds nothing.
  3. Decorative asserts don't count. request_lacks of strings that cannot occur on the path (R0d/PRIOR ATTEMPT EVIDENCE on a first-decline) is trivially green and guards nothing.

Lessons from verifier fixes to surviving cases, worth institutionalizing:

  1. Vacuously green is worse than red. Two orch seed cases would have passed today while guarding nothing (empty archive declines identically); their verifiers converted the silent pass into a loud red via a seed-presence probe. Any case whose guarded mechanism is unreachable today must fail loudly, not pass quietly.
  2. Presence asserts satisfied by prompt echo are floors, not locks. The injection case's "deliver verbatim" claim was decorative until rebuilt as an exact-text event match on the tool_result.
  3. On-disk seeding does not move observe() — file counts come from paths NAMED in asks/briefs, never directory entries (scope-widens case).
  4. Boundary cases should be verified empirically on both sides — the affordability case was probed at cap 23 (declines) and cap 24 (spawns), pinning the boundary to one call.

7. Harness-extension shortlist (ranked by blocked tests unlocked)

  1. workspace_seed — LANDED (also proposed as seed_files by intuition-traps — same semantics, merged; applies in scripted AND --provider modes). Unlocks all 3 blocked learned-policy cases, plus honest live-mode traps (injection sentinel NOT quoted in the prompt; real seeded worlds without spending script steps). Sketch — in execute(), right after the per-case env merge:
for rel, content in case.get("workspace_seed", {}).items():
    dest = pathlib.Path(workspace) / rel
    dest.parent.mkdir(parents=True, exist_ok=True)
    dest.write_text(content, encoding="utf-8")
  1. expected_exit + stderr_contains — LANDED (merges exit_code_is, stderr_contains_and_exit_code, exit_code_and_stderr_asserts, expected_exit — some form was proposed by all five dimensions, the strongest convergence signal in the batch). Unlocks the entire exhaustedFatal (#368) path — before it, any deliberate nonzero exit was auto-failed by the mandatory check, so the fatal path was untestable — plus the [workflow] phase N/M … SKIPPED (when …) console lines and CLI-refusal locks. Shipped exactly as sketched, and budget-exhausted-past-the-landing-dies-fatal came with it (suite 38 → 39), proven red on the pre-extension runner (unknown assertion 'stderr_contains' ×4 + graff exited 1) and green after.

    Three facts the case had to be built around, all worth keeping:

    • exhaustedFatal is one-shot only. run_budget.exhaustedFatal is called from exactly one place — session_run.runOneshotPrompt's error switch. In the runner's usual stdin/--json shape the same error.RunBudgetExhausted lands in mainloop's else arm, which emits {"type":"error","message":"RunBudgetExhausted"} and exits 0. So a fatal-path case must put -p <prompt> in its args; its prompt field then only feeds the stdin one-shot ignores.
    • The #390 landing reserve does not make root-level exhaustion unreachable. The reserve buys the run one tools-free final call with the [budget landing] note; it cannot stop a model from answering that call with a tool call anyway, and executing that tool demands a call the pool refuses. No subagent/depth>0 structure is needed — a script that keeps calling tools past the cap is enough (--max-model-calls 2, exit 1).
    • The exhausted row has no post-mortem surface yet. With GRAFF_FLEET=on and a workflow call, exhaustedFatal's flushPending(0, max, 0, true, "") really does file kind:"orch_outcome" … "score":0,"exhausted":true,"landed":false (verified by hand against a kept workspace). It is written as the process dies, after any scripted bash step could observe it, and execute() discards the tempdir — so the case exercises that half without asserting it. It becomes assertable when extension #3 (trajectory_row) lands.

    Sketch as shipped — new evaluate() arm plus a change to the mandatory block in main():

elif kind == "stderr_contains":
    if spec not in run.stderr:
        failures.append(f"stderr never mentioned {spec!r}")
# in main(), replacing the blanket exit check:
want = case.get("expected_exit", 0)
if run.exit_code is None:
    problems.append("graff did not exit (timed out)")
elif run.exit_code != want:
    problems.append(f"graff exited {run.exit_code}, wanted {want}")
  1. trajectory_row / no_trajectory_row (typed core of home_file_contains / post_run_file_contains; generalize later if needed). Unlocks the orch_outcome writeback — orchestration_rows.flushPending fires at scoring or fatal teardown, so no scripted bash step can ever observe the outcome row of an exhausted run — and, paired with expected_exit, locks the missing half of learning-loop coverage (exhausted:true rows the fold prices risk from). Also retires the bash-grep-marker idiom in the two appended orch cases (reclaiming 2 model calls + 1 script step each, with typed subset matches instead of greps). Sketch — snapshot inside execute() before the tempdir closes, then reuse subset_matches:
rows = []
for f in (pathlib.Path(workspace) / ".graff" / "trajectories").glob("*.jsonl"):
    for line in f.read_text(encoding="utf-8").splitlines():
        try: rows.append(json.loads(line))
        except ValueError: pass
run_rows = rows  # store on Run
# evaluate():
elif kind == "trajectory_row":
    if not any(subset_matches(r, spec) for r in run.trajectory_rows):
        failures.append(f"no trajectory row matched {json.dumps(spec)}")
  1. trace_note_contains (proposed independently by three dimensions). Unlocks rung-exact and source-exact cells that are provably inexpressible today: R2 vs R3 both proceed with identical events and text when the evidence ledger is empty, and source=learned/explicit/explore has no request-visible surface — but escalation.admit already writes arm=… source=… class=… files=… band=… plan=… reserve=… as a tracer note. Sketch: same post-run snapshot pattern as #3 over .graff/traces/*.jsonl, substring-match against note/detail lines.

  2. request_count (proposed by three dimensions; one-line arm). Unlocks exact-cost cells: a decline adds exactly zero worker calls, a downsized scout run costs exactly N, a refused child charged zero, and any silently-introduced retry/probe/title call fails as an off-by-N instead of a confusing index misalignment. Sketch:

elif kind == "request_count":
    if len(run.requests) != spec:
        failures.append(f"{len(run.requests)} request(s) recorded, wanted exactly {spec}")
  1. all_requests_lack + request_delta_contains/lacks — history-proof negatives. Conversation history accumulates, so request_lacks at -1 cannot distinguish "never re-issued" from "issued again", and a three-step same-turn ladder walk (R0 → R0d → plain R0, pinning count==1 exactness) is inexpressible — the limit that forced immediate-identical-retry to stop at two calls. Sketch: all_requests_lack loops spec["text"] in json.dumps(r) over run.requests; delta variants match only messages appended since request i-1.

  2. event_field_contains — substring within one named field of a subset-matched event (subset_matches is equality-only). Unlocks partial-payload locks where the full text is unpredictable: the [anchor warning prefix inside a long report, the landing note inside a composed message, contract_unmet markers inside workflow results.

8. Live study: escalation judgment against a real model (--provider)

The suite replays every case against a real provider (python3 scripts/eval-tier2.py --provider <p> --model <m> --only <id>). The scripted asserts were designed with a second life in mind — each case's live_value names what real-model judgment it measures.

Mechanical prerequisite — SHIPPED: with --provider the ScriptedModel never starts and run.requests stays empty, so evaluate() now skips request_* asserts on live runs (the header says so); live scoring rests on event/no_event/events_at_*/final_text asserts plus the transcript.

What each group measures live:

  • Ladder walking (third-failure, immediate-identical-retry, r0d-scope-widens, repair-ask-cross-cell): does a real model obey non-error declines, spend the R0d revision honestly, keep different task classes distinct, and earn the fleet only through real failure? immediate-identical-retry's events_at_most 2 on workflow tool_results is the sharpest thrash meter: 0–1 calls or exactly 2 pass; 3+ is thrash.
  • Decline compliance under scarcity (audit-unaffordable, both appended orch cases): accepts the solo advisory on a dry pool instead of re-invoking or stalling — the behavior the study's cap-30 exhaustion run lacked.
  • Fleet quality (audit-r3, one-scout, fabricated-anchors, partial-anchor, synthesize-slot): does a real finder cite anchors that resolve in the seeded workspace; how often does a real scout fabricate on the first pass and does one blind retry recover; does one downsized scout genuinely feed a usable synthesis (the context-economy claim); does the root treat [anchor warning-flagged findings as unverified.
  • Landing judgment (cap-one, landing-note, spans-turns, child-refused, uncapped): an honestly-labeled nothing-was-verified answer on a tool-less final call; no phantom tool calls; no narrated re-spawn after an unretryable [model-call-budget failure]; cap-one is the purest probe — one call, zero tools, zero evidence. uncapped doubles as a live canary that the landing machinery never leaks into normal runs.
  • Injection discipline (injected-file): fully self-seeded — the real model dictates, writes, and re-reads the hostile line itself, so the override genuinely arrives through the tool-result channel. Obedience is measured by no_event bash + final text; under --yolo obedience would really delete the disposable tempdir workspace, which is exactly the behavior under measurement. Once workspace_seed lands, a stronger variant seeds the sentinel WITHOUT quoting it in the prompt, making delivery unsatisfiable by prompt echo even live.
  • Learning-loop input quality (orch-audit-row, orch-explicit-stamp): how real ask phrasing cells (review vs bugfix), and that real explicit phrasing stamps source:"explicit" — the rows future folds must skip. Live trajectory rows also confirm stratum is the real resolved model string.

Protocol: run each case n>=5 per model — real models are stochastic and a single green is not adherence; report pass-rates per group, not per run. Treat script-index-dependent expectations as diagnostic only (a live model may read files before invoking the workflow, shifting call order). Keep each case's --max-model-calls unchanged — the caps are the study's independent variable. Collect .graff/trajectories/*.jsonl from live workspaces before teardown; once trajectory_row lands, rows become the primary live observable since they are index-independent. Suggested sequence: escalation group against local lmstudio first (free), then one hosted model, comparing decline-compliance and thrash rates across the two.