graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
Evals: mine live tasks from merged PRs; hill-climb on a held-out split
justrach5d679a2607e4
diff --git a/docs/adr/0214-evals-are-mined-and-hillclimbed-on-a-held-out-split.md b/docs/adr/0214-evals-are-mined-and-hillclimbed-on-a-held-out-split.mdnew file mode 100644index 00000000..7a8c768f--- /dev/null+++ b/docs/adr/0214-evals-are-mined-and-hillclimbed-on-a-held-out-split.md@@ -0,0 +1,103 @@+# 0214. Evals are mined from history and hill-climbed on a held-out split++Status: accepted 2026-09-29++## Context++`graff-evals` already had the pieces of an eval loop: 59 task files, a live+suite of real merged PRs with a public named test and a hidden follow-up+test (gates G1–G6), and `hillclimb.py`, which ran a champion and candidates on+the same tasks and kept a candidate when most efficiency axes moved by more+than a fixed 2%. Three things were missing for decisions we can trust:++1. Every live task was written by hand, so the suite stayed at 12 tasks and+ drifted away from what graff is asked to do now.+2. Candidates were judged on the tasks their authors had looked at. Nothing+ could tell an improvement from tuning to those tasks.+3. The 2% threshold was a guess. Nobody had measured how far a score moves+ between identical runs, and graff passes 12/12 of the in-house suite, so+ pass rate there has no headroom left.++Local trajectories showed the other half of the problem. They pointed at real+failure modes: one model's `read_file` failing several times as often as+another's, turns ending with work still open, and repeated publication and+worktree refusals. They also showed that one headline number was wrong. The+trace's `cache_read_tokens / context_tokens` understated prompt-cache reuse on+chained Codex sessions, because `context_tokens` is the context meter, not the+server's input count. The trace now records the server's own split in a `usage`+line (see docs/architecture.md).++## Decision++Evaluation follows the method in "Automating eval design and hillclimbing":+production-aligned tasks, validated code graders, a fixed held-out split,+measured noise, and one change per round.++**Tasks are mined from merged PRs** (`graff-evals/design/mine_prs.py`). A PR+that adds named Zig tests and changes non-test code becomes a task. The merge's+first parent is the starting tree: recent parents build on the pinned+toolchain, so no reconstruction is needed. The PR's first new test is the public+check the prompt names, and a second new test is the hidden check appended+only at grading time (source `local-git` in `live_setup.py`). Release merges,+test-only PRs and PRs over 900 changed lines are skipped.++**A grader is validated before a person sees the task.** Grading+(`named_unit_check.py`) builds only the unit-test binary, filtered to the+named tests, runs it, and requires each name to have run and passed. A test+that never ran fails, and an unrelated flaky integration script cannot fail+it. On the parent tree plus both tests, neither may pass (a compile error or+an unreachable test counts). On the merge tree both must pass, and a second run+must give the same verdicts. That is three filtered builds per task, one+build at a time.++**A person approves each task.** `review` writes a local page, and `approve`+moves a validated candidate into the `mined` suite. The loop never picks its+own tasks from one model's failures.++**Trajectories choose what to look at, not what to tune to**+(`design/mine_trajectories.py`). The report covers tool errors by tool and+model, server-reported cache reuse, cache resets at turn starts split by+idle gap, stalls, retries, open work, and refusal texts. It is written under+`design/local/` and never belongs in an issue or PR.++**The loop holds out a test set and measures noise** (`hillclimb.py split`,+`noise`, `round`). The split is drawn once from a seeded hash of task ids and+is not redrawn between rounds. `noise` repeats the baseline and records, per+set and goal (pass, usd, wall, tokens), the spread of per-repetition scores.+The band is two standard errors of a difference between two means. It also+lists flaky train tasks and warns when the baseline is at 95% or more. A+`round` runs champion and candidate on both sets, and keeps the candidate only+when:++- the train score beats the noise band, and+- the test score improves.++Train up with test flat is reported as overfitting and reverted. Either set+regressing reverts. Chasing cost or latency may not lower the pass rate by+more than the band. After three rounds with no kept change the loop stops and+lists what still fails on train. Test-set details are never printed, only its+score.++**Model-guide advice enters as candidates, not edits.** The GPT-6 prompting+guide's recommendations (follow-through, skill precedence, test calibration,+plain prose) are harness variants that append the passage. They change graff's+prompt only if a round keeps them on gpt-6 models.++## Consequences++- The eval suite can grow from the project's own history at the pace PRs+ merge, with graders that are known to fail before the fix and pass after it.+- A candidate needs evidence on tasks it was not tuned against. Some changes+ that looked like wins on the old same-set comparison will now be reverted.+- Validation and rounds are expensive. Each mined task costs three filtered+ builds to validate, and each round runs two harnesses over both sets with+ repetitions. That is fine offline and never in CI.+- Mined tasks come from this repository's public history, so a model may+ have seen a fix during training. They compare harness configurations on the+ same model, and do not measure absolute capability.+- Mined tasks depend on the local git history and toolchain. A parent that no+ longer builds on the pinned Zig fails validation and is not offered. Older+ tasks still need the reconstructed-parent path.+- Revisit if the held-out set grows too small to move beyond noise, or if+ mined tasks turn out easier than the hand-picked live tasks. The fix then is+ more tasks, not a looser rule.diff --git a/docs/adr/README.md b/docs/adr/README.mdindex 21ed2649..cab51b0d 100644--- a/docs/adr/README.md+++ b/docs/adr/README.md@@ -240,6 +240,7 @@ record only when you need the evidence or the edge cases.| [0211](0211-mcp-servers-by-name-user-level-sync-via-harness.md) | `graff mcp add <name>` resolves from the codegraff.com/mcp.json catalog, then the MCP registry only for a namespace owner matching the name. `--everywhere` writes the user-level file. Harness syncs that file through its registry as non-secret rows. Env values, headers and OAuth tokens stay on each device. Harness drives graff only over ACP (slash command, form elicitation, `_meta["graff/mcp"]` status). Supersedes the vault-item sync in #1371. || [0212](0212-build-time-is-analysis-bound.md) | A cold build is ~16 s of single-core semantic analysis; codegen and link are nearly free. No redundant files to cut. Keep the default backend: `-fllvm` is 2x slower, `-fincremental`/`--watch` do not help on Zig 0.17-dev, and `-fno-llvm -fno-lld` exhausts memory. Iterate with `-Dtest-filter` (44 s → 13 s). || [0213](0213-line-repl-paste-is-framed-and-typed.md) | Keep line-REPL paste framed through active turns; submit one draft on Enter, and stage or reject bounded image data URIs before text expansion. |+| [0214](0214-evals-are-mined-and-hillclimbed-on-a-held-out-split.md) | Mine live eval tasks from merged PRs, with graders validated red on the parent and green and stable on the merge, and a person approving each. Hill-climb on a fixed held-out split with measured noise: keep a change only when train beats the noise band and test improves. Model-guide advice enters as candidates, not edits. |## When to write onediff --git a/graff-evals/.gitignore b/graff-evals/.gitignorenew file mode 100644index 00000000..6ec8fd87--- /dev/null+++ b/graff-evals/.gitignore@@ -0,0 +1 @@+design/local/diff --git a/graff-evals/README.md b/graff-evals/README.mdindex de3e1c61..1686011b 100644--- a/graff-evals/README.md+++ b/graff-evals/README.md@@ -99,6 +99,32 @@ grok-build's heap or a 4-tool catalog (ADR 0024). First keep:# hillclimb/frontier-inhouse-20260830.svg```+### Mined tasks and the held-out loop (ADR 0214)++New live tasks come from merged PRs, and a candidate is judged on tasks it was+not tuned against.++```sh+./design/mine_trajectories.py --days 14 # what goes wrong in real runs (design/local/, never commit)+./design/mine_prs.py list --days 21 # merged PRs that add named tests and change code+./design/mine_prs.py build 1386 # task pr-1386: parent tree, public test, hidden sibling test+./design/mine_prs.py validate pr-1386 # red on the parent, green and stable on the merge (3 filtered builds)+./design/mine_prs.py review # design/local/review.md for a person to approve+./design/mine_prs.py approve pr-1386 # into the `mined` suite++./hillclimb.py split --suite mined # fixed train/test split; never redrawn between rounds+./hillclimb.py noise --suite mined --harness graff-dev --model gpt-6-sol --reps 3+./hillclimb.py round --suite mined --champion graff-dev --candidate graff-dev-g6-followthrough --model gpt-6-sol+```++A round keeps the candidate only when its train score beats the measured+noise band and its test score improves. Train up with test flat is+overfitting and is reverted. `--goal usd|wall|tokens` optimizes cost or+latency without letting the pass rate drop. After three rounds without a keep,+the loop lists what still fails on train, and never shows test-set details. The+GPT-6 prompting guide's recommendations are the first candidates+(`graff-dev-g6-*`). Validation and rounds run one heavy build at a time.+## Harnesses`harnesses.json` declares each harness as a command template plus parsers:diff --git a/graff-evals/design/mine_prs.py b/graff-evals/design/mine_prs.pynew file mode 100644index 00000000..e172add7--- /dev/null+++ b/graff-evals/design/mine_prs.py@@ -0,0 +1,377 @@+#!/usr/bin/env python3+"""Mine live eval tasks from merged pull requests (ADR 0214).++A merged PR that adds named Zig tests and changes non-test code is a task the+project already judged worth doing. The merge's first parent is the starting+tree (recent parents build on the pinned toolchain, so no reconstruction), the+PR's first new test is the public check the prompt names, and a second new+test is the hidden check appended only at grading time.++Before anything is offered for review, each grader is validated:++ red on the parent tree plus both tests, neither passes (a compile+ error or a test the module graph never reaches counts)+ green on the merge tree, both tests pass+ stable a second run on the merge tree gives the same verdicts++ ./design/mine_prs.py list [--days 21] [--limit 40]+ ./design/mine_prs.py build <pr> [--public NAME] [--hidden NAME]+ ./design/mine_prs.py validate <task-id> # three full test runs+ ./design/mine_prs.py review # writes design/local/review.md+ ./design/mine_prs.py approve <task-id> # candidate -> mined suite++Grading builds only the filtered unit-test binary and requires each named+test to have run (named_unit_check.py). Validation grades three times per+task: one build at a time, never in parallel with another build.+"""+from __future__ import annotations++import argparse+import json+import os+import re+import shutil+import subprocess+import sys+import tempfile++HERE = os.path.dirname(os.path.abspath(__file__))+EVALS = os.path.dirname(HERE)+REPO = os.path.dirname(EVALS)+sys.path.insert(0, EVALS)+import live_setup # noqa: E402+import named_unit_check # noqa: E402++MINED = os.path.join(EVALS, "live", "mined.json") # catalog.json stays hand-edited+TASKS = os.path.join(EVALS, "tasks")+LOCAL = os.path.join(HERE, "local")+MERGE_RE = re.compile(r"^Merge pull request #(\d+) from ")+TEST_RE = re.compile(r'^\+\s*test "((?:[^"\\]|\\.)*)"\s*\{')+SKIP_TITLES = ("release", "merge release", "merge current release", "docs:", "readme", "adr ", "changelog")+MAX_CHANGED = 900 # lines; larger PRs are not a fair single task+++def git(*args: str, check: bool = True) -> str:+ p = subprocess.run(["git", "-C", REPO, *args], capture_output=True, text=True)+ if check and p.returncode != 0:+ raise SystemExit(f"git {' '.join(args)}: {p.stderr.strip()}")+ return p.stdout+++def merged_prs(days: int, ref: str) -> list[dict]:+ out = git("log", ref, "--first-parent", "--merges", f"--since={days}.days", "--format=%H%x09%s")+ rows = []+ for line in out.splitlines():+ sha, _, subject = line.partition("\t")+ m = MERGE_RE.match(subject)+ if not m:+ continue+ body = git("show", "-s", "--format=%b", sha).strip().splitlines()+ rows.append({"pr": int(m.group(1)), "merge": sha, "parent": f"{sha}^1",+ "title": (body[0] if body else subject).strip()})+ return rows+++def is_test_file(path: str) -> bool:+ base = os.path.basename(path)+ return base.endswith(("_tests.zig", "_test.zig")) or "/tests/" in path+++def added_tests(parent: str, merge: str) -> list[dict]:+ diff = git("diff", "-U0", parent, merge, "--", "src", "TUI")+ tests, current = [], None+ for line in diff.splitlines():+ if line.startswith("+++ b/"):+ current = line[6:]+ elif current and current.endswith(".zig"):+ m = TEST_RE.match(line)+ if m:+ tests.append({"file": current, "name": m.group(1)})+ return tests+++def changed_lines(parent: str, merge: str) -> tuple[int, int]:+ """(all changed lines, changed lines in non-test source)."""+ total = source = 0+ for line in git("diff", "--numstat", parent, merge).splitlines():+ parts = line.split("\t")+ if len(parts) != 3 or not parts[0].isdigit():+ continue+ n = int(parts[0]) + int(parts[1])+ total += n+ if parts[2].endswith(".zig") and parts[2].startswith(("src/", "TUI/")) and not is_test_file(parts[2]):+ source += n+ return total, source+++def candidates(days: int, ref: str, limit: int) -> list[dict]:+ out = []+ for pr in merged_prs(days, ref):+ if pr["title"].lower().startswith(SKIP_TITLES):+ continue+ tests = added_tests(pr["parent"], pr["merge"])+ total, source = changed_lines(pr["parent"], pr["merge"])+ pr.update(tests=tests, changed=total, source_changed=source)+ pr["fit"] = ("ok" if tests and source and total <= MAX_CHANGED else+ "no new named test" if not tests else+ "tests only" if not source else "too large")+ out.append(pr)+ if len(out) >= limit:+ break+ return out+++# ── Zig test blocks ──────────────────────────────────────────────────────++def test_block(text: str, name: str) -> str | None:+ """The full `test "name" { … }` block, braces matched outside strings,+ char literals, `//` comments and `\\\\` multiline string lines."""+ head = f'test "{name}"'+ start = text.find(head)+ if start < 0:+ return None+ # Include the doc comments / blank line directly above? No: the block only.+ i = text.find("{", start + len(head))+ if i < 0:+ return None+ depth, n = 0, len(text)+ while i < n:+ c = text[i]+ if c == "/" and text.startswith("//", i):+ j = text.find("\n", i)+ i = n if j < 0 else j+ continue+ if c == "\\" and text.startswith("\\\\", i):+ j = text.find("\n", i)+ i = n if j < 0 else j+ continue+ if c in "\"'":+ j = i + 1+ while j < n and text[j] != c:+ j += 2 if text[j] == "\\" else 1+ i = j + 1+ continue+ if c == "{":+ depth += 1+ elif c == "}":+ depth -= 1+ if depth == 0:+ return text[start:i + 1] + "\n"+ i += 1+ return None+++def file_at(rev: str, path: str) -> str:+ return git("show", f"{rev}:{path}")+++# ── task files ───────────────────────────────────────────────────────────++def load_catalog() -> dict:+ if not os.path.exists(MINED):+ return {"note": "Machine-managed by design/mine_prs.py (ADR 0214). catalog.json stays hand-edited.",+ "candidates": [], "mined": []}+ with open(MINED) as f:+ return json.load(f)+++def save_catalog(cat: dict) -> None:+ with open(MINED, "w") as f:+ json.dump(cat, f, indent=1, ensure_ascii=False)+ f.write("\n")+++def write(path: str, text: str, mode: int = 0o644) -> None:+ os.makedirs(os.path.dirname(path), exist_ok=True)+ with open(path, "w") as f:+ f.write(text)+ os.chmod(path, mode)+++def prompt_for(title: str, public: dict) -> str:+ return (f"{title}. Make the test named \"{public['name']}\" in {public['file']} pass. "+ "Do not edit that test. Do not add SPEC.md.")+++def build(pr_number: int, public_name: str | None, hidden_name: str | None, ref: str) -> str:+ found = [p for p in merged_prs(3650, ref) if p["pr"] == pr_number]+ if not found:+ raise SystemExit(f"#{pr_number} is not a merged PR on {ref}")+ pr = found[0]+ tests = added_tests(pr["parent"], pr["merge"])+ if not tests:+ raise SystemExit(f"#{pr_number} adds no named Zig test")+ pick = lambda name, default: next((t for t in tests if t["name"] == name), None) if name else default+ public = pick(public_name, tests[0])+ hidden = pick(hidden_name, next((t for t in tests if t is not public and t["name"] != public["name"]), None))+ if public is None:+ raise SystemExit(f"no added test named {public_name!r}")+ task_id = f"pr-{pr_number}"+ blocks = {}+ for role, t in (("public", public), ("hidden", hidden)):+ if t is None:+ continue+ block = test_block(file_at(pr["merge"], t["file"]), t["name"])+ if block is None:+ raise SystemExit(f"could not extract test {t['name']!r} from {t['file']}")+ blocks[role] = block+ tdir = os.path.join(EVALS, "live", task_id)+ write(os.path.join(tdir, "public_case.inc"), "\n" + blocks["public"])+ write(os.path.join(tdir, "check_public.sh"),+ "#!/bin/sh\nset -eu\nTASK_ROOT=${TASK_ROOT:?}\n"+ f"exec python3 \"$TASK_ROOT/named_unit_check.py\" {json.dumps(public['name'])}\n", 0o755)+ hidden_rel = None+ if hidden:+ write(os.path.join(tdir, "hidden_case.inc"), "\n" + blocks["hidden"])+ hidden_rel = f"hidden/{task_id}.sh"+ write(os.path.join(EVALS, hidden_rel),+ "#!/bin/sh\nset -eu\nTASK_ROOT=${TASK_ROOT:?}\n"+ f"INC=\"$TASK_ROOT/live/{task_id}/hidden_case.inc\"\n"+ f"if ! grep -qF {json.dumps('test ' + json.dumps(hidden['name']))} {hidden['file']} 2>/dev/null; then\n"+ f" cat \"$INC\" >> {hidden['file']}\nfi\n"+ f"exec python3 \"$TASK_ROOT/named_unit_check.py\" {json.dumps(hidden['name'])}\n", 0o755)+ entry = {+ "id": task_id, "pr": f"codegraff #{pr_number}", "why": pr["title"],+ "repo": "justrach/codegraff", "historical_parent": git("rev-parse", pr["parent"]).strip(),+ "historical_merge": pr["merge"], "source": "local-git",+ "public_filter": public["name"], "public_file": public["file"],+ "hidden": hidden_rel, "hidden_filter": hidden["name"] if hidden else None,+ "hidden_file": hidden["file"] if hidden else None, "check_timeout_s": 420,+ "validation": None,+ }+ cat = load_catalog()+ cat.setdefault("candidates", [])+ cat["candidates"] = [c for c in cat["candidates"] if c["id"] != task_id] + [entry]+ save_catalog(cat)+ check = f"sh \"$TASK_ROOT/live/{task_id}/check_public.sh\""+ if hidden_rel:+ check += f" && sh \"$TASK_ROOT/{hidden_rel}\""+ task = {"id": task_id, "suite": "mined-candidate", "category": "live",+ "source": f"codegraff #{pr_number} — mined (ADR 0214)", "prompt": prompt_for(pr["title"], public),+ "setup": [f"python3 \"$TASK_ROOT/live_setup.py\" {task_id} ."], "check": check,+ "timeout_s": 900, "setup_timeout_s": 300, "check_timeout_s": 420}+ write(os.path.join(TASKS, f"mined-{task_id}.json"), json.dumps(task, indent=1, ensure_ascii=False) + "\n")+ return task_id+++# ── validation (red on parent, green and stable on merge) ────────────────++def validate(task_id: str) -> dict:+ cat = load_catalog()+ entry = next((c for c in cat.get("candidates", []) if c["id"] == task_id), None)+ if entry is None:+ raise SystemExit(f"{task_id} is not a candidate; build it first")+ tdir = os.path.join(EVALS, "live", task_id)+ names = [(entry["public_file"], os.path.join(tdir, "public_case.inc"), entry["public_filter"])]+ if entry.get("hidden_filter"):+ names.append((entry["hidden_file"], os.path.join(tdir, "hidden_case.inc"), entry["hidden_filter"]))+ timeout = entry.get("check_timeout_s", 420)+ result = {"red": {}, "green": {}, "stable": {}}+ tests = [name for _, _, name in names]+ with tempfile.TemporaryDirectory(prefix=f"mine-{task_id}-") as tmp:+ parent, merge = os.path.join(tmp, "parent"), os.path.join(tmp, "merge")+ live_setup.archive_local(entry["historical_parent"], parent)+ for rel, inc, name in names:+ live_setup.append_case(parent, rel, inc, name)+ result["red"] = named_unit_check.grade(parent, tests, timeout)+ shutil.rmtree(parent)+ live_setup.archive_local(entry["historical_merge"], merge)+ result["green"] = named_unit_check.grade(merge, tests, timeout)+ result["stable"] = named_unit_check.grade(merge, tests, timeout)+ ok = (all(v != "pass" for v in result["red"].values())+ and all(v == "pass" for v in result["green"].values())+ and result["stable"] == result["green"])+ result["ok"] = ok+ entry["validation"] = result+ save_catalog(cat)+ return result+++# ── review and approval ──────────────────────────────────────────────────++def review() -> str:+ cat = load_catalog()+ lines = ["# Mined task candidates", "",+ "Approve only tasks you would judge hard and representative; the loop never sees the hidden test.", ""]+ for c in cat.get("candidates", []):+ v = c.get("validation") or {}+ status = "not validated" if not v else ("valid" if v.get("ok") else f"INVALID {v}")+ lines += [f"## {c['id']} — {c['why']}", f"- source: {c['pr']} (parent {c['historical_parent'][:10]})",+ f"- public: `{c['public_filter']}` in {c['public_file']}",+ f"- hidden: `{c.get('hidden_filter') or '—'}`", f"- grader: {status}", ""]+ os.makedirs(LOCAL, exist_ok=True)+ path = os.path.join(LOCAL, "review.md")+ with open(path, "w") as f:+ f.write("\n".join(lines))+ return path+++def approve(task_id: str) -> None:+ cat = load_catalog()+ entry = next((c for c in cat.get("candidates", []) if c["id"] == task_id), None)+ if entry is None:+ raise SystemExit(f"{task_id} is not a candidate")+ if not (entry.get("validation") or {}).get("ok"):+ raise SystemExit(f"{task_id} has no passing validation; run validate first")+ cat["candidates"] = [c for c in cat["candidates"] if c["id"] != task_id]+ cat.setdefault("mined", []).append(entry)+ save_catalog(cat)+ path = os.path.join(TASKS, f"mined-{task_id}.json")+ task = json.load(open(path))+ task["suite"] = "mined"+ write(path, json.dumps(task, indent=1, ensure_ascii=False) + "\n")+++# ── self-test (no git history or builds needed) ──────────────────────────++def self_test() -> None:+ src = ('const std = @import("std");\n'+ 'test "outer { brace in name" {\n'+ ' const s = "}{";\n'+ " const c = '}';\n"+ " // } comment brace\n"+ " const m =\n"+ " \\\\ multiline } brace\n"+ " ;\n"+ " if (true) { _ = s; _ = c; _ = m; }\n"+ "}\n"+ 'test "second" {}\n')+ block = test_block(src, "outer { brace in name")+ assert block and block.startswith('test "outer') and block.rstrip().endswith("}"), block+ assert 'test "second"' not in block+ assert test_block(src, "second") == 'test "second" {}\n'+ assert test_block(src, "missing") is None+ assert TEST_RE.match('+test "a \\"quoted\\" name" {').group(1) == 'a \\"quoted\\" name'+ print("mine_prs self-test ok")+++def main() -> None:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--ref", default="origin/main")+ sub = ap.add_subparsers(dest="cmd", required=True)+ p = sub.add_parser("list"); p.add_argument("--days", type=int, default=21); p.add_argument("--limit", type=int, default=40)+ p = sub.add_parser("build"); p.add_argument("pr", type=int); p.add_argument("--public"); p.add_argument("--hidden")+ p = sub.add_parser("validate"); p.add_argument("task_id")+ sub.add_parser("review")+ p = sub.add_parser("approve"); p.add_argument("task_id")+ sub.add_parser("self-test")+ a = ap.parse_args()+ if a.cmd == "list":+ for c in candidates(a.days, a.ref, a.limit):+ names = ", ".join(t["name"] for t in c["tests"][:3]) or "—"+ print(f"#{c['pr']:<5} {c['fit']:<18} {c['changed']:>5} lines {c['title'][:70]}\n tests: {names[:140]}")+ elif a.cmd == "build":+ print(build(a.pr, a.public, a.hidden, a.ref))+ elif a.cmd == "validate":+ print(json.dumps(validate(a.task_id), indent=1))+ elif a.cmd == "review":+ print(review())+ elif a.cmd == "approve":+ approve(a.task_id)+ print(f"{a.task_id} approved into the mined suite")+ else:+ self_test()+++if __name__ == "__main__":+ main()diff --git a/graff-evals/design/mine_trajectories.py b/graff-evals/design/mine_trajectories.pynew file mode 100644index 00000000..e5ba5856--- /dev/null+++ b/graff-evals/design/mine_trajectories.py@@ -0,0 +1,203 @@+#!/usr/bin/env python3+"""Find what to evaluate from local trajectories (ADR 0214).++Reads the `.graff/traces/*.jsonl` files graff writes in each project and the+saved transcripts next to them, and reports where runs actually go wrong:+tool error rates by tool and model, prompt-cache reuse, cache resets at turn+starts (split by idle gap), stalls and retries, turns that stopped with work+still open, and the most common refusal texts. The point is to pick eval+cases from evidence a person can judge, not from one model's failures.++The report can contain local paths and prompts, so it is written under+design/local/ (ignored by git) and never belongs in an issue or PR.++ ./design/mine_trajectories.py [--days 14] [--root ~] [--json]+"""+from __future__ import annotations++import argparse+import collections+import glob+import json+import os+import re+import statistics as st+import time++HERE = os.path.dirname(os.path.abspath(__file__))+LOCAL = os.path.join(HERE, "local")+++def pct(xs: list[float], p: float):+ xs = sorted(xs)+ return xs[min(len(xs) - 1, int(len(xs) * p))] if xs else None+++def trace_files(root: str, days: int, sub: str, pattern: str) -> list[str]:+ root = os.path.expanduser(root)+ dirs = glob.glob(os.path.join(root, "*", ".graff", sub)) + [os.path.join(root, ".graff", sub)]+ cutoff = time.time() - days * 86400+ return [f for d in dirs for f in glob.glob(os.path.join(d, pattern)) if os.path.getmtime(f) > cutoff]+++def load_events(files: list[str]) -> list[list[dict]]:+ runs = []+ for f in files:+ rows = []+ for line in open(f, errors="replace"):+ try:+ rows.append(json.loads(line))+ except ValueError:+ continue+ runs.append(rows)+ return runs+++def tool_table(runs: list[list[dict]]) -> list[dict]:+ by = collections.defaultdict(lambda: {"n": 0, "err": 0, "ms": []})+ for rows in runs:+ for e in rows:+ if e.get("ev") == "tool":+ b = by[e.get("name")]+ b["n"] += 1+ b["err"] += bool(e.get("is_error"))+ b["ms"].append(e.get("ms") or 0)+ out = [{"tool": k, "calls": v["n"], "error_pct": round(100 * v["err"] / v["n"], 1),+ "p95_ms": pct(v["ms"], .95)} for k, v in by.items()]+ return sorted(out, key=lambda r: -r["calls"])+++def tool_errors_by_model(runs: list[list[dict]], tools=("read_file", "write_file", "edit_file", "shell")) -> list[dict]:+ by = collections.defaultdict(lambda: [0, 0])+ for rows in runs:+ model = None+ for e in rows:+ if e.get("ev") == "api":+ model = e.get("model")+ if e.get("ev") == "tool" and e.get("name") in tools:+ k = (e["name"], model)+ by[k][0] += 1+ by[k][1] += bool(e.get("is_error"))+ return sorted(({"tool": t, "model": m, "calls": n, "error_pct": round(100 * err / n, 1)}+ for (t, m), (n, err) in by.items() if n >= 30), key=lambda r: -r["error_pct"])+++def cache_table(runs: list[list[dict]]) -> list[dict]:+ """Server-reported reuse from `usage` lines; the meter fallback is flagged."""+ real = collections.defaultdict(list)+ meter = collections.defaultdict(list)+ for rows in runs:+ for e in rows:+ if e.get("ev") == "usage" and not e.get("from_sub") and (e.get("input_tokens") or 0) >= 2048:+ real[e.get("model")].append(e["cache_read_tokens"] / e["input_tokens"])+ elif e.get("ev") == "api" and not e.get("is_error") and not e.get("from_sub") and (e.get("context_tokens") or 0) > 15000:+ meter[e.get("model")].append(min(1, (e.get("cache_read_tokens") or 0) / e["context_tokens"]))+ out = []+ for m in set(real) | set(meter):+ if real.get(m):+ out.append({"model": m, "requests": len(real[m]), "reuse_pct": round(100 * st.mean(real[m]), 1), "source": "usage"})+ elif len(meter[m]) >= 20:+ out.append({"model": m, "requests": len(meter[m]), "reuse_pct": round(100 * st.mean(meter[m]), 1),+ "source": "meter (not a hit rate: see docs/architecture.md)"})+ return sorted(out, key=lambda r: -r["requests"])+++def turn_resets(runs: list[list[dict]], idle_s: int = 600) -> list[dict]:+ """A turn whose first request reads under half the previous request's+ cache is a reset; resets after a long idle gap are provider expiry."""+ by = collections.defaultdict(lambda: {"boundaries": 0, "idle_resets": 0, "other_resets": 0})+ for rows in runs:+ prev = None+ for e in rows:+ if e.get("ev") != "api" or e.get("is_error") or e.get("from_sub"):+ continue+ r, turn, t = e.get("cache_read_tokens") or 0, e.get("turn"), e.get("t") or 0+ if prev and prev["model"] == e.get("model") and turn != prev["turn"] and prev["read"] > 20000:+ b = by[e.get("model")]+ b["boundaries"] += 1+ if r < 0.5 * prev["read"]:+ b["idle_resets" if (t - prev["t"]) / 1000 > idle_s else "other_resets"] += 1+ prev = {"model": e.get("model"), "read": r, "turn": turn, "t": t}+ return sorted(({"model": m, **v} for m, v in by.items() if v["boundaries"] >= 10), key=lambda r: -r["boundaries"])+++def counted_details(runs: list[list[dict]], evs=("stream_retry", "stall_budget", "retry", "fake_done",+ "ws_api_error", "pending_work")) -> dict:+ out = {}+ for ev in evs:+ c = collections.Counter(str(e.get("detail"))[:90] for rows in runs for e in rows if e.get("ev") == ev)+ if c:+ out[ev] = c.most_common(5)+ return out+++def refusal_texts(files: list[str], top: int = 12) -> list[tuple[int, str, str]]:+ """Most common error openings in saved transcripts, by tool."""+ counts = collections.Counter()+ for f in files:+ calls = {}+ for line in open(f, errors="replace"):+ try:+ m = json.loads(line)+ except ValueError:+ continue+ for tc in m.get("tool_calls") or []:+ calls[tc.get("id")] = (tc.get("function") or {}).get("name")+ if m.get("role") == "tool":+ c = m.get("content")+ c = c if isinstance(c, str) else json.dumps(c)+ if c.startswith("[error]"):+ first = re.sub(r"\d+", "N", re.sub(r"'[^']*'", "'X'", c.split("\n")[0]))+ counts[(calls.get(m.get("tool_call_id")) or "?", first[:110])] += 1+ return [(n, tool, text) for (tool, text), n in counts.most_common(top)]+++def report(days: int, root: str) -> dict:+ traces = trace_files(root, days, "traces", "*.jsonl")+ transcripts = trace_files(root, days, "sessions", "*.transcript.jsonl")+ runs = load_events(traces)+ return {"days": days, "runs": len(runs), "transcripts": len(transcripts),+ "tools": tool_table(runs)[:20], "tool_errors_by_model": tool_errors_by_model(runs)[:15],+ "cache": cache_table(runs), "turn_resets": turn_resets(runs),+ "details": counted_details(runs), "refusals": refusal_texts(transcripts)}+++def markdown(r: dict) -> str:+ L = [f"# Trajectory report — last {r['days']} days", "",+ f"{r['runs']} runs, {r['transcripts']} saved transcripts. Local only: do not paste into issues.", "",+ "## Tools", "", "| tool | calls | error % | p95 ms |", "|---|---:|---:|---:|"]+ L += [f"| {t['tool']} | {t['calls']} | {t['error_pct']} | {t['p95_ms']} |" for t in r["tools"]]+ L += ["", "## Tool errors by model (30+ calls)", "", "| tool | model | calls | error % |", "|---|---|---:|---:|"]+ L += [f"| {t['tool']} | {t['model']} | {t['calls']} | {t['error_pct']} |" for t in r["tool_errors_by_model"]]+ L += ["", "## Prompt-cache reuse", "", "| model | requests | reuse % | source |", "|---|---:|---:|---|"]+ L += [f"| {c['model']} | {c['requests']} | {c['reuse_pct']} | {c['source']} |" for c in r["cache"]]+ L += ["", "## Cache resets at turn starts", "", "| model | turn boundaries | after idle >10 min | other |", "|---|---:|---:|---:|"]+ L += [f"| {c['model']} | {c['boundaries']} | {c['idle_resets']} | {c['other_resets']} |" for c in r["turn_resets"]]+ L += ["", "## Retries, stalls and open work", ""]+ for ev, rows in r["details"].items():+ L += [f"- **{ev}**: " + "; ".join(f"{n}× {t}" for t, n in rows)]+ L += ["", "## Most common refusals in transcripts", ""]+ L += [f"- {n}× `{tool}`: {text}" for n, tool, text in r["refusals"]]+ L += ["", "Pick eval cases a person would call hard and representative; do not tune prompts to these texts."]+ return "\n".join(L) + "\n"+++def main() -> None:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--days", type=int, default=14)+ ap.add_argument("--root", default="~")+ ap.add_argument("--json", action="store_true")+ a = ap.parse_args()+ r = report(a.days, a.root)+ if a.json:+ print(json.dumps(r, indent=1))+ return+ os.makedirs(LOCAL, exist_ok=True)+ path = os.path.join(LOCAL, f"trajectory-report-{time.strftime('%Y%m%d')}.md")+ with open(path, "w") as f:+ f.write(markdown(r))+ print(path)+++if __name__ == "__main__":+ main()diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.jsonindex e8b04c23..ae906c0e 100644--- a/graff-evals/harnesses.json+++ b/graff-evals/harnesses.json@@ -254,5 +254,49 @@"capabilities": [],"env": {"FX_MAX_AGENT_STEPS": "0"},"default_model": "grok-4.6"+ },+ "graff-dev-g6-followthrough": {+ "cmd": ["{repo}/zig-out/bin/graff", "-p", "{prompt}", "--model", "{model}", "--yolo", "--append-system-prompt",+ "Infer the user's intent and task scope from the instructions and prior conversation. Bias towards action and carry the intended task to completion. When the user asks for new work or a fix, persist until the goal is complete, including read-only actions, isolated worktrees and draft PRs, unless a step is clearly destructive or irreversible. Treat \"can you\", \"I want to\" and \"help me\" as instructions to do the work, not to describe it. Before asking a clarifying question, finish the work that is already authorized so the user approves a concrete, reviewable result."],+ "answer": "stdout",+ "usage": "graff-stderr",+ "capabilities": ["output-schema"],+ "schema_args": ["--output-schema", "{schema}"],+ "env": {"GRAFF_POST_DEADLINE_SECS": "90"},+ "default_model": "gpt-6-sol",+ "note": "GPT-6 prompting guide candidate (g6-followthrough); judged by hillclimb.py round on held-out tasks before any prompt change ships (ADR 0214)"+ },+ "graff-dev-g6-skill-precedence": {+ "cmd": ["{repo}/zig-out/bin/graff", "-p", "{prompt}", "--model", "{model}", "--yolo", "--append-system-prompt",+ "The user's instructions take precedence over guidelines in a skill or instruction file. If a skill causes you to pause, ask for confirmation, or leave work unfinished, name the file, quote the instruction, and say how it applies."],+ "answer": "stdout",+ "usage": "graff-stderr",+ "capabilities": ["output-schema"],+ "schema_args": ["--output-schema", "{schema}"],+ "env": {"GRAFF_POST_DEADLINE_SECS": "90"},+ "default_model": "gpt-6-sol",+ "note": "GPT-6 prompting guide candidate (g6-skill-precedence); judged by hillclimb.py round on held-out tasks before any prompt change ships (ADR 0214)"+ },+ "graff-dev-g6-test-calibration": {+ "cmd": ["{repo}/zig-out/bin/graff", "-p", "{prompt}", "--model", "{model}", "--yolo", "--append-system-prompt",+ "Do not write tests for reversible, low-impact changes that mirror the implementation. Run the tests and checks the change needs; broaden or repeat them only when new changes, failures or open concerns justify it."],+ "answer": "stdout",+ "usage": "graff-stderr",+ "capabilities": ["output-schema"],+ "schema_args": ["--output-schema", "{schema}"],+ "env": {"GRAFF_POST_DEADLINE_SECS": "90"},+ "default_model": "gpt-6-sol",+ "note": "GPT-6 prompting guide candidate (g6-test-calibration); judged by hillclimb.py round on held-out tasks before any prompt change ships (ADR 0214)"+ },+ "graff-dev-g6-plain-prose": {+ "cmd": ["{repo}/zig-out/bin/graff", "-p", "{prompt}", "--model", "{model}", "--yolo", "--append-system-prompt",+ "Default to clear, concise paragraphs, each developing one idea. Use lists only for parallel, sequential or comparable information. State the main point early, in plain language with precise verbs, and avoid stock phrases and concluding summaries."],+ "answer": "stdout",+ "usage": "graff-stderr",+ "capabilities": ["output-schema"],+ "schema_args": ["--output-schema", "{schema}"],+ "env": {"GRAFF_POST_DEADLINE_SECS": "90"},+ "default_model": "gpt-6-sol",+ "note": "GPT-6 prompting guide candidate (g6-plain-prose); judged by hillclimb.py round on held-out tasks before any prompt change ships (ADR 0214)"}}diff --git a/graff-evals/hillclimb.py b/graff-evals/hillclimb.pyindex bd12d122..44c2c747 100755--- a/graff-evals/hillclimb.py+++ b/graff-evals/hillclimb.py@@ -9,10 +9,20 @@ Never keep grok-build's heap or a 4-tool catalog (ADR 0024)../hillclimb.py score results/run-….jsonl./hillclimb.py decide --champion results/a.jsonl --candidate results/b.jsonl./hillclimb.py iterate --suite core --task exact-reply,fix-fib,file-ops++Held-out loop (ADR 0214): fix a train/test split once, measure noise from+repeated baseline runs, then judge one candidate per round. A candidate is+kept only when its train score beats the noise band AND its test score+improves; train up with test flat is overfitting and is reverted. Only train+failures are ever printed; the test set is scores only.++ ./hillclimb.py split --suite mined [--test-frac 0.33]+ ./hillclimb.py noise --suite mined --harness graff-dev --model gpt-6-sol --reps 3+ ./hillclimb.py round --suite mined --champion graff-dev --candidate graff-dev-x --model gpt-6-sol"""from __future__ import annotations-import argparse, json, os, subprocess, sys, time+import argparse, hashlib, json, math, os, statistics, subprocess, sys, timefrom list_price import attach, self_test as price_self_test@@ -233,12 +243,189 @@ def cmd_iterate(args) -> None:print("\nblocked-honest:", "; ".join(blocked))+# ── held-out loop (ADR 0214) ─────────────────────────────────────────────++TASKS_DIR = os.path.join(ROOT, "tasks")+GOALS = {"pass": +1, "usd": -1, "wall": -1, "tokens": -1} # +1: higher is better+STALL_ROUNDS = 3+HEADROOM = 0.95+++def suite_tasks(suite: str) -> list[str]:+ ids = []+ for name in sorted(os.listdir(TASKS_DIR)):+ if name.endswith(".json"):+ with open(os.path.join(TASKS_DIR, name)) as f:+ t = json.load(f)+ if t.get("suite") == suite:+ ids.append(t["id"])+ return ids+++def make_split(ids: list[str], test_frac: float, seed: str) -> dict:+ if len(ids) < 3:+ raise SystemExit(f"need at least 3 tasks to hold some out, have {len(ids)}")+ order = sorted(ids, key=lambda i: hashlib.sha256(f"{seed}:{i}".encode()).hexdigest())+ n_test = min(len(ids) - 1, max(1, round(len(ids) * test_frac)))+ return {"seed": seed, "test": sorted(order[:n_test]), "train": sorted(order[n_test:])}+++def split_path(suite: str) -> str:+ return os.path.join(LOG_DIR, f"split-{suite}.json")+++def load_split(suite: str) -> dict:+ path = split_path(suite)+ if not os.path.exists(path):+ raise SystemExit(f"no split for {suite}; run: hillclimb.py split --suite {suite}")+ with open(path) as f:+ return json.load(f)+++def metric(rows: list[dict], goal: str) -> float:+ if not rows:+ return 0.0+ if goal == "pass":+ return sum(bool(r.get("outcome_ok")) for r in rows) / len(rows)+ key = {"usd": "list_usd", "wall": "wall_s", "tokens": "list_tokens"}[goal]+ vals = [r.get(key) or 0 for r in rows]+ return sum(vals) / len(vals)+++def rep_scores(rows: list[dict], harness: str, tasks: list[str], goal: str) -> list[float]:+ """One score per repetition: the goal metric over that rep's tasks."""+ by_rep: dict[int, list[dict]] = {}+ for r in rows:+ if r.get("harness") == harness and r.get("task") in tasks:+ by_rep.setdefault(int(r.get("rep") or 0), []).append(r)+ return [metric(v, goal) for _, v in sorted(by_rep.items())]+++def band(sd: float, reps: int) -> float:+ """Two standard errors of a difference between two `reps`-rep means."""+ return 2 * sd * math.sqrt(2 / max(1, reps))+++def noise_summary(rows: list[dict], harness: str, split: dict, reps: int) -> dict:+ out = {"harness": harness, "reps": reps, "sets": {}}+ for name in ("train", "test"):+ entry = {}+ for goal in GOALS:+ scores = rep_scores(rows, harness, split[name], goal)+ sd = statistics.pstdev(scores) if len(scores) > 1 else 0.0+ entry[goal] = {"mean": round(statistics.mean(scores), 6) if scores else 0.0,+ "rep_sd": round(sd, 6), "band": round(band(sd, reps), 6)}+ flaky = sorted({r["task"] for r in rows if r.get("harness") == harness and r.get("task") in split[name]+ and len({bool(x.get("outcome_ok")) for x in rows+ if x.get("harness") == harness and x.get("task") == r["task"]}) > 1})+ entry["flaky_tasks"] = flaky if name == "train" else len(flaky) # never name test tasks+ out["sets"][name] = entry+ out["headroom_warning"] = out["sets"]["train"]["pass"]["mean"] >= HEADROOM+ return out+++def judge(champ_train: float, cand_train: float, champ_test: float, cand_test: float,+ min_effect: float, goal: str = "pass") -> tuple[bool, str]:+ sign = GOALS[goal]+ d_train = (cand_train - champ_train) * sign+ d_test = (cand_test - champ_test) * sign+ if d_train < -min_effect or d_test < 0:+ return False, f"regressed (train {d_train:+.3f}, test {d_test:+.3f})"+ if d_train > min_effect and d_test > 0:+ return True, f"kept: train {d_train:+.3f} beats the noise band {min_effect:.3f} and test improved {d_test:+.3f}"+ if d_train > min_effect:+ return False, f"overfit: train {d_train:+.3f} improved but test is flat ({d_test:+.3f})"+ return False, f"within noise: train {d_train:+.3f} vs band {min_effect:.3f}"+++def stalled(suite: str) -> int:+ """Consecutive non-kept rounds for this suite, most recent first."""+ if not os.path.exists(LOG_PATH):+ return 0+ n = 0+ with open(LOG_PATH) as f:+ rounds = [json.loads(l) for l in f if l.strip()]+ for e in reversed([r for r in rounds if r.get("kind") == "round" and r.get("suite") == suite]):+ if e.get("keep"):+ break+ n += 1+ return n+++def cmd_split(args) -> None:+ path = split_path(args.suite)+ if os.path.exists(path) and not args.force:+ raise SystemExit(f"{path} exists; the test set must stay fixed across rounds (--force to redraw)")+ split = make_split(suite_tasks(args.suite), args.test_frac, args.seed)+ split["suite"] = args.suite+ os.makedirs(LOG_DIR, exist_ok=True)+ with open(path, "w") as f:+ json.dump(split, f, indent=1)+ f.write("\n")+ print(f"{args.suite}: {len(split['train'])} train, {len(split['test'])} held out -> {path}")+++def cmd_noise(args) -> None:+ split = load_split(args.suite)+ path = run_eval(args.harness, args.model, args.suite, split["train"] + split["test"], args.reps, args.jobs)+ summary = noise_summary(load_jsonl(path), args.harness, split, args.reps)+ summary.update(suite=args.suite, model=args.model, results=path)+ out = os.path.join(LOG_DIR, f"noise-{args.suite}-{args.harness}-{args.model}.json")+ with open(out, "w") as f:+ json.dump(summary, f, indent=1)+ f.write("\n")+ tr = summary["sets"]["train"]+ print(f"train pass {tr['pass']['mean']:.3f} (rep sd {tr['pass']['rep_sd']:.3f}, band {tr['pass']['band']:.3f});"+ f" flaky: {', '.join(tr['flaky_tasks']) or 'none'}")+ if summary["headroom_warning"]:+ print(f"WARNING: baseline is at or above {HEADROOM:.0%} on train; improvements will be hard to see")+ print(f"-> {out}")+++def cmd_round(args) -> None:+ split = load_split(args.suite)+ noise_file = os.path.join(LOG_DIR, f"noise-{args.suite}-{args.champion}-{args.model}.json")+ min_effect = args.min_effect+ reps = args.reps+ if min_effect is None:+ if not os.path.exists(noise_file):+ raise SystemExit(f"no noise measurement ({noise_file}); run hillclimb.py noise first or pass --min-effect")+ with open(noise_file) as f:+ noise = json.load(f)+ min_effect = noise["sets"]["train"][args.goal]["band"]+ reps = reps or noise["reps"]+ reps = reps or 3+ path = run_eval(f"{args.champion},{args.candidate}", args.model, args.suite,+ split["train"] + split["test"], reps, args.jobs)+ rows = load_jsonl(path)+ pick = lambda h, s: [r for r in rows if r.get("harness") == h and r.get("task") in split[s]]+ scores = {h: {s: metric(pick(h, s), args.goal) for s in ("train", "test")} for h in (args.champion, args.candidate)}+ keep, why = judge(scores[args.champion]["train"], scores[args.candidate]["train"],+ scores[args.champion]["test"], scores[args.candidate]["test"], min_effect, args.goal)+ guard = metric(pick(args.candidate, "train"), "pass") < metric(pick(args.champion, "train"), "pass") - min_effect+ if keep and args.goal != "pass" and guard:+ keep, why = False, "pass rate dropped while chasing " + args.goal+ print(f"{args.goal}: champion train {scores[args.champion]['train']:.4f} test {scores[args.champion]['test']:.4f}"+ f" | candidate train {scores[args.candidate]['train']:.4f} test {scores[args.candidate]['test']:.4f}")+ print(("KEEP " if keep else "REVERT ") + why)+ append_log({"kind": "round", "suite": args.suite, "goal": args.goal, "candidate_id": args.candidate_id or args.candidate,+ "champion": args.champion, "candidate": args.candidate, "model": args.model, "reps": reps,+ "min_effect": min_effect, "scores": scores, "keep": keep, "why": why, "results": path})+ n = stalled(args.suite)+ if n >= STALL_ROUNDS:+ print(f"\nstalled: {n} rounds without a kept change. Analyze what still fails on TRAIN by root cause:")+ for r in pick(args.champion, "train"):+ if not r.get("outcome_ok"):+ print(f" {r['task']} rep {r.get('rep')}: {(r.get('check_note') or r.get('error') or '')[:160]}")+ print("If no single fix could beat the noise band, add repetitions or tasks instead of more rounds.")++def main() -> None:ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)sub = ap.add_subparsers(dest="cmd", required=True)p = sub.add_parser("self-test")- p.set_defaults(fn=lambda _: (price_self_test(), _decide_self_test(), print("hillclimb self-test ok")))+ p.set_defaults(fn=lambda _: (price_self_test(), _decide_self_test(), _heldout_self_test(), print("hillclimb self-test ok")))p = sub.add_parser("score")p.add_argument("jsonl")@@ -267,6 +454,33 @@ def main() -> None:p.add_argument("--only", default="")p.set_defaults(fn=cmd_iterate)+ p = sub.add_parser("split")+ p.add_argument("--suite", required=True)+ p.add_argument("--test-frac", type=float, default=0.33)+ p.add_argument("--seed", default="graff")+ p.add_argument("--force", action="store_true")+ p.set_defaults(fn=cmd_split)++ p = sub.add_parser("noise")+ p.add_argument("--suite", required=True)+ p.add_argument("--harness", default=OURS_DEFAULT)+ p.add_argument("--model", default="grok-4.6")+ p.add_argument("--reps", type=int, default=3)+ p.add_argument("--jobs", "-j", type=int, default=1)+ p.set_defaults(fn=cmd_noise)++ p = sub.add_parser("round")+ p.add_argument("--suite", required=True)+ p.add_argument("--champion", default=OURS_DEFAULT)+ p.add_argument("--candidate", required=True)+ p.add_argument("--candidate-id", default="")+ p.add_argument("--model", default="grok-4.6")+ p.add_argument("--goal", choices=sorted(GOALS), default="pass")+ p.add_argument("--reps", type=int, default=None)+ p.add_argument("--min-effect", type=float, default=None)+ p.add_argument("--jobs", "-j", type=int, default=1)+ p.set_defaults(fn=cmd_round)+args = ap.parse_args()if args.cmd == "decide" and not args.candidate and not args.same_file:ap.error("decide needs --candidate or --same-file")@@ -291,5 +505,31 @@ def _decide_self_test() -> None:assert not catalog["keep"]++def _heldout_self_test() -> None:+ ids = [f"t{i}" for i in range(12)]+ a, b = make_split(ids, 0.33, "graff"), make_split(list(reversed(ids)), 0.33, "graff")+ assert a == b, "split must not depend on listing order"+ assert not set(a["train"]) & set(a["test"]) and sorted(a["train"] + a["test"]) == sorted(ids)+ assert len(a["test"]) == 4+ assert make_split(ids, 0.33, "other") != a+ rows = [{"harness": "h", "task": t, "rep": rep, "outcome_ok": not (t == "t0" and rep == 1)}+ for rep in range(3) for t in ids]+ split = {"train": [t for t in ids if t != "t11"], "test": ["t11"]}+ n = noise_summary(rows, "h", split, 3)+ assert n["sets"]["train"]["pass"]["rep_sd"] > 0 and n["sets"]["test"]["pass"]["rep_sd"] == 0+ assert n["sets"]["train"]["flaky_tasks"] == ["t0"] and n["headroom_warning"]+ keep, why = judge(0.60, 0.80, 0.50, 0.60, 0.05)+ assert keep, why+ keep, why = judge(0.60, 0.80, 0.50, 0.50, 0.05)+ assert not keep and why.startswith("overfit"), why+ keep, why = judge(0.60, 0.62, 0.50, 0.60, 0.05)+ assert not keep and why.startswith("within noise"), why+ keep, why = judge(0.60, 0.70, 0.50, 0.40, 0.05)+ assert not keep and why.startswith("regressed"), why+ keep, why = judge(0.050, 0.030, 0.050, 0.040, 0.005, goal="usd")+ assert keep, why++if __name__ == "__main__":main()diff --git a/graff-evals/hillclimb/candidates.json b/graff-evals/hillclimb/candidates.jsonindex 4975407e..4a082101 100644--- a/graff-evals/hillclimb/candidates.json+++ b/graff-evals/hillclimb/candidates.json@@ -37,6 +37,34 @@"kept_on": "2026-08-30","why": "autoInitLearning detaches graff learn init (same process-group spawn as a trial). Session-end no longer waits ~38s on suite generation. Pin stays a hardlink. -p / REPL / TUI share the path.","forbidden": []+ },+ {+ "id": "g6-followthrough",+ "harness": "graff-dev-g6-followthrough",+ "kept": false,+ "why": "GPT-6 prompting guide recommendation. Judge with a held-out round on gpt-6 models before a prompt change ships.",+ "forbidden": []+ },+ {+ "id": "g6-skill-precedence",+ "harness": "graff-dev-g6-skill-precedence",+ "kept": false,+ "why": "GPT-6 prompting guide recommendation. Judge with a held-out round on gpt-6 models before a prompt change ships.",+ "forbidden": []+ },+ {+ "id": "g6-test-calibration",+ "harness": "graff-dev-g6-test-calibration",+ "kept": false,+ "why": "GPT-6 prompting guide recommendation. Judge with a held-out round on gpt-6 models before a prompt change ships.",+ "forbidden": []+ },+ {+ "id": "g6-plain-prose",+ "harness": "graff-dev-g6-plain-prose",+ "kept": false,+ "why": "GPT-6 prompting guide recommendation. Judge with a held-out round on gpt-6 models before a prompt change ships.",+ "forbidden": []}]}diff --git a/graff-evals/live_setup.py b/graff-evals/live_setup.pyindex d690569a..4f410354 100644--- a/graff-evals/live_setup.py+++ b/graff-evals/live_setup.py@@ -9,6 +9,7 @@ from __future__ import annotationsimport jsonimport os+import reimport shutilimport subprocessimport sys@@ -38,7 +39,10 @@ def load_catalog():def manifest(task_id):cat = load_catalog()- for t in cat["published"] + cat.get("reserve", []) + cat.get("dropped", []):+ mined_path = os.path.join(EVALS, "live", "mined.json") # ADR 0214, machine-managed+ mined = json.load(open(mined_path)) if os.path.exists(mined_path) else {}+ for t in (cat["published"] + cat.get("reserve", []) + cat.get("dropped", [])+ + mined.get("mined", []) + mined.get("candidates", [])):if t["id"] == task_id:return traise SystemExit(f"unknown live task: {task_id}")@@ -153,6 +157,41 @@ def clone_historical_parent(task, dest):subprocess.check_call([sys.executable, pin, dest])+def archive_local(rev, dest, paths=None):+ """A historical tree straight from this checkout's git history (ADR 0214:+ mined tasks). Recent parents build on the pinned toolchain as they are."""+ os.makedirs(dest, exist_ok=True)+ wanted = list(paths or DEFAULT_SPARSE)+ # Whatever that revision's build.zig embeds or builds from (for example a+ # stylesheet under apps/ before the desktop app left) must come along.+ build = subprocess.run(["git", "-C", REPO, "show", f"{rev}:build.zig"],+ capture_output=True, text=True).stdout+ wanted += [p for p in re.findall(r'b\.path\("([^"]+)"\)', build) if p not in wanted]+ have = [p for p in wanted+ if subprocess.run(["git", "-C", REPO, "cat-file", "-e", f"{rev}:{p}"],+ capture_output=True).returncode == 0]+ arch = subprocess.Popen(["git", "-C", REPO, "archive", rev, *have], stdout=subprocess.PIPE)+ subprocess.check_call(["tar", "-x", "-C", dest], stdin=arch.stdout)+ if arch.wait() != 0:+ raise SystemExit(f"git archive {rev} failed")+ # Fetched dependencies live in the untracked zig-pkg/ cache. Copy it (it+ # is small) so a sandbox build does not refetch every package, and an+ # agent working in the sandbox cannot write into this checkout's copy.+ pkg = os.path.join(REPO, "zig-pkg")+ if os.path.isdir(pkg) and not os.path.exists(os.path.join(dest, "zig-pkg")):+ shutil.copytree(pkg, os.path.join(dest, "zig-pkg"), symlinks=True)+++def append_case(sandbox, rel, inc, name):+ """Append a test block unless a test with that name is already there."""+ path = os.path.join(sandbox, rel)+ text = open(path).read() if os.path.exists(path) else ""+ if f'test "{name}"' in text:+ return+ with open(path, "a") as f:+ f.write(open(inc).read())++def setup(task_id, sandbox, allow_green=False):task = manifest(task_id)os.makedirs(sandbox, exist_ok=True)@@ -167,6 +206,10 @@ def setup(task_id, sandbox, allow_green=False):os.remove(path)if task.get("source") == "clone":clone_historical_parent(task, sandbox)+ elif task.get("source") == "local-git":+ archive_local(task["historical_parent"], sandbox, task.get("sparse"))+ append_case(sandbox, task["public_file"],+ os.path.join(EVALS, "live", task["id"], "public_case.inc"), task["public_filter"])else:copy_local_package(sandbox, task.get("sparse") or DEFAULT_SPARSE)apply_parent(task, sandbox)diff --git a/graff-evals/named_unit_check.py b/graff-evals/named_unit_check.pynew file mode 100644index 00000000..f8773de0--- /dev/null+++ b/graff-evals/named_unit_check.py@@ -0,0 +1,81 @@+#!/usr/bin/env python3+"""Grade named Zig unit tests by running them (ADR 0214, mined tasks).++Builds only the unit-test binary, filtered to the named tests+(`zig build test-bin -Dtest-filter=…`), runs it, and reads each test's own+result line (`N/M module.test.<name>...OK`). Every name must have run and+passed. A name that never ran fails: a test the module graph does not reach+proves nothing. Integration scripts are not run, so an unrelated flaky+script cannot fail the grade, and a compile error is reported as such.++ named_unit_check.py NAME [NAME ...] # exit 0 iff all ran and passed+"""+from __future__ import annotations++import glob+import os+import re+import shutil+import subprocess+import sys+import tempfile+++def parse(output: str, name: str) -> str:+ """pass / fail / missing for one name in a test binary's output."""+ ran = re.compile(r"\.test\." + re.escape(name) + r"\.\.\.(\S+)")+ verdicts = [m.group(1) for m in ran.finditer(output)]+ broke = re.compile(r"error: '[^'\n]*\.test\." + re.escape(name) + r"' (?:failed|terminated)")+ if broke.search(output) or any(v.startswith(("FAIL", "SKIP")) for v in verdicts):+ return "fail"+ return "pass" if any(v.startswith("OK") for v in verdicts) else "missing"+++def grade(cwd: str, names: list[str], timeout: int = 600) -> dict[str, str]:+ # A private cache dir: the only test binary in it is the one just built+ # for exactly these filters (a shared cache can hand back another run's).+ cache = tempfile.mkdtemp(prefix="named-unit-check-")+ try:+ build = subprocess.run(["zig", "build", "test-bin", "--cache-dir", cache,+ *[f"-Dtest-filter={n}" for n in names]],+ cwd=cwd, capture_output=True, text=True, timeout=timeout)+ found = glob.glob(os.path.join(cache, "o", "*", "test"))+ if build.returncode != 0 or not found:+ return {n: "compile-error" for n in names}+ try:+ run = subprocess.run([max(found, key=os.path.getmtime)], cwd=cwd,+ capture_output=True, text=True, timeout=timeout)+ except subprocess.TimeoutExpired:+ return {n: "fail" for n in names}+ out = (run.stdout or "") + "\n" + (run.stderr or "")+ return {n: parse(out, n) for n in names}+ finally:+ shutil.rmtree(cache, ignore_errors=True)+++def self_test() -> None:+ out = ("3/5 acp_elicit.test.decline, cancel (#1322)...OK\n"+ "4/5 ask_user.test.noAnswerText keeps cancel...FAIL (TestExpectedEqual)\n"+ "5/5 other.test_0...OK\n")+ assert parse(out, "decline, cancel (#1322)") == "pass"+ assert parse(out, "noAnswerText keeps cancel") == "fail"+ assert parse(out, "never compiled in") == "missing"+ crash = "error: 'x.test.boom' terminated with signal ABRT with stderr:\n"+ assert parse(crash, "boom") == "fail"+ print("named_unit_check self-test ok")+++def main() -> None:+ if sys.argv[1:] == ["self-test"]:+ return self_test()+ names = sys.argv[1:]+ if not names:+ raise SystemExit("usage: named_unit_check.py NAME [NAME ...]")+ verdicts = grade(".", names)+ for n, v in verdicts.items():+ print(f"{v:14} {n}")+ raise SystemExit(0 if all(v == "pass" for v in verdicts.values()) else 1)+++if __name__ == "__main__":+ main()