ππΌββοΈ The blackboard for coding agents - agentic development environment for claude code, codex, cursor, opencode, grok and more.
Rust adeai-agentsclaudeclaude-codecodexcursorgeminigrokopencodespec-driven-developmentvibe-codingvim
feat(benchmark): add --dataset flag and transcript capture docs (#126)
Fynn Fluegge9580c47c51ba
diff --git a/benchmark/README.md b/benchmark/README.mdindex 448a695..263182f 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -299,6 +299,111 @@ cat swebench_output/*/results.json | \---+### Capturing the agent's session transcript++The `predictions.jsonl` / `results.json` only capture the **git diff and metrics** β not what the+agent actually said or did. To inspect the full turn-by-turn conversation (tool calls, reasoning,+and β importantly β whether an injected skill/plugin actually influenced behavior), read the Claude+Code **session transcript**.++**Where Claude Code writes transcripts.** For every session, Claude Code appends a JSONL file:++```+~/.claude/projects/<encoded-cwd>/<session-id>.jsonl+```++`<encoded-cwd>` is the working directory with every `/` and `.` replaced by `-`. Each line is one+event (`type` = `user` / `assistant` / β¦) carrying the full message content, tool uses, and a `cwd`+field. On the host you can find your own sessions there; the same layout exists **inside the+container** for the agent Claude Code runs.++**In sandbox mode the transcript lives inside the container and is destroyed when the container+stops.** The agent runs as user `bench` with `HOME=/home/bench`, working in `/testbed` (when+`skip_worktree = true`) or in its worktree under `/testbed/.agtx/worktrees/β¦`. So the transcript is at:++```+# skip_worktree = true (agent works in /testbed)+/home/bench/.claude/projects/-testbed/<session-id>.jsonl+```++Containers are removed automatically at the end of each task, so you **must copy the transcript out+while the container is still alive**. A single `docker cp` "before the run finishes" is unreliable β+the container often stops before you catch it. Instead, run a **polling loop that re-copies every few+seconds** in a separate shell (start it right after launching the benchmark):++```bash+C=swebench-astropy-astropy-12907+DEST=/tmp/transcript_capture++# Poll while the container is alive, re-copying the projects tree each pass.+for i in $(seq 1 180); do+ docker ps --filter name=$C -q | grep -q . || { echo "container gone"; break; }+ docker cp $C:/home/bench/.claude/projects "$DEST" 2>/dev/null+ sleep 5+done++# List what landed+find "$DEST" -name '*.jsonl'+```++> **`skip_worktree = true` β one session per phase.** Each agtx phase (Planning / Running / Review)+> is a **separate** Claude Code session, so you get one `<session-id>.jsonl` per phase, not one file+> for the whole task. `void` runs as a single session.++#### Analysing the transcript (do NOT use a naive grep)++β οΈ **A raw `grep` across the JSONL gives false positives.** SessionStart-hook plugins (focus,+ponytail, caveman) inject their entire `SKILL.md` into the transcript as an `attachment` event β and+that injected text *documents* the very tags you're grepping for. So+`grep -c 'context-commit' *.jsonl` counts the **injected instructions**, not what the model actually+emitted. In one run a raw grep reported 34 CSP-tag hits while the model emitted **zero** β all 34+were inside the injected skill.++To measure real behavior you must parse the JSONL, keep only `type == "assistant"` events, and count+tags in their `text` content blocks:++```bash+python3 - "$DEST"/**/-testbed/*.jsonl << 'PY'+import json, sys+TAGS = ["context-commit","context-scope","context-checkpoint","context-transient","context-revoke"]+for path in sys.argv[1:]:+ injected = emitted = turns = 0+ counts = {}+ for line in open(path):+ try: d = json.loads(line)+ except Exception: continue+ raw = json.dumps(d)+ # SessionStart-injected SKILL.md lands as an attachment β exclude it+ if d.get("type") == "attachment" and "Always-on context discipline" in raw:+ injected += 1+ if d.get("type") != "assistant":+ continue+ turns += 1+ content = d.get("message", {}).get("content", [])+ text = "".join(c.get("text","") for c in content+ if isinstance(c, dict) and c.get("type") == "text") \+ if isinstance(content, list) else (content if isinstance(content, str) else "")+ for t in TAGS:+ n = text.count("<" + t)+ if n: counts[t] = counts.get(t, 0) + n+ print(f"{path}")+ print(f" SKILL.md injected (attachment): {injected}")+ print(f" assistant turns: {turns}")+ print(f" tags EMITTED BY MODEL: {counts or 'NONE'}")+PY+```++> **Note:** the CSP tags are HTML-style (`<context-commit>`) so the Claude Code TUI **hides them in+> the rendered pane** β `tmux capture-pane` will never show them. You must read the JSONL to see+> whether the model actually emitted them.++> **`void` quirk:** with the `void` plugin the phase prompt is *pasted* into the input box but not+> always submitted (no artifact/finish-marker to drive it), which shows as "Pane stable but no finish+> marker". If a run stalls there, submit it manually:+> `docker exec $C tmux -L agtx send-keys -t testbed:1 Enter`.++---+### Cleanup#### Non-sandbox cleanupdiff --git a/benchmark/skills/benchmark/SKILL.md b/benchmark/skills/benchmark/SKILL.mdindex 4fd76b7..0b00540 100644--- a/benchmark/skills/benchmark/SKILL.md+++ b/benchmark/skills/benchmark/SKILL.md@@ -233,6 +233,62 @@ cat swebench_output/*/results.json | \---+## Capturing the Agent Session Transcript++`predictions.jsonl` / `results.json` capture only the git diff and metrics β NOT what the agent said or did. To verify *behavior* (e.g. whether an injected skill/plugin actually changed the agent's output), read the Claude Code session transcript.++**Where Claude Code writes it:** `~/.claude/projects/{encoded-cwd}/{session-id}.jsonl`, where `{encoded-cwd}` is the working directory with every `/` and `.` replaced by `-`. Each line is one event (`type` = `user`/`assistant`/β¦) with full message content, tool uses, and a `cwd` field.++**In sandbox mode the transcript lives inside the container** (agent runs as `bench`, `HOME=/home/bench`, cwd `/testbed` when `skip_worktree = true`) and is **destroyed when the container stops**:+```+/home/bench/.claude/projects/-testbed/{session-id}.jsonl+```++Containers are removed at task end, and a single `docker cp` "before the run finishes" is unreliable β the container often stops first. Run a **polling loop that re-copies every few seconds** while the container is alive (start it right after launching the benchmark):+```bash+C=swebench-astropy-astropy-12907+DEST=/tmp/transcript_capture+for i in $(seq 1 180); do+ docker ps --filter name=$C -q | grep -q . || { echo "container gone"; break; }+ docker cp $C:/home/bench/.claude/projects "$DEST" 2>/dev/null+ sleep 5+done+find "$DEST" -name '*.jsonl'+```++With `skip_worktree = true`, each agtx phase (Planning / Running / Review) is a **separate** Claude Code session β one `.jsonl` per phase. `void` runs as a single session.++**Analysing the transcript β do NOT use a naive grep.** SessionStart-hook plugins (focus, ponytail, caveman) inject their whole `SKILL.md` into the transcript as an `attachment` event, and that injected text *documents* the very tags you'd grep for. So `grep -c 'context-commit' *.jsonl` counts the **injected instructions**, not model output. In one run a raw grep reported 34 CSP-tag hits while the model emitted **zero** β all 34 were inside the injected skill. Parse the JSONL, keep only `type == "assistant"` events, exclude the injected `attachment`, and count tags in the `text` blocks:+```bash+python3 - "$DEST"/**/-testbed/*.jsonl << 'PY'+import json, sys+TAGS = ["context-commit","context-scope","context-checkpoint","context-transient","context-revoke"]+for path in sys.argv[1:]:+ injected = turns = 0; counts = {}+ for line in open(path):+ try: d = json.loads(line)+ except Exception: continue+ if d.get("type") == "attachment" and "Always-on context discipline" in json.dumps(d):+ injected += 1+ if d.get("type") != "assistant": continue+ turns += 1+ content = d.get("message", {}).get("content", [])+ text = "".join(c.get("text","") for c in content+ if isinstance(c, dict) and c.get("type") == "text") \+ if isinstance(content, list) else (content if isinstance(content, str) else "")+ for t in TAGS:+ n = text.count("<" + t)+ if n: counts[t] = counts.get(t, 0) + n+ print(f"{path}\n injected SKILL.md: {injected} assistant turns: {turns} tags EMITTED BY MODEL: {counts or 'NONE'}")+PY+```++**Gotchas:**+- CSP tags are HTML-style (`<context-commit>`) β the Claude Code TUI **hides them in the rendered pane**; `tmux capture-pane` never shows them. Read the JSONL.+- **`void` quirk:** the phase prompt is *pasted* but not always submitted (no artifact/finish-marker), showing as "Pane stable but no finish marker". Submit manually: `docker exec $C tmux -L agtx send-keys -t testbed:1 Enter`.++---+## EvaluationAfter the run, copy-paste the printed commands or run manually:diff --git a/benchmark/swebench/benchmark.py b/benchmark/swebench/benchmark.pyindex 1a1dd0d..9f14d43 100755--- a/benchmark/swebench/benchmark.py+++ b/benchmark/swebench/benchmark.py@@ -1714,9 +1714,9 @@ class BenchmarkOrchestrator:# Dataset loader# ----------------------------------------------------------------------------def load_swebench(split: str = "test") -> list[dict]:- print(f"Loading SWE-bench Lite ({split} split)...")- dataset = load_dataset("princeton-nlp/SWE-bench_Lite", split=split)+def load_swebench(split: str = "test", dataset_name: str = "princeton-nlp/SWE-bench_Lite") -> list[dict]:+ print(f"Loading {dataset_name} ({split} split)...")+ dataset = load_dataset(dataset_name, split=split)return list(dataset)@@ -1798,6 +1798,13 @@ other agtx project settings. Example:default="test",help="HuggingFace dataset split (default: test)",)+ parser.add_argument(+ "--dataset",+ default="princeton-nlp/SWE-bench_Lite",+ dest="dataset",+ help="HuggingFace dataset name (default: princeton-nlp/SWE-bench_Lite). "+ "Use princeton-nlp/SWE-bench_Verified for multi-file/long-exploration tasks.",+ )parser.add_argument("--verbose", "-v",action="store_true",@@ -1871,7 +1878,7 @@ other agtx project settings. Example:if args.smoke_test and args.instance_ids is None and args.instances is None:instances = [{"instance_id": "smoke-test", "repo": "", "base_commit": "HEAD", "problem_statement": ""}]else:- instances = load_swebench(args.split)+ instances = load_swebench(args.split, args.dataset)if args.instance_ids:id_set = set(args.instance_ids)instances = [i for i in instances if i["instance_id"] in id_set]@@ -1907,7 +1914,7 @@ other agtx project settings. Example:run_id = f"{config_path.stem}-{int(time.time())}"print("\nTo evaluate:")print(f" uv run python -m swebench.harness.run_evaluation \\")- print(f" --dataset_name princeton-nlp/SWE-bench_Lite \\")+ print(f" --dataset_name {args.dataset} \\")print(f" --predictions_path {orchestrator.store.predictions_path} \\")print(f" --run_id {run_id}")print("\nTo report (after evaluation):")