Skip to content

armgit / agtx

Public Imported from GitHub
0

πŸ„πŸΌβ€β™‚οΈ The blackboard for coding agents - agentic development environment for claude code, codex, cursor, opencode, grok and more.

feat(benchmark): add --dataset flag and transcript capture docs (#126)

Fynn Fluegge9580c47c51ba
diff --git a/benchmark/README.md b/benchmark/README.md
index 448a695..263182f 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -299,6 +299,111 @@ cat swebench_output/*/results.json | \
---
+### Capturing the agent's session transcript
+
+The `predictions.jsonl` / `results.json` only capture the **git diff and metrics** β€” not what the
+agent actually said or did. To inspect the full turn-by-turn conversation (tool calls, reasoning,
+and β€” importantly β€” whether an injected skill/plugin actually influenced behavior), read the Claude
+Code **session transcript**.
+
+**Where Claude Code writes transcripts.** For every session, Claude Code appends a JSONL file:
+
+```
+~/.claude/projects/<encoded-cwd>/<session-id>.jsonl
+```
+
+`<encoded-cwd>` is the working directory with every `/` and `.` replaced by `-`. Each line is one
+event (`type` = `user` / `assistant` / …) carrying the full message content, tool uses, and a `cwd`
+field. On the host you can find your own sessions there; the same layout exists **inside the
+container** for the agent Claude Code runs.
+
+**In sandbox mode the transcript lives inside the container and is destroyed when the container
+stops.** The agent runs as user `bench` with `HOME=/home/bench`, working in `/testbed` (when
+`skip_worktree = true`) or in its worktree under `/testbed/.agtx/worktrees/…`. So the transcript is at:
+
+```
+# skip_worktree = true (agent works in /testbed)
+/home/bench/.claude/projects/-testbed/<session-id>.jsonl
+```
+
+Containers are removed automatically at the end of each task, so you **must copy the transcript out
+while the container is still alive**. A single `docker cp` "before the run finishes" is unreliable β€”
+the container often stops before you catch it. Instead, run a **polling loop that re-copies every few
+seconds** in a separate shell (start it right after launching the benchmark):
+
+```bash
+C=swebench-astropy-astropy-12907
+DEST=/tmp/transcript_capture
+
+# Poll while the container is alive, re-copying the projects tree each pass.
+for i in $(seq 1 180); do
+ docker ps --filter name=$C -q | grep -q . || { echo "container gone"; break; }
+ docker cp $C:/home/bench/.claude/projects "$DEST" 2>/dev/null
+ sleep 5
+done
+
+# List what landed
+find "$DEST" -name '*.jsonl'
+```
+
+> **`skip_worktree = true` β†’ one session per phase.** Each agtx phase (Planning / Running / Review)
+> is a **separate** Claude Code session, so you get one `<session-id>.jsonl` per phase, not one file
+> for the whole task. `void` runs as a single session.
+
+#### Analysing the transcript (do NOT use a naive grep)
+
+⚠️ **A raw `grep` across the JSONL gives false positives.** SessionStart-hook plugins (focus,
+ponytail, caveman) inject their entire `SKILL.md` into the transcript as an `attachment` event β€” and
+that injected text *documents* the very tags you're grepping for. So
+`grep -c 'context-commit' *.jsonl` counts the **injected instructions**, not what the model actually
+emitted. In one run a raw grep reported 34 CSP-tag hits while the model emitted **zero** β€” all 34
+were inside the injected skill.
+
+To measure real behavior you must parse the JSONL, keep only `type == "assistant"` events, and count
+tags in their `text` content blocks:
+
+```bash
+python3 - "$DEST"/**/-testbed/*.jsonl << 'PY'
+import json, sys
+TAGS = ["context-commit","context-scope","context-checkpoint","context-transient","context-revoke"]
+for path in sys.argv[1:]:
+ injected = emitted = turns = 0
+ counts = {}
+ for line in open(path):
+ try: d = json.loads(line)
+ except Exception: continue
+ raw = json.dumps(d)
+ # SessionStart-injected SKILL.md lands as an attachment β€” exclude it
+ if d.get("type") == "attachment" and "Always-on context discipline" in raw:
+ injected += 1
+ if d.get("type") != "assistant":
+ continue
+ turns += 1
+ content = d.get("message", {}).get("content", [])
+ text = "".join(c.get("text","") for c in content
+ if isinstance(c, dict) and c.get("type") == "text") \
+ if isinstance(content, list) else (content if isinstance(content, str) else "")
+ for t in TAGS:
+ n = text.count("<" + t)
+ if n: counts[t] = counts.get(t, 0) + n
+ print(f"{path}")
+ print(f" SKILL.md injected (attachment): {injected}")
+ print(f" assistant turns: {turns}")
+ print(f" tags EMITTED BY MODEL: {counts or 'NONE'}")
+PY
+```
+
+> **Note:** the CSP tags are HTML-style (`<context-commit>`) so the Claude Code TUI **hides them in
+> the rendered pane** β€” `tmux capture-pane` will never show them. You must read the JSONL to see
+> whether the model actually emitted them.
+
+> **`void` quirk:** with the `void` plugin the phase prompt is *pasted* into the input box but not
+> always submitted (no artifact/finish-marker to drive it), which shows as "Pane stable but no finish
+> marker". If a run stalls there, submit it manually:
+> `docker exec $C tmux -L agtx send-keys -t testbed:1 Enter`.
+
+---
+
### Cleanup
#### Non-sandbox cleanup
diff --git a/benchmark/skills/benchmark/SKILL.md b/benchmark/skills/benchmark/SKILL.md
index 4fd76b7..0b00540 100644
--- a/benchmark/skills/benchmark/SKILL.md
+++ b/benchmark/skills/benchmark/SKILL.md
@@ -233,6 +233,62 @@ cat swebench_output/*/results.json | \
---
+## Capturing the Agent Session Transcript
+
+`predictions.jsonl` / `results.json` capture only the git diff and metrics β€” NOT what the agent said or did. To verify *behavior* (e.g. whether an injected skill/plugin actually changed the agent's output), read the Claude Code session transcript.
+
+**Where Claude Code writes it:** `~/.claude/projects/{encoded-cwd}/{session-id}.jsonl`, where `{encoded-cwd}` is the working directory with every `/` and `.` replaced by `-`. Each line is one event (`type` = `user`/`assistant`/…) with full message content, tool uses, and a `cwd` field.
+
+**In sandbox mode the transcript lives inside the container** (agent runs as `bench`, `HOME=/home/bench`, cwd `/testbed` when `skip_worktree = true`) and is **destroyed when the container stops**:
+```
+/home/bench/.claude/projects/-testbed/{session-id}.jsonl
+```
+
+Containers are removed at task end, and a single `docker cp` "before the run finishes" is unreliable β€” the container often stops first. Run a **polling loop that re-copies every few seconds** while the container is alive (start it right after launching the benchmark):
+```bash
+C=swebench-astropy-astropy-12907
+DEST=/tmp/transcript_capture
+for i in $(seq 1 180); do
+ docker ps --filter name=$C -q | grep -q . || { echo "container gone"; break; }
+ docker cp $C:/home/bench/.claude/projects "$DEST" 2>/dev/null
+ sleep 5
+done
+find "$DEST" -name '*.jsonl'
+```
+
+With `skip_worktree = true`, each agtx phase (Planning / Running / Review) is a **separate** Claude Code session β†’ one `.jsonl` per phase. `void` runs as a single session.
+
+**Analysing the transcript β€” do NOT use a naive grep.** SessionStart-hook plugins (focus, ponytail, caveman) inject their whole `SKILL.md` into the transcript as an `attachment` event, and that injected text *documents* the very tags you'd grep for. So `grep -c 'context-commit' *.jsonl` counts the **injected instructions**, not model output. In one run a raw grep reported 34 CSP-tag hits while the model emitted **zero** β€” all 34 were inside the injected skill. Parse the JSONL, keep only `type == "assistant"` events, exclude the injected `attachment`, and count tags in the `text` blocks:
+```bash
+python3 - "$DEST"/**/-testbed/*.jsonl << 'PY'
+import json, sys
+TAGS = ["context-commit","context-scope","context-checkpoint","context-transient","context-revoke"]
+for path in sys.argv[1:]:
+ injected = turns = 0; counts = {}
+ for line in open(path):
+ try: d = json.loads(line)
+ except Exception: continue
+ if d.get("type") == "attachment" and "Always-on context discipline" in json.dumps(d):
+ injected += 1
+ if d.get("type") != "assistant": continue
+ turns += 1
+ content = d.get("message", {}).get("content", [])
+ text = "".join(c.get("text","") for c in content
+ if isinstance(c, dict) and c.get("type") == "text") \
+ if isinstance(content, list) else (content if isinstance(content, str) else "")
+ for t in TAGS:
+ n = text.count("<" + t)
+ if n: counts[t] = counts.get(t, 0) + n
+ print(f"{path}\n injected SKILL.md: {injected} assistant turns: {turns} tags EMITTED BY MODEL: {counts or 'NONE'}")
+PY
+```
+
+**Gotchas:**
+- CSP tags are HTML-style (`<context-commit>`) β†’ the Claude Code TUI **hides them in the rendered pane**; `tmux capture-pane` never shows them. Read the JSONL.
+- **`void` quirk:** the phase prompt is *pasted* but not always submitted (no artifact/finish-marker), showing as "Pane stable but no finish marker". Submit manually: `docker exec $C tmux -L agtx send-keys -t testbed:1 Enter`.
+
+---
+
## Evaluation
After the run, copy-paste the printed commands or run manually:
diff --git a/benchmark/swebench/benchmark.py b/benchmark/swebench/benchmark.py
index 1a1dd0d..9f14d43 100755
--- a/benchmark/swebench/benchmark.py
+++ b/benchmark/swebench/benchmark.py
@@ -1714,9 +1714,9 @@ class BenchmarkOrchestrator:
# Dataset loader
# ---------------------------------------------------------------------------
-def load_swebench(split: str = "test") -> list[dict]:
- print(f"Loading SWE-bench Lite ({split} split)...")
- dataset = load_dataset("princeton-nlp/SWE-bench_Lite", split=split)
+def load_swebench(split: str = "test", dataset_name: str = "princeton-nlp/SWE-bench_Lite") -> list[dict]:
+ print(f"Loading {dataset_name} ({split} split)...")
+ dataset = load_dataset(dataset_name, split=split)
return list(dataset)
@@ -1798,6 +1798,13 @@ other agtx project settings. Example:
default="test",
help="HuggingFace dataset split (default: test)",
)
+ parser.add_argument(
+ "--dataset",
+ default="princeton-nlp/SWE-bench_Lite",
+ dest="dataset",
+ help="HuggingFace dataset name (default: princeton-nlp/SWE-bench_Lite). "
+ "Use princeton-nlp/SWE-bench_Verified for multi-file/long-exploration tasks.",
+ )
parser.add_argument(
"--verbose", "-v",
action="store_true",
@@ -1871,7 +1878,7 @@ other agtx project settings. Example:
if args.smoke_test and args.instance_ids is None and args.instances is None:
instances = [{"instance_id": "smoke-test", "repo": "", "base_commit": "HEAD", "problem_statement": ""}]
else:
- instances = load_swebench(args.split)
+ instances = load_swebench(args.split, args.dataset)
if args.instance_ids:
id_set = set(args.instance_ids)
instances = [i for i in instances if i["instance_id"] in id_set]
@@ -1907,7 +1914,7 @@ other agtx project settings. Example:
run_id = f"{config_path.stem}-{int(time.time())}"
print("\nTo evaluate:")
print(f" uv run python -m swebench.harness.run_evaluation \\")
- print(f" --dataset_name princeton-nlp/SWE-bench_Lite \\")
+ print(f" --dataset_name {args.dataset} \\")
print(f" --predictions_path {orchestrator.store.predictions_path} \\")
print(f" --run_id {run_id}")
print("\nTo report (after evaluation):")